๐ง
Research & Innovation
Handling Long Sequences in LLMs (16 minute read)
Linear Attention Sequence Parallel (LASP) introduces a new strategy for efficiently managing long sequences in language models, surpassing traditional methods with its innovative use of linear attention.
Mixture of Depths (23 minute read)
One drawback of modern transformers is that each token uses the same amount of predictive compute. However, some tokens are much easier to predict than others. This work from DeepMind allows models to exit early during generation to spend less flops on certain tokens, effectively opening the door to dynamic compute - with a fixed maximum. The results are 50% fewer flops at generation time for equivalent performance.
๐จโ๐ป
Engineering & Resources
Image Customization with InstantStyle (GitHub Repo)
InstantStyle introduces a new approach to image personalization, overcoming the challenge of style consistency without the need for complex tuning. By cleverly separating style and content in images and focusing on style-specific areas, this framework ensures detailed and consistent visual stylization, blending style intensity with text control seamlessly.
Text to SQL dataset (6 minute read)
23m tokens of text to SQL data is now available on HuggingFace. Gretel has collected a significant dataset to help generate SQL queries based on natural language tasks. This can help in RAG applications and in synthetic data generation.
Image Generation with Two-Phase Inference (GitHub Repo)
TGATE introduces an efficient approach to generating images by dividing the process into planning and refining phases. This method not only simplifies the generation process by fixing certain outputs early on but also surprisingly improves image quality.
How to win at Vertical AI (9 minute read)
The true potential of AI lies in vertical B2B applications and how AI agents and open APIs are pivotal in rebundling and creating new business value. Vertical AI's short-term advantage comes from domain-specific models, while long-term success requires horizontal integration into broader ecosystems. AI agents enable the rebundling of workflows, revolutionizing managerial processes and creating new competitive advantages in various industries.
Where AI Thrives, Religion May Struggle (2 minute read)
A study led by Joshua Conrad Jackson and Adam Waytz suggests that increased exposure to AI and robotics may contribute to a decline in religious beliefs. Countries with more robots saw a greater decrease in religiosity. The study found that people with jobs who were highly exposed to AI were significantly less likely to believe in God. These correlations point to the idea that automation technologies might influence religious decline.
Write OpenAPI with TypeSpec (5 minute read)
TypeSpec, an API definition language developed at Microsoft, offers a more concise and readable way to write OpenAPI compared to JSON or YAML. Drawing from TypeScript's syntax, it addresses OpenAPI's verbosity and lack of reusable components by allowing the definition of API patterns as reusable components, thus simplifying code generation and governance at scale. TypeSpec's flexibility and productivity enhancements could make API-first development practices more appealing.