TLDR Web Dev 2024-05-15

This author successfully replaced OpenAI's API with an open-source alternative to reduce the cost of running large-scale AI applications. They tried using Ollama on a local machine to generate text summaries, but limitations with concurrent processing led to them using vLLM (a fast inference runner). To handle large volumes of requests, the author used a Kubernetes cluster to deploy and load balance vLLM.

Caches need to be maintained and updated. The process of cache invalidation can be challenging due to factors like thundering herds (a sudden surge of requests) and cache inconsistency (different parts of the data updating at different times). To handle cache invalidation properly, it's important to prioritize using a robust invalidation protocol and manage data dependencies.

The Asynchronous State Handler pattern improves UX in web apps by decoupling data fetching from the UI. It wraps asynchronous queries with meta-queries that track the status (loading, success, error) of the data fetching process. This allows the UI to react dynamically to these states, displaying loading indicators or error messages when needed. The pattern can be implemented in React using a custom hook.

Communicating in clear deltas (i.e., quantifying before and after states) is a great way to get your point across effectively. For example, in performance reviews, rather than vague statements like "improved performance," specify the percentage reduction in page load time and its impact on key metrics like user engagement. Convincing others to adopt your ideas is easier when you quantify the current problem and the expected improvement from your proposed solution.

Over a 12-year career at Google, Addy Osmani learned to put users first, collaborate well by actively sharing knowledge, and embrace lifelong learning. He encourages readers to just get started and iterate rather than trying to strive for perfection.

Gemini Flash is a new lightweight model from Google that features multimodal reasoning and a long context window of up to one million tokens.

Socket Security protects applications from hidden malware in open source code. It goes beyond traditional scanners to find new threats and integrates with GitHub for developer fixes.

Google introduced upgraded versions of its Gemini AI model, including a public preview of Gemini 1.5 Pro with a 2 million token context window at I/O 2024. For Android developers, the company highlighted Gemini Nano, an on-device AI model for faster local processing. Flutter and Dart also received updates, with Flutter now supporting compilation to WebAssembly and Dart introducing the initial stages of macros.

Google is integrating Gemini into various products, enhancing Search with AI Overviews and introducing Ask Photos in Google Photos for intelligent image searches. It has showcased AI agents that can perform complex tasks like shopping returns or helping users relocate.

Most generative AI projects fail to reach production or deliver significant revenue. Successful GenAI projects prioritize understanding business problems before selecting tools, recognize GenAI is part of a larger tech stack, keep humans involved to guide AI, and rely on trustworthy, traceable data for the best results.

Needle in a Needlestack (NIAN) is a new benchmark that was created to test how well LLMs process information in their context window - GPT-4o has significantly outperformed other models in this test.

URLhaus is a database of malicious URLs that are being used for malware distribution.

Veo is a new video generation AI model from Google Deepmind that can generate 1080p resolution videos that can go beyond a minute long.

Google revealed several new tools and features for web developers at I/O 2024 specifically related to Google Gemini, Web Assembly, and WebGPU, along with enhancements to Chrome DevTools, Angular, and Google Maps.