TLDR Data 2026-06-25
Netflix’s New Batch Compute ⚙️, Fix Data Early 🪟, Finding Topics Automatically 🔍
How Netflix Simplified Batch Compute with Kueue (6 minute read)
Netflix simplified its batch compute platform by replacing its Compute Managed Batch system with Kueue, a Kubernetes-native job queuing and scheduling system. The migration was done by maintaining full API compatibility, converting their tenant hierarchy into Kueue Cohorts + ClusterQueues/LocalQueues, and adding powerful features like preemption-based fair sharing.
Inside One Engineer's Journey to Master Long-Running Agents (16 minute read)
Long-running coding agents work better when they are grounded in structured context and persistent artifacts: repo research, knowledge bases, plans, progress files, verification reports, and review notes. Agentic Orchestrator turns those handoffs into a state-machine workflow, giving agents enough shared “data” to tackle bigger tasks without losing coherence.
Client-Side Load Balancing at a Million Requests Per Second (13 minute read)
Zalando built a high-performance client-side load balancer in-process for their Product Read API to handle over a million requests per second of internal fan-out traffic, replacing the shared edge ingress Skipper for batch requests. They implemented consistent hashing matching Skipper, Kubernetes watch-based discovery, N-ring fade-in for smooth scale-ups, and AZ-aware routing.
Real-Time Personalisation at Scale: How Zepto Understands What You Want, Right Now (9 minute read)
Zepto built a Dual Sequence ReRanker for real-time personalisation that combines a user's long-term history with their current in-session behavior using separate transformer encoders for history and session sequences, target-aware pooling (dynamically rebuilding the user profile per candidate item), a learned fusion gate, and real-time signals (trending counters, calendar context).
Broken Windows of Data (11 minute read)
Data warehouse quality needs to be built into the full development lifecycle, not left to final review. Teams can stop small inconsistencies from spreading into shared definitions, dashboards, and business logic by shifting checks earlier through modeling reviews, local validation, CI/CD, AI-assisted review, human judgment, and monitoring.
Why Technically Excellent Data Teams Still Fail (7 minute read)
Many technically excellent data teams remain irrelevant because strong execution is no longer enough. The real value comes from moving beyond just delivering data to actively driving decisions through clear perspective (opinionated analysis) and action (influencing business outcomes).
Can We Agree on a Storage/Workload Architecture Taxonomy? (9 minute read)
Data storage architectures (OLTP, OLAP, HTAP, and LTAP) can be classified into a taxonomy evaluated by system count, workloads, data visibility, and durable copies. This framework organizes modern data systems into Single Tier, Internal Tiering, Hybrid, Materializing, and Shared Tiering models like Databricks LTAP and Apache Fluss.
Define your metrics once on Databricks (Sponsor)
Most data teams redefine the same metric in dbt, BI, notebooks, and customer apps. Cube's free June 30 webinar shows how to define metrics once in a semantic layer on Databricks, then serve them everywhere - dashboards, embedded apps, AI Analyst. Live demo included.
- Register Free- Add to Calendar
SQL Concepts Lab (Tool)
SQL Concepts Lab is an interactive, browser-based tutorial built with DuckDB-WASM that lets you learn and experiment with core SQL concepts directly in your browser without any setup. It provides hands-on exercises covering fundamental SQL in a live environment, making it a practical way to understand SQL through immediate feedback and experimentation.
dbtrail (GitHub Repo)
dbtrail gives MySQL point-in-time recovery by streaming binlogs and indexing row changes with before-and-after images. Teams inspect changes, create reversal SQL, browse history, or run time-travel queries, with MCP for AI recovery.
SQLBuild (GitHub Repo)
SQLBuild adds change-aware execution to dbt by fingerprinting models and reusing production tables. It stores warehouse state and supports audits, incremental jobs, Python nodes, lineage, contracts, adapters, and DuckDB CI support.
Curated deep dives, tools and trends in big data, data science and data engineering 📊
Join 570,000 readers for
one daily email