TLDR Data 2026-08-13
Vercel’s Migration to DynamoDB 🪢, Stripe’s Self-Healing Databases 🛠️, Electric Joins Databricks ⚡
Stripe Uses Graph Search and State Machines to Automate Database Remediation (8 minute read)
Stripe automated MongoDB incident recovery by modeling its fleet as a graph and generating remediation plans at runtime. The system adapts to different shard layouts, cuts pager alerts by about 30%, and avoids an estimated 12 days of unhealthy shard states annually. Dijkstra-based planning replaced brittle workflows and allows partial recovery when no full path exists.
Filtered Vector Search: What ACORN Fixes, and What Fixes ACORN (8 minute read)
Metadata filters can break vector search by splitting the HNSW graph into disconnected islands. Qdrant compares filterable HNSW, ACORN expansion, full scans, and planner choices on one million vectors, showing when extra index-time edges beat query-time repair and when ACORN helps.
How we migrated the database behind every Vercel build (9 minute read)
Vercel migrated the state behind its build warm pool from Redis to DynamoDB because critical billing and container mappings needed durable storage, using dual writes, shadow reads, feature flags, and production comparisons to cut over safely. The migration exposed a hidden dependency on Redis's ultra-low latency, forcing Vercel to redesign an N+1-style supply loop around concurrency rather than serialized reads.
Agents are coming for data (just slowly) (7 minute read)
AI agents are well suited to the boring, mechanical parts of data work, such as testing assumptions, diagnosing schema changes, maintaining context, and fixing pipelines, while proactive automated insights remain much harder because relevance and false positives are difficult to control. Data teams should prepare with strong evals, explicit business context, and low-latency infrastructure that can handle bursty, parallel agent workloads.
The "X is Dead" Fallacy (11 minute read)
Data warehousing, SQL, modeling, semantic layers, and BI are not dying. Implementations change while core functions remain. “AI will kill X” claims often mistake automation for replacement: LLMs can generate code but not replace business semantics, governance, or architectural judgment. The real shift is which responsibilities become automated, commoditized, or move between layers.
Why Knowledge Graph Projects Fail — and How to Make Them Succeed (9 minute read)
Knowledge graphs often fail when treated like relational databases: direct SQL-to-RDF translation, row-oriented design, weak deduplication, and oversized scopes create sparse, slow systems. Better practice starts with specific use cases, staged or federated graphs, controlled APIs, SHACL structure, and explicit provenance. Query graphs on demand and keep the logical model independent of backend.
Electric joins Databricks to bring WASM Postgres to AI agent sandboxes (3 minute read)
Databricks is acquiring Electric to bring PGlite, a lightweight WASM version of Postgres, into AI agent sandboxes, giving agents fast local databases while syncing shared state back to Lakebase. The goal is to let distributed agents work independently with low latency while maintaining a centralized, governed source of truth in Lakebase.
Introducing sqlfmt: an SQL gofmt-style formatter (5 minute read)
sqlfmt is a new opinionated SQL formatter inspired by gofmt that enforces Dimitri Fontaine's “river alignment” style with lowercase keywords, consistent indentation, commas, and comments. It uses tokenization rather than an AST for robustness, runs as a CLI or in-browser via WASM, and verifies formatting does not change query semantics.
Atomic Batch Publishing in NATS 2.12: All-or-Nothing Message Guarantees (4 minute read)
NATS 2.12 adds atomic batch publishing for JetStream streams, letting producers commit related messages as an all-or-nothing unit. Event-sourced systems, multi-subject fan-out, and state replacement workflows can avoid partial writes without building custom recovery logic, while consistency checks such as expected sequence headers preserve concurrency safety.
Maestro: Netflix's open-source workflow orchestrator keeps moving (10 minute read)
Netflix's Maestro is an Apache-2.0 workflow orchestrator for large-scale data, ML, and batch pipelines, with project topics spanning data-pipelines, workflow-orchestration, MLOps, scheduling, and agentic-workflow. For teams reviewing Airflow alternatives or companion orchestration patterns, it is worth tracking as a production-derived open-source control plane.
How Standardizing Product Telemetry Reduced Time to Insight by 97% (4 minute read)
Salesforce standardized product telemetry through a Product Data Platform, replacing fragmented, team-specific pipelines with a common schema that automatically generates trusted product adoption metrics. The platform processes 45 billion rows per day across 19,000 events and 2,000+ product features, cutting time to insight and reducing manual instrumentation work to a few hours.
AI Model Drift: How to Keep Models Reliable (8 minute read)
AI model drift is declining quality as production data, user behavior, prompts, retrieval corpora, or business rules diverge from training assumptions. Teams should monitor data drift, concept drift, pipeline failures, and prompt, embedding, or output drift in LLM and RAG systems. Detection combines baselines, distribution checks, telemetry, task completion, and user feedback.
Curated deep dives, tools and trends in big data, data science and data engineering 📊
Join 590,000 readers for
one daily email