TLDR Data 2026-06-22

DoorDash replaced fragile CDC pipelines with WAIL after Debezium hit Cassandra scale limits. WAIL logs mutation intent to Kafka and the database, then a smart consumer verifies state, applies schema rules, and publishes events, improving recovery and scaling.

OpenAI's Kepler has moved beyond text-to-SQL into a context-rich analyst for 600+ PB. Daily Codex jobs crawl code to infer grain, lineage, freshness, usage, and hidden semantics. Scoped memories capture corrections at user, team, and global levels, while AST-normalized LLM grading checks SQL and result equivalence. Next: memory pruning, fine-tuning, and dashboard self-validation.

Lyft built an internal Metric Semantic Layer to solve metric definition drift and ensure consistent business logic across teams. It enforces governance through “Golden Metrics,” dual ownership (Business + Operational owners), versioned updates, and access via Python APIs, self-service UI, Amundsen catalog, and AI agents, providing a single source of truth that propagates changes automatically.

Zepto improved high-scale ClickHouse ingestion by optimizing the open-source Kafka Connect connector. The team rewrote key internals, lifted throughput by 45%, removed severe GC pauses, added smarter batching, and contributed two major fixes upstream.

MotherDuck argues DuckDB's local-first, single-node design fits agents that spin up isolated environments, branch workloads, and run analytical queries. It reports roughly 3 ms median latency and claims a $2.40/hour instance beats a $64/hour Snowflake 2XL by about 5x on ClickBench. The vision is agent swarms handling profiling, quality evals, context, anomalies, and lineage.

Databricks' big 2026 announcements are about simplifying data architecture: Lakehouse//RT aims to serve real-time apps and dashboards directly from the lakehouse, while LTAP tries to unify transactional and analytical workloads on one governed copy of data instead of relying on separate databases, CDC, ETL, and serving layers.

Genie Ops points toward self-healing pipelines, but context, governance, and interoperability remain hard. Teams need clearer credential management, event orchestration, agent control, and “git for data” patterns like cloning, rollback, and sandboxed edits. Without standards, it gets messy.

DuckDB 1.5.4 is a patch release focused on bug fixes, security hardening, and performance improvements across areas like VARIANT handling, MERGE INTO, JSON, Parquet, Arrow, gzip, and CLI behaviour.

AWS announced a context stack for AI agents: AWS Context, S3 Annotations, and skill assets in Glue Data Catalog. AWS Context builds and improves a knowledge graph from enterprise data, rules, and domain knowledge, with IAM and Lake Formation access control. Metadata sits in Iceberg on S3 Tables and is exposed through Athena, Redshift, Spark, and MCP tools.

A Ray-native framework delivers 200+ composable data-curation operators for cleaning, deduplicating, synthesizing, and analyzing text, image, audio, video, and multimodal training data. Teams define reusable pipelines as YAML recipes, combine operators into custom workflows, and scale execution from local machines to large distributed clusters.

ClickHouse is marking ten years since going open source, reflecting on how it grew from internal web analytics experiments into a major open source analytical database built from scratch for real-time, high-volume workloads and open contribution.

Nordnet added a real-time Data Quality Health Badge in Looker, showing green, yellow, or red dashboard trust signals. It catches dbt failures, silent crashes, freshness issues, and volume anomalies, consolidates repeat alerts, and uses dbt plus Looker lineage to show blast radius.

AI works better on a dbt project when the repo explains how the project works, what conventions to follow, and which tools to use.

Analytics engineering in 2026 shifts from model production to system design, governance, and AI context engineering as AI automates much of the SQL, tests, and dbt scaffolding and 72% of engineers already use AI-assisted coding.