TLDR Data 2026-11-02

Most hard lessons in data engineering never get written down. They live in someone's head after a long night in production. This Wednesday, 20+ teams share theirs:- Agents and business meaning: DatusAI and PuppyGraph on ontologies.- Fresh context, controlled access: Fresha on real-time state, Toss on permissions.- Reliability at scale: Coinbase, AppLovin and Intuit on production architecture.- Performance and cost: AWS on EKS, Apple's managed analytics service, Carbon Arc's 1 PB migration.- GPUs for analytics: NVIDIA on why GPU execution is now practical at scale.Free and virtual, with recordings for everyone who registers.

Register free · See the agenda

DuckDB 2.0 adds client-server and AI features, while Polars improves streaming and processing of larger-than-memory datasets. On a 23 GB dataset with 2 GB RAM, Polars 2.0 ran out of memory, but 2.1 finished in 438 seconds, nearly twice as fast as DuckDB's 837 seconds.

PostHog tests AI agents on real tasks, such as finding the bugs behind 33 past incidents, where its review agent caught 62%, or checking whether playback failures affected real users. It isolates each test so agents cannot copy earlier answers and is building resettable environments with fixed data to make comparisons fair.

StarTree adapts Pinot's HNSW vector index for Iceberg tables, keeping Parquet source records in object storage while storing and serving a derived index alongside them. This enables hybrid similarity and SQL-filtered queries without a second copy of the records. The trade-off: indexes are roughly as large as the embeddings, and freshness follows Iceberg snapshot synchronization rather than real-time ingestion.

Logical replication lets you upgrade or migrate PostgreSQL with only seconds of downtime: run the target cluster in parallel, stream changes, then cut over when ready. Its real advantage over in-place upgrades is rollback safety and cross-environment portability. Plan around what it does not move automatically, especially schema changes and sequences.

Data products outlasted data-mesh hype by becoming a delivery discipline rather than an enterprise reorganisation: a governed output, accountable owner, defined consumer, and evidence of value. AI agents raise the stakes because they cannot recover missing context through informal knowledge. Start with a business-critical workflow, then make meaning, quality, access, and permitted use machine-readable.

Jepsen found 96 issues in ParticleDB, including data loss, crashes, and incorrect SQL results, mostly without any server failures. Most were fixed during testing, but serious transaction bugs remained. Orkes says a newer release fixes them, though Jepsen has not verified this.

Nobody deletes anything from SharePoint, so your agent reads old versions, contradictions, half-finished files and rogue customer data. Deasy qualifies every file and keeps the bad ones out of reach. Takes minutes. Works for every agent after that.

See it →

Kafgres, a Kafka-compatible Postgres extension, reached about 700 MB/s after caching query plans, relaxing commit durability, and replacing timer-based polling with socket readiness checks. It could suit many workloads, but the single-broker setup lacks the resilience of a Kafka cluster spread across availability zones.

Nodal is an open-source toolkit that turns analysts' business definitions into reusable context for AI analytics agents. It plans queries and checks results against approved metrics, filters, and joins, flagging uncertainty for expert review.

Apache Iceberg V4 adds primary keys, unique rules, and checks, such as requiring positive order amounts, though most will initially be unenforced. Each snapshot records whether the data meets those rules, separately from whether writers must enforce them on new writes.

In most cases, flipping the reward labels in an agent's memory barely changes how it learns. This test hid a rule only the labels could reveal. With true labels, GPT-6 Sol scored 94.4 out of 100 on its last six trials, and with flipped labels 7.8. Claude Haiku 5.5 improved less, and Qwen3 4B barely moved.

Active-active GoldenGate only works when data ownership and conflict policy are designed table by table before replication starts. Simulations showed that even convergent strategies like last-writer-wins can silently lose updates, while delta merging preserves each decrement but cannot prevent both regions from accepting a request against the same last unit before replication catches up. The lesson is blunt: if business rules cannot tolerate reconciliation, do not rely on multi-master replication to fix the problem later.

A sponsored P99 CONF round-up exposes the costs behind several database designs: object storage trades low-cost durability for cold-read latency, read-optimized engines trade RAM and write amplification for tail latency, and vector search trades quantization and cache complexity for scale. The reported benchmarks are workload-specific, but the recurring lesson is: removing one bottleneck moves the constraint elsewhere.

Moving Databricks from Python 3.10 to 3.12 broke Zalando's pickled MLflow models.

Successful PostgreSQL backups can still fail to restore because of missing transaction logs, copied corruption, or lost encryption keys.