TLDR Data 2026-10-05

No one wants to use a tool they don't trust, and for data teams, trust in AI starts with the data underneath it. When no one can say where a model's inputs came from, whether they passed quality checks, or who can access them, projects stall in pilot. Collibra's free 12-question AI trust assessment shows where your org stands:

• Score your org across the 6 core capabilities that take AI from pilot to production
• Pinpoint the roadblocks holding back adoption, from data quality to governance gaps

It takes 5 minutes. Get your AI trust score

OpenAI built an internal AI data agent that answers complex data questions by combining warehouse queries with rich context from schemas, lineage, code, company knowledge, memory, and live sources, while showing its assumptions, SQL, citations, and confidence. The big lesson is that reliable data agents depend less on raw model intelligence and more on giving them the right context, continuously learning from corrections, and testing them with strong evals.

Datadog built Distributed DataFusion to let a single Apache DataFusion query run across multiple machines, giving heavy interactive queries much more compute while keeping lightweight queries on one machine to avoid unnecessary overhead. In production, some expensive queries ran up to 10× faster at the same total cost. Datadog released the system as an open-source framework that teams can adapt to their own data sources and infrastructure.

LinkedIn replaced three systems for finding and analyzing metric metadata with a single ClickHouse index. It now handles more than 150,000 queries a minute across 13 billion metrics, averaging 68 milliseconds per query. The team explains how it migrated behind an existing gateway and cut memory use to about a fifth of the old setup.

Context is becoming a primary software artifact for AI coding agents: teams must generate, lint, version, and test context packages with the same rigor as application code. Managing context drift, establishing AI bills of materials, and instrumenting production retrospectives are now essential data engineering disciplines to prevent agent hallucinations and silent architectural degradation.

LinkedIn outgrew Kafka's partition model at extreme scale and built Northguard to separate ordering, replication, and placement, making data movement and recovery less painful. Xinfra then hid the migration behind a compatibility layer so applications could move gradually, showing how infrastructure rewrites become practical when clients are insulated from backend changes.

Learn how to analyze structured and unstructured data (like text, images, and PDFs) together in BigQuery, without moving data. Incorporate techniques like multimodal processing, automatic embedding generation, and hybrid search; then go further with generative AI functions and ObjectTables to extract deeper, unified insights. Watch the webinar session or read the blog.

Amazon Aurora PostgreSQL now queries Apache Iceberg and Parquet tables directly in Amazon S3 data lakes without requiring separate ETL or reverse-ETL pipelines. Powered by an embedded DuckDB engine, the feature supports standard PostgreSQL syntax, predicate pushdown, and column pruning across operational and historical analytical data with zero additional Aurora licensing fees.

Cloudflare has released Clef and Clef-flash, ‘Jev-like' open-source System 1 models that turn text or images into choices and probabilities. They can help route support tickets, classify websites, or decide when an agent should ask a human. Both run on Workers AI or locally, and Cloudflare is offering help fine-tuning them for specific tasks.

Apache Iceberg 1.12.0 broadens production support for major v3 capabilities, including Variant, geospatial types, row lineage, deletion vectors, encrypted metadata, and REST catalog operations. However, it also removes platform support and deprecated APIs, so treat this as a substantial upgrade and test compatibility carefully before rollout.

A six-week analysis of 43,261 Fable 5 calls found that the model name stayed the same while the delivered reasoning budget varied sharply, with many invocations showing no thinking at all. The broader warning is an inference gap: benchmark performance may depend on far more sequential reasoning than production users actually receive.

Microsoft and Google are backing Apache Ossie, an open semantic-model interchange specification supported by 60+ data and AI vendors. Using standard JSON/YAML schemas, Ossie enables portable dataset, metric, and relationship definitions across disparate BI platforms, eliminating redundant semantic modeling and metric drift.

A production AWS Bedrock guardrail pipeline for AI agents reduced action latency from 13,874 ms to 1,824 ms without altering models or accuracy. The bottleneck was execution topology: independent safety checks ran sequentially. By parallelizing independent layers and short-circuiting cheap reject rules early, the team dramatically reduced execution latency and compute costs.

String-heavy aggregations waste time hashing and comparing the same text values repeatedly. DuckDB shows that replacing those columns with small integer surrogate keys through dimension tables can make grouping cheaper, reduce memory pressure, and lower spill risk. The broader takeaway: normalize low-cardinality strings before large analytical aggregations when performance matters.

Jev completed Pokémon Red in 37 hours and 40 minutes, making 16,150 decisions with a reported model cost of about $1.65.

An early look at dynamic ATTACH statements in DuckDB 2.0 Alpha.