TLDR Data 2026-08-31

Last year's Astronomer virtual conference drew 8,000+ data engineers. This year, Orchestrate Everythingwill kick off with Marc Lamberti's free crash course on orchestrating AI with Airflow. You'll get a free certification exam code ($150 value) for joining.

Then, see what production AI orchestration actually looks like with sessions from builders at Lyft, Ramp, and Wix. Sept 16. Register now.

Free to attend. Built for data engineers at every level.

Register free.

Twilio's Hudi pipelines store checkpoints in S3, so ordinary Kafka consumer lag misses when lake data is actually stale. This design reads the latest valid Hudi checkpoint, seeks Kafka partitions, and computes freshness from the oldest uncommitted record, turning petabyte-scale SLA drift into a breach ratio operators can alert on.

Pinterest scaled candidate generation by conditioning retrieval embeddings on explicit contexts such as interests, boards, and pins. The system combines conditioned sequence modeling, semantic IDs, unified models, slot routing, request deduplication, and GPU serving so one home feed can represent multiple user intents without exploding serving cost.

Kafka remains a strong log abstraction, but many streaming workloads ultimately reconstruct tables in object storage anyway. This walkthrough contrasts log-first pipelines with table-first streaming designs such as Fluss, showing where update semantics, compaction, snapshots, and catalog integration may belong closer to hot storage instead of a downstream lakehouse job.

A company moving four large Salesforce instances into Snowflake wants to replace Fivetran after cheaper tools failed at scale. The strongest alternatives suggested were Salesforce Data Cloud's native zero-copy Snowflake integration or a custom dlt pipeline using Salesforce's Bulk API.

Major cloud incidents repeatedly stem from unavoidable risks such as resource saturation, networking failures, security controls, and uncommon changes like migrations. Reliability measures add their own complexity, so outages cannot be eliminated and teams must invest in strong incident response.

Grab operationalizes data reliability by treating contract breaches as "Data Production Issues" (DPIs) with a standardized incident lifecycle. The key design move is a stable "Data Health API" that decouples incident orchestration from platform-specific internals, categorizing failures (upstream, job, or data) to automate root-cause assignment. This transforms data quality from passive alerting into an operational governance model where incidents only close upon verified recovery.

Target encoding, rolling averages, and shifted features can silently leak the answer when computed before a proper split or across mixed entities. The safe workflow is to split first, calculate training-set encodings out of fold, fit other statistics only on training data, use group-aware transforms, and give unseen categories honest fallbacks.

Most AI tools don't understand your dbt project. They break lineage and miss dependent tests, creating more work. dbt Wizard CLI reads contracts and dependencies before editing, and coordinates changes across models and YAML configurations. It validates its own work. Engineers get dbt-native accuracy without a platform migration. Learn more →

Spark 4.1.0 adds declarative pipelines, real-time structured streaming mode, Arrow-native UDF and UDTF support, SQL scripting and VARIANT as GA features, and Spark ML on Connect for Python. The release targets lower-latency streaming and a cleaner PySpark and SQL experience while improving caching, memory management, and large-workload stability.

DataFusion 55.0.0 ships ten weeks of work from 175 contributors, with MERGE INTO support, range partitioning, runtime row-group pruning, and broad performance gains. Highlights include TopK pruning, faster IN-list evaluation, fewer object-store requests for CSV, nested-column Parquet read reduction, and lower memory use for fixed-size binary GROUP BY operations.

Ablation results in tabular foundation models can be misleading when redundancy looks like causal importance. Measurements of direct and total effects across four models found little evidence of downstream self-repair, suggesting data teams should treat stable outputs after ablation as a reason to examine causality more carefully, not as proof that a layer is irrelevant.

X's legal action against Nitter shows how closed platforms can revoke access at any time, limiting competition and innovation. AT Protocol offers an alternative by giving users portable identities and letting apps freely replicate, query, and build on shared live data.

Better data, expert input, and iterative labeling often improve ML systems more than model tuning.

This geospatial modeling framework automatically selects and combines satellite, tabular, and foundation-model data for high-resolution predictions.

Lorikeet sped up some Postgres queries by up to 1,000x with a simple typecast that fixed an index-breaking mismatch between timestamp and timestamptz under Row Level Security.