TLDR Data 2025-10-16
Fivetran + dbt Merge Official 🤝, Streaming DuckDB 🦆, LinkedIn’s Incremental ML 🔁
AI Use Cases for Greater Impact [GUIDE] (Sponsor)
Don't waste time building agents that won't scale.
This guide reveals where AI agents drive tangible ROI (and where they don't) with real-life use cases.Download
the guidefor a full walkthrough:
- Which AI agent use case will bring my team the most value?
- How do I get agents beyond trials?
- What are some examples of successful AI agent use cases?
Start building smarter AI agents with this ultimate guide.
Building the Incremental and Online Training Platform at LinkedIn (15 minute read)
LinkedIn built an incremental and online training platform that shifts from batch to continuous ML updates, boosting Feed, Ads, and Jobs metrics by 2–4%. It combines nearline feature attribution, Flink transformations, Ray-based ingestion, static training graphs, and strong state/monitoring to enable fast, fault-tolerant model updates. Key for data engineers: refactored data pipelines for feature consistency, high-throughput streaming ingestion, standardized runtimes, and tight integration across stream processing, orchestration, and storage.
Evolving Signals-Joiner with Custom Joins in Apache Flink (10 minute read)
Reddit rebuilt its Flink-based Signals-Joiner to boost real-time ML signal enrichment for anti-abuse systems by replacing rigid Tumbling Window joins with a custom, key-aligned windowing strategy using CoProcessFunction. By unifying signal streams, buffering early arrivals, and aligning windows per content ID, it eliminated window boundary issues, improved enrichment rates toward 100%, and made the pipeline more maintainable and transparent, showing how custom Flink joins can outperform built-ins for complex streaming enrichment use cases.
We Built an Open Source S3 Tables Alternative (8 minute read)
RisingWave and Lakekeeper offer an open-source alternative to S3 Tables, letting data teams deploy a fully managed Apache Iceberg stack with a single command. The setup combines a REST catalog, SQL-first ingestion, and a lightweight DataFusion-based compaction engine, enabling interoperability with DuckDB, Trino, and Spark without vendor lock-in or Spark maintenance. It supports time travel, partition-aware design, and automatic compaction, making Iceberg feel like a database while remaining self-hostable. Key operational notes include single-writer constraints, limited DDL flexibility, and the need to tune commit intervals, compaction, and partitioning to manage small files efficiently.
Streaming Patterns with DuckDB (6 minute read)
DuckDB, despite its OLAP roots, can effectively support streaming analytics using adapted architectural patterns such as the Materialized View and Streaming Engine approaches, handling over one million rows per second in recent tests. Integrations with lakehouse formats like DuckLake, as well as community extensions for direct Kafka ingestion, further expand its real-time analytics capabilities. While DuckDB lacks native materialized views and full stateful streaming features, its simplicity and performance make it well-suited for many analytical streaming use cases.
Spark Config Madness (3 minute read)
Spark on AWS Glue-managed Iceberg can fully support CTAS, MERGE, UPDATE, DELETE, and INSERT operations using the official AWS Iceberg Glue packages. Spark config can be further enhanced to handle S3-backed data with minor additional settings, and leverage the default AWS credential chain for security and production alignment. However, the ongoing complexity of such implementation highlights the value of managed Spark or database services in reducing engineering overhead.
The Feature We Were Afraid to Talk About (7 minute read)
dltHub discovered that pure LLM-based automation is not reliable enough for production data workflows. Its first version of data scaffold used LLMs to generate pipeline scaffolds directly from documentation, which seemed promising but frequently produced subtle hallucinations that broke pipelines and wasted debugging time. In v2, it flipped the approach by using deterministic parsers and validators first to extract verifiable facts, then applying LLMs only for semantic nuance, with pointers back to original docs for grounding. This hybrid method significantly improved accuracy and reliability, showing that the key to robust AI-assisted tooling is minimizing LLM usage to the parts where it truly adds value.
SQLMesh, dbt, and Fivetran... What's Next? (5 minute read)
Fivetran's acquisition of dbt, following recent integrations with Tobiko Data and Census, marks a significant consolidation in the modern data stack ecosystem. This move combines key transformation, ingestion, and modeling tools, potentially streamlining workflows but raising concerns about shrinking open-source offerings and feature divergence between dbt Core and dbt Fusion. Anticipate greater platform unification, but also prepare for evolving standards.
Learn Advanced Analytical Techniques Used in the Intelligence Community (Sponsor)
The Era of Open Data Infrastructure (6 minute read)
Fivetran and dbt Labs are merging to create an open data infrastructure platform, anchored by Apache Iceberg as the industry-standard, engine-agnostic table format. This unified approach enables standardized, reliable data ingestion, transformation, and activation, delivering SLAs, complete lineage, and seamless portability across compute engines. The combination addresses enterprise data utilization bottlenecks, dramatically improving model trustworthiness, governance, and operational agility in analytics and AI workflows.
Cross-Cloud Data Replication Over Private Networks with Confluent (13 minute read)
Confluent Cloud introduces cross-cloud data replication over private networks, enabling secure, offset-preserving Kafka topic and schema mirroring across AWS, Azure, and Google Cloud using Cluster Linking and Schema Linking. This fully managed solution eliminates complex VPN setups, reduces egress costs, and ensures compliance via local data residency, supporting seamless disaster recovery with near-zero RPO and RTO.
Visualize Data Lineage Using Amazon SageMaker Catalog for Amazon EMR, AWS Glue, and Amazon Redshift (5 minute read)
Amazon SageMaker Unified Studio now offers automated, end-to-end data lineage visualization across AWS Glue, Amazon Redshift, and Amazon EMR. The OpenLineage-compatible SageMaker Catalog captures and versions lineage events, enabling in-depth traceability, auditability, and historical comparisons of data transformations and asset evolution.
Systems Thinking for Scaling Responsible Multi-Agent Architectures (50 minute video)
Rapidly scaling multi-agent AI systems introduces complex, often unintended risks and feedback loops, underscoring the need for responsible engineering practices. Applying systems thinking, specifically Causal Flow Diagrams and frameworks like Cynefin, enables teams to anticipate emergent behaviors, balance efficiency vs. human impact, and dynamically adjust reward functions or guardrails. Tools such as LIME, SHAP, Arize, and telemetry can enhance observability and explainability, while architectural patterns (orchestrated, decentralized, and human-in-the-loop) should be adapted to context and risk.
From Dark Data to Bright Insights: The Dawn of Smart Storage (6 minute read)
Google Cloud has unveiled auto annotate and object contexts for Cloud Storage, leveraging AI to automatically generate metadata and semantic insights for unstructured data. Auto annotate, now in experimental release, delivers object-level metadata (labels, detections, and PII flags) at scale, while object contexts provide native, flexible tagging and metadata lineage, fully integrated with Cloud Storage, IAM, and BigQuery. It is currently in a limited experimental release.
SQL Shader (Tool)
DuckDB-WASM SQL Shader is a browser tool that turns SQL queries into real-time procedural graphics to explore database engine performance.
Fact Graph (GitHub Repo)
Fact Graph is a production-ready knowledge graph of US tax law for JavaScript and JVM languages.
Curated deep dives, tools and trends in big data, data science and data engineering 📊
Join 630,000 readers for
one daily email