TLDR Data 2025-05-05

Ever spent hours debugging a broken CSV import pipeline? Got bigger ambitions than fixing malformed UTF-16 files or chasing rogue semicolons?

OneSchema's FileFeeds automatically handles all those edge cases that make you question your career choices – irregular formats, mysterious encoding issues, and everything else that makes your recurring CSV imports break at the worst possible time.

👉 Simply define your schema with built-in validations, let non-technical users clean a sample file (which records the transformation patterns), and connect to your API/SFTP/email.

✅ Your data pipeline now runs without your intervention, validating everything before it touches your database.

👋 Say goodbye to custom CSV-loading scripts

Apache Iceberg is transforming data storage with its open table format, gaining traction among various data vendors and tools. The upcoming Iceberg V3 introduces several enhancements, including support for geospatial data types, semistructured data, and better table encryption. These updates streamline performance and metadata management, positioning Iceberg as a critical foundation for centralizing and governing large-scale data operations across diverse systems. The cooperation between Iceberg and Delta communities is paving the way for further innovation.

Integrating with Spark via a vectorized reader, Vortex on Ice leverages zone maps and push-down compute to minimize I/O and decompression. Built with Rust and bridged to Java using JNI for performance (3× faster than JNA), it was tested on Azure with TPC-H datasets, outperforming Iceberg's Parquet reader. The integration, developed with Microsoft's Gray Systems Lab, supports Iceberg's metadata for seamless adoption.

Amazon DynamoDB excels in serverless, cloud-native applications, offering predictable performance with sub-20ms latencies, automatic partitioning, and high durability through multi-AZ replication. It handled up to 146 million requests per second during 2024 Prime Day. DynamoDB supports strong and eventual consistency modes, simplifying read-after-write scenarios, but its rigid key-value structure and lack of native joins make it less suitable for complex queries or relational data. Unpredictable or frequently changing access patterns can lead to costly table scans or table repopulation.

DuckDB's stream windowing functions enable temporal analysis of streaming data, computing time-based aggregates like tumbling, hopping, and sliding windows on datasets such as railway schedules. These functions, built on SQL window function syntax, support flexible window definitions (e.g., RANGE BETWEEN INTERVAL 15 MINUTES PRECEDING AND CURRENT ROW) for real-time analytics. They leverage DuckDB's columnar storage and vectorized execution for high performance. This post shows examples of queries for train delay analysis and passenger counts.

The undo feature in data systems, inspired by user-friendly tools like Google Docs, is challenging to implement in data engineering due to the complexity of reversing operations like table drops or row deletions in systems like Snowflake or BigQuery. Apache Iceberg supports lightweight undo via time travel and snapshots, allowing reversion to previous table states, while Delta Lake uses transaction logs for similar functionality, though both require careful management to avoid performance overhead. Emerging patterns like copy-on-write and logical deletion (e.g., marking rows as deleted) enable reversible operations, but scaling undo for massive datasets remains an open challenge.

The decision to move from Snowflake to Trino was driven by mismatches between Snowflake's design and the company's needs for fast-evolving, semi-structured data. While Snowflake excels in structured data environments, it struggled with high compute costs when querying semi-structured JSON data. Trino, by contrast, offered more control, scalability, and a fixed cost model, ultimately aligning better with their architecture and business needs, allowing for seamless integration across different systems.

Modern data strategies demand proactive lineage to combat the challenges of complex, dynamic data ecosystems. Transforming data into modular products with defined boundaries and ownership enhances trust, governance, and collaboration, aligning with AI-native architectures. This approach turns lineage from a reactive debugging tool into a proactive operational asset, offering context-bound insights and streamlined data flow understanding. By embedding context-bound metadata, every data product becomes a business-readable map, streamlining impact analysis and compliance. This is crucial for scaling teams and ensuring data reliability.

Apache Airflow®, the world's leading open source data orchestration platform, delivered on the top-requested community features in the recent 3.0 release. In this web series, Astronomer will cover everything you need to know so you can start taking advantage of the latest capabilities fast. Register for one session or all four - it's free

DiceDB is a trending open-source in-memory database optimized for modern hardware. Commonly used as a cache, it offers a familiar interface while enabling real-time data updates through query subscriptions. It delivers higher throughput and lower median latencies, making it ideal for modern workloads. DiceDB can be easily deployed and tested as a Docker container.

MongoDB has released a repository containing over 100 step-by-step notebooks focused on AI Agents and Retrieval-Augmented Generation (RAG). The collection provides comprehensive coverage from basic chatbot development to more complex applications like Airbnb-style agents.

Apache Iceberg's Format Version 3 delivers substantial improvements including enhanced compression techniques, more flexible schema evolution capabilities, and optimized partition management for better query performance. The update expands metadata functionality while strengthening integration with processing frameworks like Spark, Flink, and Trino. New transaction guarantees ensure data consistency in concurrent environments, further establishing Iceberg as a leading solution for managing large-scale data lakes.

Google Research's Amplify Initiative has launched a global, open data platform to enhance AI model training through diverse, culturally relevant datasets. This platform enables local communities to co-create annotated data, addressing critical local needs by reflecting global linguistic and cultural diversity. An initial pilot in Sub-Saharan Africa amassed 8,091 queries in seven languages. Plans are in place to expand this collaborative approach to Brazil and India. Amplify aims to fill AI-generated data gaps, emphasizing responsible data practices and improving models' contextual responsiveness across different geographies.

Multi-agent systems can be orchestrated in real-time using Kafka and Flink. The orchestrator pattern, which utilizes Flink for processing and routing with Kafka for shared memory, enables dynamic message handling between agents, enhancing scalability and adaptability. Agents are structured as stateful microservices reacting to events, facilitating efficient collaboration and reducing dependency bottlenecks. This architecture is exemplified with a real-world AI-driven sales development representative system use case.

Redis has reintroduced its core key-value database under an open source license, aiming to re-engage the developer community and enhance collaboration.

The Spark PDF project offers a custom data source for Apache Spark that allows users to read PDF files directly into Spark DataFrames.