TLDR Data 2025-10-09

Klaviyo revamped its event publishing system to eliminate data loss during network hiccups, Kafka timeouts, or serialization errors, processing up to 170,000 events/sec at peak. The solution implements a dual failure capture strategy: automatic retries write failed events to a self-hosted Kafka DLQ (retained for 7 days), while persistent failures or serialization bugs route events to S3 for infinite retention and manual recovery.

All three formats (Hudi, Delta Lake, and Iceberg) support ACID, Copy-On-Write, schema evolution, and time travel. Hudi excels with Merge-On-Read, advanced indexing, partial updates, non-blocking concurrency, and automated compaction/clustering. Delta shines in Databricks/Z-order integration but uses experimental features and proprietary elements, while Iceberg leads in partition evolution and read/write support, yet demands manual maintenance, slower metadata, and lacks CDC/primary keys.

OpenAI's engineering team built a stream processing platform using PyFlink on Kubernetes with Apache Kafka as the event streaming backbone to handle massive data volumes for AI systems, shifting from batch to real-time processing for fresher data. Kafka acts as the multi-primary event backbone for logs, training data, and experiments.

Tools like DuckDB and Polars are challenging distributed systems (e.g., Spark, AWS EC2, and Databricks) for most workloads, as datasets are rarely "Big Data" and single-node solutions offer cost savings and simplicity amid rising cloud expenses.

Growth no longer rewards the widest net. Modern Go-To-Market (GTM) teams win with precision, not volume, building revenue on infrastructure like pipelines, warehouses, and customer data platforms that turn signals into action. However, not all data is created equal, as the insights draw from five distinct data sources, each with unique engineering challenges, governance requirements, and strategic value.

Data teams often face vague or misaligned project requests from business partners. Instead of asking "What data do you need?", they should use targeted questions to uncover pain points, decision gaps, and opportunities. For example, ask what recent win they want to scale (to build rapport and amplify successes), and when a lack of data led to a bad decision (to reveal gaps and value perceptions).

You can gauge open source momentum by tracking GitHub activity, especially pull requests, which signal innovation speed and community engagement. Comparing trends across areas like analytics engines, event streaming, orchestration, and lakehouse formats can reveal where the ecosystem is moving.

Meta has open-sourced OpenZL, a format-aware, lossless compression framework that outperforms generic compressors by leveraging explicit data structure descriptions. Tailored for structured data like database tables, timeseries, and ML tensors, OpenZL achieves higher compression ratios and speed while maintaining a single universal decompressor, reducing operational complexity. The offline trainer generates data-specific compression configs, enabling rapid adaptation to schema changes without re-deployment.

Versionless Spark decouples clients from servers via a stable Spark Connect API for automatic upgrades, using environment versioning with base images (Spark Connect and Python deps). Its AI-powered Release Stability System (RSS) facilitates upgrades via workload fingerprints, historical metadata, ML-driven error triage, and anomaly detection, resulting in a 99.99% success rate across 2 billion jobs transitioned from DBR 14 to 17 (including Spark 4) with features like collation and bloom filters unlocked.

Arc Core is a high-performance time-series data warehouse designed for rapid data ingestion, achieving 1.89 million records per second on native deployment. It utilizes DuckDB, Parquet, and MinIO, making it a great choice for those who require efficient storage and querying of time-series data.

The "temporal-spatial locality hypothesis" states that data written around the same time is likely read around the same time, justifying storage proximity for efficiency, using the example of faster forward scans (read-ahead caching) versus slower backward ones (database needs to block on IO to fetch the next page). The hypothesis holds trivially in time-ordered systems like streaming and time-series data, but hash-based stores like DynamoDB reject it via random keys to avoid write hotspots, trading read spatial locality for write scalability.

IBM and NVIDIA have integrated cuDF with the Velox execution engine to enable GPU-native SQL query execution in systems like Presto and Apache Spark. Velox rewrites plans to use GPU operators (joins, scans, and aggregations) and supports UCX-based exchange for multi-GPU data routing. In benchmarks, single-node Presto showed an order-of-magnitude performance improvement over CPU.

Partitioning S3 data lakes by year/month/day seems logical, but often degrades performance, creating many small files that increase scan costs and slow queries.

Replacing Airflow with Dagster enabled smarter partitioning, event-driven monitoring, and pure SQL data loading, significantly improving lakehouse efficiency and capabilities.