TLDR Data 2025-07-17
Databricks vs Snowflake ⚔️, Scaling PostgreSQL Queues 📈, 1K API Connectors 🔌
ClickHouse 25.6 Summer Bonus: CoalescingMergeTree Table Engine (12 minute read)
CoalescingMergeTree in ClickHouse efficiently merges sparse records over time to keep only the latest version of each entity, making queries much faster at scale. The recommended approach is to use a standard MergeTree table for full historical data and a CoalescingMergeTree table for up-to-date, consolidated state to ensure both detailed history and efficient access to the current state.
Lessons From Scaling PostgreSQL Queues to 100k Events Per Second (15 minute read)
RudderStack chose PostgreSQL as its main streaming engine instead of specialized tools like Apache Kafka, scaling it to handle 100,000 events per second by overcoming challenges like table bloat, slow queries, and index bottlenecks through targeted index optimization and careful compaction strategies. It further improved performance with application-level caching, write amplification awareness, and custom PostgreSQL settings such as tuned WAL buffers, checkpointing, and autovacuum.
Engineering Real-Time Architecture At Scale: The Tech Behind Eloelo's Multi-Million Scale Push Notifications (21 minute read)
Eloelo engineered a scalable push notification system capable of delivering millions of messages quickly and reliably by leveraging a distributed message queue, microservices, and partitioned processing. It optimized for high throughput and low latency using batching, horizontal scaling, worker pools, and robust error handling to ensure timely delivery even during massive traffic spikes.
Long Read; Databricks vs. Snowflake; The Final Chapter (22 minute read)
Databricks and Snowflake are facing significant challenges as their core business models are being threatened by evolving data workloads, particularly in real-time processing. As they pivot to capture new markets, data engineers must navigate a landscape where integration and complex data platforms become essential. Emerging technologies like Apache Flink and ClickHouse present new competitive dynamics.
Accuracy Is Dead: Calibration, Discrimination, and Other Metrics You Actually Need (7 minute read)
Traditional accuracy metrics are increasingly inadequate for evaluating machine learning models, especially in production environments. Emphasizing advanced metrics like calibration (how well predicted probabilities reflect real outcomes), discrimination (the model's ability to distinguish between classes), and bespoke performance indicators provides deeper operational insight.
The Influence of Prompt Bloat on LLM Output Quality (8 minute read)
When prompts become excessively long or cluttered, it reduces output quality by overwhelming the model with irrelevant details and diluting key context. To address this, use advanced techniques like Meta Prompting, which uses another model to refine prompts; Gradient Prompt Optimization, which treats prompts as parameters to be optimized; and frameworks like DSPy or ScaleDown to automate the prompt optimization process.
🛠️ Get hands-on with AWS Data and AI in free builder workshops (Sponsor)
The next generation of Amazon SageMaker is the center for all your data, analytics and AI.
Get hands-on in free builder workshops and discover how to develop and scale AI models, build in a unified IDE, and reduce data silos - all with built-in governance.
Grab your seat today!
dltHub Workspace (Tool)
dltHub Workspace is a new environment for building, debugging, and operating dlt data pipelines, launching with LLM-native development that can scaffold connectors for 1K+ REST API sources. It lets a single developer move from pipeline code to data ingestion and notebook-ready reports in one flow, with outputs tailored to data users. A large and growing library of source, destination, and pipeline templates is available, and more features are coming soon.
Introducing Amazon S3 Vectors: First Cloud Storage with Native Vector Support at Scale (Preview) (9 minute read)
Amazon S3 Vectors introduces a new cloud storage solution optimized for large vector datasets, enabling businesses to store and query AI-ready data at scale with up to 90% lower costs. It features vector buckets and dedicated APIs for efficient similarity searches, automatic data optimization, and integration with Amazon Bedrock and OpenSearch, facilitating scalable generative AI applications and reducing the complexity of managing vector databases.
PostgreSQL Storage: Comparing Storage Options (5 minute read)
This case study compares 4 PostgreSQL storage options from row-based (heap) tables, columnar storage, CSV, to Parquet files by evaluating their query performance and storage efficiency. For fast single-row access, indexed heaps are the quickest, while column stores and Parquet files offer the best data compression.
Kubernetes Finally Solves Its Biggest Problem: Managing Databases (8 minute read)
Kubernetes Operators, in conjunction with tools like CloudNativePG and Atlas, enable fully declarative, GitOps-native lifecycle management of stateful workloads, including database schema evolution, solving a long-standing pain point for data teams. Operators use reconciliation loops to automate installation, upgrades, and migrations safely, even across complex environments, enabling scalable stateful infrastructure.
MCP: A Practical Security Blueprint for Developers (4 minute read)
Model Context Protocol (MCP) significantly enhances AI agentic coding environments by seamlessly integrating external tools and context, but it introduces critical security risks, including command injection, prompt injection, token theft, and over-permissive access. Real-world exploits, such as flaws in Anthropic's MCP Inspector and third-party servers, underscore the urgency for robust authentication, encrypted connections, strict permissioning, and elimination of plaintext secrets.
A Recap on May/June Stability (7 minute read)
The Neon platform experienced significant operational strain in May and June due to unexpected surges in database and branch creation driven by agentic AI partners, leading to multiple incidents. To address these challenges, the team implemented a horizontally-scalable architecture called "Cells" and made configuration adjustments to manage load effectively, ultimately improving stability and performance for future database operations.
Curated deep dives, tools and trends in big data, data science and data engineering 📊
Join 630,000 readers for
one daily email