kafka2delta enables one Spark query for 800 Postgres tables into Delta Lake
The author explains how transitioning from running one Spark Structured Streaming query per table to running a single query per database reduced driver memory consumption by roughly 75 percent when replicating hundreds of PostgreSQL tables into Delta Lake. The approach uses the open-source package kafka2delta to handle per-micro-batch schema evolution, window functions for latest-change extraction, and dual MERGE statements for upserts and deletes. To prevent replication slot bloat on quiet tables, the setup uses Debezium heartbeats. Open article →