kafka2delta enables one Spark query for 800 Postgres tables into Delta Lake

itnext.io8 min read

The author explains how transitioning from running one Spark Structured Streaming query per table to running a single query per database reduced driver memory consumption by roughly 75 percent when replicating hundreds of PostgreSQL tables into Delta Lake. The approach uses the open-source package kafka2delta to handle per-micro-batch schema evolution, window functions for latest-change extraction, and dual MERGE statements for upserts and deletes. To prevent replication slot bloat on quiet tables, the setup uses Debezium heartbeats. Open article →