Spark Structured Streaming watermark misconfiguration caused 146GB checkpoint

An Apache Spark Structured Streaming pipeline experienced severe performance degradation and a 146GB state store checkpoint due to an overly broad seven-day watermark configured for an upstream service with a 20-minute retry SLA. The author explains how tightening the watermark to match the SLA and migrating from the default in-memory state store to RocksDB reduced state size to 11GB, dropped micro-batch processing lag from 93 minutes to 35 seconds, and significantly accelerated restart times. Open article →