Building durable streams on Postgres

Hatchet has implemented durable streams on Postgres to provide in-order, low-latency streaming for distributed workflows and chat interfaces. To avoid the scalability bottlenecks of long-running transactions and the visibility issues of autoincrementing IDs under concurrent writes, the system uses a pessimistic locking setup with row-level locks and batch message reservations. The backend communicates via gRPC while utilizing a REST side-channel for payloads exceeding the 4MB gRPC size limit. Open article →