DuckDB 2.0 passes benchmark while Polars 2.0 fails

The author evaluates the 2.0 releases of data processing tools Polars and DuckDB by running a resource-constrained benchmark on large S3 datasets. DuckDB 2.0 successfully completed the test by leveraging network I/O and out-of-core spilling, while Polars 2.0 initially failed due to memory limitations before a later 2.1 update added group-by spill-to-disk support. Open article →