Uber drives Lance multi-base path layout

lancedb.com13 min read

The author introduces a multi-base path layout in the Lance table format, designed to distribute a single dataset across multiple S3 buckets or storage locations while preserving relative-path portability. Developed to meet Uber's requirement for petabyte-scale throughput scaling, the design utilizes an explicit array of base paths and base IDs in the manifest rather than rewriting absolute paths. According to the author, this architecture also unlocks new use cases such as hot-tiering, multi-region distribution, selective disaster recovery, and shallow cloning. Open article →