Lance File 2.1 reaches stable release with compression and random access

LanceDB has declared version 2.1 of the Lance file format stable after a period in beta. The release adds cascading encoding and compression while preserving the random access performance that LanceDB says is central to the format.
- Lance File 2.1 is now stable after a beta period.
- The format adds compression without, according to LanceDB, affecting random access performance.
- Lance 0.38.0 is the first release to fully support reading 2.1 files.
- Existing 2.0 datasets can remain unchanged and will continue to be maintained.
- LanceDB expects 2.1 to be the last version requiring a dataset copy to upgrade.
Stable format with backward compatibility
In a blog post dated October 3, 2025, LanceDB said it had documented the 2.1 specification and would treat potential breaking changes as part of version 2.2. The company is committing to backwards compatibility for 2.1.
The main limitation of Lance File 2.0 was its lack of compression. LanceDB said its goal for 2.1 was to add cascading encoding and compression without sacrificing random access performance. The company reported that it achieved that goal.
Discussing a recent paper that measured random access performance, LanceDB wrote: “I guess this means we’re still the fastest.” That statement is the company’s own characterization of the result.
What Lance File 2.1 adds
Alongside compression, the format introduces several other changes. LanceDB says 2.1 can reduce IOPS when reading nested data, including lists and structs. It also supports distinguishing between null structs and null values.
An optional repetition index cache can further reduce IOPS for variable-width data, although LanceDB says this comes at the cost of increased memory usage. These changes are intended to improve storage and reading efficiency for particular data layouts and workloads.
The likely beneficiaries are workflows that scan smaller columns, which LanceDB describes as typically being limited by disk bandwidth. Vector embeddings, images and audio may see less total impact because they are already pre-compressed and can make up most of a dataset. LanceDB recommends converting a subset of data first to test whether the change produces a meaningful reduction in size or performance.
Upgrade guidance and the path to 2.2
Storage versions are set for each dataset. Users satisfied with the performance of a 2.0 dataset do not need to upgrade, and LanceDB said it will continue to maintain 2.0. To use 2.1 features, however, a dataset must be copied and written with data_storage_version="2.1".
Lance 0.38.0 is the first release with full support for reading 2.1 files. LanceDB recommends upgrading software to 0.38.0 or higher before writing 2.1 files, because older readers may cause problems when working with files created by newer versions. Version 2.1 is not the default in 0.38.0, so it must be selected explicitly.
Work on 2.2 has begun. LanceDB expects 2.1 to be the last version requiring a dataset copy for an upgrade and hopes to support mixed-version datasets in 2.2. Planned areas include additional compression coverage, variable-width struct packing and improved JSON encoding. Until then, teams keeping older Lance versions in a mixed environment should explicitly set the storage version to 2.0 when creating datasets.
LanceDB: Lance File 2.1 is Now Stable


