Artificial intelligence news.
Straight from the original announcements.

Lance v2 targets AI workloads with a flexible columnar format

November 19, 2026, 08:16By the AI News Bites Editorial TeamSource: LanceDB
Lance v2 targets AI workloads with a flexible columnar format
Image: LanceDB

LanceDB has outlined Lance v2, a new columnar container format designed to address limitations the company sees in Parquet and Lance v1 for modern AI and machine learning workloads. The design focuses on point lookups, large values, wide schemas and extensible encodings.

Key points
  • Lance v2 is designed for AI and machine learning workloads.
  • The format targets point lookups, wide columns and very wide schemas.
  • Encodings are handled through extensions and protobuf “any” messages.
  • Lance v2 removes row groups and allows pages to be stored separately.
  • The announcement does not provide release timing or performance measurements.

Designed around AI and machine learning workloads

In a blog post dated April 13, 2024, Lance engineer Weston Pace said Lance was created because readers and writers for existing column formats did not handle AI and machine learning workloads efficiently. According to Lance, Lance v1 addressed some of those problems but still struggled in several cases.

The company highlights point lookups as one of the central use cases. These queries access a small set of rows and are important when using a secondary index. Semantic search and full text search in LanceDB ultimately involve point lookups, according to the post.

LanceDB says Parquet is poorly suited to this pattern because its encodings are not designed to be “sliceable.” Accessing one row can require loading an entire page, a problem the company says is especially significant for multi-modal workloads with large values.

The post also describes challenges involving wide columns, such as semantic search embeddings and images, and very wide schemas. Finance workloads may have a column per ticker, while feature stores can contain thousands of features for one record. Even when column projection is selective, readers still need to load schema metadata for all columns in a file, creating costs for low-latency workloads.

Encodings become extensions

Lance v2 treats encodings as extensions rather than building them into the format. The file readers and writers are unaware of the encoding used. Instead, a data producer chooses an encoding supplied by a plugin, while encoding descriptions are stored as protobuf “any” messages.

This approach allows new encodings to be created without changing the file format, file writer or file reader. LanceDB says an encoding developer needs to publish a .proto file describing the encoding and provide at least one encoder and decoder implementation.

The format itself also has no type system. From Lance’s perspective, a column is a collection of pages with an encoding, and each page is a collection of buffers. The readers and writers described in the post use the Arrow type system to convert those pages into typed arrays.

If a reader encounters an unsupported encoding, the user receives an error stating that “this file uses encoding X and the reader has not been configured with a decoder.” The user can then install or implement a decoder without requiring a change to the Lance format.

Lance v2 removes row groups

Another major design choice is the removal of row groups, which Lance describes as a solution that has “outlived their usefulness.” The company says small row groups can create excess metadata, poor performance and undersized pages, while large row groups can require substantial RAM during writing and reading.

LanceDB also argues that row groups are a poor fit for multi-threaded parallelism within a process. Reading several row groups at once can launch many simultaneous I/O operations, while computation may have to wait for those operations to complete. The post favors pipeline parallelism instead of data parallelism.

Lance v2 allows pages within a column to be stored separately rather than next to one another. Each column writer maintains its own buffer and flushes a page when enough data has accumulated. According to Lance, this enables maximally sized data pages with minimal RAM buffering in the writer and reader.

For users building AI systems, the proposed format is intended to make indexed access to large values and wide datasets more practical, while giving encoding developers a way to add capabilities without modifying the underlying container format. The announcement describes the design and its goals, rather than providing release timing or performance measurements.

Related news

Lance introduces community governance for its open format
For developersNovember 19, 2026, 08:16

Lance introduces community governance for its open format

Lance has introduced a formal governance structure for its open format community, defining contribution paths, leadership roles and consensus-based decisions. The…

LanceDB Combines Lance Namespace and Ray for Scalable AI Pipelines
For developersNovember 19, 2026, 08:16

LanceDB Combines Lance Namespace and Ray for Scalable AI Pipelines

LanceDB has presented an end-to-end approach for production AI workloads that combines Lance Namespace, Ray and LanceDB. The company says the integration connects…

Lance File 2.1 targets smaller storage and faster random access
For developersNovember 19, 2026, 08:16

Lance File 2.1 targets smaller storage and faster random access

Lance File 2.1 has entered beta with a redesigned encoding system intended to reduce file size and improve access to individual values. Lance says the update builds on…