Lance: Rebuilding columnar storage for random access and AI workloads

This is part three in our six-part series exploring alternatives to Apache Parquet. So far we’ve covered Parquet for on-disk storage and Apache Arrow for in-memory processing. Today we look at Lance, an open-source format built to handle the access patterns that Parquet was never designed for.

Parquet was designed around analytical scans, reading a few columns across many rows, as fast as possible. Many modern workloads look different: Machine learning (ML) training shuffles and samples individual rows, feature lookups fetch a handful of records at a time, and embeddings and images sit next to ordinary tabular data.

The underlying storage has changed dramatically too, from spinning disks to NVMe SSDs and cloud object stores. Lance, written in Rust, is designed to take advantage of these shifts.

On a modern NVMe drive, bandwidth is rarely the bottleneck for point lookups. The drive in the Lance paper’s test setup handled around 850,000 random 4 KiB reads per second. At that scale, what matters is how many separate I/O operations each lookup needs.

Lance chooses a structural encoding, which its specification calls a layout, for each page of data. There are five layouts to choose from:

• Mini-block layout (used for values under 256 bytes) packs values into small chunks of less than 32 KiB of compressed data. Reading a whole chunk to get one value is cheap, so Lance can use any compression here, including “opaque” schemes where individual values can’t be decoded on their own.
• Full-zip layout (used for larger values, like embeddings) interleaves each value with its structural information in a single buffer. A repetition index maps each row directly to its byte offset, so a single value can be read without touching its neighbors. Only “transparent” compression, where one value can be decoded in isolation, is allowed here.
• Constant layout stores a page where every value is the same (or null) as that single value.
• Blob layout, which users must opt into, stores very large objects such as images or video outside the page and keeps only their locations.
• Sparse layout, available only in the experimental 2.3 format, handles data containing relatively few non-null values.

Lance also drops row groups entirely. Instead, an I/O scheduler splits each column’s reads into appropriately sized requests and runs them in parallel, keeping scan throughput high without a row group size to tune.

Lance is also a table format. Each change to a dataset (an append, a delete, a new column) is committed as a new, immutable manifest pointing to the relevant data files. Deletes go into separate deletion files, and new columns can be added without rewriting existing files. Older versions stay readable, supporting reproducible training runs and easy rollback. Vector and scalar indexes live alongside the data but can be rebuilt independently of it.

Lance reads directly into Arrow’s in-memory format, so DuckDB, Polars, pandas, and PyTorch can work with Lance data without conversion.

One important thing to mention: Lance is now developed as an independent open-source project, separate from LanceDB, the multimodal lakehouse built on the Lance file format.

Lance isn’t a drop-in replacement for Parquet. Parquet’s ecosystem is far broader, with support in nearly every query engine and many independent implementations. Lance’s strength is narrower: datasets that need fast random access, vector search, large binary objects, and built-in versioning. The Lance paper reports that its latest format version often matches or exceeds Parquet’s scan performance while delivering much faster random access, suggesting that the trade-off between scans and lookups isn’t as fundamental as it once seemed.

In our next post, we’ll look at BtrBlocks, a research format that takes a different approach, focusing on fast decompression through cascading lightweight encodings.

Adam Reeve, Open-Source Software Developer