Lance format documentation
Visit the Lance format documentation to learn more about its design, features, and how it enables the multimodal lakehouse.
Advantages of the Lance format
| Advantage | Description |
|---|---|
| Multimodal storage | Efficiently holds vectors, images, videos, audio, text, and more |
| Version control | Built-in data versioning for reproducible ML experiments and data lineage |
| ML-optimized | Designed for training and inference workloads with fast random access |
| Query performance | Columnar storage enables blazing-fast vector search and analytics |
| Cloud-native | Seamless integration with cloud object stores (S3, GCS, Azure Blob) |
Key concepts
The following concepts are core to the Lance format:Arrow-native, columnar storage and interoperability with the open lakehouse ecosystem (including other file formats and compute engines).
Zero-copy data evolution, meaning you can easily add derived columns (like features or embeddings) at a later time, without full table rewrites. Only new data is written; expensive existing data (like images/videos) remain untouched.
Data versioning
Data in Lance tables are versioned — this helps keep LanceDB scalable and consistent. We do not immediately blow away old versions when creating new ones because other clients might be in the middle of querying the old version. It’s important to retain older versions for as long as they might be queried. Each version contains metadata and just the new/updated data in your transaction. So if you have 100 versions, they aren’t 100 duplicates of the same data. However, they do have 100x the metadata overhead of a single version, which can result in slower queries.Data compaction
As you insert more data, your dataset will grow and you’ll need to perform compaction to maintain query throughput (i.e., keep latencies down to a minimum). Compaction is the process of merging fragments together to reduce the amount of metadata that needs to be managed, and to reduce the number of files that need to be opened while scanning the dataset. Running compaction on a Lance dataset will do the following:- Remove deleted rows from fragments
- Remove dropped columns from fragments
- Merge small fragments into larger ones
Data deletion and recovery
Although Lance allows you to delete rows from a dataset, it does not actually delete the data immediately. It simply marks the row as deleted in theDataFile that represents a fragment.
For a given version of the dataset, each fragment can have up to one deletion file (if no rows were ever
deleted from that fragment, it will not have a deletion file). This is important to keep in mind because
it means that the data is still there, and can be recovered if needed, as long as that version still
exists based on your backup policy.
Learn more about Lance
Lance is a separate open source project. Check out its documentation to learn more.