Lance format

Lance format overview

Lance is an open-source, columnar format designed for multimodal AI. It defines how data is stored and organized into versioned tables. LanceDB Embedded and LanceDB Enterprise both use Lance as their storage foundation. The format is developed as an independent open-source project, with its own specification and tools.

Multimodal data

A Lance table can hold images, video, audio, text, embeddings and structured metadata together. For example, a video dataset can keep each clip alongside its caption, labels and embeddings, so applications can search and work with those fields together.

Data access, versioning and column updates

Efficient data access. Read individual rows for retrieval and training, or scan columns for analytics.
Versioned data. Read an earlier table version to reproduce an experiment or inspect previous results, while that version is retained.
Independent column updates. Add embeddings or other derived features without rewriting the existing image, video or text columns.

Opening Lance datasets in Python

Any .lance dataset directory is a LanceDB table. Pass the parent directory to lancedb.connect() and open the table by its directory stem:

import lancedb

db = lancedb.connect("/data")  # pass storage_options=... for S3, GCS or Azure
tbl = db.open_table("frames")  # opens /data/frames.lance; version=N pins a version

Use lancedb for search, retrieval, data curation, multimodal blob access and application code. Install pylance (import lance) only when building query engines or connectors, calling format-level fragment APIs directly, or converting plain-scan lazy or legacy blob columns with to_pandas() (see Multimodal data and blobs for pylance-free conversions).

`pylance` (`import lance`)	`lancedb` (`import lancedb`)
`lance.dataset("/data/frames.lance")`	`lancedb.connect("/data").open_table("frames")`
`lance.write_dataset(data, "/data/frames.lance")`	`lancedb.connect("/data").create_table("frames", data)`
`lance.vector.vec_to_table(mat)`	`db.create_table(name, pa.table({"vector": list(mat)}))`
`lance.blob_field("video")`	`lancedb.blob("video")`
`ds.take_blobs("video", ids=row_ids)`	`tbl.fetch_blob_files("video", row_ids)` (seekable handles) or `tbl.fetch_blobs("video", row_ids)` (`pa.LargeBinaryArray`)
`ds.to_table(columns=cols, filter=expr, limit=n)`	`tbl.search().where(expr).select(cols).limit(n).to_arrow()`
`ds.to_batches(batch_size=k)`	`tbl.search().to_batches(k)`
`ds.take(offsets)`	`tbl.__getitems__(offsets)` (preserves offset order) or `tbl.take_offsets(offsets).to_arrow()` (unordered)
`ds.versions()`, `ds.checkout_version(v)`, `ds.restore()`	`tbl.list_versions()`, `tbl.checkout(v)`, `tbl.restore()`
`ds.optimize.compact_files(...)`, `ds.cleanup_old_versions(...)`	`tbl.optimize(compaction_options=..., cleanup_older_than=...)`

Learn more

Lance format specification: how the format works.
Multimodal data and blobs: declare lancedb.blob() columns and read blobs on demand.
Table versioning: read and manage table versions.
Computed columns: add derived features to your data.
Architecture: how LanceDB Embedded and Enterprise build on Lance.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Lance format overview

Multimodal data

Data access, versioning and column updates

Opening Lance datasets in Python

Learn more

​Lance format overview

​Multimodal data

​Data access, versioning and column updates

​Opening Lance datasets in Python

​Learn more

Lance format overview

Multimodal data

Data access, versioning and column updates

Opening Lance datasets in Python

Learn more