> ## Documentation Index > Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt > Use this file to discover all available pages before exploring further. # Tables > The object most operations act on: creating, opening, renaming, cloning and dropping one. **Applies to:** Embedded ✅, Enterprise ✅ A table stores data in rows and columns and keeps a history of committed versions. Queries read table data, writes update it, and indexes accelerate searches over its columns. This page covers table management, data types, schema changes and version history. For loading records, see [Ingestion](https://docs.lancedb.com/ingestion.md). For changes to columns, see [Schema evolution](https://docs.lancedb.com/management/table.md#schema). > [!NOTE] > > In LanceDB Enterprise, tables are also created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](https://docs.lancedb.com/sql/ddl/table). ## Table structure and metadata | | | | - | - | | **Columns** | including vectors, blobs, and [computed](https://docs.lancedb.com/feature-engineering/add-computed-column.md) columns | | **Versions** | every write commits one, so reads can [travel back](https://docs.lancedb.com/management/table.md#versioning) | | **Indexes** | vector, full-text and scalar, built over its columns | ## Create a table ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} tbl = db.create_table("documents", data) tbl = db.create_table("documents", schema=schema) # empty, typed ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const tbl = await db.createTable("documents", data); const tbl = await db.createEmptyTable("documents", schema); ``` ```sql SQL theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} CREATE TABLE documents (id BIGINT, title VARCHAR) ``` `data` can be a list of objects, an Arrow table or reader, a pandas or polars frame, or Pydantic models. [Ingesting data](https://docs.lancedb.com/ingestion.md) covers each, plus `exist_ok` and `mode="overwrite"` for re-runnable scripts. > [!WARNING] > > An explicit schema is where nullability is decided. In the embedded library, > creation is also the only point at which a table can be given **stable row > ids**; in LanceDB Enterprise, whether a new table gets them is the deployment's > setting. See [Limits](https://docs.lancedb.com/basics/limits.md#schema). ## Open and list tables ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} tbl = db.open_table("documents") db.table_names() ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const tbl = await db.openTable("documents"); await db.tableNames(); ``` Each accepts `namespace_path` to act inside a [namespace](https://docs.lancedb.com/management/namespace.md) rather than at the database root. For Python remote connections created with `lancedb.connect("db://...")`, use `db.table_exists(["documents"])` to check whether a table exists. For an async remote connection, use `await db.table_exists(["documents"])`. Include namespace segments before the table name, for example `["analytics", "documents"]`. These checks return `False` when the table is missing and propagate other errors, including permission errors. Local synchronous directory connections still raise `NotImplementedError` for `table_exists`. On a connection from `connect_namespace("dir", ...)` or `connect_namespace("rest", ...)`, it fails with an `AttributeError`. For these connections, open the table and handle the error, or look for its name in a listing. A missing table raises `ValueError` on a local directory and `lance_namespace.TableNotFoundError` on a connection from `connect_namespace`. A listing returns one page of names at a time, so check every page. TypeScript has no `tableExists`. ## Rename and clone tables ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} db.rename_table("documents", "docs") db.clone_table( "docs_snapshot", source_uri="s3://bucket/db/documents.lance", source_version=42, # or source_tag="release-1" is_shallow=True, ) ``` Renaming is not supported on a local directory connection. In LanceDB Enterprise, whether a table can be renamed or cloned at all depends on how the deployment stores its catalog. A **shallow** clone (the default) references the source's data files rather than copying them, so it is fast and cheap regardless of table size. It is the practical way to branch an experiment off a known version, or to pin a training set to the exact version a model saw. > [!WARNING] > > A shallow clone depends on the source's files continuing to exist. If the source > is [compacted](https://docs.lancedb.com/maintenance/compaction.md) past the retention window, the version > the clone points at can be pruned. LanceDB Embedded connections support shallow clones > only; `is_shallow=False` is not implemented. If the copy must outlive the > source's history, copy its data into an independent table instead. A deployment > that supports cloning also makes shallow clones only, and one whose catalog > does not support it refuses every clone. > [!NOTE] > > **Enterprise job type:** on an Enterprise cluster that drops tables in the background, > the teardown runs as a job: `drop_table`, which fences the table's directory > with a marker and deletes its content, or `drop_object`, depending on how the > deployment stores its catalog. See > [Object teardown](https://docs.lancedb.com/basics/jobs.md#teardown). ## Drop a table ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} db.drop_table("documents") job = db.drop_table_async("documents") db.drop_all_tables() ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} await db.dropTable("documents"); const job = await db.dropTableAsync("documents"); await db.dropAllTables(); ``` ```sql SQL theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} DROP TABLE documents ``` > [!WARNING] > > Dropping removes the table and **every version of it**. > [Versioning](https://docs.lancedb.com/management/table.md#versioning) does not survive a drop, and neither > `drop_all_tables` nor `drop_database` asks for confirmation. Both work only on > a local database; on an Enterprise connection the SDK raises instead. On a large table the drop can take a while, so `drop_table_async` returns a [job](https://docs.lancedb.com/basics/jobs.md) rather than blocking when the deployment drops tables in the background. In the embedded library, and on an Enterprise cluster that drops inline, the returned job has already finished. ## Table management APIs | | Python | TypeScript | SQL | | - | - | - | - | | create | `create_table` | `createTable` | `CREATE TABLE` | | create empty | `create_table(schema=)` | `createEmptyTable` | `CREATE TABLE` | | open | `open_table` | `openTable` | -- | | list | `table_names`, `list_tables` | `tableNames`, `listTables` | -- | | exists | `table_exists` | -- | -- | | rename | `rename_table` | `renameTable` | -- | | clone | `clone_table` | `cloneTable` | -- | | drop | `drop_table`, `drop_table_async` | `dropTable`, `dropTableAsync` | `DROP TABLE` | | drop all | `drop_all_tables` | `dropAllTables` | -- | ## Data types A table's schema is an **Arrow schema**: there is no separate LanceDB type system mapped onto storage, so a column is the same object to the query engine as it is to the format. | Category | Types | | - | - | | Numeric | `int8`-`int64` and unsigned, `float16`, `float32`, `float64` | | String and binary | `utf8`, `large_utf8`, `binary`, `large_binary`, `fixed_size_binary(n)` | | Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp`, `duration` | | Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` | | Lance extensions | vectors, [blobs](#multimodal), bfloat16 | A vector column is a `fixed_size_list`, an ordinary Arrow type, which is why it is selectable and joinable but why arithmetic over it is list arithmetic rather than scalar. See [SQL data types](https://docs.lancedb.com/sql/data-types.md) for the full reference, including the extension types and pseudo-columns. ## Multimodal data Store images, audio, video, and documents alongside structured columns and embeddings in the same versioned table. Use `blob` columns for these payloads. Small values stay inline in the data file, up to 64 KiB by default, and larger values are stored outside it. ### Blob columns Early access A `blob` column keeps metadata separate from the payload bytes, so queries can filter rows without loading the blobs. Most queries return descriptors, which contain metadata about the blob instead of its bytes. You can [read the bytes in the query results](#multimodal-blob-query-results) or [fetch them by row ID](#multimodal-read-blobs). In Python, plain `to_pandas()` scans can also return `BlobFile` handles when `pylance` is installed. When you write raw bytes, Lance chooses a storage layout based on the payload size. A descriptor's `kind` tells you which one was used: | Storage mode | `kind` | Default placement | Storage behavior | | - | - | - | - | | `Inline` | `0` | Up to 64 KiB | Stored in the main data file. | | `Packed` | `1` | Above 64 KiB, up to 4 MiB | Packed with other blobs into a shared sidecar file, up to 1 GiB per file by default. | | `Dedicated` | `2` | Above 4 MiB | Stored in its own raw sidecar file. | | `Managed` | `4` or `5` | Out-of-line values when managed storage is enabled | Stored in a Lance-owned object independently of the data file. | Managed storage must be enabled separately. See [Managed blobs](#managed-blobs) for details. `kind=5` identifies a Managed object using an explicitly registered base. Lance uses the default thresholds unless you customize them. Explicitly configured thresholds are stored in the table schema and preserved on append. #### Setup Import the SDK and open a local database. The examples in this section use `db`. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import lancedb db = lancedb.connect("./blob_db") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import * as arrow from "apache-arrow"; import { Buffer } from "node:buffer"; import * as lancedb from "@lancedb/lancedb"; const db = await lancedb.connect("./blob_db"); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} use std::sync::Arc; use arrow_array::{ Array, Int64Array, LargeBinaryArray, RecordBatch, RecordBatchIterator, StringArray, UInt64Array, }; use arrow_schema::{DataType, Field, Schema}; use futures_util::TryStreamExt; use lancedb::blob::{BlobRangeRequest, blob}; use lancedb::connect; use lancedb::database::CreateTableMode; use lancedb::query::{ExecutableQuery, QueryBase}; let db = connect("./blob_db").execute().await.unwrap(); ``` #### Define a blob schema Declare a blob column with `lancedb.blob()` in Python and TypeScript, or `blob()` in Rust: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import lancedb import pyarrow as pa schema = pa.schema( [ pa.field("id", pa.int64()), pa.field("label", pa.string()), lancedb.blob("image"), ] ) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const blobSchema = new arrow.Schema([ new arrow.Field("id", new arrow.Int64()), new arrow.Field("label", new arrow.Utf8()), lancedb.blob("image"), ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let blob_schema = Arc::new(Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new("label", DataType::Utf8, true), blob("image", true), ])); ``` #### Ingest blobs Pass raw bytes for each blob value. In Python, a blob column accepts `bytes`. In TypeScript, it accepts a `Buffer`. The Python example creates two JPEGs with EXIF metadata, which we'll read using a blob handle below. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import io from PIL import Image def jpeg_with_make(color, make): image = Image.new("RGB", (8, 8), color=color) exif = Image.Exif() exif[0x010F] = make buf = io.BytesIO() image.save(buf, format="JPEG", exif=exif) return buf.getvalue() tbl = db.create_table( "photos", data=[ {"id": 1, "label": "red", "image": jpeg_with_make("red", "TestCam")}, {"id": 2, "label": "blue", "image": jpeg_with_make("blue", "OtherCam")}, ], schema=schema, mode="overwrite", ) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const blobTable = await db.createEmptyTable("photos", blobSchema, { mode: "overwrite", }); await blobTable.add([ { id: 1n, label: "door", image: Buffer.from("inline_image_bytes") }, { id: 2n, label: "window", image: Buffer.from("second_image_bytes") }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let blob_table = db .create_empty_table("photos", blob_schema) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); let input_schema = Arc::new(Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new("label", DataType::Utf8, true), Field::new("image", DataType::LargeBinary, true), ])); let input_batch = RecordBatch::try_new( input_schema.clone(), vec![ Arc::new(Int64Array::from(vec![1_i64, 2_i64])), Arc::new(StringArray::from(vec![Some("door"), Some("window")])), Arc::new(LargeBinaryArray::from_iter_values([ b"inline_image_bytes".as_slice(), b"second_image_bytes".as_slice(), ])), ], ) .unwrap(); let input_reader: Box = Box::new( RecordBatchIterator::new(vec![Ok(input_batch)].into_iter(), input_schema), ); blob_table.add(input_reader).execute().await.unwrap(); ``` #### Read bytes in query results Python and TypeScript can return blob bytes in query results without collecting `_rowid`. Rust uses [`fetch_blobs`](#multimodal-read-blobs) with row IDs instead. ##### Python `to_list(blob_mode="bytes")` returns the `image` bytes from the `photos` table above: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} rows = ( tbl.search().where("id = 1").select(["id", "image"]).to_list(blob_mode="bytes") ) ``` In the Python SDK, `Table.to_pandas()`, `Table.to_arrow()`, and the query builder methods `to_pandas()`, `to_arrow()`, `to_list()`, and `to_batches()` accept a `blob_mode` argument. Sync and async query builders support the same modes: | `blob_mode` | `to_pandas()` | `to_arrow()` and `to_list()` | | - | - | - | | `"lazy"` | Default. Plain scans return seekable `BlobFile` handles. Vector, full-text, hybrid, and ordered queries return descriptor dicts. | Default. Returns descriptors carrying `_lance_row_id` so the result table can be passed to `fetch_blobs` or `fetch_blob_files`. | | `"descriptions"` | Returns descriptor dicts with `kind`, `position`, `size`, `blob_id`, and `blob_uri`. | Returns the 5-field descriptor without `_lance_row_id`. | | `"bytes"` | Materializes each blob as `bytes` in the DataFrame. | Materializes each blob as `large_binary` in Arrow or `bytes` in Python dicts. | For supported local v2 blob projections, `to_pandas(blob_mode="bytes")` and `to_pandas(blob_mode="descriptions")` use native Arrow conversion without pylance. Pandas conversions that need the Lance scanner, including plain-scan `"lazy"` handles and legacy blobs, still require pylance. `to_arrow()`, `to_list()`, `to_batches()`, and the fetch methods do not require pylance. `blob_mode="bytes"` requires a LanceDB Embedded table snapshot, so Enterprise and in-memory tables raise `NotImplementedError`. On an Enterprise table, `Table.to_pandas()` and `Table.to_arrow()` aren't supported, and a query's `to_pandas()` also raises in the default `"lazy"` mode because it would open `BlobFile` handles. Query with `blob_mode="descriptions"`, then read bytes with `fetch_blobs` or `fetch_blob_files`. `blob_mode="bytes"` materializes top-level and struct-nested blob columns. It raises `ValueError` when the selected columns include blobs inside a list. Select those columns with `blob_mode="descriptions"`, or leave them out of the projection. Extra keyword arguments on `to_pandas()`, such as `split_blocks` and `self_destruct`, are forwarded to PyArrow. ###### Stream blob results `to_batches(blob_mode="bytes")` fetches blob v2 payloads one batch at a time and returns them as Arrow `large_binary` values. Iterate over the reader and call `batch.to_pandas()` to convert each batch to a DataFrame without collecting all payloads at once. For sync queries, call `query.to_batches(batch_size=64, blob_mode="bytes")` and iterate with `for`. For async queries, call `await query.to_batches(max_batch_length=64, blob_mode="bytes")` and iterate with `async for`. Sync full-text and hybrid queries do not support `to_batches()`. The default `"lazy"` mode returns descriptors and keeps automatic `_rowid` columns visible for manual fetches. `"descriptions"` returns the five-field descriptors. Both `"bytes"` and `"descriptions"` hide automatic row IDs; explicitly requested row IDs remain visible. Bytes mode has the same restrictions as `to_arrow(blob_mode="bytes")`, including no legacy blobs, blobs inside lists, Enterprise tables, or namespace `QueryTable` pushdown. If concurrent writers can overwrite the table, use a separate reader checked out to a fixed version before querying. Otherwise, an overwrite can invalidate the row IDs used to fetch bytes and raise `ValueError`. ###### Optional pandas examples The lazy-handle calls below require pylance. The query example expects `tbl_async`, `query_vector`, and a `video` column, which the `photos` table does not define. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} df_lazy = tbl.to_pandas() df_bytes = tbl.to_pandas(blob_mode="bytes") df_desc = tbl.to_pandas(blob_mode="descriptions") arrow_bytes = tbl.to_arrow(blob_mode="bytes") df_typed = tbl.to_pandas(split_blocks=True, self_destruct=True) ``` Sync and async query builders accept the same `blob_mode` values: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} df_lazy = ( tbl.search().where("id = 1").select(["id", "video"]).to_pandas(blob_mode="lazy") ) df_bytes = await ( tbl_async.query() .where("id = 1") .select(["id", "video"]) .to_pandas(blob_mode="bytes") ) df_vec = ( tbl.search(query_vector) .limit(2) .select(["id", "video"]) .to_pandas(blob_mode="bytes", split_blocks=True, self_destruct=True) ) ``` ##### TypeScript In TypeScript, `toArray()` and `toArrow()` accept a `blobMode` option: | `blobMode` | Result | | - | - | | `"descriptions"` | Default. Returns descriptors. | | `"bytes"` | Returns each top-level blob as bytes, or `null` for a null blob. Blobs nested in a struct or list keep their descriptors. | With `"bytes"`, the rows and the bytes come from the same table version. If the table changes while the query runs, the query runs again, up to three times: ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const withBytes = await blobTable .query() .select(["id", "image"]) .toArray({ blobMode: "bytes" }); const firstImage: Uint8Array | null = withBytes[0].image; ``` #### Read blobs on demand Query the rows you want and pass their `_rowid` values to a fetch method. `fetch_blob_files` opens a seekable handle and loads bytes only when you read. The Python example opens the JPEG from the ingest step and reads its EXIF Make tag with Pillow. TypeScript and Rust read the first bytes from the handle. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} from PIL import ExifTags rows = tbl.search().where("id = 1").select(["label"]).with_row_id(True).to_list() with tbl.fetch_blob_files("image", [rows[0]["_rowid"]])[0] as image_file: make = Image.open(image_file).getexif().get(ExifTags.Base.Make) print(rows[0]["label"], make) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const rows = await blobTable .query() .select(["id", "label"]) .withRowId() .toArray(); const rowIds = rows.map((row) => row._rowid as bigint); const [image] = await blobTable.fetchBlobFiles("image", [rowIds[0]!]); const header = await image!.readRange(0n, 6n); await image!.close(); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let batches = blob_table .query() .with_row_id() .execute() .await .unwrap() .try_collect::>() .await .unwrap(); let row_ids = batches[0] .column_by_name("_rowid") .unwrap() .as_any() .downcast_ref::() .unwrap() .values() .to_vec(); let handles = blob_table .fetch_blob_files("image", &row_ids[..1]) .await .unwrap(); let header = handles[0].as_ref().unwrap().read_range(0..6).await.unwrap(); handles[0].as_ref().unwrap().close().await.unwrap(); ``` The other fetch methods load bytes directly: | Need | Python | TypeScript | Rust | | - | - | - | - | | Whole payloads for a set of rows | `fetch_blobs` | `fetchBlobs` | `fetch_blobs` | | Byte slices across many rows in one call | `fetch_blob_ranges` | Not available. Use `fetchBlobFiles` and `readRanges`. | `fetch_blob_ranges` | * `fetch_blobs` returns the full bytes for each row ID in order. Null blobs produce null entries, and empty blobs produce empty byte strings. * `fetch_blob_files` loads bytes when you call `read`, `read_range`, or `read_ranges`. Range reads don't move the file cursor. Null blobs produce null handles. * `fetch_blob_ranges` takes `(row_id, offset, length)` requests and returns one slice per request, in order. A request list can repeat a row to read more than one slice of that blob. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} rows = tbl.search().select(["id"]).with_row_id(True).to_list() row_ids = [row["_rowid"] for row in rows] payloads = tbl.fetch_blobs("image", row_ids) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const payloads = await blobTable.fetchBlobs("image", rowIds); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let payloads = blob_table.fetch_blobs("image", &row_ids).await.unwrap(); ``` Python and Rust read slices across rows with `fetch_blob_ranges`. On a handle, `read_ranges` reads more than one slice of one blob. Python takes `(offset, length)` pairs, TypeScript takes `{ start, end }` ranges with an exclusive `end`, and Rust takes `start..end` ranges: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} slices = tbl.fetch_blob_ranges("image", [(row_ids[0], 0, 2)]) with tbl.fetch_blob_files("image", [row_ids[0]])[0] as image_file: parts = image_file.read_ranges([(0, 2)]) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const [secondImage] = await blobTable.fetchBlobFiles("image", [rowIds[1]!]); const parts = await secondImage!.readRanges([ { start: 0n, end: 6n }, { start: 7n, end: 12n }, ]); await secondImage!.close(); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let slices = blob_table .fetch_blob_ranges( "image", [ BlobRangeRequest::new(row_ids[0], 0, 6), BlobRangeRequest::new(row_ids[1], 0, 6), BlobRangeRequest::new(row_ids[1], 7, 5), ], ) .await .unwrap(); let second_handles = blob_table .fetch_blob_files("image", &row_ids[1..2]) .await .unwrap(); let second_image = second_handles[0].as_ref().unwrap(); let parts = second_image.read_ranges(&[0..6, 7..12]).await.unwrap(); second_image.close().await.unwrap(); ``` #### Notes and limits Blob fetches fail if any requested row ID is missing or deleted. With stable row IDs enabled, IDs remain valid across compaction. A blob field can sit at the top level of the schema or inside a struct or list. Struct-nested blob columns use dotted paths such as `clip.video` in the fetch APIs. Blobs inside lists can be written and queried as descriptors. A table row ID does not address one list element, so `blob_columns()`, `fetch_blobs`, `fetch_blob_files`, and `blob_mode="bytes"` can't return those bytes. See the Lance [blob guide](https://lance.org/guide/blob/) for the storage format. ### Managed blobs (Blob v2) Set the table schema metadata `lance.blob.managed` to `true` on a table with [`lancedb.blob()`](#multimodal-blob-v2) columns to store small values inline and larger values in managed objects under the table's `_blobs/` directory. Medium values can share a packed object, and large values use a dedicated object. The table owns these objects. You do not upload them separately, keep an object catalog, or delete their files yourself. Compaction preserves each descriptor and its payload object, so merging table fragments does not rewrite the blob payloads. > [!WARNING] > > Use an SDK and Enterprise deployment that support Blob v2 Managed storage. > Upgrade every reader, writer, and maintenance worker before activating Managed > blobs on a table, and reopen existing table handles after the upgrade. Older > clients cannot safely maintain a table containing Managed blobs. #### Write and read blobs This example uses a local connection. The same schema, `add`, query, and `fetch_blobs` calls work with an Enterprise database connection: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import pyarrow as pa db = lancedb.connect("./media") table = db.create_table( "images", schema=pa.schema([ pa.field("id", pa.int64(), nullable=False), lancedb.blob("image"), ], metadata={"lance.blob.managed": "true"}), ) table.add([ {"id": 1, "image": b""}, {"id": 2, "image": bytes(range(256)) * 400}, {"id": 3, "image": None}, ]) # Query descriptors first, then fetch only the selected rows' payloads. hits = ( table.search() .where("id <= 2") .select(["id", "image"]) .with_row_id(True) .to_arrow() ) images = table.fetch_blobs("image", hits).to_pylist() by_id = dict(zip(hits["id"].to_pylist(), images, strict=True)) assert by_id[1] == b"" assert len(by_id[2]) == 102400 ``` `fetch_blobs` materializes the selected payloads in memory, so fetch bounded batches when blobs are large. Row IDs belong to their table and snapshot. Keep descriptors opaque in application code, because their representation can differ between inline values and Managed objects. #### Derived blobs and maintenance Enterprise Functions can consume blob columns and return new blobs. Their output payloads are managed by the destination table. LanceDB Embedded tables can also derive blob columns with [SQL computed columns](https://docs.lancedb.com/feature-engineering/add-computed-column.md). Deleting a row or replacing its blob does not immediately remove the old payload: retained table versions may still reference it. [Garbage Collection](https://docs.lancedb.com/maintenance/cleanup.md#managed-blobs) reclaims objects after those references expire. A packed object remains while any of its contained blobs is still referenced. ### Legacy large-binary blobs Use [`lancedb.blob()`](#multimodal-blob-v2) for new tables. The sections below cover existing tables that use the V1 large-binary Blob API (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust with `{"lance-encoding:blob": "true"}` field metadata). Changing a field's metadata alone does not migrate an existing V1 column to Blob v2. #### 1. Define a blob schema V1 blob columns use a `large_binary` field marked with `{"lance-encoding:blob": "true"}` metadata: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import pyarrow as pa # Define schema with Blob API metadata for lazy loading schema = pa.schema( [ pa.field("id", pa.int64()), pa.field( "video", pa.large_binary(), metadata={"lance-encoding:blob": "true"}, # Enable Blob API ), ] ) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const legacyBlobSchema = new arrow.Schema([ new arrow.Field("id", new arrow.Int64()), new arrow.Field( "video", new arrow.LargeBinary(), true, new Map([["lance-encoding:blob", "true"]]), ), ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let legacy_blob_metadata = HashMap::from([( "lance-encoding:blob".to_string(), "true".to_string(), )]); let legacy_blob_schema = Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new("video", DataType::LargeBinary, true).with_metadata(legacy_blob_metadata), ]); ``` #### 2. Ingest large blobs Write raw bytes to a V1 blob column: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import lancedb db = lancedb.connect("./blob_db") # Create sample data data = [ {"id": 1, "video": b"fake_video_bytes_1"}, {"id": 2, "video": b"fake_video_bytes_2"}, ] # Create the table tbl = db.create_table("videos", data=data, schema=schema) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const legacyBlobData = lancedb.makeArrowTable( [ { id: 1, video: Buffer.from("fake_video_bytes_1") }, { id: 2, video: Buffer.from("fake_video_bytes_2") }, ], { schema: legacyBlobSchema }, ); const legacyBlobTable = await db.createTable( "legacy_videos", legacyBlobData, { mode: "overwrite", }, ); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let legacy_blob_rows = vec![ (1_i64, b"fake_video_bytes_1".to_vec()), (2_i64, b"fake_video_bytes_2".to_vec()), ]; let legacy_blob_schema = Arc::new(legacy_blob_schema); let legacy_blob_batch = RecordBatch::try_new( legacy_blob_schema.clone(), vec![ Arc::new(Int64Array::from_iter_values( legacy_blob_rows.iter().map(|row| row.0), )), Arc::new(LargeBinaryArray::from_iter_values( legacy_blob_rows.iter().map(|row| row.1.as_slice()), )), ], ) .unwrap(); let legacy_blob_reader: Box = Box::new( RecordBatchIterator::new(vec![Ok(legacy_blob_batch)].into_iter(), legacy_blob_schema), ); let legacy_blob_table = db .create_table("legacy_videos", legacy_blob_reader) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); ``` #### 3. Convert blob tables to pandas On LanceDB Embedded tables with V1 blob columns, `to_pandas()` in Python accepts `blob_mode` (`"lazy"`, `"bytes"`, or `"descriptions"`). Enterprise tables raise `NotImplementedError`, and `"bytes"` is not supported on in-memory tables. Extra keyword arguments such as `split_blocks` or `self_destruct` are forwarded to the PyArrow / Lance pandas conversion: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Default: blob columns come back lazily df_lazy = tbl.to_pandas() # Materialize blob bytes eagerly df_bytes = tbl.to_pandas(blob_mode="bytes") # Return descriptors instead of payloads df_desc = tbl.to_pandas(blob_mode="descriptions") # Forward extra kwargs to PyArrow's to_pandas df_typed = tbl.to_pandas(split_blocks=True, self_destruct=True) ``` On query builders with V1 blob columns, plain scans support `"lazy"`, `"bytes"`, and `"descriptions"`. Vector, FTS, hybrid, and ordered queries cannot materialize V1 blob columns through `to_pandas()`, so omit those columns from the projection: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Plain scan query: blob_mode is supported end to end df_lazy = ( tbl.search() .where("id = 1") .select(["id", "video"]) .to_pandas(blob_mode="lazy") ) # Same call shape works on async query builders df_bytes = await ( tbl_async.query() .where("id = 1") .select(["id", "video"]) .to_pandas(blob_mode="bytes") ) # Vector / FTS / hybrid queries can't materialize blob columns, # so omit them from the projection df_vec = ( tbl.search(query_vector) .limit(10) .select(["id", "vector"]) .to_pandas(split_blocks=True, self_destruct=True) ) ``` ### Store binary data Use a `binary` column when the query should return the raw bytes. Declare it as `pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust. #### 1. Setup and imports Import LanceDB and Arrow in your SDK. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import io import lancedb import numpy as np import pyarrow as pa from PIL import Image ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import * as arrow from "apache-arrow"; import { Buffer } from "node:buffer"; import * as lancedb from "@lancedb/lancedb"; ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} use std::collections::HashMap; use std::sync::Arc; use arrow_array::types::Float32Type; use arrow_array::{ Array, BinaryArray, FixedSizeListArray, Int32Array, Int64Array, LargeBinaryArray, RecordBatch, RecordBatchIterator, StringArray, }; use arrow_schema::{DataType, Field, Schema}; use futures_util::TryStreamExt; use lancedb::connect; use lancedb::database::CreateTableMode; use lancedb::query::{ExecutableQuery, QueryBase}; ``` #### 2. Prepare data Encode each image or file into raw bytes before writing. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} def create_dummy_image(color): img = Image.new("RGB", (100, 100), color=color) buf = io.BytesIO() img.save(buf, format="PNG") return buf.getvalue() data = [ { "id": 1, "filename": "red_square.png", "vector": np.random.rand(128).astype(np.float32), "image_blob": create_dummy_image("red"), "label": "red", }, { "id": 2, "filename": "blue_square.png", "vector": np.random.rand(128).astype(np.float32), "image_blob": create_dummy_image("blue"), "label": "blue", }, ] ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const createDummyImage = (color: string): Uint8Array => { const pngHeader = Uint8Array.from([137, 80, 78, 71, 13, 10, 26, 10]); return Buffer.concat([Buffer.from(pngHeader), Buffer.from(color, "utf8")]); }; const data = [ { id: 1, filename: "red_square.png", vector: Array.from({ length: 128 }, (_, i) => (i % 16) / 16), image_blob: createDummyImage("red"), label: "red", }, { id: 2, filename: "blue_square.png", vector: Array.from({ length: 128 }, (_, i) => ((i + 8) % 16) / 16), image_blob: createDummyImage("blue"), label: "blue", }, ]; ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let create_dummy_image = |color: u8| -> Vec { let mut png_like = vec![137, 80, 78, 71, 13, 10, 26, 10]; png_like.push(color); png_like }; let data = vec![ ( 1_i32, "red_square.png", vec![0.1_f32; 128], create_dummy_image(1), "red", ), ( 2_i32, "blue_square.png", vec![0.2_f32; 128], create_dummy_image(2), "blue", ), ]; ``` #### 3. Define the schema Declare the schema explicitly so Arrow and LanceDB store the field as `binary` instead of inferring a string or integer list. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} schema = pa.schema( [ pa.field("id", pa.int32()), pa.field("filename", pa.string()), pa.field("vector", pa.list_(pa.float32(), 128)), pa.field("image_blob", pa.binary()), pa.field("label", pa.string()), ] ) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const schema = new arrow.Schema([ new arrow.Field("id", new arrow.Int32()), new arrow.Field("filename", new arrow.Utf8()), new arrow.Field( "vector", new arrow.FixedSizeList( 128, new arrow.Field("item", new arrow.Float32(), true), ), ), new arrow.Field("image_blob", new arrow.Binary()), new arrow.Field("label", new arrow.Utf8()), ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let schema = Schema::new(vec![ Field::new("id", DataType::Int32, false), Field::new("filename", DataType::Utf8, false), Field::new( "vector", DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 128), false, ), Field::new("image_blob", DataType::Binary, false), Field::new("label", DataType::Utf8, false), ]); ``` #### 4. Ingest data Create the table from the rows and schema. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} tbl = db.create_table("images", data=data, schema=schema, mode="overwrite") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const multimodalData = lancedb.makeArrowTable(data, { schema }); const tbl = await db.createTable("images", multimodalData, { mode: "overwrite", }); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let schema = Arc::new(schema); let image_batch = RecordBatch::try_new( schema.clone(), vec![ Arc::new(Int32Array::from_iter_values(data.iter().map(|row| row.0))), Arc::new(StringArray::from_iter_values(data.iter().map(|row| row.1))), Arc::new( FixedSizeListArray::from_iter_primitive::( data.iter() .map(|row| Some(row.2.iter().copied().map(Some).collect::>())), 128, ), ), Arc::new(BinaryArray::from_iter_values( data.iter().map(|row| row.3.as_slice()), )), Arc::new(StringArray::from_iter_values(data.iter().map(|row| row.4))), ], ) .unwrap(); let image_reader: Box = Box::new( RecordBatchIterator::new(vec![Ok(image_batch)].into_iter(), schema.clone()), ); let table = db .create_table("images", image_reader) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); ``` ### Retrieve binary data When you search the table, select the `binary` column alongside your metadata columns. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} query_vector = np.random.rand(128).astype(np.float32) results = tbl.search(query_vector).limit(1).to_pandas() ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const queryVector = Array.from({ length: 128 }, (_, i) => (i % 16) / 16); const results = await tbl.search(queryVector).limit(1).toArray(); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let query_vector = vec![0.1_f32; 128]; let results = table .query() .nearest_to(query_vector) .unwrap() .limit(1) .execute() .await .unwrap() .try_collect::>() .await .unwrap(); ``` #### Convert bytes back to objects Decode the returned bytes into an image or media buffer in your application. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} for _, row in results.iterrows(): image_bytes = row["image_blob"] image = Image.open(io.BytesIO(image_bytes)) print(f"Retrieved image: {row['filename']}, Size: {image.size}") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} for (const row of results) { const imageBytes = row.image_blob as Uint8Array; console.log( `Retrieved image: ${row.filename}, Byte length: ${imageBytes.length}`, ); } ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} for batch in &results { let filenames = batch .column_by_name("filename") .unwrap() .as_any() .downcast_ref::() .unwrap(); let images = batch .column_by_name("image_blob") .unwrap() .as_any() .downcast_ref::() .unwrap(); for row in 0..batch.num_rows() { let image_bytes = images.value(row); println!( "Retrieved image: {}, Byte length: {}", filenames.value(row), image_bytes.len() ); } } ``` ### Other modalities For PDFs, audio recordings, and other files, keep searchable fields in separate columns so you can filter rows before fetching blob bytes. ### bfloat16 Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`. Useful as a vector element type when halving the memory matters more than the precision does. ## Schema evolution Schema evolution enables non-breaking modifications to a database table's structure (such as adding columns, altering data types, or dropping fields) to adapt to evolving data requirements without service interruptions. LanceDB supports ACID-compliant schema evolution through granular operations (add/alter/drop columns), allowing you to: * Iterate Safely: Modify schemas in production with versioned datasets and backward compatibility * Scale Seamlessly: Handle ML model iterations, regulatory changes, or feature additions * Optimize Continuously: Remove unused fields or enforce new constraints without downtime > [!NOTE] > > In SQL, schema changes are [`ALTER TABLE`](https://docs.lancedb.com/sql/ddl/table#alter). ### Schema evolution operations LanceDB supports four primary schema evolution operations: 1. **Adding new columns**: Extend your table with additional attributes 2. **Altering existing columns**: Change column names, data types, or nullability 3. **Updating field metadata**: Attach or change per-column Arrow metadata 4. **Dropping columns**: Remove unnecessary columns from your schema > [!TIP] > > Schema evolution operations are applied immediately but do not typically require rewriting all data. However, data type changes may involve more substantial operations. Each schema evolution operation commits a new table version and returns status metadata such as the committed `version`. Run these operations from a mutable table handle; if you checked out an older version for reads, call `checkout_latest` / `checkoutLatest` before modifying the schema. ### Add new columns You can add new columns to a table with the [`add_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.add_columns) method in Python, [`addColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#addcolumns) in TypeScript/JavaScript, or `add_columns` in Rust. New columns are populated based on SQL expressions you provide. On local tables, the Python SDK also accepts precomputed values from DataFrames and Arrow readers; see [Create a computed column](https://docs.lancedb.com/feature-engineering/add-computed-column.md#computed-vs-transforms) for the supported inputs and row-order requirements. #### Set up the example table First, let's create a sample table with product data to demonstrate schema evolution: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import numpy as np data = [ { "id": 1, "name": "Laptop", "price": 1200.00, "vector": np.random.random(128).tolist(), }, { "id": 2, "name": "Smartphone", "price": 800.00, "vector": np.random.random(128).tolist(), }, { "id": 3, "name": "Headphones", "price": 150.00, "vector": np.random.random(128).tolist(), }, ] table = db.create_table("schema_evolution_add_example", data, mode="overwrite") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const schemaAddData = [ { id: 1, name: "Laptop", price: 1200.0, vector: Array.from({ length: 128 }, () => Math.random()), }, { id: 2, name: "Smartphone", price: 800.0, vector: Array.from({ length: 128 }, () => Math.random()), }, { id: 3, name: "Headphones", price: 150.0, vector: Array.from({ length: 128 }, () => Math.random()), }, ]; const schemaAddTable = await db.createTable( "schema_evolution_add_example", schemaAddData, { mode: "overwrite" }, ); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let schema_add_schema = Arc::new(Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new("name", DataType::Utf8, false), Field::new("price", DataType::Float64, false), Field::new( "vector", DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 128), false, ), ])); let schema_add_batch = RecordBatch::try_new( schema_add_schema.clone(), vec![ Arc::new(Int64Array::from(vec![1, 2, 3])), Arc::new(StringArray::from(vec!["Laptop", "Smartphone", "Headphones"])), Arc::new(Float64Array::from(vec![1200.0, 800.0, 150.0])), Arc::new( FixedSizeListArray::from_iter_primitive::( vec![ Some(vec![Some(0.1_f32); 128]), Some(vec![Some(0.2_f32); 128]), Some(vec![Some(0.3_f32); 128]), ], 128, ), ), ], ) .unwrap(); let schema_add_reader: Box = Box::new(RecordBatchIterator::new( vec![Ok(schema_add_batch)].into_iter(), schema_add_schema.clone(), )); let schema_add_table = db .create_table("schema_evolution_add_example", schema_add_reader) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); ``` > [!NOTE] > > Columns derived from existing data (SQL expressions over other columns) are > covered in [Create a computed column](https://docs.lancedb.com/feature-engineering/add-computed-column.md), > because deriving values is a feature-engineering job rather than a schema one. #### Add columns with default values Add boolean columns with default values for status tracking: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Add a stock status column with default value table.add_columns({"in_stock": "cast(true as boolean)"}) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Add a stock status column with default value await schemaAddTable.addColumns([ { name: "in_stock", valueSql: "cast(true as boolean)", }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Add a stock status column with default value schema_add_table .add_columns( NewColumnTransform::SqlExpressions(vec![( "in_stock".to_string(), "cast(true as boolean)".to_string(), )]), None, ) .await .unwrap(); ``` #### Add nullable columns Add timestamp columns that can contain NULL values: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Add a nullable timestamp column table.add_columns({"last_ordered": "cast(NULL as timestamp)"}) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Add a nullable timestamp column await schemaAddTable.addColumns([ { name: "last_ordered", valueSql: "cast(NULL as timestamp)", }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Add a nullable timestamp column schema_add_table .add_columns( NewColumnTransform::SqlExpressions(vec![( "last_ordered".to_string(), "cast(NULL as timestamp)".to_string(), )]), None, ) .await .unwrap(); ``` > [!WARNING] > > When adding columns that should contain NULL values, be sure to cast the NULL to the appropriate type, e.g., `cast(NULL as timestamp)`. ### Alter existing columns You can alter columns using the [`alter_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.alter_columns) method in Python, [`alterColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#altercolumns) in TypeScript/JavaScript, or `alter_columns` in Rust. This allows you to: * Rename a column * Change a column's data type * Modify nullability (whether a column can contain NULL values) #### Set up the example table Create a table with a custom schema to demonstrate column alterations: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table_name = "schema_evolution_alter_example" if data is None: data = [ { "id": 1, "name": "Laptop", "price": 1200, "discount_price": 1080.0, "vector": np.random.random(128).tolist(), }, { "id": 2, "name": "Smartphone", "price": 800, "discount_price": 720.0, "vector": np.random.random(128).tolist(), }, ] schema = pa.schema( { "id": pa.int64(), "name": pa.string(), "price": pa.int32(), "discount_price": pa.float64(), "vector": pa.list_(pa.float32(), 128), } ) table = db.create_table(table_name, data, schema=schema, mode="overwrite") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const schemaAlter = new arrow.Schema([ new arrow.Field("id", new arrow.Int64()), new arrow.Field("name", new arrow.Utf8()), new arrow.Field("price", new arrow.Int32()), new arrow.Field("discount_price", new arrow.Float64()), new arrow.Field( "vector", new arrow.FixedSizeList( 128, new arrow.Field("item", new arrow.Float32(), true), ), ), ]); const schemaAlterData = lancedb.makeArrowTable( [ { id: 1, name: "Laptop", price: 1200, discount_price: 1080.0, vector: Array.from({ length: 128 }, () => Math.random()), }, { id: 2, name: "Smartphone", price: 800, discount_price: 720.0, vector: Array.from({ length: 128 }, () => Math.random()), }, ], { schema: schemaAlter }, ); const schemaAlterTable = await db.createTable( "schema_evolution_alter_example", schemaAlterData, { mode: "overwrite" }, ); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let schema_alter_schema = Arc::new(Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new("name", DataType::Utf8, false), Field::new("price", DataType::Int32, false), Field::new("discount_price", DataType::Float64, false), Field::new( "vector", DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 128), false, ), ])); let schema_alter_batch = RecordBatch::try_new( schema_alter_schema.clone(), vec![ Arc::new(Int64Array::from(vec![1, 2])), Arc::new(StringArray::from(vec!["Laptop", "Smartphone"])), Arc::new(Int32Array::from(vec![1200, 800])), Arc::new(Float64Array::from(vec![1080.0, 720.0])), Arc::new( FixedSizeListArray::from_iter_primitive::( vec![Some(vec![Some(0.1_f32); 128]), Some(vec![Some(0.2_f32); 128])], 128, ), ), ], ) .unwrap(); let schema_alter_reader: Box = Box::new(RecordBatchIterator::new( vec![Ok(schema_alter_batch)].into_iter(), schema_alter_schema.clone(), )); let schema_alter_table = db .create_table("schema_evolution_alter_example", schema_alter_reader) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); ``` #### Rename columns Change column names to better reflect their purpose: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Rename discount_price to sale_price table.alter_columns({"path": "discount_price", "rename": "sale_price"}) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Rename discount_price to sale_price await schemaAlterTable.alterColumns([ { path: "discount_price", rename: "sale_price" }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Rename discount_price to sale_price schema_alter_table .alter_columns(&[ColumnAlteration::new("discount_price".to_string()) .rename("sale_price".to_string())]) .await .unwrap(); ``` #### Change data types Convert column data types for better performance or compatibility: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Change price from int32 to int64 for larger numbers table.alter_columns({"path": "price", "data_type": pa.int64()}) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Change price from int32 to int64 for larger numbers await schemaAlterTable.alterColumns([ { path: "price", dataType: new arrow.Int64() }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Change price from int32 to int64 for larger numbers schema_alter_table .alter_columns(&[ColumnAlteration::new("price".to_string()).cast_to(DataType::Int64)]) .await .unwrap(); ``` #### Make columns nullable You can alter columns to contain NULL values: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Make the name column nullable table.alter_columns({"path": "name", "nullable": True}) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Make the name column nullable await schemaAlterTable.alterColumns([{ path: "name", nullable: true }]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Make the name column nullable schema_alter_table .alter_columns(&[ColumnAlteration::new("name".to_string()).set_nullable(true)]) .await .unwrap(); ``` Changing a column to nullable affects future writes and merges too: missing values are accepted only when the target column is nullable. #### Multiple changes at once Apply several alterations in a single operation: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Rename, change type, and make nullable in one operation table.alter_columns( { "path": "sale_price", "rename": "final_price", "data_type": pa.float64(), "nullable": True, } ) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Rename, change type, and make nullable in one operation await schemaAlterTable.alterColumns([ { path: "sale_price", rename: "final_price", dataType: new arrow.Float64(), nullable: true, }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Rename, change type, and make nullable in one operation schema_alter_table .alter_columns(&[ColumnAlteration::new("sale_price".to_string()) .rename("final_price".to_string()) .cast_to(DataType::Float64) .set_nullable(true)]) .await .unwrap(); ``` #### Expression-based type changes For transformations that are not simple casts (for example, converting `"$100"` to an integer), use a SQL-expression column add, then drop and rename: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # For custom transforms, create a new column from a SQL expression. expression_table = db.create_table( "schema_evolution_expression_example", [{"id": 1, "price_text": "$100"}], mode="overwrite", ) expression_table.add_columns( {"price_numeric": "cast(replace(price_text, '$', '') as int)"} ) expression_table.drop_columns(["price_text"]) expression_table.alter_columns({"path": "price_numeric", "rename": "price"}) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // For custom transforms, create a new column from a SQL expression. const expressionTable = await db.createTable( "schema_evolution_expression_example", [{ id: 1, price_text: "$100" }], { mode: "overwrite" }, ); await expressionTable.addColumns([ { name: "price_numeric", valueSql: "cast(replace(price_text, '$', '') as int)", }, ]); await expressionTable.dropColumns(["price_text"]); await expressionTable.alterColumns([ { path: "price_numeric", rename: "price" }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // For custom transforms, create a new column from a SQL expression. let expression_schema = Arc::new(Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new("price_text", DataType::Utf8, false), ])); let expression_batch = RecordBatch::try_new( expression_schema.clone(), vec![ Arc::new(Int64Array::from(vec![1])), Arc::new(StringArray::from(vec!["$100"])), ], ) .unwrap(); let expression_reader: Box = Box::new(RecordBatchIterator::new( vec![Ok(expression_batch)].into_iter(), expression_schema.clone(), )); let expression_table = db .create_table("schema_evolution_expression_example", expression_reader) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); expression_table .add_columns( NewColumnTransform::SqlExpressions(vec![( "price_numeric".to_string(), "cast(replace(price_text, '$', '') as int)".to_string(), )]), None, ) .await .unwrap(); expression_table.drop_columns(&["price_text"]).await.unwrap(); expression_table .alter_columns(&[ColumnAlteration::new("price_numeric".to_string()) .rename("price".to_string())]) .await .unwrap(); ``` #### Alter embedding types and dimensions It's quite common to need to change an embedding column's schema, in case a new model becomes available with a different embedding dimension. * In Python, the example shows an in-place type update when the cast is compatible. * In TypeScript and Rust, the example shows a dimension change (`384 -> 1024`), which cannot be cast in-place. For dimension changes, use this 3-step pattern: add a new column with the target type, drop the old column, then rename the new column to the original name. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} data = [ { "id": 1, "embedding": np.random.random(vector_dim).tolist(), }, ] table = db.create_table(table_name, data, mode="overwrite") table.alter_columns( dict(path="embedding", data_type=pa.list_(pa.float32(), vector_dim)) ) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const oldDim = 384; const newDim = 1024; const vectorSchema = new arrow.Schema([ new arrow.Field("id", new arrow.Int64()), new arrow.Field( "embedding", new arrow.FixedSizeList( oldDim, new arrow.Field("item", new arrow.Float16(), true), ), true, ), ]); const vectorData = lancedb.makeArrowTable( [{ id: 1, embedding: Array.from({ length: oldDim }, () => Math.random()) }], { schema: vectorSchema }, ); const vectorTable = await db.createTable("vector_alter_example", vectorData, { mode: "overwrite", }); // Changing FixedSizeList dimensions (384 -> 1024) is not supported via alterColumns. // Use addColumns + dropColumns + alterColumns(rename) to replace the column. await vectorTable.addColumns([ { name: "embedding_v2", valueSql: `arrow_cast(NULL, 'FixedSizeList(${newDim}, Float16)')`, }, ]); await vectorTable.dropColumns(["embedding"]); await vectorTable.alterColumns([{ path: "embedding_v2", rename: "embedding" }]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let old_dim = 384; let new_dim = 1024; let vector_schema = Arc::new(Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new( "embedding", DataType::FixedSizeList( Arc::new(Field::new("item", DataType::Float32, true)), old_dim, ), true, ), ])); let vector_batch = RecordBatch::try_new( vector_schema.clone(), vec![ Arc::new(Int64Array::from(vec![1])), Arc::new( FixedSizeListArray::from_iter_primitive::( vec![Some(vec![Some(0.1_f32); old_dim as usize])], old_dim, ), ), ], ) .unwrap(); let vector_reader: Box = Box::new(RecordBatchIterator::new(vec![Ok(vector_batch)].into_iter(), vector_schema.clone())); let vector_table = db .create_table("vector_alter_example", vector_reader) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); // Changing FixedSizeList dimensions (384 -> 1024) is not supported via alter_columns. // Use add_columns + drop_columns + alter_columns(rename) to replace the column. vector_table .add_columns( NewColumnTransform::SqlExpressions(vec![( "embedding_v2".to_string(), format!("arrow_cast(NULL, 'FixedSizeList({}, Float32)')", new_dim), )]), None, ) .await .unwrap(); vector_table.drop_columns(&["embedding"]).await.unwrap(); vector_table .alter_columns(&[ColumnAlteration::new("embedding_v2".to_string()) .rename("embedding".to_string())]) .await .unwrap(); ``` > [!WARNING] > > **`FixedSizeList` Dimension Changes in TypeScript and Rust** > > `alterColumns` / `alter_columns` can cast between compatible types, but changing `FixedSizeList` dimensions (for example `384 -> 1024`) is not a compatible cast. > For such cases, use `addColumns` / `add_columns` (with `arrow_cast`), then `dropColumns` / `drop_columns`, then rename the replacement column. > [!WARNING] > > Changing data types requires rewriting the column data and may be resource-intensive for large tables. Renaming columns or changing nullability is more efficient as it only updates metadata. ### Update field metadata Each column in a LanceDB table can carry a small key/value map of Arrow field metadata, which is useful for annotating columns with units, provenance, PII flags, embedding model versions, or any other schema-level context your application needs. Use [`update_field_metadata`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.update_field_metadata) in Python, [`updateFieldMetadata`](https://lancedb.github.io/lancedb/js/classes/Table/#updatefieldmetadata) in TypeScript/JavaScript, or `update_field_metadata` in Rust to add, change, or remove these key/value pairs without rewriting the column data. Each call commits a new table version and returns the new `version`. Each update targets one field by **dot-path**: top-level columns are addressed by name (for example `"embedding"`), and nested fields by their full path (for example `"address.zip"`). By default, the keys you pass are **merged** into the field's existing metadata: keys you do not mention are preserved, and passing `None` (Python) or `null` (TypeScript) deletes a key. Set `replace: true` to swap the field's entire metadata map instead of merging. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Set two metadata keys on the `category` field. res = table.update_field_metadata( {"path": "category", "metadata": {"unit": "label", "pii": "false"}} ) print(res.version) # Merge: add a new key, delete one with None, keep the rest. table.update_field_metadata( {"path": "category", "metadata": {"source": "import", "pii": None}} ) # Arrow stores field metadata as bytes. assert table.schema.field("category").metadata == { b"unit": b"label", b"source": b"import", } ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Set two metadata keys on the `category` field. const res = await fieldMetadataTable.updateFieldMetadata([ { path: "category", metadata: { unit: "label", pii: "false" } }, ]); console.log(res.version); // Merge: add a new key, delete one via null, keep the rest. await fieldMetadataTable.updateFieldMetadata([ { path: "category", metadata: { source: "import", pii: null } }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Set two metadata keys on the `category` field. let res = field_metadata_table .update_field_metadata(&[FieldMetadataUpdate::new("category") .set("unit", "label") .set("pii", "false")]) .await .unwrap(); println!("version: {}", res.version); // Merge: add a new key, delete one with `.remove`, keep the rest. field_metadata_table .update_field_metadata(&[FieldMetadataUpdate::new("category") .set("source", "import") .remove("pii")]) .await .unwrap(); ``` To overwrite a field's metadata entirely instead of merging, set `replace` to `true`: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.update_field_metadata( { "path": "category", "metadata": {"owner": "search-team"}, "replace": True, } ) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} await fieldMetadataTable.updateFieldMetadata([ { path: "category", metadata: { owner: "search-team" }, replace: true, }, ]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} field_metadata_table .update_field_metadata(&[FieldMetadataUpdate::new("category") .set("owner", "search-team") .replace()]) .await .unwrap(); ``` > [!TIP] > > You can pass multiple updates in a single call to change metadata on several fields at once. > Each call commits a single new table version. #### Canonical metadata keys You can use any string as a metadata key. By convention, LanceDB treats keys prefixed with `lancedb:` as canonical. LanceDB Enterprise displays these keys in the UI, and agents can rely on finding this information in a consistent place. Use them when they fit your use case: | Key | Purpose | Example | | - | - | - | | `lancedb:description` | Human-readable description of the field. | `"CLIP embedding of the product image"` | | `lancedb:tag:` | User-defined key/value tag, where the suffix names the tag category. | `"lancedb:tag:model": "clip"` | | `lancedb:status` | Life cycle state of the column: `production`, `candidate`, `deprecated`, or `archived`. | `"lancedb:status": "production"` | > [!NOTE] > > These keys are conventions, not enforced constraints. LanceDB does not validate their values, > but following the conventions keeps your tables consistent with Enterprise tooling. ### Drop columns You can remove columns using the [`drop_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.drop_columns) method in Python, [`dropColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#dropcolumns) in TypeScript/JavaScript, or `drop_columns` in Rust. #### Set Up the example table Create a table with temporary columns that we'll remove: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} if data is None: data = [ { "id": 1, "name": "Laptop", "price": 1200.00, "temp_col1": "X", "temp_col2": 100, "vector": np.random.random(128).tolist(), }, { "id": 2, "name": "Smartphone", "price": 800.00, "temp_col1": "Y", "temp_col2": 200, "vector": np.random.random(128).tolist(), }, { "id": 3, "name": "Headphones", "price": 150.00, "temp_col1": "Z", "temp_col2": 300, "vector": np.random.random(128).tolist(), }, ] table = db.create_table("schema_evolution_drop_example", data, mode="overwrite") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const schemaDropData = [ { id: 1, name: "Laptop", price: 1200.0, temp_col1: "X", temp_col2: 100, vector: Array.from({ length: 128 }, () => Math.random()), }, { id: 2, name: "Smartphone", price: 800.0, temp_col1: "Y", temp_col2: 200, vector: Array.from({ length: 128 }, () => Math.random()), }, { id: 3, name: "Headphones", price: 150.0, temp_col1: "Z", temp_col2: 300, vector: Array.from({ length: 128 }, () => Math.random()), }, ]; const schemaDropTable = await db.createTable( "schema_evolution_drop_example", schemaDropData, { mode: "overwrite" }, ); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let schema_drop_schema = Arc::new(Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new("name", DataType::Utf8, false), Field::new("price", DataType::Float64, false), Field::new("temp_col1", DataType::Utf8, false), Field::new("temp_col2", DataType::Int32, false), Field::new( "vector", DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 128), false, ), ])); let schema_drop_batch = RecordBatch::try_new( schema_drop_schema.clone(), vec![ Arc::new(Int64Array::from(vec![1, 2, 3])), Arc::new(StringArray::from(vec!["Laptop", "Smartphone", "Headphones"])), Arc::new(Float64Array::from(vec![1200.0, 800.0, 150.0])), Arc::new(StringArray::from(vec!["X", "Y", "Z"])), Arc::new(Int32Array::from(vec![100, 200, 300])), Arc::new( FixedSizeListArray::from_iter_primitive::( vec![ Some(vec![Some(0.1_f32); 128]), Some(vec![Some(0.2_f32); 128]), Some(vec![Some(0.3_f32); 128]), ], 128, ), ), ], ) .unwrap(); let schema_drop_reader: Box = Box::new(RecordBatchIterator::new( vec![Ok(schema_drop_batch)].into_iter(), schema_drop_schema.clone(), )); let schema_drop_table = db .create_table("schema_evolution_drop_example", schema_drop_reader) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); ``` #### Drop single columns Remove individual columns that are no longer needed: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Remove the first temporary column table.drop_columns(["temp_col1"]) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Remove the first temporary column await schemaDropTable.dropColumns(["temp_col1"]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Remove the first temporary column schema_drop_table.drop_columns(&["temp_col1"]).await.unwrap(); ``` #### Drop multiple columns Remove several columns at once for efficiency: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Remove the second temporary column table.drop_columns(["temp_col2"]) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Remove the second temporary column await schemaDropTable.dropColumns(["temp_col2"]); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} // Remove the second temporary column schema_drop_table.drop_columns(&["temp_col2"]).await.unwrap(); ``` > [!WARNING] > > Dropping a column changes the latest version only. Until > [cleanup](https://docs.lancedb.com/maintenance/cleanup.md) prunes the versions from before the drop, > [restoring](#versioning) one of them brings the column and its values back, > along with the rest of the table as it was then. Once those versions are > pruned, the drop cannot be undone, so check before dropping a column you may > need. ## Versioning This page shows the core table-versioning APIs used in the code snippets for Python, TypeScript, and Rust. Each operation below maps directly to methods shown in the examples. ### Basic Versioning Example Let's create a table with sample data to demonstrate LanceDB's versioning capabilities: #### Set Up the Table First, let's create a table with some sample data: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table_name = "quotes_versioning_example" data = [ {"id": 1, "author": "Richard", "quote": "Wubba Lubba Dub Dub!"}, {"id": 2, "author": "Morty", "quote": "Rick, what's going on?"}, { "id": 3, "author": "Richard", "quote": "I turned myself into a pickle, Morty!", }, ] # Define schema schema = pa.schema( [ pa.field("id", pa.int64()), pa.field("author", pa.string()), pa.field("quote", pa.string()), ] ) table = db.create_table(table_name, data, schema=schema, mode="overwrite") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const tableName = "quotes_versioning_example"; const data = [ { id: 1, author: "Richard", quote: "Wubba Lubba Dub Dub!" }, { id: 2, author: "Morty", quote: "Rick, what's going on?" }, { id: 3, author: "Richard", quote: "I turned myself into a pickle, Morty!", }, ]; const table = await db.createTable(tableName, data, { mode: "overwrite" }); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let table_name = "quotes_versioning_example"; let data = vec![ (1, "Richard", "Wubba Lubba Dub Dub!"), (2, "Morty", "Rick, what's going on?"), (3, "Richard", "I turned myself into a pickle, Morty!"), ]; let table = db .create_table(table_name, make_quotes_reader(data)) .mode(CreateTableMode::Overwrite) .execute() .await .unwrap(); ``` **Rust helper used in this page** ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} fn make_quotes_reader(rows: Vec<(i64, &str, &str)>) -> Box { let ids: Vec = rows.iter().map(|(id, _, _)| *id).collect(); let authors: Vec<&str> = rows.iter().map(|(_, author, _)| *author).collect(); let quotes: Vec<&str> = rows.iter().map(|(_, _, quote)| *quote).collect(); let schema = Arc::new(Schema::new(vec![ Field::new("id", DataType::Int64, false), Field::new("author", DataType::Utf8, false), Field::new("quote", DataType::Utf8, false), ])); let batch = RecordBatch::try_new( schema.clone(), vec![ Arc::new(Int64Array::from(ids)), Arc::new(StringArray::from(authors)), Arc::new(StringArray::from(quotes)), ], ) .unwrap(); let reader = RecordBatchIterator::new(vec![Ok(batch)].into_iter(), schema); Box::new(reader) } ``` #### Check Initial Version After creating the table, let's check the initial version information: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} versions = table.list_versions() current_version = table.version print(f"Number of versions after creation: {len(versions)}") print(f"Current version: {current_version}") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const versions = await table.listVersions(); const currentVersion = await table.version(); console.log(`Number of versions after creation: ${versions.length}`); console.log(`Current version: ${currentVersion}`); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let versions = table.list_versions().await.unwrap(); let current_version = table.version().await.unwrap(); println!("Number of versions after creation: {}", versions.len()); println!("Current version: {}", current_version); ``` ### Modify Data When you modify data through operations like update or delete, LanceDB automatically creates new versions. #### Update Existing Data Let's update some existing records to see versioning in action: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.update(where="author='Richard'", values={"author": "Richard Daniel Sanchez"}) rows_after_update = table.count_rows("author = 'Richard Daniel Sanchez'") print(f"Rows updated to Richard Daniel Sanchez: {rows_after_update}") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} await table.update({ where: "author = 'Richard'", values: { author: "Richard Daniel Sanchez" }, }); const rowsAfterUpdate = await table.countRows( "author = 'Richard Daniel Sanchez'", ); console.log(`Rows updated to Richard Daniel Sanchez: ${rowsAfterUpdate}`); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table .update() .only_if("author = 'Richard'") .column("author", "'Richard Daniel Sanchez'") .execute() .await .unwrap(); let rows_after_update = table .count_rows(Some("author = 'Richard Daniel Sanchez'".to_string())) .await .unwrap(); println!( "Rows updated to Richard Daniel Sanchez: {}", rows_after_update ); ``` #### Add New Data Now let's add more records to the table: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} more_data = [ { "id": 4, "author": "Richard Daniel Sanchez", "quote": "That's the way the news goes!", }, {"id": 5, "author": "Morty", "quote": "Aww geez, Rick!"}, ] table.add(more_data) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const moreData = [ { id: 4, author: "Richard Daniel Sanchez", quote: "That's the way the news goes!", }, { id: 5, author: "Morty", quote: "Aww geez, Rick!" }, ]; await table.add(moreData); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let more_data = vec![ (4, "Richard Daniel Sanchez", "That's the way the news goes!"), (5, "Morty", "Aww geez, Rick!"), ]; table .add(make_quotes_reader(more_data)) .execute() .await .unwrap(); ``` #### Check Version Changes Let's see how the versions have changed after our modifications: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} versions = table.list_versions() version_count_after_mod = len(versions) version_after_mod = table.version print(f"Number of versions after modifications: {version_count_after_mod}") print(f"Current version: {version_after_mod}") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const versionsAfterMod = await table.listVersions(); const versionCountAfterMod = versionsAfterMod.length; const versionAfterMod = await table.version(); console.log( `Number of versions after modifications: ${versionCountAfterMod}`, ); console.log(`Current version: ${versionAfterMod}`); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let versions_after_mod = table.list_versions().await.unwrap(); let version_count_after_mod = versions_after_mod.len(); let version_after_mod = table.version().await.unwrap(); println!( "Number of versions after modifications: {}", version_count_after_mod ); println!("Current version: {}", version_after_mod); ``` ### Rollback to Previous Versions LanceDB supports fast rollbacks to any previous version without data duplication. #### View All Versions First, let's see all the versions we've created: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} versions = table.list_versions() for v in versions: print(f"Version {v['version']}, created at {v['timestamp']}") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const allVersions = await table.listVersions(); for (const v of allVersions) { console.log(`Version ${v.version}, created at ${v.timestamp}`); } ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let all_versions = table.list_versions().await.unwrap(); for v in &all_versions { println!("Version {}, created at {}", v.version, v.timestamp); } ``` #### Restore a Version Snapshot Now let's restore a captured version snapshot: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.restore(version_after_mod) versions = table.list_versions() version_count_after_rollback = len(versions) print(f"Total number of versions after rollback: {version_count_after_rollback}") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} await table.checkout(versionAfterMod); await table.restore(); const versionsAfterRollback = await table.listVersions(); const versionCountAfterRollback = versionsAfterRollback.length; console.log( `Total number of versions after rollback: ${versionCountAfterRollback}`, ); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.checkout(version_after_mod).await.unwrap(); table.restore().await.unwrap(); let versions_after_rollback = table.list_versions().await.unwrap(); let version_count_after_rollback = versions_after_rollback.len(); println!( "Total number of versions after rollback: {}", version_count_after_rollback ); ``` ### Tag-Based Versioning Numeric table versions like `v3` or `v17` are precise but hard to remember. Tags let you attach human-readable labels (e.g., `"prod"`, `"baseline"`, `"q3-evaluation"`) to specific versions and check those out by name. They are conceptually similar to git tags, and unlike numeric versions, **tagged versions are never pruned**. A tagged version that is older than the retention window and is not the latest stops cleanup instead (see the cleanup note at the bottom of this page). The tags API supports the standard CRUD operations (create, list, update, delete), plus checking out by tag name. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Create a tag pointing at a specific version table.tags.create("baseline", 1) table.tags.create("with-edits", table.version) # List all tags on this table print(table.tags.list()) # Look up the version a tag points at print(table.tags.get_version("baseline")) # Move an existing tag to a different version table.tags.update("baseline", 2) # Check out a version by tag name table.checkout("baseline") print(table.version) # Delete a tag (does not delete the underlying version) table.tags.delete("with-edits") # Return to the latest version table.checkout_latest() ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} const tags = await tagsTable.tags(); // Create a tag pointing at a specific version await tags.create("baseline", 1); await tags.create("with-edits", await tagsTable.version()); // List all tags on this table console.log(await tags.list()); // Look up the version a tag points at console.log(await tags.getVersion("baseline")); // Move an existing tag to a different version await tags.update("baseline", 2); // Check out a version by tag name await tagsTable.checkout("baseline"); console.log(await tagsTable.version()); // Delete a tag (does not delete the underlying version) await tags.delete("with-edits"); // Return to the latest version await tagsTable.checkoutLatest(); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} let mut tags = tags_table.tags().await.unwrap(); // Create a tag pointing at a specific version tags.create("baseline", 1).await.unwrap(); let current_version = tags_table.version().await.unwrap(); tags.create("with-edits", current_version).await.unwrap(); // List all tags on this table let all_tags = tags.list().await.unwrap(); println!("Tags: {:?}", all_tags); // Look up the version a tag points at let baseline_version = tags.get_version("baseline").await.unwrap(); println!("baseline -> v{}", baseline_version); // Move an existing tag to a different version tags.update("baseline", 2).await.unwrap(); // Check out a version by tag name (separate method in Rust) tags_table.checkout_tag("baseline").await.unwrap(); println!("Current version: {}", tags_table.version().await.unwrap()); // Delete a tag (does not delete the underlying version) tags.delete("with-edits").await.unwrap(); // Return to the latest version tags_table.checkout_latest().await.unwrap(); ``` > [!NOTE] > > Deleting a tag only removes the label, not the version it points to. After > deletion, the underlying table version becomes eligible for cleanup again. ### Delete Data From the Table Let's demonstrate how deletions also create new versions: #### Go Back to Latest Version First, let's return to the latest version: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.checkout_latest() ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} await table.checkoutLatest(); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.checkout_latest().await.unwrap(); ``` #### Delete Data Now let's delete some data to see how it affects versioning: ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.delete("author = 'Morty'") rows_after_deletion = table.count_rows() print(f"Number of rows after deletion: {rows_after_deletion}") ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} await table.delete("author = 'Morty'"); const rowsAfterDeletion = await table.countRows(); console.log(`Number of rows after deletion: ${rowsAfterDeletion}`); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.delete("author = 'Morty'").await.unwrap(); let rows_after_deletion = table.count_rows(None).await.unwrap(); println!("Number of rows after deletion: {}", rows_after_deletion); ``` #### Version History and Operations On a fresh table, the snippets in this guide produce this version sequence: 1. `v1`: create table (`create_table` / `createTable` / `create_table`) 2. `v2`: update rows (`update`) 3. `v3`: add rows (`add`) 4. `v4`: restore snapshot (`restore`) from `version_after_mod`/`versionAfterMod` 5. `v5`: delete rows (`delete`) Read-only and checkout operations shown here (`list_versions`/`listVersions`, `version`, `checkout`, `checkout_latest`/`checkoutLatest`) do not create new versions. The version metadata fields can differ by backend. Direct table-backed version listing exposes a timestamp, while namespace-backed listing may expose fields such as `manifest_path`, `manifest_size`, `e_tag`, and `timestamp_millis`. In deployments that use managed versioning, prefer the table version APIs exposed by LanceDB Enterprise or the namespace service instead of mixing in lower-level Lance file operations. > [!NOTE] > > **System Operations** > > System operations like `optimize()`, index updates, and table compaction also increment table version numbers. > In LanceDB Embedded, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default), > which is when old-version files are removed and disk space is reclaimed. A deployment prunes them in the background > when its configuration enables [cleanup](https://docs.lancedb.com/maintenance/cleanup.md); `optimize()` is not available there. > > **Tagged versions are never pruned.** A version with a tag pointing at it is > retained regardless of age. When a tagged version is older than the retention window and > is not the latest, cleanup refuses to run instead of pruning around it: `optimize()` raises > an error naming the tag and prunes nothing. A tag on the latest version does not cause > this, but `optimize()` compacts before it prunes, and compacting commits a new version > whenever it merges files, so a tag on what was the latest version can become an older > one by the time cleanup runs; the error then comes after compacting, and the index > update does not run. To let cleanup proceed, [move the tag](#versioning-tag-based-versioning) > to a version inside the retention window, delete it, or widen the window. ## Privileges A table's `OWNERSHIP`, `SELECT`, `INSERT`, `UPDATE` and `DELETE` are granted through the REST ACL API or SQL `GRANT`/`REVOKE`: ```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE events TO ROLE etl ``` See [Privileges](https://docs.lancedb.com/governance/privileges.md#sql) for the full statement syntax and every target it reaches. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.