Tables - LanceDB

Applies to: Embedded ✅, Enterprise ✅ A table stores data in rows and columns and keeps a history of committed versions. Queries read table data, writes update it, and indexes accelerate searches over its columns. This page covers table management, data types, schema changes and version history. For loading records, see Ingestion. For changes to columns, see Schema evolution.

In LanceDB Enterprise, tables are also created and dropped in SQL with CREATE TABLE and DROP TABLE.

Table structure and metadata


Columns	including vectors, blobs, and computed columns
Versions	every write commits one, so reads can travel back
Indexes	vector, full-text and scalar, built over its columns

Create a table

tbl = db.create_table("documents", data)
tbl = db.create_table("documents", schema=schema)     # empty, typed

const tbl = await db.createTable("documents", data);
const tbl = await db.createEmptyTable("documents", schema);

CREATE TABLE documents (id BIGINT, title VARCHAR)

data can be a list of objects, an Arrow table or reader, a pandas or polars frame, or Pydantic models. Ingesting data covers each, plus exist_ok and mode="overwrite" for re-runnable scripts.

An explicit schema is where nullability is decided. In the embedded library, creation is also the only point at which a table can be given stable row ids; in LanceDB Enterprise, whether a new table gets them is the deployment’s setting. See Limits.

Open and list tables

tbl = db.open_table("documents")
db.table_names()

const tbl = await db.openTable("documents");
await db.tableNames();

Each accepts namespace_path to act inside a namespace rather than at the database root. For Python remote connections created with lancedb.connect("db://..."), use db.table_exists(["documents"]) to check whether a table exists. For an async remote connection, use await db.table_exists(["documents"]). Include namespace segments before the table name, for example ["analytics", "documents"]. These checks return False when the table is missing and propagate other errors, including permission errors. Local synchronous directory connections still raise NotImplementedError for table_exists. On a connection from connect_namespace("dir", ...) or connect_namespace("rest", ...), it fails with an AttributeError. For these connections, open the table and handle the error, or look for its name in a listing. A missing table raises ValueError on a local directory and lance_namespace.TableNotFoundError on a connection from connect_namespace. A listing returns one page of names at a time, so check every page. TypeScript has no tableExists.

Rename and clone tables

db.rename_table("documents", "docs")

db.clone_table(
    "docs_snapshot",
    source_uri="s3://bucket/db/documents.lance",
    source_version=42,       # or source_tag="release-1"
    is_shallow=True,
)

Renaming is not supported on a local directory connection. In LanceDB Enterprise, whether a table can be renamed or cloned at all depends on how the deployment stores its catalog. A shallow clone (the default) references the source’s data files rather than copying them, so it is fast and cheap regardless of table size. It is the practical way to branch an experiment off a known version, or to pin a training set to the exact version a model saw.

A shallow clone depends on the source’s files continuing to exist. If the source is compacted past the retention window, the version the clone points at can be pruned. LanceDB Embedded connections support shallow clones only; is_shallow=False is not implemented. If the copy must outlive the source’s history, copy its data into an independent table instead. A deployment that supports cloning also makes shallow clones only, and one whose catalog does not support it refuses every clone.

Enterprise job type: on an Enterprise cluster that drops tables in the background, the teardown runs as a job: drop_table, which fences the table’s directory with a marker and deletes its content, or drop_object, depending on how the deployment stores its catalog. See Object teardown.

Drop a table

db.drop_table("documents")
job = db.drop_table_async("documents")
db.drop_all_tables()

await db.dropTable("documents");
const job = await db.dropTableAsync("documents");
await db.dropAllTables();

DROP TABLE documents

Dropping removes the table and every version of it. Versioning does not survive a drop, and neither drop_all_tables nor drop_database asks for confirmation. Both work only on a local database; on an Enterprise connection the SDK raises instead.

On a large table the drop can take a while, so drop_table_async returns a job rather than blocking when the deployment drops tables in the background. In the embedded library, and on an Enterprise cluster that drops inline, the returned job has already finished.

Table management APIs

	Python	TypeScript	SQL
create	`create_table`	`createTable`	`CREATE TABLE`
create empty	`create_table(schema=)`	`createEmptyTable`	`CREATE TABLE`
open	`open_table`	`openTable`	—
list	`table_names`, `list_tables`	`tableNames`, `listTables`	—
exists	`table_exists`	—	—
rename	`rename_table`	`renameTable`	—
clone	`clone_table`	`cloneTable`	—
drop	`drop_table`, `drop_table_async`	`dropTable`, `dropTableAsync`	`DROP TABLE`
drop all	`drop_all_tables`	`dropAllTables`	—

Data types

A table’s schema is an Arrow schema: there is no separate LanceDB type system mapped onto storage, so a column is the same object to the query engine as it is to the format.

Category	Types
Numeric	`int8`-`int64` and unsigned, `float16`, `float32`, `float64`
String and binary	`utf8`, `large_utf8`, `binary`, `large_binary`, `fixed_size_binary(n)`
Date and time	`date32`, `date64`, `time32`, `time64`, `timestamp`, `duration`
Nested	`struct`, `list`, `large_list`, `fixed_size_list`, `map`
Lance extensions	vectors, blobs, bfloat16

A vector column is a fixed_size_list<float32, dim>, an ordinary Arrow type, which is why it is selectable and joinable but why arithmetic over it is list arithmetic rather than scalar. See SQL data types for the full reference, including the extension types and pseudo-columns.

Multimodal data

Store images, audio, video, and documents alongside structured columns and embeddings in the same versioned table. Use blob columns for these payloads. Small values stay inline in the data file, up to 64 KiB by default, and larger values are stored outside it.

Blob columns

Early access A blob column keeps metadata separate from the payload bytes, so queries can filter rows without loading the blobs. Most queries return descriptors, which contain metadata about the blob instead of its bytes. You can read the bytes in the query results or fetch them by row ID. In Python, plain to_pandas() scans can also return BlobFile handles when pylance is installed. When you write raw bytes, Lance chooses a storage layout based on the payload size. A descriptor’s kind tells you which one was used:

Storage mode	`kind`	Default placement	Storage behavior
`Inline`	`0`	Up to 64 KiB	Stored in the main data file.
`Packed`	`1`	Above 64 KiB, up to 4 MiB	Packed with other blobs into a shared sidecar file, up to 1 GiB per file by default.
`Dedicated`	`2`	Above 4 MiB	Stored in its own raw sidecar file.
`Managed`	`4` or `5`	Out-of-line values when managed storage is enabled	Stored in a Lance-owned object independently of the data file.

Managed storage must be enabled separately. See Managed blobs for details. kind=5 identifies a Managed object using an explicitly registered base. Lance uses the default thresholds unless you customize them. Explicitly configured thresholds are stored in the table schema and preserved on append.

Setup

Import the SDK and open a local database. The examples in this section use db.

import lancedb

db = lancedb.connect("./blob_db")

import * as arrow from "apache-arrow";
import { Buffer } from "node:buffer";
import * as lancedb from "@lancedb/lancedb";

const db = await lancedb.connect("./blob_db");

use std::sync::Arc;

use arrow_array::{
    Array, Int64Array, LargeBinaryArray, RecordBatch, RecordBatchIterator, StringArray, UInt64Array,
};
use arrow_schema::{DataType, Field, Schema};
use futures_util::TryStreamExt;
use lancedb::blob::{BlobRangeRequest, blob};
use lancedb::connect;
use lancedb::database::CreateTableMode;
use lancedb::query::{ExecutableQuery, QueryBase};

let db = connect("./blob_db").execute().await.unwrap();

Define a blob schema

Declare a blob column with lancedb.blob() in Python and TypeScript, or blob() in Rust:

import lancedb
import pyarrow as pa

schema = pa.schema(
    [
        pa.field("id", pa.int64()),
        pa.field("label", pa.string()),
        lancedb.blob("image"),
    ]
)

const blobSchema = new arrow.Schema([
  new arrow.Field("id", new arrow.Int64()),
  new arrow.Field("label", new arrow.Utf8()),
  lancedb.blob("image"),
]);

let blob_schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("label", DataType::Utf8, true),
    blob("image", true),
]));

Ingest blobs

Pass raw bytes for each blob value. In Python, a blob column accepts bytes. In TypeScript, it accepts a Buffer. The Python example creates two JPEGs with EXIF metadata, which we’ll read using a blob handle below.

import io
from PIL import Image

def jpeg_with_make(color, make):
    image = Image.new("RGB", (8, 8), color=color)
    exif = Image.Exif()
    exif[0x010F] = make
    buf = io.BytesIO()
    image.save(buf, format="JPEG", exif=exif)
    return buf.getvalue()

tbl = db.create_table(
    "photos",
    data=[
        {"id": 1, "label": "red", "image": jpeg_with_make("red", "TestCam")},
        {"id": 2, "label": "blue", "image": jpeg_with_make("blue", "OtherCam")},
    ],
    schema=schema,
    mode="overwrite",
)

const blobTable = await db.createEmptyTable("photos", blobSchema, {
  mode: "overwrite",
});

await blobTable.add([
  { id: 1n, label: "door", image: Buffer.from("inline_image_bytes") },
  { id: 2n, label: "window", image: Buffer.from("second_image_bytes") },
]);

let blob_table = db
    .create_empty_table("photos", blob_schema)
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

let input_schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("label", DataType::Utf8, true),
    Field::new("image", DataType::LargeBinary, true),
]));
let input_batch = RecordBatch::try_new(
    input_schema.clone(),
    vec![
        Arc::new(Int64Array::from(vec![1_i64, 2_i64])),
        Arc::new(StringArray::from(vec![Some("door"), Some("window")])),
        Arc::new(LargeBinaryArray::from_iter_values([
            b"inline_image_bytes".as_slice(),
            b"second_image_bytes".as_slice(),
        ])),
    ],
)
.unwrap();
let input_reader: Box<dyn arrow_array::RecordBatchReader + Send> = Box::new(
    RecordBatchIterator::new(vec![Ok(input_batch)].into_iter(), input_schema),
);
blob_table.add(input_reader).execute().await.unwrap();

Read bytes in query results

Python and TypeScript can return blob bytes in query results without collecting _rowid. Rust uses fetch_blobs with row IDs instead. to_list(blob_mode="bytes") returns the image bytes from the photos table above:

rows = (
    tbl.search().where("id = 1").select(["id", "image"]).to_list(blob_mode="bytes")
)

In the Python SDK, Table.to_pandas(), Table.to_arrow(), and the query builder methods to_pandas(), to_arrow(), to_list(), and to_batches() accept a blob_mode argument. Sync and async query builders support the same modes:

`blob_mode`	`to_pandas()`	`to_arrow()` and `to_list()`
`"lazy"`	Default. Plain scans return seekable `BlobFile` handles. Vector, full-text, hybrid, and ordered queries return descriptor dicts.	Default. Returns descriptors carrying `_lance_row_id` so the result table can be passed to `fetch_blobs` or `fetch_blob_files`.
`"descriptions"`	Returns descriptor dicts with `kind`, `position`, `size`, `blob_id`, and `blob_uri`.	Returns the 5-field descriptor without `_lance_row_id`.
`"bytes"`	Materializes each blob as `bytes` in the DataFrame.	Materializes each blob as `large_binary` in Arrow or `bytes` in Python dicts.

For supported local v2 blob projections, to_pandas(blob_mode="bytes") and to_pandas(blob_mode="descriptions") use native Arrow conversion without pylance. Pandas conversions that need the Lance scanner, including plain-scan "lazy" handles and legacy blobs, still require pylance. to_arrow(), to_list(), to_batches(), and the fetch methods do not require pylance. blob_mode="bytes" requires a LanceDB Embedded table snapshot, so Enterprise and in-memory tables raise NotImplementedError. On an Enterprise table, Table.to_pandas() and Table.to_arrow() aren’t supported, and a query’s to_pandas() also raises in the default "lazy" mode because it would open BlobFile handles. Query with blob_mode="descriptions", then read bytes with fetch_blobs or fetch_blob_files. blob_mode="bytes" materializes top-level and struct-nested blob columns. It raises ValueError when the selected columns include blobs inside a list. Select those columns with blob_mode="descriptions", or leave them out of the projection. Extra keyword arguments on to_pandas(), such as split_blocks and self_destruct, are forwarded to PyArrow. to_batches(blob_mode="bytes") fetches blob v2 payloads one batch at a time and returns them as Arrow large_binary values. Iterate over the reader and call batch.to_pandas() to convert each batch to a DataFrame without collecting all payloads at once. For sync queries, call query.to_batches(batch_size=64, blob_mode="bytes") and iterate with for. For async queries, call await query.to_batches(max_batch_length=64, blob_mode="bytes") and iterate with async for. Sync full-text and hybrid queries do not support to_batches(). The default "lazy" mode returns descriptors and keeps automatic _rowid columns visible for manual fetches. "descriptions" returns the five-field descriptors. Both "bytes" and "descriptions" hide automatic row IDs; explicitly requested row IDs remain visible. Bytes mode has the same restrictions as to_arrow(blob_mode="bytes"), including no legacy blobs, blobs inside lists, Enterprise tables, or namespace QueryTable pushdown. If concurrent writers can overwrite the table, use a separate reader checked out to a fixed version before querying. Otherwise, an overwrite can invalidate the row IDs used to fetch bytes and raise ValueError. The lazy-handle calls below require pylance. The query example expects tbl_async, query_vector, and a video column, which the photos table does not define.

df_lazy = tbl.to_pandas()

df_bytes = tbl.to_pandas(blob_mode="bytes")

df_desc = tbl.to_pandas(blob_mode="descriptions")

arrow_bytes = tbl.to_arrow(blob_mode="bytes")

df_typed = tbl.to_pandas(split_blocks=True, self_destruct=True)

Sync and async query builders accept the same blob_mode values:

df_lazy = (
    tbl.search().where("id = 1").select(["id", "video"]).to_pandas(blob_mode="lazy")
)

df_bytes = await (
    tbl_async.query()
    .where("id = 1")
    .select(["id", "video"])
    .to_pandas(blob_mode="bytes")
)

df_vec = (
    tbl.search(query_vector)
    .limit(2)
    .select(["id", "video"])
    .to_pandas(blob_mode="bytes", split_blocks=True, self_destruct=True)
)

In TypeScript, toArray() and toArrow() accept a blobMode option:

`blobMode`	Result
`"descriptions"`	Default. Returns descriptors.
`"bytes"`	Returns each top-level blob as bytes, or `null` for a null blob. Blobs nested in a struct or list keep their descriptors.

With "bytes", the rows and the bytes come from the same table version. If the table changes while the query runs, the query runs again, up to three times:

const withBytes = await blobTable
  .query()
  .select(["id", "image"])
  .toArray({ blobMode: "bytes" });
const firstImage: Uint8Array | null = withBytes[0].image;

Read blobs on demand

Query the rows you want and pass their _rowid values to a fetch method. fetch_blob_files opens a seekable handle and loads bytes only when you read. The Python example opens the JPEG from the ingest step and reads its EXIF Make tag with Pillow. TypeScript and Rust read the first bytes from the handle.

from PIL import ExifTags

rows = tbl.search().where("id = 1").select(["label"]).with_row_id(True).to_list()
with tbl.fetch_blob_files("image", [rows[0]["_rowid"]])[0] as image_file:
    make = Image.open(image_file).getexif().get(ExifTags.Base.Make)
print(rows[0]["label"], make)

const rows = await blobTable
  .query()
  .select(["id", "label"])
  .withRowId()
  .toArray();
const rowIds = rows.map((row) => row._rowid as bigint);

const [image] = await blobTable.fetchBlobFiles("image", [rowIds[0]!]);
const header = await image!.readRange(0n, 6n);
await image!.close();

let batches = blob_table
    .query()
    .with_row_id()
    .execute()
    .await
    .unwrap()
    .try_collect::<Vec<_>>()
    .await
    .unwrap();
let row_ids = batches[0]
    .column_by_name("_rowid")
    .unwrap()
    .as_any()
    .downcast_ref::<UInt64Array>()
    .unwrap()
    .values()
    .to_vec();

let handles = blob_table
    .fetch_blob_files("image", &row_ids[..1])
    .await
    .unwrap();
let header = handles[0].as_ref().unwrap().read_range(0..6).await.unwrap();
handles[0].as_ref().unwrap().close().await.unwrap();

The other fetch methods load bytes directly:

Need	Python	TypeScript	Rust
Whole payloads for a set of rows	`fetch_blobs`	`fetchBlobs`	`fetch_blobs`
Byte slices across many rows in one call	`fetch_blob_ranges`	Not available. Use `fetchBlobFiles` and `readRanges`.	`fetch_blob_ranges`

fetch_blobs returns the full bytes for each row ID in order. Null blobs produce null entries, and empty blobs produce empty byte strings.
fetch_blob_files loads bytes when you call read, read_range, or read_ranges. Range reads don’t move the file cursor. Null blobs produce null handles.
fetch_blob_ranges takes (row_id, offset, length) requests and returns one slice per request, in order. A request list can repeat a row to read more than one slice of that blob.

rows = tbl.search().select(["id"]).with_row_id(True).to_list()
row_ids = [row["_rowid"] for row in rows]
payloads = tbl.fetch_blobs("image", row_ids)

const payloads = await blobTable.fetchBlobs("image", rowIds);

let payloads = blob_table.fetch_blobs("image", &row_ids).await.unwrap();

Python and Rust read slices across rows with fetch_blob_ranges. On a handle, read_ranges reads more than one slice of one blob. Python takes (offset, length) pairs, TypeScript takes { start, end } ranges with an exclusive end, and Rust takes start..end ranges:

slices = tbl.fetch_blob_ranges("image", [(row_ids[0], 0, 2)])

with tbl.fetch_blob_files("image", [row_ids[0]])[0] as image_file:
    parts = image_file.read_ranges([(0, 2)])

const [secondImage] = await blobTable.fetchBlobFiles("image", [rowIds[1]!]);
const parts = await secondImage!.readRanges([
  { start: 0n, end: 6n },
  { start: 7n, end: 12n },
]);
await secondImage!.close();

let slices = blob_table
    .fetch_blob_ranges(
        "image",
        [
            BlobRangeRequest::new(row_ids[0], 0, 6),
            BlobRangeRequest::new(row_ids[1], 0, 6),
            BlobRangeRequest::new(row_ids[1], 7, 5),
        ],
    )
    .await
    .unwrap();

let second_handles = blob_table
    .fetch_blob_files("image", &row_ids[1..2])
    .await
    .unwrap();
let second_image = second_handles[0].as_ref().unwrap();
let parts = second_image.read_ranges(&[0..6, 7..12]).await.unwrap();
second_image.close().await.unwrap();

Notes and limits

Blob fetches fail if any requested row ID is missing or deleted. With stable row IDs enabled, IDs remain valid across compaction. A blob field can sit at the top level of the schema or inside a struct or list. Struct-nested blob columns use dotted paths such as clip.video in the fetch APIs. Blobs inside lists can be written and queried as descriptors. A table row ID does not address one list element, so blob_columns(), fetch_blobs, fetch_blob_files, and blob_mode="bytes" can’t return those bytes. See the Lance blob guide for the storage format.

Managed blobs (Blob v2)

Set the table schema metadata lance.blob.managed to true on a table with lancedb.blob() columns to store small values inline and larger values in managed objects under the table’s _blobs/ directory. Medium values can share a packed object, and large values use a dedicated object. The table owns these objects. You do not upload them separately, keep an object catalog, or delete their files yourself. Compaction preserves each descriptor and its payload object, so merging table fragments does not rewrite the blob payloads.

Use an SDK and Enterprise deployment that support Blob v2 Managed storage. Upgrade every reader, writer, and maintenance worker before activating Managed blobs on a table, and reopen existing table handles after the upgrade. Older clients cannot safely maintain a table containing Managed blobs.

Write and read blobs

This example uses a local connection. The same schema, add, query, and fetch_blobs calls work with an Enterprise database connection:

import pyarrow as pa

db = lancedb.connect("./media")
table = db.create_table(
    "images",
    schema=pa.schema([
        pa.field("id", pa.int64(), nullable=False),
        lancedb.blob("image"),
    ], metadata={"lance.blob.managed": "true"}),
)
table.add([
    {"id": 1, "image": b""},
    {"id": 2, "image": bytes(range(256)) * 400},
    {"id": 3, "image": None},
])

# Query descriptors first, then fetch only the selected rows' payloads.
hits = (
    table.search()
    .where("id <= 2")
    .select(["id", "image"])
    .with_row_id(True)
    .to_arrow()
)
images = table.fetch_blobs("image", hits).to_pylist()
by_id = dict(zip(hits["id"].to_pylist(), images, strict=True))
assert by_id[1] == b""
assert len(by_id[2]) == 102400

fetch_blobs materializes the selected payloads in memory, so fetch bounded batches when blobs are large. Row IDs belong to their table and snapshot. Keep descriptors opaque in application code, because their representation can differ between inline values and Managed objects.

Derived blobs and maintenance

Enterprise Functions can consume blob columns and return new blobs. Their output payloads are managed by the destination table. LanceDB Embedded tables can also derive blob columns with SQL computed columns. Deleting a row or replacing its blob does not immediately remove the old payload: retained table versions may still reference it. Garbage Collection reclaims objects after those references expire. A packed object remains while any of its contained blobs is still referenced.

Legacy large-binary blobs

Use lancedb.blob() for new tables. The sections below cover existing tables that use the V1 large-binary Blob API (pa.large_binary() in Python, LargeBinary in TypeScript, and DataType::LargeBinary in Rust with {"lance-encoding:blob": "true"} field metadata). Changing a field’s metadata alone does not migrate an existing V1 column to Blob v2.

1. Define a blob schema

V1 blob columns use a large_binary field marked with {"lance-encoding:blob": "true"} metadata:

import pyarrow as pa

# Define schema with Blob API metadata for lazy loading
schema = pa.schema(
    [
        pa.field("id", pa.int64()),
        pa.field(
            "video",
            pa.large_binary(),
            metadata={"lance-encoding:blob": "true"},  # Enable Blob API
        ),
    ]
)

const legacyBlobSchema = new arrow.Schema([
  new arrow.Field("id", new arrow.Int64()),
  new arrow.Field(
    "video",
    new arrow.LargeBinary(),
    true,
    new Map([["lance-encoding:blob", "true"]]),
  ),
]);

let legacy_blob_metadata = HashMap::from([(
    "lance-encoding:blob".to_string(),
    "true".to_string(),
)]);
let legacy_blob_schema = Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("video", DataType::LargeBinary, true).with_metadata(legacy_blob_metadata),
]);

2. Ingest large blobs

Write raw bytes to a V1 blob column:

import lancedb

db = lancedb.connect("./blob_db")

# Create sample data
data = [
    {"id": 1, "video": b"fake_video_bytes_1"},
    {"id": 2, "video": b"fake_video_bytes_2"},
]

# Create the table
tbl = db.create_table("videos", data=data, schema=schema)

const legacyBlobData = lancedb.makeArrowTable(
  [
    { id: 1, video: Buffer.from("fake_video_bytes_1") },
    { id: 2, video: Buffer.from("fake_video_bytes_2") },
  ],
  { schema: legacyBlobSchema },
);
const legacyBlobTable = await db.createTable(
  "legacy_videos",
  legacyBlobData,
  {
    mode: "overwrite",
  },
);

let legacy_blob_rows = vec![
    (1_i64, b"fake_video_bytes_1".to_vec()),
    (2_i64, b"fake_video_bytes_2".to_vec()),
];

let legacy_blob_schema = Arc::new(legacy_blob_schema);
let legacy_blob_batch = RecordBatch::try_new(
    legacy_blob_schema.clone(),
    vec![
        Arc::new(Int64Array::from_iter_values(
            legacy_blob_rows.iter().map(|row| row.0),
        )),
        Arc::new(LargeBinaryArray::from_iter_values(
            legacy_blob_rows.iter().map(|row| row.1.as_slice()),
        )),
    ],
)
.unwrap();
let legacy_blob_reader: Box<dyn arrow_array::RecordBatchReader + Send> = Box::new(
    RecordBatchIterator::new(vec![Ok(legacy_blob_batch)].into_iter(), legacy_blob_schema),
);
let legacy_blob_table = db
    .create_table("legacy_videos", legacy_blob_reader)
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

3. Convert blob tables to pandas

On LanceDB Embedded tables with V1 blob columns, to_pandas() in Python accepts blob_mode ("lazy", "bytes", or "descriptions"). Enterprise tables raise NotImplementedError, and "bytes" is not supported on in-memory tables. Extra keyword arguments such as split_blocks or self_destruct are forwarded to the PyArrow / Lance pandas conversion:

# Default: blob columns come back lazily
df_lazy = tbl.to_pandas()

# Materialize blob bytes eagerly
df_bytes = tbl.to_pandas(blob_mode="bytes")

# Return descriptors instead of payloads
df_desc = tbl.to_pandas(blob_mode="descriptions")

# Forward extra kwargs to PyArrow's to_pandas
df_typed = tbl.to_pandas(split_blocks=True, self_destruct=True)

On query builders with V1 blob columns, plain scans support "lazy", "bytes", and "descriptions". Vector, FTS, hybrid, and ordered queries cannot materialize V1 blob columns through to_pandas(), so omit those columns from the projection:

# Plain scan query: blob_mode is supported end to end
df_lazy = (
    tbl.search()
    .where("id = 1")
    .select(["id", "video"])
    .to_pandas(blob_mode="lazy")
)

# Same call shape works on async query builders
df_bytes = await (
    tbl_async.query()
    .where("id = 1")
    .select(["id", "video"])
    .to_pandas(blob_mode="bytes")
)

# Vector / FTS / hybrid queries can't materialize blob columns,
# so omit them from the projection
df_vec = (
    tbl.search(query_vector)
    .limit(10)
    .select(["id", "vector"])
    .to_pandas(split_blocks=True, self_destruct=True)
)

Store binary data

Use a binary column when the query should return the raw bytes. Declare it as pa.binary() in Python, Binary in TypeScript, and DataType::Binary in Rust.

1. Setup and imports

Import LanceDB and Arrow in your SDK.

import io
import lancedb
import numpy as np
import pyarrow as pa
from PIL import Image

import * as arrow from "apache-arrow";
import { Buffer } from "node:buffer";
import * as lancedb from "@lancedb/lancedb";

use std::collections::HashMap;
use std::sync::Arc;

use arrow_array::types::Float32Type;
use arrow_array::{
    Array, BinaryArray, FixedSizeListArray, Int32Array, Int64Array, LargeBinaryArray, RecordBatch,
    RecordBatchIterator, StringArray,
};
use arrow_schema::{DataType, Field, Schema};
use futures_util::TryStreamExt;
use lancedb::connect;
use lancedb::database::CreateTableMode;
use lancedb::query::{ExecutableQuery, QueryBase};

2. Prepare data

Encode each image or file into raw bytes before writing.

def create_dummy_image(color):
    img = Image.new("RGB", (100, 100), color=color)
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    return buf.getvalue()

data = [
    {
        "id": 1,
        "filename": "red_square.png",
        "vector": np.random.rand(128).astype(np.float32),
        "image_blob": create_dummy_image("red"),
        "label": "red",
    },
    {
        "id": 2,
        "filename": "blue_square.png",
        "vector": np.random.rand(128).astype(np.float32),
        "image_blob": create_dummy_image("blue"),
        "label": "blue",
    },
]

const createDummyImage = (color: string): Uint8Array => {
  const pngHeader = Uint8Array.from([137, 80, 78, 71, 13, 10, 26, 10]);
  return Buffer.concat([Buffer.from(pngHeader), Buffer.from(color, "utf8")]);
};

const data = [
  {
    id: 1,
    filename: "red_square.png",
    vector: Array.from({ length: 128 }, (_, i) => (i % 16) / 16),
    image_blob: createDummyImage("red"),
    label: "red",
  },
  {
    id: 2,
    filename: "blue_square.png",
    vector: Array.from({ length: 128 }, (_, i) => ((i + 8) % 16) / 16),
    image_blob: createDummyImage("blue"),
    label: "blue",
  },
];

let create_dummy_image = |color: u8| -> Vec<u8> {
    let mut png_like = vec![137, 80, 78, 71, 13, 10, 26, 10];
    png_like.push(color);
    png_like
};

let data = vec![
    (
        1_i32,
        "red_square.png",
        vec![0.1_f32; 128],
        create_dummy_image(1),
        "red",
    ),
    (
        2_i32,
        "blue_square.png",
        vec![0.2_f32; 128],
        create_dummy_image(2),
        "blue",
    ),
];

3. Define the schema

Declare the schema explicitly so Arrow and LanceDB store the field as binary instead of inferring a string or integer list.

schema = pa.schema(
    [
        pa.field("id", pa.int32()),
        pa.field("filename", pa.string()),
        pa.field("vector", pa.list_(pa.float32(), 128)),
        pa.field("image_blob", pa.binary()),
        pa.field("label", pa.string()),
    ]
)

const schema = new arrow.Schema([
  new arrow.Field("id", new arrow.Int32()),
  new arrow.Field("filename", new arrow.Utf8()),
  new arrow.Field(
    "vector",
    new arrow.FixedSizeList(
      128,
      new arrow.Field("item", new arrow.Float32(), true),
    ),
  ),
  new arrow.Field("image_blob", new arrow.Binary()),
  new arrow.Field("label", new arrow.Utf8()),
]);

let schema = Schema::new(vec![
    Field::new("id", DataType::Int32, false),
    Field::new("filename", DataType::Utf8, false),
    Field::new(
        "vector",
        DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 128),
        false,
    ),
    Field::new("image_blob", DataType::Binary, false),
    Field::new("label", DataType::Utf8, false),
]);

4. Ingest data

Create the table from the rows and schema.

tbl = db.create_table("images", data=data, schema=schema, mode="overwrite")

const multimodalData = lancedb.makeArrowTable(data, { schema });
const tbl = await db.createTable("images", multimodalData, {
  mode: "overwrite",
});

let schema = Arc::new(schema);
let image_batch = RecordBatch::try_new(
    schema.clone(),
    vec![
        Arc::new(Int32Array::from_iter_values(data.iter().map(|row| row.0))),
        Arc::new(StringArray::from_iter_values(data.iter().map(|row| row.1))),
        Arc::new(
            FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
                data.iter()
                    .map(|row| Some(row.2.iter().copied().map(Some).collect::<Vec<_>>())),
                128,
            ),
        ),
        Arc::new(BinaryArray::from_iter_values(
            data.iter().map(|row| row.3.as_slice()),
        )),
        Arc::new(StringArray::from_iter_values(data.iter().map(|row| row.4))),
    ],
)
.unwrap();
let image_reader: Box<dyn arrow_array::RecordBatchReader + Send> = Box::new(
    RecordBatchIterator::new(vec![Ok(image_batch)].into_iter(), schema.clone()),
);
let table = db
    .create_table("images", image_reader)
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

Retrieve binary data

When you search the table, select the binary column alongside your metadata columns.

query_vector = np.random.rand(128).astype(np.float32)
results = tbl.search(query_vector).limit(1).to_pandas()

const queryVector = Array.from({ length: 128 }, (_, i) => (i % 16) / 16);
const results = await tbl.search(queryVector).limit(1).toArray();

let query_vector = vec![0.1_f32; 128];
let results = table
    .query()
    .nearest_to(query_vector)
    .unwrap()
    .limit(1)
    .execute()
    .await
    .unwrap()
    .try_collect::<Vec<_>>()
    .await
    .unwrap();

Convert bytes back to objects

Decode the returned bytes into an image or media buffer in your application.

for _, row in results.iterrows():
    image_bytes = row["image_blob"]
    image = Image.open(io.BytesIO(image_bytes))
    print(f"Retrieved image: {row['filename']}, Size: {image.size}")

for (const row of results) {
  const imageBytes = row.image_blob as Uint8Array;
  console.log(
    `Retrieved image: ${row.filename}, Byte length: ${imageBytes.length}`,
  );
}

for batch in &results {
    let filenames = batch
        .column_by_name("filename")
        .unwrap()
        .as_any()
        .downcast_ref::<StringArray>()
        .unwrap();
    let images = batch
        .column_by_name("image_blob")
        .unwrap()
        .as_any()
        .downcast_ref::<BinaryArray>()
        .unwrap();

    for row in 0..batch.num_rows() {
        let image_bytes = images.value(row);
        println!(
            "Retrieved image: {}, Byte length: {}",
            filenames.value(row),
            image_bytes.len()
        );
    }
}

Other modalities

For PDFs, audio recordings, and other files, keep searchable fields in separate columns so you can filter rows before fetching blob bytes.

bfloat16

Sixteen-bit floats in the bfloat16 layout, stored as fixed_size_binary(2). Useful as a vector element type when halving the memory matters more than the precision does.

Schema evolution

Schema evolution enables non-breaking modifications to a database table’s structure (such as adding columns, altering data types, or dropping fields) to adapt to evolving data requirements without service interruptions. LanceDB supports ACID-compliant schema evolution through granular operations (add/alter/drop columns), allowing you to:

Iterate Safely: Modify schemas in production with versioned datasets and backward compatibility
Scale Seamlessly: Handle ML model iterations, regulatory changes, or feature additions
Optimize Continuously: Remove unused fields or enforce new constraints without downtime

In SQL, schema changes are ALTER TABLE.

Schema evolution operations

LanceDB supports four primary schema evolution operations:

Adding new columns: Extend your table with additional attributes
Altering existing columns: Change column names, data types, or nullability
Updating field metadata: Attach or change per-column Arrow metadata
Dropping columns: Remove unnecessary columns from your schema

Schema evolution operations are applied immediately but do not typically require rewriting all data. However, data type changes may involve more substantial operations.

Each schema evolution operation commits a new table version and returns status metadata such as the committed version. Run these operations from a mutable table handle; if you checked out an older version for reads, call checkout_latest / checkoutLatest before modifying the schema.

Add new columns

You can add new columns to a table with the add_columns method in Python, addColumns in TypeScript/JavaScript, or add_columns in Rust. New columns are populated based on SQL expressions you provide. On local tables, the Python SDK also accepts precomputed values from DataFrames and Arrow readers; see Create a computed column for the supported inputs and row-order requirements.

Set up the example table

First, let’s create a sample table with product data to demonstrate schema evolution:

import numpy as np

data = [
    {
        "id": 1,
        "name": "Laptop",
        "price": 1200.00,
        "vector": np.random.random(128).tolist(),
    },
    {
        "id": 2,
        "name": "Smartphone",
        "price": 800.00,
        "vector": np.random.random(128).tolist(),
    },
    {
        "id": 3,
        "name": "Headphones",
        "price": 150.00,
        "vector": np.random.random(128).tolist(),
    },
]
table = db.create_table("schema_evolution_add_example", data, mode="overwrite")

const schemaAddData = [
  {
    id: 1,
    name: "Laptop",
    price: 1200.0,
    vector: Array.from({ length: 128 }, () => Math.random()),
  },
  {
    id: 2,
    name: "Smartphone",
    price: 800.0,
    vector: Array.from({ length: 128 }, () => Math.random()),
  },
  {
    id: 3,
    name: "Headphones",
    price: 150.0,
    vector: Array.from({ length: 128 }, () => Math.random()),
  },
];
const schemaAddTable = await db.createTable(
  "schema_evolution_add_example",
  schemaAddData,
  { mode: "overwrite" },
);

let schema_add_schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("name", DataType::Utf8, false),
    Field::new("price", DataType::Float64, false),
    Field::new(
        "vector",
        DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 128),
        false,
    ),
]));
let schema_add_batch = RecordBatch::try_new(
    schema_add_schema.clone(),
    vec![
        Arc::new(Int64Array::from(vec![1, 2, 3])),
        Arc::new(StringArray::from(vec!["Laptop", "Smartphone", "Headphones"])),
        Arc::new(Float64Array::from(vec![1200.0, 800.0, 150.0])),
        Arc::new(
            FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
                vec![
                    Some(vec![Some(0.1_f32); 128]),
                    Some(vec![Some(0.2_f32); 128]),
                    Some(vec![Some(0.3_f32); 128]),
                ],
                128,
            ),
        ),
    ],
)
.unwrap();
let schema_add_reader: Box<dyn RecordBatchReader + Send> = Box::new(RecordBatchIterator::new(
    vec![Ok(schema_add_batch)].into_iter(),
    schema_add_schema.clone(),
));
let schema_add_table = db
    .create_table("schema_evolution_add_example", schema_add_reader)
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

Columns derived from existing data (SQL expressions over other columns) are covered in Create a computed column, because deriving values is a feature-engineering job rather than a schema one.

Add columns with default values

Add boolean columns with default values for status tracking:

# Add a stock status column with default value
table.add_columns({"in_stock": "cast(true as boolean)"})

// Add a stock status column with default value
await schemaAddTable.addColumns([
  {
    name: "in_stock",
    valueSql: "cast(true as boolean)",
  },
]);

// Add a stock status column with default value
schema_add_table
    .add_columns(
        NewColumnTransform::SqlExpressions(vec![(
            "in_stock".to_string(),
            "cast(true as boolean)".to_string(),
        )]),
        None,
    )
    .await
    .unwrap();

Add nullable columns

Add timestamp columns that can contain NULL values:

# Add a nullable timestamp column
table.add_columns({"last_ordered": "cast(NULL as timestamp)"})

// Add a nullable timestamp column
await schemaAddTable.addColumns([
  {
    name: "last_ordered",
    valueSql: "cast(NULL as timestamp)",
  },
]);

// Add a nullable timestamp column
schema_add_table
    .add_columns(
        NewColumnTransform::SqlExpressions(vec![(
            "last_ordered".to_string(),
            "cast(NULL as timestamp)".to_string(),
        )]),
        None,
    )
    .await
    .unwrap();

When adding columns that should contain NULL values, be sure to cast the NULL to the appropriate type, e.g., cast(NULL as timestamp).

Alter existing columns

You can alter columns using the alter_columns method in Python, alterColumns in TypeScript/JavaScript, or alter_columns in Rust. This allows you to:

Rename a column
Change a column’s data type
Modify nullability (whether a column can contain NULL values)

Set up the example table

Create a table with a custom schema to demonstrate column alterations:

table_name = "schema_evolution_alter_example"
if data is None:
    data = [
        {
            "id": 1,
            "name": "Laptop",
            "price": 1200,
            "discount_price": 1080.0,
            "vector": np.random.random(128).tolist(),
        },
        {
            "id": 2,
            "name": "Smartphone",
            "price": 800,
            "discount_price": 720.0,
            "vector": np.random.random(128).tolist(),
        },
    ]
schema = pa.schema(
    {
        "id": pa.int64(),
        "name": pa.string(),
        "price": pa.int32(),
        "discount_price": pa.float64(),
        "vector": pa.list_(pa.float32(), 128),
    }
)
table = db.create_table(table_name, data, schema=schema, mode="overwrite")

const schemaAlter = new arrow.Schema([
  new arrow.Field("id", new arrow.Int64()),
  new arrow.Field("name", new arrow.Utf8()),
  new arrow.Field("price", new arrow.Int32()),
  new arrow.Field("discount_price", new arrow.Float64()),
  new arrow.Field(
    "vector",
    new arrow.FixedSizeList(
      128,
      new arrow.Field("item", new arrow.Float32(), true),
    ),
  ),
]);
const schemaAlterData = lancedb.makeArrowTable(
  [
    {
      id: 1,
      name: "Laptop",
      price: 1200,
      discount_price: 1080.0,
      vector: Array.from({ length: 128 }, () => Math.random()),
    },
    {
      id: 2,
      name: "Smartphone",
      price: 800,
      discount_price: 720.0,
      vector: Array.from({ length: 128 }, () => Math.random()),
    },
  ],
  { schema: schemaAlter },
);
const schemaAlterTable = await db.createTable(
  "schema_evolution_alter_example",
  schemaAlterData,
  { mode: "overwrite" },
);

let schema_alter_schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("name", DataType::Utf8, false),
    Field::new("price", DataType::Int32, false),
    Field::new("discount_price", DataType::Float64, false),
    Field::new(
        "vector",
        DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 128),
        false,
    ),
]));
let schema_alter_batch = RecordBatch::try_new(
    schema_alter_schema.clone(),
    vec![
        Arc::new(Int64Array::from(vec![1, 2])),
        Arc::new(StringArray::from(vec!["Laptop", "Smartphone"])),
        Arc::new(Int32Array::from(vec![1200, 800])),
        Arc::new(Float64Array::from(vec![1080.0, 720.0])),
        Arc::new(
            FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
                vec![Some(vec![Some(0.1_f32); 128]), Some(vec![Some(0.2_f32); 128])],
                128,
            ),
        ),
    ],
)
.unwrap();
let schema_alter_reader: Box<dyn RecordBatchReader + Send> = Box::new(RecordBatchIterator::new(
    vec![Ok(schema_alter_batch)].into_iter(),
    schema_alter_schema.clone(),
));
let schema_alter_table = db
    .create_table("schema_evolution_alter_example", schema_alter_reader)
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

Rename columns

Change column names to better reflect their purpose:

# Rename discount_price to sale_price
table.alter_columns({"path": "discount_price", "rename": "sale_price"})

// Rename discount_price to sale_price
await schemaAlterTable.alterColumns([
  { path: "discount_price", rename: "sale_price" },
]);

// Rename discount_price to sale_price
schema_alter_table
    .alter_columns(&[ColumnAlteration::new("discount_price".to_string())
        .rename("sale_price".to_string())])
    .await
    .unwrap();

Change data types

Convert column data types for better performance or compatibility:

# Change price from int32 to int64 for larger numbers
table.alter_columns({"path": "price", "data_type": pa.int64()})

// Change price from int32 to int64 for larger numbers
await schemaAlterTable.alterColumns([
  { path: "price", dataType: new arrow.Int64() },
]);

// Change price from int32 to int64 for larger numbers
schema_alter_table
    .alter_columns(&[ColumnAlteration::new("price".to_string()).cast_to(DataType::Int64)])
    .await
    .unwrap();

Make columns nullable

You can alter columns to contain NULL values:

# Make the name column nullable
table.alter_columns({"path": "name", "nullable": True})

// Make the name column nullable
await schemaAlterTable.alterColumns([{ path: "name", nullable: true }]);

// Make the name column nullable
schema_alter_table
    .alter_columns(&[ColumnAlteration::new("name".to_string()).set_nullable(true)])
    .await
    .unwrap();

Changing a column to nullable affects future writes and merges too: missing values are accepted only when the target column is nullable.

Multiple changes at once

Apply several alterations in a single operation:

# Rename, change type, and make nullable in one operation
table.alter_columns(
    {
        "path": "sale_price",
        "rename": "final_price",
        "data_type": pa.float64(),
        "nullable": True,
    }
)

// Rename, change type, and make nullable in one operation
await schemaAlterTable.alterColumns([
  {
    path: "sale_price",
    rename: "final_price",
    dataType: new arrow.Float64(),
    nullable: true,
  },
]);

// Rename, change type, and make nullable in one operation
schema_alter_table
    .alter_columns(&[ColumnAlteration::new("sale_price".to_string())
        .rename("final_price".to_string())
        .cast_to(DataType::Float64)
        .set_nullable(true)])
    .await
    .unwrap();

Expression-based type changes

For transformations that are not simple casts (for example, converting "$100" to an integer), use a SQL-expression column add, then drop and rename:

# For custom transforms, create a new column from a SQL expression.
expression_table = db.create_table(
    "schema_evolution_expression_example",
    [{"id": 1, "price_text": "$100"}],
    mode="overwrite",
)

expression_table.add_columns(
    {"price_numeric": "cast(replace(price_text, '$', '') as int)"}
)
expression_table.drop_columns(["price_text"])
expression_table.alter_columns({"path": "price_numeric", "rename": "price"})

// For custom transforms, create a new column from a SQL expression.
const expressionTable = await db.createTable(
  "schema_evolution_expression_example",
  [{ id: 1, price_text: "$100" }],
  { mode: "overwrite" },
);

await expressionTable.addColumns([
  {
    name: "price_numeric",
    valueSql: "cast(replace(price_text, '$', '') as int)",
  },
]);
await expressionTable.dropColumns(["price_text"]);
await expressionTable.alterColumns([
  { path: "price_numeric", rename: "price" },
]);

// For custom transforms, create a new column from a SQL expression.
let expression_schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("price_text", DataType::Utf8, false),
]));
let expression_batch = RecordBatch::try_new(
    expression_schema.clone(),
    vec![
        Arc::new(Int64Array::from(vec![1])),
        Arc::new(StringArray::from(vec!["$100"])),
    ],
)
.unwrap();
let expression_reader: Box<dyn RecordBatchReader + Send> = Box::new(RecordBatchIterator::new(
    vec![Ok(expression_batch)].into_iter(),
    expression_schema.clone(),
));
let expression_table = db
    .create_table("schema_evolution_expression_example", expression_reader)
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

expression_table
    .add_columns(
        NewColumnTransform::SqlExpressions(vec![(
            "price_numeric".to_string(),
            "cast(replace(price_text, '$', '') as int)".to_string(),
        )]),
        None,
    )
    .await
    .unwrap();
expression_table.drop_columns(&["price_text"]).await.unwrap();
expression_table
    .alter_columns(&[ColumnAlteration::new("price_numeric".to_string())
        .rename("price".to_string())])
    .await
    .unwrap();

Alter embedding types and dimensions

It’s quite common to need to change an embedding column’s schema, in case a new model becomes available with a different embedding dimension.

In Python, the example shows an in-place type update when the cast is compatible.
In TypeScript and Rust, the example shows a dimension change (384 -> 1024), which cannot be cast in-place.

For dimension changes, use this 3-step pattern: add a new column with the target type, drop the old column, then rename the new column to the original name.

data = [
    {
        "id": 1,
        "embedding": np.random.random(vector_dim).tolist(),
    },
]
table = db.create_table(table_name, data, mode="overwrite")

table.alter_columns(
    dict(path="embedding", data_type=pa.list_(pa.float32(), vector_dim))
)

const oldDim = 384;
const newDim = 1024;
const vectorSchema = new arrow.Schema([
  new arrow.Field("id", new arrow.Int64()),
  new arrow.Field(
    "embedding",
    new arrow.FixedSizeList(
      oldDim,
      new arrow.Field("item", new arrow.Float16(), true),
    ),
    true,
  ),
]);
const vectorData = lancedb.makeArrowTable(
  [{ id: 1, embedding: Array.from({ length: oldDim }, () => Math.random()) }],
  { schema: vectorSchema },
);
const vectorTable = await db.createTable("vector_alter_example", vectorData, {
  mode: "overwrite",
});

// Changing FixedSizeList dimensions (384 -> 1024) is not supported via alterColumns.
// Use addColumns + dropColumns + alterColumns(rename) to replace the column.
await vectorTable.addColumns([
  {
    name: "embedding_v2",
    valueSql: `arrow_cast(NULL, 'FixedSizeList(${newDim}, Float16)')`,
  },
]);
await vectorTable.dropColumns(["embedding"]);
await vectorTable.alterColumns([{ path: "embedding_v2", rename: "embedding" }]);

let old_dim = 384;
let new_dim = 1024;
let vector_schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new(
        "embedding",
        DataType::FixedSizeList(
            Arc::new(Field::new("item", DataType::Float32, true)),
            old_dim,
        ),
        true,
    ),
]));
let vector_batch = RecordBatch::try_new(
    vector_schema.clone(),
    vec![
        Arc::new(Int64Array::from(vec![1])),
        Arc::new(
            FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
                vec![Some(vec![Some(0.1_f32); old_dim as usize])],
                old_dim,
            ),
        ),
    ],
)
.unwrap();
let vector_reader: Box<dyn RecordBatchReader + Send> =
    Box::new(RecordBatchIterator::new(vec![Ok(vector_batch)].into_iter(), vector_schema.clone()));
let vector_table = db
    .create_table("vector_alter_example", vector_reader)
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

// Changing FixedSizeList dimensions (384 -> 1024) is not supported via alter_columns.
// Use add_columns + drop_columns + alter_columns(rename) to replace the column.
vector_table
    .add_columns(
        NewColumnTransform::SqlExpressions(vec![(
            "embedding_v2".to_string(),
            format!("arrow_cast(NULL, 'FixedSizeList({}, Float32)')", new_dim),
        )]),
        None,
    )
    .await
    .unwrap();
vector_table.drop_columns(&["embedding"]).await.unwrap();
vector_table
    .alter_columns(&[ColumnAlteration::new("embedding_v2".to_string())
        .rename("embedding".to_string())])
    .await
    .unwrap();

FixedSizeList Dimension Changes in TypeScript and RustalterColumns / alter_columns can cast between compatible types, but changing FixedSizeList dimensions (for example 384 -> 1024) is not a compatible cast. For such cases, use addColumns / add_columns (with arrow_cast), then dropColumns / drop_columns, then rename the replacement column.

Changing data types requires rewriting the column data and may be resource-intensive for large tables. Renaming columns or changing nullability is more efficient as it only updates metadata.

Update field metadata

Each column in a LanceDB table can carry a small key/value map of Arrow field metadata, which is useful for annotating columns with units, provenance, PII flags, embedding model versions, or any other schema-level context your application needs. Use update_field_metadata in Python, updateFieldMetadata in TypeScript/JavaScript, or update_field_metadata in Rust to add, change, or remove these key/value pairs without rewriting the column data. Each call commits a new table version and returns the new version. Each update targets one field by dot-path: top-level columns are addressed by name (for example "embedding"), and nested fields by their full path (for example "address.zip"). By default, the keys you pass are merged into the field’s existing metadata: keys you do not mention are preserved, and passing None (Python) or null (TypeScript) deletes a key. Set replace: true to swap the field’s entire metadata map instead of merging.

# Set two metadata keys on the `category` field.
res = table.update_field_metadata(
    {"path": "category", "metadata": {"unit": "label", "pii": "false"}}
)
print(res.version)

# Merge: add a new key, delete one with None, keep the rest.
table.update_field_metadata(
    {"path": "category", "metadata": {"source": "import", "pii": None}}
)

# Arrow stores field metadata as bytes.
assert table.schema.field("category").metadata == {
    b"unit": b"label",
    b"source": b"import",
}

// Set two metadata keys on the `category` field.
const res = await fieldMetadataTable.updateFieldMetadata([
  { path: "category", metadata: { unit: "label", pii: "false" } },
]);
console.log(res.version);

// Merge: add a new key, delete one via null, keep the rest.
await fieldMetadataTable.updateFieldMetadata([
  { path: "category", metadata: { source: "import", pii: null } },
]);

// Set two metadata keys on the `category` field.
let res = field_metadata_table
    .update_field_metadata(&[FieldMetadataUpdate::new("category")
        .set("unit", "label")
        .set("pii", "false")])
    .await
    .unwrap();
println!("version: {}", res.version);

// Merge: add a new key, delete one with `.remove`, keep the rest.
field_metadata_table
    .update_field_metadata(&[FieldMetadataUpdate::new("category")
        .set("source", "import")
        .remove("pii")])
    .await
    .unwrap();

To overwrite a field’s metadata entirely instead of merging, set replace to true:

table.update_field_metadata(
    {
        "path": "category",
        "metadata": {"owner": "search-team"},
        "replace": True,
    }
)

await fieldMetadataTable.updateFieldMetadata([
  {
    path: "category",
    metadata: { owner: "search-team" },
    replace: true,
  },
]);

field_metadata_table
    .update_field_metadata(&[FieldMetadataUpdate::new("category")
        .set("owner", "search-team")
        .replace()])
    .await
    .unwrap();

You can pass multiple updates in a single call to change metadata on several fields at once. Each call commits a single new table version.

Canonical metadata keys

You can use any string as a metadata key. By convention, LanceDB treats keys prefixed with lancedb: as canonical. LanceDB Enterprise displays these keys in the UI, and agents can rely on finding this information in a consistent place. Use them when they fit your use case:

Key	Purpose	Example
`lancedb:description`	Human-readable description of the field.	`"CLIP embedding of the product image"`
`lancedb:tag:<name>`	User-defined key/value tag, where the suffix names the tag category.	`"lancedb:tag:model": "clip"`
`lancedb:status`	Life cycle state of the column: `production`, `candidate`, `deprecated`, or `archived`.	`"lancedb:status": "production"`

These keys are conventions, not enforced constraints. LanceDB does not validate their values, but following the conventions keeps your tables consistent with Enterprise tooling.

Drop columns

You can remove columns using the drop_columns method in Python, dropColumns in TypeScript/JavaScript, or drop_columns in Rust.

Set Up the example table

Create a table with temporary columns that we’ll remove:

if data is None:
    data = [
        {
            "id": 1,
            "name": "Laptop",
            "price": 1200.00,
            "temp_col1": "X",
            "temp_col2": 100,
            "vector": np.random.random(128).tolist(),
        },
        {
            "id": 2,
            "name": "Smartphone",
            "price": 800.00,
            "temp_col1": "Y",
            "temp_col2": 200,
            "vector": np.random.random(128).tolist(),
        },
        {
            "id": 3,
            "name": "Headphones",
            "price": 150.00,
            "temp_col1": "Z",
            "temp_col2": 300,
            "vector": np.random.random(128).tolist(),
        },
    ]
table = db.create_table("schema_evolution_drop_example", data, mode="overwrite")

const schemaDropData = [
  {
    id: 1,
    name: "Laptop",
    price: 1200.0,
    temp_col1: "X",
    temp_col2: 100,
    vector: Array.from({ length: 128 }, () => Math.random()),
  },
  {
    id: 2,
    name: "Smartphone",
    price: 800.0,
    temp_col1: "Y",
    temp_col2: 200,
    vector: Array.from({ length: 128 }, () => Math.random()),
  },
  {
    id: 3,
    name: "Headphones",
    price: 150.0,
    temp_col1: "Z",
    temp_col2: 300,
    vector: Array.from({ length: 128 }, () => Math.random()),
  },
];
const schemaDropTable = await db.createTable(
  "schema_evolution_drop_example",
  schemaDropData,
  { mode: "overwrite" },
);

let schema_drop_schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("name", DataType::Utf8, false),
    Field::new("price", DataType::Float64, false),
    Field::new("temp_col1", DataType::Utf8, false),
    Field::new("temp_col2", DataType::Int32, false),
    Field::new(
        "vector",
        DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 128),
        false,
    ),
]));
let schema_drop_batch = RecordBatch::try_new(
    schema_drop_schema.clone(),
    vec![
        Arc::new(Int64Array::from(vec![1, 2, 3])),
        Arc::new(StringArray::from(vec!["Laptop", "Smartphone", "Headphones"])),
        Arc::new(Float64Array::from(vec![1200.0, 800.0, 150.0])),
        Arc::new(StringArray::from(vec!["X", "Y", "Z"])),
        Arc::new(Int32Array::from(vec![100, 200, 300])),
        Arc::new(
            FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
                vec![
                    Some(vec![Some(0.1_f32); 128]),
                    Some(vec![Some(0.2_f32); 128]),
                    Some(vec![Some(0.3_f32); 128]),
                ],
                128,
            ),
        ),
    ],
)
.unwrap();
let schema_drop_reader: Box<dyn RecordBatchReader + Send> = Box::new(RecordBatchIterator::new(
    vec![Ok(schema_drop_batch)].into_iter(),
    schema_drop_schema.clone(),
));
let schema_drop_table = db
    .create_table("schema_evolution_drop_example", schema_drop_reader)
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

Drop single columns

Remove individual columns that are no longer needed:

# Remove the first temporary column
table.drop_columns(["temp_col1"])

// Remove the first temporary column
await schemaDropTable.dropColumns(["temp_col1"]);

// Remove the first temporary column
schema_drop_table.drop_columns(&["temp_col1"]).await.unwrap();

Drop multiple columns

Remove several columns at once for efficiency:

# Remove the second temporary column
table.drop_columns(["temp_col2"])

// Remove the second temporary column
await schemaDropTable.dropColumns(["temp_col2"]);

// Remove the second temporary column
schema_drop_table.drop_columns(&["temp_col2"]).await.unwrap();

Dropping a column changes the latest version only. Until cleanup prunes the versions from before the drop, restoring one of them brings the column and its values back, along with the rest of the table as it was then. Once those versions are pruned, the drop cannot be undone, so check before dropping a column you may need.

Versioning

This page shows the core table-versioning APIs used in the code snippets for Python, TypeScript, and Rust. Each operation below maps directly to methods shown in the examples.

Basic Versioning Example

Let’s create a table with sample data to demonstrate LanceDB’s versioning capabilities:

Set Up the Table

First, let’s create a table with some sample data:

table_name = "quotes_versioning_example"
data = [
    {"id": 1, "author": "Richard", "quote": "Wubba Lubba Dub Dub!"},
    {"id": 2, "author": "Morty", "quote": "Rick, what's going on?"},
    {
        "id": 3,
        "author": "Richard",
        "quote": "I turned myself into a pickle, Morty!",
    },
]

# Define schema
schema = pa.schema(
    [
        pa.field("id", pa.int64()),
        pa.field("author", pa.string()),
        pa.field("quote", pa.string()),
    ]
)

table = db.create_table(table_name, data, schema=schema, mode="overwrite")

const tableName = "quotes_versioning_example";
const data = [
  { id: 1, author: "Richard", quote: "Wubba Lubba Dub Dub!" },
  { id: 2, author: "Morty", quote: "Rick, what's going on?" },
  {
    id: 3,
    author: "Richard",
    quote: "I turned myself into a pickle, Morty!",
  },
];
const table = await db.createTable(tableName, data, { mode: "overwrite" });

let table_name = "quotes_versioning_example";
let data = vec![
    (1, "Richard", "Wubba Lubba Dub Dub!"),
    (2, "Morty", "Rick, what's going on?"),
    (3, "Richard", "I turned myself into a pickle, Morty!"),
];

let table = db
    .create_table(table_name, make_quotes_reader(data))
    .mode(CreateTableMode::Overwrite)
    .execute()
    .await
    .unwrap();

Show Rust helper used in this page

fn make_quotes_reader(rows: Vec<(i64, &str, &str)>) -> Box<dyn RecordBatchReader + Send> {
    let ids: Vec<i64> = rows.iter().map(|(id, _, _)| *id).collect();
    let authors: Vec<&str> = rows.iter().map(|(_, author, _)| *author).collect();
    let quotes: Vec<&str> = rows.iter().map(|(_, _, quote)| *quote).collect();

    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int64, false),
        Field::new("author", DataType::Utf8, false),
        Field::new("quote", DataType::Utf8, false),
    ]));

    let batch = RecordBatch::try_new(
        schema.clone(),
        vec![
            Arc::new(Int64Array::from(ids)),
            Arc::new(StringArray::from(authors)),
            Arc::new(StringArray::from(quotes)),
        ],
    )
    .unwrap();
    let reader = RecordBatchIterator::new(vec![Ok(batch)].into_iter(), schema);
    Box::new(reader)
}

Check Initial Version

After creating the table, let’s check the initial version information:

versions = table.list_versions()
current_version = table.version
print(f"Number of versions after creation: {len(versions)}")
print(f"Current version: {current_version}")

const versions = await table.listVersions();
const currentVersion = await table.version();
console.log(`Number of versions after creation: ${versions.length}`);
console.log(`Current version: ${currentVersion}`);

let versions = table.list_versions().await.unwrap();
let current_version = table.version().await.unwrap();
println!("Number of versions after creation: {}", versions.len());
println!("Current version: {}", current_version);

Modify Data

When you modify data through operations like update or delete, LanceDB automatically creates new versions.

Update Existing Data

Let’s update some existing records to see versioning in action:

table.update(where="author='Richard'", values={"author": "Richard Daniel Sanchez"})
rows_after_update = table.count_rows("author = 'Richard Daniel Sanchez'")
print(f"Rows updated to Richard Daniel Sanchez: {rows_after_update}")

await table.update({
  where: "author = 'Richard'",
  values: { author: "Richard Daniel Sanchez" },
});
const rowsAfterUpdate = await table.countRows(
  "author = 'Richard Daniel Sanchez'",
);
console.log(`Rows updated to Richard Daniel Sanchez: ${rowsAfterUpdate}`);

table
    .update()
    .only_if("author = 'Richard'")
    .column("author", "'Richard Daniel Sanchez'")
    .execute()
    .await
    .unwrap();
let rows_after_update = table
    .count_rows(Some("author = 'Richard Daniel Sanchez'".to_string()))
    .await
    .unwrap();
println!(
    "Rows updated to Richard Daniel Sanchez: {}",
    rows_after_update
);

Add New Data

Now let’s add more records to the table:

more_data = [
    {
        "id": 4,
        "author": "Richard Daniel Sanchez",
        "quote": "That's the way the news goes!",
    },
    {"id": 5, "author": "Morty", "quote": "Aww geez, Rick!"},
]
table.add(more_data)

const moreData = [
  {
    id: 4,
    author: "Richard Daniel Sanchez",
    quote: "That's the way the news goes!",
  },
  { id: 5, author: "Morty", quote: "Aww geez, Rick!" },
];
await table.add(moreData);

let more_data = vec![
    (4, "Richard Daniel Sanchez", "That's the way the news goes!"),
    (5, "Morty", "Aww geez, Rick!"),
];
table
    .add(make_quotes_reader(more_data))
    .execute()
    .await
    .unwrap();

Check Version Changes

Let’s see how the versions have changed after our modifications:

versions = table.list_versions()
version_count_after_mod = len(versions)
version_after_mod = table.version
print(f"Number of versions after modifications: {version_count_after_mod}")
print(f"Current version: {version_after_mod}")

const versionsAfterMod = await table.listVersions();
const versionCountAfterMod = versionsAfterMod.length;
const versionAfterMod = await table.version();
console.log(
  `Number of versions after modifications: ${versionCountAfterMod}`,
);
console.log(`Current version: ${versionAfterMod}`);

let versions_after_mod = table.list_versions().await.unwrap();
let version_count_after_mod = versions_after_mod.len();
let version_after_mod = table.version().await.unwrap();
println!(
    "Number of versions after modifications: {}",
    version_count_after_mod
);
println!("Current version: {}", version_after_mod);

Rollback to Previous Versions

LanceDB supports fast rollbacks to any previous version without data duplication.

View All Versions

First, let’s see all the versions we’ve created:

versions = table.list_versions()
for v in versions:
    print(f"Version {v['version']}, created at {v['timestamp']}")

const allVersions = await table.listVersions();
for (const v of allVersions) {
  console.log(`Version ${v.version}, created at ${v.timestamp}`);
}

let all_versions = table.list_versions().await.unwrap();
for v in &all_versions {
    println!("Version {}, created at {}", v.version, v.timestamp);
}

Restore a Version Snapshot

Now let’s restore a captured version snapshot:

table.restore(version_after_mod)
versions = table.list_versions()
version_count_after_rollback = len(versions)
print(f"Total number of versions after rollback: {version_count_after_rollback}")

await table.checkout(versionAfterMod);
await table.restore();
const versionsAfterRollback = await table.listVersions();
const versionCountAfterRollback = versionsAfterRollback.length;
console.log(
  `Total number of versions after rollback: ${versionCountAfterRollback}`,
);

table.checkout(version_after_mod).await.unwrap();
table.restore().await.unwrap();
let versions_after_rollback = table.list_versions().await.unwrap();
let version_count_after_rollback = versions_after_rollback.len();
println!(
    "Total number of versions after rollback: {}",
    version_count_after_rollback
);

Tag-Based Versioning

Numeric table versions like v3 or v17 are precise but hard to remember. Tags let you attach human-readable labels (e.g., "prod", "baseline", "q3-evaluation") to specific versions and check those out by name. They are conceptually similar to git tags, and unlike numeric versions, tagged versions are never pruned. A tagged version that is older than the retention window and is not the latest stops cleanup instead (see the cleanup note at the bottom of this page). The tags API supports the standard CRUD operations (create, list, update, delete), plus checking out by tag name.

# Create a tag pointing at a specific version
table.tags.create("baseline", 1)
table.tags.create("with-edits", table.version)

# List all tags on this table
print(table.tags.list())

# Look up the version a tag points at
print(table.tags.get_version("baseline"))

# Move an existing tag to a different version
table.tags.update("baseline", 2)

# Check out a version by tag name
table.checkout("baseline")
print(table.version)

# Delete a tag (does not delete the underlying version)
table.tags.delete("with-edits")

# Return to the latest version
table.checkout_latest()

const tags = await tagsTable.tags();

// Create a tag pointing at a specific version
await tags.create("baseline", 1);
await tags.create("with-edits", await tagsTable.version());

// List all tags on this table
console.log(await tags.list());

// Look up the version a tag points at
console.log(await tags.getVersion("baseline"));

// Move an existing tag to a different version
await tags.update("baseline", 2);

// Check out a version by tag name
await tagsTable.checkout("baseline");
console.log(await tagsTable.version());

// Delete a tag (does not delete the underlying version)
await tags.delete("with-edits");

// Return to the latest version
await tagsTable.checkoutLatest();

let mut tags = tags_table.tags().await.unwrap();

// Create a tag pointing at a specific version
tags.create("baseline", 1).await.unwrap();
let current_version = tags_table.version().await.unwrap();
tags.create("with-edits", current_version).await.unwrap();

// List all tags on this table
let all_tags = tags.list().await.unwrap();
println!("Tags: {:?}", all_tags);

// Look up the version a tag points at
let baseline_version = tags.get_version("baseline").await.unwrap();
println!("baseline -> v{}", baseline_version);

// Move an existing tag to a different version
tags.update("baseline", 2).await.unwrap();

// Check out a version by tag name (separate method in Rust)
tags_table.checkout_tag("baseline").await.unwrap();
println!("Current version: {}", tags_table.version().await.unwrap());

// Delete a tag (does not delete the underlying version)
tags.delete("with-edits").await.unwrap();

// Return to the latest version
tags_table.checkout_latest().await.unwrap();

Deleting a tag only removes the label, not the version it points to. After deletion, the underlying table version becomes eligible for cleanup again.

Delete Data From the Table

Let’s demonstrate how deletions also create new versions:

Go Back to Latest Version

First, let’s return to the latest version:

table.checkout_latest()

await table.checkoutLatest();

table.checkout_latest().await.unwrap();

Delete Data

Now let’s delete some data to see how it affects versioning:

table.delete("author = 'Morty'")
rows_after_deletion = table.count_rows()
print(f"Number of rows after deletion: {rows_after_deletion}")

await table.delete("author = 'Morty'");
const rowsAfterDeletion = await table.countRows();
console.log(`Number of rows after deletion: ${rowsAfterDeletion}`);

table.delete("author = 'Morty'").await.unwrap();
let rows_after_deletion = table.count_rows(None).await.unwrap();
println!("Number of rows after deletion: {}", rows_after_deletion);

Version History and Operations

On a fresh table, the snippets in this guide produce this version sequence:

v1: create table (create_table / createTable / create_table)
v2: update rows (update)
v3: add rows (add)
v4: restore snapshot (restore) from version_after_mod/versionAfterMod
v5: delete rows (delete)

Read-only and checkout operations shown here (list_versions/listVersions, version, checkout, checkout_latest/checkoutLatest) do not create new versions. The version metadata fields can differ by backend. Direct table-backed version listing exposes a timestamp, while namespace-backed listing may expose fields such as manifest_path, manifest_size, e_tag, and timestamp_millis. In deployments that use managed versioning, prefer the table version APIs exposed by LanceDB Enterprise or the namespace service instead of mixing in lower-level Lance file operations.

System OperationsSystem operations like optimize(), index updates, and table compaction also increment table version numbers. In LanceDB Embedded, optimize() can prune older versions based on its retention setting (cleanup_older_than, 7 days by default), which is when old-version files are removed and disk space is reclaimed. A deployment prunes them in the background when its configuration enables cleanup; optimize() is not available there.Tagged versions are never pruned. A version with a tag pointing at it is retained regardless of age. When a tagged version is older than the retention window and is not the latest, cleanup refuses to run instead of pruning around it: optimize() raises an error naming the tag and prunes nothing. A tag on the latest version does not cause this, but optimize() compacts before it prunes, and compacting commits a new version whenever it merges files, so a tag on what was the latest version can become an older one by the time cleanup runs; the error then comes after compacting, and the index update does not run. To let cleanup proceed, move the tag to a version inside the retention window, delete it, or widen the window.

Privileges

A table’s OWNERSHIP, SELECT, INSERT, UPDATE and DELETE are granted through the REST ACL API or SQL GRANT/REVOKE:

GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE events TO ROLE etl

See Privileges for the full statement syntax and every target it reaches.

​Table structure and metadata

​Create a table

​Open and list tables

​Rename and clone tables

​Drop a table

​Table management APIs

​Data types

​Multimodal data

​Blob columns

​Setup

​Define a blob schema

​Ingest blobs

​Read bytes in query results

​Read blobs on demand

​Notes and limits

​Managed blobs (Blob v2)

​Write and read blobs

​Derived blobs and maintenance

​Legacy large-binary blobs

​1. Define a blob schema

​2. Ingest large blobs

​3. Convert blob tables to pandas

​Store binary data

​1. Setup and imports

​2. Prepare data

​3. Define the schema

​4. Ingest data

​Retrieve binary data

​Convert bytes back to objects

​Other modalities

​bfloat16

​Schema evolution

​Schema evolution operations

​Add new columns

​Set up the example table

​Add columns with default values

​Add nullable columns

​Alter existing columns

​Set up the example table

​Rename columns

​Change data types

​Make columns nullable

​Multiple changes at once

​Expression-based type changes

​Alter embedding types and dimensions

​Update field metadata

​Canonical metadata keys

​Drop columns

​Set Up the example table

​Drop single columns

​Drop multiple columns

​Versioning

​Basic Versioning Example

​Set Up the Table

​Check Initial Version

​Modify Data

​Update Existing Data

​Add New Data

​Check Version Changes

​Rollback to Previous Versions

​View All Versions

​Restore a Version Snapshot

​Tag-Based Versioning

​Delete Data From the Table

​Go Back to Latest Version

​Delete Data

​Version History and Operations

​Privileges

Table structure and metadata

Create a table

Open and list tables

Rename and clone tables

Drop a table

Table management APIs

Data types

Multimodal data

Blob columns

Setup

Define a blob schema

Ingest blobs

Read bytes in query results

Read blobs on demand

Notes and limits

Managed blobs (Blob v2)

Write and read blobs

Derived blobs and maintenance

Legacy large-binary blobs

1. Define a blob schema

2. Ingest large blobs

3. Convert blob tables to pandas

Store binary data

1. Setup and imports

2. Prepare data

3. Define the schema

4. Ingest data

Retrieve binary data

Convert bytes back to objects

Other modalities

bfloat16

Schema evolution

Schema evolution operations

Add new columns

Set up the example table

Add columns with default values

Add nullable columns

Alter existing columns

Set up the example table

Rename columns

Change data types

Make columns nullable

Multiple changes at once

Expression-based type changes

Alter embedding types and dimensions

Update field metadata

Canonical metadata keys

Drop columns

Set Up the example table

Drop single columns

Drop multiple columns

Versioning

Basic Versioning Example

Set Up the Table

Check Initial Version

Modify Data

Update Existing Data

Add New Data

Check Version Changes

Rollback to Previous Versions

View All Versions

Restore a Version Snapshot

Tag-Based Versioning

Delete Data From the Table

Go Back to Latest Version

Delete Data

Version History and Operations

Privileges