# Run experiments on branches
Source: https://docs.lancedb.com/agent-branch-experiments
Use LanceDB branches to isolate agent-driven experiments from main, evaluate them on a fixed test set, and promote only the winner.
LanceDB supports [branching](/tables/branching) to isolate experiments from the main table when working with agents.
Branches are useful when you instruct an agent to try several approaches without
affecting the production data on the `main` branch.
Each experiment gets its own writable table history.
For example, you can generate and compare embeddings from two text embedding
models on the given table:
| Branch | Model | Added column |
| -------------- | ---------------------------------------- | --------------- |
| `embed-minilm` | `sentence-transformers/all-MiniLM-L6-v2` | `vector_minilm` |
| `embed-nomic` | `nomic-embed-text` from Ollama | `vector_nomic` |
The branches share a logical table, but each has its own schema and history.
The original table handle still points to `main`. LanceDB does not have a
process-wide current branch.
Install the model packages, then download the `nomic-embed-text` model from Ollama
(or any other model you prefer):
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv add sentence-transformers ollama
ollama pull nomic-embed-text
```
It's always recommended to measure the results of an experiment on a fixed evaluation set.
Here's an example of how you could ask the agent to run both experiments and compare the results:
```text Agent prompt theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
Use the lancedb skill and the LanceDB branching documentation to compare two
embedding experiments on the `camelot_multimodal` table.
1. Fork `embed-minilm` from `main`. Build normalized embeddings from
`"{role}. {description}"` with
`sentence-transformers/all-MiniLM-L6-v2` and add them as `vector_minilm`.
2. Fork `embed-nomic` from `main`. Embed the identical text with Ollama
`nomic-embed-text` and add the results as `vector_nomic`.
3. Process rows in batches and write only through each branch-scoped table
handle. Verify that neither new column appears on `main`.
4. Evaluate both branches with the queries "wise magical advisor",
"treacherous rebel", and "virtuous Grail knight". Report the top three names,
latency, and whether the expected character ranks first.
5. Do not modify `main` and do not delete either branch. Recommend a winner
based on the results.
```
The vector dimensions may differ because the columns live on separate branches.
Use a different column name for each model. Within an experiment, use the same
model for the stored text and the search queries.
## Apply the winning experiment to `main`
The branch experiments leave you with the results side by side, and they
deliberately never touch `main`. Once you've picked a winner, apply it to
`main` yourself by rerunning that experiment's validated transformation
directly against the `main` table. Rerunning the reviewed operation is the
reliable path on both OSS and Enterprise: it replays exactly the transformation
you validated on the branch, and it works the same regardless of how the branch
evolved.
Suppose `nomic-embed-text` wins. Rerun the same Nomic embedding transformation
you validated on `embed-nomic` against `main` in batches, then call
`table.optimize()` on OSS. Verify a bounded vector search on `main`, and keep
the losing branch around until you're confident in the result.
See [Branches](/tables/branching) for more on how branches, versions, and tags
relate.
## More experiments you can run
Swapping embedding models on a branch while working with agents is only one example
of what you can do with the LanceDB skill. The table below shows other experiments
you can run with the skill.
| Hypothesis | Change on the branch | What to evaluate |
| ---------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------ |
| A different embedding model improves retrieval | Add a new vector column | Recall, ranking quality, latency, and cost |
| A new parser or OCR model improves source data | Reprocess files and backfill metadata | Validation failures and a reviewed sample |
| A new search setup works better | Build an index or change hybrid-search and reranking settings | Relevance and latency on a fixed query set |
| A curation rule improves the corpus | Deduplicate, classify, or filter records | False positives, false negatives, and row counts |
| A migration is safe to deploy | Add columns or run a backfill | Schema compatibility and application checks |
For each experiment, ask the agent to state the hypothesis, use a fixed
evaluation set, report the comparison, and, if you want, work on an alternate
branch that's isolated from `main` until you decide you want the new derived
column in `main`.
Once you've chosen a winner, rerun that validated transformation to ingest it into `main`
yourself, on either LanceDB OSS or Enterprise. See our documentation on [branches](/tables/branching)
for more information.
# SDKs and REST API Reference
Source: https://docs.lancedb.com/api-reference/index
SDK and REST API reference for LanceDB Enterprise and OSS.
For detailed information of the available functions and methods in your preferred language's SDKs,
refer to the API documentation linked below.
If you're looking for a REST API reference, visit the [REST API](/api-reference/rest) page.
If you're looking for conceptual and practical namespace guidance before diving into method signatures, see
[Namespaces and Catalog Model](/namespaces) and [Using Namespaces in SDKs](/namespaces/usage).
## Supported SDKs
Python, Typescript and Rust SDKs are officially supported by LanceDB. You can use these SDKs to interact with both LanceDB OSS and Enterprise deployments.
| Reference | Description |
| :-------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas & numpy integration |
| [Typescript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` |
| [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent-storage and high performance |
## REST API SDKs
Enterprise
REST API-based SDKs provide a convenient way to interact with LanceDB Enterprise deployments using the Lance REST Namespace API.
| Reference | Description |
| :------------------------------------------------------------------------ | ------------------------------- |
| [Java](https://lancedb.github.io/lancedb/java/java/) | REST API Enterprise SDK in Java |
## Community-driven SDKs
In addition to the officially supported SDKs, the LanceDB community may contribute SDKs in other languages.
These SDKs may not have the same level of support or feature parity as the official ones supported by LanceDB, but they can be an option
for users working in languages other than those listed above.
| Reference | Description |
| :---------------------------------------------------------------------------------------- | -------------------------------------------------- |
| [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK for LanceDB |
| [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings for LanceDB |
| [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK for LanceDB |
| [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package for LanceDB |
| [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings for LanceDB |
# REST API Reference
Source: https://docs.lancedb.com/api-reference/rest/index
API reference for LanceDB
[Lance REST Namespace](https://lance.org/format/namespace/) spec
is an OpenAPI protocol that enables reading, writing and managing Lance tables by connecting
those metadata services or building a custom metadata server in a standardized way.
LanceDB OSS allows you to interface with Lance tables via the REST Namespace.
LanceDB Enterprise provides an extended REST API with
additional endpoints for managing tables and data.
If you have specific needs or questions about the Enterprise REST API Namespace,
please [contact us](mailto:support@lancedb.com).
## Authentication
Enterprise
All HTTP requests to LanceDB APIs must contain an x-api-key header that specifies a valid API key and
must be encoded as JSON or Arrow RPC.
To authenticate to the Enterprise REST API, you need the endpoint for your deployment and a valid API key for that deployment.
### Get your Enterprise credentials
1. Obtain the following values from your LanceDB administrator or the LanceDB team that provisioned your Enterprise deployment:
* an API key
* your Enterprise REST endpoint
* your database name, if your deployment uses a private endpoint or `host_override`
2. Export those values in your terminal:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
export LANCEDB_API_KEY=""
export LANCEDB_URI="https://your-enterprise-endpoint.com"
export LANCEDB_DATABASE="your-database-name"
```
3. If your Enterprise deployment is private, connect through the private network endpoint provided for your deployment. For example, Azure Private Link deployments commonly use a private IP or an internal DNS name as the endpoint.
### Verify authentication
4. Check that you can reach the deployment and list tables:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl -X GET "$LANCEDB_URI/v1/tables" \
-H "Content-Type: application/json" \
-H "x-api-key: $LANCEDB_API_KEY" \
-H "x-lancedb-database: $LANCEDB_DATABASE"
```
If your deployment endpoint already includes the database host name, you can omit the `x-lancedb-database` header.
5. Create a table to confirm write access. Let's call it `words`.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl -X POST "$LANCEDB_URI/v1/tables/words" \
-H "Content-Type: application/vnd.apache.arrow.stream" \
-H "x-api-key: $LANCEDB_API_KEY" \
-H "x-lancedb-database: $LANCEDB_DATABASE"
```
6. Check that the table has been created:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl -X GET "$LANCEDB_URI/v1/tables" \
-H "Content-Type: application/json" \
-H "x-api-key: $LANCEDB_API_KEY" \
-H "x-lancedb-database: $LANCEDB_DATABASE"
```
That's it -- you're connected! Now, you can start adding data and querying it.
You can visit the tutorial section to build your own applications with LanceDB.
Check out our tutorials on building various applications with LanceDB.
# Create a materialized view
Source: https://docs.lancedb.com/api-reference/rest/materializedview/create-a-materialized-view
/api-reference/rest/openapi.yml post /v1/materialized_view/{id}/create
Create a materialized view at identifier `id`. The view may be
query-backed, UDTF-backed, or chunker-backed, controlled by the
`kind` discriminator.
# Trigger an async materialized view refresh
Source: https://docs.lancedb.com/api-reference/rest/materializedview/trigger-an-async-materialized-view-refresh
/api-reference/rest/openapi.yml post /v1/materialized_view/{id}/refresh
Trigger an asynchronous refresh job for materialized view `id`.
Returns a job ID for tracking.
# Check if a namespace exists
Source: https://docs.lancedb.com/api-reference/rest/namespace/check-if-a-namespace-exists
/api-reference/rest/openapi.yml post /v1/namespace/{id}/exists
Check if namespace `id` exists.
This operation must behave exactly like the DescribeNamespace API,
except it does not contain a response body.
REST NAMESPACE ONLY
REST namespace conveys the result through the HTTP status code with no response body.
The REST response maps to the `NamespaceExistsResponse` model as follows:
- a `200` response means the namespace exists; a `404` response means it does not
- response headers map to `context` via the `header.` prefix (see the `Context` schema)
# Create a new namespace
Source: https://docs.lancedb.com/api-reference/rest/namespace/create-a-new-namespace
/api-reference/rest/openapi.yml post /v1/namespace/{id}/create
Create new namespace `id`.
During the creation process, the implementation may modify user-provided `properties`,
such as adding additional properties like `created_at` to user-provided properties,
omitting any specific property, or performing actions based on any property value.
# Describe a namespace
Source: https://docs.lancedb.com/api-reference/rest/namespace/describe-a-namespace
/api-reference/rest/openapi.yml post /v1/namespace/{id}/describe
Describe the detailed information for namespace `id`.
# Drop a namespace
Source: https://docs.lancedb.com/api-reference/rest/namespace/drop-a-namespace
/api-reference/rest/openapi.yml post /v1/namespace/{id}/drop
Drop namespace `id` from its parent namespace.
# List namespaces
Source: https://docs.lancedb.com/api-reference/rest/namespace/list-namespaces
/api-reference/rest/openapi.yml get /v1/namespace/{id}/list
List all child namespace names of the parent namespace `id`.
REST NAMESPACE ONLY
REST namespace uses GET to perform this operation without a request body.
It passes in the `ListNamespacesRequest` information in the following way:
- `id`: pass through path parameter of the same name
- `page_token`: pass through query parameter of the same name
- `limit`: pass through query parameter of the same name
# List tables in a namespace
Source: https://docs.lancedb.com/api-reference/rest/namespace/list-tables-in-a-namespace
/api-reference/rest/openapi.yml get /v1/namespace/{id}/table/list
List all child table names of the parent namespace `id`.
REST NAMESPACE ONLY
REST namespace uses GET to perform this operation without a request body.
It passes in the `ListTablesRequest` information in the following way:
- `id`: pass through path parameter of the same name
- `page_token`: pass through query parameter of the same name
- `limit`: pass through query parameter of the same name
- `include_declared`: pass through query parameter of the same name
# Add new columns to table schema
Source: https://docs.lancedb.com/api-reference/rest/table/add-new-columns-to-table-schema
/api-reference/rest/openapi.yml post /v1/table/{id}/add_columns
Add new columns to table `id` using SQL expressions or default values.
# Analyze query execution plan
Source: https://docs.lancedb.com/api-reference/rest/table/analyze-query-execution-plan
/api-reference/rest/openapi.yml post /v1/table/{id}/analyze_plan
Analyze the query execution plan for a query against table `id`.
Returns detailed statistics and analysis of the query execution plan.
REST NAMESPACE ONLY
REST namespace returns the response as a plain string
instead of the `AnalyzeTableQueryPlanResponse` JSON object.
# Atomically commit a batch of mixed table operations
Source: https://docs.lancedb.com/api-reference/rest/table/atomically-commit-a-batch-of-mixed-table-operations
/api-reference/rest/openapi.yml post /v1/table/batch-commit
Atomically commit a batch of table operations. This is a generalized version
of `BatchCreateTableVersions` that supports mixed operation types within a
single atomic transaction at the metadata layer.
Supported operation types:
- `DeclareTable`: Declare (reserve) a new table
- `CreateTableVersion`: Create a new version entry for a table
- `DeleteTableVersions`: Delete version ranges from a table
- `DeregisterTable`: Deregister (soft-delete) a table
All operations are committed atomically: either all succeed or none are applied.
# Atomically create versions for multiple tables
Source: https://docs.lancedb.com/api-reference/rest/table/atomically-create-versions-for-multiple-tables
/api-reference/rest/openapi.yml post /v1/table/version/batch-create
Atomically create new version entries for multiple tables.
This operation is atomic: either all table versions are created successfully,
or none are created. If any version creation fails (e.g., due to conflict),
the entire batch operation fails.
Each entry in the request specifies the table identifier and version details.
This supports `put_if_not_exists` semantics for each version entry.
# Check if a table exists
Source: https://docs.lancedb.com/api-reference/rest/table/check-if-a-table-exists
/api-reference/rest/openapi.yml post /v1/table/{id}/exists
Check if table `id` exists.
This operation should behave exactly like DescribeTable,
except it does not contain a response body.
REST NAMESPACE ONLY
REST namespace conveys the result through the HTTP status code with no response body.
The REST response maps to the `TableExistsResponse` model as follows:
- a `200` response means the table exists; a `404` response means it does not
- response headers map to `context` via the `header.` prefix (see the `Context` schema)
For DirectoryNamespace implementation, a table exists if either:
- The table has Lance data versions (regular table created with CreateTable)
- A `.lance-reserved` file exists in the table directory (declared table created with DeclareTable)
# Count rows in a table
Source: https://docs.lancedb.com/api-reference/rest/table/count-rows-in-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/count_rows
Count the number of rows in table `id`
REST NAMESPACE ONLY
REST namespace returns the response as a plain integer
instead of the `CountTableRowsResponse` JSON object.
The REST response maps to the `CountTableRowsResponse` model as follows:
- the integer response body maps to `count`
- response headers map to `context` via the `header.` prefix (see the `Context` schema)
# Create a new branch
Source: https://docs.lancedb.com/api-reference/rest/table/create-a-new-branch
/api-reference/rest/openapi.yml post /v1/table/{id}/branches/create
Create a new branch for table `id` starting from a source ref (another
branch and/or version), defaulting to the latest version of the main branch.
# Create a new table version
Source: https://docs.lancedb.com/api-reference/rest/table/create-a-new-table-version
/api-reference/rest/openapi.yml post /v1/table/{id}/version/create
Create a new version entry for table `id`.
This operation supports `put_if_not_exists` semantics.
The operation will fail with 409 Conflict if the version already exists.
# Create a new tag
Source: https://docs.lancedb.com/api-reference/rest/table/create-a-new-tag
/api-reference/rest/openapi.yml post /v1/table/{id}/tags/create
Create a new tag for table `id` that points to a specific version.
# Create a scalar index on a table
Source: https://docs.lancedb.com/api-reference/rest/table/create-a-scalar-index-on-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/create_scalar_index
Create a scalar index on a table field for faster filtering operations.
Supports scalar indexes (BTREE, BITMAP, LABEL_LIST, FTS, etc.).
This is an alias for CreateTableIndex specifically for scalar indexes.
Index creation is handled asynchronously.
Use the `ListTableIndices` and `DescribeTableIndexStats` operations to monitor index creation progress.
# Create a table with the given name
Source: https://docs.lancedb.com/api-reference/rest/table/create-a-table-with-the-given-name
/api-reference/rest/openapi.yml post /v1/table/{id}/create
Create table `id` in the namespace with the given data in Arrow IPC stream.
The schema of the Arrow IPC stream is used as the table schema.
If the stream is empty, the API creates a new empty table.
REST NAMESPACE ONLY
REST namespace uses Arrow IPC stream as the request body.
It passes in the `CreateTableRequest` information in the following way:
- `id`: pass through path parameter of the same name
- `mode`: pass through query parameter of the same name
- `properties`: serialize as a single JSON-encoded query parameter such as
`properties={"user":"alice","team":"eng"}`; these are business logic properties
managed by the namespace implementation outside Lance context
- `storage_options`: serialize as a single JSON-encoded query parameter such as
`storage_options={"aws_region":"us-east-1","timeout":"30s"}`; these configure
write-time overrides for data and metadata written during table creation
# Create an index on a table
Source: https://docs.lancedb.com/api-reference/rest/table/create-an-index-on-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/create_index
Create an index on a table field for faster search operations.
Supports vector indexes (IVF_FLAT, IVF_HNSW_SQ, IVF_PQ, etc.) and scalar indexes (BTREE, BITMAP, FTS, etc.).
Index creation is handled asynchronously.
Use the `ListTableIndices` and `DescribeTableIndexStats` operations to monitor index creation progress.
# Declare a table
Source: https://docs.lancedb.com/api-reference/rest/table/declare-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/declare
Declare a table with the given name without touching storage.
This is a metadata-only operation that records the table existence and sets up aspects like access control.
For DirectoryNamespace implementation, this creates a `.lance-reserved` file in the table directory
to mark the table's existence without creating actual Lance data files.
# Delete a branch
Source: https://docs.lancedb.com/api-reference/rest/table/delete-a-branch
/api-reference/rest/openapi.yml post /v1/table/{id}/branches/delete
Delete an existing branch from table `id`.
# Delete a tag
Source: https://docs.lancedb.com/api-reference/rest/table/delete-a-tag
/api-reference/rest/openapi.yml post /v1/table/{id}/tags/delete
Delete an existing tag from table `id`.
# Delete rows from a table
Source: https://docs.lancedb.com/api-reference/rest/table/delete-rows-from-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/delete
Delete rows from table `id`.
# Delete table version records
Source: https://docs.lancedb.com/api-reference/rest/table/delete-table-version-records
/api-reference/rest/openapi.yml post /v1/table/{id}/version/delete
Delete version metadata records for table `id`.
This operation deletes version tracking records, NOT the actual table data.
It supports deleting ranges of versions for efficient bulk cleanup.
Special range values:
- `start_version: 0` with `end_version: -1` means delete ALL version records
# Deregister a table
Source: https://docs.lancedb.com/api-reference/rest/table/deregister-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/deregister
Deregister table `id` from its namespace.
# Describe a specific table version
Source: https://docs.lancedb.com/api-reference/rest/table/describe-a-specific-table-version
/api-reference/rest/openapi.yml post /v1/table/{id}/version/describe
Describe the detailed information for a specific version of table `id`.
Returns the manifest path and metadata for the specified version.
# Describe information of a table
Source: https://docs.lancedb.com/api-reference/rest/table/describe-information-of-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/describe
Describe the detailed information for table `id`.
REST NAMESPACE ONLY
REST namespace passes `with_table_uri`, `load_detailed_metadata`, and `check_declared` as query parameters instead of in the request body.
# Drop a specific index
Source: https://docs.lancedb.com/api-reference/rest/table/drop-a-specific-index
/api-reference/rest/openapi.yml post /v1/table/{id}/index/{index_name}/drop
Drop the specified index from table `id`.
REST NAMESPACE ONLY
REST namespace does not use a request body for this operation.
The `DropTableIndexRequest` information is passed in the following way:
- `id`: pass through path parameter of the same name
- `index_name`: pass through path parameter of the same name
# Drop a table
Source: https://docs.lancedb.com/api-reference/rest/table/drop-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/drop
Drop table `id` and delete its data.
REST NAMESPACE ONLY
REST namespace does not use a request body for this operation.
The `DropTableRequest` information is passed in the following way:
- `id`: pass through path parameter of the same name
# Get query execution plan explanation
Source: https://docs.lancedb.com/api-reference/rest/table/get-query-execution-plan-explanation
/api-reference/rest/openapi.yml post /v1/table/{id}/explain_plan
Get the query execution plan for a query against table `id`.
Returns a human-readable explanation of how the query will be executed.
REST NAMESPACE ONLY
REST namespace returns the response as a plain string
instead of the `ExplainTableQueryPlanResponse` JSON object.
# Get table index statistics
Source: https://docs.lancedb.com/api-reference/rest/table/get-table-index-statistics
/api-reference/rest/openapi.yml post /v1/table/{id}/index/{index_name}/stats
Get statistics for a specific index on a table. Returns information about
the index type, distance type (for vector indices), and row counts.
# Get table statistics
Source: https://docs.lancedb.com/api-reference/rest/table/get-table-statistics
/api-reference/rest/openapi.yml post /v1/table/{id}/stats
Get statistics for table `id`, including row counts, data sizes, and column statistics.
# Get version for a specific tag
Source: https://docs.lancedb.com/api-reference/rest/table/get-version-for-a-specific-tag
/api-reference/rest/openapi.yml post /v1/table/{id}/tags/version
Get the version number that a specific tag points to for table `id`.
# Insert records into a table
Source: https://docs.lancedb.com/api-reference/rest/table/insert-records-into-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/insert
Insert new records into table `id`.
For tables that have been declared but not yet created on storage
(is_only_declared=true), this operation will create the table with
the provided data.
REST NAMESPACE ONLY
REST namespace uses Arrow IPC stream as the request body.
It passes in the `InsertIntoTableRequest` information in the following way:
- `id`: pass through path parameter of the same name
- `mode`: pass through query parameter of the same name
# List all branches for a table
Source: https://docs.lancedb.com/api-reference/rest/table/list-all-branches-for-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/branches/list
List all branches that have been created for table `id`.
Returns a map of branch names to their contents.
REST NAMESPACE ONLY
REST namespace does not use a request body for this operation.
The `ListTableBranchesRequest` information is passed in the following way:
- `id`: pass through path parameter of the same name
- `page_token`: pass through query parameter of the same name
- `limit`: pass through query parameter of the same name
# List all tables
Source: https://docs.lancedb.com/api-reference/rest/table/list-all-tables
/api-reference/rest/openapi.yml get /v1/table
List all tables across all namespaces.
REST NAMESPACE ONLY
REST namespace uses GET to perform this operation without a request body.
It passes in the `ListAllTablesRequest` information in the following way:
- `page_token`: pass through query parameter of the same name
- `limit`: pass through query parameter of the same name
- `delimiter`: pass through query parameter of the same name
- `include_declared`: pass through query parameter of the same name
# List all tags for a table
Source: https://docs.lancedb.com/api-reference/rest/table/list-all-tags-for-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/tags/list
List all tags that have been created for table `id`.
Returns a map of tag names to their corresponding version numbers and metadata.
REST NAMESPACE ONLY
REST namespace does not use a request body for this operation.
The `ListTableTagsRequest` information is passed in the following way:
- `id`: pass through path parameter of the same name
- `page_token`: pass through query parameter of the same name
- `limit`: pass through query parameter of the same name
# List all versions of a table
Source: https://docs.lancedb.com/api-reference/rest/table/list-all-versions-of-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/version/list
List all versions (commits) of table `id` with their metadata.
Use `descending=true` to guarantee versions are returned in descending order (latest to oldest).
Otherwise, the ordering is implementation-defined.
REST NAMESPACE ONLY
REST namespace does not use a request body for this operation.
The `ListTableVersionsRequest` information is passed in the following way:
- `id`: pass through path parameter of the same name
- `page_token`: pass through query parameter of the same name
- `limit`: pass through query parameter of the same name
- `descending`: pass through query parameter of the same name
# List indexes on a table
Source: https://docs.lancedb.com/api-reference/rest/table/list-indexes-on-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/index/list
List all indices created on a table. Returns information about each index
including name, columns, status, and UUID.
# Merge insert (upsert) records into a table
Source: https://docs.lancedb.com/api-reference/rest/table/merge-insert-upsert-records-into-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/merge_insert
Performs a merge insert (upsert) operation on table `id`.
This operation updates existing rows
based on a matching column and inserts new rows that don't match.
It returns the number of rows inserted and updated.
For tables that have been declared but not yet created on storage
(is_only_declared=true), this operation will create the table with
the provided data (since there are no existing rows to merge with).
REST NAMESPACE ONLY
REST namespace uses Arrow IPC stream as the request body.
It passes in the `MergeInsertIntoTableRequest` information in the following way:
- `id`: pass through path parameter of the same name
- `on`: pass through query parameter of the same name
- `when_matched_update_all`: pass through query parameter of the same name
- `when_matched_update_all_filt`: pass through query parameter of the same name
- `when_not_matched_insert_all`: pass through query parameter of the same name
- `when_not_matched_by_source_delete`: pass through query parameter of the same name
- `when_not_matched_by_source_delete_filt`: pass through query parameter of the same name
# Modify existing columns
Source: https://docs.lancedb.com/api-reference/rest/table/modify-existing-columns
/api-reference/rest/openapi.yml post /v1/table/{id}/alter_columns
Modify existing columns in table `id`, such as renaming or changing data types.
# Query a table
Source: https://docs.lancedb.com/api-reference/rest/table/query-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/query
Query table `id` with vector search, full text search and optional SQL filtering.
Returns results in Arrow IPC file or stream format.
REST NAMESPACE ONLY
REST namespace returns the response as Arrow IPC file binary data
instead of the `QueryTableResponse` JSON object.
The REST response maps to the `QueryTableResponse` model as follows:
- the Arrow IPC file binary body maps to `data`
- response headers map to `context` via the `header.` prefix (see the `Context` schema)
# Register a table to a namespace
Source: https://docs.lancedb.com/api-reference/rest/table/register-a-table-to-a-namespace
/api-reference/rest/openapi.yml post /v1/table/{id}/register
Register an existing table at a given storage location as `id`.
# Remove columns from table
Source: https://docs.lancedb.com/api-reference/rest/table/remove-columns-from-table
/api-reference/rest/openapi.yml post /v1/table/{id}/drop_columns
Remove specified columns from table `id`.
# Rename a table
Source: https://docs.lancedb.com/api-reference/rest/table/rename-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/rename
Rename table `id` to a new name.
# Restore table to a specific version
Source: https://docs.lancedb.com/api-reference/rest/table/restore-table-to-a-specific-version
/api-reference/rest/openapi.yml post /v1/table/{id}/restore
Restore table `id` to a specific version.
# Trigger an async column backfill job
Source: https://docs.lancedb.com/api-reference/rest/table/trigger-an-async-column-backfill-job
/api-reference/rest/openapi.yml post /v1/table/{id}/backfill_column
Trigger an asynchronous backfill job for a computed column on table `id`.
The column must be a virtual (UDF-backed) column. Returns a job ID for tracking.
# Update a tag to point to a different version
Source: https://docs.lancedb.com/api-reference/rest/table/update-a-tag-to-point-to-a-different-version
/api-reference/rest/openapi.yml post /v1/table/{id}/tags/update
Update an existing tag for table `id` to point to a different version.
# Update per-field metadata
Source: https://docs.lancedb.com/api-reference/rest/table/update-per-field-metadata
/api-reference/rest/openapi.yml post /v1/table/{id}/update_field_metadata
Update the Arrow field (column) metadata for table `id`.
Each entry targets a field by `path` and merges the provided key-value
pairs into that field's existing metadata, or replaces it when `replace`
is true. A null metadata value deletes that key.
# Update rows in a table
Source: https://docs.lancedb.com/api-reference/rest/table/update-rows-in-a-table
/api-reference/rest/openapi.yml post /v1/table/{id}/update
Update existing rows in table `id`.
# Update table schema metadata
Source: https://docs.lancedb.com/api-reference/rest/table/update-table-schema-metadata
/api-reference/rest/openapi.yml post /v1/table/{id}/schema_metadata/update
Replace the schema metadata for table `id` with the provided key-value pairs.
REST NAMESPACE ONLY
REST namespace uses a direct object (map of string to string) as both request and response body
instead of the wrapped `UpdateTableSchemaMetadataRequest` and `UpdateTableSchemaMetadataResponse`.
# Alter information of a transaction.
Source: https://docs.lancedb.com/api-reference/rest/transaction/alter-information-of-a-transaction
/api-reference/rest/openapi.yml post /v1/transaction/{id}/alter
Alter a transaction with a list of actions such as setting status or properties.
The server should either succeed and apply all actions, or fail and apply no action.
# Describe information about a transaction
Source: https://docs.lancedb.com/api-reference/rest/transaction/describe-information-about-a-transaction
/api-reference/rest/openapi.yml post /v1/transaction/{id}/describe
Return a detailed information for a given transaction
# Tutorial: Use the LanceDB agent skill
Source: https://docs.lancedb.com/build-with-ai-agents
Install the LanceDB skill and use an AI coding agent to quickly build a multimodal ingestion pipeline.
The LanceDB agent skill gives coding agents a maintained reference for the
Python and TypeScript APIs. It also covers portable OSS and Enterprise code,
ingestion performance, and branch operations. Install it in your project:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Install the lancedb skill
npx skills add lancedb/lancedb
# Update the skill
npx skills update lancedb
```
The skill supplements the agent's training data with current LanceDB
instructions.
The installer downloads the
[`lancedb` skills plugin](https://github.com/lancedb/lancedb/tree/main/plugins/lancedb) and makes it
available to the agents you select. The installation directory depends on the
agent client. Universal agents typically use `.agents/skills`.
The installer also creates or updates `skills-lock.json`. This file records
where the skill came from, its path in the source repository, and a hash of the
installed content.
```json theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
{
"version": 1,
"skills": {
"lancedb": {
"source": "lancedb/lancedb",
"sourceType": "github",
"skillPath": "plugins/lancedb/skills/lancedb/SKILL.md",
"computedHash": "..."
}
}
}
```
Commit the lockfile if you want skill updates to go through code review. Before
committing a new hash, inspect the changes to `SKILL.md` and its reference
files. The repository history will then show which snapshot each revision used.
## Get started with the LanceDB agent skill
This tutorial uses the Camelot dataset from the [quickstart](/quickstart), with
a portrait added for each character. Each LanceDB row contains validated
metadata and raw JPEG bytes. Text, images, and any embeddings you add later
remain in the same table.
### 1. Download the multimodal dataset
From a new project directory, download the JSON file and portraits:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
mkdir -p data/img
BASE_URL="https://docs.lancedb.com/static/assets/tutorials/build-with-ai-agents/camelot/data"
curl -fsSL "$BASE_URL/camelot.json" -o data/camelot.json
for image in \
arthur.jpg \
guinevere.jpg \
merlin.jpg \
mordred.jpg \
sir_galahad.jpg \
sir_gawain.jpg \
sir_lancelot.jpg \
sir_percival.jpg
do
curl -fsSL "$BASE_URL/img/$image" -o "data/img/$image"
done
```
Each JSON record has this shape:
```json theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
{
"id": 2,
"name": "Merlin",
"role": "Wizard and Advisor",
"description": "A powerful wizard and prophet who mentors Arthur.",
"stats": {
"strength": 2,
"courage": 4,
"magic": 5,
"wisdom": 5
},
"img": "data/img/merlin.jpg"
}
```
JSON input may have missing fields, unexpected fields, or values of the wrong
type. The LanceDB skill tells the agent to validate each record with strict
Pydantic models before writing it.
After the agent writes the pipeline, inspect the schema, batching, and write
path rather than assuming it followed the skill correctly.
### 2. Prompt your agent to build the pipeline
Install the Python packages used by the example:
```bash icon=terminal theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv init
uv add lancedb pyarrow pydantic
```
If you're using LanceDB Enterprise, ask the agent to ingest into an Enterprise table,
provide the relevant environment variables for connecting to your Enterprise deployment
in a local `.env` file, and point the agent to it.
```text .env theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
LANCEDB_URI=db://your_project_name
LANCEDB_API_KEY=your_api_key_here
LANCEDB_REGION=us-east-1
LANCEDB_HOST_OVERRIDE=https://hostname@ip_address
```
If you're using LanceDB OSS, no connection settings are required, as it runs as an
embedded retrieval library. A simple prompt like this should work:
```text Agent prompt theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# If using OSS
Use the lancedb skill to ingest the dataset in `data/` into a LanceDB
OSS table.
# If using enterprise
Use the lancedb skill to ingest the dataset in `data/` into a LanceDB
Enterprise table using the connection information in `.env`.
```
Agents that scan `.agents/skills` should find the skill automatically. If your
agent does not find it automatically, simply ask it to use the `lancedb` skill in the prompt.
That should be enough! The agent will create `ingest_multimodal.py`, or similar.
The following sections inspect the script to verify that it follows the skill's guidance.
#### Data validation
The skill encourages the agent to validate each record with Pydantic before writing it.
The agent should ideally define a schema for the table and a nested schema for the
`stats` field.
In this case, our agent correctly defined `Character` and `Stats` Pydantic models
and validated the JSON before adding it to the table.
#### Batched ingestion
Naively calling `table.add()` once per row is slow, and is considered an anti-pattern
in LanceDB. The skill encourages the agent to collect incoming rows into batches and
write them with a single `table.add()` call. When you use the skill, the agent should
produce something like this:
The script calls `Character.model_validate(...)` before adding a record to the
batch. If validation fails, that batch is never written. The function yields up
to `batch_size` rows at a time, providing an iterable of batches for the ingestion step,
shown next.
#### Table maintenance
For LanceDB OSS, the skill instructs the agent to call
`table.optimize()` after the ingestion loop. This compacts small fragments, cleans up
old versions according to the retention policy, and incorporates new data into indexes.
If you're using LanceDB Enterprise, the skill mentions that this step is not needed
because LanceDB Enterprise handles maintenance automatically.
This example dataset has only eight rows, so the default batch size writes it in
one call. Larger inputs should still avoid single-row write commits.
### 3. Run the OSS pipeline
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv run python ingest_multimodal.py
```
The table now contains the validated character records and their JPEG bytes.
Here are the first three rows:
| Image | Character | Role |
| ----------------------------- | --------------- | ------------------ |
| ![King Arthur]()
| King Arthur | King of Camelot |
| ![Merlin]()
| Merlin | Wizard and Advisor |
| ![Queen Guinevere]()
| Queen Guinevere | Queen of Camelot |
Once your pipeline works, you can [run experiments on branches](/agent-branch-experiments)
to try new embedding models, parsers, or search settings without touching `main`.
## Takeaways
The example in this tutorial was small, but similar ideas apply to other workflows, too.
Give the agent the data source, the constraints it must respect, and the
artifacts it should return.
The skill supplies LanceDB-specific guidance, but it's the user's responsibility to
ensure the output makes sense for the application.
### Try the skill with your own dataset
The skills shown in this tutorial should generalize reasonably well to other use cases.
If you find any issues, open [an issue](https://github.com/lancedb/lancedb/issues) on GitHub,
clearly describing the intended behavior.
You can choose OSS or Enterprise based on how the work will run:
Start with [LanceDB OSS](/quickstart) during the early stages of a project
when an agent is helping you prototype,
explore a dataset, or run small workflows on a subset of the data. The application owns
the storage and lifecycle work, so ask the agent to validate inputs, write in
batches, and it will include table maintenance operations such as `optimize()`
where appropriate.
Choose [LanceDB Enterprise](/enterprise) when the resulting table becomes
shared production infrastructure, and the workload needs distributed capacity,
private deployment, or platform-managed operations. The underlying data format and table
API stay the same, so the pipeline does not need to be redesigned. The agent
instead connects to a remote `db://` table and lets the cluster handle
maintenance and background work.
# ADE20K
Source: https://docs.lancedb.com/datasets/ade20k
A Lance-formatted version of the full ADE20K scene parsing benchmark, sourced from 1aurent/ADE20K. Each row is one scene image with its inline JPEG bytes, a per-pixel semantic segmentation map encoded as PNG bytes, an optional instance map, scene…
Source dataset card and downloadable files for `lance-format/ade20k-lance`.
A Lance-formatted version of the full [ADE20K scene parsing benchmark](https://groups.csail.mit.edu/vision/datasets/ADE20K/), sourced from [`1aurent/ADE20K`](https://huggingface.co/datasets/1aurent/ADE20K). Each row is one scene image with its inline JPEG bytes, a per-pixel semantic segmentation map encoded as PNG bytes, an optional instance map, scene class labels, the full per-polygon object-name list, an OpenCLIP image embedding, and pre-built indices — all available directly from the Hub at `hf://datasets/lance-format/ade20k-lance/data`.
## Key features
* **Inline image and segmentation bytes** — both the JPEG image and the RGB-encoded PNG segmentation map ride on the same row, so an annotated example is a single row read with no sidecar files.
* **Per-polygon object metadata** — `object_names` keeps the full list (one entry per annotated polygon), `objects_present` is the deduped set used for class-presence filters, and `num_objects` is precomputed.
* **CLIP image embeddings** (`image_emb`, OpenCLIP ViT-B/32, 512-d, cosine-normalized) for visual retrieval over scenes.
* **Indices shipped on disk** — `IVF_PQ` on `image_emb`, `BTREE` on `num_objects`, and `LABEL_LIST` on `objects_present` for fast `array_has_any` / `array_has_all` predicates.
## Splits
| Split | Rows |
| ------------------ | ------ |
| `train.lance` | 25,574 |
| `validation.lance` | 2,000 |
## Schema
| Column | Type | Notes |
| ----------------- | ------------------------------- | --------------------------------------------------------------------------- |
| `id` | `int64` | Row index within split |
| `image` | `large_binary` | Inline JPEG bytes |
| `segmentation` | `large_binary` | Inline PNG bytes — semantic segmentation map (RGB encoding per ADE20K spec) |
| `instance` | `large_binary?` | Inline PNG bytes — instance map; null if not provided |
| `filename` | `string` | ADE20K relative filename |
| `scene` | `list` | Scene class labels (e.g. `["bathroom"]`) |
| `object_names` | `list` | Per-polygon object names (one entry per polygon, not deduped) |
| `objects_present` | `list` | Deduped object names — feeds the `LABEL_LIST` index |
| `num_objects` | `int32` | Number of annotated objects |
| `image_emb` | `fixed_size_list` | OpenCLIP ViT-B/32 image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `BTREE` on `num_objects` — fast range filters on scene complexity
* `LABEL_LIST` on `objects_present` — supports `array_has_any` / `array_has_all` for class-presence filtering
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/ade20k-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["filename"], row["scene"], row["num_objects"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/ade20k-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/ade20k-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access, ANN search, and any mutation are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/ade20k-lance --repo-type dataset --local-dir ./ade20k-lance
> ```
>
> Then point Lance or LanceDB at `./ade20k-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes approximate-nearest-neighbor scene retrieval a single call. In production you would encode a query image through the same OpenCLIP ViT-B/32 model used at ingest and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the embedding stored on row 42 as a runnable stand-in, so the snippet works without loading any model.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/ade20k-lance/data")
tbl = db.open_table("validation")
seed = (
tbl.search()
.select(["image_emb", "filename", "scene"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["filename", "scene", "objects_present"])
.limit(10)
.to_list()
)
print("query scene:", seed["scene"])
for r in hits:
print(f" {r['filename']} scene={r['scene']} objs={r['objects_present'][:5]}")
```
Because the embeddings are cosine-normalized, the first hit will typically be the source image itself — a useful sanity check. Tune `nprobes` and `refine_factor` to trade recall against latency for your workload.
## Curate
Curation for a semantic-segmentation workflow usually means picking scenes that contain specific classes, possibly bounded by complexity. The `LABEL_LIST` index on `objects_present` makes class-presence predicates trivial, and Lance evaluates them inside the same scan as a structural filter on `num_objects`. The bounded `.limit(500)` keeps the result small and inspectable, and the `segmentation` blob is left out of the projection so the candidate scan is dominated by metadata, not PNG bytes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/ade20k-lance/data")
tbl = db.open_table("validation")
candidates = (
tbl.search()
.where(
"array_has_all(objects_present, ['bed', 'window']) AND num_objects >= 8",
prefilter=True,
)
.select(["id", "filename", "scene", "objects_present", "num_objects"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates; first scene: {candidates[0]['scene']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `id`s, or feed into the Evolve and Train workflows below. Swapping `array_has_all` for `array_has_any` widens the recall; replacing the structural predicate with `num_objects BETWEEN 3 AND 6` selects simpler scenes for an ablation slice.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `has_person` flag and a `scene_label` string pulled out of the `scene` list, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./ade20k-lance/data") # local copy required for writes
tbl = db.open_table("validation")
tbl.add_columns({
"has_person": "array_has_any(objects_present, ['person'])",
"scene_label": "element_at(scene, 1)",
"complexity_bucket": "CASE WHEN num_objects < 5 THEN 'sparse' "
"WHEN num_objects < 15 THEN 'medium' ELSE 'dense' END",
})
```
If the values you want to attach already live in another table (offline panoptic ids, predictions from a baseline segmenter, a second-pass embedding), merge them in by joining on `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"id": pa.array([0, 1, 2], type=pa.int64()),
"baseline_miou": pa.array([0.41, 0.55, 0.62]),
})
tbl.merge(predictions, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., re-running a segmentation model over the image bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a semantic-segmentation run, project the JPEG bytes and the segmentation PNG bytes; both are decoded inside the training step. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/ade20k-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "segmentation"])
loader = DataLoader(train_ds, batch_size=8, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the JPEG and PNG byte columns; decode both,
# remap the ADE20K RGB-encoded mask to class ids, forward, loss...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "objects_present"]` to `select_columns(...)` on the next run skips JPEG and PNG decoding entirely and reads only the cached 512-d vectors plus the deduped class list, which is the right shape for training a lightweight scene classifier or a class-presence probe on top of frozen features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/ade20k-lance/data")
tbl = db.open_table("validation")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./ade20k-lance/data")
local_tbl = local_db.open_table("validation")
local_tbl.tags.create("segmenter-baseline-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("validation", version="segmenter-baseline-v1")
tbl_v5 = db.open_table("validation", version=5)
```
Pinning supports two workflows. A serving pipeline locked to `segmenter-baseline-v1` keeps reading the exact same segmentation maps and class lists while the dataset evolves in parallel; newly merged predictions or evolved columns do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images, so changes in mIoU reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/ade20k-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("array_has_any(objects_present, ['bed', 'sofa', 'chair']) AND num_objects >= 5")
.select(["id", "image", "segmentation", "filename", "scene",
"objects_present", "num_objects", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./ade20k-indoor-subset")
local_db.create_table("train", batches)
```
The resulting `./ade20k-indoor-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/ade20k-lance/data` for `./ade20k-indoor-subset`.
## Source & license
Converted from [`1aurent/ADE20K`](https://huggingface.co/datasets/1aurent/ADE20K). ADE20K is released under the [BSD 3-Clause license](https://ade20k.csail.mit.edu/terms/) by the MIT CSAIL Computer Vision group.
## Citation
```
@inproceedings{zhou2017scene,
title={Scene Parsing through ADE20K Dataset},
author={Zhou, Bolei and Zhao, Hang and Puig, Xavier and Fidler, Sanja and Barriuso, Adela and Torralba, Antonio},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2017}
}
```
# ChartQA
Source: https://docs.lancedb.com/datasets/chartqa
A Lance-formatted version of ChartQA, a benchmark for question answering over scientific and business charts that demands a mix of logical and visual reasoning, redistributed via lmms-lab/ChartQA. Each row carries the chart image as inline JPEG…
Source dataset card and downloadable files for `lance-format/chartqa-lance`.
A Lance-formatted version of [ChartQA](https://github.com/vis-nlp/ChartQA), a benchmark for question answering over scientific and business charts that demands a mix of logical and visual reasoning, redistributed via [`lmms-lab/ChartQA`](https://huggingface.co/datasets/lmms-lab/ChartQA). Each row carries the chart image as inline JPEG bytes, the natural-language question and reference answer(s), a question-type tag (`human` vs `augmented`), and paired CLIP embeddings for the image and the question — all available directly from the Hub at `hf://datasets/lance-format/chartqa-lance/data`.
## Key features
* **Inline chart image bytes** in the `image` column — no sidecar files, no image folders.
* **Paired CLIP embeddings in the same row** — `image_emb` and `question_emb` (ViT-B/32, 512-dim, cosine-normalized) — so visual and textual retrieval are one indexed lookup.
* **All reference answers preserved in `answers`** alongside a canonical `answer` string used for full-text search.
* **Pre-built ANN, FTS, and scalar indices** covering both embedding columns, the question and answer strings, and the `type` tag.
## Splits
| Split | Rows | Notes |
| ------------ | ----- | ----------------------------------------- |
| `test.lance` | 2,500 | Public test slice from `lmms-lab/ChartQA` |
> The `lmms-lab/ChartQA` redistribution exposes the test split only. Train and validation live in the original ChartQA release; extend `chartqa/dataprep.py` with additional sources to add them.
## Schema
| Column | Type | Notes |
| -------------- | ------------------------------- | ------------------------------------------------ |
| `id` | `int64` | Row index within split (natural join key) |
| `image` | `large_binary` | Inline JPEG bytes |
| `image_id` | `string?` | Source does not assign explicit ids — null |
| `question_id` | `string?` | Source does not assign explicit ids — null |
| `question` | `string` | Natural-language question |
| `answers` | `list` | Reference answer(s), typically a single string |
| `answer` | `string` | First reference answer — canonical, used for FTS |
| `type` | `string?` | Question type (`human` vs `augmented`) |
| `image_emb` | `fixed_size_list` | CLIP image embedding (cosine-normalized) |
| `question_emb` | `fixed_size_list` | CLIP text embedding of the question |
## Pre-built indices
* `IVF_PQ` on `image_emb` — image-side vector search (cosine)
* `IVF_PQ` on `question_emb` — text-side vector search (cosine)
* `INVERTED` (FTS) on `question` and `answer` — keyword and hybrid search
* `BITMAP` on `type` — fast filtering by question type
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/chartqa-lance", split="test", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["answer"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/chartqa-lance/data")
tbl = db.open_table("test")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/chartqa-lance/data/test.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/chartqa-lance --repo-type dataset --local-dir ./chartqa-lance
> ```
>
> Then point Lance or LanceDB at `./chartqa-lance/data`.
## Search
The bundled `IVF_PQ` index on `question_emb` makes question-to-question retrieval a single call: encode a query with the same CLIP model used at ingest (ViT-B/32, cosine-normalized) and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the `question_emb` already stored in row 42 as a runnable stand-in, so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/chartqa-lance/data")
tbl = db.open_table("test")
seed = (
tbl.search()
.select(["question_emb", "question"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="question_emb")
.metric("cosine")
.select(["question", "answer", "type"])
.limit(10)
.to_list()
)
print("query:", seed["question"])
for r in hits:
print(f" [{r['type']}] {r['question'][:70]} -> {r['answer']}")
```
Swap `vector_column_name="question_emb"` for `image_emb` to do question-to-chart retrieval against the visual embedding instead — useful for finding charts whose layout is similar to a given prompt encoding.
Because the dataset also ships an `INVERTED` index on `question` and `answer`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges the two result lists and reranks them in a single call, which is useful when a phrase like "percentage" or "highest bar" must literally appear in the question but you still want CLIP to do the heavy lifting on semantic similarity.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid", vector_column_name="question_emb")
.vector(seed["question_emb"])
.text("percentage")
.select(["question", "answer", "type"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" [{r['type']}] {r['question'][:70]} -> {r['answer']}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
A typical curation pass combines a content predicate on the question text with a structural predicate on the question-type tag. Stacking both inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(500)` makes it cheap to inspect before committing the subset to anything downstream. The example below collects human-authored questions that mention a percentage, which is a common slice for evaluating numeric-reasoning behaviour.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/chartqa-lance/data")
tbl = db.open_table("test")
candidates = (
tbl.search("percentage OR percent")
.where("type = 'human'", prefilter=True)
.select(["id", "question", "answer", "type"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates; first: {candidates[0]['question'][:80]}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of row ids, or feed into the Evolve and Train workflows below. The `image` column is never read, so the network traffic for a 500-row candidate scan is dominated by question and answer text rather than chart JPEGs.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds `answer_length`, an `is_yes_no` flag, and an `is_numeric` flag, any of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./chartqa-lance/data") # local copy required for writes
tbl = db.open_table("test")
tbl.add_columns({
"answer_length": "length(answer)",
"is_yes_no": "lower(answer) IN ('yes', 'no')",
"is_numeric": "regexp_match(answer, '^-?[0-9]+(\\.[0-9]+)?%?$') IS NOT NULL",
})
```
If the values you want to attach already live in another table (model predictions on the test set, reasoning-chain annotations, a difficulty score), merge them in by joining on the `id` column:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"id": pa.array([0, 1, 2]),
"pred_answer": pa.array(["12%", "Yes", "34"]),
"is_correct": pa.array([True, True, False]),
})
tbl.merge(predictions, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a chart-OCR model over the image bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For fine-tuning a VLM on chart QA, project the chart bytes plus the question and answer; columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/chartqa-lance/data")
tbl = db.open_table("test")
train_ds = Permutation.identity(tbl).select_columns(["image", "question", "answer"])
loader = DataLoader(train_ds, batch_size=32, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; decode the JPEG bytes,
# tokenize the question/answer pair, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "question_emb", "answer"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a lightweight answer-classifier or a linear probe on top of frozen features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges predictions, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/chartqa-lance/data")
tbl = db.open_table("test")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./chartqa-lance/data")
local_tbl = local_db.open_table("test")
local_tbl.tags.create("eval-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("test", version="eval-v1")
tbl_v5 = db.open_table("test", version=5)
```
Pinning supports two workflows. An evaluation harness locked to `eval-v1` keeps producing comparable scores while the dataset evolves in parallel — newly added prediction columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same charts and questions, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/chartqa-lance/data")
remote_tbl = remote_db.open_table("test")
batches = (
remote_tbl.search("percentage OR percent")
.where("type = 'human'")
.select(["id", "image", "question", "answer", "type", "image_emb", "question_emb"])
.to_batches()
)
local_db = lancedb.connect("./chartqa-human-subset")
local_db.create_table("test", batches)
```
The resulting `./chartqa-human-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/chartqa-lance/data` for `./chartqa-human-subset`.
## Source & license
Converted from [`lmms-lab/ChartQA`](https://huggingface.co/datasets/lmms-lab/ChartQA). The original ChartQA dataset is released under the GNU GPL-3.0 license by Masry et al.
## Citation
```
@inproceedings{masry2022chartqa,
title={ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning},
author={Masry, Ahmed and Long, Do Xuan and Tan, Jia Qing and Joty, Shafiq and Hoque, Enamul},
booktitle={Findings of the Association for Computational Linguistics: ACL 2022},
year={2022}
}
```
# CIFAR-10
Source: https://docs.lancedb.com/datasets/cifar10
A Lance-formatted version of CIFAR-10 covering 60,000 32×32 RGB images across ten balanced object classes. Each row carries inline PNG bytes, the integer label, the human-readable class name, and a cosine-normalized CLIP image embedding, all backed…
Source dataset card and downloadable files for `lance-format/cifar10-lance`.
A Lance-formatted version of [CIFAR-10](https://huggingface.co/datasets/uoft-cs/cifar10) covering 60,000 32×32 RGB images across ten balanced object classes. Each row carries inline PNG bytes, the integer label, the human-readable class name, and a cosine-normalized CLIP image embedding, all backed by a bundled `IVF_PQ` vector index plus scalar indices on the label columns and available directly from the Hub at `hf://datasets/lance-format/cifar10-lance/data`.
## Key features
* **Inline PNG bytes** in the `image` column — no sidecar files, no image folders.
* **Pre-computed CLIP image embeddings** (OpenCLIP `ViT-B-32` / `laion2b_s34b_b79k`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index.
* **Scalar indices on both label columns** — `BTREE` on `label` and `BITMAP` on `label_name` — so class filters and class-conditioned search are constant-time lookups.
* **One columnar dataset** — scan labels cheaply, then fetch image bytes only for the rows you want.
## Splits
| Split | Rows |
| ------------- | ------ |
| `train.lance` | 50,000 |
| `test.lance` | 10,000 |
## Schema
| Column | Type | Notes |
| ------------ | ------------------------------- | ----------------------------------------------------------------------------------------------- |
| `id` | `int64` | Row index within the split (natural join key for merges) |
| `image` | `large_binary` | Inline PNG bytes (32×32 RGB) |
| `label` | `int32` | Class id (0–9) |
| `label_name` | `string` | One of `airplane`, `automobile`, `bird`, `cat`, `deer`, `dog`, `frog`, `horse`, `ship`, `truck` |
| `image_emb` | `fixed_size_list` | CLIP image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `BTREE` on `label` — fast equality and range filters on the class id
* `BITMAP` on `label_name` — fast filters across the ten class names
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable if your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/cifar10-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["label"], row["label_name"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/cifar10-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/cifar10-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/cifar10-lance --repo-type dataset --local-dir ./cifar10-lance
> ```
>
> Then point Lance or LanceDB at `./cifar10-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` turns nearest-neighbor lookup on the 512-d CLIP space into a single call. In production you would encode a query image (or, for cross-modal text→image lookup, a tokenized prompt) through OpenCLIP `ViT-B-32` at runtime and pass the resulting vector to `tbl.search(...)`. The example below uses the embedding already stored in row 42 as a runnable stand-in so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/cifar10-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb", "label_name"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "label", "label_name"])
.limit(10)
.to_list()
)
print("query class:", seed["label_name"])
for r in hits:
print(f" id={r['id']:>5} {r['label_name']}")
```
Because CIFAR-10 has only ten classes and the embeddings are cosine-normalized, near-neighbors of a seed image cluster tightly inside the seed's own class. Tune `metric`, `nprobes`, and `refine_factor` to trade recall against latency.
## Curate
A typical curation pass for a classification workflow narrows the table to a single class (or a small set of confusable classes) before sampling. Because both label columns are indexed, the filter resolves without scanning the embedding or image bytes; the bounded `.limit(500)` keeps the output small enough to inspect or hand off as a manifest of row ids.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/cifar10-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where("label_name IN ('cat', 'dog')", prefilter=True)
.select(["id", "label", "label_name"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} cat/dog candidates")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `id`s, or feed into the Evolve and Train workflows below. The `image` and `image_emb` columns are never read, so the network traffic for a 500-row candidate scan is dominated by the tiny label payload.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `is_animal` flag and an `is_target_class` flag, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./cifar10-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"is_animal": "label_name IN ('bird', 'cat', 'deer', 'dog', 'frog', 'horse')",
"is_target_class": "label = 3",
})
```
If the values you want to attach already live in another table (offline labels from a stronger model, classifier predictions, per-row confidence scores), merge them in by joining on the `id` column:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"id": pa.array([0, 1, 2], type=pa.int64()),
"pred_label": pa.array([3, 8, 0], type=pa.int32()),
"pred_conf": pa.array([0.91, 0.74, 0.99]),
})
tbl.merge(predictions, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second image encoder over the inline PNG bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetch, shuffling, and batching behave as in any PyTorch pipeline. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/cifar10-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "label"])
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb stays on disk.
# decode the PNG bytes, apply augmentations, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "label"]` to `select_columns(...)` on the next run skips PNG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a linear probe or a lightweight reranker on top of frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/cifar10-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./cifar10-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel; newly added prediction columns or relabelings do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and labels, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/cifar10-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("label_name IN ('cat', 'dog')")
.select(["id", "image", "label", "label_name", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./cifar10-cats-dogs")
local_db.create_table("train", batches)
```
The resulting `./cifar10-cats-dogs` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/cifar10-lance/data` for `./cifar10-cats-dogs`.
## Source & license
Converted from [`uoft-cs/cifar10`](https://huggingface.co/datasets/uoft-cs/cifar10). CIFAR-10 was collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton at the University of Toronto.
## Citation
```
@techreport{krizhevsky2009cifar10,
title={Learning multiple layers of features from tiny images},
author={Krizhevsky, Alex and Hinton, Geoffrey},
year={2009},
institution={University of Toronto}
}
```
# COCO Captions 2017
Source: https://docs.lancedb.com/datasets/coco-captions-2017
A Lance-formatted version of the COCO Captions 2017 corpus, redistributed via lmms-lab/COCO-Caption2017. Each row is one image with 5–7 human-written captions, a cosine-normalized CLIP image embedding, and a cosine-normalized CLIP text embedding of…
Source dataset card and downloadable files for `lance-format/coco-captions-2017-lance`.
A Lance-formatted version of the [COCO Captions 2017](https://cocodataset.org/) corpus, redistributed via [`lmms-lab/COCO-Caption2017`](https://huggingface.co/datasets/lmms-lab/COCO-Caption2017). Each row is one image with **5–7 human-written captions**, a cosine-normalized CLIP image embedding, and a cosine-normalized CLIP text embedding of the canonical caption — all stored inline and available directly from the Hub at `hf://datasets/lance-format/coco-captions-2017-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files, no image folders.
* **Paired CLIP embeddings in the same row** — `image_emb` and `text_emb` (ViT-B/32, 512-dim, cosine-normalized) — so cross-modal retrieval is one indexed lookup.
* **All 5–7 raw captions kept in `captions`** alongside a `caption` canonical string used for full-text search.
* **Pre-built ANN, FTS, and scalar indices** covering both embedding columns, the canonical caption, and `image_id`.
## Splits
| Split | Rows | Notes |
| ------------ | ------ | -------------------------------------------------- |
| `val.lance` | 5,000 | Canonical COCO 2017 val set |
| `test.lance` | 40,700 | Public test slice from `lmms-lab/COCO-Caption2017` |
> The 2017 train split (118 k images, \~18 GB of source JPEGs) is intentionally not bundled here because the `lmms-lab/COCO-Caption2017` redistribution does not include it. To extend with train, run `coco_captions_2017/dataprep.py` against your local COCO 2017 train mirror.
## Schema
| Column | Type | Notes |
| ----------- | ------------------------------- | -------------------------------------------- |
| `id` | `int64` | Row index within split (natural join key) |
| `image` | `large_binary` | Inline JPEG bytes |
| `image_id` | `string` | COCO image id |
| `filename` | `string` | Original filename (e.g. `000000179765.jpg`) |
| `captions` | `list` | All 5–7 captions for the image |
| `caption` | `string` | First caption — canonical text used for FTS |
| `image_emb` | `fixed_size_list` | CLIP image embedding (cosine-normalized) |
| `text_emb` | `fixed_size_list` | CLIP text embedding of the canonical caption |
## Pre-built indices
* `IVF_PQ` on `image_emb` — image-side vector search (cosine)
* `IVF_PQ` on `text_emb` — text-side vector search (cosine)
* `INVERTED` (FTS) on `caption` — keyword and hybrid search
* `BTREE` on `image_id` — fast lookup by COCO image id
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/coco-captions-2017-lance", split="val", streaming=True)
for row in hf_ds.take(3):
print(row["caption"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/coco-captions-2017-lance/data")
tbl = db.open_table("val")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/coco-captions-2017-lance/data/val.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/coco-captions-2017-lance --repo-type dataset --local-dir ./coco-captions-2017-lance
> ```
>
> Then point Lance or LanceDB at `./coco-captions-2017-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes cross-modal text→image retrieval a single call: encode a text query with the same CLIP model used at ingest (ViT-B/32, cosine-normalized), then pass the resulting 512-d vector to `tbl.search(...)` and target `image_emb`. The example below uses the `text_emb` already stored in row 42 as a runnable stand-in for "the CLIP encoding of a caption", so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/coco-captions-2017-lance/data")
tbl = db.open_table("val")
seed = (
tbl.search()
.select(["text_emb", "caption"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["text_emb"], vector_column_name="image_emb")
.metric("cosine")
.select(["image_id", "caption"])
.limit(10)
.to_list()
)
print("query caption:", seed["caption"])
for r in hits:
print(f" {r['image_id']:>12} {r['caption'][:70]}")
```
Because OpenAI-style CLIP embeddings are normalized, cosine is the right metric and the first hit will typically be the source image itself — a useful sanity check. Swap `vector_column_name="image_emb"` for `text_emb` to do text→text retrieval against the canonical captions instead.
Because the dataset also ships an `INVERTED` index on `caption`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges the two result lists and reranks them in a single call, which is useful when a phrase like "yellow taxi" must literally appear in the caption but you still want CLIP to do the heavy lifting on visual similarity.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid", vector_column_name="image_emb")
.vector(seed["text_emb"])
.text("a man riding a surfboard")
.select(["image_id", "caption"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" {r['image_id']:>12} {r['caption'][:70]}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
A typical curation pass for a captioning or contrastive-training workflow combines a content filter on the captions with a structural filter on the image. Stacking both inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(500)` makes it cheap to inspect before committing the subset to anything downstream.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/coco-captions-2017-lance/data")
tbl = db.open_table("val")
candidates = (
tbl.search("surfer OR surfboard OR wave")
.where("array_length(captions) >= 5", prefilter=True)
.select(["image_id", "caption", "captions"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates; first caption: {candidates[0]['caption'][:80]}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `image_id`s, or feed into the Evolve and Train workflows below. The `image` column is never read, so the network traffic for a 500-row candidate scan is dominated by caption text rather than JPEG bytes.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds `num_captions` and a `long_caption` flag, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./coco-captions-2017-lance/data") # local copy required for writes
tbl = db.open_table("val")
tbl.add_columns({
"num_captions": "array_length(captions)",
"long_caption": "length(caption) >= 80",
})
```
If the values you want to attach already live in another table (offline labels, classifier predictions, a second-pass caption from a different model), merge them in by joining on `image_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
labels = pa.table({
"image_id": pa.array(["179765", "000139"]),
"scene_label": pa.array(["beach", "kitchen"]),
})
tbl.merge(labels, on="image_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second CLIP variant over the image bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a CLIP-style contrastive run, project the JPEG bytes and a sampled caption; for a reranker or probe on top of frozen features, project the precomputed embeddings instead.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/coco-captions-2017-lance/data")
tbl = db.open_table("val")
train_ds = Permutation.identity(tbl).select_columns(["image", "caption"])
loader = DataLoader(train_ds, batch_size=128, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; decode the JPEG bytes,
# tokenize the captions, encode, contrastive loss...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "text_emb"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a lightweight reranker or a linear probe.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/coco-captions-2017-lance/data")
tbl = db.open_table("val")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./coco-captions-2017-lance/data")
local_tbl = local_db.open_table("val")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("val", version="clip-vitb32-v1")
tbl_v5 = db.open_table("val", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel — newly added embeddings or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and captions, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/coco-captions-2017-lance/data")
remote_tbl = remote_db.open_table("test")
batches = (
remote_tbl.search("surfer OR surfboard OR wave")
.where("array_length(captions) >= 5")
.select(["image_id", "image", "caption", "captions", "image_emb", "text_emb"])
.to_batches()
)
local_db = lancedb.connect("./coco-surf-subset")
local_db.create_table("train", batches)
```
The resulting `./coco-surf-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/coco-captions-2017-lance/data` for `./coco-surf-subset`.
## Source & license
Converted from [`lmms-lab/COCO-Caption2017`](https://huggingface.co/datasets/lmms-lab/COCO-Caption2017). Original COCO 2017 annotations are released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); the underlying images are subject to Flickr terms of service. Please review the [COCO Terms of Use](https://cocodataset.org/#termsofuse) before redistribution.
## Citation
```
@inproceedings{lin2014microsoft,
title={Microsoft COCO: Common objects in context},
author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
booktitle={European Conference on Computer Vision (ECCV)},
year={2014}
}
```
# COCO 2017 Detection
Source: https://docs.lancedb.com/datasets/coco-detection-2017
A Lance-formatted version of the COCO 2017 object detection benchmark, sourced from detection-datasets/coco. Each row is one image with its inline JPEG bytes, the full per-image list of bounding boxes, COCO 80-class category ids and names…
Source dataset card and downloadable files for `lance-format/coco-detection-2017-lance`.
A Lance-formatted version of the [COCO 2017 object detection benchmark](https://cocodataset.org/), sourced from [`detection-datasets/coco`](https://huggingface.co/datasets/detection-datasets/coco). Each row is one image with its inline JPEG bytes, the full per-image list of bounding boxes, COCO 80-class category ids and names, per-object areas, an OpenCLIP image embedding, and pre-built indices — all available directly from the Hub at `hf://datasets/lance-format/coco-detection-2017-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files, no image folders.
* **Per-object annotations as parallel list columns** — `bboxes`, `categories`, `category_names`, and `areas` are aligned position-for-position, so iterating boxes alongside their labels is a single row read.
* **Pre-aggregated annotation summaries** — `num_objects` (int) and `categories_present` (deduped string list) precompute the predicates curation queries hit most.
* **CLIP image embeddings** (`image_emb`, OpenCLIP ViT-B/32, 512-d, cosine-normalized) with a bundled `IVF_PQ` index for visual retrieval.
## Splits
| Split | Rows |
| ------------- | -------- |
| `train.lance` | 117,000+ |
| `val.lance` | 4,950+ |
Total annotated boxes: \~860k train / \~37k val.
## Schema
| Column | Type | Notes |
| -------------------- | ------------------------------- | ------------------------------------------------------------------------ |
| `id` | `int64` | Row index within split |
| `image` | `large_binary` | Inline JPEG bytes |
| `image_id` | `int64` | COCO image id (natural join key) |
| `width`, `height` | `int32` | Image dimensions in pixels |
| `bboxes` | `list>` | Each box is `[x_min, y_min, x_max, y_max]` in absolute pixel coordinates |
| `categories` | `list` | COCO 80-class id (0–79), aligned with `bboxes` |
| `category_names` | `list` | Human-readable class name per object (e.g. `person`, `dog`) |
| `areas` | `list` | Bounding-box area in pixels², aligned with `bboxes` |
| `num_objects` | `int32` | Number of annotated objects in the image |
| `categories_present` | `list` | Deduped class names — feeds the `LABEL_LIST` index |
| `image_emb` | `fixed_size_list` | OpenCLIP ViT-B/32 image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `BTREE` on `image_id` — fast lookup by COCO image id
* `BTREE` on `num_objects` — range filters on image complexity
* `LABEL_LIST` on `categories_present` — supports `array_has_any` / `array_has_all` for class-presence filtering
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/coco-detection-2017-lance", split="val", streaming=True)
for row in hf_ds.take(3):
print(row["image_id"], row["num_objects"], row["categories_present"][:5])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/coco-detection-2017-lance/data")
tbl = db.open_table("val")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/coco-detection-2017-lance/data/val.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access, ANN search, and any mutation are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/coco-detection-2017-lance --repo-type dataset --local-dir ./coco-detection-2017-lance
> ```
>
> Then point Lance or LanceDB at `./coco-detection-2017-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes approximate-nearest-neighbor visual retrieval a single call. In production you would encode a query image through the same OpenCLIP ViT-B/32 model used at ingest and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the embedding stored on row 42 as a runnable stand-in, so the snippet works without loading any model.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/coco-detection-2017-lance/data")
tbl = db.open_table("val")
seed = (
tbl.search()
.select(["image_emb", "image_id", "categories_present"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["image_id", "categories_present", "num_objects"])
.limit(10)
.to_list()
)
print("query categories:", seed["categories_present"])
for r in hits:
print(f" image_id={r['image_id']:>10} n={r['num_objects']:>3} cats={r['categories_present'][:5]}")
```
Because the embeddings are cosine-normalized, the first hit will typically be the source image itself — a useful sanity check. Tune `nprobes` and `refine_factor` to trade recall against latency for your workload.
## Curate
Curation for a detection workflow usually means picking images that contain a specific class combination, possibly bounded by scene complexity. The `LABEL_LIST` index on `categories_present` makes class-presence predicates trivial, and Lance evaluates them inside the same scan as range filters on `num_objects` or `width`/`height`. The bounded `.limit(500)` keeps the result small and inspectable, and the `image` column is left out of the projection so the candidate scan is dominated by annotation metadata, not JPEG bytes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/coco-detection-2017-lance/data")
tbl = db.open_table("val")
candidates = (
tbl.search()
.where(
"array_has_all(categories_present, ['person', 'frisbee']) "
"AND num_objects BETWEEN 3 AND 12",
prefilter=True,
)
.select(["image_id", "categories_present", "num_objects", "width", "height"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates; first image_id: {candidates[0]['image_id']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `image_id`s, or feed into the Evolve and Train workflows below. Swapping `array_has_all` for `array_has_any` widens recall to images containing any of the listed classes; replacing the structural predicate with `num_objects >= 10` selects busy scenes for crowd-detection ablations.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `has_person` flag, an `aspect_ratio`, and a `max_box_area` that surfaces the largest annotated object area per image — all of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./coco-detection-2017-lance/data") # local copy required for writes
tbl = db.open_table("val")
tbl.add_columns({
"has_person": "array_has_any(categories_present, ['person'])",
"aspect_ratio": "CAST(width AS DOUBLE) / CAST(height AS DOUBLE)",
"max_box_area": "array_max(areas)",
"crowded": "num_objects >= 10",
})
```
If the values you want to attach already live in another table (offline predictions from a baseline detector, per-image difficulty scores, or a second-pass embedding), merge them in by joining on `image_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"image_id": pa.array([397133, 37777, 252219], type=pa.int64()),
"baseline_map": pa.array([0.31, 0.48, 0.22]),
})
tbl.merge(predictions, on="image_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second detector over the image bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a detector training run, project the JPEG bytes alongside the parallel annotation columns the loss consumes — boxes, category ids, and (optionally) areas. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/coco-detection-2017-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(
["image", "bboxes", "categories", "areas"]
)
loader = DataLoader(train_ds, batch_size=16, shuffle=True, num_workers=4,
collate_fn=lambda b: b) # detection targets are ragged
for batch in loader:
# batch is a list of dicts: decode each JPEG, stack the bboxes / categories
# into the target dictionary your detector expects, forward, loss...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "categories_present"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors plus the deduped class list, which is the right shape for training a lightweight multi-label classifier or a class-presence probe on top of frozen features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/coco-detection-2017-lance/data")
tbl = db.open_table("val")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./coco-detection-2017-lance/data")
local_tbl = local_db.open_table("val")
local_tbl.tags.create("detector-baseline-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("val", version="detector-baseline-v1")
tbl_v5 = db.open_table("val", version=5)
```
Pinning supports two workflows. An evaluation harness locked to `detector-baseline-v1` keeps scoring against the exact same boxes and category ids while the dataset evolves in parallel; newly merged predictions or evolved columns do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and annotations, so changes in mAP reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/coco-detection-2017-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("array_has_any(categories_present, ['dog', 'cat']) AND num_objects >= 2")
.select(["image_id", "image", "bboxes", "categories", "category_names",
"areas", "num_objects", "categories_present", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./coco-pets-subset")
local_db.create_table("train", batches)
```
The resulting `./coco-pets-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/coco-detection-2017-lance/data` for `./coco-pets-subset`.
## Source & license
Converted from [`detection-datasets/coco`](https://huggingface.co/datasets/detection-datasets/coco). COCO annotations are released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); the underlying images are subject to Flickr terms of service. See the [COCO Terms of Use](https://cocodataset.org/#termsofuse) before redistribution.
## Citation
```
@inproceedings{lin2014microsoft,
title={Microsoft COCO: Common objects in context},
author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
booktitle={European Conference on Computer Vision (ECCV)},
year={2014}
}
```
# DocVQA
Source: https://docs.lancedb.com/datasets/docvqa
A Lance-formatted version of DocVQA, a benchmark for visual question answering over document images such as industry and government scans, multi-page reports, forms, and receipts, redistributed via lmms-lab/DocVQA (DocVQA config). Each row carries…
Source dataset card and downloadable files for `lance-format/docvqa-lance`.
A Lance-formatted version of [DocVQA](https://www.docvqa.org/), a benchmark for visual question answering over document images such as industry and government scans, multi-page reports, forms, and receipts, redistributed via [`lmms-lab/DocVQA`](https://huggingface.co/datasets/lmms-lab/DocVQA) (`DocVQA` config). Each row carries the page image as inline JPEG bytes, the question and reference answer span(s), the original DocVQA question-type tags, UCSF Industry Documents Library provenance, and paired CLIP embeddings for the image and the question — all available directly from the Hub at `hf://datasets/lance-format/docvqa-lance/data`.
## Key features
* **Inline page image bytes** in the `image` column — no sidecar files, no document folders.
* **Paired CLIP embeddings in the same row** — `image_emb` and `question_emb` (ViT-B/32, 512-dim, cosine-normalized) — so visual and textual retrieval are one indexed lookup.
* **All reference answer spans preserved in `answers`** alongside a canonical `answer` string used for full-text search.
* **Pre-built ANN, FTS, scalar, and label-list indices** covering both embedding columns, the question and answer text, the document ids, and the `question_types` tag list.
## Splits
| Split | Rows | Notes |
| ------------------ | ----- | ---------------------------------------- |
| `validation.lance` | 5,349 | Canonical DocVQA validation set |
| `test.lance` | 5,188 | Public test slice from `lmms-lab/DocVQA` |
## Schema
| Column | Type | Notes |
| ----------------------- | ------------------------------- | -------------------------------------------------------- |
| `id` | `int64` | Row index within split (natural join key) |
| `image` | `large_binary` | Inline JPEG bytes (page image) |
| `image_id` | `string?` | DocVQA `docId` (alias of `doc_id`) |
| `question_id` | `string?` | DocVQA `questionId` |
| `question` | `string` | Natural-language question |
| `answers` | `list` | Reference answer span(s) |
| `answer` | `string` | First reference answer — canonical, used for FTS |
| `doc_id` | `string?` | DocVQA document id |
| `ucsf_document_id` | `string?` | UCSF Industry Documents Library id |
| `ucsf_document_page_no` | `string?` | Page number within the source document |
| `data_split` | `string?` | Original split label from the source |
| `question_types` | `list` | DocVQA question-type tags (`form`, `figure`, `table`, …) |
| `image_emb` | `fixed_size_list` | CLIP image embedding (cosine-normalized) |
| `question_emb` | `fixed_size_list` | CLIP text embedding of the question |
## Pre-built indices
* `IVF_PQ` on `image_emb` — image-side vector search (cosine)
* `IVF_PQ` on `question_emb` — text-side vector search (cosine)
* `INVERTED` (FTS) on `question` and `answer` — keyword and hybrid search
* `BTREE` on `image_id`, `question_id`, `doc_id` — fast lookup by document or question id
* `LABEL_LIST` on `question_types` — set-membership filtering over question-type tags
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/docvqa-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["answer"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/docvqa-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/docvqa-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/docvqa-lance --repo-type dataset --local-dir ./docvqa-lance
> ```
>
> Then point Lance or LanceDB at `./docvqa-lance/data`.
## Search
The bundled `IVF_PQ` index on `question_emb` makes question-to-question retrieval a single call: encode a query with the same CLIP model used at ingest (ViT-B/32, cosine-normalized) and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the `question_emb` already stored in row 42 as a runnable stand-in, so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/docvqa-lance/data")
tbl = db.open_table("validation")
seed = (
tbl.search()
.select(["question_emb", "question"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="question_emb")
.metric("cosine")
.select(["question_id", "question", "answer", "question_types"])
.limit(10)
.to_list()
)
print("query:", seed["question"])
for r in hits:
print(f" {r['question_id']:>8} {r['question'][:60]} -> {r['answer']}")
```
Swap `vector_column_name="question_emb"` for `image_emb` to retrieve pages whose visual layout is similar to a given embedding — useful when you want to find other forms or invoices that look like a seed page.
Because the dataset also ships an `INVERTED` index on `question` and `answer`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges the two result lists and reranks them in a single call, which is useful when a phrase like "invoice total" or "date of birth" must literally appear in the question but you still want CLIP to do the heavy lifting on semantic similarity.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid", vector_column_name="question_emb")
.vector(seed["question_emb"])
.text("invoice total")
.select(["question_id", "question", "answer"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" {r['question_id']:>8} {r['question'][:60]} -> {r['answer']}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
A typical curation pass for a document-VQA workflow combines a content filter on the question with a structural filter on the question-type tags. Stacking both inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(500)` makes it cheap to inspect before committing the subset to anything downstream. The example below collects form-style questions that mention a date, which is a common slice for evaluating form-understanding behaviour.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/docvqa-lance/data")
tbl = db.open_table("validation")
candidates = (
tbl.search("date")
.where("array_has_any(question_types, ['form'])", prefilter=True)
.select(["question_id", "doc_id", "question", "answer", "question_types"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates; first: {candidates[0]['question'][:80]}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `question_id`s, or feed into the Evolve and Train workflows below. The `image` column is never read, so the network traffic for a 500-row candidate scan is dominated by question and answer text rather than page JPEGs.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds `answer_length`, an `is_form_question` flag, and a `has_table` flag, any of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./docvqa-lance/data") # local copy required for writes
tbl = db.open_table("validation")
tbl.add_columns({
"answer_length": "length(answer)",
"is_form_question": "array_has_any(question_types, ['form'])",
"has_table": "array_has_any(question_types, ['table/list'])",
})
```
If the values you want to attach already live in another table (OCR-extracted page text, model predictions, layout-detector outputs), merge them in by joining on `question_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"question_id": pa.array(["49153", "49154", "49155"]),
"pred_answer": pa.array(["$1,234.56", "John Doe", "2018-04-12"]),
"is_correct": pa.array([True, True, False]),
})
tbl.merge(predictions, on="question_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running OCR or a layout model over the page bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For fine-tuning a document-VLM, project the page bytes plus the question and answer; columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/docvqa-lance/data")
tbl = db.open_table("validation")
train_ds = Permutation.identity(tbl).select_columns(["image", "question", "answer"])
loader = DataLoader(train_ds, batch_size=16, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; decode the JPEG bytes,
# tokenize the question/answer pair, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "question_emb", "answer"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a lightweight answer-classifier or a linear probe on top of frozen features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges predictions, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/docvqa-lance/data")
tbl = db.open_table("validation")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./docvqa-lance/data")
local_tbl = local_db.open_table("validation")
local_tbl.tags.create("eval-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("validation", version="eval-v1")
tbl_v5 = db.open_table("validation", version=5)
```
Pinning supports two workflows. An evaluation harness locked to `eval-v1` keeps producing comparable scores while the dataset evolves in parallel — newly added prediction columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same pages and questions, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/docvqa-lance/data")
remote_tbl = remote_db.open_table("validation")
batches = (
remote_tbl.search("date")
.where("array_has_any(question_types, ['form'])")
.select(["id", "image", "question_id", "doc_id", "question", "answer",
"question_types", "image_emb", "question_emb"])
.to_batches()
)
local_db = lancedb.connect("./docvqa-forms-subset")
local_db.create_table("validation", batches)
```
The resulting `./docvqa-forms-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/docvqa-lance/data` for `./docvqa-forms-subset`.
## Source & license
Converted from [`lmms-lab/DocVQA`](https://huggingface.co/datasets/lmms-lab/DocVQA). DocVQA is released under the MIT license; the underlying documents come from the [UCSF Industry Documents Library](https://www.industrydocuments.ucsf.edu/) — review their access conditions before redistribution.
## Citation
```
@inproceedings{mathew2021docvqa,
title={DocVQA: A Dataset for VQA on Document Images},
author={Mathew, Minesh and Karatzas, Dimosthenis and Jawahar, CV},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year={2021}
}
```
# EuroSAT
Source: https://docs.lancedb.com/datasets/eurosat
A Lance-formatted version of EuroSAT, the canonical Sentinel-2 RGB land-cover benchmark, sourced from blanchon/EuroSAT_RGB. Each row is a single 64×64 RGB tile with its integer class id, the human-readable class name, and a cosine-normalized…
Source dataset card and downloadable files for `lance-format/eurosat-lance`.
A Lance-formatted version of [EuroSAT](https://github.com/phelber/eurosat), the canonical Sentinel-2 RGB land-cover benchmark, sourced from [`blanchon/EuroSAT_RGB`](https://huggingface.co/datasets/blanchon/EuroSAT_RGB). Each row is a single 64×64 RGB tile with its integer class id, the human-readable class name, and a cosine-normalized OpenCLIP image embedding — all stored inline and available directly from the Hub at `hf://datasets/lance-format/eurosat-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar TIF folders, no per-class subdirectories.
* **Pre-computed OpenCLIP image embeddings** (`image_emb`, ViT-B/32, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index for similarity search.
* **Both label representations** — integer `label` (0-9) and string `label_name` — with scalar indices on both for fast class filters.
* **One columnar dataset** — scan labels and embeddings cheaply, fetch tile bytes only for the rows you actually need.
## Splits
| Split | Rows | Notes |
| ------------------ | ------ | ------------------- |
| `train.lance` | 16,200 | Training split |
| `validation.lance` | 5,400 | Validation split |
| `test.lance` | 5,400 | Held-out test split |
## Schema
| Column | Type | Notes |
| ------------ | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | `int64` | Row index within the split (natural join key) |
| `image` | `large_binary` | Inline JPEG bytes (64×64 RGB Sentinel-2 tile) |
| `label` | `int32` | Class id (0-9) |
| `label_name` | `string` | One of `Annual_Crop`, `Forest`, `Herbaceous_Vegetation`, `Highway`, `Industrial_Buildings`, `Pasture`, `Permanent_Crop`, `Residential_Buildings`, `River`, `SeaLake` |
| `image_emb` | `fixed_size_list` | OpenCLIP `ViT-B-32` image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `BTREE` on `label` — fast equality / range filters by class id
* `BITMAP` on `label_name` — fast set-membership filters by class name
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/eurosat-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["label_name"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/eurosat-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/eurosat-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/eurosat-lance --repo-type dataset --local-dir ./eurosat-lance
> ```
>
> Then point Lance or LanceDB at `./eurosat-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes visually-similar-tile retrieval a single call. In production you would encode a query tile through the same OpenCLIP `ViT-B-32` model used at ingest (cosine-normalized) and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the embedding already stored in row 42 as a runnable stand-in, so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/eurosat-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb", "label_name"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "label_name"])
.limit(10)
.to_list()
)
print(f"reference tile class: {seed['label_name']}")
for r in hits:
print(f" id={r['id']:>6} {r['label_name']}")
```
Because the embeddings are cosine-normalized at ingest, `metric="cosine")` is the right choice and the first hit will typically be the seed tile itself — a useful sanity check. Tune `nprobes` and `refine_factor` to trade recall against latency for your workload.
## Curate
A typical curation pass for a land-cover classification or retrieval study narrows the dataset to a single class and then retrieves the visually closest tiles to a seed. Lance evaluates the vector search and the metadata filter inside a single query, so the candidate set comes back already filtered. The example below pulls the 500 forest tiles most similar to a chosen seed; the bounded `.limit(500)` keeps the output small enough to inspect or hand off.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/eurosat-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb"])
.limit(1)
.offset(0)
.to_list()[0]
)
candidates = (
tbl.search(seed["image_emb"])
.where("label_name = 'Forest'", prefilter=True)
.select(["id", "label", "label_name"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} Forest candidates")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of row ids, or feed into the Evolve and Train workflows below. The `image` column is never read, so the network traffic for a 500-row candidate scan is dominated by the small metadata payload rather than JPEG bytes.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a coarse `is_urban` flag that captures whether a tile belongs to one of the built-environment classes, useful as a direct predicate in later `where` clauses without re-evaluating the class set on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./eurosat-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"is_urban": "label_name IN ('Highway', 'Industrial_Buildings', 'Residential_Buildings')",
})
```
If the values you want to attach already live in another table (a coarse climate label per class, an external aesthetic score, model predictions from a separate eval), merge them in by joining on `label_name`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
climate = pa.table({
"label_name": pa.array(["Forest", "Pasture", "SeaLake", "River"]),
"climate_zone": pa.array(["temperate", "temperate", "marine", "freshwater"]),
})
tbl.merge(climate, on="label_name")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running an alternative remote-sensing model over the tile bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/eurosat-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "label"])
loader = DataLoader(train_ds, batch_size=128, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb stays on disk.
# decode the JPEG bytes, forward through a CNN or ViT, cross-entropy loss...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "label"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a linear probe or a lightweight classifier head on top of frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/eurosat-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./eurosat-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel; newly added columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same tiles, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/eurosat-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("label_name IN ('Forest', 'River', 'SeaLake')")
.select(["id", "image", "label", "label_name", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./eurosat-natural-subset")
local_db.create_table("train", batches)
```
The resulting `./eurosat-natural-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/eurosat-lance/data` for `./eurosat-natural-subset`.
## Source & license
Converted from [`blanchon/EuroSAT_RGB`](https://huggingface.co/datasets/blanchon/EuroSAT_RGB). EuroSAT is released under the MIT license by Helber et al. The underlying Sentinel-2 imagery is © European Space Agency, made available under the [Copernicus open data policy](https://www.copernicus.eu/en/access-data/copyright-and-licences).
## Citation
```
@inproceedings{helber2019eurosat,
title={EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification},
author={Helber, Patrick and Bischke, Benjamin and Dengel, Andreas and Borth, Damian},
journal={IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},
year={2019}
}
```
# Fashion-MNIST
Source: https://docs.lancedb.com/datasets/fashion-mnist
A Lance-formatted version of Fashion-MNIST covering 70,000 28×28 grayscale clothing images across ten balanced apparel classes. Each row carries inline PNG bytes, the integer label, the human-readable class name, and a cosine-normalized CLIP image…
Source dataset card and downloadable files for `lance-format/fashion-mnist-lance`.
A Lance-formatted version of [Fashion-MNIST](https://huggingface.co/datasets/zalando-datasets/fashion_mnist) covering 70,000 28×28 grayscale clothing images across ten balanced apparel classes. Each row carries inline PNG bytes, the integer label, the human-readable class name, and a cosine-normalized CLIP image embedding, all backed by a bundled `IVF_PQ` vector index plus scalar indices on the label columns and available directly from the Hub at `hf://datasets/lance-format/fashion-mnist-lance/data`.
## Key features
* **Inline PNG bytes** in the `image` column — no sidecar files, no image folders.
* **Pre-computed CLIP image embeddings** (OpenCLIP `ViT-B-32` / `laion2b_s34b_b79k`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index.
* **Scalar indices on both label columns** — `BTREE` on `label` and `BITMAP` on `label_name` — so apparel-class filters and class-conditioned search are constant-time lookups.
* **One columnar dataset** — scan labels cheaply, then fetch image bytes only for the rows you want.
## Splits
| Split | Rows |
| ------------- | ------ |
| `train.lance` | 60,000 |
| `test.lance` | 10,000 |
## Schema
| Column | Type | Notes |
| ------------ | ------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `id` | `int64` | Row index within the split (natural join key for merges) |
| `image` | `large_binary` | Inline PNG bytes (28×28 grayscale) |
| `label` | `int32` | Class id (0–9) |
| `label_name` | `string` | One of `T-shirt_top`, `Trouser`, `Pullover`, `Dress`, `Coat`, `Sandal`, `Shirt`, `Sneaker`, `Bag`, `Ankle_boot` |
| `image_emb` | `fixed_size_list` | CLIP image embedding (cosine-normalized) |
> The original Fashion-MNIST class strings `T-shirt/top` and `Ankle boot` are sanitized to `T-shirt_top` and `Ankle_boot` for use as filename-safe identifiers, so SQL filters on `label_name` should reference the underscored form.
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `BTREE` on `label` — fast equality and range filters on the class id
* `BITMAP` on `label_name` — fast filters across the ten class names
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable if your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/fashion-mnist-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["label"], row["label_name"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/fashion-mnist-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/fashion-mnist-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/fashion-mnist-lance --repo-type dataset --local-dir ./fashion-mnist-lance
> ```
>
> Then point Lance or LanceDB at `./fashion-mnist-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` turns nearest-neighbor lookup on the 512-d CLIP space into a single call. In production you would encode a query image (or, for cross-modal text→image lookup, a tokenized prompt like "a black ankle boot") through OpenCLIP `ViT-B-32` at runtime and pass the resulting vector to `tbl.search(...)`. The example below uses the embedding already stored in row 42 as a runnable stand-in so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/fashion-mnist-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb", "label_name"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "label", "label_name"])
.limit(10)
.to_list()
)
print("query class:", seed["label_name"])
for r in hits:
print(f" id={r['id']:>5} {r['label_name']}")
```
Because the embeddings are cosine-normalized and CLIP separates apparel categories cleanly, near-neighbors of a seed image are typically dominated by the seed's own class, with the most confusable garments (Shirt vs T-shirt\_top, Sneaker vs Sandal) showing up next. Tune `metric`, `nprobes`, and `refine_factor` to trade recall against latency.
## Curate
A typical curation pass for an apparel-classification workflow narrows the table to a confusable subset of classes (for example, the three upper-body garments that share silhouettes) before sampling. Because both label columns are indexed, the filter resolves without scanning the embedding or image bytes; the bounded `.limit(500)` keeps the output small enough to inspect or hand off as a manifest of row ids.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/fashion-mnist-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where("label_name IN ('Shirt', 'T-shirt_top', 'Pullover')", prefilter=True)
.select(["id", "label", "label_name"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} upper-body-garment candidates")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `id`s, or feed into the Evolve and Train workflows below. The `image` and `image_emb` columns are never read, so the network traffic for a 500-row candidate scan is dominated by the tiny label payload.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `is_footwear` flag that groups the three shoe-like classes and an `is_target_class` flag for one-vs-rest experiments, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./fashion-mnist-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"is_footwear": "label_name IN ('Sandal', 'Sneaker', 'Ankle_boot')",
"is_target_class": "label = 6",
})
```
If the values you want to attach already live in another table (offline labels from a stronger model, classifier predictions, per-row confidence scores), merge them in by joining on the `id` column:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"id": pa.array([0, 1, 2], type=pa.int64()),
"pred_label": pa.array([9, 0, 3], type=pa.int32()),
"pred_conf": pa.array([0.94, 0.81, 0.77]),
})
tbl.merge(predictions, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second image encoder over the inline PNG bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetch, shuffling, and batching behave as in any PyTorch pipeline. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/fashion-mnist-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "label"])
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb stays on disk.
# decode the PNG bytes, normalize to [0, 1], forward, backward...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "label"]` to `select_columns(...)` on the next run skips PNG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a linear probe or a lightweight reranker on top of frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/fashion-mnist-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./fashion-mnist-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel; newly added prediction columns or relabelings do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and labels, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/fashion-mnist-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("label_name IN ('Shirt', 'T-shirt_top', 'Pullover')")
.select(["id", "image", "label", "label_name", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./fashion-mnist-upper-body")
local_db.create_table("train", batches)
```
The resulting `./fashion-mnist-upper-body` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/fashion-mnist-lance/data` for `./fashion-mnist-upper-body`.
## Source & license
Converted from [`zalando-datasets/fashion_mnist`](https://huggingface.co/datasets/zalando-datasets/fashion_mnist). Released under the MIT license.
## Citation
```
@online{xiao2017fashionmnist,
title={Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms},
author={Xiao, Han and Rasul, Kashif and Vollgraf, Roland},
year={2017},
eprint={1708.07747},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
# FineWeb-Edu
Source: https://docs.lancedb.com/datasets/fineweb-edu
A Lance-formatted version of FineWeb-Edu — over 1.5 billion educational web passages with cleaned text, source metadata, language detection signals, and 384-dim text embeddings — available directly from the Hub at…
Source dataset card and downloadable files for `lance-format/fineweb-edu`.
A Lance-formatted version of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) — **over 1.5 billion educational web passages** with cleaned text, source metadata, language detection signals, and 384-dim text embeddings — available directly from the Hub at `hf://datasets/lance-format/fineweb-edu/data/train.lance`.
## Key features
* **Cleaned passage text** in the `text` column with the source `url` and `title` carried alongside.
* **Language detection signals** (`language`, `language_probability`) for filtered subsets.
* **Pre-computed 384-dim text embeddings** in `text_embedding`, ready for ANN search once an index is built locally.
* **One columnar dataset** — scan metadata cheaply, project just the columns each query needs, defer the heavy `text` and `text_embedding` reads to the rows that matter.
> **No pre-built indices on the Hub copy yet.** At 1.5 B+ rows the on-disk indices are too large to ship comfortably alongside the data on the Hub. The Search, Curate, Evolve, and Train sections below describe the same APIs you'd use against a fully indexed dataset, but vector and full-text examples assume a local copy with `IVF_PQ` and `INVERTED` indices built once after download. See the Materialize-a-subset section at the end for a focused-subset workflow that makes indexing tractable.
## Splits
`train.lance`
## Schema
| Column | Type | Notes |
| ------------------------------ | ------------------------------- | ---------------------------------------------------------------------------- |
| `text` | `string` | Cleaned passage body |
| `title` | `string` | Page or article title when available |
| `url` | `string` | Canonical source URL |
| `language` | `string` | Detected language code (e.g., `en`) |
| `language_probability` | `float32` | Confidence of the language detector |
| `text_embedding` | `fixed_size_list` | Passage embedding for retrieval |
| *FineWeb-Edu quality metadata* | — | Heuristic scores and length statistics carried over from the upstream corpus |
## Pre-built indices
None bundled at present. Build the recommended indices on a local copy:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./fineweb-edu/data")
tbl = db.open_table("train")
tbl.create_index(
metric="cosine",
vector_column_name="text_embedding",
index_type="IVF_PQ",
num_partitions=2048,
num_sub_vectors=96,
)
tbl.create_fts_index("text", replace=True)
```
Both indices live next to the data, so subsequent queries against the same local path pick them up automatically.
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/fineweb-edu", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["title"] or row["url"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/fineweb-edu/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/fineweb-edu/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but at 1.5 B+ rows random access and any kind of search are dramatically faster against a local copy, and ANN / FTS require local indices anyway:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/fineweb-edu --repo-type dataset --local-dir ./fineweb-edu
> ```
>
> Then point Lance or LanceDB at `./fineweb-edu/data`. For most workflows, the Materialize-a-subset section is a better starting point than downloading the full 1.5 B-row corpus.
## Search
Once an `IVF_PQ` index exists on `text_embedding`, dense retrieval is a single call. In production you would encode a query string through the same 384-dim text encoder used at ingest and pass the resulting vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./fineweb-edu/data") # local copy with the indices from the section above
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["text_embedding", "url"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["text_embedding"])
.metric("cosine")
.where("language = 'en' AND language_probability > 0.9", prefilter=True)
.select(["title", "url", "text"])
.limit(10)
.to_list()
)
for r in hits:
print(f"{r['url']}\n {(r['title'] or '')[:80]}")
```
The result set carries only the projected columns. The `text_embedding` vector is never read on the result side, and the `text` body is fetched only for the ten passages that actually came back, keeping the working set small even though the corpus is enormous.
Because the recommended setup also builds an `INVERTED` index on `text`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges the two result lists and reranks them in a single call, which is useful when a phrase must literally appear in the passage but the dense side still does most of the ranking.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid")
.vector(seed["text_embedding"])
.text("quantum computing")
.where("language = 'en'", prefilter=True)
.select(["title", "url", "text"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f"{r['url']}\n {(r['title'] or '')[:80]}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
A typical curation pass over a web corpus starts with a metadata filter — pick high-confidence English, drop short or low-quality fragments, restrict to a domain — before any text gets read. Stacking predicates inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(1000)` makes it cheap to inspect.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/fineweb-edu/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where(
"language = 'en' "
"AND language_probability > 0.95 "
"AND length(text) >= 1000",
prefilter=True,
)
.select(["url", "title", "language_probability"])
.limit(1000)
.to_list()
)
print(f"{len(candidates)} candidates; first url: {candidates[0]['url']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of URLs, or hand to the Materialize-a-subset section below for export to a writable local copy. Neither the `text` body nor the `text_embedding` vector is read by this scan, so a 1000-row curation pass against the Hub moves only kilobytes of metadata even though the underlying table is in the billions.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `text_length` and a `long_passage` flag, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull a larger slice first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./fineweb-edu/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"text_length": "length(text)",
"long_passage": "length(text) >= 1000",
})
```
If the values you want to attach already live in another table (offline labels, topic classifications, alternate embeddings from a stronger model), merge them in by joining on `url`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
labels = pa.table({
"url": pa.array(["https://example.com/a", "https://example.com/b"]),
"topic": pa.array(["math", "history"]),
})
tbl.merge(labels, on="url")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a different embedding model over the text), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For language-model pretraining the natural projection is just the `text` column; for a retrieval probe or a reranker on top of frozen features, project the precomputed embedding instead.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("./fineweb-edu/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["text"])
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=8)
for batch in loader:
# batch carries only the projected columns; tokenize, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["text_embedding"]` to `select_columns(...)` on the next run reads only the 384-d vectors and skips the text body entirely, which is the right shape for training a lightweight retrieval head on cached embeddings. Columns added in Evolve cost nothing per batch until they are explicitly projected.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/fineweb-edu/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./fineweb-edu/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("english-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="english-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `english-v1` keeps returning stable results while the dataset evolves in parallel — newly added embeddings or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same passages, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
At 1.5 B+ rows, very few workflows want the full corpus on local disk. The practical entry point is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory. The result is a writable LanceDB database scoped to the rows that actually matter for the downstream task, sized to index and iterate cheaply.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/fineweb-edu/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where(
"language = 'en' "
"AND language_probability > 0.95 "
"AND length(text) >= 1000"
)
.select(["url", "title", "text", "language", "language_probability", "text_embedding"])
.to_batches()
)
local_db = lancedb.connect("./fineweb-edu-en")
local_db.create_table("train", batches)
```
The resulting `./fineweb-edu-en` is a first-class LanceDB database. Build the recommended indices on it once (the same `create_index` / `create_fts_index` calls shown in the Pre-built indices section, pointed at the local path), and every snippet in the Search, Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/fineweb-edu/data` for `./fineweb-edu-en`.
## Source & license
Converted from [`HuggingFaceFW/fineweb-edu`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu). FineWeb-Edu is distributed under [ODC-BY 1.0](https://opendatacommons.org/licenses/by/1-0/); individual document content remains subject to the rights of the original publishers. Review the [upstream dataset card](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) before downstream use.
## Citation
```
@misc{lozhkov2024finewebedu,
title = {FineWeb-Edu: the Finest Collection of Educational Content the Web Has to Offer},
author = {Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas},
year = {2024},
url = {https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu}
}
```
# Flickr30k
Source: https://docs.lancedb.com/datasets/flickr30k
A Lance-formatted version of Flickr30k, redistributed via lmms-lab/flickr30k. Each row is one image with 5 human-written captions, a cosine-normalized CLIP image embedding, and a cosine-normalized CLIP text embedding of the canonical caption — all…
Source dataset card and downloadable files for `lance-format/flickr30k-lance`.
A Lance-formatted version of [Flickr30k](https://shannon.cs.illinois.edu/DenotationGraph/), redistributed via [`lmms-lab/flickr30k`](https://huggingface.co/datasets/lmms-lab/flickr30k). Each row is one image with **5 human-written captions**, a cosine-normalized CLIP image embedding, and a cosine-normalized CLIP text embedding of the canonical caption — all stored inline and available directly from the Hub at `hf://datasets/lance-format/flickr30k-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files, no image folders.
* **Paired CLIP embeddings in the same row** — `image_emb` and `text_emb` (ViT-B/32, 512-dim, cosine-normalized) — so cross-modal retrieval is one indexed lookup.
* **All 5 raw captions kept in `captions`** alongside a `caption` canonical string used for full-text search.
* **Pre-built ANN, FTS, and scalar indices** covering both embedding columns, the canonical caption, and `image_id`.
## Splits
| Split | Rows | Notes |
| ------------- | ------ | --------------------------------------------------------------------------------------------------------------------------- |
| `train.lance` | 31,783 | All Flickr30k images; the `lmms-lab/flickr30k` redistribution merges the original train/val/test labels into a single split |
## Schema
| Column | Type | Notes |
| ----------- | ------------------------------- | -------------------------------------------- |
| `id` | `int64` | Row index within split (natural join key) |
| `image` | `large_binary` | Inline JPEG bytes |
| `image_id` | `string` | Original Flickr image id |
| `filename` | `string?` | Original filename (e.g. `1000092795.jpg`) |
| `captions` | `list` | All 5 captions for the image |
| `caption` | `string` | First caption — canonical text used for FTS |
| `image_emb` | `fixed_size_list` | CLIP image embedding (cosine-normalized) |
| `text_emb` | `fixed_size_list` | CLIP text embedding of the canonical caption |
## Pre-built indices
* `IVF_PQ` on `image_emb` — image-side vector search (cosine)
* `IVF_PQ` on `text_emb` — text-side vector search (cosine)
* `INVERTED` (FTS) on `caption` — keyword and hybrid search
* `BTREE` on `image_id` — fast lookup by Flickr image id
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/flickr30k-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["caption"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/flickr30k-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/flickr30k-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/flickr30k-lance --repo-type dataset --local-dir ./flickr30k-lance
> ```
>
> Then point Lance or LanceDB at `./flickr30k-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes cross-modal text→image retrieval a single call: encode a text query with the same CLIP model used at ingest (ViT-B/32, cosine-normalized), then pass the resulting 512-d vector to `tbl.search(...)` and target `image_emb`. The example below uses the `text_emb` already stored in row 42 as a runnable stand-in for "the CLIP encoding of a caption", so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/flickr30k-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["text_emb", "caption"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["text_emb"], vector_column_name="image_emb")
.metric("cosine")
.select(["image_id", "caption"])
.limit(10)
.to_list()
)
print("query caption:", seed["caption"])
for r in hits:
print(f" {r['image_id']:>12} {r['caption'][:70]}")
```
Because OpenAI-style CLIP embeddings are normalized, cosine is the right metric and the first hit will typically be the source image itself — a useful sanity check. Swap `vector_column_name="image_emb"` for `text_emb` to do text→text retrieval against the canonical captions instead.
Because the dataset also ships an `INVERTED` index on `caption`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges the two result lists and reranks them in a single call, which is useful when a phrase like "dog playing in the snow" must literally appear in the caption but you still want CLIP to do the heavy lifting on visual similarity.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid", vector_column_name="image_emb")
.vector(seed["text_emb"])
.text("dog playing in the snow")
.select(["image_id", "caption"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" {r['image_id']:>12} {r['caption'][:70]}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
A typical curation pass for a captioning or contrastive-training workflow combines a content filter on the captions with a structural filter on the row. Stacking both inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(500)` makes it cheap to inspect before committing the subset to anything downstream.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/flickr30k-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search("surfer OR surfboard OR wave")
.where("array_length(captions) = 5", prefilter=True)
.select(["image_id", "caption", "captions"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates; first caption: {candidates[0]['caption'][:80]}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `image_id`s, or feed into the Evolve and Train workflows below. The `image` column is never read, so the network traffic for a 500-row candidate scan is dominated by caption text rather than JPEG bytes.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds `num_captions` and a `long_caption` flag, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./flickr30k-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"num_captions": "array_length(captions)",
"long_caption": "length(caption) >= 80",
})
```
If the values you want to attach already live in another table (offline labels, classifier predictions, an aesthetic or NSFW score, a second-pass caption from a different model), merge them in by joining on `image_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
labels = pa.table({
"image_id": pa.array(["1000092795", "10002456"]),
"scene_label": pa.array(["outdoor", "indoor"]),
})
tbl.merge(labels, on="image_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second CLIP variant over the image bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a CLIP-style contrastive run, project the JPEG bytes and a sampled caption; for a reranker or probe on top of frozen features, project the precomputed embeddings instead.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/flickr30k-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "caption"])
loader = DataLoader(train_ds, batch_size=128, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; decode the JPEG bytes,
# tokenize the captions, encode, contrastive loss...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "text_emb"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a lightweight reranker or a linear probe.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/flickr30k-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./flickr30k-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel — newly added embeddings or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and captions, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/flickr30k-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search("surfer OR surfboard OR wave")
.where("array_length(captions) = 5")
.select(["image_id", "image", "caption", "captions", "image_emb", "text_emb"])
.to_batches()
)
local_db = lancedb.connect("./flickr30k-surf-subset")
local_db.create_table("train", batches)
```
The resulting `./flickr30k-surf-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/flickr30k-lance/data` for `./flickr30k-surf-subset`.
## Source & license
Converted from [`lmms-lab/flickr30k`](https://huggingface.co/datasets/lmms-lab/flickr30k), which is itself a parquet redistribution of the [original Flickr30k corpus](https://shannon.cs.illinois.edu/DenotationGraph/). Original images come from Flickr; review the Flickr30k licensing terms before redistribution.
## Citation
```
@article{young2014image,
title={From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions},
author={Young, Peter and Lai, Alice and Hodosh, Micah and Hockenmaier, Julia},
journal={Transactions of the Association for Computational Linguistics},
volume={2},
pages={67--78},
year={2014}
}
```
# Food-101
Source: https://docs.lancedb.com/datasets/food101
A Lance-formatted version of Food-101, the fine-grained dish-classification benchmark of 101,000 photos spread evenly across 101 dish classes, sourced from ethz/food101. Each row carries the inline JPEG bytes, the integer label, the human-readable…
Source dataset card and downloadable files for `lance-format/food101-lance`.
A Lance-formatted version of [Food-101](https://www.kaggle.com/datasets/dansbecker/food-101), the fine-grained dish-classification benchmark of 101,000 photos spread evenly across 101 dish classes, sourced from [`ethz/food101`](https://huggingface.co/datasets/ethz/food101). Each row carries the inline JPEG bytes, the integer `label`, the human-readable `label_name`, and a cosine-normalized CLIP image embedding, all available directly from the Hub at `hf://datasets/lance-format/food101-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files, no image folders.
* **Pre-computed CLIP image embeddings** (`image_emb`, OpenCLIP `ViT-B-32`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index for similarity search.
* **Both numeric and string labels** (`label`, `label_name`) so filters can target either the class id or the dish name without an external mapping table.
* **Scalar indices on both label columns** so class-based curation is a quick predicate rather than a full scan.
## Splits
| Split | Rows | Notes |
| ------------------ | ------ | ----------------------------------------------------- |
| `train.lance` | 75,750 | Canonical Food-101 train split (750 images per class) |
| `validation.lance` | 25,250 | Canonical Food-101 test split (250 images per class) |
## Schema
| Column | Type | Notes |
| ------------ | ------------------------------- | --------------------------------------------------------------------------- |
| `id` | `int64` | Row index within split (natural join key for merges) |
| `image` | `large_binary` | Inline JPEG bytes (256x256, quality 92) |
| `label` | `int32` | Class id (0–100) |
| `label_name` | `string` | One of 101 dish names, underscore-spaced (`apple_pie`, `baby_back_ribs`, …) |
| `image_emb` | `fixed_size_list` | OpenCLIP `ViT-B-32` image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `BTREE` on `label` — fast lookup by class id
* `BITMAP` on `label_name` — fast lookup by class name
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/food101-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["label_name"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/food101-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/food101-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/food101-lance --repo-type dataset --local-dir ./food101-lance
> ```
>
> Then point Lance or LanceDB at `./food101-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes approximate-nearest-neighbor search a single call. In production you would encode a query photo through the same OpenCLIP `ViT-B-32` model used at ingest and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the embedding stored in row 0 as a runnable stand-in so the snippet works without a model loaded; the first hit is expected to be the seed image itself, which is a useful sanity check on the index.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/food101-lance/data")
tbl = db.open_table("validation")
seed = (
tbl.search()
.select(["image_emb", "label_name"])
.limit(1)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "label_name"])
.limit(10)
.to_list()
)
print("seed dish:", seed["label_name"])
for r in hits:
print(f" {r['id']:>6} {r['label_name']}")
```
Tune `metric`, `nprobes`, and `refine_factor` to trade recall against latency for your workload.
## Curate
A typical curation pass for a fine-grained classifier combines a class-based filter with the bundled vector search to assemble a small, focused candidate set. The `BITMAP` index on `label_name` makes the predicate effectively free, and the bounded `.limit(200)` keeps the result small enough to inspect or hand off to a training run.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/food101-lance/data")
tbl = db.open_table("validation")
candidates = (
tbl.search()
.where("label_name IN ('sushi', 'sashimi', 'ramen')")
.select(["id", "label", "label_name"])
.limit(200)
.to_list()
)
print(f"{len(candidates)} candidate Japanese-cuisine rows; first: {candidates[0]['label_name']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `id`s, or feed into the Evolve and Train workflows below. The `image` and `image_emb` columns are never read by this query, so the network traffic is dominated by the small label fields rather than JPEG bytes or vectors.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a coarse cuisine bucket and an `is_target_dish` flag for a focused training run, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./food101-lance/data") # local copy required for writes
tbl = db.open_table("validation")
tbl.add_columns({
"is_target_dish": "label_name IN ('sushi', 'ramen', 'pho')",
"is_dessert": "label_name IN ('apple_pie', 'cheesecake', 'tiramisu', 'ice_cream', 'donuts')",
})
```
If the values you want to attach already live in another table (offline labels, a second classifier's predictions, human-verified taste tags), merge them in by joining on `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"id": pa.array([0, 1, 2]),
"model_v2_pred": pa.array(["sushi", "sashimi", "sushi"]),
})
tbl.merge(predictions, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second embedding model over the JPEG bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetch, shuffling, and batching behave as in any PyTorch pipeline. For a from-scratch image classifier, project the JPEG bytes and the integer label; for a linear probe or reranker on top of frozen CLIP features, swap the projection to the embedding column and skip JPEG decoding entirely.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/food101-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "label"])
loader = DataLoader(train_ds, batch_size=128, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb stays on disk.
# decode the JPEG bytes, forward, cross-entropy against `label`...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "label"]` to `select_columns(...)` on the next run reads only the cached 512-d vectors and the label, which is the right shape for a linear probe.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges predictions, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/food101-lance/data")
tbl = db.open_table("validation")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./food101-lance/data")
local_tbl = local_db.open_table("validation")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("validation", version="clip-vitb32-v1")
tbl_v5 = db.open_table("validation", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel; newly added columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and labels, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/food101-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("label_name IN ('sushi', 'sashimi', 'ramen', 'pho', 'dumplings')")
.select(["id", "image", "label", "label_name", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./food101-asian-subset")
local_db.create_table("train", batches)
```
The resulting `./food101-asian-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/food101-lance/data` for `./food101-asian-subset`.
## Source & license
Converted from [`ethz/food101`](https://huggingface.co/datasets/ethz/food101). The Food-101 dataset is by Bossard et al. (ETH Zurich) — see the [original dataset page](https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/) for licensing details.
## Citation
```
@inproceedings{bossard2014food,
title={Food-101 -- Mining Discriminative Components with Random Forests},
author={Bossard, Lukas and Guillaumin, Matthieu and Van Gool, Luc},
booktitle={European Conference on Computer Vision (ECCV)},
year={2014}
}
```
# GQA testdev-balanced
Source: https://docs.lancedb.com/datasets/gqa-testdev-balanced
A Lance-formatted version of the canonical GQA testdev_balanced slice — 12,578 compositional VQA questions joined against the matching 398 images — sourced from lmms-lab/GQA. The original redistribution ships instructions and images as separate…
Source dataset card and downloadable files for `lance-format/gqa-testdev-balanced-lance`.
A Lance-formatted version of the canonical GQA `testdev_balanced` slice — 12,578 compositional VQA questions joined against the matching 398 images — sourced from [`lmms-lab/GQA`](https://huggingface.co/datasets/lmms-lab/GQA). The original redistribution ships instructions and images as separate parquet configs; here they are pre-joined on `image_id`, so each row carries the question text, the short answer, the GQA reasoning-program tags, paired CLIP image and question embeddings, and the inline JPEG bytes — all available directly from the Hub at `hf://datasets/lance-format/gqa-testdev-balanced-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column, duplicated across rows that share an `image_id` so each Q/A row is self-contained.
* **Paired CLIP embeddings in the same row** — `image_emb` and `question_emb` (512-dim, cosine-normalized) — for cross-modal retrieval as one indexed lookup.
* **Compositional reasoning metadata** — `structural`, `semantic`, and `detailed` question-type tags plus the `semantic_str` reasoning program.
* **Pre-built ANN, FTS, scalar, and bitmap indices** covering both embeddings, the question and short answer, the reasoning-type tags, and the image/question ids.
## Splits
| Split | Rows | Distinct images |
| --------------- | ------ | --------------- |
| `testdev.lance` | 12,578 | 398 |
The train\_balanced (\~943 k Q's × 72 k images) and val\_balanced splits are not bundled by default; pass `--instr-config` / `--images-config` to `gqa/dataprep.py` to extend.
## Schema
| Column | Type | Notes |
| ----------------------------- | ------------------------------- | ------------------------------------------------------------------- |
| `id` | `int64` | Row index within split |
| `image` | `large_binary` | Inline JPEG bytes (duplicated across rows that share an `image_id`) |
| `image_id` | `string` | GQA scene-graph image id |
| `question_id` | `string` | GQA question id |
| `question` | `string` | Compositional natural-language question |
| `answers` | `list` | One-element list (the GQA short answer) |
| `answer` | `string` | Canonical short answer (used for FTS) |
| `full_answer` | `string?` | Full-sentence answer |
| `structural` | `string?` | One of `verify`, `query`, `compare`, `choose`, `logical` |
| `semantic` | `string?` | One of `attr`, `cat`, `global`, `obj`, `rel` |
| `detailed` | `string?` | Fine-grained type (e.g. `weatherVerifyC`) |
| `is_balanced` | `bool` | GQA balanced subset flag |
| `group_global`, `group_local` | `string?` | GQA reasoning-group ids |
| `semantic_str` | `string?` | Compact description of the reasoning program |
| `image_emb` | `fixed_size_list` | CLIP image embedding (cosine-normalized) |
| `question_emb` | `fixed_size_list` | CLIP text embedding of the question |
## Pre-built indices
* `IVF_PQ` on `image_emb` — image-side vector search (cosine)
* `IVF_PQ` on `question_emb` — question-side vector search (cosine)
* `INVERTED` (FTS) on `question` and `answer` — keyword and hybrid search
* `BITMAP` on `structural`, `semantic`, `detailed` — fast categorical filters on the reasoning program
* `BTREE` on `image_id`, `question_id` — fast lookup by GQA id
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/gqa-testdev-balanced-lance", split="testdev", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["answer"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/gqa-testdev-balanced-lance/data")
tbl = db.open_table("testdev")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/gqa-testdev-balanced-lance/data/testdev.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/gqa-testdev-balanced-lance --repo-type dataset --local-dir ./gqa-testdev-balanced-lance
> ```
>
> Then point Lance or LanceDB at `./gqa-testdev-balanced-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes cross-modal text→image retrieval a single call: encode a question with the same CLIP model used at ingest (ViT-B/32, cosine-normalized), then pass the resulting 512-d vector to `tbl.search(...)` and target `image_emb`. The example below uses the `question_emb` already stored in row 42 as a runnable stand-in for "the CLIP encoding of a question", so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/gqa-testdev-balanced-lance/data")
tbl = db.open_table("testdev")
seed = (
tbl.search()
.select(["question_emb", "question", "answer"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="image_emb")
.metric("cosine")
.select(["image_id", "question", "answer", "structural"])
.limit(10)
.to_list()
)
print("query question:", seed["question"], "->", seed["answer"])
for r in hits:
print(f" {r['image_id']:>12} [{r['structural']}] {r['question'][:70]}")
```
Because the CLIP embeddings are cosine-normalized, cosine is the right metric and the first hit will often be the source row itself — a useful sanity check. Swap `vector_column_name="image_emb"` for `question_emb` to find paraphrased or topically related questions instead.
The dataset also ships an `INVERTED` index on `question` and `answer`, so the same query can be issued as a hybrid search that combines the dense vector with a literal keyword match. This is useful when a noun like "umbrella" must appear in the question text but you still want CLIP to handle visual similarity over the candidate set.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid", vector_column_name="image_emb")
.vector(seed["question_emb"])
.text("umbrella")
.select(["image_id", "question", "answer"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" {r['image_id']:>12} {r['question'][:70]} -> {r['answer']}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
A typical curation pass for a compositional-reasoning study combines a predicate on the question text (or the GQA short answer) with a structural filter on the reasoning program, so the candidate set is both topically and structurally consistent. Stacking both inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(500)` makes it cheap to inspect before committing the subset to anything downstream.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/gqa-testdev-balanced-lance/data")
tbl = db.open_table("testdev")
candidates = (
tbl.search()
.where(
"structural = 'verify' AND answer IN ('yes', 'no') AND question LIKE 'Is %'",
prefilter=True,
)
.select(["question_id", "image_id", "question", "answer", "semantic"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} verify-style yes/no candidates; first: {candidates[0]['question']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `question_id`s, or feed into the Evolve and Train workflows below. The `image` column is never read, so the network traffic for a 500-row candidate scan is dominated by the question and answer strings rather than JPEG bytes.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `is_binary_answer` flag and a `question_length` integer, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./gqa-testdev-balanced-lance/data") # local copy required for writes
tbl = db.open_table("testdev")
tbl.add_columns({
"is_binary_answer": "answer IN ('yes', 'no')",
"question_length": "length(question)",
"answer_length": "length(answer)",
})
```
If the values you want to attach already live in another table (offline labels, scene-graph features, or per-question predictions from an external model), merge them in by joining on `question_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"question_id": pa.array(["20240268", "20240269"]),
"model_answer": pa.array(["yes", "left"]),
"model_confidence": pa.array([0.91, 0.62]),
})
tbl.merge(predictions, on="question_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation, Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a VQA fine-tune, project the JPEG bytes, the question, and the short answer; columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/gqa-testdev-balanced-lance/data")
tbl = db.open_table("testdev")
train_ds = Permutation.identity(tbl).select_columns(["image", "question", "answer"])
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; decode the JPEG bytes,
# tokenize the question, forward through the VLM, compute the loss...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "question_emb", "answer"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for a lightweight reasoning probe over frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/gqa-testdev-balanced-lance/data")
tbl = db.open_table("testdev")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./gqa-testdev-balanced-lance/data")
local_tbl = local_db.open_table("testdev")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("testdev", version="clip-vitb32-v1")
tbl_v5 = db.open_table("testdev", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel — newly added model predictions or reasoning annotations do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and questions, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/gqa-testdev-balanced-lance/data")
remote_tbl = remote_db.open_table("testdev")
batches = (
remote_tbl.search()
.where("structural = 'verify' AND answer IN ('yes', 'no')")
.select(["question_id", "image_id", "image", "question", "answer", "image_emb", "question_emb"])
.to_batches()
)
local_db = lancedb.connect("./gqa-yesno-subset")
local_db.create_table("testdev", batches)
```
The resulting `./gqa-yesno-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/gqa-testdev-balanced-lance/data` for `./gqa-yesno-subset`.
## Source & license
Converted from [`lmms-lab/GQA`](https://huggingface.co/datasets/lmms-lab/GQA). GQA is released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) by Hudson and Manning (Stanford NLP).
## Citation
```
@inproceedings{hudson2019gqa,
title={GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering},
author={Hudson, Drew A. and Manning, Christopher D.},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2019}
}
```
# Handwriting OCR
Source: https://docs.lancedb.com/datasets/handwriting-ocr
This Lance-formatted version of the Doctor's Handwritten Prescription BD dataset contains 4,680 cropped PNG images of handwritten medicine names from Bangladesh. Each row keeps the original image bytes with the medicine and generic-name labels…
Source dataset card and downloadable files for `lance-format/handwriting-ocr`.
This Lance-formatted version of the [Doctor's Handwritten Prescription BD dataset](https://www.kaggle.com/datasets/mamun1113/doctors-handwritten-prescription-bd-dataset) contains 4,680 cropped PNG images of handwritten medicine names from Bangladesh. Each row keeps the original image bytes with the medicine and generic-name labels, plus deterministic search metadata derived from those labels. The dataset contains three source-preserved splits: train, validation, and test.
> \[!NOTE]
> **Training note:** The same samples appear repeatedly because a model (during training) should learn from multiple *different handwritten crops* of the same medicine, because they capture different writers, pen strokes, sizes, and image quality. The dataset would need to be shuffled and sampled appropriately before running downstream training tasks
## Key features
* **Original inline PNG bytes** in `image`, with no image folders or sidecar files required.
* **Three preserved splits**: 3,120 training, 780 validation, and 780 test examples.
* **Medicine annotations**: 78 medicine names mapped to 15 generic names.
* **Built-in text retrieval** over `searchable_summary`, with scalar indices matching the companion OCR retrieval project.
## Splits
| Split | Rows | Source directory |
| ------------ | ----: | ---------------- |
| `train` | 3,120 | `Training` |
| `validation` | 780 | `Validation` |
| `test` | 780 | `Testing` |
## Schema
| Column | Type | Notes |
| -------------------- | -------------- | ------------------------------------------------------------------ |
| `id` | `string` | Stable split-qualified row identifier |
| `image` | `large_binary` | Original inline PNG bytes |
| `medicine_name` | `string` | Source medicine-name label |
| `generic_name` | `string` | Source generic-name label |
| `normalized_text` | `string` | Deterministic normalization of `medicine_name` |
| `category` | `string` | `medication` for every source row |
| `is_medical` | `bool` | `true` for every source row |
| `needs_human_review` | `bool` | `false`; this conversion does not run OCR confidence assessment |
| `searchable_summary` | `string` | Label-derived text: `Medication mention: ()` |
| `source_dataset` | `string` | Upstream Kaggle dataset identifier |
| `split` | `string` | Original source split name: `Training`, `Validation`, or `Testing` |
| `image_filename` | `string` | Original PNG filename within the source split |
`normalized_text`, `category`, `is_medical`, `needs_human_review`, and `searchable_summary` are deterministic conversion metadata. They are not OCR or clinical-model predictions.
## Pre-built indices
* `INVERTED` (FTS) on `searchable_summary`
* `BTREE` on `id`
* `BITMAP` on `category`
* `BITMAP` on `needs_human_review`
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets through the standard Hugging Face `datasets` interface when you want a streaming sample or your pipeline already uses `Dataset` / `IterableDataset`.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset(
"lance-format/handwriting-ocr",
split="train",
streaming=True,
)
for row in hf_ds.take(3):
print(row["medicine_name"], row["generic_name"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the downloaded dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, and Versioning examples below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with lower-level APIs. Use it to inspect the schema, fragments, and pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("./data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip - for production use, download locally first.** Streaming from the Hub works for exploration, but repeated random access and training are faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/handwriting-ocr --repo-type dataset --local-dir ./handwriting-ocr
> ```
>
> Then run the examples below from `./handwriting-ocr`, where `./data` contains the downloaded Lance splits.
## Search
The bundled full-text index makes medicine and generic-name lookups efficient without an embedding model. `searchable_summary` combines the two labels, so a text query can return the matching word image and its annotations in one bounded result set.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./data")
tbl = db.open_table("train")
hits = (
tbl.search("Paracetamol")
.select(["id", "medicine_name", "generic_name", "searchable_summary"])
.limit(10)
.to_list()
)
for row in hits:
print(row["medicine_name"], "-", row["generic_name"])
```
The FTS index operates on label-derived text, not a transcription produced from the image. It is useful for inspecting the handwritten examples associated with known medicine or generic names.
## Curate
Filters produce small, explicit candidate sets for experiments. The example selects a bounded set of Paracetamol examples while avoiding image bytes until a later step needs them.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where("generic_name = 'Paracetamol'")
.select(["id", "image_filename", "medicine_name", "generic_name"])
.limit(100)
.to_list()
)
```
This list can be reviewed directly, saved as an experiment manifest, or used as the input to an annotation or training workflow.
## Evolve
Lance can append new columns without rewriting the original images or annotations. For example, a local copy can add a flag for a specific generic name using a SQL expression, then merge independently produced labels by `id`.
> **Note**: Mutations require a local copy. The examples in this card assume the dataset was downloaded into the current directory, as shown above.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
import pyarrow as pa
db = lancedb.connect("./data")
tbl = db.open_table("train")
tbl.add_columns({"is_paracetamol": "generic_name = 'Paracetamol'"})
review_labels = pa.table({
"id": ["train_00000"],
"review_status": ["reviewed"],
})
tbl.merge(review_labels, on="id")
```
The source columns and pre-built indices remain intact. New labels become available to queries as soon as the write commits.
## Train
Projection lets a training loop read only the columns it needs. The image bytes and medicine label below are projected through a standard PyTorch `DataLoader`; fields added later do not add read cost unless selected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("./data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "medicine_name"])
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=4)
for batch in loader:
# Decode PNG bytes and train an OCR or classification model.
...
```
This keeps the original image representation available for training while avoiding unrelated columns on each epoch.
## Versioning
Every Lance mutation creates a version, so the table history can identify the exact split and annotation state used by a retrieval service or experiment. The downloaded dataset can list its history and create tags for reproducible local workflows.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
tbl.tags.create("labels-v1", tbl.version)
```
A pinned version or tag keeps retrieval and training tied to a known dataset state even as review labels or derived columns are added later.
## Materialize a subset
Use a filtered query to materialize a compact local LanceDB table for writes or intensive training. The query streams matching rows and projected columns without materializing the full result set in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
source_db = lancedb.connect("./data")
source_tbl = source_db.open_table("train")
batches = (
source_tbl.search()
.where("generic_name = 'Paracetamol'")
.select(["id", "image", "medicine_name", "generic_name", "searchable_summary"])
.to_batches()
)
local_db = lancedb.connect("./paracetamol-subset")
local_db.create_table("train", batches)
```
The resulting local table can replace `./data` in the Evolve, Train, and Versioning examples when a compact writable subset is sufficient.
## Citation
Please cite both the upstream dataset and the associated research paper:
```bibtex theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@inproceedings{mia2024deep,
title={A Deep Neural Network Approach with Pioneering Local Dataset to Recognize Doctor's Handwritten Prescription in Bangladesh},
author={Mia, Abdur Rahim and Chowdhury, Mohammad Abdullah-Al-Sajid and Mamun, Abdullah Al and Ruddra, Aurunave Mollik and Tanny, Nawshin Tabassum},
booktitle={2024 International Conference on Advances in Computing, Communication, Electrical, and Smart Systems},
pages={1--6},
year={2024},
publisher={IEEE},
doi={10.1109/iCACCESS61735.2024.10499631}
}
```
Source dataset: [Mamun1113, Doctor's Handwritten Prescription BD dataset](https://www.kaggle.com/datasets/mamun1113/doctors-handwritten-prescription-bd-dataset). Associated paper: [IEEE Xplore](https://ieeexplore.ieee.org/document/10499631).
## License
The upstream Kaggle metadata identifies the database license as [Open Database License (ODbL) v1.0](https://opendatacommons.org/licenses/odbl/1.0/) and states that the contents are copyright the original authors. The upstream page also describes the dataset as free for educational and research use. Downstream users should review the upstream dataset page and publication, satisfy the ODbL obligations, and obtain any additional permissions needed for their intended use.
# HotpotQA distractor
Source: https://docs.lancedb.com/datasets/hotpotqa-distractor
A Lance-formatted version of HotpotQA using the distractor config — multi-hop reading-comprehension questions where each answer requires combining facts from two Wikipedia paragraphs, with 10 candidate paragraphs per question (gold + 8…
Source dataset card and downloadable files for `lance-format/hotpotqa-distractor-lance`.
A Lance-formatted version of [HotpotQA](https://hotpotqa.github.io/) using the `distractor` config — multi-hop reading-comprehension questions where each answer requires combining facts from two Wikipedia paragraphs, with 10 candidate paragraphs per question (gold + 8 distractors). The dataset ships with MiniLM question embeddings, flattened context text for full-text search, and pre-built ANN/FTS indices, available directly from the Hub at `hf://datasets/lance-format/hotpotqa-distractor-lance/data`.
## Key features
* **Multi-hop questions with gold supporting facts** — each row carries the question, the canonical short answer, and the `(title, sent_id)` pointers into the paragraphs that justify it.
* **Ten candidate paragraphs per question** in the parallel `context_titles` / `context_sentences` columns, plus a flattened `context_text` field that feeds the FTS index.
* **Pre-computed 384-dim question embeddings** (`question_emb`, `sentence-transformers/all-MiniLM-L6-v2`, cosine-normalized) with a bundled `IVF_PQ` index for semantic question lookup.
* **One columnar dataset** — scan metadata cheaply, then read the heavy context text only for the rows you actually want.
## Splits
| Split | Rows |
| ------------------ | ------ |
| `train.lance` | 90,447 |
| `validation.lance` | 7,405 |
## Schema
| Column | Type | Notes |
| ---------------------- | ------------------------------- | -------------------------------------------- |
| `id` | `string` | HotpotQA question id |
| `question` | `string` | The question |
| `answer` | `string` | Reference short answer (`yes` / `no` / span) |
| `type` | `string?` | `bridge` or `comparison` |
| `level` | `string?` | `easy` / `medium` / `hard` |
| `supporting_titles` | `list` | Wikipedia titles that contain the gold facts |
| `supporting_sent_ids` | `list` | Sentence indices into those titles |
| `context_titles` | `list` | All 10 paragraph titles (gold + distractors) |
| `context_sentences` | `list>` | Sentences per paragraph |
| `context_text` | `string` | Flattened paragraphs — feeds the FTS index |
| `num_supporting_facts` | `int32` | Number of gold supporting facts |
| `question_emb` | `fixed_size_list` | MiniLM question embedding |
## Pre-built indices
* `IVF_PQ` on `question_emb` — semantic question lookup (cosine)
* `INVERTED` (FTS) on `question` and `context_text` — keyword and hybrid search
* `BTREE` on `id`, `answer` — stable lookup by identifier
* `BITMAP` on `type`, `level` — cheap predicate evaluation for question class
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/hotpotqa-distractor-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["answer"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. Each `.lance` file in `data/` is a table — open by name (`train`, `validation`). The same handle is used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/hotpotqa-distractor-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/hotpotqa-distractor-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/hotpotqa-distractor-lance --repo-type dataset --local-dir ./hotpotqa-distractor-lance
> ```
>
> Then point Lance or LanceDB at `./hotpotqa-distractor-lance/data`.
## Search
The bundled `IVF_PQ` index on `question_emb` makes nearest-neighbour question lookup a single call. In production you would encode an incoming user question through the same 384-dim MiniLM encoder used at ingest and pass the resulting vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in so the snippet works without loading a model.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/hotpotqa-distractor-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["question_emb", "question"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="question_emb")
.metric("cosine")
.where("level = 'hard'", prefilter=True)
.select(["question", "answer", "supporting_titles", "type"])
.limit(10)
.to_list()
)
for r in hits:
print(f"[{r['type']}] {r['question']} -> {r['answer']}")
```
The result set carries only the projected columns; the 384-d `question_emb` is never read on the result side, and the long `context_text` body is left untouched, keeping the working set small even when the underlying scan touches every row of the train split.
Because the dataset also ships an `INVERTED` index on both `question` and `context_text`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query against the full paragraph text. LanceDB merges the two result lists and reranks them in a single call, which is useful when a named entity must literally appear in one of the supporting paragraphs but the dense side still does most of the ranking.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid")
.vector(seed["question_emb"])
.text("inception dunkirk")
.select(["question", "answer", "supporting_titles"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(r["question"], "->", r["answer"])
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency for your workload.
## Curate
Building a focused evaluation slice usually means stacking predicates over the question metadata before any context text gets read. Lance evaluates the filter inside a single scan, so the candidate set comes back already filtered, and the bounded `.limit(2000)` keeps the output small enough to inspect. The example below assembles a set of hard, multi-hop comparison questions for which the gold answer is a real span rather than `yes`/`no`.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/hotpotqa-distractor-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where(
"type = 'comparison' "
"AND level = 'hard' "
"AND num_supporting_facts >= 2 "
"AND answer NOT IN ('yes', 'no') "
"AND length(question) >= 40",
prefilter=True,
)
.select(["id", "question", "answer", "supporting_titles"])
.limit(2000)
.to_list()
)
print(f"{len(candidates)} candidates; first: {candidates[0]['question']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of question ids, or hand to the Evolve and Train sections below. Neither `context_text` nor `context_sentences` is read by this scan, so a 2000-row curation pass against the Hub moves only kilobytes of metadata.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `question_length` column and a `is_multi_hop` flag, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./hotpotqa-distractor-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"question_length": "length(question)",
"is_multi_hop": "num_supporting_facts >= 2",
})
```
If the values you want to attach already live in another table (offline retriever scores, reranker logits, alternate embeddings from a stronger model), merge them in by joining on the question `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
retriever_scores = pa.table({
"id": pa.array(["5a8b57f25542995d1e6f1371", "5a8c7595554299585d9e36b6"]),
"bm25_top1_score": pa.array([12.7, 9.4]),
})
tbl.merge(retriever_scores, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a different encoder over the question text), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a multi-hop QA model the natural projection is the question plus the flattened context and the gold answer; for a question-encoder retraining loop the precomputed embedding is enough on its own.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/hotpotqa-distractor-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["question", "context_text", "answer"])
loader = DataLoader(train_ds, batch_size=16, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; tokenize, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["question_emb", "answer"]` to `select_columns(...)` on the next run reads only the 384-d vectors and the short answer string, which is the right shape for fine-tuning a retrieval head on cached embeddings. Columns added in Evolve cost nothing per batch until they are explicitly projected.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/hotpotqa-distractor-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./hotpotqa-distractor-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("hard-multihop-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="hard-multihop-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A QA system locked to `hard-multihop-v1` keeps returning stable supporting facts while the dataset evolves in parallel — newly added retriever scores or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same questions and contexts, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/hotpotqa-distractor-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where(
"type = 'comparison' "
"AND level = 'hard' "
"AND num_supporting_facts >= 2"
)
.select(["id", "question", "answer", "supporting_titles", "context_text", "question_emb"])
.to_batches()
)
local_db = lancedb.connect("./hotpotqa-hard-comparison")
local_db.create_table("train", batches)
```
The resulting `./hotpotqa-hard-comparison` is a first-class LanceDB database. Every snippet in the Search, Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/hotpotqa-distractor-lance/data` for `./hotpotqa-hard-comparison`.
## Source & license
Converted from [`hotpot_qa`](https://huggingface.co/datasets/hotpot_qa) (`distractor` config). HotpotQA is released under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
## Citation
```
@inproceedings{yang2018hotpotqa,
title={HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering},
author={Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William W. and Salakhutdinov, Ruslan and Manning, Christopher D.},
booktitle={Empirical Methods in Natural Language Processing (EMNLP)},
year={2018}
}
```
# ImageNet-1k Validation
Source: https://docs.lancedb.com/datasets/imagenet-1k-val
A Lance-formatted version of the canonical 50,000-image ImageNet-1k (ILSVRC2012) validation split, sourced from benjamin-paine/imagenet-1k. Each row is one image with its integer class id, a string class name, and a cosine-normalized OpenCLIP image…
Source dataset card and downloadable files for `lance-format/imagenet-1k-val-lance`.
A Lance-formatted version of the canonical 50,000-image ImageNet-1k (ILSVRC2012) validation split, sourced from [`benjamin-paine/imagenet-1k`](https://huggingface.co/datasets/benjamin-paine/imagenet-1k). Each row is one image with its integer class id, a string class name, and a cosine-normalized OpenCLIP image embedding — all stored inline and available directly from the Hub at `hf://datasets/lance-format/imagenet-1k-val-lance/data`. The 1.28 M ImageNet-1k train split (\~155 GB) is intentionally out of scope for this redistribution; the val split is the canonical evaluation slice for classification benchmarks and is small enough (\~7 GB Lance) to ride entirely in inline storage alongside its embeddings.
## Key features
* **Inline JPEG bytes** in the `image` column — no per-class folders, no sidecar files.
* **Pre-computed OpenCLIP image embeddings** (`image_emb`, ViT-B/32 trained on `laion2b_s34b_b79k`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index for similarity search.
* **Both label representations** — integer `label` (0-999) and string `label_name` (first synonym of the WordNet synset, e.g. `golden_retriever`) — with scalar indices on both for fast class filters.
* **One columnar dataset** — scan labels and embeddings cheaply, fetch image bytes only for the rows you actually need.
## Splits
A single split, shipped as `validation.lance` (50,000 rows).
## Schema
| Column | Type | Notes |
| ------------ | ------------------------------- | ----------------------------------------------------------------------------- |
| `id` | `int64` | Row index within the split, 0-49,999 (natural join key) |
| `image` | `large_binary` | Inline JPEG bytes |
| `label` | `int32` | Class id (0-999) |
| `label_name` | `string` | First synonym of the synset, underscore-spaced (e.g. `golden_retriever`) |
| `image_emb` | `fixed_size_list` | OpenCLIP `ViT-B-32` / `laion2b_s34b_b79k` image embedding (cosine-normalized) |
The full comma-separated WordNet synset descriptions for each class are stored in the dataset metadata under `lance:class_names`.
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine, `num_partitions=64`)
* `BTREE` on `label` — fast equality / range filters by class id
* `BITMAP` on `label_name` — fast set-membership filters by class name
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/imagenet-1k-val-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["label_name"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/imagenet-1k-val-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/imagenet-1k-val-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/imagenet-1k-val-lance --repo-type dataset --local-dir ./imagenet-1k-val-lance
> ```
>
> Then point Lance or LanceDB at `./imagenet-1k-val-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes nearest-neighbor retrieval over the validation set a single call. In production you would encode a query image through the same OpenCLIP `ViT-B-32` / `laion2b_s34b_b79k` model used at ingest (cosine-normalized) and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the embedding already stored in row 42 as a runnable stand-in, so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/imagenet-1k-val-lance/data")
tbl = db.open_table("validation")
seed = (
tbl.search()
.select(["image_emb", "label_name"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "label_name"])
.limit(10)
.to_list()
)
print(f"reference class: {seed['label_name']}")
for r in hits:
print(f" id={r['id']:>6} {r['label_name']}")
```
Because the embeddings are cosine-normalized at ingest, `metric="cosine"` is the right choice and the first hit will typically be the seed image itself — a useful sanity check. Tune `nprobes` and `refine_factor` to trade recall against latency for your workload.
## Curate
A typical curation pass for an ImageNet-style classification or robustness study narrows the validation set to a single class (or a synset prefix) and then materializes a small candidate set for inspection. Stacking the filter and the projection inside a single scan keeps the result small and explicit, and the bounded `.limit(200)` makes it cheap to inspect before committing the subset to anything downstream.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/imagenet-1k-val-lance/data")
tbl = db.open_table("validation")
candidates = (
tbl.search()
.where("label_name = 'golden_retriever'")
.select(["id", "label", "label_name"])
.limit(200)
.to_list()
)
print(f"{len(candidates)} golden_retriever validation rows")
```
The `BITMAP` index on `label_name` resolves the predicate without scanning, and the `image` column is never read, so the network traffic for the candidate scan is dominated by the small metadata payload rather than JPEG bytes. The result is a plain list of dictionaries, ready to inspect, persist as a manifest of row ids, or feed into the Evolve and Train workflows below. To grab a family of related classes, replace the equality with a `LIKE` predicate such as `label_name LIKE 'tabby%'` or an `IN` set over a curated synset list.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a coarse `is_dog` flag over a curated set of canine synsets, which can then be used directly in later `where` clauses without re-listing the class set on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./imagenet-1k-val-lance/data") # local copy required for writes
tbl = db.open_table("validation")
tbl.add_columns({
"is_dog": "label_name IN ('golden_retriever', 'Labrador_retriever', 'beagle', 'pug', 'poodle')",
})
```
If the values you want to attach already live in another table — per-class hypernyms from WordNet, ImageNet-A / ImageNet-R membership flags, model-prediction logs from an external eval run — merge them in by joining on `label_name`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
hypernyms = pa.table({
"label_name": pa.array(["golden_retriever", "tabby", "espresso"]),
"hypernym": pa.array(["dog", "cat", "beverage"]),
})
tbl.merge(hypernyms, on="label_name")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running an alternative vision backbone over the image bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop — or, more commonly for this split, an evaluation loop — read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/imagenet-1k-val-lance/data")
tbl = db.open_table("validation")
eval_ds = Permutation.identity(tbl).select_columns(["image", "label"])
loader = DataLoader(eval_ds, batch_size=128, shuffle=False, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb stays on disk.
# decode the JPEG bytes, forward through your classifier, accumulate top-1 / top-5...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "label"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a linear probe or a lightweight classifier head on top of frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/imagenet-1k-val-lance/data")
tbl = db.open_table("validation")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./imagenet-1k-val-lance/data")
local_tbl = local_db.open_table("validation")
local_tbl.tags.create("clip-vitb32-laion2b-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("validation", version="clip-vitb32-laion2b-v1")
tbl_v5 = db.open_table("validation", version=5)
```
Pinning supports two workflows. An evaluation harness locked to `clip-vitb32-laion2b-v1` keeps reporting numbers against a fixed snapshot of labels and embeddings even as the dataset evolves in parallel; newly added columns or relabelings do not change what the tag resolves to. A research experiment pinned to the same tag can be rerun later against the exact same images, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training or evaluation loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/imagenet-1k-val-lance/data")
remote_tbl = remote_db.open_table("validation")
batches = (
remote_tbl.search()
.where("label_name IN ('golden_retriever', 'Labrador_retriever', 'beagle', 'pug', 'poodle')")
.select(["id", "image", "label", "label_name", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./imagenet-dogs-subset")
local_db.create_table("validation", batches)
```
The resulting `./imagenet-dogs-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/imagenet-1k-val-lance/data` for `./imagenet-dogs-subset`.
## Source & license
Converted from [`benjamin-paine/imagenet-1k`](https://huggingface.co/datasets/benjamin-paine/imagenet-1k), itself a redistribution of the [ILSVRC2012 ImageNet-1k validation split](https://image-net.org/challenges/LSVRC/2012/). All use is subject to the [ImageNet terms of access](https://image-net.org/download.php) — **for research use only**.
## Citation
```
@inproceedings{deng2009imagenet,
title={ImageNet: A Large-Scale Hierarchical Image Database},
author={Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2009}
}
```
# Datasets
Source: https://docs.lancedb.com/datasets/index
Browse Lance-format datasets ready to query directly from the Hugging Face Hub.
The [`lance-format`](https://huggingface.co/lance-format) organization on Hugging Face publishes a growing
catalog of multimodal datasets in Lance format. Each one bundles the raw data (images, audio, video, or text),
pre-computed embeddings, and on-disk vector / full-text indices as first-class columns in the same dataset —
so vector search, full-text search, and filtered scans work directly via `hf://` URIs without downloading.
This is powered under the hood by the [Lance format's native Hugging Face integration](https://lance.org/integrations/huggingface/)
(via the [`pylance`](https://pypi.org/project/pylance/) library). LanceDB sits on top of Lance and gives you a
convenient table-style interface to query these datasets straight from the Hub:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format//data")
tbl = db.open_table("train")
# Vector search, full-text search, or filtered scans — directly on the Hub
results = tbl.search(query).limit(10).to_list()
```
Click any card below for usage examples, schema, and pre-built indices. For a complete walkthrough of the
integration itself, see the [Hugging Face Hub integration page](/integrations/ai/huggingface).
## Image Classification
`lance-format/mnist-lance` — A Lance-formatted version of the classic MNIST handwritten-digit dataset covering 70,000 28×28 grayscale digits across ten balanced classes. Each row carries inline PNG bytes, the digit label, the human-readable class name, and a cosine-normalized…
`lance-format/cifar10-lance` — A Lance-formatted version of CIFAR-10 covering 60,000 32×32 RGB images across ten balanced object classes. Each row carries inline PNG bytes, the integer label, the human-readable class name, and a cosine-normalized CLIP image embedding, all backed…
`lance-format/fashion-mnist-lance` — A Lance-formatted version of Fashion-MNIST covering 70,000 28×28 grayscale clothing images across ten balanced apparel classes. Each row carries inline PNG bytes, the integer label, the human-readable class name, and a cosine-normalized CLIP image…
`lance-format/food101-lance` — A Lance-formatted version of Food-101, the fine-grained dish-classification benchmark of 101,000 photos spread evenly across 101 dish classes, sourced from ethz/food101. Each row carries the inline JPEG bytes, the integer label, the human-readable…
`lance-format/oxford-pets-lance` — A Lance-formatted version of the Oxford-IIIT Pet dataset — 7,390 cat and dog photos across 37 breeds — sourced from pcuenq/oxford-pets. Each row carries the inline JPEG bytes, the breed name, a species flag distinguishing cats from dogs, and a…
`lance-format/stanford-cars-lance` — A Lance-formatted version of the Stanford Cars fine-grained benchmark — 8,144 photographs across 196 make/model/year classes — sourced from Multimodal-Fatima/StanfordCars\_train. Each row carries the inline JPEG bytes, the integer class id, a…
`lance-format/imagenet-1k-val-lance` — A Lance-formatted version of the canonical 50,000-image ImageNet-1k (ILSVRC2012) validation split, sourced from benjamin-paine/imagenet-1k. Each row is one image with its integer class id, a string class name, and a cosine-normalized OpenCLIP image…
`lance-format/eurosat-lance` — A Lance-formatted version of EuroSAT, the canonical Sentinel-2 RGB land-cover benchmark, sourced from blanchon/EuroSAT\_RGB. Each row is a single 64×64 RGB tile with its integer class id, the human-readable class name, and a cosine-normalized…
## OCR
`lance-format/handwriting-ocr` — This Lance-formatted version of the Doctor's Handwritten Prescription BD dataset contains 4,680 cropped PNG images of handwritten medicine names from Bangladesh. Each row keeps the original image bytes with the medicine and generic-name labels…
## Object Detection & Segmentation
`lance-format/coco-detection-2017-lance` — A Lance-formatted version of the COCO 2017 object detection benchmark, sourced from detection-datasets/coco. Each row is one image with its inline JPEG bytes, the full per-image list of bounding boxes, COCO 80-class category ids and names…
`lance-format/pascal-voc-2012-segmentation-lance` — A Lance-formatted version of the Pascal VOC 2012 semantic segmentation split, sourced from nateraw/pascal-voc-2012. Each row pairs an inline JPEG image with the per-pixel PNG segmentation mask and a cosine-normalized OpenCLIP ViT-B-32 image…
`lance-format/ade20k-lance` — A Lance-formatted version of the full ADE20K scene parsing benchmark, sourced from 1aurent/ADE20K. Each row is one scene image with its inline JPEG bytes, a per-pixel semantic segmentation map encoded as PNG bytes, an optional instance map, scene…
`lance-format/kitti-2d-detection-lance` — A Lance-formatted version of the KITTI 2D Object Detection benchmark, sourced from nateraw/kitti so no manual signup or download from cvlibs.net is required. Each row is a single driving frame with inline JPEG bytes, the full set of 2D and 3D…
## Image Retrieval
`lance-format/coco-captions-2017-lance` — A Lance-formatted version of the COCO Captions 2017 corpus, redistributed via lmms-lab/COCO-Caption2017. Each row is one image with 5–7 human-written captions, a cosine-normalized CLIP image embedding, and a cosine-normalized CLIP text embedding of…
`lance-format/flickr30k-lance` — A Lance-formatted version of Flickr30k, redistributed via lmms-lab/flickr30k. Each row is one image with 5 human-written captions, a cosine-normalized CLIP image embedding, and a cosine-normalized CLIP text embedding of the canonical caption — all…
`lance-format/laion-1m` — A Lance-formatted slice of the LAION image-text corpus (\~1M rows) with inline JPEG bytes, CLIP image embeddings (img\_emb), full metadata, and a pre-built ANN index — all available directly from the Hub at…
## Visual Question Answering
`lance-format/chartqa-lance` — A Lance-formatted version of ChartQA, a benchmark for question answering over scientific and business charts that demands a mix of logical and visual reasoning, redistributed via lmms-lab/ChartQA. Each row carries the chart image as inline JPEG…
`lance-format/docvqa-lance` — A Lance-formatted version of DocVQA, a benchmark for visual question answering over document images such as industry and government scans, multi-page reports, forms, and receipts, redistributed via lmms-lab/DocVQA (DocVQA config). Each row carries…
`lance-format/textvqa-lance` — A Lance-formatted version of TextVQA — visual question answering where the question requires reading text in the image (street signs, product labels, screen captures) — sourced from lmms-lab/textvqa. Each row carries the image bytes, the question…
`lance-format/vqav2-lance` — A Lance-formatted version of VQAv2 — open-ended visual question answering on COCO images — sourced from lmms-lab/VQAv2. Each row is one (image, question, 10 annotator answers) triple with paired CLIP image and question embeddings drawn from the…
`lance-format/gqa-testdev-balanced-lance` — A Lance-formatted version of the canonical GQA testdev\_balanced slice — 12,578 compositional VQA questions joined against the matching 398 images — sourced from lmms-lab/GQA. The original redistribution ships instructions and images as separate…
## Text QA
`lance-format/squad-v2-lance` — A Lance-formatted version of SQuAD v2 — the Stanford Question Answering Dataset with both answerable and deliberately unanswerable questions over Wikipedia passages — with MiniLM question embeddings stored inline and ready for retrieval at…
`lance-format/trivia-qa-lance` — A Lance-formatted version of TriviaQA (rc.nocontext config) — a large reading-comprehension dataset of trivia questions paired with a canonical answer, accepted aliases, and entity-type metadata — with MiniLM question embeddings stored inline and…
`lance-format/hotpotqa-distractor-lance` — A Lance-formatted version of HotpotQA using the distractor config — multi-hop reading-comprehension questions where each answer requires combining facts from two Wikipedia paragraphs, with 10 candidate paragraphs per question (gold + 8…
`lance-format/natural-questions-val-lance` — A Lance-formatted version of the Natural Questions validation split — 7,830 real Google search queries paired with the full Wikipedia article a human used to answer them, plus 1–5 annotator labels per question. MiniLM question embeddings are stored…
`lance-format/ms-marco-v2.1-lance` — A Lance-formatted version of MS MARCO v2.1 — Microsoft's machine-reading-comprehension benchmark built from anonymized Bing query logs. Each row is one user query, the up-to-10 candidate passages Bing retrieved for it with relevance flags, and the…
## Text Corpora
`lance-format/fineweb-edu` — A Lance-formatted version of FineWeb-Edu — over 1.5 billion educational web passages with cleaned text, source metadata, language detection signals, and 384-dim text embeddings — available directly from the Hub at…
## Speech
`lance-format/librispeech-clean-lance` — A Lance-formatted version of the LibriSpeech ASR clean configuration, sourced from openslr/librispeech\_asr. Each row is one utterance with inline FLAC audio bytes, the reference transcript, a sentence-transformers embedding of that transcript, and…
## Video
`lance-format/openvid-lance` — A Lance-formatted version of the OpenVid-1M corpus — 937,957 high-quality clips with inline MP4 bytes, 1024-dim video embeddings, captions, and rich per-clip quality signals — available directly from the Hub at…
## Robotics
`lance-format/lerobot-pusht-lance` — A Lance-formatted version of lerobot/pusht — the canonical PushT benchmark from the Diffusion Policy paper — packaged using the same three-table layout as lance-format/lerobot-xvla-soft-fold so consumers can flip between datasets without changing…
`lance-format/lerobot-xvla-soft-fold` — A Lance-formatted version of lerobot/xvla-soft-fold — a multi-camera robotics dataset from the X-VLA project — packaged as three Lance tables for efficient frame-level training, episode-level trajectory loading, and direct access to the original…
## Share your own dataset
Got a multimodal dataset you want to publish? Convert it to Lance and push it to the Hub!
Anyone who opens it gets vector search, full-text search, and filtered scans on the data out of the box,
without recreating the embeddings or indexes on their end.
A step-by-step walkthrough on the LanceDB blog covering CLI setup, packaging your dataset, pushing to your namespace, and writing a dataset card.
Or browse the [latest trending Lance datasets](https://huggingface.co/datasets?format=format:lance\&sort=trending) on Hugging Face.
# KITTI 2D Detection
Source: https://docs.lancedb.com/datasets/kitti-2d-detection
A Lance-formatted version of the KITTI 2D Object Detection benchmark, sourced from nateraw/kitti so no manual signup or download from cvlibs.net is required. Each row is a single driving frame with inline JPEG bytes, the full set of 2D and 3D…
Source dataset card and downloadable files for `lance-format/kitti-2d-detection-lance`.
A Lance-formatted version of the [KITTI 2D Object Detection benchmark](https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d), sourced from [`nateraw/kitti`](https://huggingface.co/datasets/nateraw/kitti) so no manual signup or download from cvlibs.net is required. Each row is a single driving frame with inline JPEG bytes, the full set of 2D and 3D object annotations stored as parallel per-object lists, plus a cosine-normalized OpenCLIP `ViT-B-32` image embedding — all available directly from the Hub at `hf://datasets/lance-format/kitti-2d-detection-lance/data`.
KITTI is the canonical autonomous-driving detection benchmark with 8 object classes drawn from real street scenes around Karlsruhe. It is widely used for AV perception research and serves as a small-scale companion to nuScenes and Waymo.
## Key features
* **Inline JPEG bytes** in the `image` column — no parallel `image_2/` and `label_2/` folders to keep in sync.
* **Per-object 2D and 3D annotations on the same row** — bounding boxes, observation angles, 3D dimensions, locations, yaw, occlusion and truncation flags travel as parallel list columns of equal length.
* **Pre-computed CLIP image embeddings** (`image_emb`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index for visual similarity over driving scenes.
* **Scalar and label-list indices** on `num_objects` and `types_present` make per-class and crowdedness filters cheap on the Hub copy and locally.
## Splits
| Split | Rows | Notes |
| ------------- | ----- | --------------------------------------- |
| `train.lance` | 7,481 | Official KITTI training set with labels |
The KITTI `test` split has no public labels and is intentionally not bundled. Add it via `--splits train test` in `kitti/dataprep.py` if you want the unlabeled images for inference.
## Schema
| Column | Type | Notes |
| --------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `id` | `int64` | Row index within split (natural join key for merges) |
| `image` | `large_binary` | Inline JPEG bytes (re-encoded from the source PNG) |
| `bboxes` | `list>` | 2D box per object — `[left, top, right, bottom]` in pixel coords |
| `alphas` | `list` | Observation angle per object (radians, KITTI convention) |
| `dimensions` | `list>` | 3D box `(h, w, l)` per object, in metres |
| `locations` | `list>` | 3D centre `(x, y, z)` per object in camera coords, in metres |
| `rotation_y` | `list` | Yaw per object in camera coords (radians) |
| `occluded` | `list` | KITTI occlusion flag (0=visible, 1=partly, 2=largely, 3=unknown) |
| `truncated` | `list` | Truncation fraction per object (0.0-1.0) |
| `types` | `list` | Class name per object (`Car`, `Van`, `Truck`, `Pedestrian`, `Person_sitting`, `Cyclist`, `Tram`, `Misc`, `DontCare`) |
| `num_objects` | `int32` | Number of annotated objects in the frame |
| `types_present` | `list` | Deduped class names — feeds the LABEL\_LIST index |
| `image_emb` | `fixed_size_list` | OpenCLIP `ViT-B-32` image embedding (cosine-normalized) |
All `list<...>` annotation columns on the same row are aligned — index `i` across `bboxes`, `alphas`, `dimensions`, `locations`, `rotation_y`, `occluded`, `truncated`, and `types` describes the same physical object.
## Pre-built indices
* `IVF_PQ` on `image_emb` — `metric=cosine`
* `BTREE` on `num_objects`
* `LABEL_LIST` on `types_present`
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/kitti-2d-detection-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["id"], row["num_objects"], row["types_present"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/kitti-2d-detection-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/kitti-2d-detection-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/kitti-2d-detection-lance --repo-type dataset --local-dir ./kitti-2d-detection-lance
> ```
>
> Then point Lance or LanceDB at `./kitti-2d-detection-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes visual nearest-neighbour search over driving scenes a single call. In production you would encode a query frame (or a scene prototype) through OpenCLIP `ViT-B-32` at runtime and pass the resulting 512-d cosine-normalized vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/kitti-2d-detection-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb", "types_present"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "num_objects", "types_present"])
.limit(10)
.to_list()
)
print("query scene types:", seed["types_present"])
for r in hits:
print(f" id={r['id']:>5} n={r['num_objects']:>2} {r['types_present']}")
```
Because the embeddings are cosine-normalized, `metric="cosine"` is the natural choice and the first hit is typically the seed row itself. Visual neighbours tend to share scene-level structure (highway vs. urban intersection vs. parked-cars row) before they share class composition, which is what makes the cross between `image_emb` and the `types_present` / `num_objects` indices useful for the curation patterns below.
## Curate
KITTI's parallel per-object list columns make composition-based filters natural: pick scenes by which classes are present, by how many objects are in them, or by the occlusion profile of those objects. Lance evaluates these predicates inside a single filtered scan, and the bounded `.limit(...)` keeps the candidate set small and explicit. The first snippet below finds crowded scenes that contain at least one cyclist and one pedestrian — a useful slice for vulnerable-road-user studies.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/kitti-2d-detection-lance/data")
tbl = db.open_table("train")
vru = (
tbl.search()
.where(
"array_has_all(types_present, ['Cyclist', 'Pedestrian']) AND num_objects >= 8",
prefilter=True,
)
.select(["id", "num_objects", "types_present"])
.limit(200)
.to_list()
)
print(f"{len(vru)} VRU-rich frames")
```
A second pass can combine a structural filter with visual similarity: take a crowded urban seed frame and look for visually similar frames whose object lists also contain cars. This is a one-shot retrieval against the `IVF_PQ` index, joined with the `LABEL_LIST` index on `types_present` inside a single query.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
seed = (
tbl.search()
.where("num_objects >= 10 AND array_contains(types_present, 'Car')", prefilter=True)
.select(["image_emb"])
.limit(1)
.to_list()[0]
)
similar_crowded = (
tbl.search(seed["image_emb"])
.metric("cosine")
.where("array_contains(types_present, 'Car')", prefilter=True)
.select(["id", "num_objects", "types_present"])
.limit(50)
.to_list()
)
```
The results are plain lists of dictionaries, ready to inspect, persist as manifests of `id`s, or feed into the Evolve and Train workflows below. The annotation list columns and `image_emb` are read; the JPEG bytes are not touched until you ask for them.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds per-frame counts for the two most safety-relevant classes plus a `has_vru` flag, all of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./kitti-2d-detection-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"num_cars": "array_length(array_filter(types, x -> x = 'Car'))",
"num_pedestrians": "array_length(array_filter(types, x -> x = 'Pedestrian'))",
"has_vru": "array_has_any(types_present, ['Pedestrian', 'Cyclist'])",
})
```
If the values you want to attach already live in another table — detector predictions on the same frames, LIDAR-derived per-frame features, or human re-annotation — merge them in by joining on `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"id": pa.array([0, 1, 2], type=pa.int64()),
"pred_num_cars": pa.array([3, 5, 0], type=pa.int32()),
})
tbl.merge(predictions, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (running a fresh detector over the JPEG bytes, deriving alternative embeddings), Lance also provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a 2D detector, project the JPEG bytes together with the per-object `bboxes` and `types` lists; everything else (3D annotations, CLIP embeddings) stays on disk until you opt in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/kitti-2d-detection-lance/data")
tbl = db.open_table("train")
train_ds = (
Permutation.identity(tbl)
.select_columns(["image", "bboxes", "types"])
)
loader = DataLoader(train_ds, batch_size=8, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; 3D fields and image_emb stay on disk.
# decode the JPEGs, drop DontCare boxes, build target tensors, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "types_present"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors plus the deduped class list, which is the right shape for training a lightweight scene classifier or a linear probe on top of frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges predictions, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/kitti-2d-detection-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./kitti-2d-detection-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("kitti-clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="kitti-clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A perception service locked to `kitti-clip-vitb32-v1` keeps returning stable retrieval results while the dataset evolves in parallel — newly added detector predictions or alternative embeddings do not change what the tag resolves to. A detection-training experiment pinned to the same tag can be rerun later against the exact same frames and annotations, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory. The filter below carves out a vulnerable-road-user training set — frames that contain at least one pedestrian or cyclist — and writes them to a local LanceDB database.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/kitti-2d-detection-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("array_has_any(types_present, ['Pedestrian', 'Cyclist'])")
.select(["id", "image", "bboxes", "types", "num_objects", "types_present", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./kitti-vru-subset")
local_db.create_table("train", batches)
```
The resulting `./kitti-vru-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/kitti-2d-detection-lance/data` for `./kitti-vru-subset`.
## Source & license
Converted from [`nateraw/kitti`](https://huggingface.co/datasets/nateraw/kitti). KITTI is released under the [CC BY-NC-SA 3.0 license](https://creativecommons.org/licenses/by-nc-sa/3.0/) by Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago — **non-commercial research use only**. See the [KITTI license page](https://www.cvlibs.net/datasets/kitti/) for details.
## Citation
```
@inproceedings{geiger2012are,
title={Are we ready for autonomous driving? The KITTI vision benchmark suite},
author={Geiger, Andreas and Lenz, Philip and Urtasun, Raquel},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2012}
}
```
# LAION-1M
Source: https://docs.lancedb.com/datasets/laion-1m
A Lance-formatted slice of the LAION image-text corpus (~1M rows) with inline JPEG bytes, CLIP image embeddings (img_emb), full metadata, and a pre-built ANN index — all available directly from the Hub at…
Source dataset card and downloadable files for `lance-format/laion-1m`.
A Lance-formatted slice of the [LAION](https://laion.ai/blog/laion-5b/) image-text corpus (\~1M rows) with inline JPEG bytes, CLIP image embeddings (`img_emb`), full metadata, and a pre-built ANN index — all available directly from the Hub at `hf://datasets/lance-format/laion-1m/data/train.lance`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files, no image folders.
* **Pre-computed CLIP image embeddings** (`img_emb`, 768-dim) with a bundled `IVF_PQ` index for similarity search.
* **Full LAION metadata** — captions, URLs, NSFW flags, EXIF, dimensions, similarity scores.
* **One columnar dataset** — scan metadata cheaply, then fetch image bytes only for the rows you want.
## Splits
`train.lance`
## Schema
| Column | Type | Notes |
| ---------------------------------------- | ------------------------------- | ------------------------------------- |
| `key` | `int64` | Row key (natural join key for merges) |
| `image` | `large_binary` | Inline JPEG bytes |
| `image_path` | `string` | Original filename |
| `caption` | `string` | Image caption |
| `url` | `string` | Source URL |
| `NSFW` | `int64` | 0 = safe, 1 = NSFW |
| `LICENSE` | `string` | Per-row license tag |
| `similarity` | `float64` | CLIP image–text cosine similarity |
| `width`, `height` | `int64` | Image dimensions |
| `original_width`, `original_height` | `int64` | Original dimensions before resize |
| `exif`, `md5`, `status`, `error_message` | `string` | Provenance / metadata |
| `img_emb` | `fixed_size_list` | CLIP image embedding |
## Pre-built indices
* `IVF_PQ` on `img_emb` — vector similarity search (L2)
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable if your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/laion-1m", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["caption"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, and Versioning examples below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/laion-1m/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/laion-1m --repo-type dataset --local-dir ./laion-1m
> ```
>
> Then point Lance or LanceDB at `./laion-1m/data`.
## Search
The bundled `IVF_PQ` index on `img_emb` makes approximate-nearest-neighbor search a single call. In production you would encode a user prompt or query image through CLIP at runtime and pass the resulting 768-d vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")
query = (
tbl.search()
.select(["img_emb"])
.limit(1)
.offset(42)
.to_list()[0]["img_emb"]
)
hits = (
tbl.search(query)
.metric("L2")
.select(["caption", "url", "similarity"])
.limit(10)
.to_list()
)
for r in hits:
print(f"{r['similarity']:.3f} | {r['caption'][:80]}")
```
Tune `metric`, `nprobes`, and `refine_factor` to trade recall against latency for your workload.
## Curate
Building a focused subset usually means combining similarity with metadata filters. Lance evaluates both inside a single query, so the candidate set comes back already filtered. The example below finds images visually similar to a seed row and restricts the result to safe-rated, high-resolution rows in one call. The bounded `.limit(500)` keeps the output small enough to inspect or hand off.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["img_emb", "caption"])
.limit(1)
.offset(42)
.to_list()[0]
)
candidates = (
tbl.search(seed["img_emb"])
.where('"NSFW" = 0 AND similarity > 0.3 AND width >= 512', prefilter=True)
.select(["key", "url", "caption", "similarity"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates around: {seed['caption'][:60]}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of row keys, or feed into the Evolve and Train workflows below.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a precomputed `aspect_ratio` and an `is_high_res` flag, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note**: Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./laion-1m/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"aspect_ratio": "CAST(width AS DOUBLE) / CAST(height AS DOUBLE)",
"is_high_res": "width >= 512 AND height >= 512",
})
```
If the values you want to attach already live in another table (offline labels, classifier predictions, aesthetic scores), merge them in by joining on the `key` column:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
labels = pa.table({
"key": pa.array([0, 1, 2]),
"aesthetic_score": pa.array([7.1, 6.4, 8.9]),
})
tbl.merge(labels, on="key")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running an embedding model over the image bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetch, shuffling, and batching behave as in any PyTorch pipeline. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "caption"])
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; img_emb / img_emb_dinov3 stay on disk.
# decode the JPEG bytes, tokenize the captions, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["img_emb_dinov3", "caption"]` to `select_columns(...)` on the next run reads only those columns, with no data movement or shard reorganization.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./laion-1m/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("aesthetic-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="aesthetic-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `aesthetic-v1` keeps returning stable results while the dataset evolves in parallel; newly added columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same data, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where('"NSFW" = 0 AND similarity > 0.35 AND width >= 512')
.select(["key", "image", "caption", "url", "img_emb"])
.to_batches()
)
local_db = lancedb.connect("./laion-subset")
local_db.create_table("train", batches)
```
The resulting `./laion-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/laion-1m/data` for `./laion-subset`.
## Citation
```
@article{schuhmann2022laion5b,
title={LAION-5B: An open large-scale dataset for training next generation image-text models},
author={Schuhmann, Christoph and others},
journal={NeurIPS Datasets and Benchmarks Track},
year={2022}
}
```
## License
Content inherits LAION's original licensing and safety guidelines. Review [LAION policy](https://laion.ai/blog/laion-5b/) before downstream use.
# LeRobot PushT
Source: https://docs.lancedb.com/datasets/lerobot-pusht
A Lance-formatted version of lerobot/pusht — the canonical PushT benchmark from the Diffusion Policy paper — packaged using the same three-table layout as lance-format/lerobot-xvla-soft-fold so consumers can flip between datasets without changing…
Source dataset card and downloadable files for `lance-format/lerobot-pusht-lance`.
A Lance-formatted version of [`lerobot/pusht`](https://huggingface.co/datasets/lerobot/pusht) — the canonical PushT benchmark from the [Diffusion Policy paper](https://diffusion-policy.cs.columbia.edu/) — packaged using the same three-table layout as [`lance-format/lerobot-xvla-soft-fold`](https://huggingface.co/datasets/lance-format/lerobot-xvla-soft-fold) so consumers can flip between datasets without changing code. Available directly from the Hub at `hf://datasets/lance-format/lerobot-pusht-lance/data`.
## Key features
* **Three-table layout** — `frames`, `episodes`, `videos` — so frame-level training, episode-level trajectory work, and raw video access live side-by-side without scattered parquet shards or sidecar MP4 directories.
* **Inline MP4 segments** in `episodes.lance` (one blob per camera, with `from_timestamp` / `to_timestamp` bounds) and full source MP4s in `videos.lance`, all surfaced as lazy `BlobFile` handles via `take_blobs` so metadata scans never read the bytes.
* **Frame-level observations and actions** in `frames.lance` with stable `episode_index`, `frame_index`, and `index` columns for joining or temporal iteration.
* **Schema-evolution friendly** — add alternate camera streams, language annotations, or model predictions later without rewriting the data.
## Tables
| Table | Rows \~ | Purpose |
| ---------------- | ---------------------- | ------------------------------------------------------------------------------------ |
| `frames.lance` | one row per frame | Per-frame observations, actions, episode/task indices |
| `episodes.lance` | one row per episode | Full per-episode trajectories plus per-camera MP4 segment blobs and timestamp bounds |
| `videos.lance` | one row per source MP4 | Raw source video blobs and file-level provenance (path, size, sha256) |
Use `frames.lance` for low-level training (loss-per-timestep, state-conditioned policies). Use `episodes.lance` when you need the full trajectory and the matching video segments together. Use `videos.lance` when you want direct access to the original encoded video files.
## Schemas
### `frames.lance`
| Column | Type | Notes |
| ------------------- | --------------- | ----------------------------------- |
| `observation_state` | `list` | Robot state vector for that frame |
| `action` | `list` | Action vector for that frame |
| `timestamp` | `float` | Canonical frame timestamp (seconds) |
| `frame_index` | `int64` | Frame index within episode |
| `episode_index` | `int64` | Parent episode id |
| `index` | `int64` | Global frame index |
| `task_index` | `int64` | Task id |
### `episodes.lance`
| Column | Type | Notes |
| ------------------------- | ----------------------------- | ---------------------------------------------------------------- |
| `episode_index` | `int64` | Episode id |
| `task_index` | `int64` | Task id |
| `fps` | `int32` | Frame rate of the episode video segments |
| `timestamps` | `list` | Per-frame timestamps |
| `actions` | `list>` | Per-frame action vectors |
| `observation_state` | `list>` | Per-frame robot state vectors |
| `_video_blob` | `large_binary` (blob-encoded) | Inline MP4 segment for each camera, read lazily via `take_blobs` |
| `_from_timestamp` | `float64` | Segment start time |
| `_to_timestamp` | `float64` | Segment end time |
### `videos.lance`
| Column | Type | Notes |
| --------------------------- | ----------------------------- | ------------------------------- |
| `camera_angle` | `string` | Camera key |
| `chunk_index`, `file_index` | `int32` | IDs parsed from the source path |
| `relative_path`, `filename` | `string` | Provenance |
| `file_size_bytes` | `int64` | Source MP4 size |
| `sha256` | `string` | SHA256 of the MP4 bytes |
| `video_blob` | `large_binary` (blob-encoded) | Raw source MP4 bytes |
## Pre-built indices
None bundled. Build indices on a local copy if a workload calls for them — e.g., a `BTREE` on `frames.episode_index` for fast episode lookup, or a vector index after attaching observation embeddings via Evolve.
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample. Each Lance table is a separate `datasets` config.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/lerobot-pusht-lance", split="frames", streaming=True)
for row in hf_ds.take(3):
print(row["episode_index"], row["frame_index"], row["action"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. Each `.lance` file in `data/` is a table — open by name. The same handles are used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/lerobot-pusht-lance/data")
frames = db.open_table("frames")
episodes = db.open_table("episodes")
videos = db.open_table("videos")
print(len(frames), len(episodes), len(videos))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices — or when you need the blob-level `take_blobs` entry point that streams MP4 bytes lazily from inline storage.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/lerobot-pusht-lance/data/frames.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access to video segments and any kind of indexed search are dramatically faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/lerobot-pusht-lance --repo-type dataset --local-dir ./lerobot-pusht
> ```
>
> Then point Lance or LanceDB at `./lerobot-pusht/data`.
## Search
PushT does not ship a vector index out of the box — observation states are low-dimensional and most robotics workflows look up by index rather than by similarity. The bundled identifier columns (`episode_index`, `task_index`, `frame_index`) make exact lookups a single filtered scan. The example below pulls the first few frames of episode 0 from the frames table.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/lerobot-pusht-lance/data")
frames = db.open_table("frames")
slice_ = (
frames.search()
.where("episode_index = 0 AND frame_index < 10", prefilter=True)
.select(["episode_index", "frame_index", "timestamp", "action", "observation_state"])
.limit(10)
.to_list()
)
for r in slice_:
print(r["frame_index"], r["timestamp"], r["action"])
```
For similarity-style search across states or actions, attach an embedding column via Evolve and build an `IVF_PQ` index on it (see Evolve below). For visual similarity over rendered frames, the pre-extracted-frames pattern in Train below produces a table that can carry a learned image embedding alongside the pixels.
## Curate
A typical curation pass for a robotics workflow starts with an episode-level filter — pick episodes with a particular task, length, or initial condition — and then drops down to the frames within those episodes. Stacking predicates inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(...)` makes it cheap to inspect.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/lerobot-pusht-lance/data")
episodes = db.open_table("episodes")
frames = db.open_table("frames")
# Pick a handful of episodes for the default task.
ep_rows = (
episodes.search()
.where("task_index = 0", prefilter=True)
.select(["episode_index", "fps", "observation_images_image_from_timestamp",
"observation_images_image_to_timestamp"])
.limit(10)
.with_row_id(True)
.to_list()
)
ep_ids = [r["episode_index"] for r in ep_rows]
# Pull the frames belonging to those episodes for the next stage.
frame_rows = (
frames.search()
.where(f"episode_index IN ({', '.join(map(str, ep_ids))})", prefilter=True)
.select(["episode_index", "frame_index", "timestamp", "action", "observation_state"])
.limit(2000)
.to_list()
)
print(f"{len(ep_rows)} episodes, {len(frame_rows)} frames selected")
```
Neither scan reads any video bytes. The MP4 segments live in the blob-encoded `_video_blob` columns and stay on disk until something explicitly asks for them.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `action_magnitude` and a `large_action` flag to the frames table, either of which can then be used directly in `where` clauses.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./lerobot-pusht/data") # local copy required for writes
frames = db.open_table("frames")
frames.add_columns({
"action_magnitude": "SQRT(action[1] * action[1] + action[2] * action[2])",
"large_action": "SQRT(action[1] * action[1] + action[2] * action[2]) > 5.0",
})
```
If the values you want to attach already live in another table (offline reward labels, classifier predictions, learned observation embeddings), merge them in by joining on the appropriate key — `index` for frames or `episode_index` for episodes:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
rewards = pa.table({
"index": pa.array([0, 1, 2]),
"reward_to_go": pa.array([1.4, 1.3, 1.2]),
})
frames.merge(rewards, on="index")
```
The original columns and the inline video blobs are untouched, so existing code that does not reference the new columns continues to work unchanged. For column values that require a Python computation (e.g., running a visual encoder over the decoded video frames), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
A common pattern for vision-conditioned policy training is to pre-extract decoded frame pixels once into a derived LanceDB table — one row per frame, with the per-frame `action` and `observation_state` already joined in — and train against that table with the regular projection-based dataloader. `take_blobs` is the mechanism that makes the extraction step tractable: each episode's MP4 segment is randomly addressable in `episodes.lance` (the `from_timestamp` / `to_timestamp` columns give the segment bounds), so the pass can subset bytes on demand and write decoded frames into a fresh table without an external file store. Other workflows project the `_video_blob` columns from `episodes.lance` directly and decode at the batch boundary, or skip pixels entirely and train a state-only policy on `frames.lance` — the right shape is workload-specific. The actual training loop is the same `Permutation.identity(tbl).select_columns(...)` snippet in every case; only the source table and the column list change.
For a state-only policy, the frames table is already in the right shape — no pre-extraction needed:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/lerobot-pusht-lance/data")
frames = db.open_table("frames")
train_ds = Permutation.identity(frames).select_columns(["observation_state", "action"])
loader = DataLoader(train_ds, batch_size=128, shuffle=True, num_workers=4)
```
For a vision-conditioned policy, train against a pre-extracted frames-with-pixels table that joins each frame's decoded image to its `action` and `observation_state`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("./lerobot-pusht-frames") # local table produced by the one-time extraction
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "observation_state", "action"])
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=4)
```
The inline `_video_blob` storage and `take_blobs` still earn their place outside of the training loop — visualizing an episode in a notebook, sampling for human review, one-off evaluation against a held-out task, and the pre-extraction step itself — but they are not the dataloader.
## Versioning
Every mutation to a Lance table, whether it adds a column, merges labels, or builds an index, commits a new version. Each of `frames`, `episodes`, and `videos` is versioned independently, so a column added to `frames` does not bump the version of `episodes`. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/lerobot-pusht-lance/data")
frames = db.open_table("frames")
print("frames version:", frames.version)
print("history:", frames.list_versions())
print("tags:", frames.tags.list())
```
Once you have a local copy, tag the table for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./lerobot-pusht/data")
local_frames = local_db.open_table("frames")
local_frames.tags.create("pusht-v1", local_frames.version)
```
Reopen by tag or by version number against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
frames_v1 = db.open_table("frames", version="pusht-v1")
frames_v5 = db.open_table("frames", version=5)
```
Pinning supports two workflows. A policy locked to `pusht-v1` keeps reproducing the same behavior while the dataset evolves in parallel. A training experiment pinned to the same tag can be rerun later against the exact same frames, so changes in metrics reflect model changes rather than data drift.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation, index builds) need a writable backing store, and a training pipeline benefits from a local copy with fast random access into the video blobs. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/lerobot-pusht-lance/data")
remote_frames = remote_db.open_table("frames")
batches = (
remote_frames.search()
.where("task_index = 0 AND episode_index < 50")
.select(["episode_index", "frame_index", "index", "timestamp", "action", "observation_state"])
.to_batches()
)
local_db = lancedb.connect("./pusht-task0-subset")
local_db.create_table("frames", batches)
```
The resulting `./pusht-task0-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/lerobot-pusht-lance/data` for `./pusht-task0-subset`. The same pattern applies to `episodes` and `videos` — narrow each table to the rows your workload needs, and the resulting database stays small enough to index and iterate cheaply.
## Source & license
Converted from [`lerobot/pusht`](https://huggingface.co/datasets/lerobot/pusht) (LeRobot v3.0 dataset format). PushT is released under the Apache 2.0 license by the LeRobot project and the Diffusion Policy authors.
## Citation
```
@misc{cadene2024lerobot,
title={LeRobot: State-of-the-art Machine Learning for Real-World Robotics in PyTorch},
author={R{\'e}mi Cadene and Simon Alibert and Alexander Soare and Quentin Gallou{\'e}dec and Adil Zouitine and Steven Palma and Pepijn Kooijmans and Michel Aractingi and Mustafa Shukor and Martino Russi and Francesco Capuano and Caroline Pascal and Jade Choghari and Jess Moss and Thomas Wolf},
year={2024},
url={https://github.com/huggingface/lerobot}
}
@inproceedings{chi2023diffusion,
title={Diffusion Policy: Visuomotor Policy Learning via Action Diffusion},
author={Chi, Cheng and Feng, Siyuan and Du, Yilun and Xu, Zhenjia and Cousineau, Eric and Burchfiel, Benjamin and Song, Shuran},
booktitle={Robotics: Science and Systems},
year={2023}
}
```
# LeRobot X-VLA Soft-Fold
Source: https://docs.lancedb.com/datasets/lerobot-xvla-soft-fold
A Lance-formatted version of lerobot/xvla-soft-fold — a multi-camera robotics dataset from the X-VLA project — packaged as three Lance tables for efficient frame-level training, episode-level trajectory loading, and direct access to the original…
Source dataset card and downloadable files for `lance-format/lerobot-xvla-soft-fold`.
A Lance-formatted version of [`lerobot/xvla-soft-fold`](https://huggingface.co/datasets/lerobot/xvla-soft-fold) — a multi-camera robotics dataset from the [X-VLA](https://thu-air-dream.github.io/X-VLA/) project — packaged as three Lance tables for efficient frame-level training, episode-level trajectory loading, and direct access to the original encoded videos. Available directly from the Hub at `hf://datasets/lance-format/lerobot-xvla-soft-fold/data`.
* **1,542 episodes**
* **2,852,512 frames** at **20 FPS**
* **3 camera streams per episode** — `cam_high`, `cam_left_wrist`, `cam_right_wrist`
* **Robot state and action vectors** aligned to frame timestamps
## Key features
* **Three-table layout** — `frames`, `episodes`, `videos` — so frame-level training, episode-level trajectory work, and raw video access live side-by-side without scattered parquet shards or sidecar MP4 directories.
* **Per-camera inline MP4 segments** in `episodes.lance`, with `from_timestamp` / `to_timestamp` bounds per camera and per episode, surfaced as lazy `BlobFile` handles via `take_blobs` so metadata scans never read the bytes.
* **Frame-level observations and actions** in `frames.lance` with stable `episode_index`, `frame_index`, and `index` columns for joining or temporal iteration.
* **Source MP4 provenance** in `videos.lance` (`relative_path`, `filename`, `file_size_bytes`, `sha256`) alongside the raw bytes, for integrity checks or custom decode pipelines.
## Tables
| Table | Rows | Purpose |
| ---------------- | --------- | ------------------------------------------------------------------------------------ |
| `frames.lance` | 2,852,512 | Per-frame observations, actions, episode/task indices |
| `episodes.lance` | 1,542 | Full per-episode trajectories plus per-camera MP4 segment blobs and timestamp bounds |
| `videos.lance` | 104 | Raw source MP4 files (one row per source MP4) with file-level provenance |
Use `frames.lance` for low-level training (loss-per-timestep, state-conditioned policies). Use `episodes.lance` when you need the full trajectory and the matching per-camera video segments together. Use `videos.lance` when you want direct access to the original encoded video files.
## Schemas
### `frames.lance`
| Column | Type | Notes |
| ------------------- | --------------- | ----------------------------------- |
| `observation_state` | `list` | Robot state vector for that frame |
| `action` | `list` | Action vector for that frame |
| `time_stamp` | `float` | Original source timestamp field |
| `timestamp` | `float` | Canonical frame timestamp (seconds) |
| `frame_index` | `int64` | Frame index within episode |
| `episode_index` | `int64` | Parent episode id |
| `index` | `int64` | Global frame index |
| `task_index` | `int64` | Task id |
### `episodes.lance`
| Column | Type | Notes |
| --------------------------------------------------- | ----------------------------- | ---------------------------------------- |
| `episode_index` | `int64` | Episode id |
| `task_index` | `int64` | Task id |
| `fps` | `int32` | Frame rate of the episode video segments |
| `timestamps` | `list` | Per-frame timestamps |
| `actions` | `list>` | Per-frame action vectors |
| `observation_state` | `list>` | Per-frame robot state vectors |
| `observation_images_cam_high_video_blob` | `large_binary` (blob-encoded) | Inline MP4 segment for `cam_high` |
| `observation_images_cam_high_from_timestamp` | `float64` | `cam_high` segment start time |
| `observation_images_cam_high_to_timestamp` | `float64` | `cam_high` segment end time |
| `observation_images_cam_left_wrist_video_blob` | `large_binary` (blob-encoded) | Inline MP4 segment for `cam_left_wrist` |
| `observation_images_cam_left_wrist_from_timestamp` | `float64` | `cam_left_wrist` segment start time |
| `observation_images_cam_left_wrist_to_timestamp` | `float64` | `cam_left_wrist` segment end time |
| `observation_images_cam_right_wrist_video_blob` | `large_binary` (blob-encoded) | Inline MP4 segment for `cam_right_wrist` |
| `observation_images_cam_right_wrist_from_timestamp` | `float64` | `cam_right_wrist` segment start time |
| `observation_images_cam_right_wrist_to_timestamp` | `float64` | `cam_right_wrist` segment end time |
### `videos.lance`
| Column | Type | Notes |
| --------------------------- | ----------------------------- | ------------------------------- |
| `camera_angle` | `string` | Camera key (e.g. `cam_high`) |
| `chunk_index`, `file_index` | `int32` | IDs parsed from the source path |
| `relative_path`, `filename` | `string` | Provenance |
| `file_size_bytes` | `int64` | Source MP4 size |
| `sha256` | `string` | SHA256 of the MP4 bytes |
| `video_blob` | `large_binary` (blob-encoded) | Raw source MP4 bytes |
## Pre-built indices
None bundled. Build indices on a local copy if a workload calls for them — e.g., a `BTREE` on `frames.episode_index` for fast per-episode lookup, or a vector index after attaching observation embeddings via Evolve.
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample. Each Lance table is a separate `datasets` config.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/lerobot-xvla-soft-fold", split="frames", streaming=True)
for row in hf_ds.take(3):
print(row["episode_index"], row["frame_index"], row["action"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. Each `.lance` file in `data/` is a table — open by name. The same handles are used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/lerobot-xvla-soft-fold/data")
frames = db.open_table("frames")
episodes = db.open_table("episodes")
videos = db.open_table("videos")
print(len(frames), len(episodes), len(videos))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices — or when you need the blob-level `take_blobs` entry point that streams MP4 bytes lazily from inline storage.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/lerobot-xvla-soft-fold/data/frames.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access to video segments and any kind of indexed search are dramatically faster against a local copy. The full dataset is **>50 GB**, so ensure you have sufficient disk space:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/lerobot-xvla-soft-fold --repo-type dataset --local-dir ./lerobot-xvla-soft-fold
> ```
>
> Then point Lance or LanceDB at `./lerobot-xvla-soft-fold/data`. For most workflows, the Materialize-a-subset section at the end of this card is a better starting point than downloading the full corpus.
## Search
This dataset does not ship a vector index out of the box — observation states are low-dimensional and most robotics workflows look up by index rather than by similarity. The bundled identifier columns (`episode_index`, `task_index`, `frame_index`) make exact lookups a single filtered scan. The example below pulls the first few frames of episode 30 from the frames table.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/lerobot-xvla-soft-fold/data")
frames = db.open_table("frames")
slice_ = (
frames.search()
.where("episode_index = 30 AND frame_index < 10", prefilter=True)
.select(["episode_index", "frame_index", "timestamp", "action", "observation_state"])
.limit(10)
.to_list()
)
for r in slice_:
print(r["frame_index"], r["timestamp"], r["action"])
```
For similarity-style search across states or actions, attach an embedding column via Evolve and build an `IVF_PQ` index on it. For visual similarity over rendered frames, the pre-extracted-frames pattern in Train below produces a table that can carry a learned image embedding alongside the pixels.
## Curate
A typical curation pass for a robotics workflow starts with an episode-level filter — pick episodes with a particular task, length, or initial condition — and then either iterates frames or pulls the matching video segments. Stacking predicates inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(...)` makes it cheap to inspect.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/lerobot-xvla-soft-fold/data")
episodes = db.open_table("episodes")
ep_rows = (
episodes.search()
.where("task_index = 0 AND fps = 20", prefilter=True)
.select([
"episode_index",
"observation_images_cam_high_from_timestamp",
"observation_images_cam_high_to_timestamp",
])
.limit(20)
.with_row_id(True)
.to_list()
)
print(f"{len(ep_rows)} episodes selected")
for r in ep_rows[:3]:
print(
f" ep {r['episode_index']} "
f"{r['observation_images_cam_high_from_timestamp']:.2f}s → "
f"{r['observation_images_cam_high_to_timestamp']:.2f}s"
)
```
Neither this scan nor any of the per-camera segment columns are read. The MP4 segments live in the blob-encoded `_video_blob` columns and stay on disk until something explicitly asks for them — which makes "find me the right episodes" a metadata-only operation against a multi-million-frame corpus.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `episode_duration` column to the episodes table from the existing `cam_high` timestamp bounds.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./lerobot-xvla-soft-fold/data") # local copy required for writes
episodes = db.open_table("episodes")
episodes.add_columns({
"episode_duration_s": (
"observation_images_cam_high_to_timestamp - "
"observation_images_cam_high_from_timestamp"
),
"is_long_episode": (
"(observation_images_cam_high_to_timestamp - "
" observation_images_cam_high_from_timestamp) > 120.0"
),
})
```
If the values you want to attach already live in another table (offline reward labels, classifier predictions, learned observation embeddings), merge them in by joining on the appropriate key — `index` for frames or `episode_index` for episodes:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
ep_labels = pa.table({
"episode_index": pa.array([0, 1, 2]),
"outcome": pa.array(["success", "partial", "success"]),
})
episodes.merge(ep_labels, on="episode_index")
```
The original columns and the inline video blobs are untouched, so existing code that does not reference the new columns continues to work unchanged. For column values that require a Python computation (e.g., running a visual encoder over the decoded video frames), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
A common pattern for vision-language-action training is to pre-extract decoded frame pixels once into a derived LanceDB table — one row per frame, with the per-frame `action` and `observation_state` already joined in, and one column per camera holding the decoded image — and train against that table with the regular projection-based dataloader. `take_blobs` is the mechanism that makes the extraction step tractable: each episode's per-camera MP4 segment is randomly addressable in `episodes.lance` (the `*_from_timestamp` / `*_to_timestamp` columns give the segment bounds), so the pass can subset bytes on demand and write decoded frames into a fresh table without an external file store. Other workflows project the `*_video_blob` columns from `episodes.lance` directly and decode at the batch boundary, or skip pixels entirely and train a state-only policy on `frames.lance` — the right shape is workload-specific. The actual training loop is the same `Permutation.identity(tbl).select_columns(...)` snippet in every case; only the source table and the column list change.
For a state-only policy, the frames table is already in the right shape — no pre-extraction needed:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/lerobot-xvla-soft-fold/data")
frames = db.open_table("frames")
train_ds = Permutation.identity(frames).select_columns(["observation_state", "action"])
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4)
```
For a vision-language-action policy, train against a pre-extracted frames-with-pixels table that joins each frame's three decoded camera images to its `action` and `observation_state`. Picking the cameras the model actually conditions on is then a column projection — `cam_high` alone, all three, or any subset:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("./lerobot-xvla-frames") # local table produced by the one-time extraction
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(
["cam_high", "cam_left_wrist", "cam_right_wrist", "observation_state", "action"]
)
loader = DataLoader(train_ds, batch_size=32, shuffle=True, num_workers=4)
```
The inline `_video_blob` storage and `take_blobs` still earn their place outside of the training loop — visualizing an episode in a notebook, sampling for human review, one-off evaluation, and the pre-extraction step itself — but they are not the dataloader.
## Versioning
Every mutation to a Lance table, whether it adds a column, merges labels, or builds an index, commits a new version. Each of `frames`, `episodes`, and `videos` is versioned independently, so a column added to `frames` does not bump the version of `episodes`. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/lerobot-xvla-soft-fold/data")
frames = db.open_table("frames")
print("frames version:", frames.version)
print("history:", frames.list_versions())
print("tags:", frames.tags.list())
```
Once you have a local copy, tag the table for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./lerobot-xvla-soft-fold/data")
local_frames = local_db.open_table("frames")
local_frames.tags.create("xvla-v1", local_frames.version)
```
Reopen by tag or by version number against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
frames_v1 = db.open_table("frames", version="xvla-v1")
frames_v5 = db.open_table("frames", version=5)
```
Pinning supports two workflows. A policy locked to `xvla-v1` keeps reproducing the same behavior while the dataset evolves in parallel. A training experiment pinned to the same tag can be rerun later against the exact same frames and segments, so changes in metrics reflect model changes rather than data drift.
## Materialize a subset
At >50 GB across three tables and millions of frames, few workflows want the full corpus on local disk. The practical entry point is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory — including the per-camera `_video_blob` columns on `episodes.lance`, which stream through Arrow record batches rather than being assembled in a single buffer.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/lerobot-xvla-soft-fold/data")
remote_episodes = remote_db.open_table("episodes")
batches = (
remote_episodes.search()
.where("task_index = 0 AND episode_index < 50")
.select([
"episode_index", "task_index", "fps", "timestamps", "actions", "observation_state",
"observation_images_cam_high_video_blob",
"observation_images_cam_high_from_timestamp",
"observation_images_cam_high_to_timestamp",
])
.to_batches()
)
local_db = lancedb.connect("./xvla-task0-subset")
local_db.create_table("episodes", batches)
```
The resulting `./xvla-task0-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/lerobot-xvla-soft-fold/data` for `./xvla-task0-subset`. The same pattern applies to `frames` and `videos` — narrow each table to the rows your workload needs, and the resulting database stays small enough to index and iterate cheaply.
## Source & license
Converted from [`lerobot/xvla-soft-fold`](https://huggingface.co/datasets/lerobot/xvla-soft-fold) (LeRobot v3.0 dataset format), originally released as part of the [X-VLA](https://thu-air-dream.github.io/X-VLA/) project. Apache 2.0.
## Citation
```
@article{zheng2025xvla,
title={X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model},
author={Zheng and others},
journal={arXiv preprint arXiv:2510.10274},
year={2025}
}
@misc{cadene2024lerobot,
title={LeRobot: State-of-the-art Machine Learning for Real-World Robotics in PyTorch},
author={R{\'e}mi Cadene and Simon Alibert and Alexander Soare and Quentin Gallou{\'e}dec and Adil Zouitine and Steven Palma and Pepijn Kooijmans and Michel Aractingi and Mustafa Shukor and Martino Russi and Francesco Capuano and Caroline Pascal and Jade Choghari and Jess Moss and Thomas Wolf},
year={2024},
url={https://github.com/huggingface/lerobot}
}
```
# LibriSpeech clean
Source: https://docs.lancedb.com/datasets/librispeech-clean
A Lance-formatted version of the LibriSpeech ASR clean configuration, sourced from openslr/librispeech_asr. Each row is one utterance with inline FLAC audio bytes, the reference transcript, a sentence-transformers embedding of that transcript, and…
Source dataset card and downloadable files for `lance-format/librispeech-clean-lance`.
A Lance-formatted version of the LibriSpeech ASR `clean` configuration, sourced from [`openslr/librispeech_asr`](https://huggingface.co/datasets/openslr/librispeech_asr). Each row is one utterance with inline FLAC audio bytes, the reference transcript, a sentence-transformers embedding of that transcript, and speaker/chapter metadata — all available directly from the Hub at `hf://datasets/lance-format/librispeech-clean-lance/data`.
## Key features
* **Inline FLAC bytes** in the `audio` column at 16 kHz mono, with no re-encoding from the upstream parquet.
* **Sentence-transformers embedding of the transcript** in `text_emb` (`all-MiniLM-L6-v2`, 384-dim, cosine-normalized) with a bundled `IVF_PQ` index for semantic transcript search.
* **Pre-built `INVERTED` FTS index on `text`** and `BTREE` indices on `id`, `speaker_id`, and `chapter_id` for keyword search and stable lookup by identifier.
* **Per-utterance metadata** — `speaker_id`, `chapter_id`, `num_chars`, `sampling_rate` — that downstream filters can stack on.
## Splits
| Split | Source config | Rows | Description |
| ----------------------- | ----------------- | ------ | ------------------------------ |
| `dev_clean.lance` | `dev.clean` | 2,703 | Standard ASR validation set |
| `test_clean.lance` | `test.clean` | 2,620 | Standard ASR test set |
| `train_clean_100.lance` | `train.clean.100` | 28,539 | 100-hour clean training subset |
> The 360-hour and 500-hour LibriSpeech subsets (`train.360`, `train.other.500`) are not bundled here. To extend, point `librispeech/dataprep.py` at additional splits.
## Schema
| Column | Type | Notes |
| --------------- | ------------------------------- | ------------------------------------------------------------ |
| `id` | `string` | Utterance id (e.g. `1272-128104-0000`) |
| `audio` | `large_binary` | Inline FLAC bytes (16 kHz mono) |
| `sampling_rate` | `int32` | Always 16,000 |
| `text` | `string` | Reference transcript |
| `speaker_id` | `int64` | LibriVox speaker id |
| `chapter_id` | `int64` | LibriVox chapter id |
| `num_chars` | `int32` | Length of `text` in characters |
| `text_emb` | `fixed_size_list` | sentence-transformers `all-MiniLM-L6-v2` (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `text_emb` — semantic transcript search (cosine)
* `INVERTED` (FTS) on `text` — keyword and hybrid search
* `BTREE` on `id`, `speaker_id`, `chapter_id` — fast lookup by identifier
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/librispeech-clean-lance", split="test_clean", streaming=True)
for row in hf_ds.take(3):
print(row["id"], row["text"][:80])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. Each `.lance` file in `data/` is a table — open by name (`dev_clean`, `test_clean`, `train_clean_100`). The same handle is used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/librispeech-clean-lance/data")
tbl = db.open_table("train_clean_100")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/librispeech-clean-lance/data/train_clean_100.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access, ANN search, and audio decoding are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/librispeech-clean-lance --repo-type dataset --local-dir ./librispeech-clean
> ```
>
> Then point Lance or LanceDB at `./librispeech-clean/data`.
## Search
The bundled `IVF_PQ` index on `text_emb` makes semantic transcript retrieval a single call. In production you would encode a query string through the same sentence-transformers model used at ingest (`all-MiniLM-L6-v2`, cosine-normalized), then pass the resulting 384-d vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/librispeech-clean-lance/data")
tbl = db.open_table("train_clean_100")
seed = (
tbl.search()
.select(["text_emb", "text"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["text_emb"], vector_column_name="text_emb")
.metric("cosine")
.select(["id", "speaker_id", "text"])
.limit(10)
.to_list()
)
print("query transcript:", seed["text"][:80])
for r in hits:
print(f" {r['id']} spk={r['speaker_id']} {r['text'][:80]}")
```
The `audio` blob is never touched. A top-10 semantic search moves a few kilobytes of transcript text rather than the FLAC bytes for every candidate.
Because the dataset also ships an `INVERTED` index on `text`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query — useful when a name or domain term must literally appear in the transcript but you still want the semantic side to rank the rest.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid", vector_column_name="text_emb")
.vector(seed["text_emb"])
.text("astronomy")
.select(["id", "speaker_id", "text"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" {r['id']} spk={r['speaker_id']} {r['text'][:80]}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
Building a focused subset of utterances usually means combining content with structure — pick utterances by a single speaker, or above a minimum transcript length, or matching a topic. Stacking predicates inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(500)` makes it cheap to inspect.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/librispeech-clean-lance/data")
tbl = db.open_table("train_clean_100")
candidates = (
tbl.search()
.where("speaker_id = 1272 AND num_chars >= 60", prefilter=True)
.select(["id", "chapter_id", "num_chars", "text"])
.limit(500)
.with_row_id(True)
.to_list()
)
print(f"{len(candidates)} utterances; first: {candidates[0]['text'][:80]}")
```
The scan never reads the `audio` column. Lance stores binary columns independently, so a metadata-only curation pass moves only the transcript text and scalar fields across the wire — even though the underlying table includes hours of inline FLAC audio.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `is_long_utterance` flag and a coarse `length_bucket`, either of which can then be used directly in `where` clauses without re-evaluating the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./librispeech-clean/data") # local copy required for writes
tbl = db.open_table("train_clean_100")
tbl.add_columns({
"is_long_utterance": "num_chars >= 200",
"length_bucket": (
"CASE WHEN num_chars < 80 THEN 'short' "
"WHEN num_chars < 200 THEN 'medium' ELSE 'long' END"
),
})
```
If the values you want to attach already live in another table (alternate transcripts, speaker embeddings, model predictions), merge them in by joining on `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"id": pa.array(["1272-128104-0000", "1272-128104-0001"]),
"wer": pa.array([0.04, 0.12]),
})
tbl.merge(predictions, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. For column values that require a Python computation (e.g., running a speaker embedding model over the FLAC bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
A common pattern for audio training is to pre-extract decoded features once into a derived LanceDB table — one row per training-ready window of log-mel frames or raw PCM samples — and train against that table with the regular projection-based dataloader. `take_blobs` is the mechanism that makes the extraction step tractable: each utterance's FLAC bytes are randomly addressable, so the pass can subset audio on demand and write decoded windows into a fresh table without an external file store. Other workflows project `audio` directly through `select_columns(...)` and decode at the batch boundary, or skip audio entirely and train on the cached transcript embeddings — the right shape is workload-specific. The actual training loop is the same `Permutation.identity(tbl).select_columns(...)` snippet in every case; only the source table and the column list change.
Against a pre-extracted features table:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("./librispeech-features") # local table produced by the one-time extraction
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["log_mel", "text", "speaker_id"])
loader = DataLoader(train_ds, batch_size=32, shuffle=True, num_workers=4)
```
Against the cached transcript embeddings on the source table (no audio decode):
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
src_db = lancedb.connect("hf://datasets/lance-format/librispeech-clean-lance/data")
src_tbl = src_db.open_table("train_clean_100")
train_ds = Permutation.identity(src_tbl).select_columns(["text_emb", "speaker_id"])
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4)
```
The inline `audio` storage and `take_blobs` still earn their place around the training process — listening back to an utterance in a notebook, sampling for human review, one-off evaluation against a held-out set, and the pre-extraction pass itself. Each of those reads a small, explicit set of blobs once. What the Train section above keeps off the per-batch hot path is exactly that raw-audio decode: paying it every step is what the pre-extracted features are designed to avoid.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/librispeech-clean-lance/data")
tbl = db.open_table("train_clean_100")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./librispeech-clean/data")
local_tbl = local_db.open_table("train_clean_100")
local_tbl.tags.create("minilm-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train_clean_100", version="minilm-v1")
tbl_v5 = db.open_table("train_clean_100", version=5)
```
Pinning supports two workflows. A retrieval system locked to `minilm-v1` keeps returning stable results while the dataset evolves in parallel. A training experiment pinned to the same tag can be rerun later against the exact same utterances, so changes in metrics reflect model changes rather than data drift.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training pipeline benefits from a local copy with fast random access to the FLAC bytes. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory — including the `audio` column, which streams through Arrow record batches rather than being assembled in a single buffer.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/librispeech-clean-lance/data")
remote_tbl = remote_db.open_table("train_clean_100")
batches = (
remote_tbl.search()
.where("speaker_id = 1272")
.select(["id", "audio", "sampling_rate", "text", "speaker_id", "chapter_id", "text_emb"])
.to_batches()
)
local_db = lancedb.connect("./librispeech-speaker-1272")
local_db.create_table("train", batches)
```
The resulting `./librispeech-speaker-1272` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/librispeech-clean-lance/data` for `./librispeech-speaker-1272`.
## Source & license
Converted from [`openslr/librispeech_asr`](https://huggingface.co/datasets/openslr/librispeech_asr). LibriSpeech is released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) and is built from the public-domain LibriVox audiobook corpus.
## Citation
```
@inproceedings{panayotov2015librispeech,
title={LibriSpeech: An ASR corpus based on public domain audiobooks},
author={Panayotov, Vassil and Chen, Guoguo and Povey, Daniel and Khudanpur, Sanjeev},
booktitle={IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
year={2015}
}
```
# MNIST
Source: https://docs.lancedb.com/datasets/mnist
A Lance-formatted version of the classic MNIST handwritten-digit dataset covering 70,000 28×28 grayscale digits across ten balanced classes. Each row carries inline PNG bytes, the digit label, the human-readable class name, and a cosine-normalized…
Source dataset card and downloadable files for `lance-format/mnist-lance`.
A Lance-formatted version of the classic [MNIST handwritten-digit dataset](https://huggingface.co/datasets/ylecun/mnist) covering 70,000 28×28 grayscale digits across ten balanced classes. Each row carries inline PNG bytes, the digit label, the human-readable class name, and a cosine-normalized CLIP image embedding, all backed by a bundled `IVF_PQ` vector index plus scalar indices on the label columns and available directly from the Hub at `hf://datasets/lance-format/mnist-lance/data`.
## Key features
* **Inline PNG bytes** in the `image` column — no sidecar files, no image folders.
* **Pre-computed CLIP image embeddings** (OpenCLIP `ViT-B-32` / `laion2b_s34b_b79k`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index.
* **Scalar indices on both label columns** — `BTREE` on `label` and `BITMAP` on `label_name` — so digit filters and digit-conditioned search are constant-time lookups.
* **One columnar dataset** — scan labels cheaply, then fetch image bytes only for the rows you want.
## Splits
| Split | Rows |
| ------------- | ------ |
| `train.lance` | 60,000 |
| `test.lance` | 10,000 |
## Schema
| Column | Type | Notes |
| ------------ | ------------------------------- | -------------------------------------------------------- |
| `id` | `int64` | Row index within the split (natural join key for merges) |
| `image` | `large_binary` | Inline PNG bytes (28×28 grayscale) |
| `label` | `int32` | Digit class id (0–9) |
| `label_name` | `string` | Human-readable class (`"0"`..`"9"`) |
| `image_emb` | `fixed_size_list` | CLIP image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `BTREE` on `label` — fast equality and range filters on the digit id
* `BITMAP` on `label_name` — fast filters across the ten class names
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable if your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/mnist-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["label"], row["label_name"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/mnist-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/mnist-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/mnist-lance --repo-type dataset --local-dir ./mnist-lance
> ```
>
> Then point Lance or LanceDB at `./mnist-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` turns nearest-neighbor lookup on the 512-d CLIP space into a single call. In production you would encode a query digit through OpenCLIP `ViT-B-32` at runtime and pass the resulting vector to `tbl.search(...)`. The example below uses the embedding already stored in row 42 as a runnable stand-in so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/mnist-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb", "label"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "label", "label_name"])
.limit(10)
.to_list()
)
print("query digit:", seed["label"])
for r in hits:
print(f" id={r['id']:>5} label={r['label']}")
```
Because the embeddings are cosine-normalized and MNIST digits cluster tightly in CLIP space, near-neighbors of a seed image are dominated by the seed's own digit class — a useful sanity check before swapping in a real query encoder. Tune `metric`, `nprobes`, and `refine_factor` to trade recall against latency.
## Curate
A typical curation pass for a digit-classification workflow narrows the table to a single digit (or a small set of confusable digits like 4/9 or 3/8) before sampling. Because both label columns are indexed, the filter resolves without scanning the embedding or image bytes; the bounded `.limit(500)` keeps the output small enough to inspect or hand off as a manifest of row ids.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/mnist-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where("label IN (4, 9)", prefilter=True)
.select(["id", "label", "label_name"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} 4/9 candidates")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `id`s, or feed into the Evolve and Train workflows below. The `image` and `image_emb` columns are never read, so the network traffic for a 500-row candidate scan is dominated by the tiny label payload.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `is_target_class` flag for binary one-vs-rest experiments and an `is_curvy_digit` flag that groups digits with curved strokes, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./mnist-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"is_target_class": "label = 7",
"is_curvy_digit": "label IN (0, 3, 6, 8, 9)",
})
```
If the values you want to attach already live in another table (offline labels from a stronger model, classifier predictions, per-row confidence scores), merge them in by joining on the `id` column:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"id": pa.array([0, 1, 2], type=pa.int64()),
"pred_label": pa.array([5, 0, 4], type=pa.int32()),
"pred_conf": pa.array([0.97, 0.88, 0.82]),
})
tbl.merge(predictions, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second image encoder over the inline PNG bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetch, shuffling, and batching behave as in any PyTorch pipeline. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/mnist-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "label"])
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb stays on disk.
# decode the PNG bytes, normalize to [0, 1], forward, backward...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "label"]` to `select_columns(...)` on the next run skips PNG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a linear probe or a lightweight reranker on top of frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/mnist-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./mnist-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel; newly added prediction columns or relabelings do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same digits and labels, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/mnist-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("label IN (4, 9)")
.select(["id", "image", "label", "label_name", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./mnist-4-vs-9")
local_db.create_table("train", batches)
```
The resulting `./mnist-4-vs-9` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/mnist-lance/data` for `./mnist-4-vs-9`.
## Source & license
Converted from [`ylecun/mnist`](https://huggingface.co/datasets/ylecun/mnist). MNIST is released under the MIT license. The original dataset is by Yann LeCun, Corinna Cortes, and Christopher J.C. Burges.
## Citation
```
@article{lecun1998mnist,
title={The MNIST database of handwritten digits},
author={LeCun, Yann and Cortes, Corinna and Burges, CJ},
url={http://yann.lecun.com/exdb/mnist/},
year={1998}
}
```
# MS MARCO v2.1
Source: https://docs.lancedb.com/datasets/ms-marco-v2
A Lance-formatted version of MS MARCO v2.1 — Microsoft's machine-reading-comprehension benchmark built from anonymized Bing query logs. Each row is one user query, the up-to-10 candidate passages Bing retrieved for it with relevance flags, and the…
Source dataset card and downloadable files for `lance-format/ms-marco-v2.1-lance`.
A Lance-formatted version of [MS MARCO v2.1](https://huggingface.co/datasets/microsoft/ms_marco) — Microsoft's machine-reading-comprehension benchmark built from anonymized Bing query logs. Each row is one user query, the up-to-10 candidate passages Bing retrieved for it with relevance flags, and the human-written reference answers, with MiniLM query embeddings stored inline and pre-built ANN/FTS indices, available directly from the Hub at `hf://datasets/lance-format/ms-marco-v2.1-lance/data`.
## Key features
* **Self-contained passage-ranking rows** — each query carries up to 10 candidate passages in parallel `passage_text` / `passage_url` / `passage_is_selected` columns, alongside the human-written `answers` and `well_formed_answers`.
* **First relevant passage promoted to its own field** in `selected_passage`, so RAG / answer-evaluation workflows can read the gold context without indexing into the parallel passage lists.
* **Pre-computed 384-dim query embeddings** (`query_emb`, `sentence-transformers/all-MiniLM-L6-v2`, cosine-normalized) with a bundled `IVF_PQ` index for semantic query lookup.
* **One columnar dataset** — scan query metadata cheaply, defer the heavy passage text reads to the rows that matter.
## Splits
| Split | Rows |
| ------------------ | ------- |
| `train.lance` | 808,731 |
| `validation.lance` | 101,093 |
## Schema
| Column | Type | Notes |
| --------------------- | ------------------------------- | --------------------------------------------------------------- |
| `query_id` | `int64` | MS MARCO query id |
| `query` | `string` | The user's natural-language query |
| `query_type` | `string` | One of `DESCRIPTION`, `NUMERIC`, `ENTITY`, `LOCATION`, `PERSON` |
| `answers` | `list` | Human-written reference answers |
| `well_formed_answers` | `list` | Reference answers re-written as full sentences |
| `passage_text` | `list` | Up to 10 candidate passages |
| `passage_url` | `list` | Source URLs for each candidate |
| `passage_is_selected` | `list` | `1` if Bing labelled the passage relevant |
| `selected_passage` | `string?` | First relevant passage (null if none) |
| `query_emb` | `fixed_size_list` | MiniLM query embedding |
## Pre-built indices
* `IVF_PQ` on `query_emb` — semantic query lookup (cosine)
* `INVERTED` (FTS) on `query` and `selected_passage` — keyword and hybrid search
* `BTREE` on `query_id` — stable lookup by identifier
* `BITMAP` on `query_type` — cheap predicate evaluation for query class
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/ms-marco-v2.1-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["query"], "->", row["answers"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. Each `.lance` file in `data/` is a table — open by name (`train`, `validation`). The same handle is used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/ms-marco-v2.1-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/ms-marco-v2.1-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/ms-marco-v2.1-lance --repo-type dataset --local-dir ./ms-marco-v2.1-lance
> ```
>
> Then point Lance or LanceDB at `./ms-marco-v2.1-lance/data`.
## Search
The bundled `IVF_PQ` index on `query_emb` makes nearest-neighbour query lookup a single call. In production you would encode an incoming user query through the same 384-dim MiniLM encoder used at ingest and pass the resulting vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in so the snippet works without loading a model.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/ms-marco-v2.1-lance/data")
tbl = db.open_table("validation")
seed = (
tbl.search()
.select(["query_emb", "query"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["query_emb"], vector_column_name="query_emb")
.metric("cosine")
.where("query_type = 'NUMERIC'", prefilter=True)
.select(["query_id", "query", "selected_passage", "answers"])
.limit(10)
.to_list()
)
for r in hits:
print(r["query"], "->", (r["selected_passage"] or "")[:120])
```
The result set carries only the projected columns; the 384-d `query_emb` is never read on the result side, and the full `passage_text` list is left untouched, keeping the working set small even when the underlying scan touches every row of the validation split.
Because the dataset also ships an `INVERTED` index on both `query` and `selected_passage`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query against the gold passage. LanceDB merges the two result lists and reranks them in a single call, which is useful when a phrase must literally appear in the relevant passage but the dense side still does most of the ranking.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid")
.vector(seed["query_emb"])
.text("determinant matrix")
.select(["query", "selected_passage", "answers"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(r["query"])
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency for your workload.
## Curate
A typical curation pass over MS MARCO starts by combining metadata filters with structural predicates over the parallel passage lists before any heavy text gets read. Lance evaluates the filter inside a single scan, so the candidate set comes back already filtered, and the bounded `.limit(1000)` keeps the output small enough to inspect. The example below assembles a set of numeric questions for which Bing labelled at least one passage relevant and the annotators produced a well-formed reference answer.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/ms-marco-v2.1-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where(
"query_type = 'NUMERIC' "
"AND selected_passage IS NOT NULL "
"AND array_length(well_formed_answers) > 0 "
"AND length(query) >= 30",
prefilter=True,
)
.select(["query_id", "query", "answers", "well_formed_answers"])
.limit(1000)
.to_list()
)
print(f"{len(candidates)} candidates; first: {candidates[0]['query']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `query_id`s, or hand to the Evolve and Train sections below. Neither `passage_text` nor `query_emb` is read by this scan, so a 1000-row curation pass against the Hub moves only kilobytes of metadata.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `query_length` column and a `num_selected` count over the parallel `passage_is_selected` list, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./ms-marco-v2.1-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"query_length": "length(query)",
"num_selected": "array_length(passage_is_selected)",
"has_well_formed": "array_length(well_formed_answers) > 0",
})
```
If the values you want to attach already live in another table (cross-encoder reranker scores, generated-answer judgments, alternate embeddings from a stronger model), merge them in by joining on `query_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
reranker_scores = pa.table({
"query_id": pa.array([1185869, 9083, 524332], type=pa.int64()),
"reranker_top1_score": pa.array([0.91, 0.47, 0.83]),
})
tbl.merge(reranker_scores, on="query_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a different encoder over the query text), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a reader-style QA model the natural projection is the query plus the gold passage and the answer; for a query-encoder retraining loop the precomputed embedding is enough on its own.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/ms-marco-v2.1-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["query", "selected_passage", "answers"])
loader = DataLoader(train_ds, batch_size=32, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; tokenize, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["query_emb", "passage_text", "passage_is_selected"]` to `select_columns(...)` on the next run reads only those columns, which is the right shape for training a passage reranker on cached query embeddings. Columns added in Evolve cost nothing per batch until they are explicitly projected.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/ms-marco-v2.1-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./ms-marco-v2.1-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("numeric-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="numeric-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `numeric-v1` keeps returning stable passages while the dataset evolves in parallel — newly added reranker scores or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same queries and passages, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/ms-marco-v2.1-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where(
"query_type = 'NUMERIC' "
"AND selected_passage IS NOT NULL "
"AND array_length(well_formed_answers) > 0"
)
.select(["query_id", "query", "query_type", "answers", "well_formed_answers", "selected_passage", "query_emb"])
.to_batches()
)
local_db = lancedb.connect("./ms-marco-numeric")
local_db.create_table("train", batches)
```
The resulting `./ms-marco-numeric` is a first-class LanceDB database. Every snippet in the Search, Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/ms-marco-v2.1-lance/data` for `./ms-marco-numeric`.
## Source & license
Converted from [`microsoft/ms_marco`](https://huggingface.co/datasets/microsoft/ms_marco) (`v2.1`). MS MARCO is released under the MIT license.
## Citation
```
@article{nguyen2016ms,
title={MS MARCO: A Human Generated MAchine Reading COmprehension Dataset},
author={Nguyen, Tri and Rosenberg, Mir and Song, Xia and Gao, Jianfeng and Tiwary, Saurabh and Majumder, Rangan and Deng, Li},
journal={arXiv preprint arXiv:1611.09268},
year={2016}
}
```
# Natural Questions Validation
Source: https://docs.lancedb.com/datasets/natural-questions-val
A Lance-formatted version of the Natural Questions validation split — 7,830 real Google search queries paired with the full Wikipedia article a human used to answer them, plus 1–5 annotator labels per question. MiniLM question embeddings are stored…
Source dataset card and downloadable files for `lance-format/natural-questions-val-lance`.
A Lance-formatted version of the [Natural Questions](https://ai.google.com/research/NaturalQuestions/) validation split — 7,830 real Google search queries paired with the full Wikipedia article a human used to answer them, plus 1–5 annotator labels per question. MiniLM question embeddings are stored inline and the dataset ships with pre-built ANN/FTS indices, all available directly from the Hub at `hf://datasets/lance-format/natural-questions-val-lance/data`. Sourced from [`google-research-datasets/natural_questions`](https://huggingface.co/datasets/google-research-datasets/natural_questions).
> The NQ **train** split is 143 GB (307,373 rows); it is intentionally not bundled here. Add it via `natural_questions/dataprep.py --splits train` once disk and bandwidth allow.
## Key features
* **Real Google search queries** with the full Wikipedia article that answers each one — `document_html` carries the inline UTF-8 HTML, so no sidecar files or external lookups are needed at query time.
* **Annotator answer summaries** — `short_answers` aggregates and dedupes spans across all annotators, `yes_no_answer` carries the majority vote, and the `has_short_answer` / `has_long_answer` flags make annotation-coverage filters a single predicate.
* **Pre-computed 384-dim question embeddings** (`question_emb`, `sentence-transformers/all-MiniLM-L6-v2`, cosine-normalized) with a bundled `IVF_PQ` index for semantic question lookup.
* **One columnar dataset** — scan question metadata cheaply, then read the heavy `document_html` only for the rows you actually want.
## Splits
| Split | Rows |
| ------------------ | ----- |
| `validation.lance` | 7,830 |
## Schema
| Column | Type | Notes |
| ------------------- | ------------------------------- | ------------------------------------------------------- |
| `id` | `string` | NQ example id |
| `question` | `string` | Original Google search query |
| `document_title` | `string` | Wikipedia article title |
| `document_url` | `string` | Wikipedia article URL |
| `document_html` | `large_binary` | Full HTML of the article (inline; UTF-8 bytes) |
| `short_answers` | `list` | Deduped short-answer spans across all annotators |
| `num_short_answers` | `int32` | Total annotator spans (incl. duplicates) |
| `has_short_answer` | `bool` | At least one annotator provided a short-answer span |
| `has_long_answer` | `bool` | At least one annotator selected a long-answer candidate |
| `yes_no_answer` | `string` | `YES` / `NO` / `NONE` — majority vote across annotators |
| `question_emb` | `fixed_size_list` | MiniLM question embedding |
## Pre-built indices
* `IVF_PQ` on `question_emb` — semantic question lookup (cosine)
* `INVERTED` (FTS) on `question` — keyword and hybrid search
* `BTREE` on `id`, `document_title` — stable lookup by identifier
* `BITMAP` on `yes_no_answer`, `has_short_answer`, `has_long_answer` — cheap predicate evaluation for annotation coverage
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/natural-questions-val-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["short_answers"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/natural-questions-val-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/natural-questions-val-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access, ANN search, and HTML decoding are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/natural-questions-val-lance --repo-type dataset --local-dir ./natural-questions-val-lance
> ```
>
> Then point Lance or LanceDB at `./natural-questions-val-lance/data`.
## Search
The bundled `IVF_PQ` index on `question_emb` makes nearest-neighbour question lookup a single call. In production you would encode an incoming user query through the same 384-dim MiniLM encoder used at ingest and pass the resulting vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in so the snippet works without loading a model.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/natural-questions-val-lance/data")
tbl = db.open_table("validation")
seed = (
tbl.search()
.select(["question_emb", "question"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="question_emb")
.metric("cosine")
.where("has_short_answer = TRUE", prefilter=True)
.select(["question", "short_answers", "document_title"])
.limit(10)
.to_list()
)
for r in hits:
print(r["question"], "->", r["short_answers"])
```
The result set carries only the projected columns; the 384-d `question_emb` is never read on the result side, and the heavy `document_html` is left untouched, keeping the working set small even though each row carries a full Wikipedia article inline.
Because the dataset also ships an `INVERTED` index on `question`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query against the question text. LanceDB merges the two result lists and reranks them in a single call, which is useful when a named entity must literally appear in the query but the dense side still does most of the ranking.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid")
.vector(seed["question_emb"])
.text("declaration of independence")
.select(["question", "short_answers", "document_title"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(r["question"])
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency for your workload.
## Curate
A typical curation pass over NQ starts with annotation-coverage filters before any HTML gets read. Lance evaluates the filter inside a single scan, so the candidate set comes back already filtered, and the bounded `.limit(500)` keeps the output small enough to inspect. The example below assembles a set of factoid questions with at least one short-answer span and a non-yes/no resolution.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/natural-questions-val-lance/data")
tbl = db.open_table("validation")
candidates = (
tbl.search()
.where(
"has_short_answer = TRUE "
"AND yes_no_answer = 'NONE' "
"AND array_length(short_answers) >= 1 "
"AND length(question) >= 30",
prefilter=True,
)
.select(["id", "question", "short_answers", "document_title", "document_url"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates; first: {candidates[0]['question']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of NQ example ids, or hand to the Evolve and Train sections below. The large `document_html` column is not read by this scan, so a 500-row curation pass against the Hub moves only kilobytes of metadata even though each row holds an entire Wikipedia article.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `question_length` column, a `first_short_answer_length` derived from the deduped span list, and an `is_factoid` flag that combines the annotation flags, any of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./natural-questions-val-lance/data") # local copy required for writes
tbl = db.open_table("validation")
tbl.add_columns({
"question_length": "length(question)",
"first_short_answer_length": "length(short_answers[1])",
"is_factoid": "has_short_answer = TRUE AND yes_no_answer = 'NONE'",
})
```
If the values you want to attach already live in another table (offline retriever scores, generated-answer judgments, alternate embeddings from a stronger model), merge them in by joining on `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
retriever_scores = pa.table({
"id": pa.array(["797803103333068850", "5225754983651766092"]),
"bm25_top1_score": pa.array([14.2, 8.7]),
})
tbl.merge(retriever_scores, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., extracting the long-answer paragraph from `document_html`), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For an open-domain QA reader the natural projection is the question plus the full document HTML and the answer spans; for a question-encoder retraining loop the precomputed embedding is enough on its own, and skipping `document_html` keeps each batch small.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/natural-questions-val-lance/data")
tbl = db.open_table("validation")
train_ds = Permutation.identity(tbl).select_columns(["question", "document_html", "short_answers"])
loader = DataLoader(train_ds, batch_size=4, shuffle=True, num_workers=2)
for batch in loader:
# batch carries only the projected columns; tokenize, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["question_emb", "short_answers"]` to `select_columns(...)` on the next run reads only the 384-d vectors and the answer spans, which is the right shape for fine-tuning a retrieval head on cached embeddings without paying for the multi-megabyte `document_html` per row. Columns added in Evolve cost nothing per batch until they are explicitly projected.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/natural-questions-val-lance/data")
tbl = db.open_table("validation")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./natural-questions-val-lance/data")
local_tbl = local_db.open_table("validation")
local_tbl.tags.create("factoid-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("validation", version="factoid-v1")
tbl_v5 = db.open_table("validation", version=5)
```
Pinning supports two workflows. A QA system locked to `factoid-v1` keeps returning stable answer spans while the dataset evolves in parallel — newly added retriever scores or labels do not change what the tag resolves to. An evaluation experiment pinned to the same tag can be rerun later against the exact same questions and articles, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/natural-questions-val-lance/data")
remote_tbl = remote_db.open_table("validation")
batches = (
remote_tbl.search()
.where(
"has_short_answer = TRUE "
"AND yes_no_answer = 'NONE' "
"AND array_length(short_answers) >= 1"
)
.select(["id", "question", "document_title", "document_url", "short_answers", "question_emb"])
.to_batches()
)
local_db = lancedb.connect("./nq-factoid")
local_db.create_table("validation", batches)
```
The resulting `./nq-factoid` is a first-class LanceDB database. Every snippet in the Search, Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/natural-questions-val-lance/data` for `./nq-factoid`. Note that this projection deliberately omits `document_html`; include it in the `.select(...)` list when the downstream task needs the article body.
## Source & license
Converted from [`google-research-datasets/natural_questions`](https://huggingface.co/datasets/google-research-datasets/natural_questions). NQ is released under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/) (matching the Wikipedia source).
## Citation
```
@article{kwiatkowski2019natural,
title={Natural Questions: A Benchmark for Question Answering Research},
author={Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-Wei and Dai, Andrew M. and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav},
journal={Transactions of the Association for Computational Linguistics},
year={2019}
}
```
# OpenVid-1M
Source: https://docs.lancedb.com/datasets/openvid
A Lance-formatted version of the OpenVid-1M corpus — 937,957 high-quality clips with inline MP4 bytes, 1024-dim video embeddings, captions, and rich per-clip quality signals — available directly from the Hub at…
Source dataset card and downloadable files for `lance-format/openvid-lance`.
A Lance-formatted version of the [OpenVid-1M](https://huggingface.co/datasets/nkp37/OpenVid-1M) corpus — **937,957 high-quality clips** with inline MP4 bytes, 1024-dim video embeddings, captions, and rich per-clip quality signals — available directly from the Hub at `hf://datasets/lance-format/openvid-lance/data/train.lance`.
## Key features
* **Inline MP4 bytes** in the `video_blob` column, stored in a side blob file and surfaced as lazy `BlobFile` handles via `take_blobs` — metadata scans, search, and filtering never read a single byte of video data.
* **Pre-computed 1024-dim video embeddings** in `embedding` with a bundled `IVF_PQ` ANN index.
* **Pre-built `INVERTED` (FTS) index on `caption`** for keyword and hybrid search.
* **Rich quality signals** — `aesthetic_score`, `motion_score`, `temporal_consistency_score`, `camera_motion`, `fps`, `seconds` — that downstream filters can stack on.
## Splits
`train.lance`
## Schema
| Column | Type | Notes |
| ---------------------------- | -------------------------------- | ------------------------------------------------------------------------------------- |
| `video_blob` | `large_binary` (blob-encoded) | Inline MP4 bytes; stored in a separate blob file and read lazily through `take_blobs` |
| `video_path` | `string` | Original file path / object key |
| `caption` | `string` | Text description of the clip |
| `embedding` | `fixed_size_list` | Video embedding |
| `aesthetic_score` | `float64` | Visual quality, roughly 0–6 |
| `motion_score` | `float64` | Amount of motion, 0–1 |
| `temporal_consistency_score` | `float64` | Frame-to-frame stability, 0–1 |
| `camera_motion` | `string` | `pan`, `zoom`, `static`, etc. |
| `fps` | `float64` | Frames per second |
| `seconds` | `float64` | Clip duration |
| `frame` | `int64` | Total frame count |
## Pre-built indices
* `IVF_PQ` on `embedding` — video similarity (L2)
* `INVERTED` (FTS) on `caption` — keyword and hybrid search
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you just want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/openvid-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["caption"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/openvid-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices — or when you need the blob-level `take_blobs` entry point that streams video bytes lazily from inline storage.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/openvid-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access, ANN search, and video decoding are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/openvid-lance --repo-type dataset --local-dir ./openvid
> ```
>
> Then point Lance or LanceDB at `./openvid/data`.
## Search
The bundled `IVF_PQ` index on `embedding` makes approximate-nearest-neighbor search a single call. In production you would encode a text prompt through a text-to-video model or a reference clip through the same video encoder used at ingest, and pass the resulting 1024-d vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/openvid-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["embedding", "caption"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["embedding"])
.metric("L2")
.select(["caption", "aesthetic_score", "camera_motion", "seconds"])
.limit(10)
.to_list()
)
for r in hits:
print(f"{r['aesthetic_score']:.2f} | {r['camera_motion']:>8} | {r['caption'][:60]}")
```
The result set carries only the projected columns. The `video_blob` column is never read, so the network traffic for a top-10 search is dominated by a few kilobytes of caption text, not by megabytes of MP4. The lazy blob fetch comes later — see Curate below.
Because OpenVid also ships an `INVERTED` index on `caption`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges the two result lists and reranks them in a single call.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid")
.vector(seed["embedding"])
.text("sunset over the ocean")
.select(["caption", "aesthetic_score", "seconds"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f"{r['aesthetic_score']:.2f} | {r['seconds']:.1f}s | {r['caption'][:60]}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
Curation for a video workflow almost always starts as a metadata filter — pick the dynamic, high-aesthetic, well-stabilized clips first, then decide what to do with the video bytes. Stacking predicates inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(200)` makes it cheap to inspect or hand off.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/openvid-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where(
"aesthetic_score >= 4.5 "
"AND motion_score >= 0.3 "
"AND temporal_consistency_score >= 0.9",
prefilter=True,
)
.select(["caption", "camera_motion", "aesthetic_score", "fps", "seconds"])
.limit(200)
.with_row_id(True)
.to_list()
)
print(f"{len(candidates)} clips selected")
```
The scan above never reads the `video_blob` column. Lance stores blobs in a separate side file referenced by the dataset, so column-projected reads skip them entirely until they are explicitly requested. That is what makes "find me the right clips" a metadata-only operation against a million-row video corpus.
Once the candidate set is fixed, pull the actual video bytes through pylance's `take_blobs`. It returns one `BlobFile` per row — a file-like handle that streams from inline blob storage on demand rather than reading the full clip into Python memory up front. For video specifically, this is the operation that matters: a video model trainer or a dataloader inspecting a few seconds of each clip should never have to materialize entire MP4s in memory just to inspect or decode part of them.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/openvid-lance/data/train.lance")
row_ids = [r["_rowid"] for r in candidates[:10]]
blob_files = ds.take_blobs("video_blob", ids=row_ids)
```
Each `BlobFile` implements the file protocol, so it can be passed straight to a decoder like PyAV without first being copied through a `bytes` object. The decoder seeks and reads against the underlying handle, which means a 2-second sample from a 30-second clip moves only the bytes the decoder actually touches — not the whole MP4.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import av
with av.open(blob_files[0]) as container:
stream = container.streams.video[0]
for seconds in (0.0, 1.0, 2.5):
target = int(seconds / stream.time_base)
container.seek(target, stream=stream)
frame = next(
(f for f in container.decode(stream) if f.time is not None and f.time >= seconds),
None,
)
if frame is not None:
print(f" seek {seconds:.1f}s -> {frame.width}x{frame.height} @ {frame.time:.2f}s")
```
If you only need the raw bytes (e.g., to persist a hand-picked subset to disk), call `.read()` on each handle. The lazy semantics are the same; `read()` simply materializes the full blob for that one row.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
for r_id, blob in zip(row_ids, blob_files):
with open(f"clip_{r_id}.mp4", "wb") as f:
f.write(blob.read())
```
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data — including the video blobs, which stay exactly where they are. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `duration_bucket` and a `is_high_quality` flag, either of which can then be used directly in `where` clauses without re-evaluating the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./openvid/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"duration_bucket": (
"CASE WHEN seconds < 5 THEN 'short' "
"WHEN seconds < 15 THEN 'medium' ELSE 'long' END"
),
"is_high_quality": (
"aesthetic_score >= 4.5 AND temporal_consistency_score >= 0.9"
),
})
```
If the values you want to attach already live in another table (offline labels, safety classifications, a second embedding from a different encoder), merge them in by joining on `video_path`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
labels = pa.table({
"video_path": pa.array(["s3://openvid/clips/00001.mp4", "s3://openvid/clips/00002.mp4"]),
"scene_label": pa.array(["beach", "city"]),
})
tbl.merge(labels, on="video_path")
```
The original columns and the `video_blob` side file are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running an alternative video encoder over the inline bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
A common pattern for video training is to pre-extract decoded frames once into a derived LanceDB table, and train against that table with the regular projection-based dataloader. `take_blobs` is the mechanism that makes the extraction step tractable: each clip's MP4 is randomly addressable, so the pass can subset bytes on demand and write decoded windows into a fresh table without an external file store. Other workflows project `video_blob` directly through `select_columns(...)` and decode at the batch boundary, or skip pixels entirely and train on the cached embeddings — the right shape is workload-specific. The actual training loop is the same `Permutation.identity(tbl).select_columns(...)` snippet in every case; only the source table and the column list change.
Against a pre-extracted frames table:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("./openvid-frames") # local table produced by the one-time extraction
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["frames", "caption", "aesthetic_score"])
loader = DataLoader(train_ds, batch_size=8, shuffle=True, num_workers=4)
```
Against the cached embeddings on the source table (no pre-extraction):
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
src_db = lancedb.connect("hf://datasets/lance-format/openvid-lance/data")
src_tbl = src_db.open_table("train")
train_ds = Permutation.identity(src_tbl).select_columns(["embedding", "caption"])
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4)
```
The inline `video_blob` storage and `take_blobs` still earn their place outside of the training loop — random-access inspection of a clip in a notebook, sampling for human review, one-off evaluation against a held-out set, and the pre-extraction step itself — but they are not the dataloader.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk, with the same blob handles still valid. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/openvid-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./openvid/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("quality-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="quality-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `quality-v1` keeps returning stable results while the dataset evolves in parallel — newly added columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same clips, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access into the blob file. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory — including the `video_blob` column, which streams through Arrow record batches rather than being assembled in a single buffer.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/openvid-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("aesthetic_score >= 4.5 AND motion_score >= 0.3")
.select(["caption", "embedding", "video_blob", "aesthetic_score", "camera_motion"])
.to_batches()
)
local_db = lancedb.connect("./openvid-subset")
local_db.create_table("train", batches)
```
The resulting `./openvid-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/openvid-lance/data` for `./openvid-subset`. The same `take_blobs` pattern from Curate also works against the local copy — and runs faster, because the blob side file is now on local disk.
## Citation
```
@article{nan2024openvid,
title={OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation},
author={Nan, Kepan and Xie, Rui and Zhou, Penghao and Fan, Tiehan and Yang, Zhenheng and Chen, Zhijie and Li, Xiang and Yang, Jian and Tai, Ying},
journal={arXiv preprint arXiv:2407.02371},
year={2024}
}
```
## License
Content inherits the original OpenVid-1M dataset license. Review the [upstream dataset card](https://huggingface.co/datasets/nkp37/OpenVid-1M) before downstream use.
# Oxford-IIIT Pet
Source: https://docs.lancedb.com/datasets/oxford-pets
A Lance-formatted version of the Oxford-IIIT Pet dataset — 7,390 cat and dog photos across 37 breeds — sourced from pcuenq/oxford-pets. Each row carries the inline JPEG bytes, the breed name, a species flag distinguishing cats from dogs, and a…
Source dataset card and downloadable files for `lance-format/oxford-pets-lance`.
A Lance-formatted version of the [Oxford-IIIT Pet](https://www.robots.ox.ac.uk/~vgg/data/pets/) dataset — 7,390 cat and dog photos across 37 breeds — sourced from [`pcuenq/oxford-pets`](https://huggingface.co/datasets/pcuenq/oxford-pets). Each row carries the inline JPEG bytes, the breed name, a species flag distinguishing cats from dogs, and a cosine-normalized CLIP image embedding, all available directly from the Hub at `hf://datasets/lance-format/oxford-pets-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files, no image folders.
* **Pre-computed CLIP image embeddings** (`image_emb`, OpenCLIP `ViT-B-32`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index for similarity search.
* **Both breed and species labels** (`label_name`, `is_dog`) so a query can target a specific breed, all dogs, or all cats by stacking simple predicates.
* **Bitmap indices on both label columns** make species- and breed-based curation a cheap predicate rather than a full scan.
## Splits
| Split | Rows | Notes |
| ------------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `train.lance` | 7,390 | The `pcuenq/oxford-pets` source mirror ships a single split; the canonical Oxford-IIIT trainval/test partition is not pre-applied here. |
## Schema
| Column | Type | Notes |
| ------------ | ------------------------------- | -------------------------------------------------------------------------------- |
| `id` | `int64` | Row index within split (natural join key for merges) |
| `image` | `large_binary` | Inline JPEG bytes (quality 92) |
| `label_name` | `string` | One of 37 breeds, underscore-spaced (`british_shorthair`, `golden_retriever`, …) |
| `is_dog` | `bool` | `true` for dog breeds, `false` for cat breeds |
| `path` | `string?` | Original filename from the source dataset |
| `image_emb` | `fixed_size_list` | OpenCLIP `ViT-B-32` image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `BITMAP` on `label_name` — fast lookup by breed
* `BITMAP` on `is_dog` — fast species filter
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/oxford-pets-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["label_name"], row["is_dog"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/oxford-pets-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/oxford-pets-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/oxford-pets-lance --repo-type dataset --local-dir ./oxford-pets-lance
> ```
>
> Then point Lance or LanceDB at `./oxford-pets-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes approximate-nearest-neighbor search a single call. In production you would encode a query photo through the same OpenCLIP `ViT-B-32` model used at ingest and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the embedding stored in row 0 as a runnable stand-in so the snippet works without a model loaded; on a clean run the first hit is expected to be the seed image itself, which is a useful sanity check on the index.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/oxford-pets-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb", "label_name", "is_dog"])
.limit(1)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "label_name", "is_dog"])
.limit(10)
.to_list()
)
print(f"seed: {seed['label_name']} (is_dog={seed['is_dog']})")
for r in hits:
print(f" {r['id']:>5} {r['label_name']:<22} is_dog={r['is_dog']}")
```
Tune `metric`, `nprobes`, and `refine_factor` to trade recall against latency for your workload.
## Curate
A typical curation pass for a fine-grained pet classifier stacks the species predicate and a breed predicate inside a single filtered scan. With bitmap indices on both `label_name` and `is_dog`, the result comes back in milliseconds, and the bounded `.limit(200)` keeps it small enough to inspect or hand off to a training run.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/oxford-pets-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where("is_dog = true AND label_name IN ('golden_retriever', 'beagle', 'pug')")
.select(["id", "label_name", "is_dog", "path"])
.limit(200)
.to_list()
)
print(f"{len(candidates)} candidates; first: {candidates[0]['label_name']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `id`s, or feed into the Evolve and Train workflows below. The `image` and `image_emb` columns are never read by this query, so the network traffic is dominated by the small label fields rather than JPEG bytes or vectors.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below derives a `species` string from the `is_dog` boolean and adds a coarse breed-group flag for terriers, either of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./oxford-pets-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"species": "CASE WHEN is_dog THEN 'dog' ELSE 'cat' END",
"is_terrier": "label_name LIKE '%terrier%'",
})
```
If the values you want to attach already live in another table (offline labels, classifier predictions, an integer class id), merge them in by joining on `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
class_ids = pa.table({
"id": pa.array([0, 1, 2]),
"label_int": pa.array([0, 0, 17]),
})
tbl.merge(class_ids, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second embedding model over the JPEG bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetch, shuffling, and batching behave as in any PyTorch pipeline. For a from-scratch breed classifier, project the JPEG bytes and the string breed label; for a linear probe on top of frozen CLIP features, swap the projection to the embedding column and skip JPEG decoding entirely.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/oxford-pets-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "label_name"])
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb stays on disk.
# decode the JPEG bytes, map label_name -> int via a class list, forward, cross-entropy...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "label_name"]` to `select_columns(...)` on the next run reads only the cached 512-d vectors and the label, which is the right shape for a linear probe or a lightweight reranker. Projecting `["image", "is_dog"]` reduces the task to binary species classification on the same data.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/oxford-pets-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./oxford-pets-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel; newly added columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and labels, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/oxford-pets-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("is_dog = true")
.select(["id", "image", "label_name", "is_dog", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./oxford-pets-dogs-subset")
local_db.create_table("train", batches)
```
The resulting `./oxford-pets-dogs-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/oxford-pets-lance/data` for `./oxford-pets-dogs-subset`.
## Source & license
Converted from [`pcuenq/oxford-pets`](https://huggingface.co/datasets/pcuenq/oxford-pets). Released under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
## Citation
```
@inproceedings{parkhi2012cats,
title={Cats and Dogs},
author={Parkhi, Omkar M. and Vedaldi, Andrea and Zisserman, Andrew and Jawahar, C. V.},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2012}
}
```
# Pascal VOC 2012 Segmentation
Source: https://docs.lancedb.com/datasets/pascal-voc-2012-segmentation
A Lance-formatted version of the Pascal VOC 2012 semantic segmentation split, sourced from nateraw/pascal-voc-2012. Each row pairs an inline JPEG image with the per-pixel PNG segmentation mask and a cosine-normalized OpenCLIP ViT-B-32 image…
Source dataset card and downloadable files for `lance-format/pascal-voc-2012-segmentation-lance`.
A Lance-formatted version of the [Pascal VOC 2012 semantic segmentation split](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/), sourced from [`nateraw/pascal-voc-2012`](https://huggingface.co/datasets/nateraw/pascal-voc-2012). Each row pairs an inline JPEG image with the per-pixel PNG segmentation mask and a cosine-normalized OpenCLIP `ViT-B-32` image embedding, so a single columnar table carries both annotation modalities and the features needed to retrieve, curate, and train against them — all available directly from the Hub at `hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data`.
## Key features
* **Inline JPEG bytes and inline PNG mask bytes in the same row** — image and per-pixel segmentation travel together with no sidecar folders or mask lookups.
* **Pre-computed CLIP image embeddings** (`image_emb`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index for visual similarity search.
* **Standard VOC class encoding** — mask pixel values are class ids in `0..20` plus `255` for void, identical to the official VOC palette.
* **One columnar dataset** — scan image-level metadata cheaply, then fetch image or mask bytes only for the rows you actually want.
The 20 Pascal VOC foreground classes are: `aeroplane`, `bicycle`, `bird`, `boat`, `bottle`, `bus`, `car`, `cat`, `chair`, `cow`, `diningtable`, `dog`, `horse`, `motorbike`, `person`, `pottedplant`, `sheep`, `sofa`, `train`, `tvmonitor`.
## Splits
| Split | Rows | Notes |
| ------------------ | ----- | ------------------------------------ |
| `train.lance` | 1,464 | Official VOC 2012 segmentation train |
| `validation.lance` | 1,449 | Official VOC 2012 segmentation val |
## Schema
| Column | Type | Notes |
| ----------- | ------------------------------- | -------------------------------------------------------------------------------- |
| `id` | `int64` | Row index within the split (natural join key for merges) |
| `image` | `large_binary` | Inline JPEG bytes |
| `mask` | `large_binary` | Inline PNG bytes — class id per pixel (0=background, 1-20=VOC classes, 255=void) |
| `image_emb` | `fixed_size_list` | OpenCLIP `ViT-B-32` image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — `metric=cosine`
> Note: the small split sizes (≤1,464 rows) sit below Lance's default partition count, so the helper falls back to a smaller `num_partitions` automatically. For higher recall, rebuild the index with `num_partitions=16` against a local copy.
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/pascal-voc-2012-segmentation-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["id"], len(row["image"]), len(row["mask"]))
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/pascal-voc-2012-segmentation-lance --repo-type dataset --local-dir ./pascal-voc-2012-segmentation-lance
> ```
>
> Then point Lance or LanceDB at `./pascal-voc-2012-segmentation-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes visual nearest-neighbour search a single call. In production you would encode a query image (or a class prototype) through OpenCLIP `ViT-B-32` at runtime and pass the resulting 512-d cosine-normalized vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id"])
.limit(10)
.to_list()
)
for r in hits:
print(r["id"])
```
Because the embeddings are cosine-normalized, `metric="cosine"` is the natural choice and the first hit is typically the seed row itself — a useful sanity check before tuning `nprobes` and `refine_factor` for recall.
## Curate
A typical curation pass for a segmentation workflow combines visual similarity with a structural filter on the row. Stacking both inside a single filtered scan keeps the candidate set small and explicit, and the bounded `.limit(200)` makes it cheap to inspect before committing to anything downstream. The snippet below seeds from row 42 and restricts the candidates to rows whose mask payload is non-trivially sized — a cheap proxy for masks that actually carry foreground annotation.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb"])
.limit(1)
.offset(42)
.to_list()[0]
)
candidates = (
tbl.search(seed["image_emb"])
.metric("cosine")
.where("octet_length(mask) > 2000", prefilter=True)
.select(["id"])
.limit(200)
.to_list()
)
print(f"{len(candidates)} candidates")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `id`s, or feed into the Evolve and Train workflows below. The `image` and `mask` columns are never read in the candidate scan, so the network traffic stays dominated by the embedding vectors rather than image or mask bytes.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `image_bytes` size and a `has_mask` flag, both of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./pascal-voc-2012-segmentation-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"image_bytes": "octet_length(image)",
"has_mask": "octet_length(mask) > 1024",
})
```
For class-level statistics — for example, a per-row list of class ids present in the mask, or a per-class pixel count — the values cannot be derived in SQL because they require decoding the PNG. Compute them once in an external table and join in by `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
class_stats = pa.table({
"id": pa.array([0, 1, 2], type=pa.int64()),
"classes_present": pa.array([[15], [7, 15], [9]], type=pa.list_(pa.int8())),
})
tbl.merge(class_stats, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require running a model over the image bytes (a second-pass embedding, an instance segmentation, a depth prediction), Lance also provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a segmentation run, project the JPEG bytes and the PNG mask bytes together; everything else, including the CLIP embeddings, stays on disk until you opt in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "mask"])
loader = DataLoader(train_ds, batch_size=16, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb stays on disk.
# decode the JPEGs and PNGs, build (image, label) tensors, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["image_emb"]` to `select_columns(...)` on the next run skips JPEG and PNG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a lightweight linear probe over frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges class statistics, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./pascal-voc-2012-segmentation-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("voc2012-clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="voc2012-clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `voc2012-clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel — newly added class statistics or alternative embeddings do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same image/mask pairs, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("octet_length(mask) > 2000")
.select(["id", "image", "mask", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./voc-subset")
local_db.create_table("train", batches)
```
The resulting `./voc-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/pascal-voc-2012-segmentation-lance/data` for `./voc-subset`.
## Source & license
Converted from [`nateraw/pascal-voc-2012`](https://huggingface.co/datasets/nateraw/pascal-voc-2012). The Pascal VOC dataset is released under [its own custom license](http://host.robots.ox.ac.uk/pascal/VOC/) — please review before redistribution.
## Citation
```
@misc{everingham2012pascal,
title={The Pascal Visual Object Classes Challenge: A Retrospective},
author={Everingham, Mark and Eslami, S. M. Ali and Van Gool, Luc and Williams, Christopher K. I. and Winn, John and Zisserman, Andrew},
journal={International Journal of Computer Vision},
year={2015}
}
```
# SQuAD v2
Source: https://docs.lancedb.com/datasets/squad-v2
A Lance-formatted version of SQuAD v2 — the Stanford Question Answering Dataset with both answerable and deliberately unanswerable questions over Wikipedia passages — with MiniLM question embeddings stored inline and ready for retrieval at…
Source dataset card and downloadable files for `lance-format/squad-v2-lance`.
A Lance-formatted version of [SQuAD v2](https://huggingface.co/datasets/rajpurkar/squad_v2) — the Stanford Question Answering Dataset with both answerable and deliberately unanswerable questions over Wikipedia passages — with MiniLM question embeddings stored inline and ready for retrieval at `hf://datasets/lance-format/squad-v2-lance/data`.
## Key features
* **Span-extraction QA over Wikipedia** with 130k+ training questions and an `is_impossible` flag that cleanly separates answerable from unanswerable items.
* **Pre-computed 384-dim question embeddings** (`question_emb`, `sentence-transformers/all-MiniLM-L6-v2`, cosine-normalized) with a bundled `IVF_PQ` index for semantic question retrieval.
* **Full-text inverted indices** on both `question` and `context` for keyword search alongside dense retrieval.
* **One columnar dataset** carrying questions, contexts, answer spans, and embeddings together — project only the columns each query needs.
## Splits
| Split | Rows |
| ------------------ | ------- |
| `train.lance` | 130,319 |
| `validation.lance` | 11,873 |
## Schema
| Column | Type | Notes |
| --------------- | ------------------------------- | ------------------------------------------------------ |
| `id` | `string` | SQuAD question id (natural join key for merges) |
| `title` | `string` | Wikipedia article title |
| `context` | `string` | Paragraph the question was generated from |
| `question` | `string` | The question text |
| `answers` | `list` | Accepted answer spans (empty for impossible questions) |
| `answer_starts` | `list` | Character offsets of each answer within `context` |
| `is_impossible` | `bool` | `true` for SQuAD 2.0 unanswerable questions |
| `question_emb` | `fixed_size_list` | MiniLM embedding of `question` (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `question_emb` — `metric=cosine`, vector similarity search
* `INVERTED` on `question` and `context` — full-text search
* `BTREE` on `id` and `title` — point lookups and prefix scans
* `BITMAP` on `is_impossible` — fast filtering between answerable and unanswerable
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/squad-v2-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["answers"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/squad-v2-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/squad-v2-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/squad-v2-lance --repo-type dataset --local-dir ./squad-v2-lance
> ```
>
> Then point Lance or LanceDB at `./squad-v2-lance/data`.
## Search
The bundled `IVF_PQ` index on `question_emb` turns semantic question retrieval into a single call. In production you would encode an incoming question through the same MiniLM encoder used at ingest and pass the resulting 384-dim vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in, then restricts the result to answerable items so the response always carries a usable span.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/squad-v2-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["question_emb", "question"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="question_emb")
.metric("cosine")
.where("is_impossible = false", prefilter=True)
.select(["id", "title", "question", "answers"])
.limit(10)
.to_list()
)
for r in hits:
print(f"{r['title']:30s} | {r['question'][:80]}")
```
Because the recommended setup also builds an `INVERTED` index on both `question` and `context`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges and reranks the two result lists in a single call, which is useful when a literal phrase must appear in the passage but the dense side should still drive ranking.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid")
.vector(seed["question_emb"])
.text("eiffel tower")
.where("is_impossible = false", prefilter=True)
.select(["id", "title", "question", "context", "answers"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f"{r['title']:30s} | {r['question'][:80]}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
SQuAD v2 has a natural split between answerable and unanswerable questions, and the `is_impossible` boolean — backed by a `BITMAP` index — makes either subset cheap to extract. Stacking predicates inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(1000)` makes it easy to inspect or hand off.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/squad-v2-lance/data")
tbl = db.open_table("train")
impossible = (
tbl.search()
.where("is_impossible = true AND length(question) >= 40", prefilter=True)
.select(["id", "title", "question", "context"])
.limit(1000)
.to_list()
)
print(f"{len(impossible)} hard unanswerable questions; first title: {impossible[0]['title']}")
```
The mirror query — long, well-grounded answerable questions — looks identical with the boolean flipped, and the `question_emb` vector is never read by either scan. The result is a plain list of dictionaries, ready to inspect, persist as a manifest of question ids, or hand to the Materialize-a-subset section below for export to a writable local copy.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `question_length`, a `num_answers` count, and a `has_answer` flag — any of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./squad-v2-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"question_length": "length(question)",
"num_answers": "array_length(answers)",
"has_answer": "NOT is_impossible",
})
```
If the values you want to attach already live in another table (offline reader-model predictions, alternate embeddings, span-level labels), merge them in by joining on `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
scores = pa.table({
"id": pa.array(["56be4db0acb8001400a502ec", "56be4db0acb8001400a502ed"]),
"reader_score": pa.array([0.91, 0.42]),
})
tbl.merge(scores, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a different embedding model over the questions), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a reading-comprehension model the natural projection is the question, the context, and the answer spans together; for a retriever or reranker on top of frozen features, project the precomputed embedding instead.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/squad-v2-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(
["question", "context", "answers", "answer_starts", "is_impossible"]
)
loader = DataLoader(train_ds, batch_size=32, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; question_emb stays on disk.
# tokenize question+context, build span labels from answer_starts, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["question_emb"]` (optionally with `["answers"]` for hard-negative mining) to `select_columns(...)` on the next run reads only the 384-d vectors and skips the bulky `context` strings entirely, which is the right shape for training a retrieval head or reranker on cached embeddings. Columns added in Evolve cost nothing per batch until they are explicitly projected.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/squad-v2-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./squad-v2-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("baseline-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="baseline-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `baseline-v1` keeps returning stable results while the dataset evolves in parallel — newly added reader scores or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same questions and contexts, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/squad-v2-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("is_impossible = false AND length(question) >= 30")
.select(["id", "title", "context", "question", "answers", "answer_starts", "question_emb"])
.to_batches()
)
local_db = lancedb.connect("./squad-v2-answerable")
local_db.create_table("train", batches)
```
The resulting `./squad-v2-answerable` is a first-class LanceDB database. Every snippet in the Search, Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/squad-v2-lance/data` for `./squad-v2-answerable`.
## Source & license
Converted from [`rajpurkar/squad_v2`](https://huggingface.co/datasets/rajpurkar/squad_v2). SQuAD v2 is distributed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
## Citation
```
@article{rajpurkar2018know,
title={Know What You Don't Know: Unanswerable Questions for SQuAD},
author={Rajpurkar, Pranav and Jia, Robin and Liang, Percy},
journal={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers)},
year={2018}
}
```
# Stanford Cars
Source: https://docs.lancedb.com/datasets/stanford-cars
A Lance-formatted version of the Stanford Cars fine-grained benchmark — 8,144 photographs across 196 make/model/year classes — sourced from Multimodal-Fatima/StanfordCars_train. Each row carries the inline JPEG bytes, the integer class id, a…
Source dataset card and downloadable files for `lance-format/stanford-cars-lance`.
A Lance-formatted version of the [Stanford Cars](https://web.archive.org/web/20210212183835/http://ai.stanford.edu/~jkrause/cars/car_dataset.html) fine-grained benchmark — 8,144 photographs across 196 make/model/year classes — sourced from [`Multimodal-Fatima/StanfordCars_train`](https://huggingface.co/datasets/Multimodal-Fatima/StanfordCars_train). Each row carries the inline JPEG bytes, the integer class id, a BLIP-generated caption inherited from the source mirror, and a cosine-normalized CLIP image embedding, all available directly from the Hub at `hf://datasets/lance-format/stanford-cars-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files, no image folders.
* **Pre-computed CLIP image embeddings** (`image_emb`, OpenCLIP `ViT-B-32`, 512-dim, cosine-normalized) with a bundled `IVF_PQ` index for similarity search.
* **BLIP captions in `blip_caption`** with a full-text index, so keyword search on visual descriptions composes with vector search in a single query.
* **A bundled scalar index on `label`** makes class-based curation a cheap predicate rather than a full scan.
## Splits
| Split | Rows | Notes |
| ------------- | ----- | ----------------------------------------------------------------------------------------------------------- |
| `train.lance` | 8,144 | The source mirror redistributes a single split; the original Stanford Cars test split is not included here. |
## Schema
| Column | Type | Notes |
| -------------- | ------------------------------- | ---------------------------------------------------------------------- |
| `id` | `int64` | Row index within split (natural join key for merges) |
| `image` | `large_binary` | Inline JPEG bytes (quality 92) |
| `label` | `int32` | Class id (0–195), one per Make Model Year combination |
| `blip_caption` | `string?` | BLIP-generated caption (beam=5) carried through from the source mirror |
| `image_emb` | `fixed_size_list` | OpenCLIP `ViT-B-32` image embedding (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `image_emb` — vector similarity search (cosine)
* `INVERTED` (FTS) on `blip_caption` — keyword and hybrid search
* `BTREE` on `label` — fast lookup by class id
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/stanford-cars-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["label"], row["blip_caption"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Train, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/stanford-cars-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/stanford-cars-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/stanford-cars-lance --repo-type dataset --local-dir ./stanford-cars-lance
> ```
>
> Then point Lance or LanceDB at `./stanford-cars-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes approximate-nearest-neighbor search a single call. In production you would encode a query photo through the same OpenCLIP `ViT-B-32` model used at ingest and pass the resulting 512-d vector to `tbl.search(...)`. The example below uses the embedding stored in row 0 as a runnable stand-in so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/stanford-cars-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["image_emb", "blip_caption"])
.limit(1)
.to_list()[0]
)
hits = (
tbl.search(seed["image_emb"])
.metric("cosine")
.select(["id", "label", "blip_caption"])
.limit(10)
.to_list()
)
print("seed caption:", seed["blip_caption"])
for r in hits:
print(f" {r['id']:>6} label={r['label']:>3} {r['blip_caption'][:60]}")
```
Tune `metric`, `nprobes`, and `refine_factor` to trade recall against latency for your workload.
Because the dataset also ships an `INVERTED` index on `blip_caption`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges the two result lists and reranks them in a single call, which is useful when a phrase like "red convertible" must literally appear in the caption but you still want CLIP to do the heavy lifting on visual similarity.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid")
.vector(seed["image_emb"])
.text("red convertible")
.select(["id", "label", "blip_caption"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" {r['id']:>6} label={r['label']:>3} {r['blip_caption'][:60]}")
```
## Curate
A typical curation pass for a fine-grained classifier combines a class-based filter with a content filter on the caption. Stacking both inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(200)` makes it cheap to inspect before committing the subset to anything downstream. The `BTREE` on `label` and the `INVERTED` index on `blip_caption` make both predicates effectively free.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/stanford-cars-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search("convertible OR coupe")
.where("label IN (12, 47, 89)", prefilter=True)
.select(["id", "label", "blip_caption"])
.limit(200)
.to_list()
)
print(f"{len(candidates)} candidates; first: {candidates[0]['blip_caption'][:80]}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `id`s, or feed into the Evolve and Train workflows below. The `image` column is never read, so the network traffic for a 200-row candidate scan is dominated by the small caption strings rather than JPEG bytes.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. Stanford Cars class strings often encode the model year as a trailing four-digit token in the caption; the example below uses a SQL regex to lift that year into its own column, and adds a flag for vintage cars. Either can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./stanford-cars-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"caption_year": "CAST(regexp_extract(blip_caption, '(\\d{4})', 1) AS INTEGER)",
"is_long_caption": "length(blip_caption) >= 80",
})
```
If the values you want to attach already live in another table (offline labels, classifier predictions, the Make Model Year strings from the original Stanford metadata), merge them in by joining on `id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
class_strings = pa.table({
"id": pa.array([0, 1, 2]),
"class_name": pa.array([
"AM General Hummer SUV 2000",
"Acura RL Sedan 2012",
"Acura TL Sedan 2012",
]),
})
tbl.merge(class_strings, on="id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a second captioner over the image bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern.
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetch, shuffling, and batching behave as in any PyTorch pipeline. For a from-scratch fine-grained classifier, project the JPEG bytes and the integer label; for a linear probe on top of frozen CLIP features, swap the projection to the embedding column and skip JPEG decoding entirely.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/stanford-cars-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "label"])
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; image_emb and blip_caption stay on disk.
# decode the JPEG bytes, forward, cross-entropy against `label`...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "label"]` to `select_columns(...)` on the next run reads only the cached 512-d vectors and the label, which is the right shape for a linear probe or a lightweight reranker.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/stanford-cars-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./stanford-cars-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("clip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="clip-vitb32-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `clip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel; newly added columns or captions do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images and labels, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/stanford-cars-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search("convertible OR coupe")
.select(["id", "image", "label", "blip_caption", "image_emb"])
.to_batches()
)
local_db = lancedb.connect("./stanford-cars-sports-subset")
local_db.create_table("train", batches)
```
The resulting `./stanford-cars-sports-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/stanford-cars-lance/data` for `./stanford-cars-sports-subset`.
## Source & license
Converted from [`Multimodal-Fatima/StanfordCars_train`](https://huggingface.co/datasets/Multimodal-Fatima/StanfordCars_train), itself a redistribution of the Stanford Cars dataset. The original dataset license is for non-commercial research use; review the [Stanford Cars terms](https://github.com/jhoffman/stanford-cars) before redistribution.
## Citation
```
@inproceedings{krause2013collecting,
title={Collecting a large-scale dataset of fine-grained cars},
author={Krause, Jonathan and Stark, Michael and Deng, Jia and Fei-Fei, Li},
booktitle={Workshop on Fine-Grained Visual Categorization (CVPR)},
year={2013}
}
```
# TextVQA
Source: https://docs.lancedb.com/datasets/textvqa
A Lance-formatted version of TextVQA — visual question answering where the question requires reading text in the image (street signs, product labels, screen captures) — sourced from lmms-lab/textvqa. Each row carries the image bytes, the question…
Source dataset card and downloadable files for `lance-format/textvqa-lance`.
A Lance-formatted version of [TextVQA](https://textvqa.org/) — visual question answering where the question requires *reading* text in the image (street signs, product labels, screen captures) — sourced from [`lmms-lab/textvqa`](https://huggingface.co/datasets/lmms-lab/textvqa). Each row carries the image bytes, the question, the 10 reference annotator answers, the OCR tokens detected by the source pre-processing, OpenImages-style scene tags, and paired CLIP image and question embeddings — all available directly from the Hub at `hf://datasets/lance-format/textvqa-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files or image folders.
* **Paired CLIP embeddings in the same row** — `image_emb` and `question_emb` (OpenCLIP ViT-B/32, 512-dim, cosine-normalized) — so cross-modal text→image retrieval is one indexed lookup.
* **OCR tokens travel with the row** in `ocr_tokens`, which makes OCR-aware filtering and reranking a single SQL predicate alongside the visual and textual features.
* **Pre-built ANN, FTS, and scalar indices** covering both embeddings, the question and canonical answer, and the source partition.
## Splits
| Split | Rows |
| ------------------ | ------ |
| `validation.lance` | 5,000 |
| `train.lance` | 34,602 |
## Schema
| Column | Type | Notes |
| --------------- | ------------------------------- | ------------------------------------------------------- |
| `id` | `int64` | Row index within split |
| `image` | `large_binary` | Inline JPEG bytes |
| `image_id` | `string?` | TextVQA image id |
| `question_id` | `string?` | TextVQA question id |
| `question` | `string` | The question text |
| `answers` | `list` | 10 annotator answers |
| `answer` | `string` | First annotator answer (used as canonical / FTS target) |
| `ocr_tokens` | `list` | OCR tokens detected on the image |
| `image_classes` | `list` | OpenImages-style scene tags from the source |
| `set_name` | `string?` | Source partition (`train`, `val`) |
| `image_emb` | `fixed_size_list` | OpenCLIP image embedding (cosine-normalized) |
| `question_emb` | `fixed_size_list` | OpenCLIP text embedding of the question |
## Pre-built indices
* `IVF_PQ` on `image_emb` — image-side vector search (cosine)
* `IVF_PQ` on `question_emb` — question-side vector search (cosine)
* `INVERTED` (FTS) on `question` and `answer` — keyword and hybrid search
* `BTREE` on `image_id`, `question_id`, `set_name` — fast lookup by id and partition
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/textvqa-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["answer"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/textvqa-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/textvqa-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/textvqa-lance --repo-type dataset --local-dir ./textvqa-lance
> ```
>
> Then point Lance or LanceDB at `./textvqa-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes cross-modal text→image retrieval a single call: encode a question with the same OpenCLIP model used at ingest (ViT-B/32 `laion2b_s34b_b79k`, cosine-normalized), then pass the resulting 512-d vector to `tbl.search(...)` and target `image_emb`. The example below uses the `question_emb` already stored in row 42 as a runnable stand-in for "the CLIP encoding of a question", so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/textvqa-lance/data")
tbl = db.open_table("validation")
seed = (
tbl.search()
.select(["question_emb", "question", "answer"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="image_emb")
.metric("cosine")
.select(["image_id", "question", "answer", "ocr_tokens"])
.limit(10)
.to_list()
)
print("query question:", seed["question"], "->", seed["answer"])
for r in hits:
print(f" {r['image_id']:>14} {r['question'][:60]} ocr={r['ocr_tokens'][:5]}")
```
Because the CLIP embeddings are cosine-normalized, cosine is the right metric and the first hit will often be the source row itself. Swap `vector_column_name="image_emb"` for `question_emb` to find paraphrased questions instead of visually similar images.
Because the dataset also ships an `INVERTED` index on `question` and `answer`, the same query can be issued as a hybrid search that combines the dense vector with a literal keyword match. This is particularly useful for TextVQA, where a brand name or sign content like "stop" must literally appear in the question (or eventually, the answer) while CLIP handles the visual side.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid", vector_column_name="image_emb")
.vector(seed["question_emb"])
.text("brand name")
.select(["image_id", "question", "answer", "ocr_tokens"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" {r['image_id']:>14} {r['question'][:60]} -> {r['answer']}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
Curation passes for TextVQA usually combine an OCR-driven structural filter (does this image actually contain a meaningful amount of detected text?) with a content predicate on the question or the canonical answer, so the candidate set is both visually interesting and topically focused. Stacking both inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(500)` makes it cheap to inspect before committing the subset to anything downstream.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/textvqa-lance/data")
tbl = db.open_table("validation")
candidates = (
tbl.search()
.where(
"array_length(ocr_tokens) >= 5 AND question LIKE '%brand%' AND length(answer) > 0",
prefilter=True,
)
.select(["question_id", "image_id", "question", "answer", "ocr_tokens"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} candidates; first: {candidates[0]['question']} -> {candidates[0]['answer']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `question_id`s, or feed into the Evolve and Train workflows below. The `image` column is never read, so the network traffic for a 500-row candidate scan is dominated by question, answer, and OCR-token strings rather than JPEG bytes.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `ocr_token_count`, an `is_yes_no_question` flag, and an `answer_length` integer, any of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./textvqa-lance/data") # local copy required for writes
tbl = db.open_table("validation")
tbl.add_columns({
"ocr_token_count": "array_length(ocr_tokens)",
"is_yes_no_question": "lower(answer) IN ('yes', 'no')",
"answer_length": "length(answer)",
"question_length": "length(question)",
})
```
If the values you want to attach already live in another table (a stronger OCR system's tokens, a model's predicted answer, an annotator-disagreement score), merge them in by joining on `question_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"question_id": pa.array(["34602", "34603"]),
"model_answer": pa.array(["pepsi", "exit"]),
"model_confidence": pa.array([0.88, 0.72]),
})
tbl.merge(predictions, on="question_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running an alternate OCR engine over the image bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a TextVQA fine-tune that needs OCR conditioning, project the JPEG bytes, the question, the OCR tokens, and the canonical answer; columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/textvqa-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(["image", "question", "ocr_tokens", "answer"])
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; decode the JPEG bytes,
# tokenize the question and OCR tokens, forward through the VLM,
# compute the loss against `answer`...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "question_emb", "answer"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a lightweight reranker or a linear probe on top of frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/textvqa-lance/data")
tbl = db.open_table("validation")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./textvqa-lance/data")
local_tbl = local_db.open_table("validation")
local_tbl.tags.create("openclip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("validation", version="openclip-vitb32-v1")
tbl_v5 = db.open_table("validation", version=5)
```
Pinning supports two workflows. A retrieval system locked to `openclip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel — newly added OCR systems or model predictions do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images, questions, and OCR tokens, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/textvqa-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("array_length(ocr_tokens) >= 5")
.select(["question_id", "image_id", "image", "question", "answer", "ocr_tokens", "image_emb", "question_emb"])
.to_batches()
)
local_db = lancedb.connect("./textvqa-ocr-rich")
local_db.create_table("train", batches)
```
The resulting `./textvqa-ocr-rich` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/textvqa-lance/data` for `./textvqa-ocr-rich`.
## Source & license
Converted from [`lmms-lab/textvqa`](https://huggingface.co/datasets/lmms-lab/textvqa). TextVQA is released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) by Singh et al. (Facebook AI Research).
## Citation
```
@inproceedings{singh2019towards,
title={Towards VQA models that can read},
author={Singh, Amanpreet and Natarajan, Vivek and Shah, Meet and Jiang, Yu and Chen, Xinlei and Batra, Dhruv and Parikh, Devi and Rohrbach, Marcus},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2019}
}
```
# TriviaQA
Source: https://docs.lancedb.com/datasets/trivia-qa
A Lance-formatted version of TriviaQA (rc.nocontext config) — a large reading-comprehension dataset of trivia questions paired with a canonical answer, accepted aliases, and entity-type metadata — with MiniLM question embeddings stored inline and…
Source dataset card and downloadable files for `lance-format/trivia-qa-lance`.
A Lance-formatted version of [TriviaQA](https://nlp.cs.washington.edu/triviaqa/) (`rc.nocontext` config) — a large reading-comprehension dataset of trivia questions paired with a canonical answer, accepted aliases, and entity-type metadata — with MiniLM question embeddings stored inline and ready for retrieval at `hf://datasets/lance-format/trivia-qa-lance/data`. The `rc.nocontext` slice is the standard reading-comprehension form without the multi-gigabyte `entity_pages` / `search_results` payloads, which keeps the dataset compact while preserving everything needed for closed-book QA, retrieval research, and as a search target.
## Key features
* **138k+ trivia questions** with a canonical `answer_value`, normalized form for exact-match scoring, and a list of accepted `answer_aliases`.
* **Pre-computed 384-dim question embeddings** (`question_emb`, `sentence-transformers/all-MiniLM-L6-v2`, cosine-normalized) with a bundled `IVF_PQ` index for semantic question retrieval.
* **Full-text inverted index** on `question` for keyword search and hybrid retrieval.
* **One columnar dataset** carrying questions, canonical answers, aliases, types, and embeddings together — project only the columns each query needs.
## Splits
| Split | Rows |
| ------------------ | ------- |
| `train.lance` | 138,384 |
| `validation.lance` | 17,944 |
## Schema
| Column | Type | Notes |
| ------------------- | ------------------------------- | ---------------------------------------------------------------------------- |
| `question_id` | `string` | TriviaQA question id (e.g. `tc_1`); natural join key for merges |
| `question` | `string` | The trivia question |
| `question_source` | `string` | URL or source the question came from |
| `answer_value` | `string` | Canonical answer |
| `answer_aliases` | `list` | Other accepted phrasings (e.g. `["Sinclair Lewis", "Harry Sinclair Lewis"]`) |
| `normalized_answer` | `string` | Lowercased / normalized form for exact-match scoring |
| `answer_type` | `string` | TriviaQA entity type (e.g. `WikipediaEntity`, `FreebaseEntity`) |
| `question_emb` | `fixed_size_list` | MiniLM embedding of `question` (cosine-normalized) |
## Pre-built indices
* `IVF_PQ` on `question_emb` — `metric=cosine`, vector similarity search
* `INVERTED` on `question` — full-text search
* `BTREE` on `question_id` and `answer_value` — point lookups and prefix scans
* `BITMAP` on `answer_type` — fast filtering by entity type
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/trivia-qa-lance", split="train", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["answer_value"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/trivia-qa-lance/data")
tbl = db.open_table("train")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/trivia-qa-lance/data/train.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/trivia-qa-lance --repo-type dataset --local-dir ./trivia-qa-lance
> ```
>
> Then point Lance or LanceDB at `./trivia-qa-lance/data`.
## Search
The bundled `IVF_PQ` index on `question_emb` turns semantic retrieval over trivia questions into a single call. In production you would encode an incoming question through the same MiniLM encoder used at ingest and pass the resulting 384-dim vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/trivia-qa-lance/data")
tbl = db.open_table("train")
seed = (
tbl.search()
.select(["question_emb", "question"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="question_emb")
.metric("cosine")
.select(["question_id", "question", "answer_value", "answer_aliases"])
.limit(10)
.to_list()
)
for r in hits:
print(f"{r['answer_value']:30s} | {r['question'][:80]}")
```
Because the recommended setup also builds an `INVERTED` index on `question`, the same query can be issued as a hybrid search that combines the dense vector with a keyword query. LanceDB merges and reranks the two result lists in a single call, which is useful when a specific named entity must literally appear in the question but the dense side should still drive ranking.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid")
.vector(seed["question_emb"])
.text("sistine chapel")
.select(["question_id", "question", "answer_value", "answer_aliases"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f"{r['answer_value']:30s} | {r['question'][:80]}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
TriviaQA's `answer_type` column — backed by a `BITMAP` index — makes it cheap to slice the dataset by entity category, and the question text itself is a useful predicate for filtering out very short or unusually long items. Stacking predicates inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(1000)` makes it easy to inspect or hand off.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/trivia-qa-lance/data")
tbl = db.open_table("train")
candidates = (
tbl.search()
.where(
"answer_type = 'WikipediaEntity' "
"AND length(question) BETWEEN 60 AND 300",
prefilter=True,
)
.select(["question_id", "question", "answer_value", "answer_aliases"])
.limit(1000)
.to_list()
)
print(f"{len(candidates)} candidates; first answer: {candidates[0]['answer_value']}")
```
Neither the `question_emb` vector nor the unused alias fields drive this scan, so a 1000-row curation pass against the Hub moves only the projected text columns. The result is a plain list of dictionaries, ready to inspect, persist as a manifest of question ids, or hand to the Materialize-a-subset section below for export to a writable local copy.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a `question_length`, a `num_aliases` count, and a `has_aliases` flag — any of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./trivia-qa-lance/data") # local copy required for writes
tbl = db.open_table("train")
tbl.add_columns({
"question_length": "length(question)",
"num_aliases": "array_length(answer_aliases)",
"has_aliases": "array_length(answer_aliases) > 0",
})
```
If the values you want to attach already live in another table (offline reader-model predictions, alternate embeddings, retrieval scores from a different system), merge them in by joining on `question_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
scores = pa.table({
"question_id": pa.array(["tc_1", "tc_2"]),
"retriever_score": pa.array([0.88, 0.31]),
})
tbl.merge(scores, on="question_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running a different embedding model over the questions), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a closed-book QA model the natural projection is the question, the canonical answer, and the alias list (the aliases serve as additional supervision targets during loss computation or evaluation); for a retriever or reranker on top of frozen features, project the precomputed embedding instead.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/trivia-qa-lance/data")
tbl = db.open_table("train")
train_ds = Permutation.identity(tbl).select_columns(
["question", "answer_value", "answer_aliases"]
)
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; question_emb stays on disk.
# tokenize question and answer, forward, backward...
...
```
Switching feature sets is a configuration change: passing `["question_emb", "answer_value"]` to `select_columns(...)` on the next run reads only the 384-d vectors and the canonical answer string, which is the right shape for training a retrieval head or reranker on cached embeddings. Columns added in Evolve cost nothing per batch until they are explicitly projected.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/trivia-qa-lance/data")
tbl = db.open_table("train")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./trivia-qa-lance/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("baseline-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("train", version="baseline-v1")
tbl_v5 = db.open_table("train", version=5)
```
Pinning supports two workflows. A retrieval system locked to `baseline-v1` keeps returning stable results while the dataset evolves in parallel — newly added scores or alternate embeddings do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same questions and answers, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/trivia-qa-lance/data")
remote_tbl = remote_db.open_table("train")
batches = (
remote_tbl.search()
.where("answer_type = 'WikipediaEntity' AND length(question) >= 60")
.select(
["question_id", "question", "answer_value", "answer_aliases",
"normalized_answer", "answer_type", "question_emb"]
)
.to_batches()
)
local_db = lancedb.connect("./trivia-qa-wiki")
local_db.create_table("train", batches)
```
The resulting `./trivia-qa-wiki` is a first-class LanceDB database. Every snippet in the Search, Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/trivia-qa-lance/data` for `./trivia-qa-wiki`.
## Source & license
Converted from [`mandarjoshi/trivia_qa`](https://huggingface.co/datasets/mandarjoshi/trivia_qa) (`rc.nocontext` config). TriviaQA is released under the Apache 2.0 license.
## Citation
```
@article{joshi2017triviaqa,
title={TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension},
author={Joshi, Mandar and Choi, Eunsol and Weld, Daniel S and Zettlemoyer, Luke},
journal={arXiv preprint arXiv:1705.03551},
year={2017}
}
```
# VQAv2
Source: https://docs.lancedb.com/datasets/vqav2
A Lance-formatted version of VQAv2 — open-ended visual question answering on COCO images — sourced from lmms-lab/VQAv2. Each row is one (image, question, 10 annotator answers) triple with paired CLIP image and question embeddings drawn from the…
Source dataset card and downloadable files for `lance-format/vqav2-lance`.
A Lance-formatted version of [VQAv2](https://visualqa.org/) — open-ended visual question answering on COCO images — sourced from [`lmms-lab/VQAv2`](https://huggingface.co/datasets/lmms-lab/VQAv2). Each row is one `(image, question, 10 annotator answers)` triple with paired CLIP image and question embeddings drawn from the same shared space, plus the VQAv2 `question_type` / `answer_type` taxonomy and the consensus `multiple_choice_answer` — all available directly from the Hub at `hf://datasets/lance-format/vqav2-lance/data`.
## Key features
* **Inline JPEG bytes** in the `image` column — no sidecar files or image folders.
* **Paired CLIP embeddings in the same row** — `image_emb` and `question_emb` (OpenCLIP ViT-B/32, 512-dim, cosine-normalized) — so cross-modal text→image retrieval and question-similarity retrieval both work as a single indexed lookup.
* **Both raw and consensus answers** — the 10 annotator answers in `answers` alongside the canonical `multiple_choice_answer`, with parallel `answer_confidences`.
* **Pre-built ANN, FTS, scalar, and bitmap indices** covering both embeddings, the question text, the answer taxonomy, and the COCO and VQAv2 ids.
## Splits
| Split | Rows |
| ------------------ | ------- |
| `validation.lance` | 214,354 |
The `lmms-lab/VQAv2` redistribution declares only the eval splits in its `dataset_info`, so the train shards (\~444 k rows) are not bundled here today; they can be enabled by reading `data/train-*.parquet` directly with PyArrow or by switching to `Multimodal-Fatima/VQAv2_train`. Track progress in `TRACKED_DATASETS.md`.
## Schema
| Column | Type | Notes |
| ------------------------ | ------------------------------- | ----------------------------------------------------------- |
| `id` | `int64` | Row index within split |
| `image` | `large_binary` | Inline JPEG bytes |
| `image_id` | `int64` | COCO image id |
| `question_id` | `int64` | VQAv2 question id |
| `question` | `string` | Natural-language question |
| `question_type` | `string` | First few tokens of the question (e.g. `what is`, `is the`) |
| `answer_type` | `string` | One of `yes/no`, `number`, `other` |
| `multiple_choice_answer` | `string` | Canonical (most-common) answer |
| `answers` | `list` | 10 annotator answers |
| `answer_confidences` | `list` | Parallel confidence list (`yes` / `maybe` / `no`) |
| `image_emb` | `fixed_size_list` | OpenCLIP image embedding (cosine-normalized) |
| `question_emb` | `fixed_size_list` | OpenCLIP text embedding of the question (cosine-normalized) |
Because both embeddings come from the same CLIP model, they share an embedding space and cross-modal retrieval (image→question or question→image) works without any additional alignment.
## Pre-built indices
* `IVF_PQ` on `image_emb` — image-side vector search (cosine)
* `IVF_PQ` on `question_emb` — question-side vector search (cosine)
* `INVERTED` (FTS) on `question` — keyword and hybrid search
* `BITMAP` on `question_type`, `answer_type` — fast categorical filters over the VQAv2 taxonomy
* `BTREE` on `image_id`, `question_id`, `multiple_choice_answer` — fast lookup by id and canonical answer
## Why Lance?
1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.
## Load with `datasets.load_dataset`
You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable when your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import datasets
hf_ds = datasets.load_dataset("lance-format/vqav2-lance", split="validation", streaming=True)
for row in hf_ds.take(3):
print(row["question"], "->", row["multiple_choice_answer"])
```
## Load with LanceDB
LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, Versioning, and Materialize-a-subset sections below.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/vqav2-lance/data")
tbl = db.open_table("validation")
print(len(tbl))
```
## Load with Lance
`pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect dataset internals — schema, scanner, fragments, and the list of pre-built indices.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lance
ds = lance.dataset("hf://datasets/lance-format/vqav2-lance/data/validation.lance")
print(ds.count_rows(), ds.schema.names)
print(ds.list_indices())
```
> **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:
>
> ```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> hf download lance-format/vqav2-lance --repo-type dataset --local-dir ./vqav2-lance
> ```
>
> Then point Lance or LanceDB at `./vqav2-lance/data`.
## Search
The bundled `IVF_PQ` index on `image_emb` makes cross-modal text→image retrieval a single call: encode a question with the same OpenCLIP model used at ingest (ViT-B/32 `laion2b_s34b_b79k`, cosine-normalized), then pass the resulting 512-d vector to `tbl.search(...)` and target `image_emb`. The example below uses the `question_emb` already stored in row 42 as a runnable stand-in for "the CLIP encoding of a question", so the snippet works without any model loaded.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/vqav2-lance/data")
tbl = db.open_table("validation")
seed = (
tbl.search()
.select(["question_emb", "question", "multiple_choice_answer"])
.limit(1)
.offset(42)
.to_list()[0]
)
hits = (
tbl.search(seed["question_emb"], vector_column_name="image_emb")
.metric("cosine")
.select(["image_id", "question", "multiple_choice_answer", "answer_type"])
.limit(10)
.to_list()
)
print("query question:", seed["question"], "->", seed["multiple_choice_answer"])
for r in hits:
print(f" {r['image_id']:>12} [{r['answer_type']}] {r['question'][:60]} -> {r['multiple_choice_answer']}")
```
Because the CLIP embeddings are cosine-normalized, cosine is the right metric. Swap `vector_column_name="image_emb"` for `question_emb` to do question→question retrieval against the validation set instead, which is useful for clustering paraphrases or spotting near-duplicate questions across COCO images.
Because the dataset also ships an `INVERTED` index on `question`, the same query can be issued as a hybrid search that combines the dense vector with a literal keyword match. This is useful when a noun like "dog" must appear in the question text but you still want CLIP to handle visual similarity over the candidate set.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hybrid_hits = (
tbl.search(query_type="hybrid", vector_column_name="image_emb")
.vector(seed["question_emb"])
.text("dog")
.select(["image_id", "question", "multiple_choice_answer"])
.limit(10)
.to_list()
)
for r in hybrid_hits:
print(f" {r['image_id']:>12} {r['question'][:60]} -> {r['multiple_choice_answer']}")
```
Tune `metric`, `nprobes`, and `refine_factor` on the vector side to trade recall against latency.
## Curate
A typical curation pass for VQAv2 combines a structural filter on the answer taxonomy (e.g. only yes/no questions, or only counting questions) with a content predicate on the question text or the consensus answer, so the candidate set is both categorically uniform and topically focused. Stacking both inside a single filtered scan keeps the result small and explicit, and the bounded `.limit(500)` makes it cheap to inspect before committing the subset to anything downstream.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/vqav2-lance/data")
tbl = db.open_table("validation")
candidates = (
tbl.search()
.where(
"answer_type = 'yes/no' AND question_type = 'is the' AND multiple_choice_answer IN ('yes', 'no')",
prefilter=True,
)
.select(["question_id", "image_id", "question", "multiple_choice_answer"])
.limit(500)
.to_list()
)
print(f"{len(candidates)} 'is the' yes/no candidates; first: {candidates[0]['question']} -> {candidates[0]['multiple_choice_answer']}")
```
The result is a plain list of dictionaries, ready to inspect, persist as a manifest of `question_id`s, or feed into the Evolve and Train workflows below. The `image` and embedding columns are never read, so the network traffic for a 500-row candidate scan is dominated by question and answer strings rather than JPEG bytes or vectors.
## Evolve
Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds an `is_binary_answer` flag, a `num_answer_tokens` count, and a `question_length` integer, any of which can then be used directly in `where` clauses without recomputing the predicate on every query.
> **Note:** Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full split first.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("./vqav2-lance/data") # local copy required for writes
tbl = db.open_table("validation")
tbl.add_columns({
"is_binary_answer": "multiple_choice_answer IN ('yes', 'no')",
"question_length": "length(question)",
"answer_length": "length(multiple_choice_answer)",
"num_unique_answers": "array_length(answers)",
})
```
If the values you want to attach already live in another table (a model's predicted answer, an annotator-agreement score, or a difficulty rating), merge them in by joining on `question_id`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
predictions = pa.table({
"question_id": pa.array([262148000, 262148001], type=pa.int64()),
"model_answer": pa.array(["yes", "2"]),
"model_confidence": pa.array([0.87, 0.64]),
})
tbl.merge(predictions, on="question_id")
```
The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running an alternate VLM over the image bytes), Lance provides a batch-UDF API — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/).
## Train
Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetching, shuffling, and batching behave as in any PyTorch pipeline. For a VQA fine-tune, project the JPEG bytes, the question, and the consensus answer; columns added in the Evolve section above cost nothing per batch until they are explicitly projected.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader
db = lancedb.connect("hf://datasets/lance-format/vqav2-lance/data")
tbl = db.open_table("validation")
train_ds = Permutation.identity(tbl).select_columns(["image", "question", "multiple_choice_answer"])
loader = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=4)
for batch in loader:
# batch carries only the projected columns; decode the JPEG bytes,
# tokenize the question, forward through the VLM, compute the loss
# against `multiple_choice_answer`...
...
```
Switching feature sets is a configuration change: passing `["image_emb", "question_emb", "multiple_choice_answer"]` to `select_columns(...)` on the next run skips JPEG decoding entirely and reads only the cached 512-d vectors, which is the right shape for training a lightweight reranker or a linear probe on top of frozen CLIP features.
## Versioning
Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/vqav2-lance/data")
tbl = db.open_table("validation")
print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())
```
Once you have a local copy, tag a version for reproducibility:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
local_db = lancedb.connect("./vqav2-lance/data")
local_tbl = local_db.open_table("validation")
local_tbl.tags.create("openclip-vitb32-v1", local_tbl.version)
```
A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl_v1 = db.open_table("validation", version="openclip-vitb32-v1")
tbl_v5 = db.open_table("validation", version=5)
```
Pinning supports two workflows. A retrieval system locked to `openclip-vitb32-v1` keeps returning stable results while the dataset evolves in parallel — newly added model predictions or alternative annotations do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same images, questions, and consensus answers, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.
## Materialize a subset
Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full split. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
remote_db = lancedb.connect("hf://datasets/lance-format/vqav2-lance/data")
remote_tbl = remote_db.open_table("validation")
batches = (
remote_tbl.search()
.where("answer_type = 'number'")
.select(["question_id", "image_id", "image", "question", "multiple_choice_answer", "answers", "image_emb", "question_emb"])
.to_batches()
)
local_db = lancedb.connect("./vqav2-counting-subset")
local_db.create_table("validation", batches)
```
The resulting `./vqav2-counting-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/vqav2-lance/data` for `./vqav2-counting-subset`.
## Source & license
Converted from [`lmms-lab/VQAv2`](https://huggingface.co/datasets/lmms-lab/VQAv2). VQAv2 questions and annotations are released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The underlying images come from COCO and are subject to Flickr terms of service. See the [VQAv2 download page](https://visualqa.org/download.html) for details.
## Citation
```
@inproceedings{goyal2017making,
title={Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering},
author={Goyal, Yash and Khot, Tejas and Summers-Stay, Douglas and Batra, Dhruv and Parikh, Devi},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2017}
}
```
# Demo Application Gallery
Source: https://docs.lancedb.com/demos/index
Demo apps showcasing end-to-end applications built with LanceDB for production use cases.
Explore the demo applications built with LanceDB below.
| App | Description |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [Semantic.Art](#semantic-art) | A multimodal art discovery platform using feelings, phrases, and images. |
| [Wikipedia 41M Hybrid Search](#wikipedia-41m-hybrid-search) | An interactive hybrid search demo combining full-text search and vector search. |
| [Video Search](#video-search) | A video search application that allows searching through a library of videos using natural language queries. |
## Semantic.art
multimodal
hybrid-search
vector-search
semantic-routing
Semantic.art turns real, human-made art discovery into a multimodal search experience using
feelings, phrases, and images. It's built with LanceDB hybrid search and semantic routing.
Read in detail about how Semantic.art is built in this blog post.
## Wikipedia 41M Hybrid Search
multimodal
hybrid-search
vector-search
fts
Interactive hybrid search, full-text search (FTS) and vector search demo with 41M+ Wikipedia entries.
Explore the power of combining FTS with vector search for more relevant results.
Read in detail about how the Wikipedia 41M hybrid search demo is built in this blog post.
## Video Search
multimodal
video-search
vector-search
Search through a library of videos using natural language queries. This demo showcases how to use LanceDB
to perform semantic search on video content.
# Managing Embeddings
Source: https://docs.lancedb.com/embedding/index
Use the embedding API in LanceDB -- registry, functions, schemas, and multi-language SDK support.
Modern machine learning models can be trained to convert raw data into embeddings, which are vectors
of floating point numbers. The position of an embedding in vector space captures the semantics of
the data, so vectors that are close to each other are considered similar.
LanceDB provides an embedding function registry in OSS as well as its Enterprise versions
([see below](#embeddings-in-lancedb-enterprise))
that automatically generates vector embeddings during data ingestion. Automatic query-time embedding
generation is available in LanceDB OSS, with SDK-specific query ergonomics. The API abstracts
embedding generation, allowing you to focus on your application logic.
## Embedding Registry
You can get a supported embedding function from the registry, and then use it in your table schema.
Once configured, the embedding function will automatically generate embeddings when you insert data
into the table. Query-time behavior depends on SDK: Python/TypeScript can query with text directly,
while Rust examples typically compute query embeddings explicitly before vector search.
### Using an embedding function
Create an embedding function before you attach it to table or schema metadata. Python and TypeScript fetch
provider implementations from the embedding registry, while Rust constructs the provider embedding function
directly and registers it on the connection before using it in an `EmbeddingDefinition`.
Provider configuration is SDK-specific, so copy the option names from the provider page for the SDK you use.
For example, the OpenAI model is selected with `name` in Python, `model` in TypeScript, and the model argument
to `OpenAIEmbeddingFunction::new_with_model` in Rust.
| Concept | Python | TypeScript | Rust |
| ----------- | -------------------------------------------------- | -------------------------------------------------- | --------------------------------------------------- |
| Model | `name="text-embedding-3-small"` | `{ model: "text-embedding-3-small" }` | `new_with_model(api_key, "text-embedding-3-small")` |
| Retry count | `max_retries=7` | Provider/client-specific | Provider/client-specific |
| API key | `api_key="..."`, environment variables, or `$var:` | `apiKey: "..."`, environment variables, or `$var:` | Constructor argument or environment variable |
| Device | Provider-specific, for example `device="cuda"` | Provider-specific | Provider-specific |
When ingesting data with an embedding definition, LanceDB only computes the vector column if that
column is missing from the incoming batch or present but entirely null. If you provide any non-null
values in the vector column, LanceDB treats the column as user-supplied and does not backfill the
remaining rows in that batch.
For reusable runtime configuration, the registry also supports `$var:` placeholders in embedding-function config.
This is useful for provider secrets and environment-specific settings in Python and TypeScript.
* Python uses `registry.set_var(...)`.
* TypeScript uses `registry.setVar(...)`.
* You can provide a fallback with `$var:name:default`.
* Sensitive values such as API keys should be passed through registry variables instead of hardcoding them in config.
For non-sensitive settings such as inference device selection, you can also use a default fallback:
Find the full list of arguments for each provider in the [integrations](/integrations/embedding) section.
## Multiple embedding columns
A single table can include more than one embedding definition when you want to store multiple semantic views
of the same data, or generate embeddings from different source columns. In practice, each embedding definition
maps one source column to one vector column, and the table schema can contain multiple such pairs.
The exact setup differs by SDK, but the underlying pattern is the same: define a distinct source/vector pair
for each embedding function you want applied during ingest.
In TypeScript, automatic query embedding currently uses the first embedding function stored in the
table metadata. If a table has multiple embedding definitions and you need to query a specific vector
column, compute the query embedding explicitly and pass the vector to the search builder.
## Embedding model providers
LanceDB supports most popular embedding providers.
### Text embeddings
| Provider | Model ID | Default Model |
| --------------------- | ----------------------- | ------------------------ |
| OpenAI | `openai` | `text-embedding-ada-002` |
| Sentence Transformers | `sentence-transformers` | `all-MiniLM-L6-v2` |
| Hugging Face | `huggingface` | `colbert-ir/colbertv2.0` |
| Cohere | `cohere` | `embed-english-v3.0` |
| ... | ... | ... |
### Multimodal embedding
| Provider | Model ID | Supported Inputs |
| --------- | ----------- | -------------------------- |
| OpenCLIP | `open-clip` | Text, Images |
| ImageBind | `imagebind` | Text, Images, Audio, Video |
| ... | ... | ... |
You can find all supported embedding models in the [integrations](/integrations/embedding) section.
## Embeddings in LanceDB Enterprise
Enterprise
In LanceDB Enterprise, embedding generation during data ingestion is client-side and the resulting vectors are
stored on the remote table.
The Enterprise server does not currently generate embeddings from query text on its own. Any automatic
query-time embedding happens on the client side.
### How string queries are interpreted
For the Python remote client, `table.search("hello")` can take two different paths:
* If the selected vector column has embedding metadata
(i.e., the table schema stores the source-column, vector-column, and
embedding-function mapping created from fields like `SourceField()` and
`VectorField()` during table creation), then the embeddings are computed in the Python client process.
The client uses the same local LanceDB embedding registry used by OSS tables to
reconstruct the embedding function from schema metadata, compute the query vector in
the client process, and send that vector to Enterprise for search.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
result = table.search("hello").limit(1).to_list()
# The Python client computes the query embedding locally, then sends a vector search.
```
* If the table does not have embedding metadata for that search, `table.search("hello")` in `auto` mode is
treated as an FTS query instead.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
result = table.search("hello").limit(5).to_list()
# In auto mode this is treated as an FTS query, not a vector query.
```
If you want explicit vector or hybrid behavior and the client cannot resolve an embedding function from the
table metadata, generate the query embedding yourself and pass the vector directly.
TypeScript has a similar string-query distinction in `auto` mode: if no embedding providers have been
registered in the process, `search("text")` falls back to FTS. Once an embedding provider has been
imported and registered, the client expects table embedding metadata for automatic vector search and
raises an error if the table has none.
The manual query-embedding flow below works across Enterprise SDKs and is an explicit path you can use when you
want full control over query-time behavior.
## Custom Embedding Functions
You can always implement your own embedding function:
* Python/TypeScript: subclass `TextEmbeddingFunction` (text) or `EmbeddingFunction` (multimodal).
* Rust: implement the `EmbeddingFunction` trait.
# Embeddings: Quickstart
Source: https://docs.lancedb.com/embedding/quickstart
Quickstart guide for generating and working with embeddings.
LanceDB will automatically vectorize the data both at ingestion and query time. All you need to do is specify which model to use.
Popular embedding models like OpenAI, Hugging Face, Sentence Transformers, CLIP, and more, are supported.
## Step 1: Import Required Libraries
First, import the necessary LanceDB components:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.pydantic import LanceModel, Vector
from lancedb.embeddings import get_registry
```
* `lancedb`: The main database connection and operations
* `LanceModel`: Pydantic model for defining table schemas
* `Vector`: Field type for storing vector embeddings
* `get_registry()`: Access to the embedding function registry. It has all the supported as well as custom embedding functions registered by the user
* TypeScript uses `lancedb.embedding.getRegistry()` and `lancedb.embedding.LanceSchema()` for the same registry/schema workflow
* In TypeScript, import the provider module before calling `getRegistry().get(...)`; the provider import is what registers names such as `"huggingface"` or `"openai"`
## Step 2: Connect to LanceDB
Establish a connection to your LanceDB OSS directory or Enterprise cluster:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Enter your LanceDB connection URI for OSS or Enterprise here
db = lancedb.connect(...)
```
## Step 3: Initialize the Embedding Function
Choose and configure your embedding model:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
model = get_registry().get("sentence-transformers").create(name="BAAI/bge-small-en-v1.5", )
```
This creates an embedding function from the local embedding registry. The Python snippet uses the
`sentence-transformers` provider with the BGE model; the TypeScript snippet uses the Transformers-backed
`huggingface` provider. You can:
* Change `"sentence-transformers"` to other providers like `"openai"`, `"cohere"`, etc.
* Modify the model name for different embedding models
* Set `device="cuda"` for GPU acceleration if available
## Step 4: Define Your Schema
Create a Pydantic model that defines your table structure:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
class Words(LanceModel):
text: str = model.SourceField()
vector: Vector(model.ndims()) = model.VectorField()
```
* `SourceField()`: This field will be embedded
* `VectorField()`: This stores the embeddings
* `model.ndims()`: Sets vector dimensions for your model
* In TypeScript, use `model.sourceField(...)` and `model.vectorField()` inside `LanceSchema(...)`
## Step 5: Create Table and Ingest Data
Create a table with your schema and add data:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table = db.create_table("words", schema=Words)
table.add([
{"text": "hello world"},
{"text": "goodbye world"}
])
```
The `table.add()` call automatically:
* Takes the text from each document
* Generates embeddings using your chosen model
* Stores both the original text and the vector embeddings
If your input already includes the vector column, automatic embedding only runs when that column is
absent or entirely null for the batch. Partially supplied vectors are treated as manual data, so
LanceDB preserves them instead of filling only the missing rows.
## Step 6: Query with Automatic Embedding
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. In the Python remote
client, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
the client computes the query embedding before sending the vector search. If there is no embedding metadata for
that search, `search("greetings")` in `auto` mode is treated as FTS instead.
Search your data using natural language queries:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
query = "greetings"
actual = table.search(query).limit(1).to_pydantic(Words)[0]
print(actual.text)
```
The search process:
1. Automatically converts your query text to embeddings
2. Finds the most similar vectors in your table
3. Returns the matching documents
Automatic text search depends on the table's embedding metadata. If the client cannot reconstruct an
embedding function from that metadata, compute the query embedding yourself and search with the
vector directly.
# Architecture
Source: https://docs.lancedb.com/enterprise/architecture
Learn how LanceDB Enterprise separates control, compute, and storage to serve remote tables at scale.
LanceDB Enterprise is a remote, cluster-backed service built for teams that need low-latency search, predictable operations, and durable storage beyond a single machine. Instead of tying query serving, indexing, compaction, and data persistence to the same process, Enterprise separates those concerns so each part of the system can scale and evolve independently.
At a high level, it helps to think of LanceDB Enterprise as a set of layers that interoperate with one another. Users connect to **remote tables** over the network. The **data plane** serves reads, writes and background jobs such as indexing and compaction. The **control plane** manages configuration, identity, policy, and cluster lifecycle. **Indexers** build indexes and compact data outside the request path. **Object storage** holds the durable table data, manifests, and index artifacts independently of the machines serving traffic.
This architecture matters because enterprise workloads are rarely shaped like a single benchmark. Some need thousands of concurrent queries. Others need large-scale ingestion, continuous indexing, or strict operational boundaries between user traffic and background work. LanceDB Enterprise is designed so those workloads do not all compete for the same machine, disk, or process.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
flowchart TB
RT[Remote tables]
subgraph DP[Data plane]
QS[Query serving]
IX[Indexers]
end
CP[Control plane]
OS[(Object storage)]
RT --> DP
DP --> OS
IX --> OS
CP -.->|Govern and configure| DP
CP -.->|Govern and configure| IX
```
## Compute-storage separation
In LanceDB Enterprise, storage and compute are deliberately decoupled. Table data and index artifacts live in object storage, while query-serving and background workers read from and write to that shared durable layer. This means compute can be replaced, scaled, or specialized without making any individual node the owner of the dataset.
This design has practical consequences. Query fleets can scale for interactive traffic without also scaling background indexing capacity. Heavy indexing and compaction work can run on dedicated workers instead of stealing resources from user-facing queries. Caches can accelerate hot reads without becoming the source of truth. And because the data remains in object storage, durability does not depend on the lifecycle of a particular server or local disk.
## Architecture
At a high level, the control plane governs the system and the data plane executes the work. The control plane is responsible for configuration, service discovery, identity integration, policy, and cluster lifecycle. It determines how the system should behave, but it is not the layer serving table data or executing user queries -- that's the role of the data plane.
Within the data plane, serving is separated into two kinds of nodes that are provisioned independently. Query nodes are the client-facing layer. They receive requests against remote tables, validate them, resolve the target table, plan the work, and return results.
Plan executors are the read-execution layer behind the query nodes. For read-heavy query paths, they execute cache-backed reads against object storage, which helps reduce repeated remote reads and makes performance more predictable as load grows.
Indexers handle heavyweight background work such as building indexes, merging index state, compacting data, and updating the table’s stored index and layout artifacts. Together, these components let LanceDB Enterprise scale request handling, read execution, and index-building independently instead of forcing them to compete for the same compute resources.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
flowchart TB
Client[Client applications]
subgraph CP[Control plane]
CFG[Configuration, identity, policy, lifecycle]
end
subgraph DP[Data Plane
Managed or self-hosted]
QP[Query planning + routing fleet]
PE[Executor + cache fleet]
MON[Monitoring + orchestration fleet]
WAL[MemWAL fleet]
JOBS[Jobs fleet]
QP -->|Query subplans| PE
QP -->|Send state + job requests| MON
QP -->|Fresh reads and writes| WAL
PE -.->|Send state| MON
WAL -.->|Send state| MON
MON -->|Indexing, compaction,
backfill, refresh jobs| JOBS
JOBS -->|Send state| MON
end
OS[(Object Storage)]
Client -->|Queries, writes, backfills| DP
CFG -.->|Govern and configure| DP
PE -->|Read, cache data| OS
JOBS -->|Read/write table and feature data| OS
MON -.->|Table and cluster state| OS
WAL -->|Persist| OS
```
## Remote tables
A [remote table](/tables-and-namespaces#understanding-tables) is the user-facing abstraction over this architecture. From the client side, you connect to a logical storage layer and table over the network by providing a `db://...` connection identifier. The system then resolves that logical name to the underlying storage-backed table and executes the operation inside the cluster.
This is why Enterprise feels familiar at the API level while operationally behaving differently. Your application still issues table operations and queries, but it is no longer coupled to a local storage path or a single host. Instead, the cluster takes responsibility for execution, coordination, and background upkeep. In SDK terms, `open_table(...)` returns a `RemoteTable`. Architecturally, a remote table is the bridge between the client-facing API and the storage-backed system behind it.
This design makes LanceDB Enterprise suitable for catalog-backed layouts, see [Namespaces and the Catalog Model](/namespaces) for more details. For the basic application flow, see the shared [quickstart](/quickstart).
## Read path
When a client issues a query against a remote table, the path is straightforward:
1. The request reaches a query node in the data plane.
2. The query node validates the request, resolves the table, and plans the work.
3. For read-heavy queries, the query node can send part of the read work to plan executors.
4. Plan executors read the required table data from object storage, using cache where it helps reduce repeated remote reads.
5. Results are returned to the query node, assembled, and sent back to the client.
This separation is what lets Enterprise combine a clean remote API with a serving layer that can scale horizontally and keep hot data close to execution.
## Write path
Writes follow a different path because durability comes first:
1. A client writes to a remote table.
2. The query node validates the request and commits the new table state.
3. After the write succeeds, the system emits follow-up signals for indexing, compaction, or cleanup.
Keeping the commit path centered on object storage ensures that the durable record of the table lives outside any single query node. Regardless of whether you're using Lance namespaces or an external catalog, the catalog's role is mainly to resolve table names and provide access details -- the table’s actual data and index artifacts remain in object storage.
## Background work
Indexing, compaction, and cleanup are intentionally moved off the user request path. After table changes are committed, the system can determine that additional work is needed and assign it to background workers built for heavyweight processing.
In practice, that usually looks like this:
1. Table changes create follow-up events that query nodes publish to an event queue for indexing, compaction, or cleanup.
2. A background coordinator agent consumes those events, evaluates the state of the table, and decides what should run next.
3. Indexers read the relevant table state from object storage, produce updated artifacts, and write the results back.
This separation is one of the clearest architectural reasons to use LanceDB Enterprise: the same query-serving infrastructure does not have to handle every expensive indexing or compaction task itself.
## What this means for users
For teams using LanceDB Enterprise, the architecture changes the *operational model* more than the programming model. You still work with tables and queries, but the cluster now takes responsibility for distributed execution, cache-aware reads, and long-running background jobs.
The result is a system that is easier to run under production pressure:
* You interact with remote tables instead of managing physical storage layout directly.
* Query serving, indexing, and compaction can scale independently.
* Durable state lives in object storage rather than on individual machines.
* Background indexing and compaction improve performance over time without forcing that work into the foreground request path.
This architecture gives ML and AI teams a strong storage-backed platform for training, retrieval, search, and analytics workloads that can scale beyond a single machine while still providing a familiar API and predictable performance.
# Authentication
Source: https://docs.lancedb.com/enterprise/authentication
Authentication modes for LanceDB Enterprise with an API key or OAuth credentials.
LanceDB Enterprise supports two ways for clients to authenticate against a `db://` remote table:
* **API keys** — a long-lived shared secret passed as `api_key` on connect. Works in every SDK.
* **OAuth 2.0** — short-lived bearer tokens obtained from your identity provider, refreshed automatically by the client.
OAuth is the recommended option when you want to rotate credentials centrally, plug into an existing identity provider, or run on Azure with managed identities so no secret material lives on the client.
## API key
Pass the API key from your Enterprise tenant on `connect`. This works with both the synchronous and asynchronous Python clients, as well as TypeScript and Rust.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect(
uri="db://your-database-uri",
api_key="your-api-key",
region="us-east-1",
host_override="https://your-enterprise-endpoint.com",
)
```
## OAuth
The async Python client and the TypeScript client can obtain bearer tokens from an OIDC issuer and attach them to every request. Token acquisition, caching, and refresh are handled inside the client — your application code only provides the configuration.
In Python, OAuth is supported through `lancedb.connect_async`. The synchronous `connect` entry point continues to use API key authentication for `db://` URIs. In TypeScript, `lancedb.connect` accepts `oauthConfig` directly.
### Supported flows
`OAuthFlowType` selects how the client acquires tokens:
| Flow | Python value | TypeScript value | When to use |
| ---------------------- | -------------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Client Credentials | `OAuthFlowType.CLIENT_CREDENTIALS` | `OAuthFlowType.ClientCredentials` | Service-to-service / machine-to-machine. Requires a client ID and client secret registered with your identity provider. |
| Azure Managed Identity | `OAuthFlowType.AZURE_MANAGED_IDENTITY` | `OAuthFlowType.AzureManagedIdentity` | Workloads running on Azure compute (VMs, AKS, App Service, Container Apps). Tokens are fetched from the Azure IMDS endpoint, so no client secret is stored on the client. |
### Configure OAuth
Build an OAuth config and pass it on connect. Use `oauth_config` in Python and `oauthConfig` in TypeScript.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.remote import OAuthConfig, OAuthFlowType
# Client Credentials (service-to-service)
oauth_config = OAuthConfig(
issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
client_id="your-application-client-id",
client_secret="your-application-client-secret",
scopes=["api://lancedb-api/.default"],
flow=OAuthFlowType.CLIENT_CREDENTIALS,
)
db = await lancedb.connect_async(
uri="db://your-database-uri",
region="us-east-1",
host_override="https://your-enterprise-endpoint.com",
oauth_config=oauth_config,
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
import { OAuthConfig, OAuthFlowType } from "@lancedb/lancedb";
// Client Credentials (service-to-service)
const oauthConfig: OAuthConfig = {
issuerUrl: "https://login.microsoftonline.com/{tenant}/v2.0",
clientId: "your-application-client-id",
clientSecret: "your-application-client-secret",
scopes: ["api://lancedb-api/.default"],
flow: OAuthFlowType.ClientCredentials,
};
const db = await lancedb.connect("db://your-database-uri", {
region: "us-east-1",
hostOverride: "https://your-enterprise-endpoint.com",
oauthConfig,
});
```
For workloads running on Azure compute, use a managed identity instead of a client secret:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.remote import OAuthConfig, OAuthFlowType
oauth_config = OAuthConfig(
issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
client_id="your-application-client-id",
scopes=["api://lancedb-api/.default"],
flow=OAuthFlowType.AZURE_MANAGED_IDENTITY,
# Optional: required only for user-assigned managed identities.
managed_identity_client_id="your-user-assigned-identity-client-id",
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import { OAuthConfig, OAuthFlowType } from "@lancedb/lancedb";
const oauthConfig: OAuthConfig = {
issuerUrl: "https://login.microsoftonline.com/{tenant}/v2.0",
clientId: "your-application-client-id",
scopes: ["api://lancedb-api/.default"],
flow: OAuthFlowType.AzureManagedIdentity,
// Optional: required only for user-assigned managed identities.
managedIdentityClientId: "your-user-assigned-identity-client-id",
};
```
### Configuration reference
The same configuration is available in both SDKs. Python uses `snake_case` field names; TypeScript uses `camelCase`.
| Python field / TypeScript field | Required | Description |
| -------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `issuer_url` / `issuerUrl` | Yes | OIDC issuer URL or OAuth authority URL. For Azure, use `https://login.microsoftonline.com/{tenant_id}/v2.0`. |
| `client_id` / `clientId` | Yes | Application / client ID registered with your identity provider. |
| `scopes` | Yes | List of OAuth scopes to request. For Azure managed identity, provide exactly one scope or resource, for example `["api://your-app-id/.default"]`. |
| `flow` | No | Selects the OAuth flow. Defaults to Client Credentials. |
| `client_secret` / `clientSecret` | Conditional | Required for Client Credentials. Redacted from the config's `repr` (Python) and `Debug` output (TypeScript native binding) so it does not leak into logs. |
| `managed_identity_client_id` / `managedIdentityClientId` | No | Client ID of a user-assigned managed identity. Only used with Azure Managed Identity; omit for system-assigned identities. |
| `refresh_buffer_secs` / `refreshBufferSecs` | No | How many seconds before token expiry to proactively refresh. Defaults to 300. Keep this well below the token TTL — setting it greater than or equal to the TTL forces a refresh on every request. |
Treat the client secret like any other production credential. Load it from a secret manager or environment variable rather than committing it to source control. Both SDKs deliberately omit the secret from their debug/repr output so accidental log lines do not expose the value.
# Benchmarks
Source: https://docs.lancedb.com/enterprise/benchmarks
Representative latency and scalability benchmarks for LanceDB Enterprise.
LanceDB Enterprise is designed for low-latency, high-throughput search, but observed performance depends on factors such as dataset shape, index configuration, cache warmth, filter selectivity, concurrency, and cluster sizing. The figures on this page should be read as representative benchmark results for a specific test setup, not as universal guarantees for every deployment.
In our benchmark environment, warmed-cache vector search reached around **25ms** P50 latency, and metadata-filtered search reached around **50ms** P99 latency for the filter pattern shown below.
If you want performance guidance for your own workload, reach out to [contact@lancedb.com](mailto:contact@lancedb.com). The LanceDB engineering team can help map your latency, throughput, ingestion, and filtering requirements to an appropriate Enterprise cluster design.
| Percentile | Vector Search | Vector Search w. Filtering | Full-Text Search |
| :--------: | :-----------: | :------------------------: | :--------------: |
| P50 | 25ms | 30ms | 26ms |
| P90 | 26ms | 39ms | 37ms |
| P99 | 35ms | 50ms | 42ms |
Depending on workload and tuning, Enterprise clusters can also be configured for high concurrency, including thousands of QPS in some deployments, but the right configuration varies by use case. Training, search, and analytics workloads often benefit from different cluster shapes and resource allocation strategies. To understand which parts of the system influence these results, see the [Enterprise architecture](/enterprise/architecture) guide.
## Dataset
We used two datasets for this benchmark: the [dbpedia-entities-openai-1M](https://huggingface.co/datasets/KShivendu/dbpedia-entities-openai-1M)
for vector search, and a synthetic dataset for vector search with metadata filtering.
| Name | # Vectors | Vector Dimension |
| :------------------------- | :--------: | :--------------: |
| dbpedia-entities-openai-1M | 1,000,000 | 1536 |
| synthetic dataset | 15,000,000 | 256 |
These benchmark results are most useful as a directional baseline. Different data distributions, index choices, cache behavior, and cluster settings can materially change the latency profile.
## Vector Search
We ran vector queries against `dbpedia-entities-openai-1M` with a warmed-up cache. In that benchmark setup, we observed the following latency profile:
| Percentile | Latency |
| :--------: | :-----: |
| P50 | 25ms |
| P90 | 26ms |
| P99 | 35ms |
| Max | 49ms |
## Full-Text Search
With the same dataset and a warmed-up cache, full-text search fell into the following range:
| Percentile | Latency |
| :--------: | :-----: |
| P50 | 26ms |
| P90 | 37ms |
| P99 | 42ms |
| Max | 98ms |
## Vector Search with Metadata Filtering
We created a 15M-vector dataset to evaluate metadata-aware search under more complex filtering conditions.
These filters can span a wide range of scalar columns, for example, "find Sci-fi movies since 1900".
With a warmed-up cache, slightly more selective filters, for example, "find Sci-fi movies between the years 2000 and 2012", produced the following representative results:
| Percentile | Latency |
| :--------: | :-----: |
| P50 | 30ms |
| P90 | 39ms |
| P99 | 50ms |
Broader or less selective filters, for example, "find Sci-fi movies since 1900", moved the latency range to:
| Percentile | Latency |
| :--------: | :-----: |
| P50 | 65ms |
| P90 | 76ms |
| P99 | 100ms |
These benchmarks are intended to provide consistent, reproducible reference points for LanceDB Enterprise rather than one-size-fits-all promises. We periodically re-run and update numbers as necessary, but production performance will still depend on workload shape, cluster tuning, and the architectural choices described in the [Enterprise architecture](/enterprise/architecture) guide.
# Azure deployment guide
Source: https://docs.lancedb.com/enterprise/deployment/azure
Learn how to deploy LanceDB Enterprise on Azure with AKS, Private Link, and Blob Storage.
LanceDB Enterprise can be deployed on Azure using Azure Kubernetes Service (AKS) with Azure Blob Storage for data persistence and Azure Private Link for secure connectivity.
## General Architecture Overview
```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
graph TB
subgraph "Client VPC"
Client[Client Applications]
end
subgraph "Server VPC"
PLS[Azure Private Link Service]
subgraph "AKS Cluster"
LDB[LanceDB Enterprise
Query Nodes, Plan Executors,
Lance Agent, Indexer Pods]
end
EH[Azure EventHub
for LanceDB internal
message passing]
BS[Azure Blob Storage]
WI[Azure Workload Identity]
end
Client ==>|Private Link| PLS
PLS ==> LDB
LDB <-->|Read/Write| BS
LDB -->|Async Events| EH
EH -->|Process| LDB
WI -.->|RBAC| BS
WI -.->|Assigned| LDB
style Client fill:#d7e3fc,stroke:#5c6bc0,stroke-width:2px,color:#0d1b2a
style PLS fill:#f3e5f5,stroke:#ab47bc,stroke-width:2px,color:#311432
style LDB fill:#ffe0b2,stroke:#fb8c00,stroke-width:2px,color:#4a2f11
style EH fill:#f8bbd0,stroke:#ec407a,stroke-width:2px,color:#4a0821
style BS fill:#e0f2f1,stroke:#26a69a,stroke-width:2px,color:#09312d
style WI fill:#e6f4ea,stroke:#66bb6a,stroke-width:2px,color:#1d3a1f
```
### Key Components
* **LanceDB architecture** is deployed in an AKS cluster within its own VPC
* **Client applications** connect to the cluster securely using Azure Private Link
* **AKS cluster** is granted Azure Blob Storage read/write permissions using Azure Workload Identity
* **Azure EventHub** can be used as the message queue by LanceDB Enterprise for internal message communication (alternative: self-hosted Kafka cluster in AKS)
## Read Path Architecture
```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
graph LR
subgraph "Client Network"
C[Client App]
end
subgraph "Azure AKS Cluster"
PL[Private Link
Service]
QN[Query Nodes
Phalanx]
PE[Plan Executors
Distributed Data Cache]
end
subgraph "Storage"
BS[Azure Blob
Storage]
end
C -->|Private
Connection| PL
PL --> QN
QN -->|Query
Request| PE
PE -->|Cache Miss
Read Data| BS
style C fill:#d7e3fc,stroke:#5c6bc0,color:#0d1b2a
style PL fill:#f3e5f5,stroke:#ab47bc,color:#311432
style QN fill:#ffe0b2,stroke:#fb8c00,color:#4a2f11
style PE fill:#ffecb3,stroke:#ffb74d,color:#4a2f11
style BS fill:#e0f2f1,stroke:#26a69a,color:#09312d
```
### Read Path Flow
1. **Client Application** sends query request through Private Link
2. **Query Nodes** receive and process the request
3. **Plan Executors** optimize and execute the query using distributed data cache to speed up read queries
4. **Azure Blob Storage** stores data and indices in Lance, while Plan Executors maintain distributed cache for performance
## Write Path Architecture
```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
graph LR
subgraph "Client Network"
C[Client App]
end
subgraph "Azure AKS Cluster"
PL[Private Link
Service]
QN[Query Nodes
Phalanx]
LA[Lance Agent]
IP[Indexer Pods
On-Demand]
end
subgraph "Messaging"
EH[Azure EventHub
Write Events]
end
subgraph "Storage"
BS[Azure Blob
Storage]
end
C -->|Private
Connection| PL
PL --> QN
QN -->|Sync
Write| BS
QN -->|Async
Events| EH
EH -->|Consume| LA
LA -->|Launch| IP
IP -->|Index &
Optimize| BS
style C fill:#d7e3fc,stroke:#5c6bc0,color:#0d1b2a
style PL fill:#f3e5f5,stroke:#ab47bc,color:#311432
style QN fill:#ffe0b2,stroke:#fb8c00,color:#4a2f11
style LA fill:#ffe5c3,stroke:#ffb74d,color:#4a2f11
style IP fill:#ffe5c3,stroke:#ffb74d,color:#4a2f11
style EH fill:#f8bbd0,stroke:#ec407a,color:#4a0821
style BS fill:#e0f2f1,stroke:#26a69a,color:#09312d
```
### Write Path Flow
Query nodes write data and indices synchronously to Azure Blob Storage in Lance data format while asynchronously sending data modification events to Azure EventHub (or self-hosted Kafka cluster). These write events are processed by the Lance Agent, which launches indexing pods or data optimization pods to optimize data for better read performance.
## Deployment Options
### Storage Architecture Support
```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
graph TB
subgraph "Multi-Account & Multi-Container Support"
SA1[Storage Account 1]
SA2[Storage Account 2]
SA3[Storage Account N]
SA1 --> C1A[Container A]
SA1 --> C1B[Container B]
SA1 --> C1C[Container C]
SA2 --> C2A[Container X]
SA2 --> C2B[Container Y]
SA3 --> C3A[Container 1]
SA3 --> C3B[Container 2]
end
style SA1 fill:#e0f2f1,stroke:#26a69a,color:#09312d
style SA2 fill:#e0f2f1,stroke:#26a69a,color:#09312d
style SA3 fill:#e0f2f1,stroke:#26a69a,color:#09312d
```
### Deployment Models
LanceDB Enterprise supports three deployment models on Azure:
#### 1. Fully Managed Service
* **Infrastructure and storage** in LanceDB's Azure account
* **Complete management** by LanceDB team
* **Simplest setup** for customers
#### 2. BYOC (Bring Your Own Cloud)
* **Infrastructure and storage** in customer's Azure account
* **Fully Managed by LanceDB**
* **Full control** over data residency
#### 3. Hybrid - Bring Your Own Container
* **Infrastructure** in LanceDB's account
* **Storage containers** in customer's account
For private deployments, high performance at extreme scale, or if you have strict security requirements, [contact us about LanceDB Enterprise](mailto:contact@lancedb.com).
# Deployment guide
Source: https://docs.lancedb.com/enterprise/deployment/index
Learn how to deploy LanceDB Enterprise in production environments.
There are two deployment models available for LanceDB Enterprise: **Managed** and **BYOC**.
Both models support AWS, GCP, and Azure cloud platforms.
## Managed deployment
This is a private deployment of LanceDB Enterprise.
All applications run in cloud accounts managed by LanceDB in the same location as your client applications.
This hands-off approach is recommended for users who do not wish to manage the infrastructure themselves.
To access your deployment, LanceDB can provision either a public or private load balancer.
## Bring-your-own-cloud (BYOC) deployment
With this deployment model, LanceDB Enterprise is installed into your own cloud account.
This approach is recommended when:
* Users' security requirements for data residency preclude them from having data leave their account
* Other applications need to access the object storage directly
To deploy, an identity will be provisioned in your account with permissions to manage the infrastructure.
## Custom deployments
LanceDB Enterprise installation is highly configurable and customizable to your needs.
If you have any other specific deployment requirements, please reach out to our support team
at [contact@lancedb.com](mailto:contact@lancedb.com).
# LanceDB Enterprise
Source: https://docs.lancedb.com/enterprise/index
Features and benefits of LanceDB Enterprise.
**LanceDB Enterprise** is the production deployment option for teams that want to run LanceDB as a private cloud or
bring-your-own-cloud (BYOC) **multimodal lakehouse**.
If you are new to multimodal lakehouses, the short version is this: LanceDB keeps vectors, metadata, and source data
together in open table storage, while Enterprise adds the distributed infrastructure needed to serve real production
workloads on top of that data. It is designed for teams that need more scale, more operational visibility, and more
control than a single embedded process can provide.
If you need private deployments, high performance at extreme scale, or if you have strict security requirements,
[reach out to our team](mailto:contact@lancedb.com) to set up a LanceDB Enterprise cluster in your environment.
## Why use LanceDB Enterprise?
If you are evaluating LanceDB for a production AI system, Enterprise is built around three practical needs: handling
very large vector workloads, running feature engineering close to the data, and operating the platform with production
visibility.
### 1. 100B+ row scale
LanceDB Enterprise is built for demanding workloads that exceed the capabilities of a single machine, whether from extremely large data volumes or a high number of concurrent queries. Instead of asking your
application to own caching, query scaling, and maintenance, Enterprise turns those into **platform** capabilities.
This matters when your AI application moves past a prototype and starts serving real users, larger datasets, and
more concurrent requests.
* **Low-latency tiered cache**: Enterprise keeps frequently read data closer to compute, so common queries do not need
to fetch the same data from object storage over and over again. That helps reduce wait times and makes performance
more predictable as traffic increases.
* **Horizontal query throughput**: Instead of relying on one application process to answer every query, Enterprise can
spread search traffic across multiple nodes. This lets teams add capacity as usage grows, rather than re-architecting
the application each time demand spikes.
* **Distributed search**: Coming soon. Enterprise is adding dynamic horizontal scaling to search execution to allow for low latency search at much higher volumes and concurrency.
* **Distributed indexing and compaction**: Coming soon. Enterprise is expanding support for large-table maintenance so
indexing and storage cleanup can happen as platform workflows rather than as manual operator tasks.
* **Enterprise training cache**: Coming soon. Enterprise is extending the same storage-aware caching model to training and
feature engineering pipelines so large jobs can use GPU capacity more efficiently.
### 2. Feature engineering with Geneva
For many teams, retrieval is only part of the problem. They also need a reliable way to derive new columns, run
backfills, and keep feature pipelines close to the data they already store in LanceDB. This is what [Geneva](/geneva/) enables.
* **Derived features**: Create new columns from existing data and user-defined logic without standing up a separate
feature platform first.
* **Large-table backfills**: Update or recompute features across large datasets without wiring your own distributed
batch system around OSS tables.
* **Shared workflows**: Use Geneva clusters, manifests, and jobs to manage feature engineering work in one place.
### 3. Enterprise-grade monitoring
Production retrieval systems need more than search or training performance. Teams also need to observe the system, choose how it
is deployed, and satisfy security and compliance requirements.
* **Metrics and traces**: Integrates with existing observability systems for monitoring and distributed tracing using
OpenTelemetry.
* **Deployment choice**: Run LanceDB as a managed deployment or install it inside your own cloud account with
[BYOC](/enterprise/deployment/).
* **Private networking and compliance**: Designed for production environments that need encryption at rest, private
connectivity options, and compliance coverage such as SOC 2 Type II and HIPAA.
## How is LanceDB Enterprise different from OSS?
LanceDB OSS runs inside your application process. LanceDB Enterprise runs as a distributed cluster across many
machines. Both are built on the same Lance columnar file format, so moving data from one edition to the other does
not require a data conversion step.
| Dimension | LanceDB OSS | LanceDB Enterprise | What the difference means |
| :------------------------------ | :------------------------------------ | :------------------------------------ | :---------------------------------------------------------------------------------------------------------------------- |
| **Mode** | Single process | Distributed fleet | OSS lives on one host. Enterprise spreads work across nodes and keeps serving even if one node fails. |
| **Latency from object storage** | 500–1000 ms | 50–200 ms | Enterprise mitigates network delay with an SSD cache and parallel reads. |
| **Throughput** | 10–50 QPS | Up to 10,000 QPS | A cluster can serve thousands of concurrent users; a single process cannot. |
| **Cache** | None | Distributed NVMe cache | Enterprise keeps hot data near compute and avoids repeated S3 calls. |
| **Indexing & compaction** | Manual | Platform-managed workflows | OSS requires operator-managed maintenance; Enterprise is moving more of that work into the platform as support expands. |
| **Data format** | Supports multiple available standards | Supports multiple available standards | No vendor lock-in; data moves freely between editions. |
| **Deployment** | Embedded in your code | BYOC or Managed | Enterprise meets uptime, compliance, and support goals that OSS cannot. |
### Architecture and scale
LanceDB OSS is directly embedded into your service. The process owns all CPU, memory, and storage, so scale is limited
to what one host can provide.
LanceDB Enterprise separates routing, query execution, and background work across a cluster. You can add capacity by
adding nodes, and the platform can keep serving traffic even when individual nodes are unhealthy.
Read More: [LanceDB Enterprise Architecture](/enterprise/architecture/)
### Latency of data retrieval
With LanceDB OSS, read latency depends heavily on where the data lives. If you use local disk or shared file storage, reads can be quite fast. But if you point an embedded deployment at S3, GCS, or Azure Blob, every read still takes a full round trip to remote object storage, especially when the data is cold.
LanceDB Enterprise is designed for the object-storage-backed case. It uses NVMe SSDs as a hybrid cache and spreads reads across a distributed serving layer, so repeated reads can skip the full object-store round trip. The first read fills the cache, subsequent reads come from local disk, and parallel chunked reads further reduce tail latency. This matters when the application serves interactive dashboards, real-time recommendations, or other latency-sensitive workloads on top of object storage.
Read More: [LanceDB Enterprise Benchmarks](/enterprise/benchmarks/)
### Throughput of search queries
A single LanceDB OSS process shares one CPU pool with the rest of the application. When concurrent queries hit that CPU, retrieval and similarity processes compete for cores. The server cannot process more work in parallel and any extra traffic waits in the queue, raising latency without increasing queries per second.
LanceDB Enterprise distributes queries across many execution nodes. A load balancer assigns queries to the least-loaded
node, so throughput grows as more nodes join the cluster instead of stalling at a single-process ceiling.
### Caching of commonly retrieved data
LanceDB OSS has no built-in cache. Every read repeats the same object-store round trip and pays the same latency penalty.
LanceDB Enterprise shards a cache across the fleet with consistent hashing. Popular vectors remain on local NVMe drives until they age out under a least-recently-used policy. Cache misses fall back to the object store, fill the local shard, and serve future reads faster. This design slashes both latency and egress cost for workloads with temporal locality.
### Maintenance of vector indexes
Vector indexes fragment when data is inserted, updated, or deleted. Fragmentation slows queries because the engine must
scan more blocks. LanceDB OSS offers a CLI call to compact or rebuild the index, but you must schedule it yourself.
LanceDB Enterprise is designed to move more of this maintenance into background platform workflows so operators spend
less time managing it manually. We are continuing to expand distributed indexing and compaction support for the
largest workloads.
Read More: [Indexing in LanceDB](/indexing/)
### Deployment and governance
When you work with LanceDB OSS, it is included as part of your binary, Docker, or serverless function. The footprint is small, and no extra services run beside it.
LanceDB Enterprise comes in two flavors. The BYOC deployment installs the system inside your VPC, so data never leaves
your account. The managed option hands day-to-day operations to the vendor, including patching, scaling, and ongoing
monitoring. Both enterprise modes are designed for private networking, compliance, and operational oversight.
Read More: [LanceDB Enterprise Deployment](/enterprise/deployment/)
## Usage differences between Enterprise and OSS
The [quickstart](/quickstart) guide shows both local embedded connections and Enterprise `db://...`
connections. Once connected to LanceDB, the table API is largely the same: create a table, search,
filter, evolve the schema, and store multimodal records. However, there are some semantic differences
worth understanding when your code is talking to LanceDB Enterprise.
### 1. Connection model
In LanceDB Enterprise, your app connects via a `db://...` URI and sends requests to the cluster API.
The cluster executes table operations on your behalf. Your code is coupled to a **managed service endpoint**,
whereas embedded LanceDB is directly coupled to local or object-storage paths.
### 2. Returned table type
Connecting to an Enterprise table via `open_table(...)` returns a `RemoteTable`, unlike embedded LanceDB,
which returns a `LanceTable`. `RemoteTable` is a catalog-backed table accessed through a server/cluster,
and does not support all the same methods as `LanceTable` (see below).
### 3. Materialization APIs
For Python users working with LanceDB Enterprise, `RemoteTable` does not support table-level
materialization methods like `table.to_arrow()` or `table.to_pandas()`. This protects users from
accidentally materializing tables that are too large to fit in memory.
Instead, materialize results through query/search builders, for example
`table.search(...).limit(...).to_pandas()` or `table.query(...).to_arrow()`. For quick previews, use
`table.head()`.
### 4. Maintenance lifecycle
In Enterprise, maintenance operations like `optimize` and `compact_files` are handled by the cluster
as background work. You can trigger them manually, but they are not required for performance or
correctness in the same way they are in embedded LanceDB.
That means maintenance is managed by platform behavior and cluster configuration, not by explicit
per-table maintenance calls in your application code.
### 5. Guardrails and limits
Enterprise can enforce platform-level guardrails, such as index/table limits and safety checks around
operations like `merge_insert` when too many rows are unindexed. Embedded LanceDB mostly exposes
storage/format-level behavior, and you tune many lifecycle tasks yourself.
This means an operation in LanceDB Enterprise can fail due to service-level policy, not just because
of local table shape or schema mismatch.
### 6. Cluster-managed background work
In Enterprise, async writes and reindexing workflows are handled by cluster background systems. In
embedded LanceDB, if you want ongoing upkeep, you usually schedule and run it yourself in your
application or jobs.
In practice, your app issues table operations, and the platform handles distributed orchestration for
maintenance and indexing in the background.
As a rule of thumb, all you need to remember is this: treat `db://...` as a remote service boundary,
use query builders to fetch results, and otherwise interact with your tables as you would in embedded
LanceDB.
## Which one should I use?
[It's very simple to get started with OSS](/quickstart/): Get started with `pip install lancedb` and begin ingesting
your data and vectors into LanceDB. LanceDB OSS makes sense when your dataset fits on one machine, traffic is still
modest, and your team is comfortable handling maintenance tasks such as compaction or reindexing itself.
Move to LanceDB Enterprise when retrieval becomes shared infrastructure for your business: your data or traffic has
outgrown a single machine, you need private deployment options, or you want platform support for monitoring, security,
and large-scale feature workflows.
If these sound like your use cases, [reach out to us](mailto:contact@lancedb.com) and we can help you scope your workload and arrange an Enterprise proof of concept.
# Security & Compliance
Source: https://docs.lancedb.com/enterprise/security
Learn about LanceDB Enterprise security features and best practices.
LanceDB Enterprise maintains high security standards with SOC 2 Type II, HIPAA, and GDPR compliance. Our security framework is designed to provide enterprise-grade protection for your data and workloads across deployment models.
## Security Certifications
* **SOC 2 Type II**: Independent audit confirming our security controls and operational effectiveness
* **HIPAA Compliance**: Certified to handle protected health information (PHI) in healthcare applications
* **GDPR Compliance**: Supports organizations with data privacy requirements under the General Data Protection Regulation
* **Regular Audits**: Ongoing security assessments to maintain compliance standards
### Ongoing Compliance
LanceDB maintains SOC 2 Type II, HIPAA, and GDPR compliance through ongoing audits and continuous improvement of our security practices as standards and risks evolve.
Visit the [LanceDB Trust Center](https://trust.lancedb.com/) to learn more about LanceDB's security posture, data privacy practices, and to request access to security documentation.
## LanceDB Enterprise
### Data Security
Customer data is strictly protected and remains within the confines of your account.
We maintain rigorous data isolation and encryption protocols to ensure confidentiality.
LanceDB Enterprise only receives telemetry data for monitoring system health.
At LanceDB, customer data security is paramount.
### Encryption
LanceDB Enterprise safeguards your data through encryption at rest, preventing
unauthorized access. This comprehensive encryption covers all data stored within the
object store and cache.
# LanceDB Enterprise FAQ
Source: https://docs.lancedb.com/faq/faq-enterprise
Commonly asked questions about LanceDB Enterprise.
This section provides answers to the most common questions asked about LanceDB Enterprise. For assistance with LanceDB Enterprise, please [contact us](mailto:support@lancedb.com) via email and one of our
support staff will get back to you.
### Architecture and Fault Tolerance
#### What's the impact of losing each component (query node, indexer, etc.) in the LanceDB stack?
LanceDB Enterprise employs component-level replication to ensure fault tolerance and
continuous operations. While the system remains fully functional during replica
failures, transient performance impacts (e.g., elevated latency or reduced throughput)
may occur until automated recovery completes.\
For architectural deep dives, including redundancy configurations,
please contact the LanceDB team.
#### What does plan executor cache versus not cache?
The plan executor caches the table data, not the table indices.
#### Should I use disk cache or memory cache for the plan executor?
LanceDB implements highly performant consistent hashing for our plan executors.
NVMe SSD caching is enabled by default for all deployments.
#### How is the PE (Plan Executor) fleet shared? What fault tolerance exists (how many nodes can be lost)?
LanceDB's plan executor is typically deployed with 2+ replicas for fault tolerance:
* Mirrored Caches: Each query replica maintains synchronized copies of data subsets,
enabling low-latency query execution.
* Load Balancing: Traffic is distributed evenly across replicas.
With a single replica failure, there is no downtime - the system remains
operational with degraded performance, as the remaining
replicas will handle all the traffic until the failed replica comes back online.
### Consistency
#### How is strong/weak consistency configured in the enterprise stack?
By default, LanceDB Enterprise operates in strong consistency mode.
Once a write is successfully acknowledged, a new Lance dataset version manifest
file is created. Subsequent reads always load the latest manifest file to
ensure the most up-to-date data.
However, this increases query latency and can place significant load on the storage system
under high concurrency. We offer the `weak_read_consistency_interval_seconds` parameter
to adjust consistency level (whose default value is zero). This parameter Defines the interval
(in seconds) at which the system checks for table updates from other processes.
**Recommended Setting**
To balance consistency and performance, setting `weak_read_consistency_interval_seconds` to 30–60 seconds is often a
good trade-off. This reduces unnecessary cloud storage operations while still
keeping data reasonably fresh for most applications.
Note that **this setting only affects read operations**. Write operations always remain strongly consistent.
### Indexing
#### Can I use GPU for indexing?
Yes! Please [contact](mailto:support@lancedb.com) the LanceDB team to enable GPU-based indexing for
your deployment. Then you just need to call `create_index`, and the backend will use GPU for indexing.
LanceDB is able to index a few billion vectors under 4 hours.
### Cluster Configuration
#### What are the parameters that can be configured for my LanceDB cluster?
LanceDB Enterprise offers granular control over performance, resilience, and
operational behavior through a comprehensive set of parameters: replication factors for
each component, consistency level, graceful shutdown time intervals, etc. Please
contact the LanceDB team for detailed documentation on such parameter configurations.
### Monitoring and Alerts
#### What are the metrics that LanceDB exposes for monitoring?
We have various metrics set up for monitoring each component in the LanceDB stack:
* Query node: RPS, query latency, error codes, slow take count, CPU/memory utilization, etc.
* Plan executor: SSD cache hit/miss, CPU/memory utilization, etc.
Please contact the LanceDB team for the comprehensive list of monitoring metrics.
#### How do I integrate LanceDB's monitoring metrics with my monitoring dashboard?
LanceDB uses Prometheus for metrics collection and OpenTelemetry (OTel) to export such
metrics with data enrichment. The LanceDB team will work with you to integrate the
monitoring metrics with your preferred dashboard.
# LanceDB: Frequently Asked Questions
Source: https://docs.lancedb.com/faq/faq-oss
Commonly asked questions about LanceDB OSS.
This section covers some common questions and issues that you may encounter when using LanceDB.
### Is LanceDB open source?
LanceDB OSS is a permissively licensed embedded retrieval library available under an Apache 2.0 license. We also have a LanceDB Enterprise, a commercial product that can be deployed on a private cloud or a bring-your-own-cloud (BYOC) solution. LanceDB Enterprise transforms your data lake into a high-performance multimodal lakehouse.
### What is the difference between Lance and LanceDB?
[Lance](https://github.com/lancedb/lance) is a modern lakehouse format for multimodal AI. It's perfect for building search engines, feature stores and being the foundation of large-scale ML training jobs requiring high performance IO and shuffles. It also has native support for storing, querying, and inspecting deeply nested data for robotics or large blobs like images, point clouds, and more.
LanceDB is the multimodal lakehouse that's built on top of Lance, and utilizes the underlying optimized storage format to build efficient disk-based indexes that power semantic search & retrieval applications, from RAGs to QA bots to recommender systems.
### Why invent another data format instead of using Parquet?
As we mention in our talk titled "[Lance, a modern columnar data format](https://www.youtube.com/watch?v=ixpbVyrsuL8)", Parquet and other tabular formats that derive from it are rather dated (Parquet is over 10 years old), especially when it comes to random access on vectors. We needed a format that's able to handle the complex trade-offs involved in shuffling, scanning, OLAP and filtering large datasets involving vectors, and our extensive experiments with Parquet didn't yield sufficient levels of performance for modern ML. [Our benchmarks](https://lancedb.com/blog/benchmarking-random-access-in-lance/) show that Lance is up to 1000x faster than Parquet for random access, which we believe justifies our decision to create a new data format for AI.
### Why build in Rust?
We believe that the Rust ecosystem has attained mainstream maturity and that Rust will form the underpinnings of large parts of the data and ML landscape in a few years. Performance, latency and reliability are paramount to a vector DB, and building in Rust allows us to iterate and release updates more rapidly due to Rust's safety guarantees. Both Lance (the data format) and LanceDB (the database) are written entirely in Rust. We also provide Python, JavaScript, and Rust client libraries to interact with the database.
### What makes LanceDB different?
LanceDB is among the few embedded vector DBs out there that we believe can unlock a whole new class of LLM-powered applications in the browser or via edge functions. Lance's multimodal nature allows you to store the raw data, metadata and the embeddings all at once, unlike other solutions that typically store just the embeddings and metadata.
The Lance data format that powers our storage system also provides true zero-copy access and seamless interoperability with numerous other data formats (like Pandas, Polars, Pydantic) via Apache Arrow, as well as automatic data versioning and data management without needing extra infrastructure.
### How large of a dataset can LanceDB handle?
LanceDB and its underlying data format, Lance, are built to scale to really large amounts of data. LanceDB OSS can comfortably handle millions of vectors on a single node, making it a great fit for most applications. Its disk-based indexes keep performance strong without requiring expensive infrastructure.
If you need to scale to hundreds of millions of vectors or work with terabytes of data, we recommend [LanceDB Enterprise](/enterprise). Enterprise customers regularly operate on billions of rows, backed by distributed infrastructure designed for large-scale production workloads.
### Do I need to build a vector index to run vector search?
No. LanceDB is blazing fast (due to its disk-based index) for even brute force kNN search, within reason. In our benchmarks, computing 100K pairs of 1000-dimension vectors takes less than 20ms. For small datasets of \~100K records or applications that can accept \~100ms latency, a vector index is usually not necessary.
For large-scale (>1M) or higher dimension vectors, it is beneficial to create a vector index. See the [Vector Indexes](/indexing/vector-index/) section for more details.
### How can I speed up data inserts?
LanceDB auto-parallelizes large writes when you call `table.add()` with materialized
data such as `pa.Table`, `pd.DataFrame`, or `pa.dataset()`. No extra configuration
is needed — writes are automatically split into partitions of \~1M rows or 2GB.
For best results:
* **Create an empty table first**, then call `table.add()`. The `add()` path enables
automatic write parallelism, while passing data directly to `create_table()` does not.
* **For file-based data**, use `pyarrow.dataset.dataset("path/to/data/", format="parquet")`
so LanceDB can stream from disk without loading everything into memory.
* **Avoid inserting one row at a time.** Each insert creates a new data fragment on
disk. Batch your data into Arrow tables, DataFrames, or use iterators.
See [Loading Large Datasets](/tables/create#loading-large-datasets) for full examples.
### Do I need to set a refine factor when using an index?
Yes. LanceDB uses PQ, or Product Quantization, to compress vectors and speed up search when using an ANN index. However, because PQ is a lossy compression algorithm, it tends to reduce recall while also reducing the index size. To address this trade-off, we introduce a process called **refinement**. The normal process computes distances by operating on the compressed PQ vectors. The refinement factor (*rf*) is a multiplier that takes the top-k similar PQ vectors to a given query, fetches `rf * k` *full* vectors and computes the raw vector distances between them and the query vector, reordering the top-k results based on these scores instead.
For example, if you're retrieving the top 10 results and set `refine_factor` to 25, LanceDB will fetch the 250 most similar vectors (according to PQ), compute the distances again based on the full vectors for those 250 and then re-rank based on their scores. This can significantly improve recall, with a small added latency cost (typically a few milliseconds), so it's recommended you set a `refine_factor` of anywhere between 5-50 and measure its impact on latency prior to deploying your solution.
### How can I improve IVF-PQ recall while keeping latency low?
When using an IVF-PQ index, there's a trade-off between recall and latency at query time. You can improve recall by increasing the number of probes and the `refine_factor`. In our benchmark on the GIST-1M dataset, we show that it's possible to achieve >0.95 recall with a latency of under 10 ms on most systems, using \~50 probes and a `refine_factor` of 50. This is, of course, subject to the dataset at hand and a quick sensitivity study can be performed on your own data. You can find more details on the benchmark in a past [blog post](https://medium.com/etoai/benchmarking-lancedb-92b01032874a).
![]()
### How much data can LanceDB practically manage without affecting performance?
We target good performance on \~10-50 billion rows and \~10-30 TB of data. For the best performance and
scalability guarantees, check out [LanceDB Enterprise](/enterprise).
### Does LanceDB support concurrent operations?
LanceDB can handle concurrent reads very well, and can scale horizontally. The main constraint is how well the storage layer you've chosen, scales. For writes, we support concurrent writing, though too many concurrent writers can lead to failing writes as there is a limited number of times a writer retries a commit.
If you use Python's multiprocessing, you should probably not use `fork` as Lance is multi-threaded
internally and `fork` and multi-threaded Python do not work well together.
[Refer to this discussion](https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555)
for more information.
# Frequently Asked Questions
Source: https://docs.lancedb.com/faq/index
Common questions about LanceDB
Find answers to common questions about LanceDB across different deployment options and use cases.
Reach out on [Discord](https://discord.gg/AUEWnJ7Txb) for community support or [contact us](mailto:support@lancedb.com) for Enterprise assistance.
| Category | Description |
| :---------------------------------------- | :---------------------------------------------------------------------------------- |
| [LanceDB OSS](/faq/faq-oss) | Questions about LanceDB open source deployment, installation, and community support |
| [LanceDB Enterprise](/faq/faq-enterprise) | Questions about LanceDB Enterprise features, security, compliance, and support |
# Dependency Verification
Source: https://docs.lancedb.com/geneva/deployment/dependency-verification
Diagnose and resolve package version mismatches between local and distributed worker environments.
When running Geneva UDFs on distributed workers, your code is serialized locally and executed on remote workers. If the worker environment differs from your local environment, you may encounter subtle and difficult-to-debug errors.
## Example environment mismatch errors
| Symptom | Likely Cause |
| ------------------------------------------------------------------ | ----------------------------------------- |
| `TypeError: Enum.__new__() missing 1 required positional argument` | `attrs` version mismatch |
| `TypeError: Can't instantiate abstract class` | Package structure differences |
| `ArrowInvalid: cannot cast` / serialization errors | NumPy 1.x vs 2.x mismatch |
| `ModuleNotFoundError` on workers | Package only installed locally |
| Model loading failures | PyTorch version mismatch |
| Permission denied errors | Missing API keys in envrionment variables |
These issues are notoriously difficult to debug because the error messages often don't indicate the root cause.
## The `compare_ray_environments` Tool
Geneva provides a diagnostic tool to compare your local environment against Ray workers.
If you are encountering a hang or exception you can use the following diagnosis worklflow to resolve the problem.
**Run the diagnostic tool** programatically or via the CLI.
**Check PACKAGES and ENV VARS output sections for mismatches**.
**Identify critical packages**: numpy, torch, pyarrow, attrs, pydantic.
**Identify inconsistent environment variables**: `AWS_*`, `GOOGLE_APPLICATION_CREDENTIALS`
**Fix with manifest** for quick testing:
**OPTIONAL: Build custom image** for production (if using KubeRay).
### Programmatic Usage
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.runners.ray.compare_env import compare_ray_environments
# Compare and print (requires Geneva context to be initialize via `with db.context(..)`)
result = compare_ray_environments()
# Compare environments, filtering environment variables with specified prefix
result = compare_ray_environments(env_prefix="PY")
```
### CLI Usage
```bash CLI icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Connect to existing Ray cluster
python -m geneva.runners.ray.compare_env
# Start new local Ray cluster
python -m geneva.runners.ray.compare_env --address local
# Filter env vars by prefix
python -m geneva.runners.ray.compare_env --env-prefix RAY
# Show full JSON snapshots
python -m geneva.runners.ray.compare_env --show-all
# Skip sys.path comparison
python -m geneva.runners.ray.compare_env --no-sys-path
```
## Understanding the Output
The tool outputs several sections to help you identify mismatches.
### PYTHON / PLATFORM
Shows Python version and OS information for both environments:
```
=== PYTHON / PLATFORM ===
Local:
Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
Impl : CPython
Exec : /home/user/.venv/bin/python
OS : Linux 5.15.0-generic (x86_64)
Remote:
Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
Impl : CPython
Exec : /home/ray/anaconda3/bin/python
OS : Linux 5.4.0-aws (x86_64)
```
Watch for different Python versions or different OS types (macOS local vs Linux remote).
#### Architecture Mismatch (macOS to Linux)
If you see different OS types (e.g., `Darwin` locally vs `Linux` remotely), compiled extensions may fail with `ModuleNotFoundError` or segfaults.
**Solution**: Run Geneva from the same OS/architecture as your cluster (typically Linux x86\_64). Use a Linux VM, container, or remote development environment.
### Environment Variables
Environment variables present in only one environment:
```
=== ENV VARS: keys only in LOCAL ===
+ CONDA_PREFIX
+ VIRTUAL_ENV
=== ENV VARS: keys only in REMOTE ===
+ RAY_ADDRESS
+ KUBERNETES_SERVICE_HOST
```
Missing `AWS_*` or `GOOGLE_APPLICATION_CREDENTIALS` can cause storage authentication failures.
#### Passing Environment Variables to Workers
If critical environment variables are missing on workers, you can pass them via the manifest or cluster configuration.
**Option 1: Via Manifest**
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.manifest.builder import PipManifestBuilder
import os
manifest = (
PipManifestBuilder.create("my-manifest")
.env_vars({
"AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
"AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
"MY_API_KEY": os.environ["MY_API_KEY"],
})
.build()
)
```
**Option 2: Via Cluster Configuration**
Avoid hardcoding secrets. Use `os.environ` to pass values from your local environment, or use a secrets manager in production.
### Packages
The tool shows version mismatches and packages only present in one environment:
```
=== PACKAGES: version mismatches ===
* numpy: local=1.26.4 remote=2.2.6
* torch: local=2.0.1 remote=2.8.0+cpu
* attrs: local=23.2.0 remote=24.2.0
* pyarrow: local=14.0.1 remote=17.0.0
=== PACKAGES: only in LOCAL ===
+ my-custom-package
+ dev-tools
=== PACKAGES: only in REMOTE ===
+ kuberay-client
```
Watch for major version differences (NumPy 1.x vs 2.x) and PyTorch version mismatches.
#### Common Package Issues
| Issue | Symptoms | Fix |
| -------------------- | -------------------------------------------------------------------- | ------------------------------- |
| **NumPy 1.x vs 2.x** | `ArrowInvalid`, `ValueError: cannot convert`, serialization failures | Pin `numpy==1.26.4` |
| **PyTorch mismatch** | Model loading failures, CUDA errors, unexpected inference results | Pin to matching `torch` version |
| **attrs mismatch** | `TypeError: Enum.__new__() missing 1 required positional argument` | Pin `attrs` to local version |
| **Missing package** | `ModuleNotFoundError: No module named 'xyz'` | Add package to manifest |
#### Fixing Package Mismatches
**Option 1: Manifest pip Dependencies**
Specify packages in a Geneva manifest for a quick fix:
*Pros*: Quick, reusable across sessions, stored with your database.
*Cons*: Slower startup (downloads packages), may not work for complex dependencies.
**Option 2: Custom Ray Worker Image**
For KubeRay deployments, build a custom worker image:
```dockerfile Dockerfile icon="docker" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Dockerfile.ray-worker
FROM rayproject/ray:2.30.0-py311
# Install exact versions
RUN pip install \
numpy==1.26.4 \
torch==2.0.1 \
attrs==23.2.0 \
geneva==0.8.0
# Copy any custom packages
COPY ./my_udfs /app/my_udfs
```
Then reference in RayCluster spec:
```yaml Kubernetes icon="kubernetes" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
spec:
workerGroupSpecs:
- template:
spec:
containers:
- image: your-registry/ray-worker:latest
```
*Pros*: Fastest startup, reproducible.
*Cons*: Requires image build/push workflow.
**Option 3: Conda Environment**
Use a conda environment on workers via the cluster builder:
Or specify conda channels and dependencies inline:
*Pros*: Best for complex dependencies with native libraries (ffmpeg, CUDA).
*Cons*: Slower environment creation, requires conda on workers.
## API Reference
For detailed API documentation on the environment comparison functions, see the [Geneva Diagnostics API Reference](https://lancedb.github.io/geneva/api/diagnostics).
# Deploy Geneva using Helm
Source: https://docs.lancedb.com/geneva/deployment/helm
Learn how to deploy Geneva on Kubernetes using the Geneva Helm Chart
**Feature Engineering is deployed automatically in LanceDB Enterprise**
In self-managed environments, Geneva can be installed into existing Kubernetes clusters using Helm. Please [contact LanceDB](https://lancedb.com/contact/) for access to the Helm Chart and related resources.
## Pre-requisites
* An existing Kubernetes cluster
* An existing node pool(s) for Geneva workloads. By default, Geneva uses node selector
`{"geneva.lancedb.com/ray-head": "true"}` for Ray head nodes, and
`{"geneva.lancedb.com/ray-worker-cpu": "true"}` and `{"geneva.lancedb.com/ray-worker-gpu": "true"}`
for Ray CPU worker and Ray GPU worker nodes respectively. This can be overridden in the Geneva client.
* Geneva Helm chart. Please [contact LanceDB](https://lancedb.com/contact/) for access to the Helm Chart and related resources.
For more information on deploying the required cloud resources, see the [manual deployment instructions](/geneva/deployment/).
## Geneva Helm Chart
The Helm chart includes resources required for running [Geneva](https://lancedb.com/docs/geneva/) in Kubernetes.
It includes services, service accounts, RBAC roles, etc. that are used by the Geneva client to manage resources.
## Install
1. Authenticate with Kubernetes cluster, i.e. update kubeconfig
2. Configure Helm chart values
In values.yaml, configure the service account, node selectors, and cloud resources, if applicable.
```
geneva:
# Object storage root URI
rootUri:
value: "s3://my-data-bucket"
serviceAccount:
# Service account for Geneva worker pods and services
annotations:
# Set per-CSP annotations to provide access to CSP resources, i.e.
# eks.amazonaws.com/role-arn: arn:aws:iam::0123456789:role/geneva_service_role
# iam.gke.io/gcp-service-account: geneva-service-account@my-project.iam.gserviceaccount.com
gcp:
# GCP service account email for the Geneva client.
# It should have access to the GKS cluster and "roles/storage.objectUser"
# permissions on the object storage bucket.
# e.g., geneva-client-sa@project-id.iam.gserviceaccount.com
clientServiceAccount: ""
aws:
# AWS IAM role ARN to be assumed by the Geneva client.
# This role should have an access entry to the cluster with username matching the role ARN.
# It should also have r/w access to the object storage bucket.
# e.g., arn:aws:iam::123456789012:role/geneva-client-role
clientRoleArn: ""
azure:
# Azure managed identity client ID for the Geneva client.
# This identity should have a federated credential for the LanceDB namespace
# and Storage Blob Data Contributor role on the storage account.
clientPrincipalId: ""
```
3. Install kuberay operator
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
export NAMESPACE=lancedb
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
helm install kuberay-operator kuberay/kuberay-operator -n $NAMESPACE --create-namespace
```
4. Install NVIDIA device plugin (if using GPU nodes)
For GPU support, the NVIDIA device plugin must be installed in your EKS cluster:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml > nvidia-device-plugin.yml
kubectl apply -f nvidia-device-plugin.yml
```
5. Install Geneva Helm chart
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
helm install geneva ./geneva -n $NAMESPACE --create-namespace
```
## Default cluster and manifest
In LanceDB Enterprise, backfill and refresh jobs run on a **default cluster** (the compute
pool jobs run on) and a **default manifest** (the Python dependency environment — image and
packages). Configuring these in the LanceDB Enterprise chart lets jobs run out of the box
without per-job configuration. They are set under `geneva.defaults` in the chart's
`values.yaml`:
```yaml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
geneva:
defaults:
cluster:
cluster_type: external_ray
name: deployment-default
ray_address: "ray://raycluster-kuberay-head-svc.lancedb.svc.cluster.local:10001"
manifest:
name: deployment-default
pip: [geneva, pyarrow, lancedb, pylance]
head_image: rayproject/ray:2.54.0-py312
worker_image: rayproject/ray:2.54.0-py312
skip_site_packages: true
```
If no default is configured, jobs must specify a manifest explicitly. Individual transforms can override the default manifest by pinning one
with `@udf` / `@chunker` / `@udtf` (see
[Advanced Job Configuration](/geneva/jobs/advanced-job-configuration)); to override the cluster
at runtime, use an [Advanced Execution Context](/geneva/jobs/contexts).
## Providing a Ray cluster
The LanceDB Helm chart can be configured to deploy a static KubeRay cluster, provision KubeRay clusters on demand per job, or
use an existing Ray cluster.
### Use default LanceDB Enterprise Ray cluster (default)
By default, LanceDB Enterprise will use a shared, statically provisioned Ray cluster for job execution.
This can be enabled in the Helm chart by setting the following values.
```yaml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
raycluster:
enabled: true
global:
rayclusterUri: "ray://raycluster-kuberay-head-svc.lancedb.svc.cluster.local:10001"
```
Configuration for the Ray cluster can be specified by modifying raycluster.yaml Helm values.
### Provision KubeRay clusters on demand
Set `global.rayclusterUri` to an empty value to provision ephemeral KubeRay clusters on-demand for each execution job. The default KubeRay cluster configuration
is specified in `geneva.defaults.cluster`, i.e.
```yaml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
geneva:
defaults:
cluster:
cluster_type: kuberay
name: deployment-default
kuberay:
namespace: lancedb
config_method: IN_CLUSTER
head_group:
service_account: geneva-service-account
num_cpus: 2
memory: 8Gi
image: rayproject/ray:2.54.0-py312
worker_groups:
- name: cpu
service_account: geneva-service-account
num_cpus: 4
memory: 8Gi
replicas: 2
min_replicas: 0
max_replicas: 4
idle_timeout_seconds: 60
node_selector:
geneva.lancedb.com/ray-worker-cpu: "true"
image: rayproject/ray:2.54.0-py312
```
### Use an external Ray cluster
Self-managed enterprise customers can bring an existing Ray cluster to run Geneva jobs. Simply set the rayclusterUri property in the Helm chart
to a Ray address that can be accessed from the LanceDB Enterprise deployment.
```yaml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
global:
rayclusterUri: "ray://my-ray-cluster.my-ns.svc.cluster.local:10001"
```
# Manual Deployment on Kubernetes
Source: https://docs.lancedb.com/geneva/deployment/index
Learn how to deploy Geneva on Kubernetes using KubeRay for distributed feature engineering workflows on GKE and EKS.
**Feature Engineering is deployed automatically in LanceDB Enterprise**
For manual installation in self-managed environments, follow the instructions below.
Feature Engineering can be deployed as part of LanceDB Enterprise in managed or self-managed environments. First class support is provided for Azure, AWS, and GCP, including deployment automation via Terraform and Helm.
## Prerequisites
* Kubernetes cluster with KubeRay 1.1+ operator installed
* Ray 2.43+
See below for manual installation instructions for:
* Amazon Web Services (AWS) Elastic Kubernetes Service (EKS)
* Google Cloud Platform (GCP) Google Kubernetes Engine (GKE)
## Basic Kubernetes Setup
Kubernetes resources can be deployed automatically via [Helm](/geneva/deployment/helm/) or manually
via the instructions below.
In the following sections we'll use these variables:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
NAMESPACE=lancedb # replace with your actual namespace if different
KSA_NAME=geneva-ray-runner # replace with an identity name
```
### Kubernetes Service Account (KSA)
Inside your Kubernetes cluster, you need a Kubernetes service account which provides the credentials your k8s pods (Ray) run with. Here's how to create your KSA.
#### Create a Kubernetes Service Account (KSA)
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
kubectl create namespace $NAMESPACE # skip if it already exists
kubectl create serviceaccount $KSA_NAME \
--namespace $NAMESPACE
```
You can verify using:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
kubectl get serviceaccounts -n $NAMESPACE $KSA_NAME
```
The Kubernetes service account (KSA) needs RBAC permissions inside the k8s cluster to provision Ray clusters via CRDs.
#### Create a k8s Role
Create a k8s role that can access the Ray CRD operations.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
kubectl apply -f - <
#### Geneva Security Requirements
In the following sections we'll use these variables:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
NAMESPACE=lancedb # replace with your actual namespace if different
KSA_NAME=geneva-ray-runner # replace with an identity name
PROJECT_ID=... # replace with your google cloud project name
GSA_EMAIL=${KSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
LANCEDB_URI=gs://bucket/db # replace with your own path
```
#### Google Service Account (GSA)
To give your k8s workers the ability to read and write from your LanceDB buckets, your KSA needs to be bound to a Google Cloud service account (GSA) with those grants. With this setup, any pod using the KSA will automatically get a token that lets it impersonate the GSA.
Let's set this up:
**Create a Google Cloud Service Account**
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
gcloud iam service-accounts create ${KSA_NAME} \
--project=${PROJECT_ID} \
--description="Service account for ray workloads in GKE" \
--display-name="Ray Runner GSA"
```
You can verify this using:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
gcloud iam service-accounts list --filter="displayName:Ray Runner GSA"
```
> **Warning**: You need `roles/iam.serviceAccountAdmin` or minimally `roles/iam.serviceAccountTokenCreator` rights to run these commands.
Next, you'll need to verify that your KSA is bound to your GSA and has `roles/iam.workloadIdentityUser`:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
gcloud iam service-accounts get-iam-policy $GSA_EMAIL \
--project=$PROJECT_ID \
--format="json" | jq '.bindings[] | select(.role=="roles/iam.workloadIdentityUser")'
```
Give your GSA rights to access the LanceDB bucket:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
gcloud storage buckets add-iam-policy-binding ${LANCEDB_URI} \
--member="serviceAccount:${KSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/storage.objectAdmin"
```
#### GKE Workload Identity
A GKE workload identity is required to enable k8s workloads access Google Cloud services securely and without needing to manually manage service account keys. The workload identity is attached to Google Cloud service accounts (GSA) and mapped to a Kubernetes service account (KSA). This feature needs to be enabled on the GKE cluster.
You can confirm that your workers have abilities to read/write to the LanceDB bucket:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
kubectl run gcs-test --rm -it --image=google/cloud-sdk:slim \
--serviceaccount=${KSA_NAME} \
-n ${NAMESPACE} \
-- bash
echo "hello" > test.txt
gsutil cp test.txt ${LANCEDB_URI}/demo-check/test-write.txt
```
Confirm the identity inside the pod:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email
```
## Geneva on AWS EKS
Geneva can be used to provision Ray clusters running in Amazon Web Services (AWS) Elastic Kubernetes Service (EKS).
In the following sections we'll use these variables:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
NAMESPACE=lancedb # replace with your actual namespace if different
CLUSTER=geneva # replace with your actual namespace if different
KSA_NAME=geneva-ray-runner # replace with an identity name
```
### EKS Node Groups
EKS allows you to specify templates for virtual machines in "node groups". These allow you to manage and configure resources such as the number of CPUs, number of GPUs, amount of memory, and if instances are spot or regular virtual machines.
You can define your node groups however you want but Geneva uses three specific Kubernetes labels when deploying Ray pods on EKS: `ray-head`, `ray-worker-cpu`, `ray-worker-gpu`
* **Head nodes** are where the Ray dashboard and scheduler run. They should be non-spot instances and should not have processing workloads scheduled on them. Geneva looks for nodes with the `geneva.lancedb.com/ray-head: true` k8s label for this role.
* **CPU Worker nodes** are where distributed processing that does not require GPU should be scheduled. Geneva looks for nodes with the `geneva.lancedb.com/ray-worker-cpu: true` k8s label when these nodes are requested.
* **GPU Worker nodes** are where distributed processing that require GPU should be scheduled. Geneva looks for nodes with the `geneva.lancedb.com/ray-worker-gpu: true` k8s label when these nodes are requested.
### Install KubeRay Operator Using Helm
Geneva requires the KubeRay operator to be installed in your EKS cluster.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
helm install kuberay-operator kuberay/kuberay-operator -n $NAMESPACE
```
### Install NVIDIA Device Plugin
For GPU support, the NVIDIA device plugin must be installed in your EKS cluster:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml > nvidia-device-plugin.yml
kubectl apply -f nvidia-device-plugin.yml
```
### Configure Access Control
![eks-auth]()
#### Environment IAM Principal
Geneva must be run in an environment with access to [AWS credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) with permissions to `sts:AssumeRole` on the Geneva Client IAM Role.
For example, this could be a laptop with credentials provided by environment variables, or an EC2 instance with credentials provided via Instance Profile.
#### Create IAM Role for Geneva Client
The Geneva Client IAM Role is assumed by the Geneva client to provision the Kuberay cluster and run remote jobs.
This role requires IAM permissions to access the storage bucket and Kubernetes API.
Create an IAM role with the following policy:
```json theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ClusterAccess",
"Action": [
"eks:DescribeCluster",
"eks:AccessKubernetesApi"
],
"Effect": "Allow",
"Resource": ""
},
{
"Sid": "AllowListBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::"
},
{
"Sid": "AllowAllS3ObjectActions",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:HeadObject"
],
"Resource": "arn:aws:s3:::/*"
}
]
}
```
This role should also have a trust policy with `sts:AssumeRole` permissions for any principal initiating the Geneva client.
When using Geneva, this role can be specified with the `role_name` RayCluster parameter.
#### Create EKS Access Entry
Create an [EKS access entry](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html) to allow the Geneva Client Role to access the Kubernetes API for the EKS Cluster.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
aws eks create-access-entry --cluster-name $CLUSTER --principal-arn --type STANDARD
aws eks associate-access-policy --cluster-name $CLUSTER --principal-arn --access-scope type=cluster --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy
```
#### Create EKS OIDC Provider
Create an OIDC provider for your EKS cluster. This is required to allow Kubernetes Service Accounts (KSA) to assume IAM roles. See [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html#_create_oidc_provider_console).
#### Create IAM Role for Service Account
An IAM role is required for the Kubernetes Service Account (KSA) that will be used by the Ray head and worker pods.
This role must have permissions to access the storage bucket and to describe the EKS cluster:
```json theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ClusterAccess",
"Action": [
"eks:DescribeCluster"
],
"Effect": "Allow",
"Resource": ""
},
{
"Sid": "AllowListBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::"
},
{
"Sid": "AllowAllS3ObjectActions",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:HeadObject"
],
"Resource": "arn:aws:s3:::/*"
}
]
}
```
In addition, it must have a trust policy allowing the EKS OIDC provider to assume the role from the Kubernetes Service Account:
```json theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": ""
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
":aud": "sts.amazonaws.com",
":sub": "system:serviceaccount:$NAMESPACE:$KSA_NAME"
}
}
}
]
}
```
#### Associate the IAM Role with the Kubernetes Service Account
Modify the Kubernetes Service Account created in "Basic Kubernetes setup" to associate it with the IAM role created above.
The role ARN is specified using `eks.amazonaws.com/role-arn` annotation:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
kubectl annotate serviceaccount "$KSA_NAME" \
-n "$NAMESPACE" \
"eks.amazonaws.com/role-arn=$ROLE_ARN" \
--overwrite
```
### Initialize the Ray Cluster
Initialize the Ray cluster using the node selectors and metadata from above:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.runners.ray._mgr import ray_cluster
from geneva.runners.ray.raycluster import (K8sConfigMethod, _HeadGroupSpec, _WorkerGroupSpec)
head_spec = _HeadGroupSpec(
service_account="geneva-ray-runner",
num_cpus=1,
memory=2048,
node_selector={"geneva.lancedb.com/ray-head": "true"},
)
worker_spec = _WorkerGroupSpec(
name="worker",
min_replicas=1,
service_account="geneva-ray-runner",
num_cpus=2,
memory=4096,
node_selector={"geneva.lancedb.com/ray-worker-cpu": "true"},
)
with ray_cluster(
name="my-ray-cluster",
namespace="lancedb",
cluster_name="geneva",
config_method=K8sConfigMethod.EKS_AUTH,
region="us-east-1",
use_portforwarding=True,
head_group=head_spec,
worker_groups=[worker_spec],
role_name="geneva-client-role",
) as cluster:
table.backfill("embedding")
```
# Troubleshooting Geneva Deployments
Source: https://docs.lancedb.com/geneva/deployment/troubleshooting
Learn how to diagnose and resolve common issues with Geneva deployments, including version compatibility, permissions, and serialization errors.
We'll cover common problems you may encounter when using Geneva and troubleshooting tips to solve them.
## Common Issues to Verify
Here are some areas to verify to identify the source of problems with your Geneva deployment:
* **Versions compatibility** (Ray, Python, Lance)
* **Remote Ray execution** and hardware resource availability
* **Sufficient permissions** to access data
* **Worker code** only returns serializable values (no open files, no GPU resident buffers)
## Confirming Dependency Versions
Geneva uses Ray for distributed execution. Ray requires the version deployed cluster services and clients to be exactly the same. Minor versions of Python must match both on client and cluster services (e.g. 3.10.3 and 3.10.5 are ok, but 3.10.3 and 3.12.1 are not.)
Geneva has been tested with Ray 2.44+ and Python 3.10.x and 3.12.x.
You can run this code in your notebook to verify your environment matches your expectations:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
!python --version
!pip show lancedb # need 0.22.0b0+
!echo $VIRTUAL_ENV
```
## Confirming Remote Ray Execution
Geneva allows you to specify the resources of your worker nodes. You can verify that your cluster has the resources (e.g. GPUs) available for your jobs and that remote execution is working properly.
You can get some basic information about resources available to your Ray:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
print(ray.available_resources())
```
You can verify basic remote execution via Ray:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@ray.remote
def check_remote():
return "Hello from a worker"
print(ray.get(check_remote.remote()))
```
You can also verify that versions of specific libraries are present in the execution environment:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# does ray have cuda?
@ray.remote
def check_pyarrow():
import pyarrow
return pyarrow.__version__
print(ray.get(check_pyarrow.remote()))
```
> **Note**: You should execute Geneva code from a machine or VM that has the same architecture and OS type as the nodes in your cluster. This will allow for shared libraries to be shipped. For example, if you use a Mac to host a Jupyter notebook, Geneva will push Mac libraries to your Linux cluster and likely result in module not found errors due to OS/architecture differences.
For GPU-dependent UDFs and jobs, you can verify that GPU worker nodes have the CUDA library:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# does ray have cuda?
@ray.remote(num_gpus=1)
def check_cuda():
import torch
return torch.version.cuda, torch.cuda.is_available()
print(ray.get(check_cuda.remote()))
```
## Confirming Sufficient Permissions
While your notebook or working environment may have credentials to read and write to particular buckets, your jobs need sufficient rights to read and write to them as well. Adding `import geneva` to any remote function can help verify that your workers have sufficient grants.
Here we add `import geneva` to help trigger potential permissions problems:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@ray.remote(num_gpus=1)
def check_cuda():
import geneva # this is currently required before other imports
import torch
return torch.version.cuda, torch.cuda.is_available()
print(ray.get(check_cuda.remote()))
```
### GCE Permissions Errors in Job Logs
If you are using Geneva managed Ray deployed on GKE, the errors may look like this:
```
PermissionError: [Errno 13] google::cloud::Status(PERMISSION_DENIED: Permanent error, with a last message of Caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist). error_info={reason=forbidden, domain=global, metadata={gcloud-cpp.retry.function=GetObjectMetadata, gcloud-cpp.retry.reason=permanent-error, gcloud-cpp.retry.original-message=Caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist)., http_status_code=403}}). Detail: [errno 13] Permission denied
```
This indicates that your workers and/or head node are not being run with the correct service account or with an account that has sufficient access. Please double check the service account's accesses and make sure to add your service account that has access to the buckets as a parameter to your Head and Worker specs. See `service_account="geneva-integ-test"` below:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
raycluster = ray_cluster(
name= k8s_name,
namespace=k8s_namespace,
use_portforwarding=True,
head_group=_HeadGroupSpec(
num_cpus=8,
service_account="geneva-integ-test"
),
worker_groups=[
_WorkerGroupSpec(
name="cpu",
num_cpus=60,
memory="120G",
service_account="geneva-integ-test",
),
_WorkerGroupSpec(
name="gpu",
num_cpus=8,
memory="32G",
num_gpus=1,
service_account="geneva-integ-test",
),
],
)
```
## Serialization Errors
Serialization is a critical subsystem of Geneva. In order to store UDFs and perform distributed execution, both code and data must be serializable. Errors in this area can be subtle and difficult to find.
There are a few basic rules:
1. **Python objects** passed to distributed processes or written to LanceDB must be able to be pickled or unpickled using the Python pickle or cloudpickle library.
2. **Python code** used for distributed execution, including UDFs used to calculate values written to columns must be able to be pickled or unpickled using the Python pickle or cloudpickle library.
3. **Python code or objects** need to have the same encoding and representation on the client-side and the server-side.
Below is a list of error categories and examples and how to fix them.
### Serialization Library Mismatches
Any Python code and objects must be able to be serialized by the client and deserialized on the server side, and vice versa. This includes objects that are generated on the fly such as those using the `attrs` library.
The distributed processing engine Geneva uses, Ray, also depends on the `attrs` library. Different versions may create different object signatures that are not compatible when shipped from client-side to server-side and vice versa. This means you'll need to have compatible versions of this library on both sides.
Here's an example error message. It is subtle and does not directly point to the `attrs` library:
```
...
File "/home/runner/work/geneva/geneva/.venv/lib/python3.12/site-packages/ray/util/client/common.py", line 414, in _prepare_client_task
self._ensure_ref()
File "/home/runner/work/geneva/geneva/.venv/lib/python3.12/site-packages/ray/util/client/common.py", line 384, in _ensure_ref
self._ref = ray.worker._put_pickled(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/geneva/geneva/.venv/lib/python3.12/site-packages/ray/util/client/worker.py", line 509, in _put_pickled
raise cloudpickle.loads(resp.error)
TypeError: Enum.__new__() missing 1 required positional argument: 'value'
```
This was solved by updating the `attrs` module on the client side to use the same version found on the server side.
### Objects with Unserializable Elements
Python objects may have internal references to unpickleable objects such as open file handles or open network clients with machine specific state. There are two strategies here:
1. **Remove the reference** to unpickleable objects.
2. **Keep objects with unserializable state** only on the client or only on the server. This could be moving clients into the UDF function, or converting objects into serializable versions before transmitting them.
For example, creating clients or opening files must be inside the UDF. You may see pickling-related errors like this:
```
...
raise PicklingError(
_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
```
Geneva pulls in your UDFs so they can be sent to remote worker nodes. For the method to be sent, the data must be "pickleable".
**So instead of this:**
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from google.cloud import storage
storage_client = storage.Client() # this has unpickleable state
bucket = storage_client.bucket(BUCKET_NAME) # this has a reference to storage_client
...
@udf
def udf_function(...)
...
blob = bucket.blob(video_path) # the udf's closure captures the unpickleable storage_client
...
```
**Do this:**
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from google.cloud import storage
# ...
@udf
def udf_function(...)
# ...
storage_client = storage.Client() # this has unpickleable state
bucket = storage_client.bucket(BUCKET_NAME)
blob = bucket.blob(video_path) # blob is bytes and is pickleable so is safe
# ...
```
### Disconnect or Serialization Errors with GPU Dependent UDFs
When using GPU code, the typical process loads some values and tensors from CPU memory to GPU memory. Even after moving data (`data.cpu().tolist()`), there may be references to GPU memory. While this is not a problem with local execution, when doing a distributed job it may cause problems because the GPU references are not serializable and not needed. You must take steps to eliminate references to structures in GPU memory since they cannot be serialized and sent between workers. This can be achieved by explicitly disconnecting references to the GPU memory (`data.cpu().detach().tolist()`) to get only-CPU resident fully serializable objects.
Here are some typical error messages:
```
Exception in thread Thread-27 (_proxy):
Traceback (most recent call last):
File "/home/jon/.pyenv/versions/3.10.16/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/jon/proj/geneva-deepseek-vl2/.venv/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 772, in run_closure
_threading_Thread_run(self)
File "/home/jon/.pyenv/versions/3.10.16/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/jon/proj/geneva-deepseek-vl2/src/geneva/runners/ray/_portforward.py", line 172, in _proxy
{s1: s2, s2: s1}[s].sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
Log channel is reconnecting. Logs produced while the connection was down can be found on the head node of the cluster in `ray_client_server_[port].out`
2025-04-11 02:25:21 INFO Starting proxy from pod to client
2025-04-11 02:25:21 INFO Proxy started
2025-04-11 02:25:21 INFO Proxying between and
2025-04-11 02:25:21 INFO Waiting for client connection
2025-04-11 02:25:21,828 ERROR dataclient.py:330 -- Unrecoverable error in data channel.
---------------------------------------------------------------------------
...
File ~/proj/geneva-deepseek-vl2/.venv/lib/python3.10/site-packages/grpc/_channel.py:1006, in _end_unary_response_blocking(state, call, with_call, deadline)
1004 return state.response
1005 else:
-> 1006 raise _InactiveRpcError(state)
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.NOT_FOUND
details = "Failed to serialize response!"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-04-11T02:25:22.209209484+00:00", grpc_status:5, grpc_message:"Failed to serialize response!"}"
>
Unexpected exception:
Traceback (most recent call last):
File "/home/jon/proj/geneva-deepseek-vl2/.venv/lib/python3.10/site-packages/ray/util/client/logsclient.py", line 67, in _log_main
for record in log_stream:
File "/home/jon/proj/geneva-deepseek-vl2/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 543, in __next__
return self._next()
File "/home/jon/proj/geneva-deepseek-vl2/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 969, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.NOT_FOUND
details = "Logstream proxy failed to connect. Channel for client bd854100340640fb8b5770d2bf173197 not found."
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Logstream proxy failed to connect. Channel for client bd854100340640fb8b5770d2bf173197 not found.", grpc_status:5, created_time:"2025-04-11T02:25:32.223710374+00:00"}"
>
```
# End-to-End Example
Source: https://docs.lancedb.com/geneva/end-to-end
A complete Geneva example on LanceDB Enterprise — create a table, backfill computed columns, and build a materialized view with embeddings.
This example walks through a complete Geneva workflow on LanceDB Enterprise: creating a raw
table, adding computed columns with a distributed backfill, and materializing a view with
embeddings for downstream search.
The dataset is a product catalog with titles and descriptions. We'll compute a `word_count`
feature column, then create a materialized view that adds text embeddings.
## 0. What you need to run this
All you need is an existing **LanceDB Enterprise deployment**. Distributed job execution,
clusters, and dependency manifests are managed for you — there is no Kubernetes or cluster
setup in this example.
## 1. Connect and create a table
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import os
import pyarrow as pa
import geneva
# Connect to LanceDB Enterprise
db = geneva.connect(
uri="db://my-db",
host_override=os.getenv("LANCEDB_URI"),
api_key=os.getenv("LANCEDB_API_KEY"),
)
# Create a raw product table
schema = pa.schema([
pa.field("product_id", pa.int64()),
pa.field("title", pa.string()),
pa.field("description", pa.string()),
pa.field("category", pa.string()),
pa.field("price", pa.float64()),
])
data = pa.table({
"product_id": [1, 2, 3, 4, 5],
"title": ["Chainmail Coif", "Jousting Lance Grip Tape", "Dragon-Repellent Spray", "Sword Squeegee", "Visor Windshield Wipers"],
"description": [
"Premium riveted chainmail head covering. Breathable enough for dragon fire, probably.",
"Non-slip grip tape for jousting lances. 3000 PSI tensile strength. Void where tilting is prohibited.",
"All-natural herbal spray. Dragons hate it. Effectiveness not guaranteed against actual dragons.",
"Ergonomic squeegee fits all standard broadswords. Removes blood, mud, and existential dread.",
"Hand-cranked windshield wipers for full-face visors. Never ride blind into battle again.",
],
"category": ["armor", "tournament", "defense", "maintenance", "armor"],
"price": [129.99, 45.00, 59.99, 34.99, 24.99],
})
try:
db.drop_table("products_raw")
except Exception:
pass
table = db.create_table("products_raw", data=data, schema=schema)
```
## 2. Define UDFs
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
from geneva import udf
@udf(data_type=pa.int32())
def word_count(description: str) -> int:
"""Count words in the description."""
return len(description.split())
@udf(data_type=pa.string())
def price_tier(price: float) -> str:
"""Bucket price into tiers."""
if price < 30:
return "budget"
elif price < 75:
return "mid-range"
else:
return "premium"
```
## 3. Register columns and run a backfill
Register the UDFs as virtual columns and trigger a backfill. The job runs on your
deployment's default distributed execution environment — no cluster or context to configure.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Add computed columns
table.add_columns({
"word_count": word_count,
"price_tier": price_tier,
})
# Run the backfills
table.backfill("word_count")
table.backfill("price_tier")
```
## 4. Create a materialized view with embeddings
The embedding model needs extra Python dependencies (`sentence-transformers`, `torch`). Rather
than configuring a deployment-wide environment, we bundle those dependencies **with the UDF**
using `@udf(manifest=...)`. The manifest is snapshotted onto the view, so refreshes use it
automatically.
The materialized view selects a subset of columns from the source table — here we drop `price`
and `price_tier`, keeping only what's needed for search — plus a derived `embedding` column.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import udf
from geneva.manifest import GenevaManifest
# Bundle the embedding model's dependencies and attach them to the UDF
embed_manifest = (
GenevaManifest.create_pip("embedding-deps")
.pip(["sentence-transformers==3.3.1", "torch==2.5.1"])
.build()
)
@udf(data_type=pa.list_(pa.float32(), 384), manifest=embed_manifest)
class EmbedDescription:
def __init__(self):
self.model = None
def __call__(self, description: str) -> list[float]:
if self.model is None:
from sentence_transformers import SentenceTransformer
self.model = SentenceTransformer("BAAI/bge-small-en-v1.5")
return self.model.encode(description, normalize_embeddings=True).tolist()
# Build a query that selects search columns plus the derived embedding
query = table.search(None).select({
"product_id": "product_id",
"title": "title",
"description": "description",
"category": "category",
"word_count": "word_count",
"embedding": EmbedDescription(),
})
# Create the materialized view and populate it — the embedding UDF runs on refresh
db.create_materialized_view("products_enriched", query)
enriched = db.open_table("products_enriched")
enriched.refresh()
```
## 5. Query the enriched table
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
enriched = db.open_table("products_enriched")
# Vector search
results = (
enriched.search([0.1] * 384, vector_column_name="embedding")
.limit(3)
.to_arrow()
)
# Filtered search — armor category only
armor_results = (
enriched.search([0.1] * 384, vector_column_name="embedding")
.where("category = 'armor'")
.limit(3)
.to_arrow()
)
```
## 6. Incremental refresh
As new products are added to the source table, backfill the new rows and refresh the view to
compute embeddings for them — only the new rows are processed:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Append new products
new_data = pa.table({
"product_id": [6, 7],
"title": ["Moat Floaties", "Knight-Night Sleep Mask"],
"description": [
"Inflatable arm floaties in battleship grey. Because even knights can't always swim in full plate.",
"Padded silk sleep mask embroidered with your coat of arms. Blocks out 100% of torchlight.",
],
"category": ["defense", "armor"],
"price": [29.99, 19.99],
})
table.add(new_data)
# Only null/new values are computed
table.backfill("word_count")
table.backfill("price_tier")
# Incrementally materialize the new rows (including their embeddings) into the view
enriched.refresh()
```
To keep source columns in sync automatically, mark their UDFs with `@udf(auto_backfill=True)`. See
[Backfilling](/geneva/jobs/backfilling/).
# Getting Started
Source: https://docs.lancedb.com/geneva/getting-started
Connect to LanceDB Enterprise, define a UDF, and run a distributed backfill — from a notebook or a script.
Connect to your LanceDB Enterprise deployment, define a UDF, and run a distributed
backfill — all from a notebook or a script. No cluster setup required.
## Installation
Geneva is published on [PyPI](https://pypi.org/project/geneva/). Install the latest stable
release with [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`. Newer pre-release
builds with the latest features are also available on LanceDB's Fury indexes — see
[Pre-release builds](#pre-release-builds) below.
### Prerequisites
* Python 3.10+
* [uv](https://docs.astral.sh/uv/) (recommended) or `pip`
### Install the latest stable release
```bash uv icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv pip install --upgrade geneva
```
```bash pip icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install --upgrade geneva
```
### Verify
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
python -c "import geneva; print(geneva.__version__)"
```
### Pre-release builds
To pick up the newest features ahead of a stable release, install a pre-release from LanceDB's
Fury indexes. Geneva and its dependencies are published across two indexes:
| Package | Index |
| ------------------- | ------------------------------------ |
| `geneva`, `lancedb` | `https://pypi.fury.io/lancedb/` |
| `pylance` | `https://pypi.fury.io/lance-format/` |
```bash uv icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv pip install --pre --upgrade \
--extra-index-url https://pypi.fury.io/lancedb/ \
--extra-index-url https://pypi.fury.io/lance-format \
--index-strategy unsafe-best-match \
geneva
```
```bash pip icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install --pre --upgrade \
--extra-index-url https://pypi.fury.io/lancedb/ \
--extra-index-url https://pypi.fury.io/lance-format \
geneva
```
The `--index-strategy unsafe-best-match` flag is required with `uv`. By default, `uv` only
considers package versions from the first index that lists a given package (PyPI). Since
`geneva` and `pylance` also appear on PyPI, this flag tells `uv` to pick the best match across
all indexes.
## Quickstart
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import os
import geneva
import pyarrow as pa
# Connect to LanceDB Enterprise
db = geneva.connect(
uri="db://my-db",
host_override=os.getenv("LANCEDB_URI", "http://localhost:10024"),
api_key=os.getenv("LANCEDB_API_KEY"),
)
tbl = db.open_table("my_table")
# Define a User Defined Function (UDF) that counts the words in the text column
@geneva.udf(data_type=pa.int32())
def word_count(text: str) -> int:
return len(text.split())
# Register the UDF as a new virtual column
tbl.add_columns({"word_count": word_count})
# Backfill the new column using distributed execution with incremental checkpointing
tbl.backfill("word_count")
```
## Auto-backfill
With `auto_backfill=True`, LanceDB Enterprise recomputes the column for you whenever the
data or the UDF version changes — no explicit `backfill()` call needed (see
[Backfilling](/geneva/jobs/backfilling/)).
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Change the column to use a new UDF version with auto-backfill enabled
@geneva.udf(data_type=pa.int32(), auto_backfill=True)
def word_count(text: str) -> int:
return len(text.split())
tbl.alter_columns({"path": "word_count", "udf": word_count})
# Add new rows. word_count is computed automatically in the background.
tbl.add([{"text": "hello world"}])
```
## Materialized views and chunkers
A [materialized view](/geneva/jobs/materialized-views/) applies UDFs over a query and
refreshes incrementally. A [chunker](/geneva/udfs/scalar-udtfs) view expands each source
row into many rows (1:N) — useful for splitting documents, videos, or images.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Materialized view: a query with UDF-computed columns, refreshed incrementally
query = tbl.search(None).select({"text": "text", "word_count": word_count})
view = db.create_materialized_view("my_view", query)
view.refresh()
# Chunker view: 1:N row expansion — split each row's text into one row per word
from typing import Iterator, NamedTuple
class Chunk(NamedTuple):
chunk_index: int
chunk_text: str
@geneva.chunker
def split_text(text: str) -> Iterator[Chunk]:
for i, word in enumerate(text.split()):
yield Chunk(chunk_index=i, chunk_text=word)
chunks = db.create_udtf_view(
"my_chunks",
source=tbl.search(None).select(["text"]),
udtf=split_text,
)
chunks.refresh()
```
## Connecting to object storage or a local filesystem
Geneva can also run directly against cloud object storage or a local path. In this mode, jobs run on a
[distributed execution context](/geneva/jobs/contexts) you provide.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Cloud object storage (S3, GCS, Azure, or any S3-compatible object store)
db = geneva.connect("s3://my-bucket/my-database")
# Local filesystem
db = geneva.connect("/path/to/my-database")
```
# Multimodal Feature Engineering with Geneva
Source: https://docs.lancedb.com/geneva/index
Learn how to do multimodal feature engineering in LanceDB Enterprise to transform raw data into meaningful features for AI models.
Enterprise-only
When working with multimodal data at scale, [LanceDB Enterprise](/enterprise) makes it easy
to define, extract, and transform raw data into useful information and features for your
AI applications. LanceDB Enterprise's *Multimodal Feature Engineering* package is designed to improve
the productivity of AI engineers operating at immense scale.
With an API designed to leverage LanceDB's optimized data storage and retrieval, it
streamlines prototyping extraction and transformation tasks, performing experiments, exploring your
data, scaling up execution, and moving to production.
LanceDB Multimodal Feature Engineering enables researchers to seamlessly transition from
experiments in local notebooks to fully-managed distributed job execution on datasets with
billions of rows.
Feature Engineering and the `geneva` Python package are currently only available as part of
[LanceDB Enterprise](/enterprise). Please [contact us](mailto:contact@lancedb.com) if you're interested
in scaling up your feature engineering workloads for your AI and multimodal use cases.
The `geneva` package uses Python [User Defined Functions (UDFs)](/geneva/udfs/udfs) to define features
as columns in a Lance dataset. Adding a feature is straightforward:
Prototype your Python function in your favorite environment.
Wrap the function with a small UDF decorator (see [UDFs](/geneva/udfs/udfs)).
Register the UDF as a virtual column using `Table.add_columns()`.
(Optional, advanced) Override where the job runs — see [Advanced Execution Contexts](/geneva/jobs/contexts). On LanceDB Enterprise, distributed job execution is fully managed, so most users can skip this step.
Trigger a `backfill` operation (see [Backfilling](/geneva/jobs/backfilling/)).
You can build your Python feature generator function in an IDE or a notebook using your project's Python versions and dependencies. `geneva` will automate much of the dependency and version management needed to move from prototype to scale and production.
Ready to write your first feature? Head to [Getting Started](/geneva/getting-started).
## Continue learning
Visit the following pages to learn more about featuring engineering in LanceDB Enterprise:
* **Get started**: [Getting Started](/geneva/getting-started) · [What is Feature Engineering?](/geneva/overview/) · [End-to-end example](/geneva/end-to-end)
* **UDFs**: [Using UDFs](/geneva/udfs/udfs) · [Blob helpers](/geneva/udfs/blobs/) · [Error handling](/geneva/udfs/error_handling) · [Advanced configuration](/geneva/udfs/advanced-configuration)
* **Jobs**: [Job execution overview](/geneva/jobs/) · [Backfilling](/geneva/jobs/backfilling/) · [Materialized views](/geneva/jobs/materialized-views/) · [Startup optimizations](/geneva/jobs/startup/) · [Advanced job configuration](/geneva/jobs/advanced-job-configuration/) · [Advanced execution contexts](/geneva/jobs/contexts/) · [Geneva console](/geneva/jobs/console) · [Performance](/geneva/jobs/performance/)
* **Deployment**: [Deployment overview](/geneva/deployment/) · [Helm deployment](/geneva/deployment/helm/) · [Troubleshooting](/geneva/deployment/troubleshooting/)
## API Reference
* [`geneva.connect()`](https://lancedb.github.io/geneva/api/) — connect to a Geneva database
* [Connection](https://lancedb.github.io/geneva/api/connection/) — manage tables, views, jobs, clusters, and manifests
* [Table](https://lancedb.github.io/geneva/api/table/) — add columns, backfill, search, and manage table data
* [UDF](https://lancedb.github.io/geneva/api/udf/) — define user-defined functions for feature computation
# Advanced Job Configuration
Source: https://docs.lancedb.com/geneva/jobs/advanced-job-configuration
Pin the dependency manifest a transform's distributed job runs with using @udf, @chunker, and @udtf.
Enterprise-only
On LanceDB Enterprise, backfill and refresh jobs run on a managed, distributed execution
environment configured at deployment time:
* the **default cluster** — the compute pool jobs run on, and
* the **default manifest** — the Python dependency environment (image and packages) the
distributed workers run with.
These defaults are set in the [LanceDB Helm chart](/geneva/deployment/helm) and cover most
workloads. When a transform needs dependencies that differ from the deployment default, pin a
**manifest** on the transform itself, as described below.
To override the **cluster** a job runs on — for example to route an embedding backfill to a
GPU pool — see [Advanced Execution Contexts](/geneva/jobs/contexts).
## Pinning a dependency manifest
A manifest pins the Python image and packages the distributed workers run with. Build one with
the manifest builders, then attach it to your transform with the `manifest=` argument on
`@udf`, `@chunker`, or `@udtf`. The manifest is snapshotted into the column (or view) metadata
when the transform is registered, so every backfill or refresh of that transform uses it
automatically — there is no per-call manifest argument to remember.
**Manifests are immutable at the column / view level.** When a transform is registered, its
manifest is snapshotted onto the column (or view) metadata. Changing the deployment-default
manifest — or the `GenevaManifest` object in your code — does **not** affect existing columns
or views: they keep using the snapshot taken at creation time. To move a column or view to a
new manifest, re-point it to a new (or updated) UDF / chunker / UDTF — for example with
`alter_columns()` for a column, or by recreating the view.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
from typing import Iterator, NamedTuple
from geneva import udf, chunker, udtf
from geneva.manifest import GenevaManifest
# Build a manifest that pins the dependencies these transforms need
embed_manifest = (
GenevaManifest.create_pip("embedding-deps")
.pip(["sentence-transformers==3.3.1", "torch==2.5.1"])
.build()
)
```
### `@udf(manifest=...)`
Pin dependencies for a 1:1 computed column:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.list_(pa.float32(), 384), manifest=embed_manifest)
def embed(text: str) -> list[float]:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
return model.encode(text, normalize_embeddings=True).tolist()
tbl.add_columns({"embedding": embed})
tbl.backfill("embedding") # the backfill job runs with embed_manifest
```
### `@chunker(manifest=...)`
Pin dependencies for a 1:N [chunker](/geneva/udfs/scalar-udtfs) (scalar UDTF):
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
class Chunk(NamedTuple):
chunk_index: int
chunk_text: str
@chunker(manifest=embed_manifest)
def split_document(text: str) -> Iterator[Chunk]:
for i, part in enumerate(text.split("\n\n")):
yield Chunk(chunk_index=i, chunk_text=part)
view = db.create_udtf_view("chunks", source=tbl.search(None), udtf=split_document)
view.refresh() # the refresh job runs with embed_manifest
```
### `@udtf(manifest=...)`
Pin dependencies for an N:M [batch UDTF](/geneva/udfs/batch-udtfs):
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udtf(
output_schema=pa.schema([
pa.field("label", pa.string()),
pa.field("count", pa.int64()),
]),
manifest=embed_manifest,
)
def group_stats(source) -> Iterator[pa.RecordBatch]:
df = source.to_pandas()
agg = df.groupby("label").size().reset_index(name="count")
yield pa.RecordBatch.from_pandas(agg)
view = db.create_udtf_view("summaries", source=tbl.search(None), udtf=group_stats)
view.refresh() # the refresh job runs with embed_manifest
```
## Capturing your local environment for testing
When iterating locally, you often want the workers to run with the *exact* packages from your
current environment rather than a curated pip list. `Connection.capture_local_environment()`
zips your workspace (and, optionally, your site-packages), uploads the archives through the
connection, and returns a ready-to-use `GenevaManifest` you can attach to a transform with
`manifest=`.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import os
import pyarrow as pa
import geneva
from geneva import udf
db = geneva.connect(
uri="db://my-db",
host_override=os.getenv("LANCEDB_URI"),
api_key=os.getenv("LANCEDB_API_KEY"),
)
# Capture the local workspace; rely on the worker image for site-packages
manifest = db.capture_local_environment(skip_site_packages=True)
@udf(data_type=pa.string(), manifest=manifest)
def shout(text: str) -> str:
return text.upper()
tbl = db.open_table("my_table")
tbl.add_columns({"shout": shout})
tbl.backfill("shout") # workers run with your captured environment
```
Pass `skip_site_packages=False` (the default) to also upload your local site-packages.
## Manifest resolution
For a given transform, the manifest is resolved in this order (first match wins):
1. The manifest pinned on the transform via `@udf` / `@chunker` / `@udtf` `manifest=`.
2. For a materialized view, the manifest snapshotted on the view when it was created.
3. The deployment-default manifest from the [LanceDB Helm chart](/geneva/deployment/helm).
The `manifest=` argument applies to managed enterprise (`db://`) jobs. For direct
object-storage or local-filesystem connections, configure the dependency environment
explicitly with an [Advanced Execution Context](/geneva/jobs/contexts) instead.
# Backfilling
Source: https://docs.lancedb.com/geneva/jobs/backfilling
Learn how to trigger backfill operations to populate column values in your LanceDB table using Geneva's distributed framework.
## Triggering Backfill
Triggering backfill creates a distributed job to run the UDF and populate the column values in your LanceDB table. The Geneva framework simplifies several aspects of distributed execution.
**Checkpoints**: Each batch of UDF execution is checkpointed so that partial results are not lost in case of job failures. Jobs can resume and avoid most of the expense of having to recalculate values.
## Auto-backfill
Computed columns can be explicitly backfilled or they can be configured to be backfilled
automatically as data changes. Set `auto_backfill=True` on the UDF, and the column is automatically recomputed
whenever it falls out of sync with its source data.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Mark the column's UDF for automatic backfill
@udf(data_type=pa.list_(pa.float32(), 1536), version="1", auto_backfill=True)
def embed_udf(text: str) -> list[float]:
return embedding_model.encode(text)
tbl.add_columns({"embedding": embed_udf})
# No explicit backfill() needed — adding rows triggers recomputation automatically
tbl.add(new_rows)
```
The `auto_backfill` flag is recorded in the column metadata when the column is added or
altered. LanceDB Enterprise's managed agent watches for columns that need recomputation and
dispatches a distributed backfill job for you — there is no manual trigger and no status
polling. A column is recomputed when, for example, **new rows are added** (leaving it null for
those rows) or the **UDF version changes** (you bump `version=` and `alter_columns()` to the
new function).
Auto-backfill is an enterprise feature. On direct object-storage or local-filesystem
connections there is no managed agent, so `auto_backfill=True` has no effect and you must run
`backfill()` explicitly.
## Adaptive checkpoint sizing
Geneva can automatically adjust checkpoint sizes during a backfill. It starts with small checkpoints (faster proof-of-life) and grows them as it observes stable throughput, while staying within safe bounds. Planning still uses your configured checkpoint size (`checkpoint_size`), but the actual checkpoint chunks can be smaller when adaptive sizing is enabled.
Adaptive sizing is always clamped to bounds:
* `max_checkpoint_size`: Upper bound. Defaults to the job's checkpoint size (`checkpoint_size`) and is capped at that value if you set a larger max.
* `min_checkpoint_size`: Lower bound. Defaults to 1.
When `min_checkpoint_size == max_checkpoint_size`, adaptive sizing is disabled and checkpoints are fixed-size.
You can set adaptive bounds in two places:
* On the UDF definition via `@udf(..., min_checkpoint_size=..., max_checkpoint_size=...)`
* On the backfill call via `table.backfill(..., min_checkpoint_size=..., max_checkpoint_size=...)`
Backfill-level values take precedence over UDF defaults.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(min_checkpoint_size=25, max_checkpoint_size=200)
def embed_udf(text):
...
# Override the UDF defaults for this run
tbl.backfill("embedding", min_checkpoint_size=10, max_checkpoint_size=100)
```
## Managing concurrency
One way to speed up the execution of a job to give it more resources and to have it work in parallel. There are a few settings you can use on the backfill command to tune this.
* process-level `concurrency`
* thread-level `intra_applier_concurrency`
Process level concurrency can be set with the `concurrency` parameter. This lets you specify the number of processes calculating values using the UDF. The default is 8 and should be set to the number of GPUs you would like to dedicate to your job. This can also be used based on CPU constraints. So if you have 40 machines with 4 GPUs each, you could set ths value to 160. If you set the value higher than the resources available, Geneva will try to schedule as much of the resources as it can (and potentially auto-scale to get more resources).
Thread level concurrency can be set with the `intra_applier_concurrency` parameter. This lets you specify the number of threads in each process is calculating values using the UDF. The default is 1. If you have CPU heavy jobs this may be the best setting to tweak to get more utilization out of your systems. If you set the value higher than the resources available, Geneva will try to schedule as much of the resources as it can get.
The two settings can be used in combination. So if your UDF requires 1 CPU and you set `concurrency` to 10 and `intra_applier_concurrency` to 5, you will potentially have 50 instances of the UDFs running in parallel.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# backfill embeddings with 10 * 5 = 50 instances
tbl.backfill("embedding", concurrency=10, intra_applier_concurrency=5)
```
## Managing commit visibility
Feature engineering jobs at scale can take days to complete. Two settings can help you present progress to other readers incrementally.
* Limit the number of rows processed with `num_frags`
* Perform intermediate commits with `commit_granularity`
The `num_frags` parameter lets you limit the number of fragments processed before the job is considered complete. If you have a table with 1000 fragments, you could set `num_frags` to 1 to see how your UDF performs and if to validate the values generated. You can then later run with a larger `num_frags` value or without the `num_frags` setting to complete the backfill. Any fragments prevoiusly computed are not computed again.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# only backfill 2 fragments so experiement can be done on the sample
tbl.backfill("embedding", num_frags=2)
```
The `commit_granularity` parameter lets you specify how many fragments need to be ready to commit before a intermediate commit occurs and makes partial results visible to other readers. So for our example with a table of 1000 fragments, you can set `commit_granularity` to 10 to see progress updates every 10 fragments.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# backfill all fragments and perform an intermediate commit every 10 fragments to expose incremental results.
tbl.backfill("embedding", commit_granularity=10)
```
## Filtered Backfills
Geneva allows you to specify SQL-style filters on the backfill operation. This lets you to apply backfills to a specified subset of the table's rows.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# only backfill video content whose filenames start with 'a'
tbl.backfill("content", where="starts_with(filename, 'a')")
# only backfill embeddings of only those videos with content
tbl.backfill("embedding", where="content is not null")
```
Geneva also allows you to incrementally add more rows or have jobs that just update rows that were previously skipped.
If new rows are added, we can run the same command and the new rows that meet the criteria will be updated.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# only backfill video content whose filenames start with 'a'
tbl.backfill("content", where="starts_with(filename, 'a')")
# only backfill embeddings of only those videos with content
tbl.backfill("embedding", where="content is not null")
```
Or, you can use filters to add in or overwrite content in rows previously backfilled.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# only backfill video content whose filenames start with 'a' or 'b' but only if content not pulled previously
tbl.backfill("content", where="(starts_with(filename, 'a') or starts_with(filename, 'b')) and content is null")
# only backfill embeddings of only those videos with content and no prevoius embeddings
tbl.backfill("embedding", where="content is not null and embeddding is not null")
```
Reference:
* [`backfill` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.backfill)
* [`backfill_async` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.backfill_async)
* [`plan_backfill` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.plan_backfill)
* [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator and UDF configuration options including `checkpoint_size`
# Bulk Loading & Updating Columns
Source: https://docs.lancedb.com/geneva/jobs/bulk-load-columns
Load or update column data from external sources (Parquet, Lance, IPC) into your LanceDB table using a primary-key join.
Beta — introduced in Geneva 0.13.0
## Overview
A common scenario is having column data that **already exists** in an external dataset — embeddings from a vendor, features exported from a data warehouse, or columnar data in cloud storage — that you want to load into an existing LanceDB table.
`load_columns` joins value columns from an external source into your table by primary key. It works for both use cases:
* **Adding new columns:** If the specified columns don't exist in the destination table, they are created automatically.
* **Updating existing columns:** If the columns already exist, matched rows are updated with the source values. Unmatched rows are controlled by the `on_missing` parameter.
**Destination table (before):**
| pk | col\_a | col\_b |
| -- | ------ | ------ |
| 1 | x | 10 |
| 2 | y | 20 |
| 3 | z | 30 |
**External source (Parquet / Lance / IPC):**
| pk | embedding |
| -- | --------- |
| 1 | \[.1, .2] |
| 2 | \[.3, .4] |
| 3 | \[.5, .6] |
**Destination table (after `load_columns` join on `pk`):**
| pk | col\_a | col\_b | embedding |
| -- | ------ | ------ | --------- |
| 1 | x | 10 | \[.1, .2] |
| 2 | y | 20 | \[.3, .4] |
| 3 | z | 30 | \[.5, .6] |
### When to use
* **Loading new columns:** Attach pre-computed embeddings from a vendor, or add features exported from Spark/BigQuery as Parquet.
* **Updating existing columns:** Replace outdated embeddings with a newer model's output, or refresh feature values from an updated export.
* **Partial updates:** Update a subset of rows (e.g., only rows whose embeddings were recomputed) while preserving all other values via carry semantics.
* **Format consolidation:** Merge columnar data spread across Parquet files into an existing Lance table.
## Basic usage
Supports Parquet, Lance, and IPC sources. The format is auto-detected from the URI suffix, or can be overridden with `source_format`. You can load one or more columns in a single call.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("my_db")
table = db.open_table("my_table")
# Add a new embedding column from a Parquet source
table.load_columns(
source="s3://bucket/embeddings/",
pk="document_id",
columns=["embedding"],
)
# Later, update the same column with refreshed embeddings
table.load_columns(
source="s3://bucket/embeddings_v2/",
pk="document_id",
columns=["embedding"],
)
```
For non-blocking execution, use `load_columns_async` which returns a `JobFuture` — call `.result()` to block until completion.
## Handling missing keys
When the source doesn't cover every row in the destination, the `on_missing` parameter controls what happens to unmatched rows:
| Mode | Behavior |
| ------------------- | ----------------------------------------------- |
| `"carry"` (default) | Keep existing value. NULL if the column is new. |
| `"null"` | Explicitly set to NULL. |
| `"error"` | Raise an error on the first unmatched row. |
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Default: unmatched rows keep their current value
table.load_columns(
source="s3://bucket/partial_embeddings/",
pk="document_id",
columns=["embedding"],
on_missing="carry",
)
# Strict mode: fail if source doesn't cover all rows
table.load_columns(
source="s3://bucket/embeddings/",
pk="document_id",
columns=["embedding"],
on_missing="error",
)
```
The `carry` mode is particularly important for partial and multi-pass loads — it ensures that previously loaded values are never overwritten.
## Performance tuning
### Concurrency
The `concurrency` parameter controls the number of worker processes. The default is 8 — set this to match your available cluster resources.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.load_columns(
source="s3://bucket/embeddings/",
pk="document_id",
columns=["embedding"],
concurrency=16,
)
```
### Checkpointing
Bulk load jobs checkpoint each batch for fault tolerance, using the same infrastructure as [backfill jobs](/geneva/jobs/backfilling#adaptive-checkpoint-sizing). Key parameters:
* `checkpoint_interval_seconds`: Target seconds per checkpoint batch (default 60s). The adaptive sizer grows or shrinks batch sizes to hit this target.
* `min_checkpoint_size` / `max_checkpoint_size`: Bounds for adaptive sizing.
If your job is small enough to complete without needing fault tolerance, you can get better performance by effectively disabling checkpoints. Increase `checkpoint_interval_seconds` to a large value and set `min_checkpoint_size` high enough that each worker processes its entire workload in a single batch.
### Commit visibility
For long-running jobs, `commit_granularity` controls how many fragments complete before an intermediate commit makes partial results visible to readers.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.load_columns(
source="s3://bucket/embeddings/",
pk="document_id",
columns=["embedding"],
commit_granularity=10,
)
```
### Multi-pass loads for large sources
`load_columns` builds an in-memory primary-key index from the source. If the source is too large to fit in memory, split it into chunks and run sequential calls. Carry semantics guarantee correctness across passes.
Index memory depends on primary key type:
| PK type | \~Memory per row | 100M rows | 1B rows |
| --------------------- | ---------------- | --------- | ------- |
| int64 | \~8 bytes | \~800 MB | \~8 GB |
| string (avg 32 bytes) | \~32 bytes | \~3.2 GB | \~32 GB |
Choose N so that `source_size / N` fits in memory:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow.dataset as pads
# Discover source files (metadata only, no data I/O)
source_files = pads.dataset("s3://bucket/embeddings/", format="parquet").files
# Split into N chunks and run sequentially
N = 4
total = len(source_files)
for i in range(N):
chunk = source_files[i * total // N : (i + 1) * total // N]
table.load_columns(
source=chunk,
pk="document_id",
columns=["embedding"],
)
```
Each pass reads only its assigned files, so total source I/O stays at 1x. If the source is already partitioned into subdirectories, pass each URI directly:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
for shard in range(4):
table.load_columns(
source=f"s3://bucket/embeddings/shard_{shard}/",
pk="document_id",
columns=["embedding"],
)
```
Multi-pass loads must run **sequentially**, not concurrently. Two `load_columns` calls running at the same time against the same column produce an interleaved end state. Use a plain `for` loop, not `concurrent.futures`.
## Reference
* [`load_columns` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.load_columns)
* [`load_columns_async` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.load_columns_async)
# Backfill Conflicts
Source: https://docs.lancedb.com/geneva/jobs/conflicts
Learn how Geneva handles conflicts during backfill operations and what to do when they occur.
## Overview
Geneva backfills operate on a **point-in-time snapshot** of your table. When other operations modify the table during or between backfills, conflicts can occur. Geneva >=0.9.0 automatically handles most conflict scenarios, reducing unnecessary recomputation and enabling graceful recovery.
## Safe Operations During Backfill
These operations can safely run while a backfill is in progress:
| Operation | Why It's Safe |
| -------------------------------------- | ----------------------------------------------------- |
| `merge_insert` (Insert-only) | Creates new fragments without modifying existing ones |
| `add()` / append data | Creates new fragments without modifying existing ones |
| Read operations (`search`, `to_arrow`) | Read-only, no fragment modification |
| Adding new columns | Schema change only, no fragment rewrite |
## Operations That Cause Conflicts
These operations can conflict with running backfills:
| Operation | What Happens |
| -------------------------------- | ----------------------------------------------------------- |
| `compact_files()` / `optimize()` | Reorganizes fragments, invalidating the backfill's snapshot |
| `merge_insert` with updates | Modifies existing rows, causing fragment conflicts |
| `delete()` | Modifies existing fragments |
When a conflict occurs, affected fragments fail gracefully. The backfill completes what it can, and you can re-run it to process the remaining rows.
## How Geneva Handles Conflicts
### Concurrent Backfills on Different Columns
When multiple backfills run on the same table but different columns, Geneva handles version conflicts automatically:
1. Each backfill writes to different column files (field IDs)
2. If a commit conflict occurs, Geneva retries at the latest version
3. The retry merges the new column data without overwriting other columns
This is controlled by the `GENEVA_VERSION_CONFLICT_MAX_RETRIES` environment variable (default: 10). See [Advanced Configuration](/geneva/udfs/advanced-configuration) for details.
### Compaction Between Backfills
When you run compaction between backfills (not during), Geneva handles it efficiently:
| Scenario | Behavior |
| ------------------------------------------------ | ----------------------------------------------------------- |
| Backfill, compact, re-backfill (same UDF) | Already-computed rows are skipped via `WHERE IS NULL` |
| Partial backfill, compact, resume | Incremental processing continues from where it left off |
| Backfill, `alter_columns` (new UDF), re-backfill | Full reprocessing with new UDF (intentional) |
Geneva's default behavior is to skip rows that already have values (`WHERE IS NULL`). This means compaction doesn't cause unnecessary recomputation.
## Recovery Steps
When a conflict occurs during a backfill:
1. **Wait** for any concurrent operations (compaction, updates) to complete
2. **Re-run** the backfill:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl.backfill("column_name")
```
3. **Only uncomputed rows** will be processed (rows with NULL values in the target column)
Checkpoints from the previous run are preserved, so you won't lose progress on successfully computed rows.
## Best Practices
### Sequence Your Operations
For the smoothest experience, sequence your operations:
```
1. Complete all data ingestion
2. Run backfill to compute UDF columns
3. Run compaction/optimization
```
### Use Insert-Only Operations During Backfill
If you need to add data while a backfill is running, use insert-only operations:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Safe: INSERT-only merge_insert
tbl.merge_insert("id").when_not_matched_insert_all().execute(new_data)
# Unsafe: Updates to existing rows
tbl.merge_insert("id").when_matched_update_all().execute(data) # May conflict
```
### Monitor Backfill Progress
Use async backfills to monitor progress and handle errors:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
fut = tbl.backfill_async("column_name")
while not fut.done():
time.sleep(1)
# Check for errors before subsequent operations
result = fut.result()
```
### Disable Auto-Compaction During Large Backfills
If using LanceDB Enterprise which has auto-compaction enabled, consider disabling it during large backfill operations to avoid conflicts.
## Related
* [Backfilling](/geneva/jobs/backfilling) - Triggering and configuring backfill operations
* [Advanced Configuration](/geneva/udfs/advanced-configuration) - Environment variables for retry behavior
## API Reference
* [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `add()`, `merge_insert()`, and other table mutation methods
# Geneva Console
Source: https://docs.lancedb.com/geneva/jobs/console
The Geneva Console provides a web-based interface for monitoring and managing Geneva jobs, clusters, and manifests.
![geneva-console]()
## Why a Geneva Console?
* Collaboration: The console helps multiple people work together. Individual jobs can be run in a notebook or workflow, but to collaborate on jobs, it helps to be able to see everything that's running on a given database.
* History: See what has run in the past and diagnose any problems with your jobs.
* Shared resources: The console stores definitions of clusters and manifests, so you can easily tell what resources you want to use to run your job.
## Getting Started
The Geneva console is installed with the Geneva Helm chart; [contact LanceDB](https://lancedb.com/contact/) for access to the Helm chart.
1. Install or upgrade the Geneva Helm chart (see [Helm Deployment](/geneva/deployment/helm/)).
2. In your web browser, connect to the Geneva Console UI using the external ingress/load balancer URI configured in your deployment.
3. **Backup (no external ingress):** if your deployment doesn't expose the console via ingress or a load balancer, forward port 3000 from the `geneva-console-ui` service and open `http://localhost:3000`:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
kubectl port-forward -n lancedb svc/geneva-console-ui 3000:3000
```
(Use `-n` to specify the namespace you installed the Helm chart into. We advise `lancedb`.)
4\. When prompted, enter your bucket and database, like:
```
s3://my-bucket/my-db
```
## What's in the Console?
### Jobs Overview
The heart of the console is an overview of all jobs that are running on a given database. See each job's status, progress, timing, and initiator.
### Job Details
Click on a job's ID to get more details, especially events that have happened in a job's life cycle, and metrics such as number of workers, rows, and fragments written.
### Clusters
See the Geneva clusters that you have defined to run jobs. Because clusters can be reused by name, this view can help you run a new job with the same resource constraints as a previous job.
### Manifests
See the Manifests you've defined and what packages/dependencies they contain. As with clusters, manifests are reusable, so it's easy to start a new job with the same dependencies as an old one by just specifying the manifest name.
## API Reference
* [Connection](https://lancedb.github.io/geneva/api/connection/) — `get_job()`, `list_jobs()`, `list_clusters()`, `list_manifests()`
* [Cluster](https://lancedb.github.io/geneva/api/cluster/) — `GenevaCluster` and cluster configuration classes
* [Manifest](https://lancedb.github.io/geneva/api/manifest/) — `GenevaManifest` and manifest builder classes
# Advanced Execution Contexts
Source: https://docs.lancedb.com/geneva/jobs/contexts
Configure the distributed execution backend — clusters and dependency manifests — for Geneva jobs on object-storage and local-filesystem connections.
**This page applies to direct object-storage and local-filesystem connections only.** On
LanceDB Enterprise (`db://`) connections, distributed job execution is fully managed: the
cluster and manifest are configured at deployment time (see
[Helm deployment](/geneva/deployment/helm)) and can be overridden per job with `cluster=` /
`manifest=` (see [Advanced Job Configuration](/geneva/jobs/advanced-job-configuration)). You
do not define execution contexts there.
When you connect Geneva directly to object storage (`s3://`, `gs://`, …) or a local path,
there is no managed control plane, so you configure the execution backend yourself using the
contexts below. The APIs on this page require Geneva **v0.10.0** or later.
Geneva's distributed execution backend is **Ray**. There are 3 ways to connect to a Ray cluster:
1. Local Ray
2. KubeRay: create a cluster on demand in your Kubernetes cluster.
3. Existing Ray Cluster
## Ray Clusters
### Local Ray
To execute jobs without an external Ray cluster, you can use `LocalRayContext`. This will auto-create a Ray cluster on your machine. Because it's on your laptop/desktop, this is only suitable for prototyping on small datasets. But it is the easiest way to get started. Simply define the UDF, add a column, call [`Connection.local_ray_context()`](https://lancedb.github.io/geneva/api/connection/#geneva.db.Connection.local_ray_context), and trigger the job:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import udf
from geneva.db import Connection
@udf
def filename_len(filename: str) -> int:
return len(filename)
tbl.add_columns({"filename_len": filename_len})
with Connection.local_ray_context():
tbl.backfill("filename_len")
```
Geneva will package up your local environment and send it to each worker process, so they'll have access to all the same dependencies as if you ran a simple Python script yourself.
### KubeRay
If you have a Kubernetes cluster with kuberay-operator, you can use Geneva to automatically provision RayClusters. To do so, define a Geneva cluster, representing the resource needs, Docker images, and other Ray configurations necessary to run your job. Make sure your cluster has adequate compute resources to provision the RayCluster. Here is an example Geneva cluster definition:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import geneva
from geneva.cluster import GenevaCluster, K8sConfigMethod
from geneva.cluster.builder import KubeRayClusterBuilder
db = geneva.connect("s3://my-bucket/my-db")
cluster_name = "my-geneva-cluster" # lowercase, numbers, hyphens only
service_account = "my_k8s_service_account" # k8s service account that Geneva runs as
k8s_namespace = "lancedb" # k8s namespace
cluster = (
GenevaCluster.create_kuberay(cluster_name)
.namespace(k8s_namespace)
.aws_config(region="us-east-1") # only required if using AWS
.config_method(K8sConfigMethod.LOCAL) # Load k8s config from `~/.kube.config`
# (other options include EKS_AUTH to load from AWS EKS, or IN_CLUSTER to load the
# config when running inside a pod in the cluster)
.head_group(
service_account=service_account,
cpus=2,
node_selector={"geneva.lancedb.com/ray-head":""}, # k8s label required for head in your cluster
)
.add_worker_group(
KubeRayClusterBuilder.cpu_worker()
.cpus(4)
.memory("8Gi")
.service_account(service_account)
.build()
)
.add_worker_group(
KubeRayClusterBuilder.gpu_worker() # defaults to 1 GPU
.cpus(2)
.memory("8Gi")
.service_account(service_account)
.build()
)
.build()
)
db.define_cluster(cluster_name, cluster)
# define_cluster stores the cluster metadata in persistent storage. The Cluster can then be referenced by name and provisioned when creating an execution context.
table = db.get_table("my_table")
with db.context(cluster=cluster_name):
table.backfill("my_udf")
```
See the API docs for all the parameters [`GenevaCluster.create_kuberay()`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.mgr.GenevaCluster.create_kuberay), [`add_worker_group()`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.builder.KubeRayClusterBuilder.add_worker_group), [`cpu_worker()`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.builder.KubeRayClusterBuilder.cpu_worker), and [`gpu_worker()`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.builder.KubeRayClusterBuilder.gpu_worker) can use.
#### Exit Modes
When you launch multiple async jobs in a single context, the exit mode controls whether the cluster waits for all of them to finish and how it handles failures. You can customize this behavior with the `on_exit` parameter:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.runners.ray.raycluster import ExitMode
with db.context(cluster=cluster_name, manifest=manifest_name, on_exit=ExitMode.DELETE):
fut1 = tbl.backfill_async("embedding_a")
fut2 = tbl.backfill_async("embedding_b")
# No need to call .result() — the context waits for both jobs
# Both jobs completed; cluster deleted
```
| Exit Mode | Behavior |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ExitMode.DELETE` (default) | Wait for all async jobs in the context to complete, then delete the cluster. Ideal for batch scripts that launch multiple `backfill_async()` calls in one context. |
| `ExitMode.RETAIN_ON_FAILURE` | Wait for all async jobs in the context to complete. If any job failed, the context body raised an exception, or `wait_timeout` was exceeded, retain the cluster for debugging; otherwise delete. |
| `ExitMode.RETAIN` | Never delete the cluster, regardless of job outcomes. Useful for notebooks and interactive sessions where you run multiple jobs over time. |
### External Ray cluster
If you already have a Ray cluster, Geneva can execute jobs against it too. You do so by defining a Geneva cluster with [`GenevaCluster.create_external()`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.mgr.GenevaCluster.create_external) which has the address of the cluster. Here's an example:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import geneva
from geneva.cluster import GenevaCluster
db = geneva.connect(my_db_uri)
cluster_name = "my-geneva-external-cluster"
cluster = (
GenevaCluster.create_external(cluster_name, "ray://my_ip:my_port")
.build()
)
db.define_cluster(cluster_name, cluster)
```
If you need to send environment variables to your workers, in either a KubeRay or External Ray cluster, you can use [`ray_init_kwargs`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.builder.KubeRayClusterBuilder.ray_init_kwargs), like so:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
cluster = (
GenevaCluster.create_kuberay(cluster_name) # or create_external(cluster_name)
.ray_init_kwargs({
"runtime_env": {
"env_vars": {
"MY_VAR": "value",
"AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"]
},
},
})
...
.build()
)
```
## Dependencies and Manifests
Most UDFs require some dependencies: helper libraries like `pillow` for image processing, pre-trained models like `open-clip` to calculate embeddings, or even small config files. We have three ways to get them to workers:
1. Define dependencies explicitly in a manifest
2. Bake dependencies into an image
3. Auto-upload local dependencies
### Define dependencies explicitly in a manifest
We recommend defining dependencies explicitly: it's the easiest way to understand exactly what's running, and the least error-prone. To do so, define a Manifest, like so:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.manifest import GenevaManifest
db = geneva.connect(my_db_uri)
manifest_name="dev-manifest"
manifest = (
GenevaManifest.create_pip(manifest_name)
.pip(["lancedb", "numpy"])
).build()
db.define_manifest(manifest_name, manifest)
```
After workers start up, this will run `pip install lancedb numpy` on them. You can also use [`.requirements_path(path)`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.builder.PipManifestBuilder.requirements_path) to point to a local `requirements.txt` file instead of listing packages inline. Note that attempting to use both [`.pip()`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.builder.PipManifestBuilder.pip) and `.requirements_path()` will raise an exception.
For conda-based dependencies, use [`GenevaManifest.create_conda()`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.mgr.GenevaManifest.create_conda) instead:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.manifest import GenevaManifest
manifest = (
GenevaManifest.create_conda("my-conda-manifest")
.conda({"dependencies": ["python=3.10", "numpy"]})
).build()
```
You can also use [`.conda_environment_path(path)`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.builder.CondaManifestBuilder.conda_environment_path) to point to a local `environment.yml` file. Note that attempting to use both [`.conda()`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.builder.CondaManifestBuilder.conda) and `.conda_environment_path()` will raise an exception.
### Bake dependencies into an image
Because the `pip` or `conda` methods involve installing packages, they will incur some startup costs. When your jobs are stable in production, therefore, it will be faster to build all your dependencies into the workers' images, then specify them like so:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.manifest import GenevaManifest
db = geneva.connect(my_db_uri)
manifest_name = "prod-manifest"
manifest = (
GenevaManifest.create_pip(manifest_name)
.worker_image("myregistry.example.com/my-custom-worker-image:latest")
).build()
db.define_manifest(manifest_name, manifest)
```
You can also define images in a cluster via the [`head_group`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.builder.KubeRayClusterBuilder.head_group) method and [`cpu_worker`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.builder.CpuWorkerBuilder)/[`gpu_worker`](https://lancedb.github.io/geneva/api/cluster/#geneva.cluster.builder.GpuWorkerBuilder) methods, e.g.:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
cluster = (
GenevaCluster.create_kuberay(cluster_name)
.head_group(
image="myregistry.example.com/my-custom-head-image:latest",
)
.add_worker_group(KubeRayClusterBuilder.cpu_worker()
.image("myregistry.example.com/my-custom-cpu-worker-image:latest")
)
.add_worker_group(KubeRayClusterBuilder.gpu_worker()
.image("myregistry.example.com/my-custom-gpu-worker-image:latest")
)
.build()
)
```
However, if an image is defined in both a Cluster and a Manifest, the definition in the Manifest will take priority.
### Auto-upload local dependencies
Geneva can package your local environment and send it to Ray workers. This includes the current workspace root (if you're in a python repo) or the current working directory (if you're not). [`GenevaManifest.create_site()`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.mgr.GenevaManifest.create_site) additionally uploads your Python site-packages (defined by `site.getsitepackages()`) to workers. This is not recommended for production use, as it is prone to issues like architecture mismatches of built dependencies, but it can be a good way to iterate quickly during development.
To upload site packages:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.manifest import GenevaManifest
db = geneva.connect(my_db_uri)
manifest_name = "dev-manifest"
manifest = GenevaManifest.create_site(manifest_name).build()
db.define_manifest(manifest_name, manifest)
```
### What's in a manifest?
Here's a summary of what's in a manifest and how you can define it across the three manifest builder types.
| Contents | How you can define it | Factory method |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Local working directory (or workspace root, if in a python repo) | Will be uploaded automatically. | All |
| Local python packages (site-packages) | Uploaded automatically. | [`create_site()`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.mgr.GenevaManifest.create_site) |
| Pip packages to be installed | Use [`.pip(packages: list[str])`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.builder.PipManifestBuilder.pip) or [`.requirements_path(path: str)`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.builder.PipManifestBuilder.requirements_path). See [Ray's RuntimeEnv docs](https://docs.ray.io/en/latest/ray-core/api/doc/ray.runtime_env.RuntimeEnv.html) for details. | [`create_pip()`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.mgr.GenevaManifest.create_pip) |
| Conda packages to be installed | Use [`.conda(deps: dict[str, Any])`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.builder.CondaManifestBuilder.conda) or [`.conda_environment_path(path: str)`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.builder.CondaManifestBuilder.conda_environment_path). See [Ray's RuntimeEnv docs](https://docs.ray.io/en/latest/ray-core/api/doc/ray.runtime_env.RuntimeEnv.html) for details. | [`create_conda()`](https://lancedb.github.io/geneva/api/manifest/#geneva.manifest.mgr.GenevaManifest.create_conda) |
| Local python packages outside of `site_packages` | Use `.py_modules(modules: list[str])` or `.add_py_module(module: str)`. See [Ray's RuntimeEnv docs](https://docs.ray.io/en/latest/ray-core/api/doc/ray.runtime_env.RuntimeEnv.html) for details. | All |
| Container image for head node | Use `.head_image(head_image: str)` or `default_head_image()` to use the default. Note that, if the image is also defined in the GenevaCluster, the image set here in the Manifest will take priority. | All |
| Container image for worker nodes | Use `.worker_image(worker_image: str)` or `default_worker_image()` to use the default for the current platform. As with the head image, this takes priority over any images set in the Cluster. | All |
If you want to see exactly what is being uploaded to the cluster, set `.delete_local_zips(False)` and `.local_zip_output_dir(path)` then examine the zip files in `path`.
## Putting it all together: Execution Contexts
An execution context represents the concrete execution environment (Cluster and Manifest) used to execute a distributed job.
Calling `context` will enter a context manager that will provision an execution cluster and execute the Job using the Cluster and Manifest definitions provided. Because you've already defined the cluster and manifest, you can just reference them by name. Note that providing a manifest is optional. Once completed, the context manager will automatically de-provision the cluster.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
db = geneva.connect(my_db_uri)
tbl = db.get_table("my_table")
with db.context(cluster=cluster_name, manifest=manifest_name):
tbl.backfill("embedding")
```
In a notebook environment, you can manually enter and exit the context manager in multiple steps like so:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
ctx = db.context(cluster=cluster_name, manifest=manifest_name)
ctx.__enter__()
# ... do stuff
ctx.__exit__(None,None,None)
```
## API Reference
* [Connection](https://lancedb.github.io/geneva/api/connection/) — `context()`, `local_ray_context()`, `list_clusters()`, `list_manifests()`
* [Cluster](https://lancedb.github.io/geneva/api/cluster/) — `KubeRayClusterBuilder`, `LocalRayClusterBuilder`, `ExternalRayClusterBuilder`, and worker configuration classes
* [Manifest](https://lancedb.github.io/geneva/api/manifest/) — `PipManifestBuilder`, `CondaManifestBuilder`, `SiteManifestBuilder`
# Geneva Jobs
Source: https://docs.lancedb.com/geneva/jobs/index
Learn about Geneva's job execution framework for distributed feature engineering workflows.
Geneva provides a comprehensive job execution framework for distributed feature engineering workflows. This section covers the different types of jobs and execution contexts available in Geneva.
## Job Types
### [Backfilling](/geneva/jobs/backfilling/)
Trigger distributed jobs to populate column values in your LanceDB table using UDFs. Learn about filtered backfills and incremental updates.
### [Materialized Views](/geneva/jobs/materialized-views/)
Create declarative materialized views to manage batch updates of expensive operations. Optimize data layouts for training and simplify orchestration.
### [Startup Optimizations](/geneva/jobs/startup/)
Optimize job and session startup times for faster interactive development and production workflows. Learn about caching, pre-warming, and performance tuning.
## Execution Contexts
### [Advanced Execution Contexts](/geneva/jobs/contexts/)
Understand how Geneva automatically packages and deploys your Python execution environment to worker nodes for distributed execution.
### [Geneva Console](/geneva/jobs/console/)
Set up and access the Geneva Console for monitoring and managing Geneva jobs, clusters, and execution contexts.
## Key Features
* **Distributed Processing**: Scale feature computation across multiple nodes
* **Checkpointing**: Resume jobs from failures without losing progress
* **Incremental Updates**: Only process new or modified data
* **Distributed Execution**: Run jobs on Kubernetes or standalone compute clusters
* **Environment Management**: Automatic dependency packaging and deployment
## Getting Started
1. **Choose your execution context** based on your infrastructure
2. **Define your UDFs** for feature computation
3. **Trigger backfill operations** to populate your data
4. **Monitor performance** and optimize based on usage patterns
For detailed information about each job type and execution context, explore the documentation in this section.
## API Reference
* [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `backfill_async()`, `refresh()`, `plan_backfill()`, `plan_refresh()`, and other job-triggering methods
* [Connection](https://lancedb.github.io/geneva/api/connection/) — `get_job()`, `list_jobs()`, `context()`, and `local_ray_context()`
* [Cluster](https://lancedb.github.io/geneva/api/cluster/) — configure KubeRay and local Ray execution backends
* [Manifest](https://lancedb.github.io/geneva/api/manifest/) — package Python environments for remote workers
# Job Metrics (Diagnostics)
Source: https://docs.lancedb.com/geneva/jobs/job_metrics
Use Geneva Job Metrics to monitor and troubleshoot jobs in real time.
## How to find metrics
Job metrics can be found in the [Geneva Console UI](https://docs.lancedb.com/geneva/jobs/console), by clicking on a job's ID to get to the "Job details" page.
## Core diagnostic metrics
| Metric | What it means | Common signal |
| ---------------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------- |
| `rows_checkpointed` | Rows finished by read/UDF/checkpoint stage. | High value means upstream compute is progressing. |
| `rows_ready_for_commit` | Rows ready for atomic commit (becoming visible to other DB connections). | If much lower than `rows_checkpointed`, writer path is likely bottlenecked. |
| `rows_committed` | Rows already visible to other DB connections. | If lagging far behind `rows_ready_for_commit`, commit stage may be bottlenecked. |
| `cnt_geneva_workers_active` | Current parallel UDF executors. | Lower than expected means reduced effective parallelism. |
| `cnt_geneva_workers_pending` | Deficit from desired parallelism. | Persistently high value usually means scheduling/resource pressure. |
| `read_io_time_ms` | Cumulative read IO time. | Dominant value suggests storage/read bottleneck. |
| `udf_processing_time` | Cumulative UDF execution time. | Dominant value suggests compute/UDF bottleneck. |
| `batch_checkpointing_time` | Cumulative batch checkpoint overhead. | High value suggests checkpoint overhead is expensive. |
| `writer_write_time` | Cumulative writer output time. | High value often points to object storage throughput/throttling issues. |
| `writer_queue_wait_time_ms` | Cumulative writer queue wait time. | High value can indicate writer starvation/backpressure. |
| `commit_time_ms` | Cumulative commit time. | High value means commit itself is expensive. |
| `commit_conflict_retries` | Commit retries due to version conflicts. | Non-trivial counts indicate commit contention. |
| `commit_backoff_time_ms` | Time spent backing off during commit retries. | High value indicates contention/retry pressure. |
| `commit_concurrent_writer_retries` | Retries from "Too many concurrent writers". | High value indicates writer concurrency contention. |
## Quick diagnosis workflow
1. Check `rows_checkpointed` vs `rows_ready_for_commit`.
* If `rows_checkpointed` is high but `rows_ready_for_commit` is low, fragment
writer is usually the bottleneck.
* This often indicates object storage read/write pressure (for example S3).
2. Compare read, UDF, and checkpoint timing.
* High `read_io_time_ms`: storage or scan bottleneck.
* High `udf_processing_time`: UDF compute bottleneck.
* High `batch_checkpointing_time`: checkpoint overhead bottleneck.
* Typical mitigations: increase `checkpoint_size`, increase
`max_checkpoint_size`, or compact the table to produce larger fragments.
3. Check writer timing.
* High `writer_write_time` is commonly object storage throttling/throughput
limit.
* Typical mitigations: use higher network-bandwidth node types, and keep
object storage and compute nodes in the same region.
4. Check commit pressure.
* High `commit_conflict_retries`, `commit_backoff_time_ms`, or
`commit_concurrent_writer_retries` indicates commit contention.
5. Check parallelism deficit.
* If `cnt_geneva_workers_pending` stays high while
`cnt_geneva_workers_active` stays low, the job is running below desired
parallelism due to cluster/resource constraints.
## Notes
* Timing metrics are cumulative and may overlap; do not sum them as exact wall
time.
* For completed jobs, row counters should settle to stable final values.
# Job Lifecycle
Source: https://docs.lancedb.com/geneva/jobs/lifecycle
Understanding how Geneva jobs work, their lifecycle states, and how to monitor and manage them.
Geneva uses background jobs to execute long-running operations like backfills and materialized view refreshes. This guide explains how jobs work, their lifecycle states, and how to monitor and manage them.
## Overview
Jobs in Geneva are asynchronous operations that process data in the background. There are two primary job types:
| Job Type | Purpose | Created By |
| ----------------------------- | -------------------------------- | ------------------ |
| **Backfill** | Compute column values using UDFs | `table.backfill()` |
| **Materialized View Refresh** | Update precomputed query results | `view.refresh()` |
Both job types share the same lifecycle states and monitoring capabilities.
## Job States
Every job progresses through a well-defined state machine:
```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
stateDiagram-v2
[*] --> PENDING
PENDING --> RUNNING
RUNNING --> DONE
RUNNING --> FAILED
DONE --> [*]
FAILED --> [*]
```
| State | Description |
| ----------- | ------------------------------------------------ |
| **PENDING** | Job has been created and is queued for execution |
| **RUNNING** | Job is actively processing data |
| **DONE** | Job completed successfully |
| **FAILED** | Job encountered an error during execution |
## Monitoring Jobs
The [Geneva Console](/geneva/jobs/console) provides a web-based interface for monitoring job status, progress, and history across your database. This is the recommended way to track jobs in collaborative environments.
For programmatic access, you can query job status directly via the API:
### Querying Job Status
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import geneva
db = geneva.connect("/path/to/db")
# Get job state manager
jsm = db._history
# Get a specific job
job = jsm.get(job_id)[0]
print(f"Status: {job.status}")
print(f"Started: {job.launched_at}")
print(f"Completed: {job.completed_at}")
# List jobs for a table
pending_jobs = jsm.list_jobs(table_name="my_table", status="PENDING")
running_jobs = jsm.list_jobs(table_name="my_table", status="RUNNING")
```
### Progress Metrics
Jobs report progress through metrics:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Access job metrics
for metric in job.metrics:
print(f"{metric['name']}: {metric['count']}/{metric['total']}")
```
Common metrics include:
| Metric | Description |
| --------------------- | ------------------------------------------------------------------------------------------ |
| `fragments` | [Fragments](https://lance.org/format/table/?h=fragment#fragments) scheduled for processing |
| `writer_fragments` | Fragments written to storage |
| `udf_values_computed` | Rows processed by UDFs |
| `rows_checkpointed` | Rows saved to checkpoint store |
| `rows_committed` | Rows committed to the table |
| `workers` | Workers started for parallel execution |
### Job Events
Jobs log significant events during execution:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
for event in job.events:
print(f"{event['timestamp']}: {event['message']}")
```
Example events:
* "Job started"
* "Checkpointing complete for fragment 42"
* "Partial commit: 64 fragments"
* "Job completed successfully"
## Fault Tolerance
Geneva jobs are designed to be resilient to failures:
### Checkpoint-Based Recovery
Jobs save intermediate results to a checkpoint store. If a job fails:
1. **Completed work is preserved** - Checkpointed batches are not lost
2. **Resume from checkpoint** - Restarted jobs skip already-processed data
3. **No duplicate processing** - Each batch is processed exactly once
By default, checkpoints are stored in a `_ckp/` subdirectory inside the table's storage location. At scale, you can redirect checkpoints to a separate bucket to avoid IOPS contention. See [Checkpoint Storage configuration](/geneva/udfs/advanced-configuration#checkpoint-storage) for details.
### Resuming Failed Jobs
To resume a failed job, simply re-run the same backfill or refresh command. The job will automatically detect existing checkpoints, skip already-processed fragments, and continue from where it left off.
## API Reference
* [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `backfill_async()`, `refresh()`, `JobFuture`
* [Connection](https://lancedb.github.io/geneva/api/connection/) — `get_job()`, `list_jobs()`
# Materialized Views with UDFs
Source: https://docs.lancedb.com/geneva/jobs/materialized-views
Learn how to use Geneva's materialized view feature to declaratively manage batch updates of expensive operations using UDFs.
Geneva provides a materialized view feature that can be used to declaratively manage "batch" updates of expensive operations such as populating UDF columns. These updates are triggered via refresh operation. This can be used to optimize data layouts for training and to simplify some operations that traditionally may require external procedural orchestration (airflow, prefect, dagster).
> **Note**: This is similar to how traditional databases offer a materialized view feature to declaratively manage expensive aggregation and join operations.
## Process Overview
The process is straightforward:
1. Define a query on table, optionally including UDFs in the select clause.
2. Create the materialized view using `db.create_materialized_view(...)`.
3. Populate the new materialized view table using the `refresh` operation.
Just like with backfills, this operation is incremental, checkpointed, and run in a distributed manner.
## Example
Let's walk through an example using a raw video table as a base. We want to create a materialized view off the table that adds transcription columns to a subset of the values.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import connect, udf
import pyarrow as pa
db = connect("/path/to/lancedb")
schema = pa.schema([
pa.field("video_id", pa.int64(), nullable=False),
pa.field("video_uri", pa.string(), nullable=False),
pa.field("upload_ts", pa.timestamp("us"), nullable=False),
pa.field("metadata", pa.json(), nullable=True),
])
raw_videos = db.create_table(
"raw_videos",
schema=schema,
primary_key="video_id"
)
```
Here's our UDFs, and the creation of a new empty materialized view.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf
def transcribe(video_uri) -> str:
from whisper import load_model
model = load_model("base")
return model.transcribe(uri)["text"]
@udf(data_type=pa.binary())
def load_video(video_uri: pa.Array) -> pa.Array:
videos =
return ...
q = raw_videos.search(None)
.shuffle(seed=42)
.select(
{
"video_uri": "video_uri",
"video": load_video,
"transcription": transcribe,
}
)
view_table = db.create_materialized_view("table_view", q)
```
To populate the values, we call `refresh`.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# explicitly copy values from the source table, applying UDF on cols.
db.refresh("table_view")
```
Note that the UDF is stored on the destination materialized view table.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
raw_table.add(...)
db.refresh("table_view") # only materialize new or modified rows.
```
The operation is incremental. So the next time refresh on the table is called, only new fragments with new data get materialized into the materialized view table.
Materialized views are just tables so you can query them as well as modify them by adding new `add_columns`, `backfill` particular columns and deriving other materialized views or views from them.
Reference:
* [`create_materialized_view` API](https://lancedb.github.io/geneva/api/connection/#geneva.db.Connection.create_materialized_view)
## FAQ
### Do we copy the UDFs from the source table?
No. The UDF does not but any UDF calculated values in the original table come to the materialized table via refresh. New columns defined by the UDFs in the materialized view creation are attached only to the materialized view. They can be backfilled (since the UDF belongs to the view) or refreshed.
### On MV refresh, do we force materialization of UDFs cols on the source table?
No. They are managed at the source table only. If it is null the null values are propagated. Future options may force materialization/backfill "recursively".
## API Reference
* [Connection](https://lancedb.github.io/geneva/api/connection/) — `create_materialized_view()`, `create_view()`
* [Table](https://lancedb.github.io/geneva/api/table/) — `refresh()`, `plan_refresh()`
* [Query](https://lancedb.github.io/geneva/api/query/) — `create_materialized_view()` on query builder
# Distributed Job Performance and Cluster Sizing
Source: https://docs.lancedb.com/geneva/jobs/performance
Learn how to tune Geneva distributed job performance by scaling compute resources and balancing write bandwidth.
When Geneva runs in distributed mode, jobs are deployed against a Kubernetes KubeRay instance that dynamically provisions a Ray cluster. Job execution time depends on sufficient CPU/GPU resources for *computation* and sufficient *write bandwidth* to store the output values. Tuning the performance of a job boils down to configuring the table or cluster resources.
## Geneva defaults
Geneva sets the following defaults when creating a KubeRay cluster via GenevaClusterBuilder. These apply as both Kubernetes requests and limits.
### Head Node
| Resource | Default |
| --------------- | ----------------------------------- |
| CPU | 4 |
| Memory | 8 GiB |
| GPU | 0 |
| Node Selector | `geneva.lancedb.com/ray-head: true` |
| Service Account | `geneva-service-account` |
The head node runs the Ray GCS (Global Control Store), the driver task, and the dashboard. It does not run UDF applier actors.
### CPU Workers
| Resource | Default |
| ------------- | ----------------------------------------- |
| CPU | 4 |
| Memory | 8 GiB |
| GPU | 0 |
| Node Selector | `geneva.lancedb.com/ray-worker-cpu: true` |
| Replicas | 1 (min: 0, max: 100) |
| Idle Timeout | 60 seconds |
### GPU Workers
| Resource | Default |
| ------------- | ----------------------------------------- |
| CPU | 8 |
| Memory | 16 GiB |
| GPU | 1 |
| Node Selector | `geneva.lancedb.com/ray-worker-gpu: true` |
| Replicas | 1 (min: 0, max: 100) |
| Idle Timeout | 60 seconds |
## Scaling computation resources
Geneva jobs can split and schedule computational work into smaller batches that are assigned to *tasks* which are distributed across the cluster. As each task completes, it writes its output into a checkpoint file. If a job is interrupted or run again, Geneva will look to see if a checkpoint for the computation is already present and if not will kick off computations.
Usually computation capacity is the bottleneck for job execution. To complete all of a job's tasks more quickly, you just need to increase the amount of CPU/GPU resources available.
### Estimating CPU and memory requirements
#### CPU
Geneva can run up to `concurrency` tasks in parallel. Within each task, it can run up to `intra_applier_concurrency` UDF applications concurrently. If your UDF requests `udf.num_cpus` CPUs, the peak CPU required is approximately:
`total_cpus ≈ concurrency * intra_applier_concurrency * udf.num_cpus`
`concurrency` and `intra_applier_concurrency` are parameters on `Table.backfill(...)`.
In most cases you do not need to change `udf.num_cpus`, unless the UDF itself uses multiple threads. If your UDF is single-threaded but you want to speed it up with more parallelism, prefer increasing `concurrency` first. If you stop seeing meaningful improvements, then try increasing `intra_applier_concurrency`.
#### Memory
Geneva defaults to checkpoint batches of up to 100 rows (`checkpoint_size=100`). Peak memory usage scales with both the input rows held for processing and the output rows buffered before they are written. A rough upper bound is:
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
memory_bytes ≈
input_row_size * min(checkpoint_size * worker_num_cpus, task_size) +
output_row_size * min(checkpoint_size * intra_applier_concurrency, task_size)
```
* `worker_num_cpus` is the number of CPUs on the Ray worker.
* `task_size` is a tunable parameter (rows per task). It can be set on the UDF (e.g. `@udf(..., task_size=...)`) and on `Table.backfill(..., task_size=...)` (backfill-level values take precedence). If `task_size` is larger than the number of rows in a fragment, it behaves the same as `task_size = fragment_num_rows`. By default, it is:
`task_size = num_rows / (concurrency * intra_applier_concurrency * 2)` (where `num_rows` is the number of rows in the table, or the number of rows selected by your filter)
* `input_row_size` / `output_row_size` are the average bytes per row after materialization in memory.
For primitive numeric columns (ints/floats), these sizes are usually small. For images, videos, embeddings, etc. you may need to explicitly provision more memory (e.g. via `@udf(..., memory=...)`) and/or reduce `checkpoint_size` / `task_size`.
Typical per-row sizes:
* Images: \~200KB–2MB (depends on dataset and encoding)
* Videos: \~10MB–200MB
* Embeddings: `dimension * data_type_size` bytes (e.g. float32 embeddings use 4 bytes per value, so a 1536-dim embedding is `1536 * 4 = 6144` bytes)
#### Internal Actor Resource Overhead
In addition to UDF applier actors, each backfill job creates internal actors that consume resources:
| Component | CPU | Memory | Count |
| -------------- | ------------------------------------------ | ---------- | ------------- |
| Driver task | 0.1 | — | 1 |
| JobTracker | 0.1 | 128 MiB | 1 |
| Writer actors | 0.1 | 1 GiB | 1 per applier |
| Queue actors | 0 | — | 1 per applier |
| Applier actors | UDF `num_cpus * intra_applier_concurrency` | UDF memory | `concurrency` |
#### Example
For a job with `concurrency=4`, `intra_applier_concurrency=2`, with a UDF that requests 1 CPU and 1GiB memory:
| Component | CPU | Memory | Count |
| -------------- | --------------------------- | ----------------------------------------------- | ----- |
| Driver task | 0.1 | — | 1 |
| JobTracker | 0.1 | 128 MiB | 1 |
| Writer actors | 0.1 | 1 GiB | 4 |
| Queue actors | 0 | — | 4 |
| Applier actors | 1\*2=2 | 1 GiB | 4 |
| Total | 0.1+0.1+4*0.1+4*2 = **8.6** | 128 MiB + 4 \* 1GiB + 4 \* 1GiB = **8.125 GiB** | |
### Overall Cluster Sizing
Of course, the size of a cluster will vary dramatically for each task. But if you don't know how to estimate your workload, we can recommend the following cluster sizes as a starting point:
| Cluster Size | Head Node | Workers |
| ---------------------- | ----------- | ----------------- |
| **Small** (Dev / CI) | 1 CPU, 8 GB | 4 x 2 CPU, 8 GB |
| **Medium** (Staging) | 2 CPU, 8 GB | 4 x 4 CPU, 8 GB |
| **Large** (Production) | 4 CPU, 8 GB | 8+ x 8 CPU, 16 GB |
### Validation Thresholds
The cluster builder validates memory configuration and warns or errors on suspicious values:
| Threshold | Value | Behavior |
| ---------------------------- | ------------------ | ------------------------------------------------- |
| GPU worker minimum memory | `< 4 GiB` | **Error** — build fails if below this |
| Large memory warning | `> 100 GB` | **Warning** — may exceed K8s node capacity |
| Memory-per-CPU ratio warning | `> 16 GiB per CPU` | **Warning** — unusual ratio, likely misconfigured |
### GKE node pools
GKE + KubeRay can autoscale the number of VM nodes on demand. Limitations on the amount of resources provisioned are configured via [node pools](https://cloud.google.com/kubernetes-engine/docs/how-to/node-pools#scale-node-pool). Node pools can be managed to scale vertically (type of machine) or horizontally (# of nodes).
Properly applying Kubernetes labels to the node pool machines allows you to control resources for different jobs in your cluster.
### Options on `Table.backfill(..)`
The `Table.backfill(..)` method has several optional arguments to tune performance. To saturate the CPUs in the cluster, the main arguments to change are `concurrency` which controls the number of task processes and `intra_applier_concurrency` which controls the number of task threads per task process.
`commit_granularity` controls how frequently fragments are committed so that partial results can become visible to table readers.
Setting `checkpoint_size` smaller introduces finer-grained checkpoints and can help provide more frequent proof of life as a job is being executed. This is useful if the computation on your data is expensive.
Reference:
* [`backfill` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.backfill)
* [`backfill_async` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.backfill_async)
## Balancing write bandwidth
While computation can be broken down into small tasks, new Lance column data for each fragment must be written out in a serialized fashion. Each fragment has a writer that waits for checkpointed results to arrive, sequences them, and then serially writes out the new data file.
Writers can be a bottleneck if a Lance dataset has a small number of fragments, especially if the amount of data being written out is comparatively large. Maximizing parallel write throughput can be achieved by having more fragments than nodes in the cluster.
### Symptom: Computation tasks complete but writers seem to hang
Certain jobs that take a small dataset and expand it may appear as if the writer has frozen.
An example is a table that contains a list of URLs pointing to large media files. This list is relatively small (\< 100MB) and can fit into a single fragment. A UDF that downloads will fetch all the data and then attempt to write all of it out through the single writer. This single writer can then be responsible for serially writing out 500+GB of data to a single file!
To mitigate this, you can load your initial table so that there will be multiple fragments. Each fragment with new outputs can be written in parallel with higher write throughput.
## API Reference
* [Cluster](https://lancedb.github.io/geneva/api/cluster/) — `KubeRayClusterBuilder`, `CpuWorkerBuilder`, `GpuWorkerBuilder` — configure CPU/GPU/memory resources
* [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `compact_files()`, `optimize()`
* [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator options: `num_cpus`, `num_gpus`, `memory`, `batch_size`
# Job and Session Startup Optimizations
Source: https://docs.lancedb.com/geneva/jobs/startup
Learn how to optimize Geneva job and session startup times for faster interactive development and production workflows.
During interactive sessions, there are two main actions where you would interact with Geneva.
* Compute cluster creation
* Job execution
Behind the scenes, Geneva packages your python environment and auto-provisions nodes to execute the jobs. This can be time consuming, taking on the order of 5mins to complete before any work is done. The following sections will describe what happens in these steps and how to diagnose and speed up these interactions.
## Compute cluster creation
To execute a Geneva job, you'll need to initialize a compute environment. Here's the basic steps Geneva takes to instantiate that cluster:
* User requests a cluster
* Scan workspace's python path for modules
* Generate local workspace directory zip
* Generate python site-packages directory zip(s)
* Generate other dirs zip (may include your .venv)
* Upload zips
* Provision head node
* Initialize head node
The requests to create an environment can take 5-10 mins to initiate. The most time-consuming steps are generating directory zips and uploading them. AI workloads often require many module packages and can be dependent on specific versions to work. Common modules required for GPU use to run model inferrence can easily be 5GB-10GB of compressed content. On GCE for example, this can take \~5mins to zip all this and \~1min to upload all of this data.
To speed this up, Geneva employs caching to help optimize the startup time. There are a few things you can do to make subsequent runs faster, often times \<1 minute:
### Hashing and Caching
Geneva generates a hash of each path in the python path that takes into account files and their last modified time. After the first time a directory zip is created and uploaded, the cached copy is used and no new zip is generated or uploaded. However, if there are any changes (e.g. new module added or upgraded) a new hash created and the environment's content is zipped and uploaded.
### Isolate dynamic code and modules
If you use a Jupyter notebook environment for your driver, the content of the `.ipynb` file is constantly changing. This means the hash for the directory that contains the notebook will change, even if the subdirectories do not. If your notebook is in your home directory, this could pull in large amounts unneeded code and data. To avoid this you can move your notebook into a subdirectory with no children. When your notebook is executed it is updated but only the notebook content is resent. Other path directories are unchanged, have the same hash and can skip zip and ship.
### Package dependecies into a docker image
Geneva has an option to skip the zip and ship of the site-packages. Enabling this assumes that the default docker image is overriden with a custom image that has the `site-package` content preloaded.
### Pre-provision nodes and pods:
In your kubernetes configuration, you can tag specific nodes with `geneva.lancedb.com/ray-head` k8s label. These nodes should be configured to be on non-spot instances that are always up. This makes initial kuberay cluster creation quick.
## Job execution
A backfill or materialized view jobs triggers the provisioning of worker nodes that will perform the computations and writes. A cold start can be slow because several steps must take place before the UDFs can be applied. However, once nodes and pods are warmed up, the time between submission and execution can be quick.
Here's the basic steps Geneva takes to kick off a Geneva job:
* User submits job (backfill)
* plan scans
* provision worker nodes (vms)
* load vm
* Autoscale workers nodes
* provision worker nodes (vms)
* load vm
* schedule ray actors
* download docker images
* download zips
* execute udf
* orchestrate fragment write.
In practice, planning the initial distributed step scans require loading vm and pod images. With a cold start, this can take \~5 minutes.
Here are some steps you can take to pre-warming worker nodes and pods so that exectuion can be more interactive:
**Set worker spec's replicas or min\_replicas to a value >0:** When the kuberay cluster is instantiated this also pre-provision vm's so they are ready for k8s to place pod. replicas (for initial # of worker nodes), and minWorkers (to keep a pool for nodes always up)
**Make a warmup call:** Making an initial request to ray will load the pod and zips content to the worker node so that subsequent startups will be fast.
**Prevent nodes from auto-scaling down:** During cluster creation, you can specify `idle_timeout_seconds` option -- this is the amount of time before a node needs to be idle before it is considered for de-provisioning.
# Troubleshooting Geneva Jobs
Source: https://docs.lancedb.com/geneva/jobs/troubleshooting
Diagnose and fix common issues when running Geneva backfills, materialized views, and other distributed jobs.
This page covers common problems you may hit while running Geneva jobs and how to resolve them. For deployment and cluster issues, see [Troubleshooting Geneva Deployments](/geneva/deployment/troubleshooting).
## Admission control and resource errors
Geneva runs **admission control** before starting a job to check that the cluster has enough resources. If the check fails, the job is rejected with `ResourcesUnavailableError` (or a warning if admission is not strict).
### "Job requires GPUs but cluster has no GPU worker groups configured"
Your UDF requests GPUs but the cluster has no GPU nodes.
**Check your UDF:**
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@geneva.udf(num_gpus=1) # This requires GPUs
def my_udf(x): ...
```
**Fix options:**
1. Remove the GPU requirement if the UDF can run on CPU: `@geneva.udf(num_gpus=0)` (or omit `num_gpus`).
2. Make sure you have a GPU worker group; e.g. in your ClusterBuilder:
```
KubeRayClusterBuilder.create()
...
.add_worker_group(
KubeRayClusterBuilder.gpu_worker(1)
.image(get_ray_image(ray.__version__, _py, gpu=True))
.service_account("geneva-service-account")
.node_selector({"geneva.lancedb.com/ray-worker-gpu": "true"})
.build()
)
```
3. Add GPU worker nodes to your cluster and ensure the cluster definition includes a GPU worker group with the correct node selector (e.g. `"geneva.lancedb.com/ray-worker-gpu": "true"`).
### "UDF requires X CPUs + Y GPUs but no worker group can satisfy all requirements."
Of course, if your cluster doesn't have enough GPUs, add them! But if you're surprised how many CPUs/GPUs your UDF is requesting, it may be because of concurrency. Your tasks's actual CPU/GPU requirement is `concurrency × CPUs/GPUs per task`.
**Fix:** Lower `concurrency` so that the total CPUs/GPUs needed (concurrency × CPUs/GPUs per task) does not exceed the cluster's CPU/GPU count.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Before: backfill with `num_gpus(1)` needs 8 GPUs
table.backfill("col", concurrency=8)
# After: cap at 4 GPUs
table.backfill("col", concurrency=4)
```
### "No single node can satisfy all requirements"
The UDF asks for a combination of CPU, memory, and GPU that no single node in the cluster has. This often happens on heterogeneous clusters.
**Example:** Node A has 8 CPUs and 4 GB memory; Node B has 4 CPUs and 8 GB memory. A UDF that needs 8 CPUs and 8 GB cannot be placed on either node.
**Fix options:**
1. **Reduce UDF resource requests** so they fit on your smallest target node (e.g. lower `num_cpus` or memory).
2. **Add larger nodes** that can satisfy CPU, memory, and GPU together.
3. **Use a homogeneous cluster** where nodes have the same shape.
### Job passes admission but hangs at low progress
If admission control passes but the job stalls at a low percentage:
**1. Ray dashboard – see what’s actually running**
* **Local Ray:** After starting a local cluster, Ray usually prints the dashboard URL (e.g. `http://127.0.0.1:8265`). Open that in a browser. If you didn’t capture it, the dashboard is typically on port **8265** on the host where Ray was started.
* **KubeRay (Kubernetes):** Port-forward the dashboard from the Ray head pod, then open it locally:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Find the Ray head pod (replace NAMESPACE and cluster name as needed)
kubectl get pods -n NAMESPACE -l ray.io/node-type=head
# Forward the dashboard port (Ray uses 8265 for the dashboard)
kubectl port-forward -n NAMESPACE POD_NAME 8265:8265
```
Then open **[http://localhost:8265](http://localhost:8265)** in your browser.
* **External Ray cluster:** Use the dashboard URL your cluster operator provides (often the head node’s port 8265).
**What to look for in the dashboard:**
* **Actors** (or **State** → **Actors**): A long list of actors stuck in **PENDING** means Ray cannot place them (e.g. not enough CPUs, GPUs, or memory on any node). That matches “admission passed but nothing runs.”
* **Tasks / Jobs:** If your Geneva job shows as a Ray job, open it and check for tasks that stay **PENDING** or **RUNNING** for a long time without finishing. Pending tasks often mean insufficient resources or that workers aren’t joining.
* **Cluster / Nodes:** Check that worker nodes are **ALIVE** and that **Available** resources (CPU, memory, GPU) are not zero. Dead or overloaded nodes can cause jobs to hang.
* **Unprovisioned GPUs:** Sometimes the CSP doesn't have enough GPUs available, so even though you have correctly requested them, the job may be unable to run. In this case, run on CPU if possible, or try your job again later.
**2. Memory pressure**
If nodes are full or workers are being OOM-killed, the job can stall or fail. In the Ray dashboard, check node memory usage; in Kubernetes, check pod restarts (`kubectl get pods -n NAMESPACE`) and pod logs for OOM. Also try reducing `concurrency` or the UDF’s memory request, even slightly.
**3. Writers / queues**
Geneva uses Ray actors for writers and queues. If the job is stuck at a fixed percentage, writers may be blocked (e.g. on storage or version conflicts). Check Ray dashboard **Actors** for writer-like actors that stay **RUNNING** but make no progress, and check cluster or pod logs for errors from Geneva (e.g. commit or connection errors).
You can relax or skip admission to narrow down whether the failure is admission vs. scheduling: set `GENEVA_ADMISSION__CHECK=false` or use `_admission_check=False` on the backfill (for testing only). See [Advanced configuration](/geneva/udfs/advanced-configuration/) for more details.
***
## Ray connection and startup
### "Geneva was unable to connect to the Ray head"
The client could not connect to the Ray cluster (e.g. after starting a KubeRay or external cluster).
**Common causes:**
* **Head not ready** – The Ray head pod may still be starting. Wait a bit and retry, or increase `GENEVA_RAY_INIT_MAX_RETRIES` (default 5).
* **Image/architecture mismatch** – The Ray head image may not match the node architecture (e.g. arm64 vs x64). Use an image built for the same architecture as your nodes. (the function [`get_ray_image`](https://lancedb.github.io/geneva/api/utils/) can help find the right image name.)
* **Network / firewall** – If using an external Ray cluster, ensure the `ray://` address is reachable and that no firewall is blocking the Ray client port.
**Quick check:** From the same network as the client, try connecting with `ray.init("ray://:")` in a small script to see the exact error.
### Ray client "already connected" or init fails on retry
If you see errors about the client already being connected (e.g. when re-running a notebook or script), ensure you're not holding an old Ray client connection. Restart the kernel or process so Ray is re-initialized cleanly. Geneva disconnects the client when the context exits; leaving a context open or reusing a stale connection can cause this.
### Head node out of memory
If the head node is under-provisioned (e.g. 1 CPU / 2 GB), it can OOM when:
* Many workers connect and register with GCS
* The Ray dashboard accumulates metrics
* Object store spillover occurs
**Recommendation**: Use at least 4 CPU / 8 GiB for the head node in any non-trivial deployment. This is the current default.
***
## Serialization library or `attrs` version mismatch
Ray and Geneva use cloudpickle and can be sensitive to library versions. If you see `TypeError: Enum.__new__() missing 1 required positional argument: 'value'` or similar pickling errors with no obvious non-serializable object, ensure **client and cluster use the same Python minor version and compatible library versions** (e.g. same `attrs`). Run the job from a machine with the same OS/architecture as the workers when possible so that shipped environments match.
***
## Permissions and storage
### GCS / S3 permission denied in job logs
Workers run under a Kubernetes service account (and possibly a cloud IAM role). If you see `PermissionError`, `storage.objects.get` denied, or 403 from object storage:
1. **Service account** – Confirm the Geneva Ray head and worker specs use the intended `service_account`. That account must have read/write to the bucket (e.g. GCS `roles/storage.objectUser` or equivalent S3 permissions).
2. **Workload identity** – On GKE, bind the K8s service account to a Google service account with bucket access. On EKS, use IRSA or node IAM so the pod role can access the bucket.
See [Troubleshooting Geneva Deployments](/geneva/deployment/troubleshooting) for permission examples and service account configuration.
***
## Version conflicts and commit retries
### Version conflicts during commit
Concurrent backfills or other writers can cause version conflicts when committing. Geneva retries with merging; if conflicts persist, you may see repeated retries or failures.
**Options:**
* Reduce concurrency or avoid overlapping backfills to the same table/fragments.
* Tune retries (e.g. `GENEVA_VERSION_CONFLICT_MAX_RETRIES`, default 10) if you expect transient contention. See [Advanced configuration](/geneva/udfs/advanced-configuration/).
### Writer stalls or commit timeouts
If writers are slow (e.g. under resource contention), they may be considered stalled and restarted. You can increase the idle tolerance with `GENEVA_WRITER_STALL_IDLE_ROUNDS` (default 6 rounds of 5s). For commit timeouts or transient storage errors, `GENEVA_COMMIT_MAX_RETRIES` (default 12) controls how many times Geneva retries the commit.
***
## Materialized views
### Matview refresh fails with resource errors
Refreshing a materialized view runs admission control for each UDF in the view. If any UDF’s resource requirements cannot be satisfied, the refresh fails. Ensure the cluster has enough resources for **all** UDFs used in the view (same as for a single backfill), and that no single UDF asks for more than any one node can provide.
***
## Quick checks before running a job
* **Versions** – Same Ray version on client and cluster; same Python minor (e.g. 3.10.x) on both. See [Troubleshooting Geneva Deployments](/geneva/deployment/troubleshooting#confirming-dependency-versions).
* **Remote execution** – Use `ray.available_resources()` and a simple `@ray.remote` task to confirm the cluster is reachable and has the expected CPUs/GPUs/memory.
* **Permissions** – Run a minimal remote task that `import geneva` and touches the same bucket/path as your job to surface permission issues early.
## API Reference
* [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator options: `num_gpus`, `num_cpus`, `memory`
* [Cluster](https://lancedb.github.io/geneva/api/cluster/) — `KubeRayClusterBuilder`, `CpuWorkerBuilder`, `GpuWorkerBuilder`
* [Table](https://lancedb.github.io/geneva/api/table/) — `backfill()`, `refresh()`
* [Error Handling](https://lancedb.github.io/geneva/api/error_handling/) — `FatalWorkerOOMError`, `FatalWorkerCrashError`, and other worker error types
# What is Feature Engineering?
Source: https://docs.lancedb.com/geneva/overview/index
Learn how to transform raw data into meaningful features for AI models using LanceDB's feature engineering capabilities. Scale your feature engineering workflows with distributed processing and UDFs.
This section introduces the concept of feature engineering using an example of a product
recommendation system.
For your AI models to work well — whether for search, recommendations, or anything else — you need good data. But raw data is usually messy and incomplete. Before you can train a model or use it for search, you have to turn that raw data into clear, meaningful features.
Feature engineering is the process of cleaning up your data and creating new signals that actually help your model learn or make better predictions. This step is just as important for preparing training data as it is for powering your AI in production.
It can take some work to get from raw data to useful features. Let's look at a simple example to see why this matters and how it's done.
## The Challenge: Manual Feature Engineering
Imagine you are building a product recommendation system for a large e-commerce platform. The goal is to find items that are genuinely "similar" to a product a user is currently viewing.
This notion of "similarity" must be sophisticated. It goes far beyond a simple text match on the product description. It needs to incorporate nuanced business concepts like popularity, value, brand equity, and key product attributes.
### Step 1: The Raw Data Table
We start with a raw data table in our data lakehouse. We'll call it `products_raw`. This table contains the basic, unprocessed information scraped from our product catalog. An embedding model could be applied directly to the `description` column for semantic search. However, that would only capture a fraction of the story.
**Table: `products_raw`**
| product\_id | title | description | category | price | original\_price | review\_count | avg\_rating |
| :---------- | :----------- | :------------------------------------ | :------- | :---- | :-------------- | :------------ | :---------- |
| 101 | V-Neck Tee | "A soft, 100% cotton v-neck shirt." | T-Shirt | 25.00 | 25.00 | 1200 | 4.8 |
| 102 | Designer Tee | "Limited edition organic cotton tee." | T-Shirt | 90.00 | 150.00 | 25 | 4.9 |
| 103 | New Tee | "A new v-neck t-shirt." | T-Shirt | 22.00 | 22.00 | 1 | 5.0 |
### Step 2: The Problem with Raw Data
Using this raw data to generate embeddings for a recommendation model is destined for failure. Critical business signals that define true product similarity are either misleading, hidden, or entirely absent.
* **Misleading Popularity:** The "New Tee" boasts a perfect 5.0 rating, but with only a single review. Is it truly more "popular" or "better" than the "V-Neck Tee" with 1200 reviews and a 4.8 rating? A naive model would think so. This leads to poor recommendations.
* **Missing Price Context:** The model sees a price of \$90.00 for the "Designer Tee." It has no intrinsic understanding that this represents a steep 40% discount. This is a powerful purchasing signal. The model also doesn't know how this price compares to the average price of other T-shirts.
* **Hidden Attributes:** Key attributes like "organic" or "limited edition" are buried within the free-text `description`. They are invisible to any model that doesn't perform sophisticated text analysis. Yet, they are crucial for matching user preferences.
### Step 3: Manual Feature Engineering
To address these challenges, a data scientist or ML engineer typically embarks on a manual, code-intensive journey. The goal is to extract hidden signals and craft new, meaningful features. This process often takes place in a Jupyter notebook. It involves several intricate steps.
1. **Engineer `popularity_score`:**\
To create a more accurate measure of popularity, combine the average rating with the volume of reviews. A common approach is to use logarithmic scaling. This prevents products with massive review counts from overwhelming the score.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
popularity_score = log(review_count + 1) * avg_rating
```
2. **Engineer `price_tier`:**\
To help the model understand value, bucket raw prices into clear tiers such as 'budget', 'mid-range', or 'premium'.
**Logic:**\
Use conditional logic (for example, `CASE WHEN` in SQL or `np.select` in Python) to assign a tier based on price thresholds.
3. **Engineer `discount_pct`:**\
To explicitly signal a "deal" to the model, calculate the discount percentage by combining the original and current prices.
**Logic:**
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
discount_pct = (original_price - price) / original_price
```
4. **Engineer `price_vs_cat_avg`:**\
To contextualize a product's price, compare it to the average price within its category. This requires an aggregation step to compute the average price per category. Then, calculate the feature for each product.
**Logic:**\
For each product:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
price_vs_cat_avg = price / avg_price_for_category
```
5. **Engineer `is_organic`:**\
To surface key product attributes, process the `description` text to identify important keywords.
**Logic:**\
Use a regular expression or string search for the keyword "organic" to create a boolean (`true`/`false`) flag.
Each of these steps requires careful data manipulation and domain knowledge. Iterative experimentation is also needed to ensure the resulting features are both accurate and useful for downstream models.
### Step 4: The Enriched Table
After executing this complex chain of logic, we produce a new, enriched table. The features in this table are more potent and ready to be fed into an embedding model. This produces high-quality vectors for our recommendation system.
**Table: `products_engineered`**
| product\_id | ... | popularity\_score | price\_tier | discount\_pct | price\_vs\_cat\_avg | is\_organic |
| :---------- | :-- | :---------------- | :---------- | :------------ | :------------------ | :---------- |
| 101 | ... | 33.6 | 'budget' | 0.0 | 0.54 | false |
| 102 | ... | 15.9 | 'premium' | 0.4 | 1.95 | true |
| 103 | ... | 3.5 | 'budget' | 0.0 | 0.47 | false |
## Why Use Feature Engineering?
Manual feature engineering works for small datasets. However, things change dramatically at scale. In real-world production systems, you often need to process tens or hundreds of millions of records. Sometimes this must happen in real time. The logic that was simple in a notebook, such as aggregations, conditional logic, or text processing, becomes much harder to manage and execute efficiently.
As data grows, so does complexity. You might need to generate features from new sources. These could include user behavior logs, images, or videos. This can involve running large-scale machine learning models for tasks like image captioning or text embedding. These tasks require significant compute resources and robust infrastructure.
Managing this at scale means dealing with distributed systems, scheduling, and monitoring. You must ensure that feature pipelines are reliable and reproducible. Experimenting with new features or updating existing ones can require major engineering work. This slows down iteration and innovation. Infrastructure challenges, such as orchestrating batch and streaming jobs, handling dependencies, and scaling inference, often become the main bottleneck.
In short, the biggest challenge in modern feature engineering is not just coming up with good features. It is also about building infrastructure that can handle complex, multimodal operations. The goal is to deliver fresh, high-quality features quickly and reliably at massive scale.
## API Reference
* [`geneva.connect()`](https://lancedb.github.io/geneva/api/) — connect to a Geneva database
* [Connection](https://lancedb.github.io/geneva/api/connection/) — manage tables, views, jobs, clusters, and manifests
* [Table](https://lancedb.github.io/geneva/api/table/) — add columns, backfill, search, and manage table data
* [UDF](https://lancedb.github.io/geneva/api/udf/) — define transforms as user-defined functions
# Geneva Python SDK
Source: https://docs.lancedb.com/geneva/reference
LanceDB Feature Engineering Python SDK Reference
Refer to the Geneva Python SDK reference documentation by clicking here.
# Advanced Configuration
Source: https://docs.lancedb.com/geneva/udfs/advanced-configuration
Learn about environment variables for configuring Geneva behavior.
Geneva supports various environment variables that start with `GENEVA_` to configure advanced behavior and fine-tune system settings.
All `GENEVA_` environment variables are optional and have sensible defaults. Only set them if you need to override the default behavior.
## Admission Control
Admission control validates cluster resources before starting jobs to prevent failures due to insufficient resources.
| Variable | Default | Description |
| --------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GENEVA_ADMISSION__CHECK` | `true` | Enable admission control checks. Set to `false` to skip all checks. |
| `GENEVA_ADMISSION__STRICT` | `true` | If `true`, reject the job with `ResourcesUnavailableError` when resources are insufficient. If `false`, log a warning but allow the job to proceed. |
| `GENEVA_ADMISSION__TIMEOUT` | `3.0` | Timeout in seconds for Ray API calls during admission control checks. Prevents hanging when the cluster is in a bad state. |
## Commit and Retry Configuration
Control retry behavior for commits and version conflicts.
| Variable | Default | Description |
| ------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GENEVA_COMMIT_MAX_RETRIES` | `12` | Maximum number of retries for commit operations. With exponential backoff (1s, 2s, 4s, 8s, 16s, then 16s capped), 12 retries gives \~2.5 minutes total wait time before giving up. |
| `GENEVA_VERSION_CONFLICT_MAX_RETRIES` | `10` | Maximum number of retries for version conflicts during commit. Version conflicts occur when concurrent backfills commit to the same fragments. Prevents infinite loops when concurrent commits keep conflicting. |
| `GENEVA_WRITER_STALL_IDLE_ROUNDS` | `6` | Number of idle rounds (5s each) before considering a writer stalled during drain. With many concurrent backfills, resource contention can slow writers without them being truly stalled. |
## Lance Retry Configuration
This section configures retry logic for Lance I/O operations. Retries occur on `OSError`, `ValueError`, and `RuntimeError("Too many concurrent writers")` exceptions, and are retried with exponential backoff with jitter.
| Variable | Default | Description |
| --------------------------------- | ------- | ---------------------------------------------------------------------------------------- |
| `GENEVA_RETRY_LANCE_ATTEMPTS` | `7` | Maximum number of retry attempts for Lance I/O operations. |
| `GENEVA_RETRY_LANCE_INITIAL_SECS` | `0.5` | Initial wait time in seconds for exponential backoff when retrying Lance I/O operations. |
| `GENEVA_RETRY_LANCE_MAX_SECS` | `120.0` | Maximum wait time in seconds for exponential backoff when retrying Lance I/O operations. |
## Checkpoint Storage
Checkpoint storage configuration is **experimental**. The environment variable names and behavior may change in a future release.
Configure where Geneva stores checkpoint data during job execution. Checkpoints enable fault-tolerant processing by saving intermediate results so that failed jobs can resume without reprocessing completed work.
By default, Geneva stores checkpoints in a `_ckp/` subdirectory inside the table's own storage location. This means checkpoints share the same bucket and IOPS budget as the table data. You can override this to store checkpoints in a separate location.
| Variable | Default | Description |
| ------------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `JOB__CHECKPOINT__OBJECT_STORE__PATH` | *(table dir)*`/_ckp/` | URI where checkpoint data is stored. When set, overrides the default in-table checkpoint location. Accepts any URI supported by Lance (e.g., `gs://bucket/path/checkpoints`, `s3://bucket/checkpoints`). |
This variable maps to the config path `job.checkpoint.object_store.path`. It can also be set via config files in `.config/` or `pyproject.toml` under the `[geneva]` section.
### Why use a separate checkpoint path?
At scale, checkpoint I/O and data I/O compete for the same object store IOPS budget when they share a bucket prefix. Setting `JOB__CHECKPOINT__OBJECT_STORE__PATH` to a **different bucket or prefix** decouples checkpoint I/O from data I/O, giving each its own IOPS budget and preventing shared-prefix rate limiting.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Example: separate checkpoint bucket from dataset storage
JOB__CHECKPOINT__OBJECT_STORE__PATH=gs://my-checkpoints-bucket/ckpts
```
Equivalent programmatic configuration:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.config import override_config_kv
override_config_kv({
"job.checkpoint.object_store.path": "gs://my-checkpoints-bucket/ckpts",
})
```
## Other Configuration
| Variable | Default | Description |
| ----------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GENEVA_RAY_INIT_MAX_RETRIES` | `5` | Maximum number of retry attempts for `ray.init()` connection failures. Useful when connecting to Ray clusters that may be temporarily unavailable. |
| `GENEVA_K8S_AUTH_MAX_RETRIES` | `3` | Maximum number of retries for Kubernetes authentication operations. Must be at least 1. |
| `GENEVA_CONFIG_DIR` | `./.config` | Directory path where Geneva looks for configuration files (`.yaml`, `.json`, `.toml`). Can be an absolute or relative path. |
# Batch User-Defined Table Functions (UDTFs)
Source: https://docs.lancedb.com/geneva/udfs/batch-udtfs
Use batch UDTFs for N:M transformations like deduplication, clustering, and aggregation across entire tables or partitions.
Beta — introduced in Geneva 0.11.0
Geneva's standard UDFs operate **row-at-a-time** — one input row produces exactly one output value. Batch User-Defined Table Functions (UDTFs) lift this restriction, enabling **N:M transformations** where the output can have a completely different schema and row count than the input.
| Workflow | Input | Output | Cardinality |
| -------------------- | ------ | -------------- | ----------- |
| Deduplication | N rows | M rows (M ≤ N) | N:M |
| Clustering | N rows | K cluster rows | N:K |
| Aggregation | N rows | 1 summary row | N:1 |
| Cross-row join/merge | N rows | M rows | N:M |
## Defining a Batch UDTF
Use the `@udtf` decorator on a class or function. The UDTF receives a query builder over the source data and **yields** `pa.RecordBatch` objects with an arbitrary output schema.
### Class-based
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import udtf
import pyarrow as pa
from collections.abc import Iterator
@udtf(
output_schema=pa.schema([
pa.field("row_id", pa.int64()),
pa.field("cluster_id", pa.int64()),
pa.field("is_duplicate", pa.bool_()),
]),
input_columns=["row_id", "phash"],
num_cpus=4,
memory=8 * 1024**3, # 8 GiB
)
class PHashDedupe:
def __init__(self, threshold: int = 4):
self.threshold = threshold
def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
data = source.to_arrow().select(["row_id", "phash"])
clusters = self._cluster(data)
yield pa.RecordBatch.from_pydict({
"row_id": clusters["row_id"],
"cluster_id": clusters["cluster_id"],
"is_duplicate": clusters["is_duplicate"],
})
```
### Function-based
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udtf(output_schema=pa.schema([
pa.field("label", pa.string()),
pa.field("count", pa.int64()),
pa.field("mean_score", pa.float64()),
]))
def group_stats(source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
df = source.to_pandas()
agg = df.groupby("label").agg(
count=("label", "size"),
mean_score=("score", "mean"),
).reset_index()
yield pa.RecordBatch.from_pandas(agg)
```
### Decorator Parameters
| Parameter | Type | Description |
| ----------------------------- | ------------------- | --------------------------------------------------------------------------------------------------- |
| `output_schema` | `pa.Schema` | **Required.** Arrow schema of the output table. |
| `input_columns` | `list[str] \| None` | Restrict which source columns are visible. `None` means all. |
| `partition_by` | `str \| None` | Column name for partition-parallel execution. |
| `partition_by_indexed_column` | `str \| None` | Column name with an IVF index for index-based partitioning. Mutually exclusive with `partition_by`. |
| `num_cpus` | `float` | Ray CPU resource request per worker. |
| `num_gpus` | `float` | Ray GPU resource request per worker. |
| `memory` | `int \| None` | Ray memory resource request in bytes. |
| `on_error` | | Error handling configuration (see [Error Handling](#error-handling)). |
## Creating and Refreshing a UDTF View
Batch UDTFs are always attached to a persistent view via `create_udtf_view()`. Call `refresh()` to populate or update the view.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
conn = geneva.connect("/data/mydb")
images = conn.open_table("images")
# Create the UDTF view
deduped = conn.create_udtf_view(
"deduped_images",
source=images.search(None).select(["row_id", "phash"]),
udtf=PHashDedupe(threshold=4),
)
# Populate the view
deduped.refresh()
```
UDTF views use the same cluster infrastructure as other Geneva jobs:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# KubeRay cluster
with conn.context(cluster="my-cluster", manifest="my-manifest"):
deduped.refresh()
# Local Ray
with conn.local_ray_context():
deduped.refresh()
```
### Version-aware refresh
On each refresh, Geneva checks the source table's version against the version stored in the view metadata. If the source has not changed, the refresh is skipped entirely — an O(1) check.
## Execution Modes
### Single-worker (no partitioning)
When neither `partition_by` nor `partition_by_indexed_column` is set, the UDTF runs as a **single Ray task** with access to the entire source dataset. Use this for global operations that need cross-row visibility.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udtf(output_schema=...)
class GlobalAggregation:
def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
all_data = source.to_arrow()
result = expensive_cross_row_computation(all_data)
yield result.to_batches()[0]
```
### Partition-parallel (`partition_by`)
The framework groups source data by the partition column and dispatches each partition as an independent Ray task. Use this when the computation is naturally parallelizable by some grouping key.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udtf(
output_schema=pa.schema([
("row_id_a", pa.string()),
("row_id_b", pa.string()),
("hamming_dist", pa.int32()),
]),
partition_by="partition_id",
num_cpus=2,
)
class EdgeDetection:
def __init__(self, threshold: int = 4):
self.threshold = threshold
def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
data = source.to_arrow()
edges = self._pairwise_compare(data, self.threshold)
if edges:
yield pa.RecordBatch.from_pydict(edges)
```
### Index-based partitioning (`partition_by_indexed_column`)
Instead of partitioning by a materialized column, the framework reads partition assignments directly from an existing **IVF vector index** (IVF\_FLAT, IVF\_PQ, IVF\_HNSW\_FLAT, IVF\_HNSW\_SQ, etc.). This avoids materializing a `partition_id` column and keeps partitions synchronized with the index.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.partitioning import create_ivf_flat_index
# 1. Build an IVF index on the source table
create_ivf_flat_index(images, "phash", k=16)
# 2. Define the UDTF with index-based partitioning
@udtf(
output_schema=pa.schema([
("row_id_a", pa.string()),
("row_id_b", pa.string()),
("hamming_dist", pa.int32()),
]),
partition_by_indexed_column="phash",
num_cpus=2,
)
class IndexPartitionedEdgeDetection:
def __init__(self, threshold: int = 4):
self.threshold = threshold
def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
data = source.to_arrow()
edges = self._pairwise_compare(data, self.threshold)
if edges:
yield pa.RecordBatch.from_pydict(edges)
```
| | `partition_by` | `partition_by_indexed_column` |
| ---------------------------- | -------------------------- | ------------------------------------- |
| Partition source | Column values (SQL filter) | IVF index partitions (row ID take) |
| Requires materialized column | Yes | No — reads from index metadata |
| Partition count | Number of distinct values | Number of non-empty index partitions |
| Sync with index | Manual | Automatic — always reads latest index |
`partition_by` and `partition_by_indexed_column` are **mutually exclusive**. Setting both raises `ValueError`.
## Yielding Batches
The UDTF yields one or more `pa.RecordBatch` or `pa.Table` objects. Each batch must conform to `output_schema`. The framework validates each batch, writes it, and optionally checkpoints it.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Streaming — yield per source batch (memory-efficient)
def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
for batch in source.to_batches(batch_size=1024):
yield transform(batch)
# Bulk — load all, compute, yield once
def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
all_data = source.to_arrow()
result = expensive_cross_row_computation(all_data)
yield result.to_batches()[0]
```
Use the streaming pattern for memory-efficient processing. Use the bulk pattern when the computation inherently requires all data in memory (e.g., clustering, global deduplication).
## Error Handling
Error handling operates at **partition granularity** — the unit of work is the entire `__call__()` execution for a partition (or the full table in single-worker mode).
| Mode | Behavior | Use case |
| ------------------ | ---------------------------------------------------- | --------------------------------- |
| **Fail** (default) | Exception kills the partition, refresh fails | Correctness-critical UDTFs |
| **Retry** | Retry the entire partition with configurable backoff | Transient failures (network, OOM) |
| **Skip** | Log error, continue with remaining partitions | Best-effort / tolerant workloads |
Unlike standard UDF error handling, there is no row-level skip — UDTFs yield whole batches, so the smallest error unit is the partition.
## Checkpointing
Each yielded batch is checkpointed before reporting completion. On resume after a failure, completed batches are skipped and entire partitions with a `__done__` marker are skipped.
Checkpoint keys include the source table version, so stale checkpoints from a previous source version are automatically ignored when the source changes.
Checkpointed UDTFs must be **deterministic** — the same input must yield the same batch sequence for resume to work correctly.
## Examples
### K-Means Clustering
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udtf(
output_schema=pa.schema([
pa.field("row_id", pa.int64()),
pa.field("cluster_id", pa.int64()),
pa.field("distance_to_centroid", pa.float64()),
]),
num_cpus=4,
memory=16 * 1024**3,
)
class KMeansClustering:
def __init__(self, k: int = 100):
self.k = k
def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
import numpy as np
embeddings = source.to_arrow().select(["row_id", "embedding"])
row_ids = embeddings.column("row_id").to_pylist()
vectors = np.stack(embeddings.column("embedding").to_pylist())
centroids, assignments, distances = self._fit(vectors)
chunk_size = 10_000
for start in range(0, len(row_ids), chunk_size):
end = min(start + chunk_size, len(row_ids))
yield pa.RecordBatch.from_pydict({
"row_id": row_ids[start:end],
"cluster_id": assignments[start:end].tolist(),
"distance_to_centroid": distances[start:end].tolist(),
})
```
### Aggregation
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udtf(
output_schema=pa.schema([
pa.field("label", pa.string()),
pa.field("count", pa.int64()),
pa.field("mean_score", pa.float64()),
]),
)
class GroupStats:
def __init__(self, group_by: str = "label"):
self.group_by = group_by
def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
df = source.to_pandas()
agg = df.groupby(self.group_by).agg(
count=("label", "size"),
mean_score=("score", "mean"),
).reset_index()
yield pa.RecordBatch.from_pandas(agg)
```
Reference:
* [`create_udtf_view` API](https://lancedb.github.io/geneva/api/connection/#geneva.db.Connection.create_udtf_view)
* [UDTF](https://lancedb.github.io/geneva/api/udtf/) — full `@udtf` / `@batch_udtf` decorator reference including `output_schema`, `partition_by`, `num_gpus`, and `on_error`
# Blob Types in Geneva UDFs
Source: https://docs.lancedb.com/geneva/udfs/blobs
Learn how to work with Lance Blobs in Geneva UDFs for handling large binary objects efficiently with lazy reading capabilities.
Geneva supports UDFs that take [Lance Blobs](https://docs.lancedb.com/tables/multimodal) (large binary objects) as input and has the ability to write out columns with binaries encoded as Lance Blobs. Lance blobs are an optimization intended for large objects (1's MBs -> 100MB's) and provide a file-like object that lazily reads large binary objects.
## Reading Blobs
Defining functions that read blob columns is straight forward.
For scalar UDFs, blob columns are expected to be of type `BlobFile`
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lance.blob import BlobFile
@udf
def work_on_udf(blob: BlobFile) -> int:
assert isinstance(blob, BlobFile)
data = blob.read()
# do something intresting.
return len(data)
```
## Writing Blobs
Defining UDFs that write out `Blob`s to a new column is straightforward. Here we add the standard metadata annotation to the UDF so that Geneva knows to write out Blobs.
For scalar udfs, your udf will return `bytes`, explicitly set the `data_type` to `pa.large_binary()`, and add the `field_metadata` that specifies blob encoding.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.large_binary(), field_metadata={"lance-encoding:blob": "true"})
def generate_blob(text: str, multiplier: int) -> bytes:
"""UDF that generates blob data by repeating text."""
return (text * multiplier).encode("utf-8")
```
For `pa.RecordBatch` batched UDFs you the effort is similar:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.large_binary(), field_metadata={"lance-encoding:blob": "true"})
def batch_to_blob(batch: pa.RecordBatch) -> pa.Array:
"""UDF that converts RecordBatch rows to blob data."""
import json
blobs = []
for i in range(batch.num_rows):
# do something that returns bytes
blob_data = ...
blobs.append(blob_data)
return pa.array(blobs, type=pa.large_binary())
```
## API Reference
* [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator for defining blob-processing functions
* [Table](https://lancedb.github.io/geneva/api/table/) — `add_columns()`, `backfill()`
# Error Handling in Geneva UDFs
Source: https://docs.lancedb.com/geneva/udfs/error_handling
Learn how configure retry, skip, and fail behaviors for UDFs.
Geneva provides three ways to handle errors, in increasing complexity: factory functions, exception matchers, and full Tenacity control.
## Quick Start: Factory Functions
Use factory functions for common error handling patterns:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import udf, retry_transient
import pyarrow as pa
@udf(data_type=pa.int32(), on_error=retry_transient())
def my_udf(x: int) -> int:
# Will retry on network errors (ConnectionError, TimeoutError, OSError)
return call_external_api(x)
```
Geneva provides four built-in factory functions:
| Function | Behavior |
| ------------------- | --------------------------------------------------------------------------- |
| `retry_transient()` | Retry `ConnectionError`, `TimeoutError`, `OSError` with exponential backoff |
| `retry_all()` | Retry any exception with exponential backoff |
| `skip_on_error()` | Return `None` for any exception (skip the row) |
| `fail_fast()` | Fail immediately on any exception (default behavior) |
### Customizing Retry Behavior
Factory functions accept parameters to customize behavior:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import udf, retry_transient, retry_all
# Increase max attempts
@udf(data_type=pa.int32(), on_error=retry_transient(max_attempts=5))
def more_retries(x: int) -> int:
...
# Change backoff strategy
@udf(data_type=pa.int32(), on_error=retry_all(max_attempts=3, backoff="fixed"))
def fixed_backoff(x: int) -> int:
...
```
**Parameters:**
* `max_attempts` (int): Maximum number of attempts (default: 3)
* `backoff` (str): Backoff strategy between retries
* `"exponential"` (default): 1s, 2s, 4s, 8s... with jitter, capped at 60s
* `"fixed"`: Fixed 1s delay between attempts
* `"linear"`: 1s, 2s, 3s, 4s... capped at 60s
## Custom Exception Handling: Matchers
For fine-grained control, use `Retry`, `Skip`, and `Fail` matchers:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import udf, Retry, Skip, Fail
@udf(
data_type=pa.int32(),
on_error=[
Retry(ConnectionError, TimeoutError, max_attempts=3),
Retry(ValueError, match="rate limit", max_attempts=5),
Skip(ValueError), # Other ValueErrors - skip the row
Fail(AuthError), # Auth failures - fail immediately
]
)
def custom_handling(x: int) -> int:
...
```
**How matching works:**
1. Matchers are evaluated in order (first match wins)
2. More specific matchers should come before general ones
3. Unmatched exceptions fail the job
### Exception Matchers
| Matcher | Behavior | Parameters |
| ------------ | ----------------------------- | ---------------------------------- |
| `Retry(...)` | Retry with backoff, then fail | `max_attempts`, `backoff`, `match` |
| `Skip(...)` | Return `None` for that row | `match` |
| `Fail(...)` | Fail the job immediately | `match` |
**Syntax:**
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Single exception
Retry(ConnectionError)
# Multiple exceptions
Retry(ConnectionError, TimeoutError, OSError)
# With parameters
Retry(ConnectionError, max_attempts=5, backoff="fixed")
# With message matching
Retry(ValueError, match="rate limit")
```
### Message Matching
Use the `match` parameter to filter exceptions by their message content. The pattern is a regex:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import Retry, Skip
# Simple substring (works because regex matches substrings)
Retry(ValueError, match="rate limit")
# Matches: ValueError("rate limit exceeded")
# Regex pattern
Retry(ValueError, match=r"rate.?limit")
# Matches: ValueError("rate limit")
# Matches: ValueError("ratelimit")
# Matches: ValueError("rate_limit")
# Case-insensitive matching (use (?i) flag)
Retry(ValueError, match=r"(?i)rate limit")
# Matches: ValueError("Rate Limit exceeded")
# Matches: ValueError("RATE LIMIT hit")
# Regex alternation (match multiple patterns)
Retry(ValueError, match=r"429|rate.?limit|throttl")
# Matches: ValueError("Error 429")
# Matches: ValueError("rate limit exceeded")
# Matches: ValueError("Request throttled")
```
For example, using matchers to distinguish error types:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(
data_type=pa.string(),
on_error=[
# Retry rate limits with more attempts
Retry(ValueError, match="rate limit", max_attempts=10),
# Skip invalid input
Skip(ValueError, match="invalid"),
# Fail on other ValueErrors
Fail(ValueError),
]
)
def api_call(x: str) -> str:
...
```
### Behavior Summary
| Outcome | What Happens | When to Use |
| --------- | ---------------------------------- | ---------------------------------------------------------- |
| **Retry** | Retry with backoff, then fail/skip | Transient errors: network issues, rate limits, timeouts |
| **Skip** | Return `None` for that row | Bad input data, row-specific failures, optional enrichment |
| **Fail** | Kill the job immediately | Fatal errors: auth failures, configuration errors |
## Advanced: Full Tenacity Control
For power users who need custom callbacks or complex retry conditions, omit `on_error` and use `error_handling=`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import udf
from geneva.debug.error_store import ErrorHandlingConfig, UDFRetryConfig
from tenacity import wait_random_exponential, stop_after_delay
@udf(
data_type=pa.int32(),
error_handling=ErrorHandlingConfig(
retry_config=UDFRetryConfig(
retry=my_custom_retry_condition,
stop=stop_after_delay(300),
wait=wait_random_exponential(min=1, max=120),
before_sleep=my_logging_callback,
),
),
)
def power_user_udf(x: int) -> int:
...
```
Note: `on_error=` and `error_handling=` cannot be used together.
## Restrictions
* **Skip behavior** only works with scalar UDFs (functions that process one row at a time)
* For batch UDFs that receive `RecordBatch`, use `Retry` or `Fail` only
* **All Retry matchers must use the same backoff strategy.** You cannot mix different backoff strategies in the same `on_error` list:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(on_error=[
Retry(ConnectionError, backoff="exponential"),
Retry(TimeoutError, backoff="fixed"), # Error: different backoff!
])
@udf(on_error=[
Retry(ConnectionError, backoff="fixed"),
Retry(TimeoutError, backoff="fixed"), # Same backoff - OK
])
```
* **Invalid regex patterns are rejected at construction time:**
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# This will raise ValueError due to the unclosed bracket
Retry(ValueError, match=r"[invalid")
# But this will work:
Retry(ValueError, match=r"rate.?limit")
```
## API Reference
* [Error Handling](https://lancedb.github.io/geneva/api/error_handling/) — `Retry`, `Skip`, `Fail`, `FatalWorkerOOMError`, `FatalWorkerCrashError`, and all exception matcher classes
* [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator `on_error` parameter
# Understanding Transforms
Source: https://docs.lancedb.com/geneva/udfs/index
Understand the three types of user-defined functions in Geneva — UDFs, chunkers (scalar UDTFs), and batch UDTFs — and when to use each.
Geneva provides three types of user-defined functions for transforming data. Each type has a different input/output cardinality and is suited to different workflows.
## Choosing the Right Type
* **Adding a column to each row?** Use a [**UDF**](/geneva/udfs/udfs).
* **Splitting each row into multiple rows?** Use a [**Chunker**](/geneva/udfs/scalar-udtfs).
* **Computing across rows with a different output shape?** Use a [**Batch UDTF**](/geneva/udfs/batch-udtfs).
## At a Glance
| | UDF | Chunker (Scalar UDTF) | Batch UDTF |
| --------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Cardinality** | 1:1 | 1:N | N:M |
| **Decorator** | `@udf` | `@chunker` | `@udtf` |
| **Refresh** | Incremental | Incremental | Full |
| **Parallelism** | Fragment-parallel | Fragment-parallel | Partition-parallel |
| **Inherited columns** | N/A — adds to existing rows | Automatic from query | Independent output schema |
| **Registration** | [`table.add_columns()`](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.add_columns) | [`db.create_udtf_view()`](https://lancedb.github.io/geneva/api/connection/#geneva.db.Connection.create_udtf_view) | [`db.create_udtf_view()`](https://lancedb.github.io/geneva/api/connection/#geneva.db.Connection.create_udtf_view) |
## UDFs (1:1)
Standard UDFs produce exactly **one output value per input row**. Use them to add computed columns to existing tables or materialized views.
| id | text | **embedding** |
| -- | ------------- | ------------------------ |
| 1 | "hello world" | **→ \[0.12, 0.34, ...]** |
| 2 | "foo bar" | **→ \[0.56, 0.78, ...]** |
| 3 | "baz qux" | **→ \[0.90, 0.11, ...]** |
Each input row produces exactly one output value. The new column is added to the same table.
**Use cases**: Embeddings, data enrichment, format conversion, scoring.
See [UDFs](/geneva/udfs/udfs) for the full guide.
## Chunkers (Scalar UDTFs, 1:N)
Chunkers — also called scalar UDTFs — **expand each source row into multiple output rows**. The output is a materialized view that inherits parent columns and supports incremental refresh.
**Source: `documents`**
| doc\_id | title | text |
| ------- | ------------- | ------------------------ |
| 1 | "Intro to AI" | "Machine learning is..." |
| 2 | "Data Guide" | "Data pipelines are..." |
**Derived: `chunks`** (1:N expansion via `@chunker`)
| doc\_id | title | chunk\_index | chunk\_text |
| ------- | ------------- | ------------ | --------------------- |
| 1 | "Intro to AI" | 0 | "Machine learning..." |
| 1 | "Intro to AI" | 1 | "Neural networks..." |
| 1 | "Intro to AI" | 2 | "Training data..." |
| | | | |
| 2 | "Data Guide" | 0 | "Data pipelines..." |
| 2 | "Data Guide" | 1 | "ETL processes..." |
Each source row produces **one or more** output rows. Parent columns (`doc_id`, `title`) are inherited automatically.
**Use cases**: Document chunking, video segmentation, image tiling.
See [Chunkers](/geneva/udfs/scalar-udtfs) for the full guide.
## Batch UDTFs (N:M)
Batch UDTFs read from a source table (or partition) and **produce output with an arbitrary schema and row count**. They always perform a full refresh.
**Source: `sales`**
| product | region | amount |
| ------- | ------ | ------ |
| Widget | East | 100 |
| Widget | East | 250 |
| Widget | West | 175 |
| Gadget | East | 300 |
| Gadget | West | 400 |
| Gadget | West | 150 |
**Derived: `sales_summary`** (N:M aggregation via `@udtf`)
| product | total\_amount | avg\_amount | num\_sales |
| ------- | ------------- | ----------- | ---------- |
| Widget | 525 | 175.0 | 3 |
| Gadget | 850 | 283.3 | 3 |
6 input rows become 2 output rows with a completely different schema. The output shape is determined entirely by the UDTF logic — it could be fewer rows (aggregation), more rows (clustering), or the same count with different columns.
**Use cases**: Deduplication, clustering, aggregation, cross-row joins.
See [Batch UDTFs](/geneva/udfs/batch-udtfs) for the full guide.
## API Reference
* [UDF](https://lancedb.github.io/geneva/api/udf/) — `@udf` decorator and `UDF` class
* [UDTF](https://lancedb.github.io/geneva/api/udtf/) — `@udtf`, `@chunker`, `@batch_udtf` decorators and `UDTF`/`Chunker` classes
* [Table](https://lancedb.github.io/geneva/api/table/) — `add_columns()`, `backfill()`
* [Connection](https://lancedb.github.io/geneva/api/connection/) — `create_udtf_view()`, `create_materialized_view()`
# Profiling Stateful UDF Memory
Source: https://docs.lancedb.com/geneva/udfs/profiling-memory
Find memory leaks and runaway peak usage in stateful UDFs with memray, before they cause worker OOMs in production.
Stateful UDFs are the most common source of worker memory pressure in Geneva. Unlike scalar UDFs, a stateful UDF instance lives for the **entire lifetime of a Ray actor**, processing many batches in sequence. Anything your `setup()` allocates is held for the duration of the job, and anything `__call__` retains accumulates batch after batch — sometimes silently, until a worker OOMs partway through a large backfill.
This page shows you how to profile a stateful UDF locally with [memray](https://github.com/bloomberg/memray), what to look for, and the common patterns that leak.
If your worker is being OOM-killed and you don't know why, **profile a single actor locally first**. A 5-minute memray run on your laptop is faster than another 45-minute distributed run that fails the same way.
## Why stateful UDFs leak
A stateful UDF in Geneva is a class:
A few facts make memory behavior easy to get wrong:
* **One instance per worker, many batches.** Geneva instantiates the class once per Ray actor. The same `self` processes every batch routed to that worker — potentially thousands.
* **`setup()` runs once.** Whatever you allocate there stays in memory until the actor dies. That's intentional for things like ML models, but it's a footgun for "lazy" caches that grow.
* **`self.` survives across calls.** Anything you attach to `self` inside `__call__` is retained for the rest of the actor's life.
* **Workers don't restart between batches.** Unlike a serverless function, you don't get a fresh process per invocation. Memory accumulates linearly with batch count until the worker hits its memory cap.
The result: a leak that looks tiny in unit tests (1 batch, 4 MiB) can blow up an 8-hour backfill (10 000 batches, 40 GiB).
## When to profile
Profile your UDF if **any** of these are true:
* The UDF loads a model, builds an index, or otherwise allocates more than \~100 MiB in `setup()`.
* The UDF maintains a cache, deduplication table, or running statistic in `self`.
* A worker is being OOM-killed during backfill (look for `FatalWorkerOOMError`, see [Job troubleshooting](/geneva/jobs/troubleshooting)).
* Worker RSS grows steadily during a backfill rather than staying flat after `setup()`.
You do **not** need to profile pure functional UDFs (no `self` state) or UDFs that only ever read from `self` — those can't leak by construction.
## Profiling a UDF with memray
memray ships as a `dev` dependency in Geneva, so it's already in your environment if you installed with `uv sync`.
The trick to profiling under Ray is that **workers run in separate processes**. Wrapping `pytest` with `memray run` only sees the driver, not the actors that actually run your UDF. The cleanest pattern is to have **the UDF instrument itself**, controlled by an environment variable that's only set when you want a profile.
### Step 1 — add an opt-in tracker to your UDF
The tracker is a no-op when `MY_UDF_MEMRAY_OUT_DIR` isn't set, so leaving this code in your UDF is safe for production runs.
### Step 2 — propagate the env var to Ray workers
Ray workers don't inherit driver environment variables by default. When you start a local Ray cluster from Geneva, pass `extra_env` so the variable reaches each worker:
Set `concurrency=1` while profiling. One actor processing all batches sequentially produces a single clean trace; the default of 8 produces 8 noisier traces that you'd have to merge mentally.
### Step 3 — read the trace
When backfill finishes, you'll have one (or more) `memray--.bin` files under `/tmp/my-udf-profile/`. Render and inspect them:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Quick summary — peak heap, total allocations, what's leaked
uv run -m memray summary /tmp/my-udf-profile/memray-*.bin
# Interactive flamegraph in your browser
uv run -m memray flamegraph /tmp/my-udf-profile/memray-*.bin
open /tmp/my-udf-profile/memray-*.html
# Top allocators by retained bytes
uv run -m memray tree /tmp/my-udf-profile/memray-*.bin
```
## What the numbers mean
memray reports two values you'll care about most:
* **Peak heap** (`metadata.peak_memory`) — the high-water mark. This is what triggers OOMs. A peak well above your `setup()` allocations means a batch transiently doubles memory before freeing.
* **Leaked allocations** (`get_leaked_allocation_records()`) — what was still allocated when the tracker ended. **This is not necessarily a bug** — your `setup()` model is "leaked" in this sense because it lives the actor's lifetime. The signal is *how much above expected baseline* is leaked.
A healthy stateful UDF profile, after processing many batches, looks roughly like:
```
peak heap ≈ setup() allocations + 1 batch of working memory
leaked ≈ setup() allocations (i.e. nothing extra retained from __call__)
```
An **unhealthy** profile looks like:
```
peak heap ≈ setup() + N × per-call allocation ← grows with batch count
leaked ≈ setup() + N × per-call allocation ← per-call state never freed
```
The flamegraph will show a thick stack frame anchored in `__call__` rising as you scroll through time — that's the leak.
## Going deeper: RSS vs Arrow allocations
memray gives you the *Python-side* allocation story. For real diagnosis of "where is the worker's memory actually going?", it pays to watch **process RSS** and **Arrow's own allocator** side-by-side — together they tell you which subsystem owns the bytes, often faster than reading a flamegraph.
Drop this into your UDF (or anywhere on the worker) to log a snapshot:
The three numbers answer different questions:
* **`rss_mb`** — every byte the OS has handed this Python interpreter. Includes Python heap, Arrow, native libraries, and pages the C allocator (`glibc`/`jemalloc`) is holding even though Python freed them. This is what triggers cgroup OOM-kills.
* **`arrow_live_mb`** — bytes currently held by *live PyArrow buffers* (`RecordBatch`, `Array`, `ChunkedArray`, etc.). Goes up when you create Arrow data, down when those references are dropped.
* **`gap_mb` = rss − arrow\_live** — "everything else." This is the Python heap (your own `self.cache`, model weights, dicts, lists), native libraries (PyTorch, ONNX), and allocator retention.
### Diagnostic patterns
Log the breakdown every few batches and the *shape of growth over time* tells you which subsystem to fix:
| Pattern | Diagnosis | First thing to try |
| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `rss` climbs slowly, `arrow_live` flat near zero, big growing `gap` | Allocator retention — Python freed it but `glibc` is keeping the pages | `ctypes.CDLL("libc.so.6").malloc_trim(0)` periodically (Linux only); or set `MALLOC_TRIM_THRESHOLD_=131072` |
| `rss` climbs, `arrow_live` climbs in lockstep | Real Arrow leak — your code is holding `RecordBatch` / `Array` references | Find where you're appending batches to `self`, or where checkpoint / error payloads aren't being released |
| `rss` spikes hugely on a few calls then settles, eventually one spike OOMs | Peak is too big, not a leak — a single call allocates more than the worker has | Shrink `batch_size`, `blob_read_buffer_size`, or split the work |
| `rss` flat for hours then sudden cliff upward | One pathological row — usually one huge blob (a 4K-resolution image, a 50 MB PDF) | Find the offending row by ID; add a size check at the top of `__call__` |
| `rss` rises during `setup()`, then flat for the whole run, `gap` constant | Healthy — that's your model loaded once per actor | Nothing to do |
The reference UDFs in Geneva's own integration test (`src/stress_tests/_memray_probe.py`) print exactly this breakdown every 32 calls. The workflow's stdout logs are a working example of the "clean" and "leaky" patterns — the leaky one shows the **lockstep with Python heap** signature (the second row above, but with `gap` climbing instead of `arrow_live` — because the leak is `bytearray`, not Arrow).
## Common leak patterns
### 1. The growing cache
Looks harmless. Fine on a unit test with 10 inputs. **Catastrophic on a backfill of 10M rows**, where most inputs are unique and the cache grows to fill the worker.
**Fix:** Use a bounded cache (`functools.lru_cache` with `maxsize`, or a manual size cap), or skip caching when you don't know the cardinality.
### 2. Accumulating per-call buffers
**Fix:** Don't hold references to inputs past the return of `__call__`. If you need rolling state, summarize into a small aggregate (counts, sums) instead of holding the raw batches.
### 3. Closures capturing batch arrays
**Fix:** Extract only the small values you actually need into the closure, or execute the work eagerly.
### 4. ML model state that grows
Some ML libraries retain per-call state internally (KV caches, gradient buffers, autograd graphs). If you're using PyTorch:
For Hugging Face pipelines, ensure you're in `eval()` mode and not accumulating gradients. For long-running stateful UDFs on GPUs, also see `torch.cuda.empty_cache()` between large batches.
## A confidence check
A useful "does my profiling actually work?" sanity check: temporarily introduce a deliberate leak and confirm memray catches it.
If `memray summary` doesn't show leaked bytes growing roughly with batch count after this change, your tracker isn't actually attached (most often: the env var isn't reaching workers — re-check `extra_env`).
Geneva's own test suite ships a reference implementation of this pattern in `src/stress_tests/_memray_probe.py` and `src/stress_tests/test_memray_stateful_udf.py`, plus a GitHub Actions workflow (`memray-stateful-udf-profile.yml`) that uploads the per-actor `.bin` and rendered flamegraph as a CI artifact. Feel free to copy that scaffolding for your own project's UDFs.
## Related
* [UDFs](/geneva/udfs/udfs) — defining stateful UDFs
* [Job troubleshooting](/geneva/jobs/troubleshooting) — diagnosing OOMs and other worker errors
* [Advanced configuration](/geneva/udfs/advanced-configuration) — admission control and resource limits
* [memray documentation](https://bloomberg.github.io/memray/) — flamegraph, summary, and tree report formats
# Gemini
Source: https://docs.lancedb.com/geneva/udfs/providers/gemini
Embed text and generate completions using Google's Gemini models.
See the API reference for [Gemini UDFs](https://lancedb.github.io/geneva/api/gemini/) and
[Embedding UDFs](https://lancedb.github.io/geneva/api/embeddings/) for all parameters.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install 'geneva[udf-text-gemini]'
```
Gemini UDFs make API calls that incur **per-token costs**. Each row processed results in one
or more API requests billed to your account. Review
[Gemini pricing](https://ai.google.dev/gemini-api/docs/pricing) before running on large tables.
Set the `GEMINI_API_KEY` environment variable before calling any factory function below.
The key is read **at UDF creation time** and serialized with the UDF — no cluster-level
`env_vars` configuration is needed.
## Embeddings
Embed text with optional task-type hints for retrieval, classification, and clustering scenarios.
See the [API reference](https://lancedb.github.io/geneva/api/embeddings/#geneva.udfs.text.embeddings.gemini_embedding_udf) for all parameters.
**Multiple embeddings tuned for different retrieval tasks:**
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.udfs import gemini_embedding_udf
table.add_columns({
# Full-dimension embedding for document retrieval
"embedding_doc": gemini_embedding_udf(
column="body",
model="gemini-embedding-001",
task_type="RETRIEVAL_DOCUMENT",
),
# Compact embedding for semantic similarity
"embedding_sim_256": gemini_embedding_udf(
column="body",
model="gemini-embedding-001",
task_type="SEMANTIC_SIMILARITY",
output_dimensionality=256,
),
})
```
## Generation
Generate text from Gemini models. Supports text, image, audio, video, and document inputs.
See the [API reference](https://lancedb.github.io/geneva/api/gemini/#geneva.udfs.text.gemini.gemini_udf) for all parameters.
**Enrich a table with sentiment, captions, and transcriptions at once:**
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.udfs import gemini_udf
table.add_columns({
# Classify review sentiment with a fast model
"sentiment": gemini_udf(
column="review",
prompt="Classify the sentiment as positive, negative, or neutral. Return only the label.",
model="gemini-2.5-flash",
),
# Caption product images with a more capable model
"caption": gemini_udf(
column="image",
prompt="Describe the main subject of this image in one sentence",
model="gemini-2.5-pro",
mime_type="image/jpeg",
),
# Transcribe audio clips
"transcript": gemini_udf(
column="audio",
prompt="Transcribe this audio clip",
model="gemini-2.5-flash",
mime_type="audio/mp3",
),
})
```
## API Reference
* [Gemini](https://lancedb.github.io/geneva/api/gemini/) — `gemini_udf()` parameters: `column`, `prompt`, `model`, `mime_type`, and more
* [Embeddings](https://lancedb.github.io/geneva/api/embeddings/) — `gemini_embedding_udf()` parameters: `column`, `model`, `task_type`, `dimensionality`, `normalize`
# Built-in LLM and Embedding UDFs
Source: https://docs.lancedb.com/geneva/udfs/providers/index
Geneva ships pre-built UDFs for common LLM providers so you don't have to write custom classes
for everyday embedding and generation tasks.
| Provider | Embeddings | Generation | Runs locally | Install extra |
| --------------------------------------------------------------------- | :--------: | :--------: | :----------: | ---------------------------------------- |
| [OpenAI](/geneva/udfs/providers/openai) | ✓ | ✓ | — | `geneva[udf-text-openai]` |
| [Gemini](/geneva/udfs/providers/gemini) | ✓ | ✓ | — | `geneva[udf-text-gemini]` |
| [Sentence Transformers](/geneva/udfs/providers/sentence-transformers) | ✓ | — | ✓ | `geneva[udf-text-sentence-transformers]` |
OpenAI and Gemini UDFs make remote API calls that incur per-token costs.
Sentence Transformers run locally on your workers with no API costs — see
[GPU acceleration](/geneva/udfs/providers/sentence-transformers#gpu-acceleration)
for performance tips.
## Comparing models and prompts
Because `add_columns` accepts a dictionary, you can evaluate multiple models, parameter
settings, or prompts in a single pass over your data. Each entry produces its own column,
so results sit side by side in the same table for easy comparison.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.udfs import openai_udf, gemini_udf, openai_embedding_udf
table.add_columns({
# Compare two embedding models
"emb_small": openai_embedding_udf(column="body", model="text-embedding-3-small"),
"emb_large": openai_embedding_udf(column="body", model="text-embedding-3-large"),
# Compare the same task across providers
"summary_openai": openai_udf(
column="body",
prompt="Summarize in one sentence",
model="gpt-5-mini",
),
"summary_gemini": gemini_udf(
column="body",
prompt="Summarize in one sentence",
model="gemini-2.5-flash",
),
})
```
This works for any combination — different models from the same provider, different providers,
different prompts with the same model, or different dimensionality settings. All columns are
computed in parallel during the same backfill job.
To recompute columns later (e.g., after altering a UDF or adding new rows), use `backfill`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Backfill a single column
table.backfill("emb_small")
# Backfill only rows missing a value
table.backfill("summary_openai", where="summary_openai is null")
```
## What's included
All built-in UDFs share these capabilities:
* **API key handling** — Keys are captured from your local environment at UDF creation time and securely serialized with the UDF. No cluster-level environment configuration required.
* **Retry with backoff** — Transient API errors (rate limits, timeouts, server errors) are automatically retried with exponential backoff.
* **Batch processing** — Embedding UDFs batch multiple rows per API call for better throughput.
* **L2 normalization** — Embedding UDFs support optional L2 normalization via the `normalize` parameter (disabled by default since both providers return pre-normalized vectors).
## See also
* [Working with UDFs](/geneva/udfs/index) — Write custom scalar, batched, and stateful UDFs
* [Error handling](/geneva/udfs/error_handling) — Fine-grained retry and skip policies
* [Working with blobs](/geneva/udfs/blobs) — Process binary data (images, audio, video)
## API Reference
* [Embeddings](https://lancedb.github.io/geneva/api/embeddings/) — `sentence_transformer_udf()`, `gemini_embedding_udf()`, `openai_embedding_udf()`
* [Gemini](https://lancedb.github.io/geneva/api/gemini/) — `gemini_udf()`
* [OpenAI](https://lancedb.github.io/geneva/api/openai/) — `openai_udf()`, `openai_embedding_udf()`
# OpenAI
Source: https://docs.lancedb.com/geneva/udfs/providers/openai
Embed text and generate completions using OpenAI models.
See the [API reference](https://lancedb.github.io/geneva/api/openai/) for all parameters.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install 'geneva[udf-text-openai]'
```
OpenAI UDFs make API calls that incur **per-token costs**. Each row processed results in one
or more API requests billed to your account. Review
[OpenAI pricing](https://openai.com/api/pricing/) before running on large tables.
Set the `OPENAI_API_KEY` environment variable before calling any factory function below.
The key is read **at UDF creation time** and serialized with the UDF — no cluster-level
`env_vars` configuration is needed.
## Embeddings
**Compare models by adding multiple embedding columns at once:**
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.udfs import openai_embedding_udf
table.add_columns({
# Default model — fast, 1536 dimensions
"embedding_small": openai_embedding_udf(
column="body",
model="text-embedding-3-small",
),
# Higher-quality model — 3072 dimensions
"embedding_large": openai_embedding_udf(
column="body",
model="text-embedding-3-large",
),
# Same large model, truncated to 256 dimensions for storage efficiency
"embedding_large_256": openai_embedding_udf(
column="body",
model="text-embedding-3-large",
output_dimensionality=256,
),
})
```
## Generation
Generate text from OpenAI chat completion models. Supports both text and binary (image)
input columns.
See the [API reference](https://lancedb.github.io/geneva/api/openai/#geneva.udfs.openai.openai_udf) for all parameters.
**Add a summary and an image caption in one call, using different models:**
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.udfs import openai_udf
table.add_columns({
# Fast model for bulk text summarization
"summary": openai_udf(
column="body",
prompt="Summarize this document in 3 bullet points",
model="gpt-5-mini",
),
# More capable model for nuanced image captions
"caption": openai_udf(
column="image",
prompt="Provide a 1 sentence description of the scene",
model="gpt-5",
mime_type="image/jpeg",
),
})
```
## API Reference
* [OpenAI](https://lancedb.github.io/geneva/api/openai/) — `openai_udf()` and `openai_embedding_udf()` — all parameters including `column`, `prompt`, `model`, `mime_type`, `dimensions`, and `normalize`
# Sentence Transformers
Source: https://docs.lancedb.com/geneva/udfs/providers/sentence-transformers
Embed text using any HuggingFace Sentence Transformer model locally — no API key needed.
See the [API reference](https://lancedb.github.io/geneva/api/embeddings/#geneva.udfs.text.embeddings.sentence_transformer_udf) for all parameters.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install 'geneva[udf-text-sentence-transformers]'
```
Sentence Transformer models run **locally** on your workers — there are no API calls and no
per-token costs. This makes them a good fit for large-scale embedding jobs where cost is a
concern.
## Embeddings
**Compare a lightweight and a high-quality model side by side:**
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva.udfs import sentence_transformer_udf
table.add_columns({
# Lightweight default model — fast, CPU-friendly
"embedding_mini": sentence_transformer_udf(
column="body",
model="sentence-transformers/all-MiniLM-L6-v2",
),
# Larger model with GPU acceleration
"embedding_bge": sentence_transformer_udf(
column="body",
model="BAAI/bge-large-en-v1.5",
num_gpus=1.0,
),
})
```
## GPU acceleration
Sentence Transformer models can run on CPU or GPU. Smaller models like `all-MiniLM-L6-v2`
work well on CPU, but larger models like `bge-large-en-v1.5` benefit significantly from GPU
acceleration. Use the `num_gpus` parameter to request GPU resources for a worker:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# CPU-only (default) — suitable for lightweight models
sentence_transformer_udf(column="body", model="sentence-transformers/all-MiniLM-L6-v2")
# GPU-accelerated — recommended for larger models
sentence_transformer_udf(column="body", model="BAAI/bge-large-en-v1.5", num_gpus=1.0)
```
Setting `num_gpus` to a fractional value (e.g., `0.5`) tells the
[Ray scheduler](https://docs.ray.io/en/latest/ray-core/scheduling/accelerators.html)
to co-locate multiple workers on the same physical GPU. For example, two UDFs with
`num_gpus=0.5` will be scheduled on a single GPU. Note that Ray does not enforce GPU memory
limits — it is your responsibility to ensure the combined models fit in GPU memory.
## API Reference
* [Embeddings](https://lancedb.github.io/geneva/api/embeddings/) — `sentence_transformer_udf()` — all parameters including `column`, `model`, `num_gpus`, `normalize`, and `batch_size`
# Chunkers (Scalar UDTFs)
Source: https://docs.lancedb.com/geneva/udfs/scalar-udtfs
Use chunkers (scalar UDTFs) for 1:N row expansion — split videos into clips, chunk documents, or tile images with automatic parent column inheritance and incremental refresh.
Beta — introduced in Geneva 0.11.0
Standard UDFs produce exactly **one output value per input row**. **Chunkers** — also
called scalar UDTFs — enable **1:N row expansion**: each source row can produce multiple
output rows. The results are stored as a materialized view with MV-style incremental refresh.
| Source Table | Derived Table | Expansion |
| -------------- | -------------- | ------------------ |
| 1 video row | → N clip rows | Video segmentation |
| 1 document row | → N chunk rows | Text chunking |
| 1 image row | → N tile rows | Image tiling |
For example, a chunker that splits documents into passages turns a `documents` table into a
`chunks` table, carrying the parent columns into every child row:
**Source: `documents`**
| doc\_id | title | text |
| ------- | ------------- | ------------------------ |
| 1 | "Intro to AI" | "Machine learning is..." |
| 2 | "Data Guide" | "Data pipelines are..." |
**Derived: `chunks`** (1:N expansion)
| doc\_id | title | chunk\_index | chunk\_text |
| ------- | ------------- | ------------ | --------------------- |
| 1 | "Intro to AI" | 0 | "Machine learning..." |
| 1 | "Intro to AI" | 1 | "Neural networks..." |
| 1 | "Intro to AI" | 2 | "Training data..." |
| 2 | "Data Guide" | 0 | "Data pipelines..." |
| 2 | "Data Guide" | 1 | "ETL processes..." |
Parent columns (`doc_id`, `title`) are inherited automatically; `chunk_index` and
`chunk_text` are generated by the chunker.
## Defining a Chunker
Use the `@chunker` decorator on a function that **yields** output rows. Geneva infers the output schema from the return type annotation.
Input parameters are bound to source columns **by name** — the parameter `video_path` binds to source column `video_path`, just like standard UDFs.
A chunker can yield **zero rows** for a source row. The source row is still marked as processed and will not be retried on the next refresh.
### List return pattern
If you prefer to build the full list in memory rather than yielding, you can return a `list` instead of an `Iterator`:
### Batched chunker
For vectorized processing, use `batch=True`. The function receives Arrow arrays and returns a `RecordBatch` of expanded rows. Because the return type `pa.RecordBatch` cannot be inferred, you must supply `output_schema` explicitly:
## Creating a Chunker View
Chunkers use the `create_udtf_view` API (passing the chunker as the `udtf` argument):
The `query` parameter controls which source columns are inherited. Columns listed in `.select()` are carried into every child row automatically.
## Inherited Columns
Child rows automatically include the parent's columns — no manual join required. The columns available in the child table are determined by the query's `.select()`:
### `videos` table (source)
| video\_path | duration | metadata |
| ----------- | -------- | ---------- |
| /v/a.mp4 | 120.0 | \{fps: 30} |
| /v/b.mp4 | 60.0 | \{fps: 24} |
### `clips` table (derived, 1:N)
| video\_path | metadata | clip\_start | clip\_end | clip\_bytes |
| ----------- | ---------- | ----------- | --------- | -------------- |
| /v/a.mp4 | \{fps: 30} | 0.0 | 10.0 | b"\x00\x1a..." |
| /v/a.mp4 | \{fps: 30} | 10.0 | 20.0 | b"\x00\x2b..." |
| /v/a.mp4 | \{fps: 30} | 20.0 | 30.0 | b"\x00\x3c..." |
| | | | | |
| /v/b.mp4 | \{fps: 24} | 0.0 | 10.0 | b"\x00\x4d..." |
| /v/b.mp4 | \{fps: 24} | 10.0 | 20.0 | b"\x00\x5e..." |
The first three rows come from the `/v/a.mp4` source row, the last two from `/v/b.mp4`. Inherited columns (`video_path`, `metadata`) are carried over automatically; `clip_start`, `clip_end`, and `clip_bytes` are generated by the UDTF.
## Adding Computed Columns After Creation
Since chunker views are materialized views, you can add UDF-computed columns to the child table and backfill them:
This is a powerful pattern: expand source rows with a chunker, then enrich the expanded rows with standard UDFs.
## Incremental Refresh
Chunkers support **incremental refresh**, just like standard materialized views:
* **New source rows**: The UDTF runs on new rows, inserting child rows.
* **Deleted source rows**: Child rows linked to the deleted parent are cascade-deleted.
* **Updated source rows**: Old children are deleted, UDTF re-runs, new children inserted.
Only the new source rows are processed. Existing clips from previous refreshes are untouched.
## Chaining Chunker Views
Chunker views are standard materialized views, so they can serve as the source for further views:
## Full Example: Document Chunking
For a comparison of all three function types (UDFs, Chunkers, Batch UDTFs), see [Understanding Transforms](/geneva/udfs).
Reference:
* [`chunker` API](https://lancedb.github.io/geneva/api/udtf/#geneva.chunker)
* [`create_udtf_view` API](https://lancedb.github.io/geneva/api/connection/#geneva.db.Connection.create_udtf_view)
# User-Defined Functions (UDFs)
Source: https://docs.lancedb.com/geneva/udfs/udfs
Define 1:1 transforms that add computed columns to your tables — embeddings, enrichment, scoring, and more.
UDFs are the core building block for feature engineering in Geneva. A UDF wraps a Python function and applies it to every row in a table, producing exactly **one output value per input row** (1:1). Use UDFs to compute embeddings, enrich data with external APIs, transform formats, or derive new features from existing columns.
## Defining a UDF
Converting your Python code to a Geneva UDF is simple. There are three kinds of UDFs that you can provide — scalar UDFs, batched UDFs and stateful UDFs.
In all cases, Geneva uses Python type hints from your functions to infer the input and output
[arrow data types](https://arrow.apache.org/docs/python/api/datatypes.html) that LanceDB uses.
### Scalar UDFs
The **simplest** form is a scalar UDF, which processes one row at a time:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from geneva import udf
@udf
def area_udf(x: int, y: int) -> int:
return x * y
```
This UDF will take the value of `x` and value of `y` from each row and return the product. The `@udf` wrapper is all that is needed.
### Batched UDFs
For **better performance**, you can also define batch UDFs that process multiple rows at once.
You can use `pyarrow.Array`s:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
from geneva import udf
@udf(data_type=pa.int32())
def batch_filename_len(filename: pa.Array) -> pa.Array:
lengths = [len(str(f)) for f in filename]
return pa.array(lengths, type=pa.int32())
```
Or take entire rows using `pyarrow.RecordBatch`:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
from geneva import udf
@udf(data_type=pa.int32())
def recordbatch_filename_len(batch: pa.RecordBatch) -> pa.Array:
filenames = batch["filename"]
lengths = [len(str(f)) for f in filenames]
return pa.array(lengths, type=pa.int32())
```
> **Note**: Batch UDFS require you to specify `data_type` in the `@udf` decorator for batched UDFs which defines `pyarrow.DataType` of the returned `pyarrow.Array`.
### Struct outputs
A UDF can return multiple related values as a single `struct` column by setting `data_type` to a `pa.struct(...)` and returning a tuple (matched by field order) or a `dict` keyed by field name.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import io
import pyarrow as pa
from geneva import udf
@udf(
data_type=pa.struct(
[pa.field("width", pa.int32()), pa.field("height", pa.int32())]
),
)
def dimensions(image: bytes) -> tuple[int, int]:
"""Extract image dimensions (width, height)."""
from PIL import Image
img = Image.open(io.BytesIO(image))
return img.size
```
Downstream UDFs can then read individual fields via dot notation in `input_columns` (see below).
### Struct fields and list inputs
You can pass nested `struct` fields directly into a UDF by specifying `input_columns` with dot notation. For list-typed inputs, Geneva can pass a NumPy array when the argument is annotated as `np.ndarray` (use `np.ndarray | None` for nullable lists).
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import numpy as np
import pyarrow as pa
from geneva import udf
struct_type = pa.struct([("vals", pa.list_(pa.int32()))])
schema = pa.schema([pa.field("info", struct_type)])
@udf(data_type=pa.int32(), input_columns=["info.vals"])
def sum_vals(vals: np.ndarray | None) -> int | None:
if vals is None:
return None
assert isinstance(vals, np.ndarray)
return int(np.sum(vals))
```
### Stateful UDFs
You can also define a **stateful** UDF that retains its state across calls.
This can be used to share code and **parameterize your UDFs**. In the example below, the model being used is a parameter that can be specified at UDF registration time. It can also be used to parameterize input column names of `pa.RecordBatch` batch UDFS.
This also can be used to **optimize expensive initialization** that may require heavy resources on the distributed workers. For example, this can be used to load a model to the GPU once for all records sent to a worker instead of once per record or per batch of records.
A stateful UDF is a `Callable` class, with `__call__()` method. The call method can be a scalar function or a batched function.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from typing import Callable
from openai import OpenAI
@udf(data_type=pa.list_(pa.float32(), 1536))
class OpenAIEmbedding(Callable):
def __init__(self, model: str = "text-embedding-3-small"):
self.model = model
# Per-worker openai client
self.client: OpenAI | None = None
def __call__(self, text: str) -> pa.Array:
if self.client is None:
self.client = OpenAI()
resp = self.client.embeddings.create(model=self.model, input=text)
return pa.array(resp.data[0].embeddings)
```
For common providers like OpenAI and Gemini, Geneva ships [built-in UDFs](/geneva/udfs/providers) that handle API keys, retries, and batching for you — no custom class needed.
> **Note**: The state is will be independently managed on each distributed Worker.
## UDF options
The `udf` can have extra annotations that specify resource requirements and operational characteristics.
These are just add parameters to the `udf(...)`.
### Resource requirements for UDFs
Some workers may require specific resources such as gpus, cpus and certain amounts of RAM.
You can provide these requirements by adding `num_cpus`, `num_gpus`, and `memory` parameters to the UDF.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(..., num_cpus=1, num_gpus=0.5, memory = 4 * 1024**3) # require 1 CPU, 0.5 GPU, and 4GiB RAM
def func(...):
...
```
### Operational parameters for UDFs
#### checkpoint\_size
`checkpoint_size` controls how many rows are processed before checkpointing, and therefore reporting and saving progress.
UDFs can be quite varied: some can be simple operations where thousands of calls can be completed per second, while others may be slow and require 30s per row. So a simple default like "every 1000 rows" might write once a second or once every 8 hours!
Geneva will handle this internally, using an experimental feature that will adapt checkpoint sizing as a UDF progresses. However, if you want to see writes more or less frequently, you can set this manually. There are three parameters:
* `checkpoint_size`: the seed for the initial checkpoint size
* `min_checkpoint_size`: the minimum value that Geneva will use while adapting checkpoint size
* `max_checkpoint_size`: the maximum value that Geneva will use while adapting checkpoint size
Therefore, to force a checkpoint size (and effectively disable adaptive batch sizing), set all three of these parameters to the same value.
### Error handling
Depending on the UDF, you may want Geneva to ignore rows that hit failures, retry, or fail the entire job. For simple cases, Geneva provides a simple parameter, `on_error`, with the following options:
| Function | Behavior |
| ------------------- | --------------------------------------------------------------------------- |
| `retry_transient()` | Retry `ConnectionError`, `TimeoutError`, `OSError` with exponential backoff |
| `retry_all()` | Retry any exception with exponential backoff |
| `skip_on_error()` | Return `None` for any exception (skip the row) |
| `fail_fast()` | Fail immediately on any exception (default behavior) |
If those are not specific enough, Geneva also provides [many more error handling options](/geneva/udfs/error_handling).
## Registering Features with UDFs
Registering a feature is done by providing the `Table.add_columns()` function a new column name and the Geneva UDF.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import geneva
import numpy as np
import pyarrow as pa
lancedb_uri="gs://bucket/db"
db = geneva.connect(lancedb_uri)
# Define schema for the video table
schema = pa.schema([
("filename", pa.string()),
("duration_sec", pa.float32()),
("x", pa.int32()),
("y", pa.int32()),
])
tbl = db.create_table("videos", schema=schema, mode="overwrite")
# Generate fake data
N = 10
data = {
"filename": [f"video_{i}.mp4" for i in range(N)],
"duration_sec": np.random.uniform(10, 300, size=N).astype(np.float32),
"x": np.random.choice([640, 1280, 1920], size=N),
"y": np.random.choice([360, 720, 1080], size=N),
"caption": [f"this is video {i}" for i in range(N)]
}
# Convert to Arrow Table and add to LanceDB
batch = pa.table(data, schema=schema)
tbl.add(batch)
```
Here's how to register a simple UDF:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf
def area_udf(x: int, y: int) -> int:
return x * y
@udf
def download_udf(filename: str) -> bytes:
...
# {'new column name': , ...}
# simple_udf's arguments are `x` and `y` so the input columns are
# inferred to be columns `x` amd `y`
tbl.add_columns({"area": area_udf, "content": download_udf })
```
### Registering Multi-Output UDFs
Use a multi-output UDF when one expensive read or decode can produce several features. For example, if a table stores image bytes, a single UDF can open the image once and return `height`, `width`, and an embedding column together. This avoids separate UDFs that would each read or decode the same image.
Define the output shape with `typing.NamedTuple` and annotate the UDF return type as `geneva.Columns[YourNamedTuple]`. Passing that UDF directly to [`Table.add_columns()`](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.add_columns) expands the result into multiple sibling columns using the `NamedTuple` field names.
If those names need a namespace or would conflict with existing columns, wrap the UDF with `geneva.UnpackedUDF(udf, prefix="...")` before calling `add_columns()`. The prefix is added to each materialized column name while keeping the outputs in one logical feature group.
Manage multi-output sibling columns as a group. Backfill, drop, or alter the full group together instead of changing only one sibling column.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import io
from typing import NamedTuple
import geneva
from PIL import Image
db = geneva.connect("/data/mydb")
tbl = db.open_table("images")
class ImageFeatures(NamedTuple):
height: int
width: int
embedding: list[float]
@geneva.udf
def image_features(image: bytes) -> geneva.Columns[ImageFeatures]:
img = Image.open(io.BytesIO(image)) # Read and decode the image once.
embedding = embedding_model.encode(img)
return ImageFeatures(
height=img.height,
width=img.width,
embedding=embedding,
)
# Adds sibling columns named "height", "width", and "embedding".
tbl.add_columns(image_features)
# Or add the same outputs with a prefix to avoid name conflicts.
tbl.add_columns(geneva.UnpackedUDF(image_features, prefix="image_"))
```
Batched UDFs require return type in their `udf` annotations
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.int32())
def batch_filename_len(filename: pa.Array) -> pa.Array:
...
# {'new column name': }
# batch_filename_len's input, `filename` input column is
# specified by the UDF's argument name.
tbl.add_columns({"filename_len": batch_filename_len})
```
or
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.int32())
def recordbatch_filename_len(batch: pa.RecordBatch) -> pa.Array:
...
# {'new column name': }
# batch_filename_len's input. pa.RecordBatch typed UDF
# argument pulls in all the column values for each row.
tbl.add_columns({"filename_len": recordbatch_filename_len})
```
Similarly, a stateful UDF is registered by providing an instance of the Callable object. The call method may be a per-record function or a batch function.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.list_(pa.float32(), 1536))
class OpenAIEmbedding(Callable):
...
def __call__(self, text: str) -> pa.Array:
...
# OpenAIEmbedding's call method input is inferred to be 'text' of
# type string from the __call__'s arguments, and its output type is
# a fixed size list of float32.
tbl.add_columns({"embedding": OpenAIEmbedding()})
```
## Changing data in computed columns
Let's say you backfilled data with your UDF then you noticed that your data has some issues. Here are a few scenarios:
1. All the values are incorrect due to a bug in the UDF.
2. Most values are correct but some values are incorrect due to a failure in UDF execution.
3. Values calculated correctly and you want to perform a second pass to fixup some of the values.
In scenario 1, you'll most likely want to replace the UDF with a new version and recalculate all the values. You should perform a `alter_table` and then `backfill`.
In scenario 2, you'll most likely want to re-execute `backfill` to fill in the values. If the error is in your code (certain cases not handled), you can modify the UDF, and perform an `alter_table`, and then `backfill` with some filters.
In scenario 3, you have a few options. A) You could `alter` your UDF and include the fixup operations in the UDF. You'd `alter_table` and then `backfill` recalculating all the values. B) You could have a chain of computed columns -- create a new column, calculate the "fixed" up values and have your application use the new column or a combination of the original column. This is similar to A but does not recalculate A and can incur more storage. C) You could `update` the values in the column with the fixed up values. This may be expedient but also sacrifices reproducibility.
The next section shows you how to change your column definition by `alter`ing the UDF.
## Altering UDFs
You now want to revise the code. To make the change, you'd update the UDF used to compute the column using the `alter_columns` API and the updated function. The example below replaces the definition of column `area` to use the `area_udf_v2` function.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.alter_columns({"path": "area", "udf": area_udf_v2} )
```
After making this change, the existing data already in the table does not change. However, when you perform your next basic `backfill` operation, all values would be recalculated and updated. If you only wanted some rows updated, you could perform a filtered backfill, targeting the specific rows that need the new upates.
For example, this filter would only update the rows where area was currently null.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.backfill("area", where="area is null")
```
## Auto-backfill
For columns whose values should always stay in sync with their source data, set
`auto_backfill=True` on the UDF. On LanceDB Enterprise (`db://` connections), the column is
then recomputed for you automatically — you don't need to call `backfill()` yourself.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.int32(), version="2", auto_backfill=True)
def area_udf(x: int, y: int) -> int:
return x * y
# Register a new auto-backfill column...
tbl.add_columns({"area": area_udf})
# ...or re-point an existing column to an auto-backfill UDF
tbl.alter_columns({"path": "area", "udf": area_udf})
```
### How it works
The `auto_backfill` flag is recorded in the column's metadata when the column is added or
altered. LanceDB Enterprise's managed agent watches for columns that need recomputation and
dispatches a [distributed backfill job](/geneva/jobs/backfilling/) automatically — there is no
manual trigger and no status polling. A column is recomputed when, for example:
* **New rows are added** to the table (`tbl.add(...)`), leaving the column null for those rows.
* **The UDF version changes** — you bump `version=` and `alter_columns()` to the new function.
Auto-backfill is an enterprise feature. On direct object-storage or local-filesystem
connections there is no managed agent, so `auto_backfill=True` has no effect and you must run
`backfill()` explicitly.
Reference:
* [`alter_columns` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.alter_columns)
* [`add_columns` API](https://lancedb.github.io/geneva/api/table/#geneva.table.Table.add_columns)
* [UDF](https://lancedb.github.io/geneva/api/udf/) — full `@udf` decorator reference including `data_type`, `num_gpus`, `auto_backfill`, `batch_size`, and other options
# LanceDB
Source: https://docs.lancedb.com/index
Multimodal lakehouse for AI.
**LanceDB** is a [multimodal lakehouse](https://lancedb.com/blog/multimodal-lakehouse/) for AI teams that need
one data layer for curation, feature engineering, search and retrieval, and model training.
It is built on top of [Lance](/lance), an open-source lakehouse format designed for multimodal AI data.
Move from data exploration to model training on one, unified platform without needing to manage a
fragmented stack of storage, feature, retrieval, and training systems.
## Build better models, faster
Training data and experimentation slow down when raw data, metadata, embeddings, features, and governance
artifacts live in separate systems. LanceDB keeps them together in one versioned multimodal table, so AI teams spend less
time stitching infrastructure together and more time improving datasets, testing features, and keeping GPUs fed.
![Training data lifecycle: Curation, Feature Engineering, Search and Retrieval, Training]()
Use the same table to curate training data, add derived features, retrieve examples, and feed training jobs that rely on expensive GPUs.
Training workloads can sample, shuffle, and scan projected columns from local storage or object storage, then assemble
GPU-ready batches from a tagged dataset version.
For a deeper look at how this works in training pipelines, start with [Why LanceDB for training](/training/why-lancedb).
## LanceDB suite
The LanceDB suite includes LanceDB OSS, an open-source embedded retrieval library, and LanceDB Enterprise,
a multimodal lakehouse platform for the full AI data lifecycle.
OSS is easy to set up on a local machine for search and regular-scale workflows. LanceDB Enterprise is built
for teams that need scale without building bespoke infrastructure for curation,
feature engineering, search and retrieval, and efficient training data access.
![LanceDB suite: OSS search and Enterprise multimodal lakehouse on Lance format]()
## Why teams use LanceDB
Store images, video, audio, text, annotations, embeddings, and model-generated features together in one schema-enforced table.
The same table can support dataset curation, feature backfills, experiment splits, retrieval, and training.
Training workloads mix fast random access with high-throughput sequential scans. LanceDB is designed for both, so
teams can shuffle data into GPU-ready batches more efficiently, improve input throughput, and iterate on experiments faster.
Whether the end user is a human or an agent, LanceDB powers production retrieval workloads such as semantic search,
hybrid search, RAG, agent memory, and recommendation systems. Retrieval runs against the same LanceDB tables used
for curation, feature engineering, and training workflows.
## Start with your workload
Learn why LanceDB works well as the data layer for training workloads.
Use LanceDB tables and permutations for projected, shuffled, random-access training reads.
Explore Lance-formatted multimodal datasets with raw bytes, metadata, embeddings, and indices.
Use vector search, full-text search, hybrid search, reranking, filtering, and SQL.
## From local development to production scale
LanceDB OSS and LanceDB Enterprise share the same Lance format and table model. Start locally with the embedded OSS
library, then move to Enterprise when your team needs distributed scale, managed infrastructure, private deployment,
or higher-throughput curation, feature engineering, search and retrieval, and training workflows.
### 1. LanceDB OSS
The fastest way to get started is the open-source embedded library, with client SDKs in Python, TypeScript
and Rust. Run it locally in just a few steps, which lets you explore datasets, curate data, and run search and retrieval workloads
for agents. Start here:
Get started with LanceDB in minutes.
Create tables, evolve schemas, version data, and modify rows in LanceDB.
### 2. LanceDB Enterprise
[LanceDB Enterprise](/enterprise) is a petabyte-scale (and beyond), distributed **multimodal lakehouse** platform built for
search, curation, feature engineering, and high-throughput training data access workflows on top of the same core table
abstraction. This eliminates the need for teams to build bespoke infrastructure to manage large multimodal datasets.
To set up LanceDB Enterprise in your organization, reach out to us at
[contact@lancedb.com](mailto:contact@lancedb.com).
**Built with scale, performance, and security in mind.**
LanceDB Enterprise is designed for very large-scale, high-performance, distributed workloads in
private deployments, and can operate under strict [security requirements](/enterprise/security).
Get started with LanceDB in minutes, including Enterprise `db://` connections.
# Full-Text Search (FTS) Index
Source: https://docs.lancedb.com/indexing/fts-index
Create and tune BM25-based full-text search indexes in LanceDB.
LanceDB provides performant full-text search based on BM25, allowing you to incorporate keyword-based search in your retrieval solutions. This page shows
examples on how to create and configure FTS indexes in LanceDB OSS and Enterprise, using the synchronous and asynchronous APIs.
In LanceDB Enterprise, `create_fts_index` API returns immediately, but index building happens asynchronously.
## Creating FTS Indexes
### Synchronous API
Use `create_fts_index` with synchronous LanceDB connections:
Check FTS index status using the API:
`wait_for_index(...)` waits until the named FTS index exists and `index_stats(...)` reports `num_unindexed_rows == 0`. It can time out if writes keep adding rows faster than the index catches up. If a table has multiple FTS indexes, specify the target text column when querying instead of relying on implicit selection.
### Asynchronous API
When using async connections (`connect_async`), use `create_index` with the `FTS` configuration:
The `create_fts_index` method is not available on `AsyncTable`. Use `create_index` with `FTS` config instead.
The current FTS implementation is Lance-native. Legacy Tantivy-only options, including
`use_tantivy`, are no longer accepted by the index creation APIs.
## Nested field paths
FTS indexes can target text leaves inside struct columns by passing a dotted path (for example, `payload.text`). The same path works for [`MatchQuery`](/search/full-text-search) and [`PhraseQuery`](/search/full-text-search), and for the `columns` argument on async `nearest_to_text` queries.
You can point an index at any string leaf nested in a struct, regardless of depth. The struct container itself isn't indexable: you have to name a specific text field.
LanceDB rejects paths that don't resolve to a text leaf:
* A struct container (for example, `payload`): raises `ValueError: FTS index cannot be created ...`.
* A non-text leaf such as an integer or float (for example, `payload.count`): raises the same error.
* A path that doesn't exist in the schema (for example, `payload.missing`): raises `ValueError: Field path ... not found`.
The async API accepts the same dotted paths through `create_index`:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.index import FTS
await async_table.create_index("payload.text", config=FTS(with_position=True))
```
## Configuration Options
### FTS Parameters
| Parameter | Type | Default | Description |
| :------------------ | :--------- | :---------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `with_position` | bool | `False` | Store token positions (required for phrase queries) |
| `base_tokenizer` | str | `"simple"` | Text splitting method (`simple`, `whitespace`, `raw`, `ngram`, `icu`, `jieba/*`, or `lindera/*`) |
| `language` | str | `"English"` | Language for stemming and stop-word filters. Choose CJK and mixed-language segmentation with `base_tokenizer`. |
| `max_token_length` | int | `40` | Maximum token size; longer tokens are omitted |
| `lower_case` | bool | `True` | Lowercase tokens |
| `stem` | bool | `True` | Apply stemming (`running` → `run`) |
| `remove_stop_words` | bool | `True` | Drop common stop words |
| `ascii_folding` | bool | `True` | Normalize accented characters |
| `custom_stop_words` | list\[str] | `None` | Extra stop words to drop in addition to the language defaults. Requires `remove_stop_words=True`. |
| `ngram_min_length` | int | `3` | Minimum n-gram length. Applies only when `base_tokenizer="ngram"`. |
| `ngram_max_length` | int | `3` | Maximum n-gram length. Applies only when `base_tokenizer="ngram"`. |
| `prefix_only` | bool | `False` | Index only prefix n-grams rather than all substrings. Applies only when `base_tokenizer="ngram"`. |
| `block_size` | int | `128` | Number of documents per compressed posting block. Supported values are `128` and `256`. Setting this to `256` opts in to the experimental FTS V3 layout. |
* `max_token_length` can filter out base64 blobs or long URLs.
* Disabling `with_position` reduces index size but disables phrase queries.
* `ascii_folding` helps with international text (e.g., “café” → “cafe”).
### Tokenizer choices
`base_tokenizer` controls segmentation before token filters run:
* `simple`, `whitespace`, and `raw` cover common tokenization strategies for space-delimited text.
* `ngram` indexes overlapping character spans for substring-style matching.
* `icu` uses bundled ICU4X word segmentation for mixed-language text and scripts where whitespace splitting is not enough. ICU stands for [International Components for Unicode](https://icu.unicode.org/), and this tokenizer does not need external model files.
* `jieba/*` is for Chinese word segmentation with Jieba.
* `lindera/*` loads a compiled Lindera dictionary, such as `lindera/ipadic` for Japanese or `lindera/ko-dic` for Korean.
Model-backed tokenizers such as `jieba/default`, `lindera/ipadic`, and `lindera/ko-dic` require tokenizer model files in Lance's language model home. Lance looks under the default platform data directory for `lance/language_models`, or you can set `LANCE_LANGUAGE_MODEL_HOME` to point to another model root. For example, `jieba/default` is resolved under `/jieba/default/...`.
`language` is used by token filters, not by the base tokenizer. Stemming supports Arabic, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, and Turkish. Built-in stop-word removal supports Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. For other stemming languages, set `remove_stop_words=False` or pass `custom_stop_words`.
### Posting block size
`block_size` controls the number of documents packed into each compressed posting block on disk. The default of `128` matches the current FTS layout and is the right choice for most workloads. Setting it to `256` opts in to the experimental FTS V3 format, which changes how postings are encoded and may introduce breaking changes in future releases. Any other value is rejected at index creation time.
You can set the option through either the synchronous or asynchronous API. In the async Python API, pass it on the `FTS` config, and in the TypeScript API use the camelCase `blockSize` field on `FtsOptions`:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.index import FTS
await async_table.create_index("text", config=FTS(block_size=256))
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
await table.createIndex("text", {
config: lancedb.Index.fts({ blockSize: 256 }),
});
```
### Phrase Query Configuration
Enable phrase queries by setting:
| Parameter | Required Value | Purpose |
| :------------------ | :------------- | :-------------------------------------------- |
| `with_position` | `True` | Track token positions for phrase matching |
| `remove_stop_words` | `False` | Preserve stop words for exact phrase matching |
## Indexing nested string fields
You can build an FTS index on a string field inside a struct by passing its full dotted path, like `nested.text`. The same path is used when you query the index through `fts_columns`, and the indexed column is reported back as the full path from `list_indices()`.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Schema: pa.struct([pa.field("text", pa.string())]) stored under the `nested` column.
table.create_fts_index("nested.text")
results = (
table.search("puppy", query_type="fts", fts_columns="nested.text")
.limit(5)
.to_list()
)
```
Use the canonical Lance path: dot-separate each struct field from root to leaf (for example, `metadata.author.name`). The same convention applies to scalar and vector indexes.
# GPU-Powered Vector Indexing
Source: https://docs.lancedb.com/indexing/gpu-indexing
Accelerate IVF and HNSW index builds with GPU acceleration in LanceDB.
With LanceDB's GPU-powered vector indexing you can index very large datasets in far less time
than you could with the default CPU-based indexing. In our tests, LanceDB
is capable of indexing billions of rows in under four hours on a 1-8 GPU cluster.
**Automatic GPU indexing**
Enterprise-only
Automatic GPU Indexing is currently only available in [LanceDB Enterprise](/enterprise/).
Please [contact us](mailto:contact@lancedb.com) to enable this feature for your deployment.
The vector index is created when you call `create_index`. The backend will use GPU resources
to build either the IVF or HNSW indexes. The system automatically selects the optimal GPU
configuration based on your data size and available hardware.
This process is also asynchronous by default, but you can use `wait_for_index` to convert it
into a synchronous process by waiting until the index is built.
`wait_for_index(...)` waits for the index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. It can time out if the table is receiving continuous writes while the build is trying to catch up.
GPU acceleration changes how the vector index is built, not the lifecycle after the build. Rows
appended later still need `optimize()` before they are part of the index, and `fast_search()` only
searches indexed rows.
## Manual GPU indexing in LanceDB OSS
You can use the Python SDK to manually create the `IVF_PQ` index on a GPU. You'll need
[PyTorch>2.0](https://pytorch.org/). Note that GPU-based indexing is currently only
supported by the synchronous SDK in LanceDB OSS.
Specify the values `cuda` or `mps` (on Apple Silicon) for the `accelerator` parameter
to enable GPU training on your device.
### GPU indexing on Linux
### GPU indexing on macOS (Apple Silicon)
## Performance considerations
* GPU memory usage scales with `num_partitions` and vector dimensions
* For optimal performance, ensure GPU memory exceeds dataset size
* Batch size is automatically tuned based on available GPU memory
* Indexing speed improves with larger batch sizes
## Troubleshooting
If you encounter the error `AssertionError: Torch not compiled with CUDA enabled`,
you need to [install PyTorch with CUDA support](https://pytorch.org/get-started/locally/).
# Indexing Data
Source: https://docs.lancedb.com/indexing/index
Optimize search performance in LanceDB using vector indexes, full-text search, and scalar indexes. Understand IVF-PQ indexing for efficient vector similarity search.
An **index** is a data structure that facilitates efficient scans and lookups on the embeddings of a given dataset. LanceDB provides a comprehensive suite of indexes to optimize query performance across diverse workloads:
* **Vector Index**: Optimized for searching high-dimensional data (like images, audio, or text embeddings) by efficiently finding the most similar vectors
* **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases
* **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g., timestamps, prices)
Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with:
* Vector search (prefilter or post-filter results using metadata)
* Full-text search (combining keyword matching with structured filters)
* SQL scans (optimizing WHERE clauses on scalar columns)
* Key-value lookups (enabling rapid primary key-based retrievals)
## Supported Index Types
LanceDB provides a comprehensive suite of indexing strategies for different data types and use cases:
| Index | Use Case | Description |
| :----------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
Distance metrics: `l2` `cosine` `dot` `hamming`
Quantizations: `None/Flat` `PQ` `SQ` `RQ` |
| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.
Distance metrics: `l2` `cosine` `dot`
Quantizations: `None/Flat` `SQ` `PQ` |
| `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. |
| `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. |
| `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. |
| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
| `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. |
TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization).
**Operational checks**
For vector indexes, use the same distance metric when creating the index and searching it. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` if you need to confirm the index has caught up. `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`; it can time out if writes keep adding unindexed rows.
By default, automatic vector indexing creates `IVF_PQ`, and scalar index creation defaults to
`BTree` unless you pass another scalar index config. `BTree` and `Bitmap` indexes target scalar
columns, not list columns; use `LabelList` for list containment filters.
### Quantization Types
Vector indexes can use different quantization methods to compress vectors and improve search performance:
| Quantization | Use Case | Description |
| :-------------------------- | :------------------------------------------------------------------------------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PQ` (Product Quantization) | Default choice for most vector search scenarios. Use when you need to balance index size and recall. | Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy. |
| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
| `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. |
| `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. |
## Understanding the IVF-PQ Index
An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **k-nearest neighbors (kNN)** problem.
It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below.
LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy.
## IVF-PQ
LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with **Product Quantization (PQ)** to efficiently compress embeddings.
The implementation provides several parameters to fine-tune the index's size, query throughput, latency, and recall.
### Product Quantization
Quantization is a compression technique used to reduce the dimensionality of an embedding to speed up search.
Product quantization (PQ) works by dividing a large, high-dimensional vector of size into equally sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector.
![]()
It's important to remember that quantization is a *lossy process*, i.e., the reconstructed vector is not identical to the original vector. This results in a trade-off between the size of the index and the accuracy of the search results.
As an example, consider starting with 128-dimensional vector consisting of 32-bit floats. Quantizing it to an 8-bit integer vector with 4 dimensions as in the image above, we can significantly reduce memory requirements.
Original: `128 × 32 = 4096` bits
Quantized: `4 × 8 = 32` bits
Quantization results in a **128x** reduction in memory requirements for each vector in the index, which is substantial.
### Inverted File Index (IVF) Implementation
While PQ helps with reducing the size of the index, IVF primarily addresses search performance. The primary purpose of an inverted file index is to facilitate rapid and effective nearest neighbor search by narrowing down the search space.
In IVF, the PQ vector space is divided into *Voronoi cells*, which are essentially partitions that consist of all the points in the space that are within a threshold distance of the given region's seed point. These seed points are initialized by running K-means over the stored vectors. The centroids of K-means turn into the seed points which then each define a region. These regions are then are used to create an inverted index that correlates each centroid with a list of vectors in the space, allowing a search to be restricted to just a subset of vectors in the index.
![]()
During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query.
![]()
## HNSW Index Implementation
Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly.
### Types of ANN Search Algorithms
Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one.
For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly.
There are three main types of ANN search algorithms:
* **Tree-based search algorithms**: Use a tree structure to organize and store data points.
* **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice.
* **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex.
HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a k-nearest neighbor (or k-approximate nearest neighbor) graph, which we outline below.\
HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list.
### Understanding k-Nearest Neighbor Graphs
The k-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple:
* Each vector in the dataset is given an associated vertex.
* Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex.
* For some applications (including nearest-neighbor search), the incoming edges are also added.
Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors:
* Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node).
* Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex.
* Repeat until a local optimum is found.
The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors.
Computing a k-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the k-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster.\
In fact, another data structure is not needed: This can be done "incrementally".
That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex.
One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index.
### Hierarchical Navigable Small Worlds (HNSW)
HNSW builds on k-ANN in two main ways:
* Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant.
* The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list.
This recursive structure can be thought of as separating into layers:
* At the bottom-most layer, a k-ANN graph on the whole dataset is present.
* At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present.
* At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer.
Then the greedy search routine operates as follows:
* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer.
* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method.
* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10).
# Quantization
Source: https://docs.lancedb.com/indexing/quantization
Learn about quantization when creating an index in LanceDB.
Quantization compresses high-dimensional float vectors into a smaller, approximate representation, where instead of storing every vector as a float32 or float64, it's stored in compressed form, without too much of a compromise in search quality.
Use quantization when:
* You have a large dataset with relatively high-dimensional vectors (512, 768, 1024+)
* Index build time and query latency matter
LanceDB currently exposes multiple quantized vector index types, including:
* `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples.
* `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`.
* `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details.
* `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads.
* `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search.
Two axes are being combined here: whether partitions are searched flatly or via an HNSW graph (`IVF_*` vs. `IVF_HNSW_*`), and which quantizer compresses the vectors (`PQ`, `RQ`, or `SQ`). `IVF_PQ` is the default and works well in many cases. For more drastic compression, RaBitQ (`IVF_RQ`) is a reasonable option. For higher recall at low latency, the HNSW-backed variants are usually the right pick. The ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table on the vector indexing page is the canonical decision tool.
Use the same distance metric when training the index and running queries against it. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`.
## RaBitQ quantization
RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall.
### How RaBitQ works
* Embeddings are grouped around centroids (as in other IVF indexes).
* Each residual vector is normalized and mapped to the nearest vertex of a randomly rotated hypercube on the unit sphere.
* The sign pattern of that vector is stored as bits (1 bit per dimension).
* Two small corrective factors are stored:
1. The distance from the original vector to its centroid
2. The dot product between the normalized vector and its quantized version
Compared to `IVF_PQ`, RaBitQ:
* Avoids training expensive PQ codebooks
* Builds indexes faster and handles updates more easily
* Maintains or improves recall at high dimensionality under the same storage budget
For a deeper dive into the theory and some benchmark results, see the blog post: [LanceDB's RaBitQ Quantization for Blazing Fast Vector Search](https://lancedb.com/blog/feature-rabitq-quantization/).
### Using RaBitQ
You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`.
When using `IVF_RQ`, vector dimensions must be divisible by `8`.
`num_bits` controls how many bits per dimension are used:
1 bit is the classic RaBitQ setting. You can set it to 2, 4, or 8 bits to improve fidelity for better precision or recall — the main trade-off is additional storage for the extra bits per dimension, with only a modest increase in query-time compute.
It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to how you would do in `IVF_PQ`.
Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions.
## API Reference
The full list of parameters to the algorithm are listed below.
* `distance_type`: Literal\["l2", "cosine", "dot"], defaults to "l2"\
The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product.
* `num_partitions`: Optional\[int], defaults to None\
Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count.
* `num_bits`: int, defaults to 1\
Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage.
* `max_iterations`: int, defaults to 50\
Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality.
* `sample_rate`: int, defaults to 256\
Number of samples per partition during training. Higher values may improve accuracy but increase training time.
* `target_partition_size`: Optional\[int], defaults to None\
Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence.
# Keeping Indexes Up-to-Date with Reindexing
Source: https://docs.lancedb.com/indexing/reindexing
Learn how to keep your indexes up-to-date in LanceDB using incremental indexing, including best practices for adding new records without full reindexing.
As you add new data to your LanceDB tables, your indexes may become outdated.
Reindexing is the process of updating the index to account for new data -- this applies to either a full-text search (FTS) index or a vector index. Reindexing is an important operation to run periodically as your data grows, as it has performance implications.
As data is being added and a reindex operation is running, LanceDB will combine results from the existing index with exhaustive/flat search on the new data. This is done to ensure that you're still retrieving results over all your data, but it does come at a performance cost. The more data that you add without reindexing, the impact on latency (due to exhaustive search) can be noticeable.
Rather than dropping an existing index entirely and reindexing from scratch, LanceDB supports **incremental indexing**.
## Incremental Reindexing
You can manually trigger an incremental indexing operation on updated data
using the `optimize()` method on a table.
Table optimization performs three maintenance operations:
1. **Compaction**: merges small fragments into larger ones to improve read performance
2. **Pruning/Cleanup**: removes files from versions older than a retention window (7 days by default)
3. **Index update**: adds newly-ingested data to existing vector, scalar, and FTS indexes
Enterprise
LanceDB Enterprise support incremental reindexing through an automated background process. When new data is added to a table, the system automatically triggers a new index build. As the dataset grows, indexes are asynchronously updated in the background.
* While indexes are being rebuilt, queries use brute force methods on unindexed rows, which may temporarily increase latency. To avoid this, set `fast_search=True` to search only indexed data.
* Use `index_stats()` to view the number of unindexed rows. This will be zero when indexes are fully up-to-date. If you call `wait_for_index(...)`, it polls the same status and can time out while continuous writes keep adding unindexed rows.
The benefit of using LanceDB Enterprise is that it automates the reindexing process
and operates continuously in the background, minimizing the impact on latency under high loads.
In OSS, you must manually manage the reindexing cadence based on your data growth and performance needs.
## Disk utilization
Compaction by itself does not immediately free disk space, and can temporarily increase it because new
compacted files are written before old-version files are deleted. Disk space is reclaimed when old versions
are pruned during cleanup. Set retention only as low as your rollback and time-travel requirements allow.
If you need to reclaim space more aggressively in OSS, use a shorter retention window:
```python Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from datetime import timedelta
table.optimize(cleanup_older_than=timedelta(days=1))
```
# Scalar Indexes
Source: https://docs.lancedb.com/indexing/scalar-index
Learn how to use scalar indexes in LanceDB for efficient metadata filtering and query optimization.
Scalar indexes organize data by scalar attributes (e.g., numbers, categories) and enable fast filtering of vector data. They accelerate retrieval of scalar data associated with vectors, thus enhancing query performance.
LanceDB supports four types of scalar indexes:
* `BTREE`: Stores column data in sorted order for binary search. Best for columns with many unique values.
* `BITMAP`: Uses bitmaps to track value presence. Ideal for columns with few unique values (e.g., categories, tags).
* `LABEL_LIST`: Special index for `List` and `LargeList` columns of primitive values supporting `array_contains_all` and `array_contains_any` queries.
* `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`.
## Choosing the Right Index Type
| Data Type | Filter | Index Type |
| :-------------------------------------------------------------- | :---------------------------------------- | :----------- |
| Numeric, String, Temporal | `<`, `=`, `>`, `in`, `between`, `is null` | `BTREE` |
| Boolean, numbers or strings with fewer than 1,000 unique values | `<`, `=`, `>`, `in`, `between`, `is null` | `BITMAP` |
| List of low cardinality of numbers or strings | `array_has_any`, `array_has_all` | `LABEL_LIST` |
| String or binary (`Utf8`, `LargeUtf8`, `Binary`, `LargeBinary`) | `contains(col, 'needle')` | `FM` |
## Scalar Index Operations
### 1. Build the Index
You can create multiple scalar indexes within a table. By default, the index will be `BTREE`, but you can always configure another type like `BITMAP`
If you are using LanceDB Enterprise, the `create_scalar_index` API returns immediately, but the building of the scalar index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_scalar_index()` or call `wait_for_index()` on the table.
### 2. Check Index Status
`wait_for_index(...)` waits until the named scalar indexes exist and `index_stats(...)` reports `num_unindexed_rows == 0`. If a table is receiving steady writes, that fully indexed state may not stabilize before the timeout.
### 3. Update the Index
Updating the table data (adding, deleting, or modifying records) requires that you also update the scalar index. This can be done by calling `optimize`, which will trigger an update to the existing scalar index.
New data added after creating the scalar index will still appear in search results if optimize is not used, but with increased latency due to a flat search on the unindexed portion. LanceDB Enterprise automates the optimize process, minimizing the impact on search speed.
### 4. Run Indexed Searches
The following scan will be faster if the column `book_id` has a scalar index:
Scalar indexes can also speed up scans containing a vector search or full text search, and a prefilter:
## Indexing nested fields
Scalar indexes can target a scalar field inside a struct by passing its full dotted path. The path is preserved end to end: it's the value you pass to `create_scalar_index`, it's what `list_indices()` reports under `columns`, and it's the column reference you use in filter predicates.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Schema: pa.struct([pa.field("user_id", pa.int32())]) stored under the `metadata` column.
table.create_scalar_index("metadata.user_id", name="metadata_user_id_idx")
# The same dotted path works in WHERE clauses.
table.search().where("metadata.user_id = 42").limit(1).to_list()
```
Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `metadata.author.name`). The same convention applies to FTS and vector indexes.
## FM-Index for substring search
The `FM` index is a scalar index built over string or binary columns that
accelerates substring lookups expressed as `contains(col, 'needle')`. Unlike the
tokenized [FTS index](/indexing/fts-index), which matches whole words after
tokenization, the FM-Index matches arbitrary substrings of the raw bytes — so it
works well for URLs, file paths, identifiers, log lines, or any column where you
search for a fragment rather than a word.
Use the FM-Index when:
* Filters use `contains(col, 'needle')` (substring), not equality or word search.
* The column is `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary`.
* You want substring matches without paying for tokenization, language analysis,
or BM25 scoring.
Pick `FTS` instead when you need word-level relevance ranking, phrase queries,
or language-aware tokenization.
### Create an FM-Index
Build an FM-Index with the async `create_index` API by passing the `Fm` config in
Python or `Index.fm()` in TypeScript. In Rust, use `Index::Fm(FmIndexBuilder::default())`.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.index import Fm
await tbl.create_index("text", config=Fm())
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
import { Index } from "@lancedb/lancedb";
await tbl.createIndex("text", { config: Index.fm() });
```
```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
use lancedb::index::Index;
use lancedb::index::scalar::FmIndexBuilder;
tbl.create_index(&["text"], Index::Fm(FmIndexBuilder::default()))
.execute()
.await?;
```
After the index is built, substring filters use it automatically:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.search().where("contains(text, 'needle')").limit(10).to_pandas()
```
`list_indices()` reports the index type as `"Fm"`.
## Index UUID Columns
LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`), enabling efficient lookups and filtering on UUID-based primary keys.
**To use `FixedSizeBinary`, ensure you have:**
* Python SDK version `0.22.0` or later
* TypeScript SDK version `0.19.0` or later
### 1. Define UUID Type
### 2. Generate UUID Data
### 3. Create Table with UUID Column
### 4. Create and Wait for the Index
### 5. Perform Operations with the UUID Index
## Index nested fields
You can build a scalar index on a field inside a struct column by passing the
canonical dot-separated path to `create_index`. This is useful when filters
target attributes nested under a `metadata`-style column, for example
`metadata.user_id` or `metadata.event.type`.
If a literal segment of the path itself contains a dot (for example a column
named `user.id` nested inside `metadata`), wrap that segment in backticks so
LanceDB can tell the dot apart from the path separator: `` metadata.`user.id` ``.
`list_indices()` echoes the same canonical path back, so the column you pass in
round-trips through index metadata regardless of nesting depth or escaping.
Composite indexes that cover multiple columns aren't supported yet. Each
`create_index` call must target a single (possibly nested) field path.
# Vector Indexes
Source: https://docs.lancedb.com/indexing/vector-index
Build and optimize LanceDB vector indexes, including IVF, HNSW and binary quantized indexes.
You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two kinds of vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**.
**IVF + HNSW**
In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. This combines IVF's scalability with HNSW's higher-recall ANN search within partitions.
### Manual Indexing
If using LanceDB OSS, you will have to create the vector index manually, by calling `table.create_index()`, and updating the index as new data arrives and tuning its parameters is also a manual process.
### Automatic Indexing
Enterprise-only
Vector indexing is managed **automatically** in LanceDB Enterprise. As soon as data is updated, the system updates the index and optimizates it. *This is done asynchronously as a background process*.
When you create a table in LanceDB Enterprise, LanceDB automatically:
* Infers the vector columns from the schema
* Create an optimized `IVF_PQ` index without manual configuration
* Automatically configure indexing parameters
The default distance is `l2` (Euclidean).
You can call `create_index()` with different parameters to create a new index -- this replaces any existing index.
Although the `create_index` API returns immediately, the building of the vector index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter.
Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving.
Rows appended after an index build remain outside that index until optimization refreshes it. Normal
search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that
fallback and searches only indexed rows.
## Choose the Right Index
Use this table as a quick starting point for choosing the right index type and quantization method for your use case:
| If your top priority is... | Use this index | Why | Typical compressed size vs. raw vectors |
| :------------------------------------------------------- | :-------------- | :------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------ |
| Highest recall / no quantization | `IVF_HNSW_FLAT` | Uses raw vectors inside the IVF+HNSW structure, avoiding quantization loss. | Around raw vector size plus HNSW graph overhead |
| Best recall/latency trade-off | `IVF_HNSW_SQ` | Combines IVF partitioning with HNSW graph search for strong quality at low latency. | Typically a little larger than `1/4` of raw size |
| Maximum compression | `IVF_RQ` | RaBitQ-style quantization with very strong compression. | Around `1/32` of raw size |
| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | On small-dimensional vectors, `IVF_PQ` often provides higher accuracy with similar performance compared to `IVF_RQ`. | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) |
If your vector search frequently includes metadata filters (`where(...)`), prefer `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance.
Compression ratios are practical rules of thumb and can vary with vector distribution, metric, and configuration.
For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher compression than `IVF_RQ`.
### Index Tuning
Start with these values, then tune for your workload:
* HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`)
* `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer)
* Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger.
* `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing.
* `IVF_RQ`
* `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets.
* `IVF_PQ`
* `num_partitions`: start at `num_rows // 4096` (rounded to an integer).
* `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes.
* For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance.
## Example: Construct an IVF Index
In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF\_PQ with L2 distance, which is well-suited for high-dimensional vector search.
Make sure you have enough data in your table (at least a few thousand rows) for effective index training.
### Index Configuration
Sometimes you need to configure the index beyond default parameters:
* Index Types:
* `IVF_HNSW_FLAT`: highest recall, with no vector quantization
* `IVF_HNSW_SQ`: best recall/latency trade-off
* `IVF_RQ`: best compression for large, high-dimensional datasets
* `IVF_PQ`: often higher accuracy than `IVF_RQ` for small dimensions (`<= 256`) at similar query performance
* `metrics`: default is `l2`, other available are `cosine` or `dot`
* When using `cosine` similarity, distances range from 0 (identical vectors) to 2 (maximally dissimilar)
* `num_partitions`: use index-specific starting points from the section above:
* HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576`
* `IVF_RQ` and `IVF_PQ`: `num_rows // 4096`
* `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition
count from a target number of rows per partition. If you set both `num_partitions` and
`target_partition_size`, `num_partitions` takes precedence.
* `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search.
Let's take a look at a sample request for an IVF index:
### 1. Setup
Connect to LanceDB and open the table you want to index.
### 2. Construct an IVF Index
Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target.
#### Indexing nested vector fields
If your vector column lives inside a struct, pass its full dotted path as `vector_column_name`. The same path is used at query time and is what `list_indices()` reports under `columns`:
Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes.
### Async API and Config Objects
With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings:
Use these Python config classes for the index types shown on this page:
| Index type | Python config class |
| :-------------- | :------------------ |
| `IVF_FLAT` | `IvfFlat` |
| `IVF_PQ` | `IvfPq` |
| `IVF_RQ` | `IvfRq` |
| `IVF_SQ` | `IvfSq` |
| `IVF_HNSW_FLAT` | `IvfHnswFlat` |
| `IVF_HNSW_PQ` | `IvfHnswPq` |
| `IVF_HNSW_SQ` | `IvfHnswSq` |
### 3. Query the IVF Index
Search using a random 1,536-dimensional embedding.
#### Search Configuration
Core knobs available on a vector search call:
| Parameter | Description |
| :---------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `limit` | Number of results to return (`k`). |
| `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB auto-tunes this by default. |
| `minimum_nprobes` | Partitions that are *always* scanned. Higher values raise recall at the cost of latency. |
| `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. |
| `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. |
| `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. |
**Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count.
Recommended `nprobes` behavior by index type:
| Index type | Guidance |
| :----------------------------- | :------------------------------------------------------------------------------------------------------------------- |
| `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. |
| `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. |
| `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. |
#### Advanced Search Controls
These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints.
| Method | Description |
| :----------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. |
| `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall\@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). |
**Thresholding with `distance_range`:**
**Measuring recall with `bypass_vector_index`:**
Compare ANN results against a flat-scan ground truth to compute recall\@k. This is the standard way to pick `nprobes` for your workload.
Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall measurements or small tables, not production queries.
Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules.
## Example: Construct an HNSW Index
### Index Configuration
There are four key parameters to set when constructing an HNSW index:
* `index_type`: choose `IVF_HNSW_SQ` for a strong recall/latency/size trade-off, or `IVF_HNSW_FLAT` when you want the IVF+HNSW structure without vector quantization.
* `metric`: The default is `l2` euclidean distance metric. Other available are `dot` and `cosine`.
* `m`: The number of neighbors to select for each vector in the HNSW graph.
* `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph.
### 1. Construct an HNSW Index
The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`.
### 2. Query the HNSW Index
## Example: Construct a Binary Vector Index
Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits.
### Index Configuration
* Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space.
* Index Type: `IVF_FLAT` is used for indexing binary vectors
* `metric`: the `hamming` distance is used for similarity search
* The dimension of binary vectors must be a multiple of 8. For example, a 128-dimensional vector is stored as a uint8 array of size 16.
**`IVF_FLAT` + `hamming` is the only supported path for binary vectors.**
* `hamming` distance is only valid on packed binary (uint8) data; it is rejected on float vector columns.
* Quantized index types (`IVF_PQ`, `IVF_RQ`, `IVF_SQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) do not accept binary inputs — their `distance_type` is restricted to `l2`, `cosine`, or `dot`.
### 1. Create Table and Schema
### 2. Generate and Add Data
### 3. Construct the Binary Index
### 4. Vector Search
## Check Index Status
Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**.
In the LanceDB Enterprise UI, navigate to your table page - the "Index" column reflects each column's index status: it is blank when no index exists, shows an "in progress" label while the index is being built, and shows the index type once the build completes.
Programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built.
To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_index()` or call `wait_for_index()` on the table.
Each entry returned by `list_indices()` also carries detailed per-index metadata, so you can inspect an index without a follow-up `index_stats()` call. Node.js exposes the same fields in camelCase (`num_indexed_rows` → `numIndexedRows`):
| Field | What it tells you |
| :--------------------------------------- | :------------------------------------------------------------------------- |
| `num_indexed_rows`, `num_unindexed_rows` | Index coverage over the table |
| `size_bytes` | Total size of the index files on disk |
| `num_segments`, `index_version` | On-disk layout and format version |
| `created_at` | Creation time (ms since the Unix epoch in Node.js) |
| `index_uuid`, `type_url` | Internal identifiers for the index segment |
| `index_details` | Type-specific details (e.g. IVF partition counts or quantization settings) |
These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them.
## Custom Index Names
The `{column}_idx` suffix is a default convention, not the only supported naming path. Pass `name=...` to `create_index()` to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side `IVF_PQ` and `IVF_HNSW_SQ` builds) or when you script index replacement by name. Once set, `list_indices()`, `index_stats(name)`, and `wait_for_index([name])` all reference the custom name.
# Agno
Source: https://docs.lancedb.com/integrations/ai/agno
Build a search assistant using the Agno agent framework with LanceDB as the knowledge backend.
[Agno](https://docs.agno.com/introduction) is a framework for building agentic AI applications.
It supports LanceDB as a knowledge backend, allowing you to easily ingest and retrieve external content for your agents.
When you pair Agno's `Knowledge` system with LanceDB, you get a clean Agentic RAG setup.
We'll walk through the steps below to build a YouTube transcript-aware Agno assistant that can:
* Ingest a transcript from a YouTube video via the YouTube API
* Store embeddings and metadata in LanceDB
* Retrieve context during responses with hybrid search
* Ask questions about the video content in a CLI chat loop
## Prerequisites
Install dependencies:
```bash pip icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install -U agno openai lancedb youtube-transcript-api beautifulsoup4
```
```bash uv icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv add agno openai lancedb youtube-transcript-api beautifulsoup4
```
## Step 1: Configure LanceDB-backed knowledge
First, you can initialize the core `Knowledge` object that your agent will use for retrieval.
It configures LanceDB as the vector store, enables hybrid search with native LanceDB FTS, and sets the embedding model.
## Step 2: Fetch and ingest the YouTube transcript
Next, extract a YouTube video ID, fetch the full transcript, and flatten it into text for indexing.
The snippet shown below then inserts that transcript text into the Agno knowledge base, which writes vectors and metadata to LanceDB.
This path explicitly fetches the transcript first, then inserts transcript text into LanceDB through Agno.
## Step 3: Create an Agno agent with knowledge search
The next step is to construct an Agno `Agent` and attach the knowledge base you just populated.
With `search_knowledge=True`, the agent performs retrieval before answering, so responses stay grounded in transcript context.
In Agno, retrieval is exposed as a tool call that the model can invoke at runtime.
When `search_knowledge=True`, Agno makes a knowledge-search tool (shown in output as `search_knowledge_base(...)`) available to the model; the model decides when to call it, Agno executes the tool, and the returned context is fed back into the final answer.
## Step 4: Start a CLI chat loop
You can now ask an initial question and then start an interactive loop for follow-up queries.
Each prompt runs through the same retrieval pipeline, so you can iteratively inspect what the transcript contains.
Want local-first inference? Replace OpenAI model/embedder classes with Agno's Ollama providers. See Agno's Ollama knowledge examples: [docs.agno.com/examples/models/ollama/chat/knowledge](https://docs.agno.com/examples/models/ollama/chat/knowledge).
### Question 1
The following question is asked in the CLI chat loop:
```
┏━ Message ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃
┃ Q: What kinds of data can LanceDB handle? ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
┏━ Tool Calls ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃
┃ • search_knowledge_base(query=What kinds of data can LanceDB handle?) ┃
┃ • search_knowledge_base(query=LanceDB images audio video handle kinds of data ┃
┃ can handle 'LanceDB can handle' 'kinds of data' 'images audio video' transcript) ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
┏━ Response (19.1s) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃
┃ ┃
┃ • Images, audio, video — i.e., multimodal AI data and “all manners of things ┃
┃ you don't put into traditional databases” (per the transcript). ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
We get the response based on the transcript's contents as expected.
### Question 2
Let's ask a more specific question about the CEO of LanceDB, which is also in the transcript:
```
You: What is the name of the CEO of LanceDB?
INFO Found 10 documents
┏━ Message ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃
┃ What is the name of the CEO of LanceDB? ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
┏━ Tool Calls ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃
┃ • search_knowledge_base(query=CEO of LanceDB) ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
┏━ Response (16.7s) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃
┃ ┃
┃ • According to the retrieved YouTube transcript/title, the CEO of LanceDB is ┃
┃ Chang She. ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
We get the response based on the transcript's contents and title as expected.
## Why this works well
To start, LanceDB OSS can run from a local directory, so transcript data can stay on your machine when you are using the OSS stack.
* You do not need to maintain a separate transcript parser in your application code.
* You do not need to hand-roll chunking and retrieval orchestration across multiple modules.
* One explicit Agno `Knowledge` object, backed by LanceDB, defines both ingestion and search behavior in one place.
* Fewer moving parts means the tutorial stays readable and the same pattern is easier to carry into production code.
As your application needs grow, you can migrate to LanceDB [Enterprise](/enterprise) for
convenience features like automatic compaction and reindexing and the ability to scale to
really large datasets.
# GenKit
Source: https://docs.lancedb.com/integrations/ai/genkit
### genkitx-lancedb
Genkit is an open-source framework for building end-to-end AI and RAG pipelines with a clean, TypeScript-first
developer experience. The genkitx-lancedb plugin lets you use LanceDB as a high-performance vector store
inside your Genkit flows, so you can index, search, and retrieve data efficiently as part of your AI
applications.
### Installation
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pnpm install genkitx-lancedb
```
### Usage
Adding LanceDB plugin to your genkit instance.
You can run this app with the following command:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
genkit start -- tsx --watch src/index.ts
```
This'll add LanceDB as a retriever and indexer to the genkit instance. You can see it in the GUI view
![Screenshot 2025-05-11 at 7 21 05 PM]()
**Testing retrieval on a sample table**
Let's see the raw retrieval results
![Screenshot 2025-05-11 at 7 21 05 PM]()
On running this query, you'll get 5 results fetched from the lancedb table, where each result looks something like this:
![Screenshot 2025-05-11 at 7 21 18 PM]()
## Creating a custom RAG flow
Now that we've seen how you can use LanceDB in a Genkit pipeline, let's refine the flow and create a RAG. A RAG flow will consist of an index and a retriever with its outputs postprocessed and fed into an LLM for final response
### Creating custom indexer flows
You can also create custom indexer flows, utilizing more options and features provided by LanceDB.
![Screenshot 2025-05-11 at 8 35 56 PM]()
In your console, you can see the logs
![Screenshot 2025-05-11 at 7 19 14 PM]()
### Creating custom retriever flows
You can also create custom retriever flows, utilizing more options and features provided by LanceDB.
Now using our retrieval flow, we can ask a question about the ingested PDF
![Screenshot 2025-05-11 at 7 18 45 PM]()
# Hermes Agent
Source: https://docs.lancedb.com/integrations/ai/hermes-agent
Use LanceDB as a persistent, semantic memory backend for Hermes Agent. Get durable recall across sessions with vector and hybrid search.
[Hermes Agent](https://github.com/NousResearch/hermes-agent) is a self-hosted, open-source
personal agent from [Nous Research](https://nousresearch.com). You can talk to it from a
terminal UI or reach the same agent from Telegram, Discord, and Slack, and it exposes a
dedicated slot for external *memory providers* that run alongside its built-in notes.
The [LanceDB memory plugin](https://github.com/lancedb/hermes-agent-memory) fills that slot.
It gives Hermes durable, semantic recall across sessions: state a preference or a project
convention once, and the agent can retrieve it weeks later in a brand-new session — even when
you ask for it in completely different words. Everything runs inside Hermes' own Python
process, storing a single LanceDB table on local disk. There's no memory server to operate.
**The mental model is clean**
* Hermes owns the agent loop
* LanceDB manages the durable long-term memory and offers semantic recall.
## Why LanceDB fits agent memory
Out of the box, Hermes remembers with a small curated notes file frozen into the system
prompt, plus lexical (keyword) search over past sessions. Both are useful, but keyword search
misses paraphrases of what you originally typed — the exact thing you need when recalling a
fact you phrased differently months ago.
LanceDB is an embedded retrieval library, which makes it a natural fit here:
* **No server to stand up** — it reads and writes a table on local disk, so the plugin ships
as a dependency rather than a service to operate.
* **One table holds everything** — content, metadata, and embeddings live together. A memory
becomes a structured row with a category, tags, timestamps, and provenance, not just a text
blob.
* **Query it any way you need** — vector similarity for meaning, BM25 full-text for exact
names and jargon, a hybrid of the two, or plain metadata filters to keep recall scoped to
the right workspace.
* **It scales up** — the same table abstraction carries over to larger LanceDB deployments
later, so the local setup is never a dead end.
## Install and activate
Want to try this without touching your existing Hermes setup? Run everything in an isolated
profile: `hermes profile create demo`, then add `-p demo` to the commands below. When you're
done, `rm -rf ~/.hermes/profiles/demo` removes all trace.
Skip this if you already have Hermes installed.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
```
This shallow-clones the plugin into `~/.hermes/plugins/lancedb/`.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hermes plugins install lancedb/hermes-agent-memory
```
Hermes loads plugins inside its own Python interpreter, so the dependencies go *there* — not
into a separate virtualenv. (This interpreter is shared across profiles, so you only install
once.)
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv pip install --python ~/.hermes/hermes-agent/venv/bin/python3 lancedb openai pyyaml
```
The plugin turns conversations into embeddings, so it needs an embeddings key. By default that
is OpenAI, so set `OPENAI_API_KEY` in your environment or in `~/.hermes/.env`.
Prefer a local or non-OpenAI model? The plugin uses an OpenAI-compatible client, so you can
point it at any compatible endpoint (OpenRouter, Ollama, vLLM, …) in your config — no code
change needed. See [Configuration](#configuration) below.
Switch memory on and pick this plugin:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hermes memory setup # choose "lancedb"
```
Then confirm it's actually active before you start chatting — this is the one step worth not
skipping, because Hermes quietly falls back to its built-in notes if the provider isn't set:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hermes memory status
```
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
Memory status
────────────────────────────────────────
Built-in: always active
Provider: lancedb
Plugin: installed ✓
Status: available ✓
```
You want to see `Provider: lancedb` with both `installed ✓` and `available ✓`.
## The memory tools
Once activated, the agent has four tools for working with long-term memory:
| Tool | What it does |
| :----------------- | :------------------------------------------------------------------------------------------------------------------------------------- |
| `lancedb_recall` | Semantic (vector, the default) or hybrid search over your workspace memory. Returns matching facts with scores and provenance. |
| `lancedb_remember` | Stores a durable fact when you explicitly ask. Deduplicated by content hash, so remembering the same thing twice doesn't pile up rows. |
| `lancedb_read` | Fetches a single memory by ID, optionally with the original conversation messages it was distilled from. |
| `lancedb_forget` | Deletes safely: previews candidates first, then deletes by exact ID, so nothing disappears by accident. |
Beyond these tools, the plugin also captures durable facts from your conversations
automatically — an auxiliary model distills them before context is compressed and again when a
session ends, so insights survive even when the raw messages are summarized away.
## Walkthrough
"*Teach it your project preferences*"
Let's make this concrete with the pain we opened on: re-explaining your setup to the agent every session.
We'll save a convention once and then prove a brand-new session can recall it. This example will touch all four
tools along the way.
### Remember
Ask Hermes to commit a convention to long-term memory. Saying "remember in long-term memory"
makes sure it lands in the LanceDB store, which shows up as the `⚡ lancedb_r` (`lancedb_remember`)
line below:
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
● Remember in long-term memory: for this project I only use uv, never pip, and I always add type hints to Python functions.
┊ 🧠 memory +memory: "For this project, the user only uses uv for Python package management, never pip, and always adds type hints to Python functions."
┊ ⚡ lancedb_r 0.0s
─ ⚕ Hermes ────────────────────────────────────────────────────────────────
Remembered. I've stored that project convention: use uv only, never pip, and always add type hints to Python functions.
```
### Recall
First, take Hermes' built-in notes out of the picture so recall can *only* come from LanceDB —
the two layers run side by side otherwise, and either could answer:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hermes config set memory.memory_enabled false
```
Now start a fresh session (`/new`) and ask for the convention back, worded completely
differently from how you saved it:
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
● /new
● Look through your long-term memory for my Python project preferences.
─ ⚕ Hermes ────────────────────────────────────────────────────────────────
I found one durable memory about Python project preferences:
- Project Python tooling preference is uv; pip is not used; type hints are required.
```
This is the whole point in one exchange. You asked about your *preferences* using none of the
words you stored the fact with — no *uv*, no *pip*, no *type hints* — yet a fresh session, with
none of the earlier conversation in context, pulls it straight back. Semantic search matches on
meaning rather than exact wording, which is precisely where keyword search falls short.
Turn the built-in layer back on for everyday use with `hermes config set memory.memory_enabled true`
— the two memory layers happily coexist. We only disabled it here to prove the recall came from
LanceDB.
### Read
You can also ask where a fact came from. Hermes attributes the answer to its stored memory
rather than guessing from a file in the repo (under the hood, `lancedb_read` can also return
the exact source messages a fact was distilled from):
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
● Where did that uv preference come from?
─ ⚕ Hermes ────────────────────────────────────────────────────────────────
It came from the project's persistent memory, not from a codebase file I inspected.
Specifically, the recalled memory says:
- "For this project, the user only uses uv for Python package management, never pip, and always adds type hints to Python functions."
```
### Forget
When a preference changes, ask Hermes to drop the old fact. The tool calls tell the whole
story: the two `⚡ lancedb_f` (`lancedb_forget`) lines are it previewing matches and then
deleting, and the trailing `⚡ lancedb_r` is it saving the replacement in the same breath:
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
● I've switched from uv to pixi. Forget the uv preference.
┊ ⚡ lancedb_f uv preference python package management never pip type hints 0.3s
┊ ⚡ lancedb_f 0.0s
┊ ⚡ lancedb_r 0.2s
─ ⚕ Hermes ────────────────────────────────────────────────────────────────
Done — I removed the uv preference from memory and replaced it with:
- This project uses pixi for Python package management
- Type hints on Python functions are still preferred
```
Remember, recall, read, forget: four small operations that between them cover the entire
lifecycle of a durable memory.
## Retrieval modes
Recall ships in `vector` mode by default — pure semantic search, which is what survives the
paraphrasing you saw above. If you also need exact name or jargon matching, switch to `hybrid`
(vector + BM25) and choose how the two legs are fused: RRF, a vector-biased linear blend, or a
cross-encoder reranker. Mode is set per call; fusion is a config setting.
```yaml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# ~/.hermes/config.yaml
plugins:
lancedb:
retrieval:
mode: hybrid # vector (default) | hybrid
reranker:
type: rrf # how the vector + BM25 legs are fused
# Swap RRF for a reranking pass (pulls in sentence-transformers + torch):
# type: cross-encoder
# model: cross-encoder/ettin-reranker-17m-v1
# rerank_top_n: 50
```
The cross-encoder is the one path that pulls in a local ML stack, so it stays opt-in. It
defaults to the compact 17M-parameter [ettin reranker](https://huggingface.co/cross-encoder/ettin-reranker-17m-v1).
## Inspect the store
Everything lives in one table named `memories` at `~/.hermes/lancedb/memories.lance`. Because
it's a plain LanceDB table, you can open it directly and see exactly what the agent has stored
— a `kind` column separates extracted `fact` rows from the raw `turn` rows they were drawn
from:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("~/.hermes/lancedb")
tbl = db.open_table("memories")
print(tbl.to_pandas()[["kind", "category", "content"]].head())
```
## Configuration
The plugin runs on sensible defaults once activated — you don't have to configure anything.
`~/.hermes/config.yaml` is purely for overrides. Two common ones:
Use a cheaper model for the auxiliary fact-extraction calls:
```yaml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# ~/.hermes/config.yaml
auxiliary:
lancedb_extraction:
provider: openrouter
model: google/gemini-3-flash
```
Point embeddings at a fully local endpoint (for example, Ollama) so nothing leaves your
machine:
```yaml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# ~/.hermes/config.yaml
plugins:
lancedb:
embedding:
model: nomic-embed-text
base_url: http://localhost:11434/v1
api_key_env: OLLAMA_API_KEY # any value works for local Ollama
```
Changing the embedding model (or its dimension) against an existing store requires recreating
the table — the plugin fails loudly on a dimension mismatch rather than silently returning
nothing. Every option is documented in the plugin's [`default_config.yaml`](https://github.com/lancedb/hermes-agent-memory/blob/main/src/default_config.yaml).
## Benchmark
On [LongMemEval-S](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned), a
long-conversation QA benchmark, LanceDB's semantic recall clearly beat Hermes' built-in lexical
search (0.66 vs. 0.53 answer accuracy) by finding the right messages even when the question was
worded differently from the original conversation. For the full methodology, the
per-question-type breakdown, and a reproducible harness, see the
[blog post](https://www.lancedb.com/blog/semantic-memory-for-hermes-agent-with-lancedb) and the
[benchmark harness](https://github.com/lancedb/hermes-agent-memory/tree/main/benchmarks).
## Why this works well
* **It's local-first and embedded.** The LanceDB memory table lives on your disk with no server to run;
the plugin installs as a dependency of Hermes' own environment.
* **Recall survives paraphrasing.** Semantic search matches meaning, not spelling, which is the
failure mode that sinks keyword-only session search.
* **Memories are structured and traceable.** Each fact is a row with metadata and a link back
to the messages it came from, and `forget` always previews before it deletes.
* **Nothing about it is a dead end.** As your needs grow, the same table abstraction carries
over to LanceDB [Enterprise](/enterprise) for automatic compaction, reindexing, and scale.
To try it, install the plugin, enable it with `hermes memory setup`, and run the kind of
workflow we walked through above.
# Hugging Face Hub
Source: https://docs.lancedb.com/integrations/ai/huggingface
Use LanceDB directly on Lance datasets hosted on the Hugging Face Hub for multimodal search and retrieval.
[Hugging Face Hub](https://huggingface.co/datasets?format=format:lance\&sort=trending) is a popular platform for sharing machine learning datasets, models, and other resources.
LanceDB can directly scan Lance datasets hosted on the [Hugging Face Hub](https://huggingface.co/datasets?format=format:lance) with `hf://` URIs.
This is enabled under the hood by the [lance-huggingface](https://lance.org/integrations/huggingface/)
integration that allows users to stream Lance datasets directly from Hugging Face without needing to
download them first.
For ML and AI engineers working in LanceDB, this capability is incredibly useful for quickly exploring
multimodal datasets and reusing Lance datasets shared by others, without writing custom data loaders
or preprocessing pipelines.
The snippets below use the [`lance-format/laion-1m`](https://huggingface.co/datasets/lance-format/laion-1m)
dataset published in Lance format. The dataset includes a million image-caption pairs, and the
Lance dataset can package image embeddings alongside the metadata. This makes it useful for
demonstrating LanceDB's multimodal search capabilities in combination with easy sharing via the
Hugging Face Hub.
The LAION table includes multimodal columns such as:
* `image` (inline JPEG bytes)
* `caption` (text)
* `img_emb` (image embedding vector)
* metadata fields such as `url` and `similarity`
## Install dependencies
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install lancedb pillow
```
## Open the dataset with LanceDB
LanceDB can open the dataset directly from the Hub, without needing to download it first.
Note that in LanceDB, you need to specify the table name when opening a Lance dataset,
and the Hugging Face convention is to use `train` and `test` splits for datasets.
The LAION dataset is uploaded as a single split named `train`, so we specify the table name
that contains the `*.lance` files when opening the dataset.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
table = db.open_table("train")
print(f"Opened table: {table.name}")
print(f"Rows: {len(table)}")
```
## Inspect schema and available indexes
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
print(table.schema)
```
This prints the schema of the LAION table. Note that there's an image embedding column that's
a fixed-size list of 768-dimensional floats, and a binary column containing the raw JPEG bytes of the image.
```
image_path: string
caption: string
NSFW: string
similarity: double
LICENSE: string
url: string
key: string
status: string
error_message: null
width: int64
height: int64
original_width: int64
original_height: int64
exif: string
md5: string
img_emb: fixed_size_list[768]
child 0, item: float
image: binary
```
When inspecting Lance datasets from Hugging Face, it's also a good idea to check whether the dataset author included
any pre-built indexes that you can use for search. You can check the available indexes with:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
print(table.list_indices())
```
```
[
Index(IvfPq, columns=["img_emb"], name="img_emb_idx"),
Index(FTS, columns=["caption"], name="caption_idx")
]
```
In this case, we see that we have an IVF\_PQ vector index on the `img_emb` column, and an FTS index on the `caption`
column, which means we can directly do vector search on the image embeddings and keyword search on the captions
without needing to build the indexes ourselves!
If you see an empty list, it may be because the dataset author did not include the index files when uploading
to Hugging Face. You can download the dataset locally, and build the indexes yourself. See the [indexing guide](/indexing/)
for instructions on building different types of indexes with LanceDB.
## Projection scan
Run a simple scan by projecting relevant columns to get a feel for the dataset. For example, we
can run a search without any filters or input parameters to get a small subset of the data:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
rows = (
table.search()
.select(["caption", "url", "similarity"])
.limit(3)
.to_list()
)
for i, row in enumerate(rows, start=1):
print(f"{i}. {row['caption']}")
print(f" url={row['url']}")
print(f" similarity={row['similarity']}")
```
We get the first three rows and their metadata printed out, which look like this:
```
1. Cordelia and Dudley on their wedding day last year
url=https://i.dailymail.co.uk/i/pix/2012/01/05/article-2082728-0EF8956600000578-53_233x315.jpg
similarity=0.2926466464996338
2. Statistics on challenges for automation in 2021
url=https://verloop.io/wp-content/uploads/2021/02/Challenges.jpg
similarity=0.30174341797828674
3. Teacher Gifts / Great gifts for your child's teacher. Don't know what to get? Take a look at these gifts that the teacher in your life will love!
url=https://i.pinimg.com/custom_covers/216x146/550494823141083777_1487893945.jpg
similarity=0.3362061381340027
```
## Scan and filter data
Filtered search is a common pattern to narrow down interesting subsets of the data during early
exploration. Here's an example:
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
filtered = (
table.search()
.where("height > 600")
.select(["caption", "url", "width", "height"])
.limit(3)
.to_list()
)
for row in filtered:
print(row["caption"], row["url"], row["width"], row["height"])
```
This prints out the metadata for large images with height greater than 600 pixels:
```
Luca Trousers, mustard stripe https://cdn.shopify.com/s/files/1/0151/5333/products/IMG_0791_1024x1024.jpg?v=1585142190 384 766
Baby Blue Fitted Short Sleeve T Shirt 3 https://cdn-img.prettylittlething.com/a/d/d/1/add198cab3ec30a61102437275573f4963642528_cmf6022_3.jpg 384 612
pattern cutting made easy pdf https://i.pinimg.com/736x/7c/6c/a7/7c6ca7361815a8929b3dd6ad34a03ab9.jpg 384 1045
```
## Export image bytes to local files
To work with a subset of the data locally, you can export the image bytes from the table and save them as JPEG files.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from pathlib import Path
sample = (
table.search()
.select(["image", "caption"])
.limit(3)
.to_list()
)
out_dir = Path("samples")
out_dir.mkdir(exist_ok=True)
for i, row in enumerate(sample):
out_path = out_dir / f"laion_{i}.jpg"
with open(out_path, "wb") as f:
f.write(row["image"])
print(f"Saved {out_path} | caption={row['caption']}")
```
You can now preview the images you just exported on your local machine to get a better sense of the data.
## Vector search
You can use LanceDB to run vector search directly on the data on the Hub, **without needing to download the dataset
or build your own vector index**. This makes it incredibly easy to explore the dataset and iterate on your search queries
before you decide to download a local copy for further experimentation on your end.
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Pick an arbitrary image embedding from the dataset
query_embedding = (
table.search()
.select(["img_emb"])
.limit(1)
.to_list()[0]["img_emb"]
)
results = (
table.search(query_embedding, vector_column_name="img_emb")
.select(["caption", "url", "_distance"])
.limit(3)
.to_list()
)
for row in results:
print(row["_distance"], row["caption"])
```
| distance | caption |
| ------------------- | --------------------------------------------------- |
| 0.17765313386917114 | Cordelia and Dudley on their wedding day last year |
| 0.17765313386917114 | Cordelia and Dudley on their wedding day last year |
| 0.17765313386917114 | Cordelia and Dudley on their wedding day last year |
Note that the LAION dataset is known to contain a lot of duplicate images, so you may see the same image
showing up multiple times in the search results.
## Full-text search
Run an FTS search query that uses BM25 ranking on the `caption` column (on which we already have an FTS index):
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
fts_results = (
table.search("dog running on beach", query_type="fts")
.select(["caption", "url", "_score"])
.limit(3)
.to_list()
)
```
| caption | url | \_score |
| ------------------------------- | ------------------------------------------------------------------ | --------- |
| running with dog | [https://www.doggytastic.com/wp…](https://www.doggytastic.com/wp…) | 15.73168 |
| Dog Running in Water | [https://static.wixstatic.com/m…](https://static.wixstatic.com/m…) | 14.756516 |
| Dogs on the run by heidiannemo… | [http://ih2.redbubble.net/image…](http://ih2.redbubble.net/image…) | 14.756516 |
## Download the full dataset
You may hit Hugging Face rate limits when streaming large samples from `hf://`, despite using a Hugging Face token.
For repeated queries or queries that operate on the full dataset, it's recommended to
download the dataset locally and query from disk.
Here's how to download the entire dataset via the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli):
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
huggingface-cli download lance-format/laion-1m --repo-type dataset --local-dir ./laion-1m
```
## Upload your own datasets to Hugging Face in Lance format
This section shows how you can upload your own Lance datasets to the Hugging Face Hub to share with the community.
First, install the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli) and export both `OPENAI_API_KEY` and `HF_TOKEN`.
Then, create a Lance dataset using LanceDB on a local machine, and then proceed to upload it to the Hub via a CLI command.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
export OPENAI_API_KEY=...
export HF_TOKEN=hf_...
hf auth login --token "$HF_TOKEN"
```
A typical sequence of steps is given below.
### 1. Upload your local directory to the Hub
Upload the full local directory to a specified repository on the Hugging Face Hub. The command below uploads the contents of your local LanceDB directory at `/path/to/your_local_dir` to a new repository named `your_hf_org/repo_name` under your Hugging Face account.
```bash bash icon="code" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hf upload-large-folder /path/to/your_local_dir your_hf_org/repo_name \
--repo-type dataset \
--revision main
```
The `upload-large-folder` command is designed for [uploading large datasets](https://huggingface.co/docs/huggingface_hub/en/guides/upload) (potentially terabytes in size) and will handle multipart uploads, retries, and resuming interrupted uploads.
### 2. Inspect dataset versions
Because you can query your remote dataset directly from Hugging Face with `hf://` URIs in LanceDB, you can easily inspect the dataset versions and updates on the Hub without needing to download the data locally. This is very useful to keep track of changes to the dataset and iterate on your data collection and curation process.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("hf://datasets/your_hf_org/repo_name")
table = db.open_table("table_name")
versions = table.list_versions()
print(versions)
```
This will print out the list of versions available for the dataset on the Hub, along with their metadata such as creation date and description.
### 3. Add a dataset card
The Hub dataset card allows you to communicate the schema and usage of the dataset to other developers. It sits at the repo's root in a file named `README.md` on the Hub.
This project keeps the source card text in `HF_DATASET_CARD.md`, so you can publish updates
to the dataset there and upload it as `README.md` using the following command on the HF CLI:
this requires a regular `hf upload` because it is a single-file upload to a specific target path (a custom commit message can be added if you wish).
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hf upload lancedb/magical_kingdom HF_DATASET_CARD.md README.md \
--repo-type dataset \
--commit-message "Update dataset card"
```
### 4. Update the dataset
Over time, you may want to add new rows (append) or columns (backfill) to your dataset as your needs evolve. You can make the necessary updates to your local dataset using LanceDB, and then upload the updated version back to the Hub with the same `hf upload-large-folder` command.
```bash bash icon="code" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
hf upload-large-folder /path/to/your_local_dir your_hf_org/repo_name \
--repo-type dataset \
--revision main
```
The CLI will only upload the new data that has changed since the last upload, avoiding wasted I/O while making it easy to keep your dataset up-to-date on the Hub.
That's it! Your dataset is now updated on the Hub with the new data and schema changes, and other users can query the latest version of the dataset directly from Hugging Face with `hf://` URIs in LanceDB.
## Explore more Lance datasets on Hugging Face
The LanceDB team is actively uploading useful and interesting datasets in Lance format to the Hugging Face Hub
under the [lance-format](https://huggingface.co/lance-format) organization. We actively encourage the Hugging Face
and LanceDB communities to upload their own Lance datasets to the Hub to share with others!
In the meantime, feel free to check out the Hugging Face Hub to discover more Lance datasets uploaded by the community.
Click here to explore the latest trending Lance datasets on 🤗 Hugging Face!
# Kiln AI
Source: https://docs.lancedb.com/integrations/ai/kiln
[**Kiln**](https://kiln.tech) is a free tool for building production-ready AI systems, combining an intuitive desktop application and an open-source Python library. It supports RAG pipelines, evaluations, agents, MCP tool-calling, synthetic data generation, and fine-tuning. Kiln provides deep integration with LanceDB for vector search, full-text search (BM25), and hybrid search.
## Quick Start: Build a RAG Pipeline in 5 Minutes with Kiln & LanceDB
Watch the [quick start overview on Vimeo](https://vimeo.com/1119945690).
Kiln's [app](https://kiln.tech/download) makes it easy to:
* Build a RAG pipeline with a simple drag-and-drop interface
* [Compare](#find-the-best-rag-pipeline-for-your-use-case) search index options (powered by LanceDB), document extractors, embedding models, and chunking strategies
* Create end-to-end [evaluations](https://docs.kiln.tech/docs/evaluations) to determine which search configuration works best for your use case
* Load your data from Kiln into [LanceDB Enterprise](/enterprise) for production use
* Iterate with confidence by evaluating new content, prompts, models, and embeddings in minutes instead of weeks
## Find the Best RAG Pipeline for Your Use Case
There is no universal best RAG solution—only the best solution for your specific use case. Kiln makes it easy to compare state-of-the-art configurations and find which works best for you.
Start with pre-configured templates for state-of-the-art RAG at various performance/quality/cost levels, or experiment with any combination of options:
| Area | Technologies | Description |
| :------------------ | :---------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------- |
| Search Index | LanceDB | Compare LanceDB's vector search, full-text search (BM25), and hybrid search to find the best approach for your use case. |
| Content | Kiln Document Library | Collaborate on a document library with your team to find the best content for your RAG. Track every revision and tag document sets. |
| Document Extraction | Gemini, OpenAI GPT, Qwen VL, and more | Find the most accurate document extraction models for converting PDFs, images, audio, video, and other formats into textual data for RAG. |
| Embeddings | Embedding models from Gemini, OpenAI, Nomic, Qwen, and more | Find the embedding model best suited to your use case. |
| Chunking | LlamaIndex | Find the ideal chunk size and method. |
## Get Started
To get started, download the [Kiln App](https://kiln.tech/download), create a project, and navigate to "Docs & Search".
See the [Kiln documentation for creating a RAG system](https://docs.kiln.tech/docs/documents-and-search-rag) for details on each step of the process.
## More Information
* [Kiln Homepage](https://kiln.tech)
* [Download the Kiln App](https://kiln.tech/download)
* [Kiln GitHub Repository](https://github.com/Kiln-AI/Kiln)
* [Building RAG Systems - Kiln Documentation](https://docs.kiln.tech/docs/documents-and-search-rag)
* [Python Library](https://pypi.org/project/kiln-ai/) or `pip install kiln_ai`
# LangChain
Source: https://docs.lancedb.com/integrations/ai/langchain
**LangChain** is a framework designed for building applications with large language models (LLMs) by chaining together various components. It supports a range of functionalities including memory, agents, and chat models, enabling developers to create context-aware applications.
LangChain streamlines these stages (in figure above) by providing pre-built components and tools for integration, memory management, and deployment, allowing developers to focus on application logic rather than underlying complexities.
Integration of **Langchain** with **LanceDB** enables applications to retrieve the most relevant data by comparing query vectors against stored vectors, facilitating effective information retrieval. It results in better and context aware replies and actions by the LLMs.
## Quick Start
You can load your document data using langchain's loaders, for this example we are using `TextLoader` and `OpenAIEmbeddings` as the embedding model.
## Documentation
In the above example `LanceDB` vector store class object is created using `from_documents()` method which is a `classmethod` and returns the initialized class object.
You can also use `LanceDB.from_texts(texts: List[str],embedding: Embeddings)` class method.
The exhaustive list of parameters for `LanceDB` vector store are :
| Name | type | Purpose | default |
| :------------------- | :------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------ |
| `connection` | (Optional) `Any` | `lancedb.db.LanceDBConnection` connection object to use. If not provided, a new connection will be created. | `None` |
| `embedding` | (Optional) `Embeddings` | Langchain embedding model. | Provided by user. |
| `uri` | (Optional) `str` | It specifies the directory location of **LanceDB database** and establishes a connection that can be used to interact with the database. | `/tmp/lancedb` |
| `vector_key` | (Optional) `str` | Column name to use for vector's in the table. | `'vector'` |
| `id_key` | (Optional) `str` | Column name to use for id's in the table. | `'id'` |
| `text_key` | (Optional) `str` | Column name to use for text in the table. | `'text'` |
| `table_name` | (Optional) `str` | Name of your table in the database. | `'vectorstore'` |
| `api_key` | (Optional `str`) | API key to use for LanceDB Enterprise deployment. | `None` |
| `region` | (Optional) `str` | Region to use for LanceDB Enterprise deployment. | Only for LanceDB Enterprise : `None`. |
| `mode` | (Optional) `str` | Mode to use for adding data to the table. Valid values are "append" and "overwrite". | `'overwrite'` |
| `table` | (Optional) `Any` | You can connect to an existing table of LanceDB, created outside of langchain, and utilize it. | `None` |
| `distance` | (Optional) `str` | The choice of distance metric used to calculate the similarity between vectors. | `'l2'` |
| `reranker` | (Optional) `Any` | The reranker to use for LanceDB. | `None` |
| `relevance_score_fn` | (Optional) `Callable[[float], float]` | Langchain relevance score function to be used. | `None` |
| `limit` | `int` | Set the maximum number of results to return. | `DEFAULT_K` (it is 4) |
### Methods
##### `add_texts()`
This method turn texts into embedding and add it to the database.
| Name | Purpose | defaults |
| :---------- | :-------------------------------------------------------------- | :--------------- |
| `texts` | `Iterable` of strings to add to the vectorstore. | Provided by user |
| `metadatas` | Optional `list[dict()]` of metadatas associated with the texts. | `None` |
| `ids` | Optional `list` of ids to associate with the texts. | `None` |
| `kwargs` | Other keyworded arguments provided by the user. | - |
It returns list of ids of the added texts.
***
##### create\_index()
This method creates a scalar(for non-vector cols) or a vector index on a table.
| Name | type | Purpose | defaults |
| :----------------- | :-------------- | :------------------------------------------------------------------------------------ | :------- |
| `vector_col` | `Optional[str]` | Provide if you want to create index on a vector column. | `None` |
| `col_name` | `Optional[str]` | Provide if you want to create index on a non-vector column. | `None` |
| `metric` | `Optional[str]` | Provide the metric to use for vector index. choice of metrics: 'l2', 'dot', 'cosine'. | `l2` |
| `num_partitions` | `Optional[int]` | Number of partitions to use for the index. | `256` |
| `num_sub_vectors` | `Optional[int]` | Number of sub-vectors to use for the index. | `96` |
| `index_cache_size` | `Optional[int]` | Size of the index cache. | `None` |
| `name` | `Optional[str]` | Name of the table to create index on. | `None` |
For index creation make sure your table has enough data in it. An ANN index is usually not needed for datasets \~100K vectors. For large-scale (>1M) or higher dimension vectors, it is beneficial to create an ANN index.
***
##### similarity\_search()
This method performs similarity search based on **text query**.
| Name | Type | Purpose | Default |
| -------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `query` | `str` | A `str` representing the text query that you want to search for in the vector store. | N/A |
| `k` | `Optional[int]` | It specifies the number of documents to return. | `None` |
| `filter` | `Optional[Dict[str, str]]` | It is used to filter the search results by specific metadata criteria. | `None` |
| `fts` | `Optional[bool]` | It indicates whether to perform a full-text search (FTS). | `False` |
| `name` | `Optional[str]` | It is used for specifying the name of the table to query. If not provided, it uses the default table set during the initialization of the LanceDB instance. | `None` |
| `kwargs` | `Any` | Other keyworded arguments provided by the user. | N/A |
Return documents most similar to the query **without relevance scores**.
***
##### similarity\_search\_by\_vector()
The method returns documents that are most similar to the specified **embedding (query) vector**.
| Name | Type | Purpose | Default |
| ----------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `embedding` | `List[float]` | The embedding vector you want to use to search for similar documents in the vector store. | N/A |
| `k` | `Optional[int]` | It specifies the number of documents to return. | `None` |
| `filter` | `Optional[Dict[str, str]]` | It is used to filter the search results by specific metadata criteria. | `None` |
| `name` | `Optional[str]` | It is used for specifying the name of the table to query. If not provided, it uses the default table set during the initialization of the LanceDB instance. | `None` |
| `kwargs` | `Any` | Other keyworded arguments provided by the user. | N/A |
**It does not provide relevance scores.**
***
##### similarity\_search\_with\_score()
Returns documents most similar to the **query string** along with their relevance scores.
| Name | Type | Purpose | Default |
| -------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `query` | `str` | A `str` representing the text query you want to search for in the vector store. This query will be converted into an embedding using the specified embedding function. | N/A |
| `k` | `Optional[int]` | It specifies the number of documents to return. | `None` |
| `filter` | `Optional[Dict[str, str]]` | It is used to filter the search results by specific metadata criteria. This allows you to narrow down the search results based on certain metadata attributes associated with the documents. | `None` |
| `kwargs` | `Any` | Other keyworded arguments provided by the user. | N/A |
It gets called by base class's `similarity_search_with_relevance_scores` which selects relevance score based on our `_select_relevance_score_fn`.
***
##### similarity\_search\_by\_vector\_with\_relevance\_scores()
Similarity search using **query vector**.
| Name | Type | Purpose | Default |
| ----------- | -------------------------- | ----------------------------------------------------------------------------------------- | ------- |
| `embedding` | `List[float]` | The embedding vector you want to use to search for similar documents in the vector store. | N/A |
| `k` | `Optional[int]` | It specifies the number of documents to return. | `None` |
| `filter` | `Optional[Dict[str, str]]` | It is used to filter the search results by specific metadata criteria. | `None` |
| `name` | `Optional[str]` | It is used for specifying the name of the table to query. | `None` |
| `kwargs` | `Any` | Other keyworded arguments provided by the user. | N/A |
The method returns documents most similar to the specified embedding (query) vector, along with their relevance scores.
***
##### max\_marginal\_relevance\_search()
This method returns docs selected using the maximal marginal relevance(MMR).
Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents.
| Name | Type | Purpose | Default |
| ------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
| `query` | `str` | Text to look up documents similar to. | N/A |
| `k` | `Optional[int]` | Number of Documents to return. | `4` |
| `fetch_k` | `Optional[int]` | Number of Documents to fetch to pass to MMR algorithm. | `None` |
| `lambda_mult` | `float` | Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. | `0.5` |
| `filter` | `Optional[Dict[str, str]]` | Filter by metadata. | `None` |
| `kwargs` | Other keyworded arguments provided by the user. | - | |
Similarly, `max_marginal_relevance_search_by_vector()` function returns docs most similar to the embedding passed to the function using MMR. instead of a string query you need to pass the embedding to be searched for.
***
##### add\_images()
This method adds images by automatically creating their embeddings and adds them to the vectorstore.
| Name | Type | Purpose | Default |
| ----------- | ---------------------- | -------------------------- | ------- |
| `uris` | `List[str]` | File path to the image | N/A |
| `metadatas` | `Optional[List[dict]]` | Optional list of metadatas | `None` |
| `ids` | `Optional[List[str]]` | Optional list of IDs | `None` |
It returns list of IDs of the added images.
# LlamaIndex
Source: https://docs.lancedb.com/integrations/ai/llamaIndex
## Quickstart
LlamaIndex is a well-known framework for building LLM-powered agents over your data with LLMs and workflows.
You can build your LlamaIndex pipeline and persist your metadata and embeddings in LanceDB via the `LanceDBVectorStore` class.
First, install the LlamaIndex-LanceDB integration.
pip install llama-index-vector-stores-LanceDB
Run the below script as an example.
The vector store connector will open an existing LanceDB directory or create the directory if it does not exist.
### Filtering
For metadata filtering, you can use a Lance SQL-like string filter as demonstrated in the example above. Additionally, you can also filter using the `MetadataFilters` class from LlamaIndex:
### Hybrid Search
For complete documentation, refer [here](https://docs.lancedb.com/search/hybrid-search). This example uses the `colbert` reranker. Make sure to install necessary dependencies for the reranker you choose.
In the snippet above, you can change/specify `query_type` when creating the engine/retriever
to use different search strategies, such as vector search or FTS.
## API reference
See the official LlamaIndex Vector Stores API reference for more details.
# PromptTools
Source: https://docs.lancedb.com/integrations/ai/prompttools
[PromptTools](https://github.com/hegelai/prompttools) offers a set of free, open-source tools for testing and experimenting with models, prompts, and configurations. The core idea is to enable developers to evaluate prompts using familiar interfaces like code and notebooks. You can use it to experiment with different configurations of LanceDB, and test how LanceDB integrates with the LLM of your choice.
![Open In Colab]()
# Meta Llama Synthetic Data Kit
Source: https://docs.lancedb.com/integrations/ai/synthetic-data-kit
Use Meta Llama's Synthetic Data Kit with LanceDB to generate high-quality synthetic datasets for LLM fine-tuning and training.
[Synthetic Data Kit](https://github.com/meta-llama/synthetic-data-kit) is a tool from Meta LLAMA that helps you generate high-quality synthetic datasets for fine-tuning large language models (LLMs). It simplifies the process of preparing data for fine-tuning by providing a command-line interface (CLI) with a modular four-command flow.
One of the key features of the `synthetic-data-kit` is its use of the Lance format for storing and ingesting datasets. This allows for efficient storage and retrieval of data, which is crucial when working with large datasets.
### Key Features:
* **Data Ingestion:** The toolkit can ingest various file formats, including PDF, HTML, YouTube transcripts, DOCX, PPT, and TXT.
* **Fine-tuning Format Creation:** It can create different fine-tuning formats, such as question-answer (QA) pairs, QA pairs with Chain-of-Thought (CoT), and summarization formats.
* **Data Curation:** The tool uses Llama as a judge to curate high-quality examples, ensuring the quality of the generated dataset.
* **Flexible Saving Options:** You can save the generated datasets in various formats compatible with your fine-tuning workflow, including Hugging Face, JSONL, and JSON.
### How it Works:
The synthetic-data-kit follows a simple four-step process:
1. **Ingest:** Import your input files into the toolkit. The data is stored in the Lance format for efficient processing.
2. **Create:** Generate diverse fine-tuning datasets, such as reasoning, summarization, and QA pairs, from the ingested documents.
3. **Curate:** Use Llama to filter and select high-quality examples from the generated dataset.
4. **Save-as:** Export the curated dataset in your preferred format.
### Usage
The `synthetic-data-kit` uses Lance format to store and manage the data that you ingest. The workflow is a series of commands that build on each other, starting with the `ingest` command.
Here is an example of the end-to-end workflow:
1. **Ingest Data into a LanceDB dataset**
This command takes a directory of source files and creates a LanceDB dataset from them.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
synthetic-data-kit ingest docs/report.pdf --multimodal
# This will create a Lance dataset at data/parsed/report.lance
# with 'text' and 'image' columns.
#Generate multimodal-qa pairs from the ingested data
synthetic-data-kit create data/parsed/report.lance --type multimodal-qa
```
2. **Create fine-tuning data**
This command uses the LanceDB dataset created in the previous step to generate synthetic data in the desired format.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
synthetic_data create data/parsed/report.lance
```
3. **Curate the data**
This step uses a language model to curate the generated data and ensure its quality.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
synthetic_data curate report.json
```
4. **Save the final dataset**
Finally, save the curated data to a file in the desired format.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
synthetic_data save-as report.json --save_path ./my_finetuning_data.jsonl
```
This workflow allows you to go from a collection of documents to a high-quality, fine-tuning dataset with just a few commands. The use of LanceDB in the background makes the process efficient and scalable.
### Getting Started:
To get started with the synthetic-data-kit, you can clone the [GitHub Repository](https://github.com/meta-llama/synthetic-data-kit) and install the necessary dependencies.
> **Note:** You will also need access to a Llama model, either running locally or via a hosted API.
# dlt
Source: https://docs.lancedb.com/integrations/data/dlt
[dlt](https://dlthub.com/docs/intro) is an open-source library that you can add to your Python scripts to load data from various and often messy data sources into well-structured, live datasets. dlt's [integration with LanceDB](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb) lets you ingest data from any source (databases, APIs, CSVs, dataframes, JSONs, and more) into LanceDB with a few lines of simple python code. The integration enables automatic normalization of nested data, schema inference, incremental loading and embedding the data. dlt also has integrations with several other tools like dbt, airflow, dagster etc. that can be inserted into your LanceDB workflow.
## How to ingest data into LanceDB
In this example, we will be fetching movie information from the [Open Movie Database (OMDb) API](https://www.omdbapi.com/) and loading it into a local LanceDB instance. To implement it, you will need an API key for the OMDb API (which can be created freely [here](https://www.omdbapi.com/apikey.aspx)).
1. **Install `dlt` with LanceDB extras:**
```sh theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install dlt[lancedb]
```
2. **Inside an empty directory, initialize a `dlt` project with:**
```sh theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
dlt init rest_api lancedb
```
This will add all the files necessary to create a `dlt` pipeline that can ingest data from any REST API (ex: OMDb API) and load into LanceDB.
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
├── .dlt
│ ├── config.toml
│ └── secrets.toml
├── rest_api
├── rest_api_pipeline.py
└── requirements.txt
```
dlt has a list of pre-built [sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) like [SQL databases](https://dlthub.com/docs/dlt-ecosystem/verified-sources/sql_database), [REST APIs](https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api), [Google Sheets](https://dlthub.com/docs/dlt-ecosystem/verified-sources/google_sheets), [Notion](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion) etc., that can be used out-of-the-box by running `dlt init lancedb`. Since dlt is a python library, it is also very easy to modify these pre-built sources or to write your own custom source from scratch.
3. **Specify necessary credentials and/or embedding model details:**
In order to fetch data from the OMDb API, you will need to pass a valid API key into your pipeline. Depending on whether you're using LanceDB OSS or LanceDB Enterprise, you also may need to provide the necessary credentials to connect to the LanceDB instance. These can be pasted inside `.dlt/secrets.toml`.
dlt's LanceDB integration also allows you to automatically embed the data during ingestion. Depending on the embedding model chosen, you may need to paste the necessary credentials inside `.dlt/secrets.toml`:
```toml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
[sources.rest_api]
api_key = "api_key" # Enter the API key for the OMDb API
[destination.lancedb]
embedding_model_provider = "sentence-transformers"
embedding_model = "all-MiniLM-L6-v2"
[destination.lancedb.credentials]
uri = ".lancedb"
api_key = "api_key" # API key to connect to LanceDB Enterprise. Leave out if you are using LanceDB OSS.
embedding_model_provider_api_key = "embedding_model_provider_api_key" # Not needed for providers that don't need authentication (ollama, sentence-transformers).
```
See [here](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb#configure-the-destination) for more information and for a list of available models and model providers.
4. **Write the pipeline code inside `rest_api_pipeline.py`:**
The following code shows how you can configure dlt's REST API source to connect to the [OMDb API](https://www.omdbapi.com/), fetch all movies with the word "godzilla" in the title, and load it into a LanceDB table. The REST API source allows you to pull data from any API with minimal code, to learn more read the [dlt docs](https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api).
The script above will ingest the data into LanceDB as it is, i.e. without creating any embeddings. If we want to embed one of the fields (for example, `"Title"` that contains the movie titles), then we will use dlt's `lancedb_adapter` and modify the script as follows:
* Add the following import statement:
* Modify the pipeline run like this:
This will use the embedding model specified inside `.dlt/secrets.toml` to embed the field `"Title"`.
5. **Install necessary dependencies:**
```sh theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install -r requirements.txt
```
Note: You may need to install the dependencies for your embedding models separately.
```sh theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install sentence-transformers
```
6. **Run the pipeline:**
Finally, running the following command will ingest the data into your LanceDB instance.
```sh theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
python custom_source.py
```
For more information and advanced usage of dlt's LanceDB integration, read [the dlt documentation](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb).
# DuckDB
Source: https://docs.lancedb.com/integrations/data/duckdb
Learn how to use the DuckDB-Lance extension to query Lance tables with SQL.
LanceDB integrates with [DuckDB](https://duckdb.org/) through the [Lance extension](https://github.com/lance-format/lance-duckdb) for DuckDB. In this page, we'll show how LanceDB manages table lifecycle, and DuckDB provides SQL analytics (including joins) and search over those tables.
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
## Install
Install the DuckDB CLI as per [their docs](https://duckdb.org/install) and alternatively, their Python package with `pip install duckdb`.
Then, open the DuckDB CLI and install and load the Lance extension as follows:
```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
INSTALL lance;
LOAD lance;
```
## Attach the directory namespace in DuckDB
Attach the LanceDB root directory as a Lance namespace:
```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
ATTACH './local_lancedb' AS lance_ns (TYPE LANCE);
```
In this page, tables are referenced using `lance_ns.main.`, so the table path is `lance_ns.main.lance_duck`.
## Write Lance table
Create the `lance_duck` table using SQL and populate it with sample data:
```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
CREATE OR REPLACE TABLE lance_ns.main.lance_duck AS
SELECT *
FROM (
VALUES
('duck', 'quack', [0.9, 0.7, 0.1]::FLOAT[]),
('horse', 'neigh', [0.3, 0.1, 0.5]::FLOAT[]),
('dragon', 'roar', [0.5, 0.2, 0.7]::FLOAT[])
) AS t(animal, noise, vector);
```
This table is the source of truth for all DuckDB queries below.
The examples below show SQL entered in the DuckDB CLI. You can run the same SQL from
Python as well, using LanceDB and DuckDB's Python clients in your application code.
## Query the table with SQL
```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT *
FROM lance_ns.main.lance_duck
LIMIT 5;
```
## Vector search
```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT animal, noise, vector, _distance
FROM lance_vector_search(
'lance_ns.main.lance_duck',
'vector',
[0.8, 0.7, 0.2]::FLOAT[],
k = 1,
prefilter = true
)
ORDER BY _distance ASC;
```
## Full-text search
```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT animal, noise, vector, _score
FROM lance_fts(
'lance_ns.main.lance_duck',
'animal',
'the brave knight faced the dragon',
k = 1,
prefilter = true
)
ORDER BY _score DESC;
```
## Hybrid search
```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT animal, noise, vector, _hybrid_score, _distance, _score
FROM lance_hybrid_search(
'lance_ns.main.lance_duck',
'vector',
[0.8, 0.7, 0.2]::FLOAT[],
'animal',
'the duck surprised the dragon',
k = 2,
prefilter = false,
alpha = 0.5,
oversample_factor = 4
)
ORDER BY _hybrid_score DESC;
```
## Directory namespace model
A directory namespace maps a LanceDB catalog root to namespace-qualified table identifiers in DuckDB. This keeps table discovery and table naming stable as your project grows.
To learn more about the catalog and namespace model, see [Namespaces and the Catalog Model](/namespaces).
## Advanced usage
See the [docs](https://github.com/lance-format/lance-duckdb) directory in the Lance-DuckDB extension repo
for more advanced usage on SQL and REST API clients.
# Pandas and PyArrow
Source: https://docs.lancedb.com/integrations/data/pandas_and_pyarrow
Because Lance is built on top of [Apache Arrow](https://arrow.apache.org/),
LanceDB fits naturally into Pandas-first workflows. You can ingest a `DataFrame`,
query it with LanceDB's vector operators, and keep working in Pandas without any glue code.
## Create a dataset
Start by importing LanceDB alongside your usual Pandas utilities and connect to a temporary database.
Use the familiar `pd.DataFrame` API to prepare your rows, then pass the entire frame to `db.create_table`.
## Vector search
Queries can return Pandas frames as well, so you can immediately inspect the results or pipe them into downstream analytics.
## Async API
For web services or background jobs that already rely on `asyncio`, use the asynchronous helpers to keep everything non-blocking.
# Polars
Source: https://docs.lancedb.com/integrations/data/polars_arrow
LanceDB supports [Polars](https://github.com/pola-rs/polars), a blazingly fast DataFrame library for Python written in Rust. Under the hood, both Lance and Polars speak Arrow, so passing data back and forth stays zero-copy and ergonomic.
## Create & Query a Table
Import the required libraries, including the optional Pydantic helpers if you plan to define schemas.
Build a Polars `DataFrame`, convert it to Arrow, and use it directly when creating a LanceDB table.
Run vector search and keep the results as a Polars `DataFrame` for further processing or visualization.
## Work with LazyFrames
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once.
## Define Schemas with Pydantic
You can also describe your table via `LanceModel` and continue ingesting data from Polars. This is useful when multiple teams share a schema or when you want validation.
# Pydantic
Source: https://docs.lancedb.com/integrations/data/pydantic
[Pydantic](https://docs.pydantic.dev/latest/) is a data validation library in Python.
LanceDB integrates with Pydantic for schema inference, data ingestion, and query result casting.
Using `lancedb.pydantic.LanceModel`, users can seamlessly
integrate Pydantic with the rest of the LanceDB APIs.
First, import the necessary LanceDB and Pydantic modules:
Next, define your Pydantic model by inheriting from `LanceModel` and specifying your fields including a vector field:
Set the database connection URL:
Now you can create a table, add data, and perform vector search operations:
## Vector Field
LanceDB provides a `lancedb.pydantic.Vector` method to define a
vector Field in a Pydantic Model.
This example demonstrates how LanceDB automatically converts Pydantic field types to their corresponding Apache Arrow data types. The `pydantic_to_schema()` function takes a Pydantic model and generates an Arrow schema where:
* `int` fields become `pa.int64()` (64-bit integers)
* `str` fields become `pa.utf8()` (UTF-8 encoded strings)
* `Vector(768)` becomes `pa.list_(pa.float32(), 768)` (fixed-size list of 768 float32 values)
* The `False` parameter indicates that the fields are not nullable
## Type Conversion
LanceDB automatically convert Pydantic fields to
[Apache Arrow DataType](https://arrow.apache.org/docs/python/generated/pyarrow.DataType.html#pyarrow.DataType).
Current supported type conversions:
| Pydantic Field Type | PyArrow Data Type |
| ------------------- | ----------------------------------- |
| `int` | `pyarrow.int64` |
| `float` | `pyarrow.float64` |
| `bool` | `pyarrow.bool` |
| `str` | `pyarrow.utf8()` |
| `list` | `pyarrow.List` |
| `BaseModel` | `pyarrow.Struct` |
| `Vector(n)` | `pyarrow.FixedSizeList(float32, n)` |
LanceDB supports to create Apache Arrow Schema from a
`pydantic.BaseModel`
via `lancedb.pydantic.pydantic_to_schema` method.
This example shows a more complex Pydantic model with various field types and demonstrates how LanceDB handles:
* Basic types: `int` and `str` fields
* Vector fields: `Vector(1536)` creates a fixed-size list of 1536 float32 values
* List fields: `List[int]` becomes a variable-length list of int64 values
* Schema generation: The `pydantic_to_schema()` function automatically converts all these types to their Arrow equivalents
# Voxel51
Source: https://docs.lancedb.com/integrations/data/voxel51
# FiftyOne
[FiftyOne](https://docs.voxel51.com/) is an open source toolkit that enables users to curate better data and build better models. It includes tools for data exploration, visualization, and management, as well as features for collaboration and sharing.
Any developers, data scientists, and researchers who work with computer vision and machine learning can use FiftyOne to improve the quality of their datasets and deliver insights about their models.
![example]()
**FiftyOne** provides an API to create LanceDB tables and run similarity queries, both **programmatically in Python** and via **point-and-click in the App**.
Let's get started and see how to use **LanceDB** to create a **similarity index** on your FiftyOne datasets.
## Overview
[Embeddings](/embedding/) are foundational to all of the **vector search** features. In FiftyOne, embeddings are managed by the [**FiftyOne Brain**](https://docs.voxel51.com/user_guide/brain.html) that provides powerful machine learning techniques designed to transform how you curate your data from an art into a measurable science.
> *Have you ever wanted to find the images most similar to an image in your dataset?*
The **FiftyOne Brain** makes computing **visual similarity** really easy. You can compute the similarity of samples in your dataset using an embedding model and store the results in the **brain key**.
You can then sort your samples by similarity or use this information to find potential duplicate images.
We'll be doing the following :
1. **Create Index** - In order to run similarity queries against our media, we need to **index** the data. We can do this via the `compute_similarity()` function.
* In the function, specify the **model** you want to use to generate the embedding vectors, and what **vector search engine** you want to use on the **backend** (here LanceDB).
You can also give the similarity index a name(`brain_key`), which is useful if you want to run vector searches against multiple indexes.
2. **Query** - Once you have generated your similarity index, you can query your dataset with `sort_by_similarity()`. The query can be any of the following:
* An ID (sample or patch)
* A query vector of same dimension as the index
* A list of IDs (samples or patches)
* A text prompt (search semantically)
## Prerequisites: install necessary dependencies
1. **Create and activate a virtual environment**
Install virtualenv package and run the following command in your project directory.
python -m venv fiftyone\_
From inside the project directory run the following to activate the virtual environment.
source fiftyone\_/Scripts/activate
fiftyone\_/Scripts/activate
2. **Install the following packages in the virtual environment**
To install FiftyOne, ensure you have activated any virtual environment that you are using, then run
pip install fiftyone
## Understand basic workflow
The basic workflow shown below uses LanceDB to create a similarity index on your FiftyOne datasets:
1. Load a dataset into FiftyOne.
2. Compute embedding vectors for samples or patches in your dataset, or select a model to use to generate embeddings.
3. Use the `compute_similarity()` method to generate a LanceDB table for the samples or object patches embeddings in a dataset by setting the parameter `backend="lancedb"` and specifying a `brain_key` of your choice.
4. Use this LanceDB table to query your data with `sort_by_similarity()`.
5. If desired, delete the table.
## Quick Example
Let's jump on a quick example that demonstrates this workflow.
Make sure you install torch ([guide here](https://pytorch.org/get-started/locally/)) before proceeding.
!!! note
Running the code above will download the clip model (2.6Gb)
Once the similarity index has been generated, we can query our data in FiftyOne by specifying the `brain_key`:
The returned result are of type - `DatasetView`.
`DatasetView` does not hold its contents in-memory. Views simply store the rule(s) that are applied to extract the content of interest from the underlying Dataset when the view is iterated/aggregated on.
This means, for example, that the contents of a `DatasetView` may change as the underlying Dataset is modified.
> *Can you query a view instead of dataset?*
Yes, you can also query a view.
Performing a similarity search on a `DatasetView` will only return results from the view; if the view contains samples that were not included in the index, they will never be included in the result.
This means that you can index an entire Dataset once and then perform searches on subsets of the dataset by constructing views that contain the images of interest.
## Using LanceDB backend
By default, calling `compute_similarity()` or `sort_by_similarity()` will use an sklearn backend.
To use the LanceDB backend, simply set the optional `backend` parameter of `compute_similarity()` to `"lancedb"`:
Alternatively, you can configure FiftyOne to use the LanceDB backend by setting the following environment variable.
In your terminal, set the environment variable using:
export FIFTYONE\_BRAIN\_DEFAULT\_SIMILARITY\_BACKEND=lancedb
\$Env:FIFTYONE\_BRAIN\_DEFAULT\_SIMILARITY\_BACKEND="lancedb" //powershell
set FIFTYONE\_BRAIN\_DEFAULT\_SIMILARITY\_BACKEND=lancedb //cmd
This will only run during the terminal session. Once terminal is closed, environment variable is deleted.
Alternatively, you can **permanently** configure FiftyOne to use the LanceDB backend creating a `brain_config.json` at `~/.fiftyone/brain_config.json`. The JSON file may contain any desired subset of config fields that you wish to customize.
```json theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
{
"default_similarity_backend": "lancedb"
}
```
This will override the default `brain_config` and will set it according to your customization. You can check the configuration by running the following code :
## LanceDB config parameters
The LanceDB backend supports query parameters that can be used to customize your similarity queries. These parameters include:
| Name | Purpose | Default |
| :-------------- | :--------------------------------------------------------------------------------------------------------------- | :--------------- |
| **table\_name** | The name of the LanceDB table to use. If none is provided, a new table will be created | `None` |
| **metric** | The embedding distance metric to use when creating a new table. The supported values are ("cosine", "euclidean") | `"cosine"` |
| **uri** | The database URI to use. In this Database URI, tables will be created. | `"/tmp/lancedb"` |
There are two ways to specify/customize the parameters:
1. **Using `brain_config.json` file**
```json theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
{
"similarity_backends": {
"lancedb": {
"table_name": "your-table",
"metric": "euclidean",
"uri": "/tmp/lancedb"
}
}
}
```
2. **Directly passing to `compute_similarity()` to configure a specific new index** :
For a much more in depth walkthrough of the integration, visit the LanceDB x Voxel51 [docs page](https://docs.voxel51.com/integrations/lancedb.html).
# AWS Bedrock
Source: https://docs.lancedb.com/integrations/embedding/aws
AWS Bedrock supports multiple base models for generating text embeddings. You need to setup the AWS credentials to use this embedding function.
You can do so by using `awscli` and also add your session\_token:
```shell theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
aws configure
aws configure set aws_session_token ""
```
to ensure that the credentials are set up correctly, you can run the following command:
```shell theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
aws sts get-caller-identity
```
Supported Embedding modelIDs are:
* `amazon.titan-embed-text-v1`
* `cohere.embed-english-v3`
* `cohere.embed-multilingual-v3`
Supported parameters (to be passed in `create` method) are:
| Parameter | Type | Default Value | Description |
| ----------------------- | ---- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **name** | str | "amazon.titan-embed-text-v1" | The model ID of the bedrock model to use. Supported base models for Text Embeddings: amazon.titan-embed-text-v1, cohere.embed-english-v3, cohere.embed-multilingual-v3 |
| **region** | str | "us-east-1" | Optional name of the AWS Region in which the service should be called (e.g., "us-east-1"). |
| **profile\_name** | str | None | Optional name of the AWS profile to use for calling the Bedrock service. If not specified, the default profile will be used. |
| **assumed\_role** | str | None | Optional ARN of an AWS IAM role to assume for calling the Bedrock service. If not specified, the current active credentials will be used. |
| **role\_session\_name** | str | "lancedb-embeddings" | Optional name of the AWS IAM role session to use for calling the Bedrock service. If not specified, a "lancedb-embeddings" name will be used. |
| **runtime** | bool | True | Optional choice of getting different client to perform operations with the Amazon Bedrock service. |
| **max\_retries** | int | 7 | Optional number of retries to perform when a request fails. |
Usage Example:
# Cohere
Source: https://docs.lancedb.com/integrations/embedding/cohere
Using cohere API requires cohere package, which can be installed using `pip install cohere`. Cohere embeddings are used to generate embeddings for text data. The embeddings can be used for various tasks like semantic search, clustering, and classification.
You also need to set the `COHERE_API_KEY` environment variable to use the Cohere API.
Supported models are:
* embed-english-v3.0
* embed-multilingual-v3.0
* embed-english-light-v3.0
* embed-multilingual-light-v3.0
* embed-english-v2.0
* embed-english-light-v2.0
* embed-multilingual-v2.0
Supported parameters (to be passed in `create` method) are:
| Parameter | Type | Default Value | Description |
| ------------------- | ----- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `str` | `"embed-english-v2.0"` | The model ID of the cohere model to use. Supported base models for Text Embeddings: embed-english-v3.0, embed-multilingual-v3.0, embed-english-light-v3.0, embed-multilingual-light-v3.0, embed-english-v2.0, embed-english-light-v2.0, embed-multilingual-v2.0 |
| `source_input_type` | `str` | `"search_document"` | The type of input data to be used for the source column. |
| `query_input_type` | `str` | `"search_query"` | The type of input data to be used for the query. |
Cohere supports following input types:
| Input Type | Description |
| ----------------------- | -------------------------------------- |
| "`search_document`" | Used for embeddings stored in a vector |
| | database for search use-cases. |
| "`search_query`" | Used for embeddings of search queries |
| | run against a vector DB |
| "`semantic_similarity`" | Specifies the given text will be used |
| | for Semantic Textual Similarity (STS) |
| "`classification`" | Used for embeddings passed through a |
| | text classifier. |
| "`clustering`" | Used for the embeddings run through a |
| | clustering algorithm |
Usage Example:
# ColPali
Source: https://docs.lancedb.com/integrations/embedding/colpali
We support [ColPali](https://github.com/illuin-tech/colpali) model embeddings for multimodal multi-vector retrieval. ColPali produces multiple embedding vectors per input (multi-vector), enabling more nuanced similarity matching between text queries and image documents.
Using ColPali requires the colpali-engine package, which can be installed using `pip install colpali-engine`.
ColPali produces **multi-vector** embeddings, meaning each input generates multiple embedding vectors rather than a single vector. Use `MultiVector(func.ndims())` instead of `Vector(func.ndims())` when defining your schema.
Supported models are:
* Metric-AI/ColQwen2.5-3b-multilingual-v1.0 (default)
* vidore/colpali-v1.3
* vidore/colqwen2-v1.0
* vidore/colSmol-256M
Supported parameters (to be passed in `create` method) are:
| Parameter | Type | Default Value | Description |
| --------------------- | ------------------------------ | --------------------------------------------- | ------------------------------------------------------------------------- |
| `model_name` | `str` | `"Metric-AI/ColQwen2.5-3b-multilingual-v1.0"` | The name of the model to use. |
| `device` | `str` | `"auto"` | The device for inference. Can be `"auto"`, `"cpu"`, `"cuda"`, or `"mps"`. |
| `dtype` | `str` | `"bfloat16"` | Data type for model weights (bfloat16, float16, float32, float64). |
| `pooling_strategy` | `str` | `"hierarchical"` | Token pooling strategy: `"hierarchical"`, `"lambda"`, or `None`. |
| `pool_factor` | `int` | `2` | Factor to reduce sequence length when pooling is enabled. |
| `batch_size` | `int` | `2` | Batch size for processing inputs. |
| `quantization_config` | `Optional[BitsAndBytesConfig]` | `None` | Quantization configuration for the model (requires bitsandbytes). |
This embedding function supports ingesting images as both bytes and URLs. You can query them using text.
Now we can search using text queries:
# Gemini
Source: https://docs.lancedb.com/integrations/embedding/gemini
With Google's Gemini, you can represent text (words, sentences, and blocks of text) in a vectorized form, making it easier to compare and contrast embeddings. For example, two texts that share a similar subject matter or sentiment should have similar embeddings, which can be identified through mathematical comparison techniques such as cosine similarity. For more on how and why you should use embeddings, refer to the Embeddings guide.
The Gemini Embedding Model API supports various task types:
| Task Type | Description |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "`retrieval_query`" | Specifies the given text is a query in a search/retrieval setting. |
| "`retrieval_document`" | Specifies the given text is a document in a search/retrieval setting. Using this task type requires a title but is automatically provided by Embeddings API |
| "`semantic_similarity`" | Specifies the given text will be used for Semantic Textual Similarity (STS). |
| "`classification`" | Specifies that the embeddings will be used for classification. |
| "`clustering`" | Specifies that the embeddings will be used for clustering. |
Usage Example:
# Hugging Face
Source: https://docs.lancedb.com/integrations/embedding/huggingface
We offer support for all Hugging Face models (which can be loaded via [transformers](https://huggingface.co/docs/transformers/en/index) library). The default model is `colbert-ir/colbertv2.0` which also has its own special callout - `registry.get("colbert")`. Some Hugging Face models might require custom models defined on the HuggingFace Hub in their own modeling files. You may enable this by setting `trust_remote_code=True`. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
Example usage:
# IBM watsonx
Source: https://docs.lancedb.com/integrations/embedding/ibm
Generate text embeddings using IBM's watsonx.ai platform.
## Supported Models
You can find a list of supported models at [IBM watsonx.ai Documentation](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx). The currently supported model names are:
| Model ID | Dimensions |
| ----------------------------------------- | ---------- |
| `ibm/granite-embedding-278m-multilingual` | 768 |
| `ibm/slate-125m-english-rtrvr-v2` | 768 |
| `ibm/slate-30m-english-rtrvr-v2` | 384 |
| `intfloat/multilingual-e5-large` | 1024 |
| `sentence-transformers/all-minilm-l6-v2` | 384 |
For new tables, `ibm/granite-embedding-278m-multilingual` is the recommended default. Older model IDs (such as `ibm/slate-125m-english-rtrvr` and `sentence-transformers/all-minilm-l12-v2`) remain resolvable for tables whose stored metadata references them, but they are no longer advertised for new use.
## Parameters
The following parameters can be passed to the `create` method:
| Parameter | Type | Default Value | Description |
| ----------- | ---- | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name | str | `"ibm/slate-125m-english-rtrvr"` | The model ID of the watsonx.ai model to use. Pass one of the current supported IDs above (e.g. `"ibm/granite-embedding-278m-multilingual"`) when creating new tables. |
| api\_key | str | None | Optional IBM Cloud API key (or set `WATSONX_API_KEY`) |
| project\_id | str | None | Optional watsonx project ID (or set `WATSONX_PROJECT_ID`). Mutually exclusive with `space_id`. |
| space\_id | str | None | Optional watsonx deployment space ID (or set `WATSONX_SPACE_ID`). Mutually exclusive with `project_id`. |
| url | str | None | Optional custom URL for the watsonx.ai instance |
| params | dict | None | Optional additional parameters for the embedding model (e.g. `{"truncate_input_tokens": 512}`) |
You must supply exactly one of `project_id` or `space_id` (either as an argument or via its environment variable). Setting both, or neither, raises a `ValueError`.
## Usage Example
First, the watsonx.ai library is an optional dependency, so must be installed separately:
```
pip install ibm-watsonx-ai
```
Optionally set environment variables (if not passing credentials to `create` directly):
```sh theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
export WATSONX_API_KEY="YOUR_WATSONX_API_KEY"
# Provide exactly one of the following:
export WATSONX_PROJECT_ID="YOUR_WATSONX_PROJECT_ID"
export WATSONX_SPACE_ID="YOUR_WATSONX_SPACE_ID"
```
# ImageBind
Source: https://docs.lancedb.com/integrations/embedding/imagebind
We have support for [imagebind](https://github.com/facebookresearch/ImageBind) model embeddings. You can download our version of the packaged model via - `pip install imagebind-packaged==0.1.2`.
This function is registered as `imagebind` and supports Audio, Video and Text modalities(extending to Thermal,Depth,IMU data):
| Parameter | Type | Default Value | Description |
| ----------- | ------ | ------------------ | -------------------------------------------------------------- |
| `name` | `str` | `"imagebind_huge"` | Name of the model. |
| `device` | `str` | `"cpu"` | The device to run the model on. Can be `"cpu"` or `"gpu"`. |
| `normalize` | `bool` | `False` | set to `True` to normalize your inputs before model ingestion. |
Below is an example demonstrating how the API works:
Now, we can search using any modality:
#### image search
#### audio search
#### Text search
You can add any input query and fetch the result as follows:
If you have any questions about the embeddings API, supported models, or see a relevant model missing, please raise an issue [on GitHub](https://github.com/lancedb/lancedb/issues).
# Instructor
Source: https://docs.lancedb.com/integrations/embedding/instructor
[Instructor](https://instructor-embedding.github.io/) is an instruction-finetuned text embedding model that can generate text embeddings tailored to any task (e.g. classification, retrieval, clustering, text evaluation, etc.) and domains (e.g. science, finance, etc.) by simply providing the task instruction, without any finetuning.
If you want to calculate customized embeddings for specific sentences, you can follow the unified template to write instructions.
Represent the `domain` `text_type` for `task_objective`:
* `domain` is optional, and it specifies the domain of the text, e.g. science, finance, medicine, etc.
* `text_type` is required, and it specifies the encoding unit, e.g. sentence, document, paragraph, etc.
* `task_objective` is optional, and it specifies the objective of embedding, e.g. retrieve a document, classify the sentence, etc.
More information about the model can be found at the [source URL](https://github.com/xlang-ai/instructor-embedding).
| Argument | Type | Default | Description |
| ---------------------- | ------ | -------------------------------------------------------------------- | --------------------------------------------------------- |
| `name` | `str` | "hkunlp/instructor-base" | The name of the model to use |
| `batch_size` | `int` | `32` | The batch size to use when generating embeddings |
| `device` | `str` | `"cpu"` | The device to use when generating embeddings |
| `show_progress_bar` | `bool` | `True` | Whether to show a progress bar when generating embeddings |
| `normalize_embeddings` | `bool` | `True` | Whether to normalize the embeddings |
| `quantize` | `bool` | `False` | Whether to quantize the model |
| `source_instruction` | `str` | `"represent the document for retrieval"` | The instruction for the source column |
| `query_instruction` | `str` | `"represent the document for retrieving the most similar documents"` | The instruction for the query |
# Jina
Source: https://docs.lancedb.com/integrations/embedding/jina
## Text Embedding Models
Jina embeddings are used to generate embeddings for text and image data.
You also need to set the `JINA_API_KEY` environment variable to use the Jina API.
You can find a list of supported models under [https://jina.ai/embeddings/](https://jina.ai/embeddings/)
Supported parameters (to be passed in `create` method) are:
| Parameter | Type | Default Value | Description |
| --------- | ----- | ---------------- | ------------------------------------- |
| `name` | `str` | `"jina-clip-v1"` | The model ID of the jina model to use |
Usage Example:
## Multimodal Embedding Models
Jina embeddings can also be used to embed both text and image data, only some of the models support image data and you can check the list
under [https://jina.ai/embeddings/](https://jina.ai/embeddings/)
Supported parameters (to be passed in `create` method) are:
| Parameter | Type | Default Value | Description |
| --------- | ----- | ---------------- | ------------------------------------- |
| `name` | `str` | `"jina-clip-v1"` | The model ID of the jina model to use |
Usage Example:
# Ollama
Source: https://docs.lancedb.com/integrations/embedding/ollama
Generate embeddings via the [ollama](https://github.com/ollama/ollama-python) python library. More details:
* [Ollama docs on embeddings](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embeddings)
* [Ollama blog on embeddings](https://ollama.com/blog/embedding-models)
| Parameter | Type | Default Value | Description |
| ---------------------- | -------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------ |
| `name` | `str` | `nomic-embed-text` | The name of the model. |
| `host` | `str` | `http://localhost:11434` | The Ollama host to connect to. |
| `options` | `ollama.Options` or `dict` | `None` | Additional model parameters listed in the documentation for the Modelfile such as `temperature`. |
| `keep_alive` | `float` or `str` | `"5m"` | Controls how long the model will stay loaded into memory following the request. |
| `ollama_client_kwargs` | `dict` | `{}` | kwargs that can be past to the `ollama.Client`. |
# OpenAI
Source: https://docs.lancedb.com/integrations/embedding/openai
LanceDB registers the OpenAI embeddings function in the registry by default, as `openai`. Below are the parameters that you can customize when creating the instances:
| Parameter | Type | Default Value | Description |
| ----------- | ----- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `str` | `"text-embedding-ada-002"` | The name of the model. |
| `dim` | `int` | Model default | For OpenAI's newer text-embedding-3 model, we can specify a dimensionality that is smaller than the 1536 size. This feature supports it |
| `use_azure` | bool | `False` | Set true to use Azure OpenAI SDK |
# OpenCLIP
Source: https://docs.lancedb.com/integrations/embedding/openclip
We support CLIP model embeddings using the open source alternative, [open-clip](https://github.com/mlfoundations/open_clip) which supports various customizations. It is registered as `open-clip` and supports the following customizations:
| Parameter | Type | Default Value | Description |
| ------------ | ------ | --------------------- | ----------------------------------------------------------------------- |
| `name` | `str` | `"ViT-B-32"` | The name of the model. |
| `pretrained` | `str` | `"laion2b_s34b_b79k"` | The name of the pretrained model to load. |
| `device` | `str` | `"cpu"` | The device to run the model on. Can be `"cpu"` or `"gpu"`. |
| `batch_size` | `int` | `64` | The number of images to process in a batch. |
| `normalize` | `bool` | `True` | Whether to normalize the input images before feeding them to the model. |
This embedding function supports ingesting images as both bytes and urls. You can query them using both text and other images.
LanceDB supports ingesting images directly from accessible links.
Now we can search using text from both the default vector column and the custom vector column
Because we're using a multimodal embedding function, we can also search using images
# Sentence Transformers
Source: https://docs.lancedb.com/integrations/embedding/sentence-transformers
Allows you to set parameters when registering a `sentence-transformers` object.
Sentence transformer embeddings are normalized by default. It is recommended to use normalized embeddings for similarity search.
The `trust_remote_code` parameter defaults to `True`, which allows models to execute arbitrary code from their Hugging Face repository during loading. If you are loading untrusted models, set `trust_remote_code=False` to prevent remote code execution.
| Parameter | Type | Default Value | Description |
| ------------------- | ------ | ------------------ | -------------------------------------------------------------------------------- |
| `name` | `str` | `all-MiniLM-L6-v2` | The name of the model |
| `device` | `str` | `cpu` | The device to run the model on (can be `cpu` or `gpu`) |
| `normalize` | `bool` | `True` | Whether to normalize the input text before feeding it to the model |
| `trust_remote_code` | `bool` | `True` | Whether to trust and execute remote code from the model's Huggingface repository |
```markdown theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
- sentence-transformers/all-MiniLM-L12-v2
- sentence-transformers/paraphrase-mpnet-base-v2
- sentence-transformers/gtr-t5-base
- sentence-transformers/LaBSE
- sentence-transformers/all-MiniLM-L6-v2
- sentence-transformers/bert-base-nli-max-tokens
- sentence-transformers/bert-base-nli-mean-tokens
- sentence-transformers/bert-base-nli-stsb-mean-tokens
- sentence-transformers/bert-base-wikipedia-sections-mean-tokens
- sentence-transformers/bert-large-nli-cls-token
- sentence-transformers/bert-large-nli-max-tokens
- sentence-transformers/bert-large-nli-mean-tokens
- sentence-transformers/bert-large-nli-stsb-mean-tokens
- sentence-transformers/distilbert-base-nli-max-tokens
- sentence-transformers/distilbert-base-nli-mean-tokens
- sentence-transformers/distilbert-base-nli-stsb-mean-tokens
- sentence-transformers/distilroberta-base-msmarco-v1
- sentence-transformers/distilroberta-base-msmarco-v2
- sentence-transformers/nli-bert-base-cls-pooling
- sentence-transformers/nli-bert-base-max-pooling
- sentence-transformers/nli-bert-base
- sentence-transformers/nli-bert-large-cls-pooling
- sentence-transformers/nli-bert-large-max-pooling
- sentence-transformers/nli-bert-large
- sentence-transformers/nli-distilbert-base-max-pooling
- sentence-transformers/nli-distilbert-base
- sentence-transformers/nli-roberta-base
- sentence-transformers/nli-roberta-large
- sentence-transformers/roberta-base-nli-mean-tokens
- sentence-transformers/roberta-base-nli-stsb-mean-tokens
- sentence-transformers/roberta-large-nli-mean-tokens
- sentence-transformers/roberta-large-nli-stsb-mean-tokens
- sentence-transformers/stsb-bert-base
- sentence-transformers/stsb-bert-large
- sentence-transformers/stsb-distilbert-base
- sentence-transformers/stsb-roberta-base
- sentence-transformers/stsb-roberta-large
- sentence-transformers/xlm-r-100langs-bert-base-nli-mean-tokens
- sentence-transformers/xlm-r-100langs-bert-base-nli-stsb-mean-tokens
- sentence-transformers/xlm-r-base-en-ko-nli-ststb
- sentence-transformers/xlm-r-bert-base-nli-mean-tokens
- sentence-transformers/xlm-r-bert-base-nli-stsb-mean-tokens
- sentence-transformers/xlm-r-large-en-ko-nli-ststb
- sentence-transformers/bert-base-nli-cls-token
- sentence-transformers/all-distilroberta-v1
- sentence-transformers/multi-qa-MiniLM-L6-dot-v1
- sentence-transformers/multi-qa-distilbert-cos-v1
- sentence-transformers/multi-qa-distilbert-dot-v1
- sentence-transformers/multi-qa-mpnet-base-cos-v1
- sentence-transformers/multi-qa-mpnet-base-dot-v1
- sentence-transformers/nli-distilroberta-base-v2
- sentence-transformers/all-MiniLM-L6-v1
- sentence-transformers/all-mpnet-base-v1
- sentence-transformers/all-mpnet-base-v2
- sentence-transformers/all-roberta-large-v1
- sentence-transformers/allenai-specter
- sentence-transformers/average_word_embeddings_glove.6B.300d
- sentence-transformers/average_word_embeddings_glove.840B.300d
- sentence-transformers/average_word_embeddings_komninos
- sentence-transformers/average_word_embeddings_levy_dependency
- sentence-transformers/clip-ViT-B-32-multilingual-v1
- sentence-transformers/clip-ViT-B-32
- sentence-transformers/distilbert-base-nli-stsb-quora-ranking
- sentence-transformers/distilbert-multilingual-nli-stsb-quora-ranking
- sentence-transformers/distilroberta-base-paraphrase-v1
- sentence-transformers/distiluse-base-multilingual-cased-v1
- sentence-transformers/distiluse-base-multilingual-cased-v2
- sentence-transformers/distiluse-base-multilingual-cased
- sentence-transformers/facebook-dpr-ctx_encoder-multiset-base
- sentence-transformers/facebook-dpr-ctx_encoder-single-nq-base
- sentence-transformers/facebook-dpr-question_encoder-multiset-base
- sentence-transformers/facebook-dpr-question_encoder-single-nq-base
- sentence-transformers/gtr-t5-large
- sentence-transformers/gtr-t5-xl
- sentence-transformers/gtr-t5-xxl
- sentence-transformers/msmarco-MiniLM-L-12-v3
- sentence-transformers/msmarco-MiniLM-L-6-v3
- sentence-transformers/msmarco-MiniLM-L12-cos-v5
- sentence-transformers/msmarco-MiniLM-L6-cos-v5
- sentence-transformers/msmarco-bert-base-dot-v5
- sentence-transformers/msmarco-bert-co-condensor
- sentence-transformers/msmarco-distilbert-base-dot-prod-v3
- sentence-transformers/msmarco-distilbert-base-tas-b
- sentence-transformers/msmarco-distilbert-base-v2
- sentence-transformers/msmarco-distilbert-base-v3
- sentence-transformers/msmarco-distilbert-base-v4
- sentence-transformers/msmarco-distilbert-cos-v5
- sentence-transformers/msmarco-distilbert-dot-v5
- sentence-transformers/msmarco-distilbert-multilingual-en-de-v2-tmp-lng-aligned
- sentence-transformers/msmarco-distilbert-multilingual-en-de-v2-tmp-trained-scratch
- sentence-transformers/msmarco-distilroberta-base-v2
- sentence-transformers/msmarco-roberta-base-ance-firstp
- sentence-transformers/msmarco-roberta-base-v2
- sentence-transformers/msmarco-roberta-base-v3
- sentence-transformers/multi-qa-MiniLM-L6-cos-v1
- sentence-transformers/nli-mpnet-base-v2
- sentence-transformers/nli-roberta-base-v2
- sentence-transformers/nq-distilbert-base-v1
- sentence-transformers/paraphrase-MiniLM-L12-v2
- sentence-transformers/paraphrase-MiniLM-L3-v2
- sentence-transformers/paraphrase-MiniLM-L6-v2
- sentence-transformers/paraphrase-TinyBERT-L6-v2
- sentence-transformers/paraphrase-albert-base-v2
- sentence-transformers/paraphrase-albert-small-v2
- sentence-transformers/paraphrase-distilroberta-base-v1
- sentence-transformers/paraphrase-distilroberta-base-v2
- sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
- sentence-transformers/paraphrase-multilingual-mpnet-base-v2
- sentence-transformers/paraphrase-xlm-r-multilingual-v1
- sentence-transformers/quora-distilbert-base
- sentence-transformers/quora-distilbert-multilingual
- sentence-transformers/sentence-t5-base
- sentence-transformers/sentence-t5-large
- sentence-transformers/sentence-t5-xxl
- sentence-transformers/sentence-t5-xl
- sentence-transformers/stsb-distilroberta-base-v2
- sentence-transformers/stsb-mpnet-base-v2
- sentence-transformers/stsb-roberta-base-v2
- sentence-transformers/stsb-xlm-r-multilingual
- sentence-transformers/xlm-r-distilroberta-base-paraphrase-v1
- sentence-transformers/clip-ViT-L-14
- sentence-transformers/clip-ViT-B-16
- sentence-transformers/use-cmlm-multilingual
- sentence-transformers/all-MiniLM-L12-v1
```
You can also load many other model architectures from the library. For example models from sources such as BAAI, Nomic, Salesforce Research, etc. See this HF hub page for all [supported models](https://huggingface.co/models?library=sentence-transformers).
Here is an example that uses the BAAI embedding model from the Hugging Face Hub [supported models](https://huggingface.co/models?library=sentence-transformers).
Visit sentence-transformers [HuggingFace HUB](https://huggingface.co/sentence-transformers) page for more information on the available models.
# Superlinked
Source: https://docs.lancedb.com/integrations/embedding/superlinked
[Superlinked](https://superlinked.com) is a self-hosted inference engine (SIE) for embedding, reranking, and extraction. The `sie-lancedb` package registers SIE as a first-class embedding function in LanceDB's embeddings registry, so embeddings are computed automatically on insert and search. You need a running SIE instance - see the [Superlinked quickstart](https://superlinked.com/docs) for deployment options.
## Installation
```bash Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install sie-lancedb
```
```bash TypeScript icon=js theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
npm install @superlinked/sie-lancedb @lancedb/lancedb
```
## Registered functions
Importing `sie_lancedb` registers two embedding functions in LanceDB's registry:
| Name | Purpose |
| ------------------- | -------------------------------------------------- |
| `"sie"` | Dense text embeddings |
| `"sie-multivector"` | ColBERT-style late interaction with MaxSim scoring |
Supported parameters on `.create()`:
| Parameter | Type | Description |
| ---------- | ----- | -------------------------------------------------------------------------------------------------------------- |
| `model` | `str` | Any of 85+ SIE-supported models (e.g. `BAAI/bge-m3`, `NovaSearch/stella_en_400M_v5`, `jinaai/jina-colbert-v2`) |
| `base_url` | `str` | URL of the SIE endpoint (e.g. `http://localhost:8080`) |
## Usage
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.embeddings import get_registry
from lancedb.pydantic import LanceModel, Vector
import sie_lancedb # registers "sie" and "sie-multivector"
sie = get_registry().get("sie").create(
model="BAAI/bge-m3",
base_url="http://localhost:8080",
)
class Documents(LanceModel):
text: str = sie.SourceField()
vector: Vector(sie.ndims()) = sie.VectorField()
db = lancedb.connect("~/.lancedb")
table = db.create_table("docs", schema=Documents, mode="overwrite")
table.add([
{"text": "Machine learning is a subset of AI."},
{"text": "Neural networks use multiple layers."},
{"text": "Python is popular for ML development."},
])
results = table.search("What is deep learning?").limit(3).to_list()
```
LanceDB handles embedding generation for both inserts and queries automatically, based on the `SourceField` / `VectorField` declarations on the schema.
## Hybrid search with reranker
`SIEReranker` plugs into LanceDB's hybrid search pipeline. It uses SIE's cross-encoder `score()` to rerank combined vector + full-text search results. You need a full-text search index on the column first:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from sie_lancedb import SIEReranker
# Create FTS index for hybrid search
table.create_fts_index("text", replace=True)
results = (
table.search("What is deep learning?", query_type="hybrid")
.rerank(SIEReranker(model="jinaai/jina-reranker-v2-base-multilingual"))
.limit(5)
.to_list()
)
for r in results:
print(f"{r['_relevance_score']:.3f} {r['text']}")
```
The reranker also works with pure vector or pure FTS search via `.rerank()`.
## ColBERT / multivector
`SIEMultiVectorEmbeddingFunction` (registered as `"sie-multivector"`) works with LanceDB's native `MultiVector` type and MaxSim scoring for ColBERT and ColPali models:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.pydantic import MultiVector
sie_colbert = get_registry().get("sie-multivector").create(
model="jinaai/jina-colbert-v2",
base_url="http://localhost:8080",
)
class ColBERTDocs(LanceModel):
text: str = sie_colbert.SourceField()
vector: MultiVector(sie_colbert.ndims()) = sie_colbert.VectorField()
table = db.create_table("colbert_docs", schema=ColBERTDocs, mode="overwrite")
table.add([{"text": "Machine learning is a subset of AI."}])
# MaxSim search - query and document multivectors compared token-by-token
results = table.search("What is ML?").limit(5).to_list()
```
## Entity extraction
`SIEExtractor` adds entity extraction to LanceDB's data-enrichment workflows. Extract entities from a text column and merge the results back as a structured Arrow column - enabling filtered search on extracted entities:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from sie_lancedb import SIEExtractor
extractor = SIEExtractor(
base_url="http://localhost:8080",
model="urchade/gliner_multi-v2.1",
)
extractor.enrich_table(
table,
source_column="text",
target_column="entities",
labels=["person", "technology", "organization"],
id_column="id",
)
```
The `entities` column stores structured Arrow data (`list>`), so you can filter on extracted entities in queries.
## Links
* [`sie-lancedb` on PyPI](https://pypi.org/project/sie-lancedb/)
* [`@superlinked/sie-lancedb` on npm](https://www.npmjs.com/package/@superlinked/sie-lancedb)
* [Superlinked on GitHub](https://github.com/superlinked/sie)
* [Superlinked docs](https://superlinked.com/docs)
# VoyageAI
Source: https://docs.lancedb.com/integrations/embedding/voyageai
Voyage AI provides cutting-edge embedding and rerankers.
Using voyageai API requires voyageai package, which can be installed using `pip install voyageai`. Voyage AI embeddings are used to generate embeddings for text data. The embeddings can be used for various tasks like semantic search, clustering, and classification.
You also need to set the `VOYAGE_API_KEY` environment variable to use the VoyageAI API.
Supported models are:
* voyage-4-large (best retrieval quality, 1024 default dimensions, supports 256/512/1024/2048)
* voyage-4 (balanced general-purpose, 1024 default dimensions, supports 256/512/1024/2048)
* voyage-4-lite (optimized for latency/cost, 1024 default dimensions, supports 256/512/1024/2048)
* voyage-context-3
* voyage-3.5
* voyage-3.5-lite
* voyage-3
* voyage-3-lite
* voyage-finance-2
* voyage-multilingual-2
* voyage-law-2
* voyage-code-2
* voyage-multimodal-3.5 (multimodal - supports text, images, and video)
**Multimodal Model:** `voyage-multimodal-3.5` supports text, images, and video inputs. It outputs 1024-dimensional embeddings by default, configurable via the `output_dimension` parameter (256, 512, 1024, 2048). See the [VoyageAI multimodal embeddings documentation](https://docs.voyageai.com/docs/multimodal-embeddings) for more details.
Supported parameters (to be passed in `create` method) are:
| Parameter | Type | Default Value | Description |
| ------------------ | ------ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `str` | `None` | The model ID of the model to use. Supported models: voyage-4-large, voyage-4, voyage-4-lite, voyage-3, voyage-3-lite, voyage-3.5, voyage-3.5-lite, voyage-context-3, voyage-finance-2, voyage-multilingual-2, voyage-law-2, voyage-code-2, voyage-multimodal-3.5 |
| `input_type` | `str` | `None` | Type of the input text. Default to None. Other options: query, document. |
| `truncation` | `bool` | `True` | Whether to truncate the input texts to fit within the context length. |
| `output_dimension` | `int` | `None` | Output embedding dimension. Only supported by `voyage-multimodal-3.5`. Valid options: 256, 512, 1024 (default), 2048. |
Usage Example:
### Multimodal Example
The `voyage-multimodal-3.5` model can embed text alongside images. You can use image URLs, file paths, or PIL Image objects:
# Integrations
Source: https://docs.lancedb.com/integrations/index
Connect LanceDB with popular AI providers, frameworks, and data platforms
LanceDB seamlessly plugs into the rest of your AI and data engineering stack. Use the sections
below to jump straight into the guides that matter for your workflow.
| Group | Description |
| :------------------------------------------------- | :---------------------------------------------------------------------------------------------- |
| [Embedding models](/integrations/embedding/) | Connect with popular embedding model providers including OpenAI, Cohere, Hugging Face, and more |
| [Reranking models](/integrations/reranking/) | Enhance search results with advanced reranking models and techniques |
| [AI platforms & frameworks](/integrations/ai/) | Integrate with LangChain, LlamaIndex, Kiln, and other AI development frameworks |
| [Data platforms & frameworks](/integrations/data/) | Integrate LanceDB with popular data tools and platforms like DuckDB, Pydantic and dlt |
# LeRobotDataset
Source: https://docs.lancedb.com/integrations/lerobotdataset
Use Lance-backed LeRobotDataset loaders and LanceDB to inspect, filter, and train on robotics datasets from the Hugging Face Hub.
[LeRobot](https://huggingface.co/docs/lerobot/index) is Hugging Face's open-source robotics stack for collecting data, training policies, running simulations, and sharing robotics datasets and models on the Hub.
[LeRobotDataset v3.0](https://huggingface.co/docs/lerobot/lerobot-dataset-v3) standardizes robot learning data across sensorimotor time series, actions, multi-camera video, and task metadata. Its v3 layout stores high-frequency tabular signals in Parquet, visual streams as MP4 shards, and metadata that reconstructs episode-level views from larger files.
Lance pairs well with LeRobot when you need high-performance random access, lazy multimodal blob reads, and a single table interface for curation, search, and training data preparation. The `lerobot-lancedb` package ships Lance-backed `LeRobotDataset` subclasses, and LanceDB can open Lance-formatted LeRobot datasets on the Hub directly through `hf://` URIs.
## Install
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install lancedb lance lerobot-lancedb
```
## Use Lance-backed LeRobotDataset loaders
`LeRobotLanceDataset` is useful when your Lance-backed dataset stores decoded image observations. It's a drop-in replacement for `LeRobotDataset`, so existing policy training code keeps working with the usual PyTorch dataset and dataloader patterns.
For datasets that store camera observations as MP4 video segments, use `LeRobotLanceVideoDataset` instead.
Use the image loader for Lance-backed repos that store image frames. Use the video loader for MP4-backed LeRobot datasets such as `lance-format/lerobot-pusht-lance`.
## Open LeRobot Lance tables with LanceDB
Lance-formatted LeRobot datasets published by `lance-format` expose each `.lance` file under `data/` as a LanceDB table. The PushT dataset, for example, has `frames`, `episodes`, and `videos` tables.
Opening the tables directly is handy for inspecting schemas, counting rows, sampling metadata, or building curation workflows before any data reaches the training loop.
## Filter a frame window
Most robotics workflows want a deterministic slice by `episode_index`, `frame_index`, or task metadata long before training begins. LanceDB filters those rows without touching the video blobs.
With the filtered set in hand, you can materialize a smaller local LanceDB database, add derived columns, attach embeddings, or build vector and scalar indexes for faster repeated access.
## Example Lance-formatted LeRobot datasets
A Lance-formatted version of `lerobot/pusht` with frame, episode, and video tables.
A multi-camera robotics dataset packaged as Lance tables for frame-level and episode-level access.
## More resources
Hugging Face's guide to the v3 dataset layout, streaming, transforms, and migration.
API documentation for the Lance-backed LeRobotDataset implementations.
## When to use each interface
| Interface | Best for |
| :------------------------- | :--------------------------------------------------------------------------------------- |
| `LeRobotDataset` | Standard LeRobot training loops and policy code |
| `LeRobotLanceDataset` | Drop-in training on Lance-backed image datasets |
| `LeRobotLanceVideoDataset` | Drop-in training on Lance-backed video datasets |
| LanceDB | Interactive inspection, filtering, curation, search, indexing, and materializing subsets |
| `lance.dataset(...)` | Lower-level schema, fragment, index, and blob access |
# Answer.AI Rerankers
Source: https://docs.lancedb.com/integrations/reranking/answerdotai
Use AnswerDotAI's lightweight reranking library with LanceDB. Features unified API for common reranking models, configurable model selection, and comprehensive scoring options.
# Answer.AI Rerankers
This integration uses [AnswersDotAI's rerankers](https://github.com/AnswerDotAI/rerankers) to rerank the search results, providing a lightweight, low-dependency, unified API to use all common reranking and cross-encoder models.
> **Note:** Supported query types – Hybrid, Vector, and FTS.
## Accepted Arguments
| Argument | Type | Default | Description |
| -------------- | ----- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_type` | `str` | `"colbert"` | The type of model to use. Supported model types can be found here: [https://github.com/AnswerDotAI/rerankers](https://github.com/AnswerDotAI/rerankers). |
| `model_name` | `str` | `"answerdotai/answerai-colbert-small-v1"` | The name of the reranker model to use. |
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the \`\_relevance\_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | --------------- | ----------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
### Vector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ------------------------------------------------------------------------------------ |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`). |
### FTS Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
# Cohere Reranker
Source: https://docs.lancedb.com/integrations/reranking/cohere
Integrate Cohere's powerful reranking API with LanceDB for enhanced search results. Supports English and multilingual models with configurable scoring options for vector, FTS, and hybrid search.
# Cohere Reranker
This reranker uses the [Cohere](https://cohere.ai/) API to rerank the search results. You can use this reranker by passing `CohereReranker()` to the `rerank()` method. Note that you'll either need to set the `COHERE_API_KEY` environment variable or pass the `api_key` argument to use this reranker.
> **Note:** Supported query types – Hybrid, Vector, and FTS.
```shell theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install cohere
```
## Accepted Arguments
| Argument | Type | Default | Description |
| -------------- | ----- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_name` | `str` | `"rerank-english-v2.0"` | The name of the reranker model to use. Available cohere models are: rerank-english-v2.0, rerank-multilingual-v2.0 |
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `top_n` | `str` | `None` | The number of results to return. If None, will return all results. |
| `api_key` | `str` | `None` | The API key for the Cohere API. If not provided, the `COHERE_API_KEY` environment variable is used. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the \`\_relevance\_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type |
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | --------------- | ---------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column |
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`) |
### Vector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column |
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`) |
### FTS Search
| `return_score` | Status | Description |
| -------------- | ----------- | ---------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column |
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`) |
# ColBERT Reranker
Source: https://docs.lancedb.com/integrations/reranking/colbert
Enhance search results with ColBERT's contextual reranking in LanceDB. Features efficient model deployment, device optimization, and flexible scoring options for vector, FTS, and hybrid search.
# ColBERT Reranker
This reranker uses ColBERT model to rerank the search results. You can use this reranker by passing `ColbertReranker()` to the `rerank()` method.
> **Note:** Supported query types – Hybrid, Vector, and FTS.
## Accepted Arguments
| Argument | Type | Default | Description |
| -------------- | ----- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_name` | `str` | `"colbert-ir/colbertv2.0"` | The name of the reranker model to use. |
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `device` | `str` | `None` | The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu". |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the \`\_relevance\_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | --------------- | ----------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
### Vector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ------------------------------------------------------------------------------------ |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`). |
### FTS Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
# Jina Reranker
Source: https://docs.lancedb.com/integrations/reranking/jina
Integrate Jina's multilingual reranking API with LanceDB for improved search results. Features model selection, API key management, and flexible scoring options for all search types.
# Jina Reranker
This reranker uses the [Jina](https://jina.ai/reranker/) API to rerank the search results. You can use this reranker by passing `JinaReranker()` to the `rerank()` method. Note that you'll either need to set the `JINA_API_KEY` environment variable or pass the `api_key` argument to use this reranker.
> **Note:** Supported query types – Hybrid, Vector, and FTS.
## Accepted Arguments
| Argument | Type | Default | Description |
| -------------- | ----- | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_name` | `str` | `"jina-reranker-v2-base-multilingual"` | The name of the reranker model to use. You can find the list of available models in [https://jina.ai/reranker](https://jina.ai/reranker). |
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `top_n` | `str` | `None` | The number of results to return. If None, will return all results. |
| `api_key` | `str` | `None` | The API key for the Jina API. If not provided, the `JINA_API_KEY` environment variable is used. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the \`\_relevance\_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | --------------- | ----------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
### Vector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ------------------------------------------------------------------------------------ |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`). |
### FTS Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
# OpenAI Reranker (Experimental)
Source: https://docs.lancedb.com/integrations/reranking/openai
Explore experimental search reranking using OpenAI's GPT models in LanceDB. Features configurable model selection, API key management, and comprehensive scoring options for all search types.
# OpenAI Reranker (Experimental)
This reranker uses OpenAI chat model to rerank the search results. You can use this reranker by passing `OpenAI()` to the `rerank()` method.
> **Note:** Supported query types – Hybrid, Vector, and FTS.
> **Warning:** This reranker is experimental. OpenAI does not have a dedicated reranking model, so it uses a chat model under the hood.
## Accepted Arguments
| Argument | Type | Default | Description |
| -------------- | ----- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_name` | `str` | `"gpt-4-turbo-preview"` | The name of the reranker model to use. |
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the \`\_relevance\_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
| `api_key` | `str` | `None` | The API key to use. If None, will use the OPENAI\_API\_KEY environment variable. |
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | --------------- | ----------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
### Vector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ------------------------------------------------------------------------------------ |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`). |
### FTS Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
# VoyageAI Reranker
Source: https://docs.lancedb.com/integrations/reranking/voyageai
Integrate VoyageAI's cutting-edge reranking models with LanceDB. Features model selection, API key management, and comprehensive scoring options for all search types.
# VoyageAI Reranker
Voyage AI provides cutting-edge embedding and rerankers.
This reranker uses the [VoyageAI](https://docs.voyageai.com/docs/) API to rerank the search results. You can use this reranker by passing `VoyageAIReranker()` to the `rerank()` method. Note that you'll either need to set the `VOYAGE_API_KEY` environment variable or pass the `api_key` argument to use this reranker.
> **Note:** Supported query types – Hybrid, Vector, and FTS.
## Accepted Arguments
| Argument | Type | Default | Description |
| -------------- | ------ | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_name` | `str` | `None` | The name of the reranker model to use. Available models are: rerank-2, rerank-2-lite |
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `top_n` | `str` | `None` | The number of results to return. If None, will return all results. |
| `api_key` | `str` | `None` | The API key for the Voyage AI API. If not provided, the `VOYAGE_API_KEY` environment variable is used. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the \`\_relevance\_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type |
| `truncation` | `bool` | `None` | Whether to truncate the input to satisfy the "context length limit" on the query and the documents. |
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | --------------- | ---------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Returns only have the `_relevance_score` column |
| `all` | ❌ Not Supported | Returns have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`) |
### Vector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Returns only have the `_relevance_score` column |
| `all` | ✅ Supported | Returns have vector(`_distance`) along with Hybrid Search score(`_relevance_score`) |
### FTS Search
| `return_score` | Status | Description |
| -------------- | ----------- | ---------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Returns only have the `_relevance_score` column |
| `all` | ✅ Supported | Returns have FTS(`score`) along with Hybrid Search score(`_relevance_score`) |
# Watsonx Reranker
Source: https://docs.lancedb.com/integrations/reranking/watsonx
Rerank LanceDB search results with the IBM watsonx.ai text rerank API. Supports vector, FTS, and hybrid search with configurable models, projects, and spaces.
# Watsonx Reranker
This reranker uses the [IBM watsonx.ai](https://cloud.ibm.com/docs/apis/watsonx-ai#text-rerank) text rerank API to reorder search results. Pass `WatsonxReranker()` to the `rerank()` method on a query. Credentials come from the `WATSONX_API_KEY` and `WATSONX_PROJECT_ID` (or `WATSONX_SPACE_ID`) environment variables, or can be passed explicitly as arguments.
> **Note:** Supported query types – Hybrid, Vector, and FTS.
```shell theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install ibm-watsonx-ai
```
## Accepted Arguments
| Argument | Type | Default | Description |
| ----------------------- | ----- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `model_name` | `str` | `"cross-encoder/ms-marco-minilm-l-12-v2"` | The rerank model ID. See [supported rerank models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx#rerank). |
| `column` | `str` | `"text"` | The name of the table column to use as document input. |
| `top_n` | `int` | `None` | The number of results to return. If `None`, all results are returned. |
| `return_score` | `str` | `"relevance"` | Options are `"relevance"` or `"all"`. Controls which score columns are kept on the result. |
| `api_key` | `str` | `None` | IBM Cloud API key. Falls back to the `WATSONX_API_KEY` environment variable. |
| `project_id` | `str` | `None` | watsonx.ai project ID. Falls back to `WATSONX_PROJECT_ID`. Mutually exclusive with `space_id`. |
| `space_id` | `str` | `None` | watsonx.ai deployment space ID. Falls back to `WATSONX_SPACE_ID`. Mutually exclusive with `project_id`. |
| `url` | `str` | `"https://us-south.ml.cloud.ibm.com"` | watsonx.ai service URL. |
| `truncate_input_tokens` | `int` | `None` | Truncate each document to this many tokens before scoring. |
You must supply exactly one of `project_id` or `space_id` (either as an argument or via its environment variable). Setting both, or neither, raises a `ValueError`.
## Supported scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | --------------- | ----------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
### Vector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ------------------------------------------------------------------------------------ |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`). |
### FTS Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
# Stable World Model
Source: https://docs.lancedb.com/integrations/stable-worldmodel
Use Stable World Model with LanceDB-backed datasets for reproducible world model research, fast data loading, and compact storage.
[Stable World Model](https://github.com/galilai-group/stable-worldmodel) is a research platform for collecting data, training world models, and evaluating policies with model-predictive control across standardized environments.
The LanceDB integration is built into Stable World Model's data format registry. Lance is the default backend for collected datasets, so a path ending in `.lance` gives you an append-friendly LanceDB table with episode-contiguous rows and fast indexed reads.
Random access speed is the bottleneck for world model training, since the loop repeatedly samples temporal windows from high-dimensional observations, actions, and rewards. The faster those windows arrive, the more GPU time goes into training rather than waiting on the data loader.
## Install
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install stable-worldmodel
```
Datasets and checkpoints are stored under `$STABLEWM_HOME`, which defaults to `~/.stable_worldmodel/`.
## Collect data into Lance
Stable World Model uses Lance by default when you collect to a `.lance` path.
Replace `your_expert_policy` with the expert or scripted policy you use to collect demonstrations.
Every writer accepts a `mode` argument such as `append`, `overwrite`, or `error`. The default is append, so re-running collection extends the existing dataset.
## Load a Lance dataset for training
The dataset loader autodetects the Lance format from the path.
Your model code stays focused on the world model objective while LanceDB handles the storage layout and read path.
## Evaluate with model-predictive control
After training a world model on the Lance-backed dataset, Stable World Model can evaluate it with planning solvers such as CEM.
Replace `world_model` with the trained model object from your training loop.
## Convert between formats
Stable World Model can convert between registered dataset formats. A common workflow is to collect in Lance for fast training reads, then export to the video layout for compact inspection artifacts.
## Throughput
The Stable World Model README reports the following PushT benchmark results from `scripts/benchmark/compare_h5_lance.py`:
| Format | Source | Cache | samples/s | ms/step |
| :------ | :----- | :------- | --------: | ------: |
| HDF5 | local | no-cache | 1,416.1 | 45.2 |
| HDF5 | local | cached | 1,474.0 | 43.4 |
| LanceDB | local | no-cache | 4,814.8 | 13.3 |
| LanceDB | local | cached | 4,431.3 | 14.4 |
| Video | local | - | 1,330.6 | 48.1 |
| LanceDB | s3 | no-cache | 3,183.7 | 20.1 |
| LanceDB | s3 | cached | 3,253.2 | 19.7 |
| HDF5 | s3 | no-cache | 9.1 | 7,032.5 |
| HDF5 | s3 | cached | 756.5 | 84.6 |
In that benchmark, local LanceDB reached about **3.4x** the no-cache throughput of local HDF5, while S3-backed LanceDB reached about **350x** the no-cache throughput of S3-backed HDF5. Even with cache enabled, S3-backed LanceDB was about **4.3x** faster than S3-backed HDF5.
These numbers come from the Stable World Model project's own benchmark setup, so they're best read as a reproducible directional baseline that may shift across environments, models, and storage configurations.
## Storage
The same README reports these local storage sizes for the benchmark dataset:
| Format | Local size |
| :------ | ---------: |
| HDF5 | 43.12 GB |
| LanceDB | 13.31 GB |
| Video | 496.29 MB |
LanceDB used about **69% less local storage than HDF5** in the reported benchmark, while preserving a table interface built for fast training reads and append-heavy collection.
## More resources
Installation, quick start, supported formats, benchmarks, environments, solvers, and citation.
Full upstream documentation with tutorials, API references, and guides.
# Lance format
Source: https://docs.lancedb.com/lance
Open-source lakehouse format for multimodal AI.
[Lance](https://lance.org/) is an open-source, columnar lakehouse format for multimodal AI.
It provides a file format, table format, and lightweight catalog spec, allowing developers
to build a complete open lakehouse on top of object storage.
Building on top of open foundations and optimizing the format for random access
(without compromising scan performance) enables
high-performance vector search, full-text search, indexing, and feature engineering capabilities.
[LanceDB](/enterprise) builds on these capabilities so teams can work with one multimodal data layer
instead of moving data across separate storage, search, feature, and training systems.
Visit the Lance format documentation to learn more about its design, features, and how it enables the multimodal lakehouse.
## Capabilities of the Lance format
| Capability | What it enables |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Multimodal storage | Store images, video, audio, text, embeddings, annotations, metadata, features, and more, all in one table. |
| First-class blob API | Store large binary objects such as images, video, audio, and model artifacts in blob columns with lazy reads and streaming byte access. |
| Fast random access and scans | Sample, shuffle, and retrieve individual rows efficiently without giving up high-throughput sequential reads. |
| Flexible data evolution | Add, drop, rename, or alter columns as datasets change, often without rewriting existing data files. |
| Versioned tables | Reproduce experiments, restore previous states, and tie downstream artifacts to the exact table version they used. |
| Hybrid search and indexing | Combine vector search, full-text search, and scalar filters on the same dataset with Lance indexes. |
| Open lakehouse interoperability | Build on object storage and connect Lance tables to open engines such as PyTorch, Ray, Spark, Trino, DuckDB and Polars. |
## Key concepts
The following concepts are core to the Lance format:
**Arrow-native, columnar storage** and **interoperability** with the open lakehouse ecosystem (including other file formats and compute engines).
**Zero-copy** data evolution, meaning you can easily add derived columns (like features or embeddings) at a later time, **without full table rewrites**. Only new data is written; expensive existing data (like images/videos) remain untouched.
Data is **versioned**, with each insert operation creating a new version of the dataset and an update to the manifest that tracks versions via metadata
### Data versioning
Data in Lance tables are versioned -- this helps keep LanceDB scalable and consistent.
We do not immediately blow away old versions when creating new ones because other clients might be
in the middle of querying the old version. It's important to retain older versions for as long as they
might be queried.
Each version contains metadata and just the new/updated data in your transaction. So if you have 100
versions, they aren't 100 duplicates of the same data. However, they do have 100x the metadata overhead
of a single version, which can result in slower queries.
### Data compaction
As you insert more data, your dataset will grow and you'll need to perform compaction to maintain query
throughput (i.e., keep latencies down to a minimum). Compaction is the process of merging fragments
together to reduce the amount of metadata that needs to be managed, and to reduce the number of files
that need to be opened while scanning the dataset.
Running compaction on a Lance dataset will do the following:
* Remove deleted rows from fragments
* Remove dropped columns from fragments
* Merge small fragments into larger ones
Compaction focuses on read performance, not immediate disk reclamation. During compaction, Lance writes
new compacted files while older files are still referenced by previous table versions. This means disk
usage can increase temporarily until old versions are cleaned up.
### Data deletion and recovery
Although Lance allows you to delete rows from a dataset, it does not actually delete the data immediately.
It simply marks the row as deleted in the `DataFile` that represents a fragment.
For a given version of the dataset, each fragment can have up to one deletion file (if no rows were ever
deleted from that fragment, it will not have a deletion file). This is important to keep in mind because
it means that the data is still there, and can be recovered if needed, as long as that version still
exists based on your backup policy.
Lance is a separate open source project. Check out its documentation to learn more.
# Namespaces and the Catalog Model
Source: https://docs.lancedb.com/namespaces/index
Understand LanceDB as a catalog-level abstraction over Lance's table format, and learn how namespaces help organize Lance tables.
Despite its name, LanceDB is not a "database" in the traditional sense -- it is a **Multimodal Lakehouse** that builds on the table abstraction,
similar to many other lakehouse projects. LanceDB exposes a catalog-level abstraction over the Lance table format, via a *namespace spec*.
If you're coming from traditional databases or lakehouses, you can think of a namespace as the catalog path that says where a table name lives.
Lance provides the **file** and **table** formats to store and manage your data and indexes.
LanceDB operates at the catalog layer (used to organize, discover, and operate on many Lance tables)
and provides a compute engine on top of the Lance format.
This is why many SDK methods in LanceDB, like `create_table`, `open_table`, `drop_table`, and
`rename_table`, accept namespace input. The SDK methods expose that input in the idiom of each
language: Python uses `namespace_path`, Rust uses builder methods like `.namespace(...)`, and
TypeScript uses `namespacePath` arguments.
## Namespace hierarchy
Namespaces are generalizations of catalog specs that give platform developers a clean way to present Lance tables in the structures users expect. The diagram below shows how the hierarchy can go beyond a single level.
A namespace can contain a collection of tables, and it can also contain namespaces recursively.
![]()
Before diving into examples, it helps to keep two terms in mind: the **namespace client** is the abstraction that presents a consistent namespace API, while the **namespace implementation** is the concrete backend that resolves namespaces and table locations (for example, a local directory or an external catalog).
If you want to go deeper, see the Lance format [namespace documentation](https://lance.org/format/namespace/).
## Namespace paths and names
A namespace path is a list of components. For example, `["prod", "search"]` means the `search`
namespace inside the `prod` namespace. The empty path, `[]`, means the root namespace.
Each component is a name, not a filesystem path segment. Namespace names can't be empty, and each
component can contain only letters, numbers, underscores, hyphens, and periods. That keeps the same
identifier usable across local directory namespaces and REST namespace identifiers.
## Directory namespaces
The simplest namespace model in LanceDB is a single root namespace, often represented by one
directory:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
./local_lancedb (root)
└── prod
└── search
└── user (table)
└── data (table)
```
As a user of LanceDB OSS, you might never notice namespaces at first, because LanceDB exposes the single-level hierarchy shown above, with the data stored in the `data/` directory, where the root namespace is implicit. Connecting to this namespace is as simple as connecting to the catalog root:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
# Connect to the directory namespace root
db = lancedb.connect("./local_lancedb")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
// Connect to the directory namespace root
const db = await lancedb.connect("./local_lancedb");
```
```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
use lancedb::connect;
// Connect to the directory namespace root
let db = connect("./local_lancedb").execute().await?;
```
This creates the default namespace directory (`data/`) under the specified root path.
You can also explicitly connect to a namespace using `lancedb.connect_namespace(...)` with the directory namespace implementation:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
# Local namespace-backed catalog root (DirectoryNamespace)
# See https://lance.org/format/namespace/dir/catalog-spec/
db = lancedb.connect_namespace("dir", {"root": "./local_lancedb"})
table_name = "user"
data = [{"id": 1, "vector": [0.1, 0.2], "name": "alice"}]
table = db.create_table(table_name, data=data, mode="create")
print(f"Created table: {table.name}")
# Created table: user
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
// Local namespace-backed catalog root (DirectoryNamespace)
// See https://lance.org/format/namespace/dir/catalog-spec/
const db = await lancedb.connectNamespace("dir", { root: "./local_lancedb" });
const table = await db.createTable(
"user",
[{ id: 1, vector: [0.1, 0.2], name: "alice" }],
{ mode: "create" },
);
console.log(`Created table: ${table.name}`);
// Created table: user
```
```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
use std::collections::HashMap;
use std::sync::Arc;
use arrow_schema::{DataType, Field, Schema};
// Local namespace-backed catalog root (DirectoryNamespace)
// See https://lance.org/format/namespace/dir/catalog-spec/
let mut properties = HashMap::new();
properties.insert("root".to_string(), "./local_lancedb".to_string());
let db = lancedb::connect_namespace("dir", properties)
.execute()
.await?;
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int64, false),
]));
let table = db
.create_empty_table("user", schema)
.execute()
.await?;
println!("Created table: {}", table.name());
// Created table: user
```
* For simple use cases in LanceDB OSS, you don't need to go too deep into namespaces.
* To integrate LanceDB with external catalogs and to use it as a true **multimodal lakehouse**, it's useful to understand the different namespace implementations and how to use them in your organization's setup.
## Remote or external catalog namespaces
The example above showed local directory-based namespaces. LanceDB also supports namespaces backed by remote object stores and external catalogs, via the REST namespace implementation.
For remote object stores with central metadata/catalog services (either commercial or open source),
use the REST namespace implementation. It is backed by REST routes
(for example `POST /v1/namespace/{id}/create` and `GET /v1/namespace/{id}/list`) and server-provided table locations.
For authentication, any property prefixed with `headers` is forwarded as an HTTP header
(for example `headers.Authorization` becomes `Authorization`, and `headers.X-API-Key` becomes `X-API-Key`).
LanceDB Enterprise REST requests use the `x-api-key` header for API-key authentication. Deployments
that route multiple databases through the same endpoint can also use headers such as
`x-lancedb-database` or `x-lancedb-database-prefix` for database context.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import os
import lancedb
# Remote namespace-backed catalog root (RestNamespace)
# See https://lance.org/format/namespace/rest/catalog-spec/
db = lancedb.connect_namespace(
"rest",
{
"uri": "https://.internal..com",
"headers.x-api-key": os.environ["API_KEY"],
# or:
# "headers.Authorization": f"Bearer {os.environ['REST_AUTH_TOKEN']}",
},
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
// Remote namespace-backed catalog root (RestNamespace)
// See https://lance.org/format/namespace/rest/catalog-spec/
const db = await lancedb.connectNamespace("rest", {
uri: "https://.internal..com",
headers: {
"x-api-key": process.env.API_KEY ?? "",
// or:
// Authorization: `Bearer ${process.env.REST_AUTH_TOKEN}`,
},
});
```
```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
use std::collections::HashMap;
// Remote namespace-backed catalog root (RestNamespace)
// See https://lance.org/format/namespace/rest/catalog-spec/
let mut properties = HashMap::new();
properties.insert(
"uri".to_string(),
"https://.internal..com".to_string(),
);
properties.insert(
"headers.x-api-key".to_string(),
std::env::var("API_KEY")?,
);
// or:
// properties.insert(
// "headers.Authorization".to_string(),
// format!("Bearer {}", std::env::var("REST_AUTH_TOKEN")?),
// );
let db = lancedb::connect_namespace("rest", properties)
.execute()
.await?;
```
[LanceDB Enterprise](/enterprise) operates a REST namespace server on top of the Lance format, so any REST client that can speak the REST namespace API
contract can be used to interact with it. For authentication examples in LanceDB Enterprise, visit
the [Namespaces in SDKs](/namespaces/usage#namespaces-in-lancedb-enterprise) page.
## Best practices
Below, we list some best practices for working with namespaces:
* For simple use cases and single, stand-alone applications, the directory-based root namespace is sufficient and requires no special configuration.
* For remote storage locations, introduce explicit namespaces when multiple teams, environments, or domains share the same catalog.
* Treat namespace paths as stable identifiers (for example `"prod/search"`, `"staging/recs"`).
* For maintainability reasons, avoid hard-coding object-store table paths in application code -- instead, prefer catalog identifiers + namespaces.
# Using Namespaces
Source: https://docs.lancedb.com/namespaces/usage
Use LanceDB's namespace-aware table and catalog APIs in Python, TypeScript, and Rust.
As your table organization needs grow over time and your projects become more complex, you can use namespaces to organize your tables in a way that reflects your business domains, teams, or environments.
As described in the [Namespaces and Catalog Model](/namespaces) section, namespaces are LanceDB's way of generalizing catalog specs, providing developers a clean way to manage hierarchical organization of tables in the catalog. The SDKs treat a namespace as a path and can use it for table resolution when you use LanceDB outside the root namespace.
## Table operations with namespace paths
Let's imagine a scenario where your table management needs have evolved, and you now have the following multi-level structure to organize your tables outside the root namespace.
```
./local_lancedb (root)
└── prod
└── search
└── user (table)
└── data (table)
└── recommendations
└── user (table)
└── data (data)
```
Below, we show how you would express table operations within that namespace. Each item in the namespace
list (`["prod", "search"]`) represents a level in the namespace hierarchy, and the table name is
specified when you create, open, list, or drop it.
The SDK methods expose the namespace path in the idiom of each language:
* Python: pass `namespace_path=["prod", "search"]` to table operations.
* Rust: call builder methods such as `.namespace(vec!["prod".to_string(), "search".to_string()])`.
* TypeScript: pass a `namespacePath` array, for example `await db.openTable("user", ["prod", "search"])`.
Using namespaces is **optional** in LanceDB, and most basic use cases do not require to work with them.
An empty namespace (`[]`), which is the default, means "root namespace", and the data will be stored in
the `data/` directory under the specified root path.
## Namespace management APIs
You can open/create/drop tables inside a namespace path (like `["prod", "search"]`).
All three SDKs expose namespace lifecycle operations directly.
In Python, use `lancedb.connect_namespace(...)` when calling namespace lifecycle methods such as
`create_namespace`, `list_namespaces`, `describe_namespace`, and `drop_namespace`.
In TypeScript, use `lancedb.connectNamespace(...)` and call `createNamespace`, `listNamespaces`,
`describeNamespace`, and `dropNamespace` on the returned `Connection`.
In Rust, use `lancedb::connect_namespace(...)` and call `create_namespace`, `list_namespaces`,
and `drop_namespace`.
Namespace creation and deletion have modes that control what happens when the target already exists,
doesn't exist, or contains data:
* Create mode: `create` fails if the namespace already exists, `exist_ok` keeps the existing namespace, and `overwrite` replaces it.
* Drop mode: `fail` reports an error when the namespace doesn't exist, and `skip` treats a missing namespace as a successful no-op.
* Drop behavior: `restrict` keeps non-empty namespaces from being dropped, and `cascade` drops child namespaces and tables first.
Namespace path components can't be empty. Each component can contain only letters, numbers,
underscores, hyphens, and periods.
Listing APIs return the immediate children of the requested namespace path. Use `limit` with the
returned `page_token` to page through large catalogs; pass an empty namespace path (`[]`) when you
want to list from the root namespace.
## Namespaces in LanceDB Enterprise
In LanceDB Enterprise deployments, configure namespace-backed federated databases in a TOML file under your deployment's `config` directory.
LanceDB Enterprise supports both directory-based (`ns_impl = "dir"`) and REST-based (`ns_impl = "rest"`) namespace implementations.
The example below shows how to configure a directory-based namespace implementation in LanceDB Enterprise.
```toml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Federated database configuration for DirectoryNamespace
# This example uses minio storage
[federated_dbs.federated_dir_test]
ns_impl = "dir"
root = "s3:///"
"storage.region" = "us-east-1"
"storage.endpoint" = "http://localhost:9000"
"storage.access_key_id" = "minioadmin"
"storage.secret_access_key" = "minioadmin"
"storage.allow_http" = "true"
# Far future expiration (year 2100)
"storage.expires_at_millis" = "4102444800000"
```
The example above uses MinIO, but the same approach applies to other cloud object storage platforms based on your deployment.
For REST-based namespace servers, you can specify the namespace implementation as `"rest"` with forwarding prefixed headers
for authentication and context propagation.
```toml theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
[federated_dbs.federated_rest_test]
ns_impl = "rest"
uri = "http://.internal.catalog.com"
forward_header_prefixes = ["X-forward"]
```
With `forward_header_prefixes = ["X-forward"]`, any incoming header starting with `X-forward` is forwarded to
`http://.internal.catalog.com`. This is useful for auth propagation, for example sending
`X-forward-authorization: Bearer xxxx`.
For the LanceDB REST API itself, requests use `x-api-key` for API-key authentication. If your endpoint
serves more than one database, LanceDB can also use headers such as `x-lancedb-database` or
`x-lancedb-database-prefix` to route the request to the right database context.
## Related references
* [Client SDK API references](/api-reference)
* [REST API Reference](/api-reference/rest)
* [Namespaces and the Catalog Model](/namespaces)
# Performance Tips and Best Practices
Source: https://docs.lancedb.com/performance
Optimize LanceDB for your workload across ingestion, indexing, querying, and maintenance.
LanceDB is performant by default. This page covers performance best practices that matter when you want to ensure you get the right performance for a specific workload. Use the table below to jump to the area relevant to what you're working on.
| When you're working on... | Read |
| -------------------------------------------------------------- | --------------------------- |
| Loading data into a table | [Ingestion](#ingestion) |
| Running filtered or vector queries at scale | [Indexing](#indexing) |
| Iterating over large result sets (training, export, migration) | [Querying](#querying) |
| Keeping a long-lived dataset healthy | [Maintenance](#maintenance) |
| Inspecting query plans | [Diagnostics](#diagnostics) |
When using Python with multiprocessing, use `spawn` rather than `fork`. LanceDB is multi-threaded internally, and `fork` plus a multi-threaded process is unsafe.
## Ingestion
If ingestion is taking longer than expected on a large dataset, the cause is almost always how `add()` is called: each call commits a new version and a new fragment, so a per-row loop pays that per-call overhead at every row. The best practice is to pick the ingestion mode that matches your data shape — bulk ingestion when the data is already materialized, or iterator ingestion when it's streamed or computed on the fly.
**Why `merge_insert()` is significantly slower than `add()`**
A merge has to scan existing data to find matches on the join key (or look them up via a scalar index, if one exists), and then delete-and-reinsert any updated rows in a single transaction; `add()` simply appends new fragments.
Use `add()` for pure appends, and reach for `merge_insert()` only when you need upsert or conditional-insert logic. When you do use it, build a scalar index on the join column first — otherwise the match step falls back to a full column scan, which is the dominant cost at scale.
### Bulk ingestion: for data you already have
For materialized inputs (Arrow Tables, DataFrames) and file-backed sources (`pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers, estimating the partition count from the data size — more partitions means more concurrent writes and higher throughput, up to the CPU core count.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.add(arrow_table) # in-memory
table.add(df) # pandas
table.add(ds.dataset("data/", format="parquet")) # streams from disk, still parallelized
```
Pass `progress=True` to watch it happen: LanceDB shows a live tqdm bar with rows written, throughput in MB/s, and active worker count.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.add(ds.dataset("data/", format="parquet"), progress=True)
```
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
71%|███████▏ | 710000/1000000 [00:12<00:05, 58.8kit/s, 42.3 MB/s | 8/8 workers]
```
For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over a hand-built `pyarrow.RecordBatchReader`: a `Dataset` can be counted and rescanned, so LanceDB knows the row count upfront (better auto-parallelism) and can retry a failed write from the start — a reader can only be consumed once, so neither is possible. Reach for the iterator path below only when the data genuinely can't be backed by files, e.g. to apply custom data transformations as you ingest.
For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path.
### Iterator ingestion: for data you transform on the fly
If each row needs work before it can be written — applying a custom transformation as you ingest, for example — the data doesn't exist as a file you can point `ds.dataset(...)` at. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead; LanceDB consumes one batch at a time as you produce them.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
def stream():
for raw in source:
vectors = model.encode(raw["text"])
yield pa.RecordBatch.from_pydict({**raw, "vector": vectors})
table.add(stream())
```
Use decently large chunks of several thousand rows or more, rather than yielding single-row batches.
**Set `write_parallelism` yourself for large inputs**
A reader can't be counted or rescanned the way a `Dataset` can (see above), so LanceDB can't auto-size parallelism for it — set `write_parallelism` explicitly for large inputs:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.add(stream(), write_parallelism=4)
```
Each partition becomes its own fragment, so don't over-allocate on a small input — budget one unit of parallelism per \~100K rows or \~1 GB of data as a rule of thumb.
### Bulk ingests into a remote table
Enterprise
This section applies only to remote tables (`db://` connections). Embedded (local) writes are unaffected.
When you write to a remote LanceDB Enterprise table, the client splits each write partition into one or more HTTP `insert` parts and uploads them under a single upload id. Each part is streamed rather than buffered, so peak memory stays bounded. But each part still has to finish within the client read timeout while the server writes it to object storage, so on very large ingests an oversized part can run past that timeout and surface as:
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
lancedb.remote.errors.HttpError: operation timed out
```
Two environment variables control how parts are cut. Both are picked up automatically by the Python and TypeScript clients, and are the only way to tune this from those SDKs:
| Variable | Default | What it controls |
| ------------------------------------ | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` | 8 GiB | Maximum size of a single insert part, in LZ4-compressed Arrow IPC bytes. Set to `0` to disable splitting and send one request per partition. |
| `LANCE_CLIENT_MAX_REQUEST_DURATION` | Half the read timeout | Maximum wall-clock time, in whole seconds, that any one insert part may stay open. Set to `0` to disable the time-based cut. |
A part is cut when it hits either limit, whichever comes first. Lower the byte budget when large ingests hit the read timeout; lower the duration when uploads are slow or throttled and the byte budget isn't the limiting factor.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
export LANCE_CLIENT_MAX_BYTES_PER_REQUEST=1073741824 # 1 GiB
export LANCE_CLIENT_MAX_REQUEST_DURATION=120 # 2 minutes
```
Splitting into more parts does not change the final table. The server stages every part under the shared upload id and merges them atomically when the write completes.
## Indexing
### Vector indexes
If vector search latency climbs with table size (i.e., queries that ran in milliseconds on a small table take seconds as it grows to millions of rows), the cause is the default brute-force scan over every vector. That works fine below \~100K vectors, but past that you should build a dedicated vector index. Pick the type by your data shape:
| Index | When to use |
| ------------- | ------------------------------------------------------------------------------------------------------------------- |
| `IVF_PQ` | General-purpose default; what Enterprise builds automatically. Higher accuracy than RQ at small dimensions (≤ 256). |
| `IVF_RQ` | Maximum compression on high-dim vectors, faster builds than PQ. |
| `IVF_HNSW_SQ` | Best recall/latency for unfiltered search; higher latency variance under selective filters. |
| `IVF_FLAT` | Required for binary vectors with `hamming`. |
The distance metric is fixed once the index is built. Pick the distance metric based on how the embedding model was trained: `cosine` (unnormalized), `dot` (already-normalized, best performance), `l2` (general-purpose, default), `hamming` (binary). For parameter tuning, see [Vector Indexing](/indexing/vector-index).
### Scalar indexes
If filtered queries slow down as the table grows — even when the filter is selective — the cause is a full column scan: without a scalar index, LanceDB evaluates the `where(...)` predicate on every row, and the same applies to `merge_insert` join keys. The best practice is to build a scalar index on every column you filter or join on, picking the type by the column's shape:
| Index | Best for |
| ----------------- | ---------------------------------------------------------------- |
| `BTREE` (default) | Numeric, string, temporal columns with mostly distinct values |
| `BITMAP` | Boolean and low-cardinality columns (\< \~1,000 distinct values) |
| `LABEL_LIST` | `List` columns queried with `array_has_any` / `array_has_all` |
See [Scalar Indexing](/indexing/scalar-index).
### Full-text search
If your full-text index is much larger than expected, or takes longer than expected to build, the cause is usually phrase-query flags being enabled when they aren't needed: `with_position=True` and `remove_stop_words=False` both significantly inflate index size and build time. The best practice is to keep the defaults for most workloads, and only enable those flags when you actually need to search for phrases. See [FTS Indexing](/indexing/fts-index) configuration options for the full set of options.
## Compaction and cleanup
Two things accumulate on a long-lived table as more and more data gets added to it:
* **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
* **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.
In OSS, you run them yourself via `optimize()`, which bundles both into a single call:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from datetime import timedelta
table.optimize() # 7-day default retention
table.optimize(cleanup_older_than=timedelta(days=1)) # reclaim space sooner
```
The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Reindexing](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches.
LanceDB Enterprise handles both compaction and cleanup automatically.
## Querying
Three knobs materially affect query latency, memory use, and recall. Be deliberate about each one on every query:
If queries return more data than you need or take longer than expected, the cause is usually projecting more columns than necessary, or letting the result count run unbounded. The best practice is to always pass both `select()` and `limit()`:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.search(emb).select(["id", "title"]).limit(20)
```
If a query with `prefilter=False` returns fewer than `limit` results — sometimes zero — the cause is that post-filter applies the predicate after the top-k is selected, so only candidates that already passed the search are filtered. Pre-filter is the default and guarantees every result satisfies the predicate. Switch to `prefilter=False` only when fewer-than-`limit` results are acceptable. See [Filtering](/search/filtering).
If recall is lower than expected, the cause is that the search-time knobs limit how many candidates the index considers. Adjust based on your index type:
* Quantized indexes (PQ, RQ, SQ): `refine_factor` pulls extra candidates and re-scores them on full vectors.
* HNSW-backed indexes: `ef` at search time. Start at `1.5 × k`, raise toward `10 × k` if recall is short.
* IVF candidate breadth: `nprobes` is auto-tuned; override only when a selective pre-filter leaves too few neighbors.
For hybrid search, the default `RRFReranker()` combines vector and FTS results into a single ranking via reciprocal rank fusion. See [Hybrid Search](/search/hybrid-search).
### Avoid materializing the whole table
If you need every row in the table (for training, export, or migration), calling `to_pandas()` or `to_arrow()` will run you out of memory on any non-trivial dataset — both materialize the full table at once. The best practice is to iterate via `table.search(...)` or `table.query(...)`, which work the same way in both LanceDB OSS and Enterprise.
In OSS, you can also stream batches through the underlying Lance dataset directly — useful when you need filter pushdown or fragment-level parallelism via `ds.scanner(...)`:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
ds = table.to_lance()
for batch in ds.to_batches(columns=["id", "text"], batch_size=10000):
process(batch)
```
**API disparity between OSS and Enterprise.**
LanceDB OSS exposes the `to_pandas()`, `to_arrow()`, and `table.to_lance()` methods for direct Lance dataset access. Enterprise's `RemoteTable` exposes none of these. Only `table.search(...)` and `table.query(...)`. In general, it's always best to go through `search()` and `query()` to keep your code portable across both OSS and Enterprise.
### Diagnostics
To analyze a slow query, inspect what the query engine actually did and the state of the indexes it touched. These two tools surface that information — use them in this order:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
print(table.search(emb).where("year > 2000").limit(10).analyze_plan())
print(table.index_stats("vector_idx")) # num_unindexed_rows should be ~0
```
`analyze_plan()` returns the execution plan with per-stage timings, so you can see where the query actually spent its time. `index_stats()` shows how many rows are still unindexed — you want `num_unindexed_rows` to be \~0. In `analyze_plan`, look for:
| Plan pattern | Fix |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `LanceScan` with high `bytes_read` / `iops` | Add a missing index, project columns with `select()`, or check that the dataset has been compacted |
| Multiple sequential filters | Reorder filter conditions |
[Optimize Query Performance](/search/optimize-queries) walks through a fully worked before/after example, including how `KNNVectorDistance` and `output_batches` change once indexes are in place.
## Where to go next
Read execution plans, find the bottleneck.
Index types, parameters, and tuning in depth.
Pre- vs post-filter, scalar indexes, list columns.
Benchmark methodology and reference latency numbers.
# Quickstart
Source: https://docs.lancedb.com/quickstart
Get started with LanceDB in minutes.
As described in [the landing page](/), LanceDB provides one data layer for
curation, feature engineering, search and retrieval, and model training. Whether you are preparing
training data, building a RAG or agentic retrieval system, reviewing examples, or adding model-generated
features, you'll work with the same underlying table and search primitives.
Let's get started in just a few steps!
## 1. Install LanceDB
Install LanceDB in your client SDK.
```bash pip icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install lancedb
```
```bash uv icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv add lancedb
# Or, in an existing virtual environment:
uv pip install lancedb
```
```bash TypeScript icon=js theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
npm install @lancedb/lancedb
```
```bash Rust icon=Rust theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
cargo add lancedb
```
### Python pre-release builds
To pick up the latest features and bug fixes
before the next stable release, install a pre-release from LanceDB's Fury index.
```bash pip icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
```
```bash uv icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv venv
uv pip install --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
# To add to pyproject.toml, use:
uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
```
Pre-release builds receive the same level of testing as stable releases, but their availability is not guaranteed
for more than 6 months after release. For real-world workloads, we recommend you use the latest stable release
as far as possible.
The default `lancedb` wheel targets `x86-64-haswell` (AVX2 + FMA + F16C). On older x86\_64 CPUs without AVX2
(Intel Sandy Bridge, Ivy Bridge, Westmere; AMD Bulldozer, Piledriver, Steamroller), `import lancedb` crashes
with `Illegal instruction`.
Install `lancedb-compat` instead. It exposes the same API (`import lancedb` still works), is built at the
`x86-64-v2` baseline, and uses runtime SIMD dispatch to still leverage AVX2, FMA, or AVX-512 when available:
```bash pip icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install lancedb-compat
```
The two packages share the same `lancedb/` namespace and conflict at install time, so pick one. To switch,
uninstall the other first (`pip uninstall lancedb && pip install lancedb-compat`).
## 2. Connect to a LanceDB database
LanceDB supports several URI patterns to connect to a database.
* A local filesystem path (when using it as an embedded library)
* A `db://...` URI (when using LanceDB Enterprise)
* An object storage URI: `s3://...`, `gs://...`, or `az://...` (when connecting directly from the client SDK)
### Connect via local directory path
The simplest way to begin is to use LanceDB as an embedded library. Import LanceDB in your
client SDK of choice and point to a local directory path.
### Connect via object storage URIs
You can also connect directly to object storage from the client SDK:
For credentials, endpoints, and provider-specific options, see
[Configuring storage](/storage/configuration).
### Connect to LanceDB Enterprise
If you're using LanceDB Enterprise, you can connect to the remote database using the
`db://` URI along with the API key, region, and cluster endpoint you received from the
LanceDB team. Pass the cluster endpoint via `host_override` so the client routes
requests to your deployment.
`host_override` is the full URL of your cluster endpoint, including the scheme
(`https://`) and a port if your deployment listens on a non-default one
(e.g. `https://your-enterprise-endpoint.com:443`). If you don't have the
endpoint, [contact the LanceDB team](mailto:contact@lancedb.com).
To learn more about `RemoteTable` semantics and how Enterprise differs operationally from
embedded LanceDB, see the [Enterprise overview](/enterprise).
## 3. Create a new table
Let's create a small table of characters from the kingdom of Camelot. Each row stores source text,
metadata, structured fields, and a vector embedding in the same LanceDB table.
The embeddings we use in this example are synthetic and for demonstration purposes only. In a real AI
data workflow, you would generate them from text, images, audio, or video using an embedding model of choice.
Each row has source text, metadata, structured fields, and a vector:
```json theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
{
"id": "2",
"name": "Merlin",
"role": "Wizard",
"description": "Advisor and prophet with deep magical knowledge.",
"stats": { "strength": 2, "magic": 5, "leadership": 4, "wisdom": 5 },
"vector": [0.2, 0.9, 0.4, 0.9]
}
```
The full raw records are included below:
You can now create a LanceDB table from those records. The code below creates a LanceDB table
with the appropriate schema and ingests the data.
## 4. Semantic search
Search is a useful capability for all kinds of AI data pipelines. Below, we do a vector similarity
search for samples similar to a "*wise magical advisor*" (transforming the natural language query to
an embedding), and project only the columns needed by the next step.
Search (which requires random access) is a ubiquitous access pattern that appears in many workloads:
whether you're building a RAG or recommendation system, serving agent memory, or curating a training
dataset.
The example for Python above shows how to convert results to a Polars DataFrame.
Depending on your language, you can collect query results as a list/array of objects or DataFrames
to be used downstream in your application.
Use the `to_pandas()` method to convert query results into a Pandas DataFrame.
## 5. Curation
Searching for relevant results can be more useful when combined with metadata filters.
In this tiny example, we filter to examples with high `magic` stats.
When working with large datasets, it's common to use the same pattern to filter on quality labels,
train/eval splits, numeric fields, categorical values, timestamp windows, or generated tags and labels.
## 6. Add a derived feature
Feature engineering is the process of cleaning up your data and creating new signals that
help your model learn, make better predictions, or your agent retrieve more useful information.
In the example below, we add a `power_score` column from the structured `stats` fields.
Lance supports data evolution, so you can add new columns without rewriting the entire table.
Next, you can query a compact view of the new feature:
| name | role | power\_score |
| ------------ | ------ | ------------ |
| King Arthur | King | 3.5 |
| Merlin | Wizard | 4.0 |
| Sir Lancelot | Knight | 3.0 |
The same workflow is used for data preparation tasks when adding derived features, cached model signals, review scores, or dataset
quality indicators.
## 7. Store multimodal data
Multimodal data is a first-class citizen in LanceDB. Binary data (image, audio, video, etc.) is
stored as blobs or inline Arrow binary types in a LanceDB column, and they benefit from the same
table operations and data versioning semantics as other data types. All the data is governed
in the same table, so you can search, filter, and retrieve multimodal records together with structured
fields, metadata, and embeddings.
In this example, the
[`lancedb/magical_kingdom`](https://huggingface.co/datasets/lancedb/magical_kingdom) dataset stores
character images, descriptions, structured stats, image embeddings, and text embeddings together.
Say we downloaded the image for Sir Lancelot from that dataset locally. You can read the image bytes
in your client SDK and store them in a LanceDB column. The image bytes can be used for downstream tasks
like retrieval, evaluation, or training.
These snippets load the local image file and store the bytes in an `image` column:
For more examples, see the [multimodal data](/tables/multimodal) section.
## Code
See the full code for these examples (including helper functions) in the
`quickstart` file for the appropriate client language in the
[files provided in the repo](https://github.com/lancedb/docs/tree/main/tests).
## What's next?
You've learned how to install LanceDB, connect, create one table for AI data, retrieve related
examples, curate with metadata, add a derived feature, and represent multimodal records. These same
primitives apply across the AI data lifecycle, from data preparation and feature engineering to
retrieval, evaluation, and training.
Continue to the table and search guides to build on this example with schema options, appends,
updates, versioning, indexing, full-text search, hybrid search, and reranking.
Build on this quickstart with table creation, updates, and schema tips.
Learn how to build Retrieval-Augmented Generation (RAG) applications using LanceDB.
Create vector, full-text, and scalar indexes to speed up queries on larger datasets.
Use LanceDB for projected, shuffled, random-access reads in training workflows.
# Cross Encoder Reranker
Source: https://docs.lancedb.com/reranking/cross_encoder
Implement semantic search reranking in LanceDB using Cross Encoder models. Features configurable model selection, device optimization, and comprehensive scoring options for all search types.
# Cross Encoder Reranker
This reranker uses Cross Encoder models from sentence-transformers to rerank the search results. You can use this reranker by passing `CrossEncoderReranker()` to the `rerank()` method.
> **Note:** Supported query types – Hybrid, Vector, and FTS.
## Accepted Arguments
| Argument | Type | Default | Description |
| ------------------- | ------ | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_name` | `str` | `"cross-encoder/ms-marco-TinyBERT-L-6"` | The name of the reranker model to use. |
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `device` | `str` | `None` | The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu". |
| `trust_remote_code` | `bool` | `True` | Passed to Sentence Transformers model loading. Set this to `False` when you only want to load models that do not require custom repository code. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all" is supported, returns relevance score along with the vector and/or FTS scores depending on query type. |
The reranker loads the model locally through `sentence-transformers`, so install the local model
runtime dependencies you need, such as PyTorch and any device-specific acceleration packages.
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | --------------- | ----------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ❌ Not Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
### Vector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ------------------------------------------------------------------------------------ |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have vector(`_distance`) along with Hybrid Search score(`_relevance_score`). |
### FTS Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
# Building Custom Rerankers
Source: https://docs.lancedb.com/reranking/custom-reranker
Learn how to create custom rerankers in LanceDB by extending the base Reranker class.
You can build your own custom reranker in LanceDB by subclassing the base `Reranker` class. At a
minimum, you need to implement `rerank_hybrid()`, which is the logic that combines vector and
full-text search results. Beyond that, you can optionally implement `rerank_vector()` and
`rerank_fts()` if you want your reranker to also handle pure vector or pure full-text searches.
Decide up front which surfaces — hybrid, pure vector, or pure full-text — your reranker should
cover, and only override the ones you need. The base class leaves `rerank_vector()` and
`rerank_fts()` unimplemented, so calling `.rerank(...)` on a single-modality search you haven't
overridden raises `NotImplementedError` rather than silently returning unsorted results. That's a
useful guard, but worth knowing about before you wire up a query path you didn't plan for.
The Python base class exposes hybrid, vector-only, and FTS-only rerank hooks. TypeScript and Rust
currently expose the custom reranker interface for hybrid reranking. In Rust, a custom reranker must
also satisfy the trait bounds `Debug + Send + Sync`.
## Interface
The `Reranker` base interface comes with a `merge_results()` method that can be used to combine the
results of semantic and full-text search. This is a vanilla merging algorithm that simply concatenates
the results and removes the duplicates without taking the scores into consideration. It only keeps the
first copy of the row encountered. This works well in cases that don't require the scores of semantic
and full-text search to combine the results. If you want to use the scores or want to support
`return_score="all"`, you'll need to implement your own merging algorithm. The base
`return_score` option accepts only `"relevance"` and `"all"`.
Whichever methods you override, your reranker has one job on the way out: attach a
`_relevance_score` column with the most relevant rows at the top. LanceDB will reject the result
if that column is missing, and downstream `.limit(...)` calls trust the order you return, so
sort descending before handing the table back.
For vector-only reranking in Python, pass a text query to `.rerank(..., query_string="...")`.
The vector query itself may be numeric, but `rerank_vector(query, vector_results)` still receives
a string query so your reranker can score each candidate against the user's text intent.
Below, we show the pseudocode of a custom reranker that combines the results of semantic and full-text
search using a linear combination of the scores:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.rerankers import Reranker
import pyarrow as pa
class MyReranker(Reranker):
def __init__(self, param1, param2, ..., return_score="relevance"):
super().__init__(return_score)
self.param1 = param1
self.param2 = param2
def rerank_hybrid(self, query: str, vector_results: pa.Table, fts_results: pa.Table):
# Use the built-in merging function
combined_result = self.merge_results(vector_results, fts_results)
# Do something with the combined results
# ...
# Return the combined results
return combined_result
def rerank_vector(self, query: str, vector_results: pa.Table):
# Do something with the vector results
# ...
# Return the vector results
return vector_results
def rerank_fts(self, query: str, fts_results: pa.Table):
# Do something with the FTS results
# ...
# Return the FTS results
return fts_results
```
## Example
As an example, let's build custom reranker that enhances the Cohere Reranker by accepting a filter
query, and accepts any other `CohereReranker` params as `kwargs`.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from typing import List, Union
import pandas as pd
from lancedb.rerankers import CohereReranker
class ModifiedCohereReranker(CohereReranker):
def __init__(self, filters: Union[str, List[str]], **kwargs):
super().__init__(**kwargs)
filters = filters if isinstance(filters, list) else [filters]
self.filters = filters
def rerank_hybrid(self, query: str, vector_results: pa.Table, fts_results: pa.Table)-> pa.Table:
combined_result = super().rerank_hybrid(query, vector_results, fts_results)
df = combined_result.to_pandas()
for filter in self.filters:
df = df.query("not text.str.contains(@filter)")
return pa.Table.from_pandas(df)
def rerank_vector(self, query: str, vector_results: pa.Table)-> pa.Table:
vector_results = super().rerank_vector(query, vector_results)
df = vector_results.to_pandas()
for filter in self.filters:
df = df.query("not text.str.contains(@filter)")
return pa.Table.from_pandas(df)
def rerank_fts(self, query: str, fts_results: pa.Table)-> pa.Table:
fts_results = super().rerank_fts(query, fts_results)
df = fts_results.to_pandas()
for filter in self.filters:
df = df.query("not text.str.contains(@filter)")
return pa.Table.from_pandas(df)
```
Under the hood, `vector_results` and `fts_results` are PyArrow tables. You can learn more about
PyArrow tables [here](https://arrow.apache.org/docs/python). The advantage of PyArrow tables is their
interoperability -- you can easily convert them to Pandas/Polars DataFrames, `PyDict`, `PyList`, etc.
The benefits are also bidirectional -- just as you can easily convert PyArrow tables *to* Pandas
DataFrames using the `to_pandas()` method -- you can perform DataFrame transformations
and just as easily convert the DataFrame back to PyArrow tables using `pa.Table.from_pandas()` method
as shown in the example above.
# Evaluating Hybrid Search Performance
Source: https://docs.lancedb.com/reranking/eval
Learn about evaluating hybrid search performance in LanceDB.
Hybrid search is an often misused and/or misunderstood term. In this section, we're using
the definition of "hybrid search" to mean using a combination of keyword-based and vector search.
Because the vector search operates in a dense embedding space and keyword-based search operate
in a sparse embedding space, their relevance scores cannot be directly compared.
Combining results from multiple searches thus requires a reranking step.
Before evaluating hybrid search, build an FTS index on the text column and use a table with embedding
metadata or explicit query vectors for the vector side. Otherwise the evaluation is measuring setup
fallbacks or errors rather than the reranker.
## Reranking strategies
There are two common approaches for reranking search results from multiple sources.
* **Score-based**: Calculate final relevance scores from the individual search algorithm scores. Examples: Reciprocal Rank Fusion (the default in LanceDB), mean reciprocal rank fusion, and weighted linear combination of semantic and keyword-based search scores.
* **Relevance-based**: Discards the existing scores and calculates the relevance of each search result-query pair. Example: Cross Encoder models
If you call `.rerank()` on a hybrid query without passing a reranker, LanceDB defaults to
`RRFReranker()` — a score-based reranker that uses Reciprocal Rank Fusion. This is the
score-based path most readers encounter first; `LinearCombinationReranker` is an alternative
score-based strategy you opt into explicitly.
By default, rerankers return `_relevance_score`. Pass `return_score="all"` when a reranker
supports it, and you also need the original vector or FTS scores for debugging.
Evaluation code can rely on returned rows being ordered by descending `_relevance_score`. Empty
reranked result sets still include the `_relevance_score` column.
The hybrid `rerank(...)` method also accepts a `normalize` argument that controls how the raw
vector and FTS scores are made comparable before reranking:
* `normalize="score"` (the default) — normalizes the raw vector and FTS scores directly.
* `normalize="rank"` — converts each result list to ranks first, then normalizes.
This choice materially affects score-based rerankers (such as `LinearCombinationReranker`), so
when you evaluate score-based strategies, treat `normalize` as a tunable hyperparameter
alongside the reranker itself.
For score-based evaluation, the main built-in knobs are:
* `RRFReranker(K=60)`: rank fusion with a positive smoothing constant.
* `MRRReranker(weight_vector=0.5, weight_fts=0.5)`: weighted reciprocal rank fusion where the weights must sum to `1.0`.
* `LinearCombinationReranker(weight=0.7, fill=1.0)`: blends vector and FTS scores directly; `fill` controls how strongly to penalize a result missing from one side.
Even though there are many more strategies for reranking, there are no "universally best"
ones that work well for all cases, because reranking quality is dataset and application specific.
Evaluating whether a reranking strategy is a good fit is also a challenge. In the next
section, we discuss an example evaluation of different reranking strategies on a sample dataset.
## Example evaluation
The table below shows our evaluation results from an experiment comparing multiple rerankers on
\~800 hybrid search queries. This is a modified version of an evaluation script by
[LlamaIndex](https://github.com/run-llama/finetune-embedding/blob/main/evaluate.ipynb) that measures
hit-rate @ top-k.
### Using OpenAI `text-embedding-ada-002`
Vector Search baseline: **0.64**
| Reranker | Top-3 | Top-5 | Top-10 |
| ------------------ | ------ | ------ | ------ |
| Linear Combination | `0.73` | `0.74` | `0.85` |
| Cross Encoder | `0.71` | `0.70` | `0.77` |
| Cohere | `0.81` | `0.81` | `0.85` |
| ColBERT | `0.68` | `0.68` | `0.73` |
![]()
### Using OpenAI `text-embedding-3-small`
Vector Search baseline: **0.59**
| Reranker | Top-3 | Top-5 | Top-10 |
| ------------------ | ------ | ------ | ------ |
| Linear Combination | `0.68` | `0.70` | `0.84` |
| Cross Encoder | `0.72` | `0.72` | `0.79` |
| Cohere | `0.79` | `0.79` | `0.84` |
| ColBERT | `0.70` | `0.70` | `0.76` |
![]()
## Conclusion
The results show that the reranking methods can significantly improve the search relevance. However,
the improvement we saw was not consistent across all rerankers. In reality, the choice of reranker
likely depends on the dataset and the application.
It's also important to note that the reranking methods are not a
replacement for the search methods they supplement. They are complementary and it's likely that you'd
have to tune them together to get the best results. The latency vs. recall tradeoff is also an
important factor to consider when choosing the reranker. Hopefully this evaluation
gives you a starting point for your own experiments with hybrid search in LanceDB!
# Reranking Search Results
Source: https://docs.lancedb.com/reranking/index
Use a reranker to improve search relevance.
Reranking is the process of re-ordering search results to improve relevance, often using a
different model than the one used for the initial search. LanceDB has built-in support for reranking
with models from Cohere, Sentence-Transformers, and more.
### Quickstart
To use a reranker, you run a search and pass the results to the `rerank()` method. The examples below
move from the simplest, model-free rerankers to a model-based one. Each is a complete, runnable script.
**1. Linear combination (simplest).** `LinearCombinationReranker` normalizes the vector and full-text
scores and blends them with a single `weight` (default `0.7`, favoring the vector score). It runs no
model, and reranks [hybrid search](/search/hybrid-search) results.
**2. Reciprocal Rank Fusion.** `RRFReranker` fuses results by rank position instead of raw score, so it
sidesteps having to make vector and full-text scores comparable. It loads no model either, and it's the
default reranker for hybrid search.
**3. Cohere (model-based).** For higher relevance, a model-based reranker scores each result against the
query with a trained model. `CohereReranker` works with vector, full-text, or hybrid search, and needs
the `cohere` package plus either `COHERE_API_KEY` in the environment or an `api_key` argument.
Reach for the model-free rerankers (`LinearCombinationReranker`, `RRFReranker`) when cost and latency
matter most; reach for a model-based one like `CohereReranker` or `CrossEncoderReranker` when you need
higher relevance and can afford to score every query and document pair with a model.
### Supported Rerankers
LanceDB supports the following rerankers out of the box. The first three are score-based and run no
model; the rest are model-based. The built-in rerankers are documented in this section; the hosted
providers that need an API key live under [integrations](/integrations/reranking).
| Reranker | Default model |
| --------------------------- | --------------------------------------- |
| `RRFReranker` | None (reciprocal rank fusion) |
| `LinearCombinationReranker` | None (weighted score blend) |
| `MRRReranker` | None (weighted reciprocal rank) |
| `CohereReranker` | `rerank-english-v3.0` |
| `CrossEncoderReranker` | `cross-encoder/ms-marco-TinyBERT-L-6` |
| `ColbertReranker` | `colbert-ir/colbertv2.0` |
| `AnswerdotaiRerankers` | `answerdotai/answerai-colbert-small-v1` |
| `JinaReranker` | `jina-reranker-v2-base-multilingual` |
| `OpenaiReranker` | `gpt-4-turbo-preview` |
| `VoyageAIReranker` | No default (model name required) |
| `WatsonxReranker` | `cross-encoder/ms-marco-minilm-l-12-v2` |
The model-based rerankers need their provider package installed, and the hosted ones
(`CohereReranker`, `JinaReranker`, `OpenaiReranker`, `VoyageAIReranker`, `WatsonxReranker`) also need an API key, passed
as an `api_key` argument or set in the provider-specific environment variable.
Rerankers add `_relevance_score` and return rows ordered by descending relevance. Python rerankers
accept `return_score="relevance"` or `return_score="all"`; use `"all"` when you want to keep the
original vector distance or FTS score columns for debugging. Model-based rerankers read from
`column="text"` by default, so either return that column in the search results or pass a different
column.
Use `refine_factor` on vector or hybrid queries when you're reranking approximate IVF-PQ results and
want a larger candidate pool before the final ranking step. A value of `3` asks LanceDB to fetch
`limit * 3` candidates, refine them with the full vectors, and keep the requested `limit`. Higher
values can improve recall, but they also increase query latency.
**SDK coverage differs across languages**
The provider-specific rerankers in the table above
(`CohereReranker`, `CrossEncoderReranker`, `ColbertReranker`, and others under `lancedb.rerankers`)
are currently **Python-only**. The TypeScript and Rust SDKs currently expose hybrid reranking through
the generic `Reranker` interface (`rerankHybrid` / `rerank_hybrid`) and the built-in `RRFReranker`.
In TypeScript, create the built-in RRF reranker with `await RRFReranker.create(k)`. To use a
model-based reranker from TypeScript or Rust, you must implement the hybrid reranker interface
yourself.
### Multi-vector reranking
Most rerankers support reranking based on multiple vectors. To rerank based on multiple vectors, you can pass a list of vectors to the `rerank` method. Here's an example of how to rerank based on multiple vector columns using the `CrossEncoderReranker`:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.rerankers import CrossEncoderReranker
reranker = CrossEncoderReranker()
query = "hello"
# `deduplicate=True` requires `_rowid` on every input result set,
# so call `.with_row_id(True)` on each search before passing it in.
res1 = table.search(query, vector_column_name="vector").limit(3).with_row_id(True)
res2 = table.search(query, vector_column_name="text_vector").limit(3).with_row_id(True)
res3 = table.search(query, vector_column_name="meta_vector").limit(3).with_row_id(True)
reranked = reranker.rerank_multivector([res1, res2, res3], deduplicate=True)
```
* Passing `deduplicate=True` to `rerank_multivector(...)` raises a `ValueError` if any of the
input result sets is missing the `_rowid` column. Therefore, it's recommended to add `.with_row_id(True)` to every
`table.search(...)` call before reranking, or omit `deduplicate=True` if you don't need it.
* `RRFReranker.rerank_multivector(...)` always requires `_rowid` on its inputs, regardless of
the `deduplicate` flag.
## Creating Custom Rerankers
LanceDB also allows you to create custom rerankers by extending the base `Reranker` class. The custom reranker
should implement the `rerank` method that takes a list of search results and returns a reranked list of
search results. This is covered in more detail in the [creating custom rerankers](/reranking/custom-reranker/) section.
# Linear Combination Reranker
Source: https://docs.lancedb.com/reranking/linear_combination
Learn about LanceDB's deprecated Linear Combination Reranker for combining semantic and full-text search scores.
# Linear Combination Reranker
> **Note:** This reranker is deprecated. Use the `RRFReranker` if you need a score-based reranker.
The Linear Combination Reranker combines the results of semantic and full-text search using a linear combination of the scores. The weights for the linear combination can be specified, and defaults to 0.7, i.e, 70% weight for semantic search and 30% weight for full-text search.
> **Note:** Supported query type – Hybrid search only.
## Accepted Arguments
| Argument | Type | Default | Description |
| -------------- | ------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `weight` | `float` | `0.7` | The weight to use for the semantic search score. The weight for the full-text search score is `1 - weight`. |
| `fill` | `float` | `1.0` | Score used when a result is missing from one side of the hybrid query. The default strongly penalizes missing vector or FTS matches. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all", returns all scores from the vector and FTS search along with the relevance score. |
`weight` must be between `0` and `1`. If either the vector or FTS side returns no rows, the reranker
returns the non-empty side with `_relevance_score` attached rather than failing.
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | ----------- | --------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column |
| `all` | ✅ Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_distance`) |
# MRR Reranker
Source: https://docs.lancedb.com/reranking/mrr
Combine and rerank search results using Mean Reciprocal Rank (MRR) algorithm in LanceDB. Supports weighted scoring for hybrid and multivector search.
# MRR Reranker
This reranker uses the Mean Reciprocal Rank (MRR) algorithm to combine and rerank search results from vector and full-text search. You can use this reranker by passing `MRRReranker()` to the `rerank()` method. The MRR algorithm calculates the average of reciprocal ranks across different search results, providing a balanced way to merge results from multiple ranking systems.
> **Note:** Supported query types – Hybrid and Multivector search.
## Accepted Arguments
| Argument | Type | Default | Description |
| --------------- | ------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `weight_vector` | `float` | `0.5` | Weight for vector search results (0.0 to 1.0). |
| `weight_fts` | `float` | `0.5` | Weight for FTS search results (0.0 to 1.0). |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score`. If "all", will return all scores from the vector and FTS search along with the relevance score. |
**Note:** `weight_vector` + `weight_fts` must equal 1.0.
For multivector reranking, input result sets need `_rowid` so LanceDB can identify the same row
across the ranked lists. Add `.with_row_id(True)` to each vector search before passing the results
to the reranker.
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
### Multivector Search
| `return_score` | Status | Description |
| -------------- | ----------- | ------------------------------------------------------------------------------ |
| `relevance` | ✅ Supported | Results only have the `_relevance_score` column. |
| `all` | ✅ Supported | Results have vector distances from all searches along with `_relevance_score`. |
# Reciprocal Rank Fusion Reranker
Source: https://docs.lancedb.com/reranking/rrf
Learn about LanceDB's default Reciprocal Rank Fusion (RRF) reranker for hybrid search. Implements the Cormack et al. algorithm for optimal search result ranking.
# Reciprocal Rank Fusion Reranker
**Reciprocal Rank Fusion (RRF)** is a model-free way to merge several ranked result lists into a
single ordering. Rather than comparing raw similarity scores (which aren't directly comparable
across, say, a vector search and a full-text search), RRF looks only at each document's *rank
position* in each list. It scores every document with the formula `1 / (rank + K)`, sums those
contributions across the lists, and re-sorts by the total. Documents that rank highly in more than
one search rise to the top. Because there's no model to load or call, it's fast and cheap, which is
why it's the default reranker for LanceDB hybrid search. The implementation follows the
[Cormack et al. paper](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf).
> **Supported query types:** hybrid search and [multi-vector reranking](/reranking#multi-vector-reranking).
> Because RRF fuses two or more ranked lists, it can't rerank a single vector or full-text result set
> on its own. Calling `rerank_vector` or `rerank_fts` on an `RRFReranker` raises `NotImplementedError`.
## Accepted Arguments
| Argument | Type | Default | Description |
| -------------- | ----- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `K` | `int` | `60` | A constant used in the RRF formula (default is 60). Experiments indicate that k = 60 was near-optimal, but that the choice is not critical. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score`. If "all", will return all scores from the vector and FTS search along with the relevance score. |
`K` must be greater than `0`. In TypeScript, construct the built-in reranker with
`await RRFReranker.create(k)` before passing it to `.rerank(...)`.
## Multi-vector reranking
`RRFReranker` can also fuse the results of several vector searches with `rerank_multivector`, applying
the same rank-fusion algorithm across more than two lists. Every input result set must include the
`_rowid` column, so add `.with_row_id(True)` to each `table.search(...)` call before reranking,
otherwise the call raises a `ValueError`. See [multi-vector reranking](/reranking#multi-vector-reranking)
for a full example.
## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
### Hybrid Search
| `return_score` | Status | Description |
| -------------- | ----------- | ----------------------------------------------------------------------------------------------------------- |
| `relevance` | ✅ Supported | Returned rows only have the `_relevance_score` column. |
| `all` | ✅ Supported | Returned rows have vector(`_distance`) and FTS(`score`) along with Hybrid Search score(`_relevance_score`). |
# Metadata Filtering in LanceDB
Source: https://docs.lancedb.com/search/filtering
Filter search results in LanceDB based on metadata fields.
LanceDB supports filtering features of query results based on metadata fields.
While joint vector and metadata search at scale presents a significant challenge,
LanceDB achieves sub-100ms latency at thousands of QPS, enabling efficient vector search
with filtering capabilities even on datasets containing billions of records.
**Pre-filtering** means LanceDB applies the metadata `where(...)` condition before running vector search, so the search only considers rows that already match the filter. **Post-filtering** means LanceDB runs vector search first and only then filters the returned candidates. Pre-filtering is enabled by default. In practice, pre-filtering is better when the filter is part of the result contract; post-filtering can be lower-latency for expensive or non-indexable filters, but it can return fewer than `limit` rows, or even zero, if the nearest neighbors do not pass the filter.
On hybrid queries, the same `where(...)` filter is applied to both the vector and full-text halves of the query. The prefilter or postfilter choice controls whether that happens before each subquery scores candidates or after the subquery top-k is produced.
## Chaining `where` clauses
In more recent LanceDB SDK versions (see the callout box below for the exact version numbers), you can call `where(...)` (Python and TypeScript) or `only_if(...)` (Rust) more than once on the same query builder. Each additional filter is combined with the previous one using logical `AND`, so `where("a > 0").where("b < 10")` is equivalent to `where("(a > 0) AND (b < 10)")`.
For a fixed predicate, writing one `where(...)` clause with `AND` is just as valid and often clearer. Chaining is mainly useful when code composes filters incrementally, such as applying a shared base predicate in a helper and then adding a per-call predicate at the query site.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Both filters apply: item is in the list AND price is above 15.
result = (
table.search([100, 102])
.where("item IN ('foo', 'bar', 'baz')")
.where("price > 15.0")
.limit(3)
.to_arrow()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Both filters apply: item is in the list AND price is above 15.
const result = await table
.search([100, 102])
.where("item IN ('foo', 'bar', 'baz')")
.where("price > 15.0")
.limit(3)
.toArray();
```
In Python SDK versions before `0.34.0` and Rust/TypeScript SDK versions before `0.31.0`, a second `where(...)` or `only_if(...)` call replaced the first filter, so only the last predicate was applied. If your code needs to run on those older versions, write a single predicate with `AND` instead of chaining calls. When upgrading to the latest SDKs, review any existing chained filters and drop earlier calls you no longer want to apply.
## Example: Metadata Filtering
To illustrate filtering capabilities, let's try four data points with combinations of vectors and metadata:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
data = [
{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0},
{"vector": [10.2, 100.8], "item": "baz", "price": 30.0},
{"vector": [1.4, 9.5], "item": "fred", "price": 40.0},
]
table = db.create_table("metadata_filter_example", data=data, mode="overwrite")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const data = [
{ vector: [3.1, 4.1], item: "foo", price: 10.0 },
{ vector: [5.9, 26.5], item: "bar", price: 20.0 },
{ vector: [10.2, 100.8], item: "baz", price: 30.0 },
{ vector: [1.4, 9.5], item: "fred", price: 40.0 },
];
const tableName = "metadata_filter_example";
const table = await db.createTable(tableName, data, {
mode: "overwrite",
});
```
### Filtering Without Vector Search
You can always filter your data without search. This is useful when you need to query based on metadata:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
filtered_no_search_result = (
table.search()
.where("(item IN ('foo', 'bar', 'baz')) AND (price > 15.0)")
.limit(3)
.to_arrow()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const filteredResult = await table
.query()
.where("(item IN ('foo', 'bar', 'baz')) AND (price > 15.0)")
.limit(3)
.toArray();
```
If your table is large, this could potentially return a very large amount of data. Please be sure to use a `limit` clause unless you're sure you want to return the whole result set.
### Pre-Filtering with Vector Search
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
filtered_result = (
table.search([100, 102])
.where("(item IN ('foo', 'bar')) AND (price > 15.0)")
.limit(3)
.to_arrow()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const results = await table
.search([100, 102])
.where("(item IN ('foo', 'bar')) AND (price > 15.0)")
.toArray();
```
### Post-Filtering with Vector Search
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
post_filtered_result = (
table.search([100, 102])
.where("(item IN ('foo', 'bar')) AND (price > 15.0)", prefilter=False)
.limit(3)
.to_arrow()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const postFilteredResult = await (table.search([100, 102]) as VectorQuery)
.where("(item IN ('foo', 'bar')) AND (price > 15.0)")
.postfilter()
.limit(3)
.toArray();
```
When querying large tables, omitting a `limit` clause may overwhelm resources and return excessive data. It's always recommended
to be mindful of the potential impact on performance and costs when working with really large tables.
## Filtering with SQL
Because it's built on top of DataFusion, LanceDB embraces the utilization of standard SQL expressions as predicates for filtering operations. SQL can be used during vector search, update, and deletion operations.
LanceDB supports a growing list of SQL expressions:
| SQL Expression | Description |
| :----------------------------------------------------------------------------------------- | :-------------------------- |
| `>, >=, <, <=, =` | Comparison operators |
| `AND`, `OR`, `NOT` | Logical operators |
| `IS NULL`, `IS NOT NULL` | Null checks |
| `IS TRUE`, `IS NOT TRUE`, `IS FALSE`, `IS NOT FALSE` | Boolean checks |
| `IN` | Value matching from a set |
| `LIKE`, `NOT LIKE` | Pattern matching |
| `CAST` | Type conversion |
| `regexp_match(column, pattern)` | Regular expression matching |
| [DataFusion Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) | Additional SQL functions |
### Simple SQL Filters
For example, the following filter string is acceptable:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl.search([100, 102]).where(
"(item IN ('foo', 'baz')) AND (price > 20.0)"
).to_arrow()
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
await table
.search([100, 102])
.where("(item IN ('foo', 'baz')) AND (price > 20.0)")
.toArray();
```
### Advanced SQL Filters
If your column name contains special characters, upper-case characters, or is a [SQL Keyword](https://docs.rs/sqlparser/latest/sqlparser/keywords/index.html),
you can use backtick (`` ` ``) to escape it. For nested fields, each segment of the
path must be wrapped in backticks.
```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
`CUBE` = 10 AND `UpperCaseName` = '3' AND `column name with space` IS NOT NULL
AND `nested with space`.`inner with space` < 2
```
Field names containing periods (.) are NOT supported.
### Dates, Timestamps, Decimals
Literals for dates, timestamps, and decimals can be written by writing the string
value after the type name. For example:
```sql SQL icon="SQL" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
date_col = date '2021-01-01'
and timestamp_col = timestamp '2021-01-01 00:00:00'
and decimal_col = decimal(8,3) '1.000'
```
For timestamp columns, the precision can be specified as a number in the type
parameter. Microsecond precision (6) is the default.
| SQL | Time unit |
| :------------- | :----------- |
| `timestamp(0)` | Seconds |
| `timestamp(3)` | Milliseconds |
| `timestamp(6)` | Microseconds |
| `timestamp(9)` | Nanoseconds |
## Apache Arrow Mapping
LanceDB internally stores data in [Apache Arrow](https://arrow.apache.org/) format.
The mapping from SQL types to Arrow types is:
| SQL type | Arrow type |
| :-------------------------------------------------------- | :----------------- |
| `boolean` | `Boolean` |
| `tinyint` / `tinyint unsigned` | `Int8` / `UInt8` |
| `smallint` / `smallint unsigned` | `Int16` / `UInt16` |
| `int` or `integer` / `int unsigned` or `integer unsigned` | `Int32` / `UInt32` |
| `bigint` / `bigint unsigned` | `Int64` / `UInt64` |
| `float` | `Float32` |
| `double` | `Float64` |
| `decimal(precision, scale)` | `Decimal128` |
| `date` | `Date32` |
| `timestamp` | `Timestamp` \[^1] |
| `string` | `Utf8` |
| `binary` | `Binary` |
## Best Practices
**Scalar Indexes**: We strongly recommend creating scalar indices on columns used for filtering, whether combined with a search operation or applied independently (e.g., for updates or deletions).
For best performance with large tables or high query volumes:
* Build a scalar index on frequently filtered columns
* Use exact column names in filters (e.g., `user_id` instead of `USER_ID`)
* Avoid complex transformations in filter expressions (keep them simple)
* When running concurrent queries, use connection pooling for better throughput
For a column of type LIST(T), you can use `LABEL_LIST` to create a scalar index. Then you should leverage DataFusion's [array functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html#array-functions) like `array_has_any` or `array_has_all` for optimized filtering.
## Limitations
Both **pre-filtering** and **post-filtering** can yield false positives. For pre-filtering, if the filter is too selective, it might eliminate relevant items that the vector search would have otherwise identified as a good match. In this case, increasing `nprobes` parameter will help reduce such false positives. It is recommended to call `bypass_vector_index()` if you know that the filter is highly selective.
Similarly, a highly selective post-filter can lead to false positives. Increasing both `nprobes` and `refine_factor` can mitigate this issue. When deciding between pre-filtering and post-filtering, pre-filtering is generally the safer choice if you're uncertain.
# Full-Text Search Examples
Source: https://docs.lancedb.com/search/fts-examples
Worked examples of fuzzy search, boosting, boolean queries, and substring search with LanceDB full-text search.
These worked examples build on the concepts from the [Full-Text Search guide](/search/full-text-search). They walk through creating sample tables, building FTS indices, and running fuzzy, phrase, boosted, boolean, and substring queries.
## Fuzzy Search and Boosting Example
### Generate Data
First, let's create a table with sample text data for testing fuzzy search:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
import numpy as np
import pandas as pd
import random
# Connect to LanceDB
db = lancedb.connect(
uri="db://your-project-slug",
api_key="your-api-key",
region="us-east-1"
)
# Generate 100 rows of random " " text
table_name = "fts-fuzzy-boosting-test"
vectors = [np.random.randn(128) for _ in range(100)]
verbs = ("runs", "hits", "jumps", "drives", "barfs")
adv = ("crazily.", "dutifully.", "foolishly.", "merrily.", "occasionally.")
adj = ("adorable", "clueless", "dirty", "odd", "stupid")
def sentence(nouns):
return " ".join(random.choice(words) for words in (nouns, verbs, adv, adj))
text = [sentence(("puppy", "car")) for _ in range(100)]
text2 = [sentence(("rabbit", "girl", "monkey")) for _ in range(100)]
count = [random.randint(1, 10000) for _ in range(100)]
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb"
const db = await lancedb.connect({
uri: "db://your-project-slug",
apiKey: "your-api-key",
region: "us-east-1"
});
// Generate 100 rows of random " " text
const tableName = "fts-fuzzy-boosting-test-ts";
const n = 100;
const verbs = ["runs", "hits", "jumps", "drives", "barfs"];
const adverbs = ["crazily", "dutifully", "foolishly", "merrily", "occasionally"];
const adjectives = ["adorable", "clueless", "dirty", "odd", "stupid"];
const pick = (words: string[]) => words[Math.floor(Math.random() * words.length)];
const sentence = (nouns: string[]) =>
[nouns, verbs, adverbs, adjectives].map(pick).join(" ");
const vectors = Array.from({ length: n }, () =>
Array.from({ length: 128 }, () => Math.random() * 2 - 1)
);
const text = Array.from({ length: n }, () => sentence(["puppy", "car"]));
const text2 = Array.from({ length: n }, () => sentence(["rabbit", "girl", "monkey"]));
const count = Array.from({ length: n }, () => Math.floor(Math.random() * 10000) + 1);
```
### Create Table
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Create table with sample data
table = db.create_table(
table_name,
data=pd.DataFrame({
"vector": vectors,
"id": [i % 2 for i in range(100)],
"text": text,
"text2": text2,
"count": count,
}),
mode="overwrite"
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Create table with sample data
const data = makeArrowTable(
vectors.map((vector, i) => ({
vector,
id: i % 2,
text: text[i],
text2: text2[i],
count: count[i],
}))
);
const table = await db.createTable(tableName, data, { mode: "overwrite" });
```
### Construct FTS Index
Create a full-text search index on the first text column:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Create FTS index on first text column
table.create_fts_index("text")
wait_for_index(table, "text_idx")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Create FTS index on first text column
await table.createIndex("text", { config: Index.fts() });
await waitForIndex(table, "text_idx");
```
Then, create an index on the second text column:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Create FTS index on second text column
table.create_fts_index("text2")
wait_for_index(table, "text2_idx")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Create FTS index on second text column
await table.createIndex("text2", { config: Index.fts() });
await waitForIndex(table, "text2_idx");
```
### Basic and Fuzzy Search
Now we can perform basic, fuzzy, and prefix match searches:
#### Basic Exact Search
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery
# Basic match (exact search)
basic_match_results = (
table.search(MatchQuery("crazily", "text"))
.select(["id", "text"])
.limit(100)
.to_pandas()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import { MatchQuery } from "@lancedb/lancedb";
// Basic match (exact search)
const basicMatchResults = await table.query()
.fullTextSearch(new MatchQuery("crazily", "text"))
.select(["id", "text"])
.limit(100)
.toArray();
```
#### Fuzzy Search with Typos
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Fuzzy match (allows typos)
fuzzy_results = (
table.search(MatchQuery("craziou", "text", fuzziness=2))
.select(["id", "text"])
.limit(100)
.to_pandas()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Fuzzy match (allows typos)
const fuzzyResults = await table.query()
.fullTextSearch(new MatchQuery("craziou", "text", {
fuzziness: 2,
}))
.select(["id", "text"])
.limit(100)
.toArray();
```
#### Prefix based Match
Prefix-based match allows you to search for documents containing words that start with a specific prefix.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Fuzzy match (allows typos)
fuzzy_results = (
table.search(MatchQuery("cra", "text", prefix_length=3))
.select(["id", "text"])
.limit(100)
.to_pandas()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Fuzzy match (allows typos)
const fuzzyResults = await table.query()
.fullTextSearch(new MatchQuery("cra", "text", {
prefixLength: 3,
}))
.select(["id", "text"])
.limit(100)
.toArray();
```
### Phrase Match
Phrase matching enables you to search for exact sequences of words. Unlike regular text search
which matches individual terms independently, phrase matching requires words to appear in the
specified order with no intervening terms.
Phrase queries are supported but only for a single column; providing multiple columns with a quoted phrase raises an error.
Phrase matching is particularly useful for:
* Searching for specific multi-word expressions
* Matching exact titles or quotes
* Finding precise word combinations in a specific order
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Exact phrase match
from lancedb.query import PhraseQuery
phrase_results = (
table.search(PhraseQuery("puppy runs", "text"))
.select(["id", "text"])
.limit(100)
.to_pandas()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import { PhraseQuery } from "@lancedb/lancedb";
// Exact phrase match
const phraseResults = await table.query()
.fullTextSearch(new PhraseQuery("puppy runs", "text"))
.select(["id", "text"])
.limit(100)
.toArray();
```
#### Flexible Phrase Match
To provide more flexible phrase matching, LanceDB supports the `slop` parameter. This allows you to match phrases where the terms appear close to each other, even if they are not directly adjacent or in the exact order, as long as they are within the specified `slop` value.
For example, the phrase query "puppy merrily" would not return any results by default. However, if you set `slop=1`, it will match phrases like "puppy jumps merrily", "puppy runs merrily", and similar variations where one word appears between "puppy" and "merrily".
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Flexible phrase match with slop=1 for 'puppy merrily'
from lancedb.query import PhraseQuery
phrase_results = (
table.search(PhraseQuery("puppy merrily", "text", slop=1))
.select(["id", "text"])
.limit(100)
.to_pandas()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import { PhraseQuery } from "@lancedb/lancedb";
// Flexible phrase match with slop=1 for 'puppy runs'
const phraseResults = await table.query()
.fullTextSearch(new PhraseQuery("puppy runs", "text", { slop: 1 }))
.select(["id", "text"])
.limit(100)
.toArray();
```
### Search with Boosting
Boosting allows you to control the relative importance of different search terms or fields
in your queries. This feature is particularly useful when you need to:
* Prioritize matches in certain columns
* Promote specific terms while demoting others
* Fine-tune relevance scoring for better search results
| Parameter | Type | Default | Description |
| --------------- | ----- | -------- | ------------------------------------------------------------------ |
| positive | Query | required | The primary query terms to match and promote in results |
| negative | Query | required | Terms to demote in the search results |
| negative\_boost | float | 0.5 | Multiplier for negative matches (lower values = stronger demotion) |
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery, BoostQuery, MultiMatchQuery
# Boost data with 'runs' in text more than 'puppy' in text
boosting_results = (
table.search(
BoostQuery(
MatchQuery("runs", "text"),
MatchQuery("puppy", "text"),
negative_boost=0.2,
),
)
.select(["id", "text"])
.limit(100)
.to_pandas()
)
# Search across both text and text2
multi_match_results = (
table.search(MultiMatchQuery("crazily", ["text", "text2"]))
.select(["id", "text", "text2"])
.limit(100)
.to_pandas()
)
# Search with field boosting
multi_match_boosting_results = (
table.search(
MultiMatchQuery("crazily", ["text", "text2"], boosts=[1.0, 2.0]),
)
.select(["id", "text", "text2"])
.limit(100)
.to_pandas()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import { MatchQuery, BoostQuery, MultiMatchQuery } from "@lancedb/lancedb";
// Boosting Example
const boostingResults = await table.query()
.fullTextSearch(new BoostQuery(new MatchQuery("runs", "text"), new MatchQuery("puppy", "text"), {
negativeBoost: 0.2,
}))
.select(["id", "text"])
.limit(100)
.toArray();
// Search across both text fields
const multiMatchResults = await table.query()
.fullTextSearch(new MultiMatchQuery("crazily", ["text", "text2"]))
.select(["id", "text", "text2"])
.limit(100)
.toArray();
// Search with field boosting
const multiMatchBoostingResults = await table.query()
.fullTextSearch(new MultiMatchQuery("crazily", ["text", "text2"], {
boosts: [1.0, 2.0],
}))
.select(["id", "text", "text2"])
.limit(100)
.toArray();
```
* Use fuzzy search when handling user input that may contain typos or variations
* Apply field boosting to prioritize matches in more important columns
* Combine fuzzy search with boosting for robust and precise search results
**Recommendations for optimal FTS performance:**
* Create full-text search indices on text columns that will be frequently searched
* For hybrid search combining text and vectors, see our [hybrid search guide](/search/hybrid-search/)
* For performance benchmarks, check our [benchmark results](/enterprise/benchmarks/)
* For complex queries, use SQL to combine FTS with other filter conditions
### Boolean Queries
LanceDB supports boolean logic in full-text search, allowing you to combine multiple queries using `and` and `or` operators. This is useful when you want to match documents that satisfy multiple conditions (intersection) or at least one of several conditions (union).
#### Combining Two Match Queries
In Python, you can combine two MatchQuery objects using either the `and` function or the `&` operator (e.g., `MatchQuery("puppy", "text") and MatchQuery("merrily", "text")`); both methods are supported and yield the same result. Similarly, you can use either the `or` function or the `|` operator to perform an or query.
In TypeScript, boolean queries are constructed using the `BooleanQuery` class with a list of \[Occur, subquery] pairs. For example, to perform an AND query:
```sql SQL icon="code" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
BooleanQuery([
[Occur.Must, new MatchQuery("puppy", "text")],
[Occur.Must, new MatchQuery("merrily", "text")],
])
```
This approach allows you to specify complex boolean logic by combining multiple subqueries with different Occur values (such as `Must`, `Should`, or `MustNot`).
**Which queries are allowed?**
A boolean query must include at least one `SHOULD` or `MUST` clause. Queries that contain only a `MUST_NOT` clause are not allowed.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery
# Example: Find documents containing both "puppy" and "merrily"
and_query = MatchQuery("puppy", "text") & MatchQuery("merrily", "text")
and_results = (
table.search(and_query)
.select(["id", "text"])
.limit(100)
.to_pandas()
)
# Example: Find documents containing either "puppy" or "merrily"
or_query = MatchQuery("puppy", "text") | MatchQuery("merrily", "text")
or_results = (
table.search(or_query)
.select(["id", "text"])
.limit(100)
.to_pandas()
)
```
```typescript TypeScript icon="square-js" expandable=true theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import { MatchQuery, BooleanQuery, Occur } from "@lancedb/lancedb";
// Flexible boolean queries with MatchQuery
// Find documents containing both "puppy" and "merrily"
const mustResults = await table
.search(
new BooleanQuery([
[Occur.Must, new MatchQuery("puppy", "text")],
[Occur.Must, new MatchQuery("merrily", "text")],
]),
)
.select(["id", "text"])
.limit(100)
.toArray();
// Find documents containing either "puppy" or "merrily"
const shouldResults = await table
.search(
new BooleanQuery([
[Occur.Should, new MatchQuery("puppy", "text")],
[Occur.Should, new MatchQuery("merrily", "text")],
]),
)
.select(["id", "text"])
.limit(100)
.toArray();
```
**How to use booleans?**
* Use `and`/`&`(Python), `Occur.Must`(Typescript) for intersection (documents must match all queries).
* Use `or`/`|`(Python), `Occur.Should`(Typescript) for union (documents must match at least one query).
## Substring Search Example
LanceDB supports searching for substrings in text columns using n-gram tokenization. This is useful for finding partial matches within text content.
### Setting Up the Table
First, create a table with sample text data and configure n-gram tokenization:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
import lancedb
db = lancedb.connect(":memory:")
data = pa.table({"text": ["hello world", "lance database", "lance is cool"]})
table = db.create_table("test", data=data)
table.create_fts_index("text", base_tokenizer="ngram")
```
### Basic Substring Search
With the default n-gram settings (minimum length of 3), you can search for substrings of length 3 or more:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
results = table.search("lan", query_type="fts").limit(10).to_list()
assert len(results) == 2
assert set(r["text"] for r in results) == {"lance database", "lance is cool"}
results = (
table.search("nce", query_type="fts").limit(10).to_list()
) # spellchecker:disable-line
assert len(results) == 2
assert set(r["text"] for r in results) == {"lance database", "lance is cool"}
```
### Handling Short Substrings
By default, the minimum n-gram length is 3, so shorter substrings like "la" won't match:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
results = table.search("la", query_type="fts").limit(10).to_list()
assert len(results) == 0
```
### Customizing N-gram Parameters
You can customize the n-gram behavior by adjusting the minimum length and using prefix-only matching:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.create_fts_index(
"text",
base_tokenizer="ngram",
replace=True,
ngram_min_length=2,
prefix_only=True,
)
```
### Testing Custom N-gram Settings
With the new settings, you can now search for shorter substrings and use prefix-only matching:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
results = table.search("lan", query_type="fts").limit(10).to_list()
assert len(results) == 2
assert set(r["text"] for r in results) == {"lance database", "lance is cool"}
results = (
table.search("nce", query_type="fts").limit(10).to_list()
) # spellchecker:disable-line
assert len(results) == 0
results = table.search("la", query_type="fts").limit(10).to_list()
assert len(results) == 2
assert set(r["text"] for r in results) == {"lance database", "lance is cool"}
```
# Full-Text Search (FTS)
Source: https://docs.lancedb.com/search/full-text-search
Learn how to implement full-text search in LanceDB using BM25 for keyword-based retrieval.
LanceDB provides support for Full-Text Search via Lance, allowing you to incorporate keyword-based search (based on BM25) in your retrieval solutions.
## Basic Usage
Consider that we have a LanceDB table named `my_table`, whose string column `text` we want to index and query via keyword search, the FTS index must be created before you can search via keywords.
### Table Setup
First, open or create the table you want to search:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.index import FTS
uri = "data/sample-lancedb"
db = lancedb.connect(uri)
table = db.create_table(
"my_table_fts",
data=[
{"vector": [3.1, 4.1], "text": "Frodo was a happy puppy"},
{"vector": [5.9, 26.5], "text": "There are several kittens playing"},
],
)
```
```ts TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
const uri = "data/sample-lancedb"
const db = await lancedb.connect(uri);
const data = [
{ vector: [3.1, 4.1], text: "Frodo was a happy puppy" },
{ vector: [5.9, 26.5], text: "There are several kittens playing" },
];
const tbl = await db.createTable("my_table", data, { mode: "overwrite" });
```
```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
let uri = "data/sample-lancedb";
let db = connect(uri).execute().await?;
let initial_data: Box = create_some_records()?;
let tbl = db
.create_table("my_table", initial_data)
.execute()
.await?;
```
### Construct FTS Index
Create a full-text search index on your text column:
In Python, this page shows the synchronous `create_fts_index(...)` form. For the
asynchronous equivalent (`await table.create_index("text", config=FTS(...))`), see
[FTS index](/indexing/fts-index).
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.create_fts_index("text")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
await tbl.createIndex("text", {
config: lancedb.Index.fts(),
});
```
```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl
.create_index(&["text"], Index::FTS(FtsIndexBuilder::default()))
.execute()
.await?;
```
### Full-text Search
Perform full-text search and retrieve results:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
results = table.search("puppy")
.limit(10)
.select(["text"])
.to_list()
# [{'text': 'Frodo was a happy puppy', '_score': 0.6931471824645996}]
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const results = await tbl
.search("puppy", "fts")
.select(["text"])
.limit(10)
.toArray();
```
```rust Rust icon="Rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
let results = tbl
.query()
.full_text_search(FullTextSearchQuery::new("puppy".to_owned()))
.select(lancedb::query::Select::Columns(vec!["text".to_owned()]))
.limit(10)
.execute()
.await?;
```
The search is conducted on all indexed columns by default, so it's useful when there are multiple indexed columns.
If you want to specify which columns to search use `fts_columns="text"`
LanceDB automatically searches on the existing FTS index if the input to the search is of type `str`. If you provide a vector as input, LanceDB will search the ANN index instead.
If a table has more than one FTS index, specify the indexed text column in the query. In Python you can use `fts_columns` or the query builder's `nearest_to_text(..., columns=...)`; in TypeScript, use `query().nearestToText(..., columns)`. The newer Lance-native FTS does not accept legacy Tantivy-only index parameters.
### Keeping the index up to date
Rows you add after building an FTS index aren't part of the index until you optimize the table. Until then, queries fall back to a flat scan over the unindexed fragments to keep results complete, which slows them down as the unindexed tail grows. Call `table.optimize()` to fold new rows into the existing index — it's the same operation used for vector indexes:
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
await tbl.add([{ vector: [3.1, 4.1], text: "Frodo was a happy puppy" }]);
await tbl.optimize();
```
```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl.add(new_data).execute().await?;
tbl.optimize(OptimizeAction::All).await?;
```
A useful rule of thumb is to call `optimize()` after roughly 100,000 row changes or 20 data-modification operations, whichever comes first. For tables with continuous ingest, schedule it on a cadence that keeps `num_unindexed_rows` (from `table.index_stats(...)`) close to zero. If you want to skip the flat scan over unindexed rows entirely — for example, on a hot read path where stale results are acceptable — call `.fast_search()` on the query so the search returns only indexed results.
## Advanced Usage
### Tokenize Table Data
By default, the text is tokenized by splitting on punctuation and whitespaces, and would filter out words that are longer than 40 characters. All words are converted to lowercase.
Stemming is useful for improving search results by reducing words to their root form, e.g. "running" to "run". LanceDB supports stemming for Arabic, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, and Turkish. You should set the `base_tokenizer` parameter rather than `tokenizer_name` because you cannot customize the tokenizer if `tokenizer_name` is specified.
Tokenization and language filters are separate settings. `base_tokenizer` controls how text is split into searchable tokens. `language` controls stemming and stop-word removal when `stem=True` or `remove_stop_words=True`; choose the tokenizer for CJK or mixed-language segmentation.
For example, to enable stemming for English:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.create_fts_index("text", language="English", replace=True)
```
The tokenizer is customizable, you can specify how the tokenizer splits the text, and how it filters out words, etc.
**Default index parameters:**
* `base_tokenizer`: `"simple"`
* `language`: English
* `with_position`: false
* `max_token_length`: 40
* `lower_case`: true
* `stem`: true
* `remove_stop_words`: true
* `ascii_folding`: true
* `custom_stop_words`: `None` — pass a `list[str]` to drop additional words beyond the language defaults. Requires `remove_stop_words=True`.
For multilingual use cases, use `base_tokenizer="icu"` for unicode-aware word segmentation on mixed-language text. ICU stands for [International Components for Unicode](https://icu.unicode.org/). The ICU tokenizer uses bundled ICU4X segmenter data, so it does not require external tokenizer model files. It is a good default when documents mix languages or include scripts where the simple tokenizer would keep an unspaced span as one large token.
The Python API also supports tokenizer implementations that load language model files. Use `base_tokenizer="jieba/default"` for Jieba tokenization, which segments Chinese text into searchable word tokens when the text is written without spaces between words. Use Lindera-backed tokenizers for dictionary-based East Asian morphological segmentation, such as `base_tokenizer="lindera/ipadic"` for Japanese or `base_tokenizer="lindera/ko-dic"` for Korean when you have installed and compiled that Lindera model. These are language-specific tokenizers; ICU is the broader mixed-language option.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.create_fts_index(
"text",
base_tokenizer="jieba/default",
stem=False,
remove_stop_words=False,
ascii_folding=False,
replace=True,
)
```
Model-backed tokenizers require tokenizer model files in Lance's language model home. Lance looks under the default platform data directory for `lance/language_models`, or you can set `LANCE_LANGUAGE_MODEL_HOME` to point to a different model root:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
export LANCE_LANGUAGE_MODEL_HOME=/path/to/lance/language_models
```
For example, `jieba/default` is resolved under `/jieba/default/...`, `lindera/ipadic` under `/lindera/ipadic/...`, and `lindera/ko-dic` under `/lindera/ko-dic/...`.
Built-in stop-word removal supports Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. If you use another stemming language, such as Arabic, Greek, Romanian, Tamil, or Turkish, set `remove_stop_words=False` or pass `custom_stop_words`.
For example, for language with accents, you can specify the tokenizer to use `ascii_folding` to remove accents, e.g. 'é' to 'e':
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.create_fts_index(
"text",
language="French",
stem=True,
ascii_folding=True,
replace=True,
)
```
### Filtering Options
LanceDB full text search supports to filter the search results by a condition, both pre-filtering and post-filtering are supported.
This can be invoked via the familiar `where` syntax.
With pre-filtering:
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
await tbl
.search("puppy")
.select(["id", "doc"])
.limit(10)
.where("meta='foo'")
.prefilter(true)
.toArray();
```
```rust Rust icon="Rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table
.query()
.full_text_search(FullTextSearchQuery::new("puppy".to_owned()))
.select(lancedb::query::Select::Columns(vec!["doc".to_owned()]))
.limit(10)
.only_if("meta='foo'")
.execute()
.await?;
```
With post-filtering:
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
await tbl
.search("apple")
.select(["id", "doc"])
.limit(10)
.where("meta='foo'")
.prefilter(false)
.toArray();
```
```rust Rust icon="Rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table
.query()
.full_text_search(FullTextSearchQuery::new(words[0].to_owned()))
.select(lancedb::query::Select::Columns(vec!["doc".to_owned()]))
.postfilter()
.limit(10)
.only_if("meta='foo'")
.execute()
.await?;
```
### Phrase vs. Terms Queries
Lance-based FTS doesn't support queries using boolean operators `OR`, `AND` in the search string.
For full-text search you can specify either a **phrase** query like `"the old man and the sea"`,
or a **terms** search query like `old man sea`.
To search for a phrase, the index must be created with `with_position=True` and `remove_stop_words=False`:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.create_fts_index("text", with_position=True, replace=True)
```
This will allow you to search for phrases, but it will also significantly increase the index size and indexing time.
### Fuzzy Search
Fuzzy search allows you to find matches even when the search terms contain typos or slight variations.
LanceDB uses the classic [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance)
to find similar terms within a specified edit distance.
| Parameter | Type | Default | Description |
| --------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| fuzziness | int | 0 | Maximum edit distance allowed for each term. If not specified, automatically set based on term length: 0 for length ≤ 2, 1 for length ≤ 5, 2 for length > 5 |
| max\_expansions | int | 50 | Maximum number of terms to consider for fuzzy matching. Higher values may improve recall but increase search time |
For a complete walkthrough that creates a sample table and demonstrates fuzzy search and relevance boosting, see the [fuzzy search example](/search/fts-examples#fuzzy-search-and-boosting-example).
### Search for Substring
LanceDB supports searching for substrings in the text column, you can set the `base_tokenizer` parameter to `"ngram"` to enable this feature, and use the parameters `ngram_min_length` and `ngram_max_length` to control the length of the substrings:
| Parameter | Type | Default | Description |
| ------------------ | ---- | ------- | -------------------------------------------------- |
| ngram\_min\_length | int | 3 | Minimum length of the n-grams to search for |
| ngram\_max\_length | int | 3 | Maximum length of the n-grams to search for |
| prefix\_only | bool | false | Whether to only search for prefixes of the n-grams |
## More Examples
For complete worked examples of fuzzy search, prefix matching, phrase matching, boosting, boolean queries, and substring search — including sample data generation and index setup — see [Full-Text Search Examples](/search/fts-examples).
## Full-Text Search on Array Fields
LanceDB supports full-text search on string array columns, enabling efficient keyword-based search across multiple values within a single field (e.g., tags, keywords).
### Setting Up the Connection
Connect to your LanceDB instance:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
# Connect to LanceDB
db = lancedb.connect(
uri="db://your-project-slug",
api_key="your-api-key",
region="us-east-1"
)
```
```typescript TypeScript icon="square-js" expandable=true theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb"
const db = await lancedb.connect({
uri: "db://your-project-slug",
apiKey: "your-api-key",
region: "us-east-1"
});
```
### Defining the Schema
Create a schema that includes an array field for tags:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table_name = "fts-array-field-test"
schema = pa.schema([
pa.field("id", pa.string()),
pa.field("tags", pa.list_(pa.string())),
pa.field("description", pa.string())
])
```
```typescript TypeScript icon="square-js" expandable=true theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const tableName = "fts-array-field-test-ts";
// Create schema
const schema = new Schema([
new Field("id", new Utf8(), false),
new Field("tags", new List(new Field("item", new Utf8()))),
new Field("description", new Utf8(), false)
]);
```
### Creating Sample Data
Generate sample data with array fields containing tags:
```python Python icon="python" expandable=true theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Generate sample data
data = {
"id": [f"doc_{i}" for i in range(10)],
"tags": [
["python", "machine learning", "data science"],
["deep learning", "neural networks", "AI"],
["database", "indexing", "search"],
["vector search", "embeddings", "AI"],
["full text search", "indexing", "database"],
["python", "web development", "flask"],
["machine learning", "deep learning", "pytorch"],
["database", "SQL", "postgresql"],
["search engine", "elasticsearch", "indexing"],
["AI", "transformers", "NLP"]
],
"description": [
"Python for data science projects",
"Deep learning fundamentals",
"Database indexing techniques",
"Vector search implementations",
"Full-text search guide",
"Web development with Python",
"Machine learning with PyTorch",
"Database management systems",
"Search engine optimization",
"AI and NLP applications"
]
}
```
```typescript TypeScript icon="square-js" expandable=true theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Generate sample data
const data = makeArrowTable(
Array(10).fill(0).map((_, i) => ({
id: `doc_${i}`,
tags: [
["python", "machine learning", "data science"],
["deep learning", "neural networks", "AI"],
["database", "indexing", "search"],
["vector search", "embeddings", "AI"],
["full text search", "indexing", "database"],
["python", "web development", "flask"],
["machine learning", "deep learning", "pytorch"],
["database", "SQL", "postgresql"],
["search engine", "elasticsearch", "indexing"],
["AI", "transformers", "NLP"]
][i],
description: [
"Python for data science projects",
"Deep learning fundamentals",
"Database indexing techniques",
"Vector search implementations",
"Full-text search guide",
"Web development with Python",
"Machine learning with PyTorch",
"Database management systems",
"Search engine optimization",
"AI and NLP applications"
][i]
})),
{ schema }
);
```
### Creating the Table and Adding Data
Create the table and populate it with the sample data:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Create table and add data
table = db.create_table(table_name, schema=schema, mode="overwrite")
table_data = pa.Table.from_pydict(data, schema=schema)
table.add(table_data)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Create table
const table = await db.createTable(tableName, data, { mode: "overwrite" });
console.log(`Created table: ${tableName}`);
```
### Building the Full-Text Search Index
Create an FTS index on the tags column to enable efficient text search:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Create FTS index
table.create_fts_index("tags")
wait_for_index(table, "tags_idx")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Create FTS index
console.log("Creating FTS index on 'tags' column...");
await table.createIndex("tags", {
config: Index.fts()
});
// Wait for index
const ftsIndexName = "tags_idx";
await waitForIndex(table, ftsIndexName);
```
### Performing Fuzzy Search
Search for terms with typos using fuzzy matching:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Search examples
print("\nSearching for 'learning' in tags with a typo:")
result = (
table.search(MatchQuery("learnin", column="tags", fuzziness=1))
.select(['id', 'tags', 'description'])
.to_arrow()
)
```
```typescript TypeScript icon="square-js"> theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Search examples
console.log("\nSearching for 'learning' in tags with a typo:");
const fuzzyResults = await table.query()
.fullTextSearch(new MatchQuery("learnin", "tags", {
fuzziness: 2,
}))
.select(["id", "tags", "description"])
.toArray();
console.log(fuzzyResults);
```
### Performing Phrase Search
Search for exact phrases within the array fields:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
print("\nSearching for 'machine learning' in tags:")
result = (
table.search(PhraseQuery("machine learning", column="tags"))
.select(['id', 'tags', 'description'])
.to_arrow()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
console.log("\nSearching for 'machine learning' in tags:");
const phraseResults = await table.query()
.fullTextSearch(new PhraseQuery("machine learning", "tags"))
.select(["id", "tags", "description"])
.toArray();
console.log(phraseResults);
```
# Hybrid Search
Source: https://docs.lancedb.com/search/hybrid-search
Learn how to perform hybrid search in LanceDB by combining vector and full-text search techniques with reranking.
In certain cases, you may want to retrieve documents that are semantically similar to a given query,
but also prioritize specific keywords. This is an example of **hybrid search**, a query method that combines
multiple search techniques.
For detailed examples, look at this [Python Notebook](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/saas_examples/python_notebook/Hybrid_search.ipynb) or the [**TypeScript Example**](https://github.com/lancedb/vectordb-recipes/tree/main/examples/saas_examples/ts_example/hybrid-search)
## Example: Hybrid Search
### 1. Setup
Import the necessary libraries and dependencies for working with LanceDB, OpenAI embeddings, and reranking.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import os
import lancedb
import openai
from lancedb.embeddings import get_registry
from lancedb.pydantic import LanceModel, Vector
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
import "@lancedb/lancedb/embedding/openai";
import { Utf8 } from "apache-arrow";
```
### 2. Connect to LanceDB
Establish a connection to your LanceDB instance, with different options for Enterprise setups or open source.
OSS
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uri = "data/sample-lancedb"
db = lancedb.connect(uri)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
import * as arrow from "apache-arrow";
const databaseDir = "data/sample-lancedb";
const db = await lancedb.connect(databaseDir);
```
Enterprise
For LanceDB Enterprise, set the `db://` URI, region and the host override to your private cloud endpoint:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
host_override = os.environ.get("LANCEDB_HOST_OVERRIDE")
db = lancedb.connect(
uri=uri,
api_key=api_key,
region=region,
host_override=host_override
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
import * as arrow from "apache-arrow";
const uri = "db://my-lancedb-instance/my-database";
const apiKey = process.env.LANCEDB_API_KEY;
const region = process.env.LANCEDB_REGION;
const hostOverride = process.env.LANCEDB_HOST_OVERRIDE;
const db = await lancedb.connect(uri, {
apiKey,
region
hostOverride,
});
```
### 3. Configure Embedding Model
Set up the any embedding model that will convert text into vector representations for semantic search.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
embeddings = get_registry().get("sentence-transformers").create()
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const embedFunc = lancedb.embedding.getRegistry().get("openai")?.create({
model: "text-embedding-ada-002",
}) as lancedb.embedding.EmbeddingFunction;
```
### 4. Create Table & Schema
Define the data structure for your documents, including both the text content and its vector representation.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
class Documents(LanceModel):
text: str = embeddings.SourceField()
vector: Vector(embeddings.ndims()) = embeddings.VectorField()
table_name = "hybrid_search_example"
table = db.create_table(table_name, schema=Documents, mode="overwrite")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const documentSchema = lancedb.embedding.LanceSchema({
text: embedFunc.sourceField(new Utf8()),
vector: embedFunc.vectorField(),
});
const tableName = "hybrid_search_example";
const table = await db.createEmptyTable(tableName, documentSchema, {
mode: "overwrite",
});
```
### 5. Add Data
Insert sample documents into your table, which will be used for both semantic and keyword search.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
data = [
{"text": "rebel spaceships striking from a hidden base"},
{"text": "have won their first victory against the evil Galactic Empire"},
{"text": "during the battle rebel spies managed to steal secret plans"},
{"text": "to the Empire's ultimate weapon the Death Star"},
]
table.add(data=data)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const data = [
{ text: "rebel spaceships striking from a hidden base" },
{ text: "have won their first victory against the evil Galactic Empire" },
{ text: "during the battle rebel spies managed to steal secret plans" },
{ text: "to the Empire's ultimate weapon the Death Star" },
];
await table.add(data);
console.log(`Created table: ${tableName} with ${data.length} rows`);
```
### 6. Build Full Text Index
Create a full-text search index on the text column to enable keyword-based search capabilities.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.create_fts_index("text")
wait_for_index(table, "text_idx")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
console.log("Creating full-text search index...");
await table.createIndex("text", {
config: lancedb.Index.fts(),
});
await waitForIndex(table as any, "text_idx");
```
### 7. Set Reranker \[Optional]
Initialize the reranker that will combine and rank results from both semantic and keyword search. By default, lancedb uses RRF reranker, but you can choose other rerankers like `Cohere`, `CrossEncoder`, or others lister in integrations section.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
reranker = RRFReranker()
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const reranker = await lancedb.rerankers.RRFReranker.create();
```
### 8. Hybrid Search
Perform a hybrid search query that combines semantic similarity with keyword matching, using the specified reranker to merge and rank the results.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
results = (
table.search(
"flower moon",
query_type="hybrid",
vector_column_name="vector",
fts_columns="text",
)
.rerank(reranker)
.limit(10)
.to_pandas()
)
print("Hybrid search results:")
print(results)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
console.log("Performing hybrid search...");
const queryVector = await embedFunc.computeQueryEmbeddings("full moon in May");
const hybridResults = await table
.query()
.fullTextSearch("flower moon")
.nearestTo(queryVector)
.rerank(reranker)
.select(["text"])
.limit(10)
.toArray();
console.log("Hybrid search results:");
console.log(hybridResults);
```
### 9. Hybrid Search - Explicit Vector and Text Query pattern
You can also pass the vector and text query explicitly. This is useful if you're not using the embedding API or if you're using a separate embedder service.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
vector_query = [0.1, 0.2, 0.3, 0.4, 0.5]
text_query = "flower moon"
(
table.search(query_type="hybrid")
.vector(vector_query)
.text(text_query)
.limit(5)
.to_pandas()
)
```
## Query controls
Hybrid queries inherit the same builder API as vector and FTS queries, so the same knobs for filtering, distance bounds, and row identity apply. These compose with `.rerank(...)` and the explicit `.vector()` / `.text()` form shown above.
Always set `.limit(...)` on production hybrid queries. LanceDB's default search limit is 10, but an
explicit cap gives you a clear top-k contract to tune before reranking.
### Returning row IDs
Pass `with_row_id(True)` (Python) or `withRowId()` (TypeScript) to include the internal `_rowid` column in the results. This is useful for joining hybrid results back to a primary table, or for deduping across multiple queries:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
results = (
table.search("flower moon", query_type="hybrid")
.with_row_id(True)
.limit(10)
.to_pandas()
)
# results now contains a `_rowid` column alongside `_relevance_score`
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const results = await table
.query()
.fullTextSearch("flower moon")
.nearestTo(queryVector)
.withRowId()
.limit(10)
.toArray();
```
### Bounding vector distance
`distance_range(lower, upper)` (Python) and `distanceRange(lower, upper)` (TypeScript) constrain the vector half of the hybrid query to the half-open interval `[lower, upper)`. This is helpful when you want to cap how far semantic candidates can drift from the query vector before reranking:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
results = (
table.search("flower moon", query_type="hybrid")
.distance_range(lower_bound=0.0, upper_bound=0.4)
.limit(10)
.to_pandas()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const results = await table
.query()
.fullTextSearch("flower moon")
.nearestTo(queryVector)
.distanceRange(0.0, 0.4)
.limit(10)
.toArray();
```
Either bound can be omitted to leave that side unbounded.
### Prefilter vs. postfilter
When the query carries a metadata filter via `where(...)`, you can choose whether the filter runs before or after the vector and FTS sub-queries. **Prefiltering** (the default) applies `where` to the candidate set before scoring, which is usually what you want — it shrinks the working set and benefits from any scalar indexes on the filter columns. **Postfiltering** runs the filter on the already-ranked top-k from each sub-query; this can be faster when the filter is non-selective or unindexed, but it may return fewer than `limit` rows because some of the top-k may be filtered out.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Prefilter (default): filter applied before scoring
table.search("flower moon", query_type="hybrid") \
.where("category = 'film'", prefilter=True) \
.limit(10) \
.to_pandas()
# Postfilter: filter applied after the sub-queries return top-k
table.search("flower moon", query_type="hybrid") \
.where("category = 'film'", prefilter=False) \
.limit(10) \
.to_pandas()
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Prefilter (default): just call .where(...)
await table.query()
.fullTextSearch("flower moon")
.nearestTo(queryVector)
.where("category = 'film'")
.limit(10)
.toArray();
// Postfilter: chain .postfilter() after .where(...)
await table.query()
.fullTextSearch("flower moon")
.nearestTo(queryVector)
.where("category = 'film'")
.postfilter()
.limit(10)
.toArray();
```
The choice gets baked into both sub-queries, so the vector and FTS halves see the filter applied the same way. Use [`explain_plan`](/search/optimize-queries#analyzing-non-vector-queries) on a hybrid query to see whether the filter pushed into the scan or ran as a separate `FilterExec` step.
## More on Reranking
You can perform hybrid search in LanceDB by combining the results of semantic and full-text search via a reranking algorithm of your choice. LanceDB comes with [**built-in rerankers**](https://docs.lancedb.com/reranking) and you can implement your own **custom reranker** as well.
By default, LanceDB uses `RRFReranker()`, which uses reciprocal rank fusion score, to combine and rerank the results of semantic and full-text search. You can customize the hyperparameters as needed or write your own custom reranker. Here's how you can use any of the available rerankers:
| Argument | Type | Default | Description |
| :---------- | :--------- | :-------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `normalize` | `str` | `"score"` | The method to normalize the scores. Can be `rank` or `score`. If `rank`, the scores are converted to ranks and then normalized. If `score`, the scores are normalized directly. |
| `reranker` | `Reranker` | `RRF()` | The reranker to use. If not specified, the default reranker is used. |
# Search
Source: https://docs.lancedb.com/search/index
Comprehensive guide to all search capabilities in LanceDB including vector search, full-text search, hybrid search, and more.
| Feature | Description |
| :------------------------------------------------ | :-------------------------------------------------------- |
| [Vector Search](/search/vector-search/) | Semantic similarity search with multiple distance metrics |
| [Multivector Search](/search/multivector-search/) | Search using multiple vector embeddings per document |
| [Full-Text Search](/search/full-text-search/) | Keyword-based search with BM25 and pre-filtering |
| [Hybrid Search](/search/hybrid-search/) | Combines vector and full-text search with reranking |
| [Filtering](/search/filtering/) | Filter results based on metadata fields |
| [SQL Queries](/search/sql/index) | SQL query capabilities for data exploration and analytics |
## Before you search
* Vector search can run without an ANN index as an exhaustive scan. That's useful while prototyping, but build a vector index before relying on low-latency searches over larger tables.
* Full-text and hybrid text search require an FTS index on the text column you query. If a table has multiple FTS indexes, specify the target column. FTS also supports phrase, boolean, boosted, multi-match, and fuzzy query forms when you need more than plain terms.
* Phrase FTS queries require an index created with token positions enabled.
* Multivector search currently uses cosine similarity and accepts either one query vector or a matrix of query vectors; every query vector must match the inner dimension of the multivector column.
* Set an explicit `.limit(...)` for production queries. The default top-k is 10 for search builders,
but spelling it out makes latency and result-count assumptions visible. Query builders also
support controls such as prefilter/postfilter, distance ranges, row-id inclusion, offset
pagination, and Arrow/Pandas/list result materialization.
# Multivector Search
Source: https://docs.lancedb.com/search/multivector-search
Learn how to perform multivector search in LanceDB to handle multiple vector embeddings per document, which is ideal for late-interaction models like ColBERT and ColPaLi.
LanceDB's multivector support enables you to store and search multiple vector embeddings for a single item.
This capability is particularly valuable when working with late-interaction models like ColBERT and ColPaLi, which generate multiple embeddings per document.
In this tutorial, you'll create a table with multiple vector embeddings per document and learn how to perform multivector search. For more end-to-end examples, see the [VectorDB recipes repository](https://github.com/lancedb/vectordb-recipes/tree/main/examples).
## Multivector Support
Each item in your dataset can have a column containing multiple vectors, which LanceDB can efficiently index and search. When performing a search, you can query with either a single vector embedding or multiple vector embeddings.
Currently, only the `cosine` metric is supported for multivector search. The vector value type can be `float16`, `float32`, or `float64`.
Each query vector must match the inner vector dimension in the multivector column. This applies to both single-vector queries and multi-vector query matrices.
## Computing Similarity
MaxSim (Maximum Similarity) is a key concept in late-interaction models that:
* Computes the maximum similarity between each query embedding and all document embeddings
* Sums these maximum similarities to get the final relevance score
* Effectively captures fine-grained semantic matches between query and document tokens
The MaxSim calculation can be expressed as:
$$
\text{MaxSim}(Q, D) = \sum_{i=1}^{|Q|} \max_{j=1}^{|D|} \text{sim}(q_i, d_j)
$$
Where $sim$ is the similarity function (e.g., cosine similarity).
$$
Q = \{q_1, q_2, ..., q_{|Q|}\}
$$
$Q$ represents the query embeddings, and $D = \{d_1, d_2, ..., d_{|D|}\}$ represents the document embeddings.
## Using Multivector Search
### 1. Setup
Connect to LanceDB and import the required libraries.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
import numpy as np
import pyarrow as pa
db = lancedb.connect(
uri="db://your-project-slug",
api_key="your-api-key",
region="your-region"
)
```
### 2. Define Schema
Define a schema that specifies a multivector field. A multivector field is a nested list structure in which each document contains multiple vectors. In this case, we'll create a schema with:
1. An ID field as an integer (int64)
2. A vector field that is a list of lists of float32 values
* The outer list represents multiple vectors per document
* Each inner list is a 256-dimensional vector
* Using float32 for memory efficiency while maintaining precision
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
db = lancedb.connect("data/multivector_demo")
schema = pa.schema(
[
pa.field("id", pa.int64()),
# float16, float32, and float64 are supported
pa.field("vector", pa.list_(pa.list_(pa.float32(), 256))),
]
)
```
### 3. Generate Multivectors
Generate sample data where each document contains multiple vector embeddings, which can represent different aspects or views of the same document.
In this example, we create **1024 documents** where each document has **2 random vectors** of **dimension 256**, simulating a real-world scenario where you might have multiple embeddings per item.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
data = [
{
"id": i,
"vector": np.random.random(size=(2, 256)).tolist(), # Each document has 2 vectors
}
for i in range(1024)
]
```
### 4. Create a Table
Create a table with the defined schema and sample data, which will store multiple vectors per document for similarity search.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl = db.create_table("multivector_example", data=data, schema=schema)
```
### 5. Build an Index
Only cosine similarity is supported for multivector search operations.
For faster search, build the standard `IVF_PQ` index over your vectors:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl.create_index(metric="cosine", vector_column_name="vector")
```
**Indexing matters more for multivector tables than for single-vector ones.**
A brute-force scan over a multivector column has to compare every query vector to every document vector in every row, so the cost grows with both the row count and the number of vectors per row.
In LanceDB OSS, the query will just run, so a large unindexed multivector table can stall a process for a long time before returning results.
On LanceDB Enterprise, the brute-force KNN safety check applies a stricter row threshold to multivector columns — roughly 10× lower than for single-vector columns. So an unindexed multivector table will start being rejected with a "vector search would use brute-force KNN" error well before a comparable single-vector table would. Build the index before you start hitting it from production traffic, even if the dataset is small enough that you'd skip indexing for a single-vector workload.
### 6. Query a Single Vector
When searching with a single query vector, it will be compared against all vectors in each document, and the similarity scores will be aggregated to find the most relevant documents.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
query = np.random.random(256)
results_single = tbl.search(query).limit(5).to_pandas()
```
### 7. Query Multiple Vectors
With multiple query vectors, LanceDB calculates similarity using late interaction, a late-interaction technique that computes relevance by finding the best-matching pairs between query and document vectors. This approach provides more nuanced matching while maintaining fast retrieval speeds.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
query_multi = np.random.random(size=(2, 256))
results_multi = tbl.search(query_multi).limit(5).to_pandas()
```
Visit the [Hugging Face embedding integration](/integrations/embedding/huggingface/) page for information on embedding models.
## Simple Example: ColBERT Embeddings
[ColBERT](https://arxiv.org/abs/2004.12832) is the most well-known late-interaction retrieval model that
represents each document and query as multiple token embeddings and scores matches by taking the best
token-to-token similarities (MaxSim) across them.
Install the dependencies before running this example:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install pylate lancedb pandas
```
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import numpy as np
import pyarrow as pa
import lancedb
from pylate import models
# 1) Load a late-interaction model via PyLate
# PyLate docs show ColBERT() + encode(..., is_query=...) :contentReference[oaicite:2]{index=2}
model = models.ColBERT(model_name_or_path="lightonai/GTE-ModernColBERT-v1")
# You can discover dim from one embedding (avoid guessing)
dim = model.encode(["hello"], is_query=True)[0].shape[1]
# 2) Create a LanceDB table with a multivector column
db = lancedb.connect("./pylate_lancedb")
schema = pa.schema([
pa.field("doc_id", pa.string()),
pa.field("text", pa.string()),
# multivector: list> :contentReference[oaicite:3]{index=3}
pa.field("mv", pa.list_(pa.list_(pa.float32(), dim))),
])
docs = [
{"doc_id": "1", "text": "The train to Tokyo leaves at 5pm."},
{"doc_id": "2", "text": "That Pho restaurant in Hanoi is highly rated."},
{"doc_id": "3", "text": "This is a noodle bar in Osaka, Japan."},
]
# 3) Encode documents with PyLate (token vectors per doc)
doc_texts = [d["text"] for d in docs]
doc_embs = model.encode(doc_texts, is_query=False) # list/array of (T, dim) per doc :contentReference[oaicite:4]{index=4}
rows = []
for d, emb in zip(docs, doc_embs):
emb = np.asarray(emb, dtype=np.float32)
rows.append({**d, "mv": emb.tolist()})
tbl = db.create_table("docs", data=rows, schema=schema, mode="overwrite")
# 4) Build an index + query using a query matrix.
# Multivector brute-force scales with rows × vectors-per-row, so build the index
# at much smaller dataset sizes than you would for single-vector search — and
# always before exposing the table to remote traffic.
tbl.create_index(vector_column_name="mv", metric="cosine")
query = "Tell me about ramen in Japan"
q_emb = np.asarray(model.encode([query], is_query=True)[0], dtype=np.float32) # (Tq, dim) :contentReference[oaicite:5]{index=5}
out = tbl.search(q_emb).limit(5).to_pandas() # multivector search accepts a matrix :contentReference[oaicite:6]{index=6}
print(out[["doc_id", "text"]])
```
Late-interaction model implementations evolve rapidly, so it's a good idea to check the latest popular models
when trying multivector search.
## Advanced Example: XTR Embeddings
[ConteXtualized Token Retriever (XTR)](https://arxiv.org/abs/2304.01982) is a late-interaction retrieval model that represents text as token-level vectors instead of a single embedding.
This lets search score token-to-token matches (MaxSim), which can improve fine-grained relevance.
The notebook linked below shows how to integrate XTR, which prioritizes critical document
tokens during the initial retrieval stage and removes the gathering stage to improve performance significantly.
By focusing on the most semantically salient tokens early in the process, XTR reduces computational complexity
while improving recall and ensuring rapid identification of candidate documents.
# Optimize Query Performance
Source: https://docs.lancedb.com/search/optimize-queries
Analyze and optimize query performance in LanceDB.
LanceDB provides two powerful tools for query analysis and optimization: `explain_plan` and `analyze_plan`. Let's take a better look at how they work:
| Method | Purpose | Description |
| :------------- | :----------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `explain_plan` | Query Analysis | Print the resolved query plan to understand how the query will be executed. Helpful for identifying slow queries or unexpected query results. |
| `analyze_plan` | Performance Tuning | Execute the query and return a physical execution plan annotated with runtime metrics including execution time, number of rows processed, and I/O stats. Essential for performance tuning and debugging. |
## Query Analysis Tools
### explain\_plan
Reveals the logical query plan before execution, helping you identify potential issues with query structure and index usage. This tool is useful for:
* Verifying query optimization strategies
* Validating index selection
* Understanding query execution order
* Detecting missing indices
### analyze\_plan
Executes the query and provides detailed runtime metrics, including:
* Operation duration (`_elapsed_compute_`)
* Data processing statistics (`_output_rows_`, `_bytes_read_`)
* Index effectiveness (`_index_comparisons_`, `_indices_loaded_`)
* Resource utilization (`_iops_`, `_requests_`)
Together, these tools offer a comprehensive view of query performance, from planning to execution. Use `explain_plan` to verify your query structure and `analyze_plan` to measure and optimize actual performance.
Metadata filters are prefiltered by default, which usually shows the filter pushed into the
`LanceScan` or index scan. If you set `prefilter=False`, expect a separate `FilterExec` after
search instead; that can be useful for some expensive filters, but it changes both latency and
the number of rows available after filtering.
## Reading the Execution Plan
To demonstrate query performance analysis, we'll use a table containing 1.2M rows sampled from the [Wikipedia dataset](https://huggingface.co/datasets/wikimedia/wikipedia). Initially, the table has no indices, allowing us to observe the impact of optimization.
Let's examine a vector search query that:
* Filters rows where `identifier` is between 0 and 1,000,000
* Returns the top 100 matches
* Projects specific columns: `chunk_index`, `title`, and `identifier`
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# explain_plan
query_explain_plan = (
table.search(query_embed)
.where("identifier > 0 AND identifier < 1000000")
.select(["chunk_index", "title", "identifier"])
.limit(100)
.explain_plan(True)
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// explain_plan
const explainPlan = await table
.search(queryEmbed)
.where("identifier > 0 AND identifier < 1000000")
.select(["chunk_index", "title", "identifier"])
.limit(100)
.explainPlan(true);
```
### Execution Plan Components
The execution plan reveals the sequence of operations performed to execute your query. Let's examine each component:
```
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance]
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title"
CoalesceBatchesExec: target_batch_size=1024
GlobalLimitExec: skip=0, fetch=100
FilterExec: _distance@3 IS NOT NULL
SortExec: TopK(fetch=100), expr=[_distance@3 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000
LanceScan: uri=***, projection=[vector, identifier], row_id=true, row_addr=false, ordered=false
```
#### 1. Base Layer (LanceScan)
* Initial data scan loading only specified columns to minimize I/O
* Unordered scan enabling parallel processing
```
LanceScan:
- projection=[vector, identifier]
- row_id=true, row_addr=false, ordered=false
```
#### 2. First Filter
* Apply requested filter on `identifier` column
* Reduces the number of vectors that need KNN computation
```
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000
```
#### 3. Vector Search
* Computes L2 (Euclidean) distances between query vector and all vectors that passed the filter
```
KNNVectorDistance: metric=l2
```
#### 4. Results Processing
* Filters out null distance results
* Sorts by distance and takes top 100 results
* Processes in batches of 1024 for optimal memory usage
```
SortExec: TopK(fetch=100)
- expr=[_distance@3 ASC NULLS LAST]
- preserve_partitioning=[false]
FilterExec: _distance@3 IS NOT NULL
GlobalLimitExec: skip=0, fetch=100
CoalesceBatchesExec: target_batch_size=1024
```
#### 5. Data Retrieval
* `RemoteTake` is a key component of Lance's I/O cache
* Handles efficient data retrieval from remote storage locations
* Fetches specific rows and columns needed for the final output
* Optimizes network bandwidth by only retrieving required data
```
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title"
```
#### 6. Final Output
* Returns only requested columns and maintains column ordering
```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance]
```
This plan demonstrates a basic search without index optimizations: it performs a full scan and filter before vector search.
## Performance Analysis
Let's use `analyze_plan` to run the query and analyze the query performance, which will help us identify potential bottlenecks:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# analyze_plan
query_analyze_plan = (
table.search(query_embed)
.where("identifier > 0 AND identifier < 1000000")
.select(["chunk_index", "title", "identifier"])
.limit(100)
.analyze_plan()
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// analyze_plan
const analyzePlan = await table
.search(queryEmbed)
.where("identifier > 0 AND identifier < 1000000")
.select(["chunk_index", "title", "identifier"])
.limit(100)
.analyzePlan();
```
### Performance Metrics Analysis
```
ProjectionExec: expr=[chunk_index@4 as chunk_index, title@5 as title, identifier@1 as identifier, _distance@3 as _distance], metrics=[output_rows=100, elapsed_compute=1.424µs]
RemoteTake: columns="vector, identifier, _rowid, _distance, chunk_index, title", metrics=[output_rows=100, elapsed_compute=175.53097ms, output_batches=1, remote_takes=100]
CoalesceBatchesExec: target_batch_size=1024, metrics=[output_rows=100, elapsed_compute=2.748µs]
GlobalLimitExec: skip=0, fetch=100, metrics=[output_rows=100, elapsed_compute=1.819µs]
FilterExec: _distance@3 IS NOT NULL, metrics=[output_rows=100, elapsed_compute=10.275µs]
SortExec: TopK(fetch=100), expr=[_distance@3 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=100, elapsed_compute=39.259451ms, row_replacements=546]
KNNVectorDistance: metric=l2, metrics=[output_rows=1099508, elapsed_compute=56.783526ms, output_batches=1076]
FilterExec: identifier@1 > 0 AND identifier@1 < 1000000, metrics=[output_rows=1099508, elapsed_compute=17.136819ms]
LanceScan: uri=***, projection=[vector, identifier], row_id=true, row_addr=false, ordered=false, metrics=[output_rows=1200000, elapsed_compute=21.348178ms, bytes_read=1852931072, iops=78, requests=78]
```
#### 1. Data Loading (LanceScan)
* Scanned 1,200,000 rows from the LanceDB table
* Read 1.86GB of data in 78 I/O operations
* Only loaded necessary columns (`vector` and `identifier`)
* Unordered scan for parallel processing
#### 2. Filtering & Search
* Applied prefilter condition (`identifier > 0 AND identifier < 1000000`)
* Reduced dataset from 1.2M to 1,099,508 rows
* KNN search used L2 (Euclidean) distance metric
* Vector comparisons processed in 1076 batches
#### 3. Results Processing
* KNN results sorted by distance (TopK with fetch=100)
* Null distances filtered out
* Batches coalesced to target size of 1024 rows
* Additional columns fetched for final results
* Remote take operation for 100 results
* Final projection of required columns
### Distributed metrics on remote tables
Enterprise
When you call `analyze_plan` against a LanceDB Enterprise table, the query runs across a pool of workers. By default the returned plan aggregates each operator's metrics into a single value, matching the local single-node output. Pass a `distributed_metrics` mode when you need to see how work was split across workers:
| Mode | What it shows | When to use it |
| :----------- | :---------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------- |
| `aggregate` | One summary value per operator (default). Same shape as local `analyze_plan`. | Everyday performance checks and comparisons against a local plan. |
| `per_worker` | Metrics reported separately for each worker that participated in the query. | Diagnosing stragglers or uneven work distribution across the cluster. |
| `full` | Both the aggregate summary and the per-worker breakdown. | Deep investigations where you need the totals *and* the per-worker detail in a single output. |
The parameter only affects remote plans — local queries always return the aggregate output regardless of what you pass.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Show a per-worker breakdown for a remote query
plan = (
table.search(query_embed)
.where("identifier > 0 AND identifier < 1000000")
.limit(100)
.analyze_plan(distributed_metrics="per_worker")
)
print(plan)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Show a per-worker breakdown for a remote query
const plan = await table
.search(queryEmbed)
.where("identifier > 0 AND identifier < 1000000")
.limit(100)
.analyzePlan("per_worker");
```
Use `per_worker` or `full` sparingly — the plan output grows with the size of your worker pool. Stick with the default `aggregate` mode for routine tuning.
### Key Observations
* Vector search is the primary bottleneck (1,099,508 vector comparisons)
* Significant I/O overhead (1.86GB data read)
* Full table scan due to lack of indices
* Substantial optimization potential through proper index implementation
## Optimized Query Execution
After creating vector and scalar indices, the execution plan shows:
```
ProjectionExec: expr=[chunk_index@3 as chunk_index, title@4 as title, identifier@2 as identifier, _distance@0 as _distance]
RemoteTake: columns="_distance, _rowid, identifier, chunk_index, title"
CoalesceBatchesExec: target_batch_size=1024
GlobalLimitExec: skip=0, fetch=100
SortExec: TopK(fetch=100), expr=[_distance@0 ASC NULLS LAST], preserve_partitioning=[false]
ANNSubIndex: name=vector_idx, k=100, deltas=1
ANNIvfPartition: uuid=83916fd5-fc45-4977-bad9-1f0737539bb9, nprobes=20, deltas=1
ScalarIndexQuery: query=AND(identifier > 0,identifier < 1000000)
```
### Optimized Plan Analysis
#### 1. Scalar Index Query
```
ScalarIndexQuery: query=AND(identifier > 0,identifier < 1000000)
metrics=[
output_rows=2
index_comparisons=2,301,824
indices_loaded=2
output_batches=1
parts_loaded=562
elapsed_compute=86.979354ms
]
```
* Range filter using scalar index
* Only 2 index files and 562 scalar index parts loaded
* 2.3M index comparisons for matches
#### 2. Vector Search
```
ANNSubIndex: name=vector_idx, k=100, deltas=1
metrics=[
output_rows=2,000
index_comparisons=25,893
indices_loaded=0
output_batches=20
parts_loaded=20
elapsed_compute=111.849043ms
]
```
* IVF index with 20 probes
* Only 20 index parts loaded
* 25,893 vector comparisons
* 2,000 matching vectors
#### 3. Results Processing
```
SortExec: TopK(fetch=100), expr=[_distance@0 ASC NULLS LAST], preserve_partitioning=[false]
GlobalLimitExec: skip=0, fetch=100
CoalesceBatchesExec: target_batch_size=1024
```
* Sorts by distance
* Limits to top 100 results
* Batches into groups of 1024
#### 4. Data Fetching
```
RemoteTake: columns="_distance, _rowid, identifier, chunk_index, title"
metrics=[output_rows=100, elapsed_compute=113.491859ms, output_batches=1, remote_takes=100]
```
* Single output batch
* One remote take per row
#### 5. Final Projection
```
ProjectionExec: expr=[chunk_index@3 as chunk_index, title@4 as title, identifier@2 as identifier, _distance@0 as _distance]
```
* Returns specified columns: chunk\_index, title, identifier, and distance
### Performance Improvements
#### 1. Initial Data Access
```
ScalarIndexQuery metrics:
- indices_loaded=2
- parts_loaded=562
- output_batches=1
```
* Before: Full table scan of 1.2M rows, 1.86GB data
* After: Only 2 indices and 562 scalar index parts loaded
* Benefit: Eliminated table scans for prefilter
#### 2. Vector Search Efficiency
```
ANNSubIndex:
- index_comparisons=25,893
- indices_loaded=0
- parts_loaded=20
- output_batches=20
```
* Before: L2 calculations on 1,099,508 vectors
* After:
* 99.8% reduction in vector comparisons
* Decreased output batches from 1,076 to 20
#### 3. Data Retrieval Optimization
```
RemoteTake:
- remote_takes=100
- output_batches=1
```
* RemoteTake operation remains consistent
## Performance Optimization Guide
### 1. Index Implementation
#### When to Create Indices
* Columns used in WHERE clauses
* Vector columns for similarity searches
* Join columns used in `merge_insert`
#### Index Type Selection
| Data Type | Recommended Index | Use Case |
| ----------- | ------------------------------------- | ---------------------------------------- |
| Vector | IVF\_PQ/IVF\_HNSW\_SQ/IVF\_HNSW\_FLAT | Approximate nearest neighbor search |
| Scalar | B-Tree | Range queries and sorting |
| Categorical | Bitmap | Multi-value filters and set operations |
| `List` | Label\_list | Multi-label classification and filtering |
Use `table.index_stats()` to monitor index coverage.
A well-optimized table should have `num_unindexed_rows ~ 0`.
### 2. Query Plan Optimization
#### Common Patterns and Fixes
| Plan Pattern | Optimization |
| ------------------------------------------- | -------------------------------------------- |
| LanceScan with high *bytes\_read* or *iops* | Add missing index |
| | Use `select()` to limit returned columns |
| | Check whether the dataset has been compacted |
| Multiple sequential filters | Reorder filter conditions |
!!! note "Regular Performance Analysis"
Regularly analyze your query plans to identify and address performance bottlenecks.
The `analyze_plan` output provides detailed metrics to guide optimization efforts.
### 3. Getting Started with Optimization
For vector search performance:
* Create ANN index on your vector column(s) as described in the [index guide](/indexing/vector-index/)
* If you often filter by metadata, create [scalar indices](/indexing/scalar-index/) on those columns
## Analyzing non-vector queries
`explain_plan` and `analyze_plan` aren't vector-specific — they're available on every query builder, including FTS and hybrid. The most common reason to look at the plan for a non-vector query is to confirm whether your `where` clause pushed into the scan (good) or ran as a separate `FilterExec` step on top of the search results (often slower, and a hint that the filter column needs a scalar index).
### FTS queries
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
plan = (
table.search("puppy", query_type="fts")
.where("category = 'animals'", prefilter=True)
.limit(10)
.explain_plan(True)
)
print(plan)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const plan = await table
.query()
.fullTextSearch("puppy")
.where("category = 'animals'")
.limit(10)
.explainPlan(true);
```
In an indexed FTS plan you should see a `MatchQuery` (or other FTS execution node) reading from the inverted index, with the metadata filter pushed down. If the plan shows a `LanceScan` followed by `FilterExec` over the entire text column, the FTS index either isn't covering the column or the filter isn't using a scalar index — both worth investigating.
### Hybrid queries
For hybrid queries, `explain_plan` returns the reranker label followed by the vector and FTS sub-plans, indented for readability:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
plan = (
table.search("flower moon", query_type="hybrid")
.where("category = 'film'", prefilter=True)
.limit(10)
.explain_plan(True)
)
print(plan)
# RRFReranker(...)
#
#
```
`analyze_plan` does the same, but executes both sub-queries and labels them as `Vector Search Plan:` and `FTS Search Plan:` in the output. This is the easiest way to see whether the filter pushed into both halves uniformly, and which half is dominating latency.
# Full-Text Search with SQL
Source: https://docs.lancedb.com/search/sql/fts-sql
Use LanceDB's full-text search capabilities via SQL queries.
Enterprise-only
This feature is currently in beta. The SQL syntax and JSON query format may change in future
releases as we continue to refine and improve the FTS SQL interface. We recommend testing
thoroughly and being prepared to update your queries as newer versions of LanceDB become available.
LanceDB provides support for full-text search via SQL queries using the `fts()` User-Defined Table Function (UDTF). This allows you to incorporate keyword-based search (based on BM25) in your SQL queries for powerful text retrieval.
The SQL `fts()` table function expects exactly two string literals: the table name and the JSON FTS query. Build the JSON query in your application, pass it as a SQL string literal, and keep filtering, grouping, or joining in the surrounding SQL.
## Table Setup
First, set up your FlightSQL client connection. See [SQL Queries documentation](/search/sql) for detailed client setup instructions.
For the examples below, we assume you have a `run_query()` helper function that executes SQL and returns results.
### Creating the Table
Create a table with text data:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
run_query("""
CREATE TABLE my_docs (
id INT,
text STRING,
category STRING,
author STRING
)
""")
```
### Inserting Data
Insert sample documents:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
run_query("""
INSERT INTO my_docs VALUES
(1, 'The happy puppy runs merrily in the park', 'animals', 'Alice'),
(2, 'A curious kitten jumps quickly over the fence', 'animals', 'Bob'),
(3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'),
(4, 'Dogs and cats are wonderful companions', 'animals', 'Charlie'),
(5, 'Puppy training requires patience and dedication', 'training', 'Alice'),
(6, 'The clever cat runs crazily around the house', 'animals', 'Bob'),
(7, 'Running in the park is excellent exercise', 'sports', 'Charlie'),
(8, 'Machine learning models process text efficiently', 'technology', 'David'),
(9, 'The fuzzy puppy loves to play with toys', 'animals', 'Alice'),
(10, 'Natural language processing enables text search', 'technology', 'David')
""")
```
### Creating FTS Index
Create a full-text search index on the text column:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
run_query("CREATE INDEX ON my_docs USING fts (text)")
```
To use phrase queries (exact phrase matching), create the index with `with_position = true`:
```sql SQL icon="SQL" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
CREATE INDEX ON my_docs USING fts (text) WITH (with_position = true)
```
Without position information, phrase queries will not work. See the [Phrase Queries](#phrase-queries) section below for details.
## Basic Full-Text Search
Use the `fts()` UDTF in SQL queries with JSON-formatted search queries:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery
# Create a match query and convert to JSON
query = MatchQuery("puppy", "text")
json_query = query.to_json()
# Execute FTS query via SQL - returns top 5 matches in arbitrary order
result = run_query(f"""
SELECT id, text, category
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
print(result.to_pandas())
# Output (4 documents match "puppy", showing all matches):
# id text category
# 0 1 The happy puppy runs merrily in the park animals
# 1 3 The puppy catches a ball with great enthusiasm sports
# 2 5 Puppy training requires patience and dedication training
# 3 9 The fuzzy puppy loves to play with toys animals
```
FTS queries compute a BM25 relevance score for each matching document and by default return the top 5 matching results in **arbitrary order**:
**For exact ordering by relevance**, select the special `_score` column and order by it:
```sql SQL icon="SQL" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
-- ✅ Returns top 5 matching results ordered by relevance (highest first)
SELECT id, text, _score FROM fts('my_docs', 'query')
ORDER BY _score DESC
LIMIT 5
```
**Key points:**
* Without `ORDER BY _score DESC`, you get the top matching results but in arbitrary order
* The `_score` column is optional - include it only when you need to see or order by relevance scores
* `_score` uses the BM25 ranking algorithm to measure relevance
## Advanced Query Types
### Fuzzy Search
Fuzzy search allows you to find matches even when the search terms contain typos:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery
# Search with fuzzy matching (allows 2 character edits)
query = MatchQuery("pupy", "text", fuzziness=2)
json_query = query.to_json()
result = run_query(f"""
SELECT id, text
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
print(result.to_pandas())
# Output - fuzzy matching finds "puppy" despite the typo "pupy":
# id text
# 0 9 The fuzzy puppy loves to play with toys
# 1 1 The happy puppy runs merrily in the park
# 2 5 Puppy training requires patience and dedication
# 3 3 The puppy catches a ball with great enthusiasm
```
### Phrase Queries
Search for exact phrases in documents:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import PhraseQuery
# Search for exact phrase
query = PhraseQuery("happy puppy", "text")
json_query = query.to_json()
result = run_query(f"""
SELECT id, text
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
```
For phrase queries to work, the FTS index must be created with `with_position=true`:
```sql SQL icon="SQL" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
CREATE INDEX ON my_docs USING fts (text) WITH (with_position = true)
```
#### Phrase Queries with Slop
Allow some flexibility in phrase matching with the `slop` parameter:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import PhraseQuery
# Allow up to 2 words between "puppy" and "park"
query = PhraseQuery("puppy park", "text", slop=2)
json_query = query.to_json()
result = run_query(f"""
SELECT id, text
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
```
### Boolean Queries
Combine multiple queries using boolean logic:
#### AND Queries
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery
# Find documents containing both "puppy" AND "happy"
query = MatchQuery("puppy", "text") & MatchQuery("happy", "text")
json_query = query.to_json()
result = run_query(f"""
SELECT id, text
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
```
#### OR Queries
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery
# Find documents containing either "puppy" OR "kitten"
query = MatchQuery("puppy", "text") | MatchQuery("kitten", "text")
json_query = query.to_json()
result = run_query(f"""
SELECT id, text, category
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
print(result.to_pandas())
# Output shows results matching either term:
# id text category
# 0 2 A curious kitten jumps quickly over the fence animals
# 1 9 The fuzzy puppy loves to play with toys animals
# 2 5 Puppy training requires patience and dedication training
# 3 1 The happy puppy runs merrily in the park animals
# 4 3 The puppy catches a ball with great enthusiasm sports
```
### Boost Queries
Control relevance by boosting or demoting certain terms:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery, BoostQuery
# Boost documents with "puppy", demote those with "kitten"
query = BoostQuery(
positive=MatchQuery("puppy", "text"),
negative=MatchQuery("kitten", "text"),
negative_boost=0.2
)
json_query = query.to_json()
result = run_query(f"""
SELECT id, text
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
```
### Multi-Match Queries
Search across multiple columns simultaneously:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MultiMatchQuery
# Search "puppy" in both text and category columns
query = MultiMatchQuery("puppy", ["text", "category"])
json_query = query.to_json()
result = run_query(f"""
SELECT id, text, category
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
```
#### Multi-Match with Field Boosting
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MultiMatchQuery
# Boost matches in "text" column 2x more than "category"
query = MultiMatchQuery("puppy", ["text", "category"], boosts=[2.0, 1.0])
json_query = query.to_json()
result = run_query(f"""
SELECT id, text, category
FROM fts('my_docs', '{json_query}')
LIMIT 5
""")
```
## Combining FTS with SQL
FTS queries can be combined with standard SQL features like WHERE clauses, GROUP BY, and JOINs:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.query import MatchQuery
query = MatchQuery("puppy", "text")
json_query = query.to_json()
# Combine FTS with WHERE clause to filter by category
result = run_query(f"""
SELECT id, text, category
FROM fts('my_docs', '{json_query}')
WHERE category = 'animals'
LIMIT 5
""")
```
## Query Parameters Reference
For detailed information about query parameters and options for `MatchQuery`, `PhraseQuery`, `BoostQuery`, and `MultiMatchQuery`, see the [Full-Text Search documentation](/search/full-text-search/).
## Related Documentation
* [Full-text search](/search/full-text-search/) - Learn about FTS capabilities and query types
* [SQL queries](/search/sql) - General SQL query documentation
* [Hybrid search](/search/hybrid-search/) - Combine FTS with vector search
# Query with SQL
Source: https://docs.lancedb.com/search/sql/index
SQL query capabilities in LanceDB Enterprise for analytical queries and data exploration.
Enterprise-only
[LanceDB Enterprise](/enterprise) comes with an SQL endpoint that can be used for analytical queries and data exploration. The SQL endpoint is designed to be compatible with the
[Arrow FlightSQL protocol](https://arrow.apache.org/docs/format/FlightSql.html), which allows you to use any Arrow FlightSQL-compatible client to query your data.
## Installing the client
There are Flight SQL clients available for most languages and tools. If you find that your
preferred language or tool is not listed here, please [reach out](mailto:contact@lancedb.com) to us and we can help you find a solution. The following examples demonstrate how to install the Python and TypeScript
clients.
```bash Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# The `flightsql-dbapi` package provides a Python DB API 2 interface to the
# LanceDB SQL endpoint. You can use it to connect to the SQL endpoint and
# execute queries directly and get back results in pyarrow format.
pip install flightsql-dbapi
```
```bash TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# LanceDB maintains a TypeScript client for the Arrow FlightSQL protocol.
# You can use it to connect to the SQL endpoint and execute queries directly.
# Results are returned in Arrow format or as plain JS/TS objects.
npm install --save @lancedb/flightsql-client
```
## Usage
LanceDB uses the powerful DataFusion query engine to execute SQL queries. This means that
you can use a wide variety of SQL syntax and functions to query your data. For more detailed
information on the SQL syntax and functions supported by DataFusion, please refer to the
[DataFusion documentation](https://datafusion.apache.org/user-guide/sql/index.html).
The FlightSQL endpoint executes one SQL statement per request and is intended for queries. Use the LanceDB SDKs for DDL and table-management operations such as creating tables, adding columns, or building indexes.
### Setting Up the Client
Establish a connection to your LanceDB Enterprise SQL endpoint using your preferred FlightSQL client:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from flightsql import FlightSQLClient
client = FlightSQLClient(
host="your-enterprise-endpoint",
port=10025,
insecure=True,
token="DATABASE_TOKEN",
metadata={"database": "your-project-slug"},
features={"metadata-reflection": "true"},
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import { Client } from "@lancedb/flightsql-client";
const client = await Client.connect({
host: "your-enterprise-endpoint:10025",
username: "lancedb",
password: "password",
});
```
### Executing a Query
Run SQL queries against your LanceDB tables. Different clients may handle the FlightSQL protocol differently:
```bash Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
def run_query(query: str):
"""Simple method to fully materialize query results"""
info = client.execute(query)
if len(info.endpoints) != 1:
raise Error("Expected exactly one endpoint")
ticket = info.endpoints[0].ticket
reader = client.do_get(ticket)
return reader.read_all()
result = run_query("SELECT * FROM flights WHERE origin = 'SFO'")
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const result = await client.query("SELECT * FROM flights WHERE origin = 'SFO'");
```
### Processing Results
Handle the query results returned by your FlightSQL client:
```bash Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
print(result)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Results are returned as plain JS/TS objects and we create an interface
// here for our expected structure so we can have strong typing. This is
// optional but recommended.
interface FlightRecord {
origin: string;
destination: string;
}
const flights = (await result.collectToObjects()) as FlightRecord[];
console.log(flights);
```
### Inspecting query plans
The SQL endpoint runs queries through DataFusion, which means DataFusion's `EXPLAIN` family of statements is available unchanged. They're the SQL counterpart of the Python/TypeScript [`explain_plan` and `analyze_plan` methods](/search/optimize-queries) and are useful for the same things: confirming index usage, checking filter pushdown, and finding the slow operator in a query that's underperforming.
| Statement | What it returns |
| :------------------------ | :---------------------------------------------------------------------------------- |
| `EXPLAIN ` | Logical and physical plan, without executing the query. |
| `EXPLAIN ANALYZE ` | Executes the query and annotates each operator with runtime metrics (rows, timing). |
| `EXPLAIN VERBOSE ` | Adds intermediate optimizer plans on top of `EXPLAIN`. |
Run them through the same client you use for regular queries — the result is a small Arrow table with `plan_type` and `plan` columns:
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
plan = run_query(
"EXPLAIN ANALYZE SELECT origin, destination "
"FROM flights WHERE origin = 'SFO' LIMIT 100"
)
for row in plan.to_pylist():
print(row["plan_type"])
print(row["plan"])
print()
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const plan = await client.query(
"EXPLAIN ANALYZE SELECT origin, destination " +
"FROM flights WHERE origin = 'SFO' LIMIT 100"
);
for (const row of (await plan.collectToObjects()) as Array<{ plan_type: string; plan: string }>) {
console.log(row.plan_type);
console.log(row.plan);
}
```
The operators that show up in the SQL plan are the same ones documented on the [Optimize Query Performance](/search/optimize-queries) page (`LanceScan`, `ScalarIndexQuery`, `KNNVectorDistance`, `ANNIvfPartition`, and so on), so the same reasoning about index coverage and filter pushdown applies - just read the plan from a SQL client instead of a query builder.
For full-text search from SQL, use the dedicated [`fts()` table function](/search/sql/fts-sql). It takes two string-literal arguments: the table name and the JSON-encoded FTS query, which can include operator-style terms, OR, phrase, and fuzzy query forms.
# Vector Search
Source: https://docs.lancedb.com/search/vector-search
Learn how to run vector search queries in LanceDB. Includes best practices, tips and examples.
Vector search is a technique used to search for similar items based on their vector representations, called embeddings. It is also known as similarity search, nearest neighbor search, or approximate nearest neighbor search.
![]()
Raw data (e.g. text, images, audio, etc.) is converted into embeddings via an embedding model, which are then stored in a multimodal lakehouse like LanceDB. To perform similarity search at scale, an index is created on the stored embeddings, which can then used to perform fast lookups.
## Supported distance metrics
Distance metrics determine how LanceDB compares vectors to find similar matches. Euclidean or `l2` is the default, and used for general-purpose similarity, `cosine` for unnormalized embeddings, `dot` for normalized embeddings (best performance), or `hamming` for binary vectors.
Ensure you always use the same distance metric that your embedding model was trained with. Most modern embedding models use cosine similarity, so `cosine` is often the best choice. However, if your vectors are normalized, you should use `dot` for best performance.
The right metric improves both search accuracy and query performance. Currently, LanceDB supports the following metrics:
| Distance metric | Mathematical form | Notes |
| --------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `l2` | $\|x-y\|_2=\sqrt{\sum_i (x_i-y_i)^2}$ | Measures the straight-line distance between two points in vector space. Calculated as the square root of the sum of squared differences between corresponding vector components. |
| `cosine` | $1-\frac{x\cdot y}{\|x\|_2\|y\|_2}$ | Measures directional difference between vectors. Computed as 1 minus cosine similarity (the dot product normalized by both vector magnitudes), so vector length does not affect the score. Use for unnormalized vectors. |
| `dot` | $x\cdot y=\sum_i x_i y_i$ | Calculates the sum of products of corresponding vector components. Provides raw similarity scores without normalization, sensitive to vector magnitudes. Use for normalized vectors for best performance. |
| `hamming` | $\sum_i \mathbf{1}[x_i\neq y_i]$ | Counts the number of positions where corresponding bits differ between binary vectors. Only applicable to binary vectors stored as packed uint8 arrays. |
For indexed search, supported distance metrics vary by index type:
| Index type | Supported distance metrics |
| --------------- | ------------------------------------ |
| `IVF_FLAT` | `["l2", "cosine", "dot", "hamming"]` |
| `IVF_PQ` | `["l2", "cosine", "dot"]` |
| `IVF_SQ` | `["l2", "cosine", "dot"]` |
| `IVF_RQ` | `["l2", "cosine", "dot"]` |
| `IVF_HNSW_FLAT` | `["l2", "cosine", "dot"]` |
| `IVF_HNSW_PQ` | `["l2", "cosine", "dot"]` |
| `IVF_HNSW_SQ` | `["l2", "cosine", "dot"]` |
### Configure Distance Metric
By default, `l2` will be used as metric type. You can specify the metric type as
`cosine` or `dot` if required (`hamming` is supported for `IVF_FLAT` index only).
**Note:** You can configure the distance metric during search only if there's no vector index. If a vector index exists, the distance metric will always be the one you specified when creating the index.
Here you can see the same search but using `cosine` similarity instead of `l2` distance. The result focuses on vector direction rather than absolute distance, which works better for normalized embeddings.
Set `.limit(...)` on vector searches you run in applications. You can page through results with `.offset(...)` and include LanceDB's internal row id with `.with_row_id()` / `.withRowId()` when you need a stable handle for follow-up operations.
## Selecting the vector column
If your table has exactly one vector column, you can omit the column name and LanceDB will pick it for you. This works for both top-level columns (such as `vector`) and vector fields nested inside a struct (such as `image.embedding`).
When LanceDB can't infer a single column, it raises a `ValueError` (Python) or rejects the query (Node/Rust). Two cases trigger this:
* No vector column: the schema has no `fixed_size_list` or `list` of floats.
* Multiple candidates: more than one column matches the query's dimension. The error lists every candidate path so you can pick one explicitly.
To disambiguate, pass the field path with dot notation. Wrap any segment that contains characters outside `[A-Za-z0-9_]` in backticks (for example, `` `image-meta`.`embedding.v1` ``).
The same field-path syntax works when creating an index on a nested vector column:
When several columns share a name across structs (for example, `image.embedding` and `text.embedding`), LanceDB still picks the one whose dimension matches your query vector. If two candidates have the same dimension, you must pass the column name explicitly.
## Vector Search With ANN Index
Instead of performing an exhaustive search on the entire database for each and every query, approximate nearest neighbour (ANN) algorithms use an index to narrow down the search space, which significantly reduces query latency.
The trade-off is that the results are not guaranteed to be the true nearest neighbors of the query, but are usually "good enough" for most use cases.
Use ANN search for large-scale applications where speed matters more than perfect recall. LanceDB uses approximate nearest neighbor algorithms to deliver fast results without examining every vector in your dataset.
When a vector index is used, `_distance` is not always the true distance between full vectors. On quantized ANN indexes, LanceDB may compute `_distance` from the compressed representation for speed. Use `refine_factor` when you want reranking on full vectors.
### Exact vs Approximate Distances
When doing vector search, the meaning of "distance" depends on whether you are using an index and whether `refine_factor` is specified as part of your query.
`nprobes` controls how many partitions are searched to find candidates, `approx_mode` controls the query-time speed/recall trade-off for RQ-quantized indexes, and `refine_factor` controls how many candidates are rescored on full vectors for better distance fidelity and reranking quality.
The table below summarizes the behavior of `_distance` in search results based on your query configuration:
| Query mode | Neighbor quality | `_distance` in results |
| :----------------------------------- | :----------------------------------------- | :---------------------------------------------------------------------------------------------- |
| No index or `.bypass_vector_index()` | Exact kNN (100% recall) | True distance on full vectors |
| Indexed ANN, no `refine_factor` | Approximate neighbors | Distance on the index representation: exact for flat indexes, approximate for quantized indexes |
| Indexed ANN + `refine_factor(1)` | Approximate neighbors (same candidate set) | Distances recomputed on full vectors for reranked candidates |
| Indexed ANN + `refine_factor(>1)` | Better recall than no refine (usually) | Distances recomputed on full vectors for reranked candidates |
For deeper tuning guidance on indexing and performance estimation, see the [vector indexes](/indexing/vector-index/#search-configuration) page,
For tuning `nprobes`, see below.
### Tuning `approx_mode`
Use `approx_mode` when you want to adjust the speed/recall trade-off for approximate vector search at query time. This setting currently applies only to RQ-quantized indexes, such as `IVF_RQ`; other index types ignore it.
The supported values are:
| Value | Behavior |
| :--------- | :-------------------------------------------------------------------------------------------------------------------------- |
| `fast` | Prefer lower query latency, which can reduce recall. |
| `normal` | Use the default balance between query latency and recall. This only has an effect for RQ indexes built with `num_bits > 1`. |
| `accurate` | Prefer higher recall, which can increase query latency. |
You can change `approx_mode` per query without rebuilding the index. For RQ indexes built with `num_bits=1`, `normal` uses the same one-bit scoring path as `fast`. If you also set `refine_factor`, LanceDB first uses `approx_mode` while finding candidates, then reranks the selected candidates on full vectors.
### Tuning `nprobes`
* `nprobes` controls how many partitions are searched at query time.
* `nprobes` improves candidate recall, but does not by itself make `_distance` exact.
* By default, LanceDB automatically tunes `nprobes` to achieve the best performance without noticeably sacrificing accuracy.
* In most cases, leave `nprobes` unset and use the auto-tuned value.
* Only tune `nprobes` manually when recall is below your target, or when you need even higher performance for your workload.
* If recall is too low, increase `nprobes` gradually, but after a certain threshold, increasing `nprobes` yields only marginal accuracy gains.
* If you need higher performance and have recall headroom, decrease `nprobes` gradually.
For filtered ANN searches, you can also set `minimum_nprobes` and `maximum_nprobes`. LanceDB starts
with the minimum and can scan more partitions up to the maximum if the filter leaves too few
candidates. Calling `nprobes(n)` fixes both values to `n`, which disables that adaptive behavior.
### Vector Search with Prefiltering
This is the default vector search setting. You can use prefiltering to boost query performance by reducing the search space before vector calculations begin. The system first applies your filter criteria to the dataset, then conducts vector search operations only on the remaining relevant subset.
This filters out rows where label ≤ 2 before doing vector search, then picks specific columns from the top 5 matches.
The `.where("label > 2")` applies a filter before vector search, `.select(["text", "keywords", "label"])` chooses specific columns to return, and `.limit(5)` restricts results to the top `5` most similar vectors.
As a result, you'll see a result with just the data you want from the most similar vectors.
### Vector Search with Postfiltering
Use postfiltering to prioritize vector similarity by searching the full dataset first, then applying metadata filters to the top results. This approach ensures you get the most similar vectors before filtering, which can be crucial when similarity is more important than metadata constraints.
Here you can see how to do vector search first to get the most similar vectors, then filter by label > 1 on those results.
The `prefilter=False` parameter tells LanceDB to apply the filter after vector search instead of before, `.where("label > 1")` filters the top results by metadata, and `.select()` chooses which columns to include.
In the end, you receive a query result with the best matches that also meet your metadata requirements.
[Post-filtering](/search/filtering/#post-filtering-with-vector-search) in LanceDB applies
the filter condition after obtaining the nearest neighbors based on vector similarity.
## Multivector Search
Use multivector search when your documents contain multiple embeddings and you need sophisticated matching between query and document vector pairs. The late interaction approach finds the most relevant combinations across all available embeddings and provides nuanced similarity scoring.
Only `cosine` similarity is supported as the distance metric for multivector search operations.
Every query vector must match the inner dimension of the multivector column; LanceDB rejects mismatched query dimensions rather than guessing how to reshape them.
Here you can see how to take 2 query vectors and find the best matching pairs between them and document vectors using late interaction. The `np.random.random(size=(2, 256))` creates a 2×256 array with two random query vectors, `.limit(5)` returns the top 5 best document-query combinations, and `.to_pandas()` provides results in a DataFrame format.
**Read more:** [Multivector search](/search/multivector-search/)
## Advanced Search Scenarios
### Search With Distance Range
Use `distance_range` search when you need vectors within particular similarity bounds rather than just the closest neighbors. The system filters results to only include vectors that fall within your specified distance thresholds from the query.
This shows three ways to search within distance ranges: bounded, upper bound only, and lower bound only.
The `distance_range()` method filters results by similarity thresholds - the first example finds vectors with distance between `0.1` and `0.5`, the second finds vectors closer than `0.5`, and the third finds vectors farther than `0.1`.
Each approach returns Arrow tables with vectors that fall within your specified distance thresholds.
### Search With Binary Vectors
Use binary vector search for scenarios involving binary embeddings, such as those produced by hashing algorithms. The system stores these efficiently as packed uint8 arrays and uses Hamming distance calculations to determine vector similarity.
The number of dimensions of the binary vector must be a multiple of 8. A vector of dimensionality 128 will be stored as a `uint8` array of size 16.
Here you can see how to set up a table for binary vectors, pack them efficiently into bytes, and search using Hamming distance.
The schema defines a 32-byte vector field (256 bits ÷ 8), `np.random.randint(0, 2, size=256)` creates binary vectors, `np.packbits()` compresses them to bytes, and `.distance_type("hamming")` specifies `hamming` distance for similarity calculation.
The search produces an Arrow table with binary vectors ranked by how many bits differ from the query.
## Scaling Vector Search
### Batch Search
Use batch search to handle multiple query vectors simultaneously. This gives you significant efficiency gains over individual queries. LanceDB processes all vectors in parallel and organizes results with a `query_index` field that maps each result set back to its originating query.
This takes 5 query embeddings and finds the top 5 matches for each one in a single batch operation.
The `load_dataset()` loads embeddings from a Hugging Face dataset, `query_embeds` contains `5` query vectors, and `.search(query_embeds)` processes all queries simultaneously.
The final query result contains all results, including a `query_index` to tell you which query each result came from.
When processing batch queries, the results include a `query_index` field
to explicitly associate each result set with its corresponding query in
the input batch.
### Search With Asynchronous Indexing
To optimize for speed over completeness, enable the `fast_search` flag in your query to skip searching unindexed data.
While vector indexing occurs asynchronously, newly added vectors are immediately
searchable through a fallback brute-force search mechanism. This ensures zero
latency between data insertion and searchability, though it may temporarily
increase query response times.
Here you can see how to turn on fast search mode to skip unindexed vectors and only look through indexed data for speed.
The `fast_search=True` parameter tells LanceDB to only search indexed vectors, skipping any recently added data that hasn't been indexed yet.
You'll obtain a query result with the top `5` matches from indexed vectors, but might miss data that was just added.
## Brute Force Search
### Search With No Index
The simplest way to perform vector search is to perform a brute force search, without an index, where the distance between the query vector and all the vectors in the database are computed, with the top-k closest vectors returned.
This is equivalent to a k-nearest neighbours (kNN) search in vector space.
Choose brute force search when you need guaranteed 100% recall, typically with smaller datasets where query speed isn't the primary concern. The system scans every vector in the table and calculates precise distances to find the exact nearest neighbors.
This carries out a brute force search through every vector in the table to find the 3 closest matches to a random 1536-dimensional query. You'll get back a list of the most similar vectors with exact distances.
![]()
As you can imagine, the brute force approach is not scalable for datasets larger than a few hundred thousand vectors, as the latency of the search grows linearly with the size of the dataset. This is where approximate nearest neighbour (ANN) algorithms come in.
### Bypass the Vector Index
Use `bypass_vector_index` to get exact, ground-truth results by performing exhaustive searches across all vectors. Instead of relying on approximate methods, the system directly compares your query against every vector in the table, ensuring 100% recall at the cost of increased query time.
This skips the approximate index and checks every single vector for exact, ground-truth results.
The `.bypass_vector_index()` method forces LanceDB to perform an exhaustive search through all vectors instead of using the approximate nearest neighbor index, ensuring exact results but at the cost of slower performance.
The output is a query result with the top 5 exact matches, guaranteeing 100% recall but taking longer to run.
This approach is particularly useful when:
* Evaluating ANN index quality
* Calculating recall metrics to tune index parameters
* Ensuring exact results for critical applications
# Configuring Cloud Storage in LanceDB
Source: https://docs.lancedb.com/storage/configuration
Configure LanceDB to use S3, GCS, Azure Blob, and S3-compatible object stores with environment variables or storage options.
When using LanceDB OSS, you can choose where to store your data. The tradeoffs between storage options are covered in the [storage architecture guide](/storage). This page shows how to configure each backend.
**LanceDB Enterprise storage configuration**
In LanceDB Enterprise, you connect with `db://...` and the cluster owns the storage credentials, so `storage_options` are not passed at runtime. Cloud auth is set at deployment time. For federated databases, the namespace service vends per-request credentials automatically. See the [quickstart](/quickstart), [Enterprise overview](/enterprise/), and [Azure deployment guide](/enterprise/deployment/azure) for the Enterprise flow.
## Object stores
LanceDB supports AWS S3 (and compatible stores), Azure Blob Storage, and Google Cloud Storage. The URI scheme in your `connect` call selects the backend.
### Configuration options
When running inside the target cloud with correct IAM bindings, LanceDB often needs no extra configuration. When running elsewhere, provide credentials via environment variables or `storage_options`.
**Storage option casing**
Keys are case-insensitive. Use lowercase in `storage_options` and uppercase in environment variables.
Table-level `storage_options` inherit every key from the connection and override on a per-key basis. Pass them to `create_table` or `open_table` for options that should apply to a single table:
**Inspect the effective options**
On `AsyncTable`, `await table.initial_storage_options()` returns the options the table was opened with, and `await table.latest_storage_options()` returns the current options after any provider-driven refresh. The deprecated `table.storage_options()` method will be removed in a future release.
#### General object store options
| Key | Description |
| :--------------------------- | :-------------------------------------------------------------- |
| `allow_http` | Allow non-TLS connections. |
| `allow_invalid_certificates` | Skip certificate validation for TLS connections. |
| `connect_timeout` | Timeout for the connect phase. |
| `timeout` | Timeout for the full request. |
| `user_agent` | User agent string sent with requests. |
| `proxy_url` | Proxy URL to route requests through. |
| `proxy_ca_certificate` | PEM-formatted CA certificate for proxy connections. |
| `proxy_excludes` | Comma-separated hosts that bypass the proxy (domains or CIDR). |
| `download_retry_count` | Number of retries when downloading objects. |
| `client_max_retries` | Maximum retries for object-store client requests. |
| `client_retry_timeout` | Total retry timeout (seconds) for object-store client requests. |
**Option support varies by backend**
These are commonly used options. Cloud-specific keys (for example `region`, `endpoint`, `service_account`, and Azure credential keys) are backend-dependent and can be provided in `storage_options` as needed.
#### New table configuration
These options control the Lance file format and features used when creating new tables. Pass them via `storage_options` at connection or table level. They are evaluated only at table creation; setting them on an existing connection does not rewrite or alter tables that already exist.
| Key | Values | Default | Description |
| :----------------------------------- | :----------------- | :------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `new_table_data_storage_version` | `legacy`, `stable` | `stable` | Lance file format version for new tables. Use `legacy` for backward compatibility with older clients, or `stable` for the current format with better performance. |
| `new_table_enable_v2_manifest_paths` | `true`, `false` | `false` | Use v2 manifest path naming. Requires LanceDB >= 0.10.0 to read. |
| `new_table_enable_stable_row_ids` | `true`, `false` | `false` | Keep row IDs stable across compaction, delete, and merge operations. |
```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
flowchart TD
A[Creating a new table] --> B{Need backward compatibility\nwith older LanceDB clients?}
B -->|Yes| C[new_table_data_storage_version: legacy]
B -->|No| D[new_table_data_storage_version: stable\nDefault — recommended]
D --> E{Need stable row IDs\nacross compaction and deletes?}
E -->|Yes| F[new_table_enable_stable_row_ids: true]
E -->|No| G[Default: false]
D --> H{All clients on LanceDB >= 0.10.0?}
H -->|Yes| I[new_table_enable_v2_manifest_paths: true]
H -->|No| J[Default: false]
```
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
# Set the Lance file format version at connection level
db = lancedb.connect(
"s3://bucket/path",
storage_options={
"new_table_data_storage_version": "stable",
},
)
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import * as lancedb from "@lancedb/lancedb";
// Set the Lance file format version at connection level
const db = await lancedb.connect("s3://bucket/path", {
storageOptions: {
newTableDataStorageVersion: "stable",
},
});
```
**Deprecated parameter**
The `data_storage_version` parameter on `create_table()` is deprecated. Use `new_table_data_storage_version` in `storage_options` instead.
## AWS S3
![]()
Set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optionally `AWS_SESSION_TOKEN` as environment variables or pass them in `storage_options`. Region is optional for AWS but required for most S3-compatible stores.
Minimum permissions usually include `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:ListBucket`, and `s3:GetBucketLocation` scoped to the relevant bucket/prefix.
### S3-compatible stores
If the endpoint is `http://` (common in local development), also set `ALLOW_HTTP=true` or pass `allow_http=True` in `storage_options`.
### S3 Express
Consult AWS networking requirements for S3 Express before enabling.
**Clean up failed multipart uploads**
LanceDB aborts multipart uploads on graceful shutdown, but crashes can leave incomplete uploads. Add an S3 lifecycle rule to delete in-progress uploads after a few days.
### Server-side encryption with KMS
To encrypt at rest with an AWS KMS key, set `aws_server_side_encryption` to `aws:kms` and `aws_sse_kms_key_id` to the key ID or ARN. The same options apply at connection or table level and combine with bucket-level default encryption.
The IAM principal needs `kms:Encrypt`, `kms:Decrypt`, and `kms:GenerateDataKey` on the configured KMS key.
## Google Cloud Storage
![]()
Provide credentials via `GOOGLE_SERVICE_ACCOUNT` (path to JSON) or include the path in `storage_options`. GCS defaults to HTTP/1; set `HTTP1_ONLY=false` if you need HTTP/2.
## Azure Blob Storage
![]()
Set `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY` as environment variables, or pass them via `storage_options`.
For SAS-token auth, set `azure_storage_account_name` and `azure_storage_sas_token`:
Other supported keys include service principal credentials (`azure_client_id`, `azure_client_secret`, `azure_tenant_id`), managed identities, and custom endpoints.
## Tigris Object Storage
![]()
Tigris exposes an S3-compatible API. Configure the endpoint and region:
Environment variables `AWS_ENDPOINT=https://t3.storage.dev` and `AWS_DEFAULT_REGION=auto` achieve the same configuration.
## Tencent COS
[Tencent Cloud Object Storage (COS)](https://www.tencentcloud.com/products/cos) is the primary object store for workloads running in the China region. Use the `cos://` URI scheme to connect directly to a COS bucket.
Supported keys include `secret_id`, `secret_key`, `region`, and `endpoint`. You can also authenticate via the `TENCENTCLOUD_SECRET_ID` and `TENCENTCLOUD_SECRET_KEY` environment variables.
**Availability**
COS is bundled in the Python wheel by default. To use it from Rust or the Node binding, build with the `cos` Cargo feature enabled.
## GooseFS
[GooseFS](https://www.tencentcloud.com/document/product/1424) is Tencent Cloud's distributed cache acceleration layer for COS and S3. It is a common choice when the same hot dataset is read repeatedly, such as vector search and AI training workloads. Connect using the `goosefs://` URI scheme.
GooseFS reads credentials and endpoint configuration from the GooseFS client environment. See the [GooseFS documentation](https://www.tencentcloud.com/document/product/1424) for cluster setup.
**Availability**
GooseFS is bundled by default in the Python wheel and the Node binding. To use it from Rust, build with the `goosefs` Cargo feature enabled.
# Storage Architecture in LanceDB
Source: https://docs.lancedb.com/storage/index
Understand LanceDB storage backends, tradeoffs, and how to pick the right option for your latency, scale, and cost goals.
LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs).
Choosing a backend is a balance between latency, scalability, cost, and operational complexity. Use this guide to pick the right fit for your workload.
## Storage backend selection guide
![]()
When architecting your system, ask yourself:
* **Latency**: How fast do I need results? What do the p50 and p95 look like?
* **Scalability**: Can I scale data volume and QPS easily?
* **Cost**: What is the all-in cost of storage plus serving?
* **Reliability/Availability**: How will replication and disaster recovery work?
## Storage backend comparison
Below is a high-level comparison ordered from lowest cost to lowest latency.
### 1. Object storage (S3 / GCS / Azure Blob)
* **Latency**: Highest; expect hundreds of milliseconds and higher p95.
* **Scalability**: Effectively unlimited storage; QPS bound by concurrency limits.
* **Cost**: Lowest overall.
* **Reliability/Availability**: Highly available, backed by cloud SLAs.
LanceDB separates storage and compute and writes immutable fragments, making it a strong fit for stateless, horizontally scalable deployments.
**Concurrent writers on S3**
S3 and S3 Express now support atomic writes natively, so LanceDB handles concurrent writers against the same table out-of-the-box — no external commit coordinator is required. Bucket-level [server-side encryption with KMS](/storage/configuration#server-side-encryption-with-kms) and [S3 Express One Zone](/storage/configuration#s3-express) are also supported on this tier.
### 2. File storage (EFS / GCS Filestore / Azure File)
* **Latency**: Better than object storage; p95 under \~\<100ms is typical.
* **Scalability**: High, but limited by provisioned IOPS per volume.
* **Cost**: More than object storage but cheaper than in-memory options; cold data can tier down automatically.
* **Reliability/Availability**: Highly available; replication/backup must be managed separately.
Keep a copy of data in object storage for disaster recovery. If zero downtime is required, provision a second network file system with replicated data.
### 3. Third-party storage (e.g., MinIO, WekaFS)
* **Latency**: Similar to EFS; typically under \<100ms.
* **Scalability**: Determined by the chosen vendor’s cluster sizing.
* **Cost**: Higher than S3; may edge above EFS at larger scales.
* **Reliability/Availability**: Shareable across many nodes; replication depends on vendor capabilities.
### 4. Block storage (EBS / GCP Persistent Disk / Azure Managed Disk)
* **Latency**: Near-local performance; often \<30ms.
* **Scalability**: Not shareable across instances; shard or copy data when scaling.
* **Cost**: Higher than networked file systems, plus potential I/O charges.
* **Reliability/Availability**: Persists through instance restarts; backups and sharding must be managed.
### 5. Local storage (SSD / NVMe)
* **Latency**: Fastest; p95 often under \<10ms.
* **Scalability**: Hard to scale in cloud environments; requires sharding or additional copies for higher QPS.
* **Cost**: Highest; tightly coupling compute and storage makes horizontal scaling difficult.
* **Reliability/Availability**: Data is tied to the instance; backups must be rigorous.
Use local disk only when you need extremely low latency and are comfortable owning the operational overhead.
## File-format choices that interact with the backend
A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full on the [configuration page](/storage/configuration#new-table-configuration); the architecture-level summary is:
* `new_table_enable_v2_manifest_paths` matters most on object stores, where opening a table with many versions is dominated by listing cost. Leave it off for backward compatibility with clients older than LanceDB 0.10.0.
* `new_table_enable_stable_row_ids` keeps row IDs stable across compaction, delete, and merge. The choice is independent of the backend but affects any system that joins on row ID.
* `new_table_data_storage_version` selects the on-disk format. The default `stable` is recommended for all new tables; pick `legacy` only when older readers must keep working.
# Monitor LanceDB with OpenTelemetry
Source: https://docs.lancedb.com/storage/monitoring
Export LanceDB object store request counts, bytes, latency, errors, and throttles to any OpenTelemetry backend.
LanceDB emits internal metrics (currently object store request counts, bytes transferred, request latency, retryable errors, and throttles) and can bridge them into any [OpenTelemetry](https://opentelemetry.io/) backend. Use this to watch how your application interacts with S3, GCS, Azure Blob, or the local filesystem in production: spot latency regressions, catch retry storms, and size your storage tier from real workload data.
The bridge is available in the Python and TypeScript SDKs. It is a thin wrapper over LanceDB's `metrics` recorder; your application supplies and configures the OpenTelemetry SDK.
This page covers LanceDB OSS. LanceDB Enterprise clusters emit their own Prometheus/OpenTelemetry metrics from the server side — see the [Enterprise overview](/enterprise/) for that flow.
## What you get
Once instrumented, LanceDB registers one observable instrument per metric on your `MeterProvider`. The current catalog covers the object store layer:
| Metric | Kind | Description |
| ---------------------------------------------- | --------- | ------------------------------------------------------------------------------- |
| `lance_object_store_requests_total` | Counter | Total object store requests, labelled by `operation` and `base` (store scheme). |
| `lance_object_store_request_duration_seconds` | Histogram | Request latency in seconds. |
| `lance_object_store_bytes_transferred_total` | Counter | Bytes read from or written to the store. |
| `lance_object_store_retryable_responses_total` | Counter | Requests that returned a retryable error (throttles, transient failures). |
| `lance_object_store_in_flight_requests` | Gauge | Currently outstanding object store requests. |
The recorder is process-global and pull-based: your configured `MetricReader` collects on its own schedule, so there is no hot-path overhead beyond the atomic aggregation that LanceDB does anyway.
**Histograms are exported Prometheus-style.** OpenTelemetry has no asynchronous histogram instrument, so each histogram surfaces as three observable counters: `_bucket` (with an `le` attribute per bucket boundary, including `+Inf`), `_count`, and `_sum`. Only `_sum` carries the histogram's unit; `_bucket` and `_count` are cumulative sample counts.
## Python
Install LanceDB with the `otel` extra to pull in the OpenTelemetry API, plus an OpenTelemetry SDK of your choice. The SDK is intentionally not bundled, so you configure it and its readers and exporters however your platform expects.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install "lancedb[otel]" opentelemetry-sdk
```
Call `instrument_lancedb_metrics()` once at startup, before opening any tables. It returns `True` when the recorder is installed and instruments are registered.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
from lancedb.otel import instrument_lancedb_metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import (
PeriodicExportingMetricReader,
)
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
OTLPMetricExporter,
)
reader = PeriodicExportingMetricReader(OTLPMetricExporter())
provider = MeterProvider(metric_readers=[reader])
instrument_lancedb_metrics(provider)
# Any object store activity from this point on is now recorded.
db = lancedb.connect("s3://my-bucket/lancedb")
```
If you omit `meter_provider`, LanceDB uses the global provider returned by `opentelemetry.metrics.get_meter_provider()`.
`instrument_lancedb_metrics()` returns `False` and emits a warning if another `metrics`-crate recorder is already installed in the process. Only one global recorder is permitted, so instrument LanceDB before any other library that installs its own recorder.
## TypeScript
The Node SDK depends on `@opentelemetry/api` directly, so no extra install step is needed to expose the entry point. You still need an OpenTelemetry SDK to actually export.
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
npm install @opentelemetry/sdk-metrics @opentelemetry/exporter-metrics-otlp-grpc
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import { connect, instrumentLanceDbMetrics } from "@lancedb/lancedb";
import { MeterProvider, PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-grpc";
const reader = new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter(),
});
const provider = new MeterProvider({ readers: [reader] });
instrumentLanceDbMetrics(provider);
const db = await connect("s3://my-bucket/lancedb");
```
`instrumentLanceDbMetrics()` also accepts no arguments, in which case it uses the global provider from `@opentelemetry/api`. Calling it more than once is safe: instruments are created only on the first successful call.
## What to watch
A few starting points for dashboards and alerts:
* **Request rate by operation:** `rate(lance_object_store_requests_total[1m])` broken down by `operation` shows read vs. write pressure and helps size ingestion and serving traffic separately.
* **Tail latency:** histogram quantiles over `lance_object_store_request_duration_seconds_bucket` catch object store slowdowns before they surface as query timeouts.
* **Retryable responses:** a rising `lance_object_store_retryable_responses_total` typically means you are being throttled and should back off or shard writes.
* **In-flight requests:** a growing `lance_object_store_in_flight_requests` gauge without a matching rise in throughput indicates queueing.
## Where to go next
Tune ingestion, indexing, and query patterns once metrics highlight a hot spot.
Configure the object store backends whose requests these metrics measure.
# Tables and Namespaces
Source: https://docs.lancedb.com/tables-and-namespaces
Learn more about the table abstraction and namespaces in LanceDB.
Despite its name, LanceDB is not a "database" in the traditional sense. It is a **Multimodal Lakehouse** built on Lance tables plus a catalog abstraction.
As you dive deeper into LanceDB, it helps to separate two ideas:
* A **table** is where your data lives and is queried.
* A **namespace** is how groups of tables are organized and resolved at the catalog level.
## Understanding tables
A table is the core data abstraction in LanceDB: a structured dataset with schema, indexes, and versioned updates.
What changes between deployments is how that table is addressed and accessed.
The mental model below clarifies table types by connection mode:
* **`LanceTable`**: direct table access (local path, `file://`, `s3://`, and similar object-store paths). This is the common mode in LanceDB OSS.
* **`RemoteTable`**: catalog-backed table access through a server/cluster (`db://...`). This is the mode you will use in LanceDB Enterprise.
![]()
From an application perspective, both expose a familiar table API: create/open tables, mutate rows, and query data.
The main difference is where resolution and execution happen (directly against storage vs through a remote catalog service).
## Semantic difference between tables and namespaces
The easiest way to think about this is:
* A **table** answers: "What data do I store and query?"
* A **namespace** answers: "Where does this table name live in my catalog hierarchy?"
In other words, tables are data objects; namespaces are catalog objects.
| Concept | Scope | Owns | Typical operations |
| --------- | ------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Table | Data layer | Schema, rows, indexes, versions | `create_table`, `open_table`, inserts/updates/deletes, search/query |
| Namespace | Catalog layer | Hierarchy of names, table grouping, table name resolution | `create_namespace`, `list_namespaces`, `drop_namespace`, table ops with `namespace` |
For simple use cases where you have a relatively flat set of tables, you can ignore namespaces and just use table paths directly.
As your application needs evolve and your tables grow in number and complexity, you may move from table-centric thinking
to catalog-centric thinking. Check out the [Namespaces and the Catalog Model](/namespaces) guide to learn more.
# Branches
Source: https://docs.lancedb.com/tables/branching
Fork isolated, writable lines of table history in LanceDB. Run experiments, backfills, and index rebuilds without disturbing production reads on main.
A branch is an isolated, writable line of history forked from `main` (or from any
other branch). Anything you do on a branch (e.g., adding rows, changing the schema,
building an index) stays on that branch, so `main` keeps serving production
reads exactly as before. Branches are a natural fit when you want to:
* Experiment with a new index, schema change, or reprocessing step without
affecting live queries on `main`.
* Run a backfill or migration you'd like to validate before applying it to
`main`.
* Hand a collaborator a frozen point-in-time fork while you keep writing to `main`.
## How branches relate to versions and tags
Every LanceDB table already tracks a linear history of [versions](/tables/versioning),
and you can [tag](/tables/versioning#tag-based-versioning) a version or `checkout`
one to read it. Branches add the missing piece: a *separate, writable* line of
history. Where a tag is a read-only label and `checkout` is a read-only view, a
branch forks from a point in history and then evolves on its own. Creating or
checking out a branch hands you a new table handle whose reads and writes are
scoped to that branch. The [comparison table](#branches-vs-tags-vs-checkout) at
the end of this page lays out when to reach for each.
Branches are supported on local and namespace-backed tables in LanceDB OSS, as
well as on LanceDB Enterprise (remote) tables.
## Connect to a table
The branch API is identical no matter how you connect — only the connection
itself differs between OSS and Enterprise. Establish a connection (`db`) and open
a `table` (see [Create a table](/tables/create)), then use the same branch calls
in every example that follows.
### LanceDB OSS
Point LanceDB at a local directory (or an object-storage URI) to use it as an
embedded library.
### LanceDB Enterprise
Enterprise
Branching on LanceDB Enterprise works the same way as on OSS, but you connect to your
Enterprise deployment with a `db://` URI, an API key, and your
region. Once you have a connection, open a table and use the same branch calls in
every example that follows.
## Work with branches
The lifecycle of a branch is short and predictable: fork it, write to it, reopen
it whenever you need it, and delete it once you're done. The examples below use a
small `quotes` table with three rows on `main`.
### Create a branch
Forking from `main` returns a table handle scoped to the new branch. `main` is
the reserved default source, so `create` needs only a name; to fork from
somewhere else, pass a branch name, a specific version, or both.
### Write to a branch
Writes go through the branch handle and stay there — the `main` handle keeps
reporting its original row count. Listing branches returns a mapping of each
branch name to its metadata, including the version it was forked from.
### Reopen a branch
A branch outlives the handle that created it. Reopen it later by name — either
from an existing table handle or straight from the connection when you open the
table. Both routes give you a writable handle tracking the branch's latest state.
### Delete a branch
Deleting a branch removes it and its branch-local history; `main` is untouched.
Before deleting a branch, make sure you've retained any results you need — see
[Apply branch-tested changes to `main`](#apply-branch-tested-changes-to-main)
below.
## Apply branch-tested changes to `main`
A branch has its own writable history. Outside of the [diff and merge
APIs](#compare-and-merge-a-branch-into-main) — which are available on
LanceDB Enterprise and promote added columns only — LanceDB does not reconcile
one branch's history with another or detect conflicts between them. To carry
other accepted work forward, rerun the validated operation against `main` or
explicitly write selected results to it.
How you apply a validated change depends on the type of work:
* **Added columns (Enterprise):** review the branch with `diff`, then promote
the new columns onto `main` with `merge`. See
[Compare and merge a branch into `main`](#compare-and-merge-a-branch-into-main).
* **Backfill or transformation:** rerun the validated job against `main`.
* **Schema change:** apply the same reviewed schema operation to `main`.
* **Index change:** build the index on `main` using the configuration validated
on the branch.
* **Selected row results:** upsert those rows into `main` using a stable unique
key.
### Upsert selected branch rows into `main`
If the result you want to retain is a set of inserted or updated rows, use
[`merge_insert`](/tables/update#merge-incoming-rows-by-key) to write them to
`main`. Despite its name, `merge_insert` is a row-ingestion operation: it
matches incoming rows by key and does not merge branch histories.
Read the rows you want from the reviewed branch, then upsert them into `main` on
your key column — `id` in this example. Here, `candidate` is the branch handle:
This operation transfers only the inserted or updated rows in its input. It does
not transfer branch history, schema changes, indexes, or branch-local deletions,
and it does not determine which rows changed after the branch was created.
Reapplying an identical payload is idempotent on the key, but the operation is
not conflict-aware. If `main` has diverged, incoming branch rows can overwrite
newer values with the same key. Read and upsert the whole branch only when that
overwrite is intentional; otherwise, filter the branch read to the rows you
intend to apply.
## Compare and merge a branch into `main`
Enterprise
On LanceDB Enterprise, branches expose two review-and-land calls that let you
inspect what a branch has changed relative to `main` and then promote its new
columns onto `main` in place — without reissuing the branch's writes.
`diff` and `merge` are available on Enterprise (remote) tables only. On local
tables both calls raise `NotSupported`. `merge` currently promotes added
columns; use the [upsert](#upsert-selected-branch-rows-into-main) or rerun
patterns above for row and index changes.
### Diff a branch
`diff` reads the branch and `main`, and returns a summary of what has changed:
which columns were added, removed, or altered; which indexes were added or
removed; row-count deltas; and — most importantly — a list of merge blockers
explaining why the branch cannot currently be merged, if any.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
diff = table.branches.diff("exp")
print(diff["addedColumns"]) # columns the branch introduced
print(diff["mergeable"]) # True when there are no blockers
print(diff["mergeBlockers"]) # list of {"code", "message"} entries
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
const diff = await table.branches.diff("exp");
console.log(diff.addedColumns); // columns the branch introduced
console.log(diff.mergeable); // true when there are no blockers
console.log(diff.mergeBlockers); // array of { code, message } entries
```
Common merge blocker codes include `BaseMoved` (the branch's parent no longer
matches `main`'s latest), `RowsChanged`, `ColumnRemoved`, `ColumnChanged`,
`NoMergeableChanges`, `NoColumnChanges`, `InputColumnDependency`, and
`ParentNotMain`. Newer server codes surface as `Unknown` so older clients
keep working.
### Merge a branch
`merge` promotes a branch's added columns onto `main`. It is a review-and-land
operation: the server re-evaluates the diff at request time, and either lands
the promotion or rejects it with the same blockers `diff` would report. A
rejected merge is not an exception — it resolves with `status="rejected"` so
you can inspect the blockers and decide what to do next.
Set `dry_run=True` (Python) or `dryRun: true` (TypeScript) to preview the
merge without landing it. The result includes a `preview.promoted_columns`
list showing which columns the merge would (or did) promote.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Preview first — this does not modify main.
preview = table.branches.merge("exp", dry_run=True)
print(preview["status"]) # "ready" | "rejected" | ...
print(preview["preview"]["promotedColumns"])
# Land the merge.
result = table.branches.merge("exp")
if result["status"] == "merged":
print("landed at main version", result["mainVersionAfter"])
elif result["status"] == "rejected":
for blocker in result["diff"]["mergeBlockers"]:
print(blocker["code"], blocker["message"])
```
```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
// Preview first — this does not modify main.
const preview = await table.branches.merge("exp", true);
console.log(preview.status); // "ready" | "rejected" | ...
console.log(preview.preview.promotedColumns);
// Land the merge.
const result = await table.branches.merge("exp");
if (result.status === "merged") {
console.log("landed at main version", result.mainVersionAfter);
} else if (result.status === "rejected") {
for (const blocker of result.diff.mergeBlockers) {
console.log(blocker.code, blocker.message);
}
}
```
Possible `status` values are `ready` (returned by `dry_run` when the merge
would land), `merged` (a real merge that landed), `rejected` (server declined;
see `diff.mergeBlockers`), `notImplemented`, and `unknown` for forward
compatibility. Merge requests are not retried on rejection — the response
carries everything you need to decide next steps.
## Build indexes on a branch
One of the most useful things a branch buys you is a safe place to build and
validate an index without affecting what's in production on the `main` branch.
Fork a branch, create your vector (ANN) and full-text search (FTS)
indexes on it, and check recall and latency. Once you have selected a
configuration, build the corresponding index on `main` through your normal
deployment workflow. Because the indexes live on the branch, queries against
`main` never see a half-built index and are never slowed down by the branch's
build.
This pattern is especially valuable on Enterprise
deployments, where `main` is typically serving production traffic while you tune
an index configuration on the side.
Schema changes such as adding, altering, or dropping columns are branch-scoped in
the same way, so you can stage and review a larger reshaping of a table before
applying the same schema operations to `main`.
## Branches vs. tags vs. versions
Now that you're familiar with branches, you can see how they complement the other ways LanceDB
give you to work with table history. Choose these approaches based on whether you need to
*label*, *read*, or *write* data at a point in history:
| Feature | Writable? | Purpose |
| --------------------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------- |
| **[Branch](#work-with-branches)** | ✅ Yes | Write on top of a point in history without touching `main`. |
| **[Tag](/tables/versioning#tag-based-versioning)** | ❌ No | Attach a human-readable label to an existing version; protects it from cleanup. |
| **[`checkout(version)`](/tables/versioning#rollback-to-previous-versions)** | ❌ No | Read a historical version of `main` without forking. Read-only until you `restore`. |
For linear version history — creating versions, listing them, rolling back, and
tagging — see the [Versioning and Reproducibility](/tables/versioning) guide.
# Consistency
Source: https://docs.lancedb.com/tables/consistency
Learn about consistency settings and versioning in LanceDB tables.
You can set `read_consistency_interval` on the connection to control how often reads check for updates from other writers.
There are three possible settings for `read_consistency_interval`:
1. **Unset (default)**: no automatic cross-process refresh checks.
2. **Zero seconds**: check for updates on every read (strongest freshness).
3. **Non-zero interval**: check for updates after the interval elapses (eventual refresh).
The value you set depends on your application's consistency needs and performance requirements.
For example, a real-time dashboard might require strong consistency, while a batch analytics job might be
fine with eventual consistency. Stronger consistency is not free — the smaller the interval, the more
often each read pays the cost of refreshing against object storage, which raises per-read latency and cost.
This setting works for both local ([LanceTable](/tables-and-namespaces#understanding-tables)) and remote
tables. It only affects read operations —
write operations are always consistent.
**Consistency in Remote Tables**
For remote tables (`db://` connections), `read_consistency_interval` is also
respected by the client. The interval is sent to the server as a freshness bound on each read:
* **Unset (default)**: no freshness header is sent; reads use the server's cached view of the table.
* **Zero seconds**: every read asks the server for the latest committed version.
* **Non-zero interval**: reads accept data at least as fresh as `now - interval`.
In addition, after any write or after a `checkout_latest` / `restore` on a table handle, subsequent
reads on that same handle carry a freshness floor so you read your own writes without extra
configuration. The floor is the later of the configured interval and the moment of the last
write or refresh, and it is shared across handles to the same table on the same connection.
Each remote table handle also tracks the highest dataset version it has observed in a read
response and sends it back with every subsequent read, so successive reads on the same handle
never observe an older version even when a load balancer routes them to query nodes with
differently-cached views. `checkout_latest` resets this watermark.
Stronger consistency is not free — the smaller the interval, the more often each read pays the cost
of refreshing against storage, which raises per-read latency and cost.
In Enterprise deployments, the server-side default freshness is still
controlled by the cluster-level `weak_read_consistency_interval_seconds` parameter; the client setting
tightens that bound on a per-connection basis.
## Configure Consistency Parameters
To set strong consistency, set the interval to 0:
For eventual consistency, use a non-zero interval:
With the default unset interval, tables do not auto-refresh from other writers.
To manually check for updates, use `checkout_latest` / `checkoutLatest`:
For reproducible reads, you can also pin a table to a specific snapshot with `checkout(...)` or
a tag, restore a table to a prior version, then return to the live table with
`checkout_latest` / `checkoutLatest`. See
[Versioning](/tables/versioning/) for the full version and tag workflow.
## Handle bad vectors
This section is currently specific to the Python SDK.
In LanceDB Python, you can use the `on_bad_vectors` parameter to choose how
invalid vector values are handled. Invalid vectors are vectors that are not valid
because:
1. They are the wrong dimension
2. They contain NaN values
3. They are null but are on a non-nullable field
By default, LanceDB will raise an error if it encounters a bad vector. You can
also choose one of the following options:
* `drop`: Ignore rows with bad vectors
* `fill`: Replace bad values (NaNs) or missing values (too few dimensions) with
the fill value specified in the `fill_value` parameter. An input like
`[1.0, NaN, 3.0]` will be replaced with `[1.0, 0.0, 3.0]` if `fill_value=0.0`.
* `null`: Replace bad vectors with null (only works if the column is nullable).
A bad vector `[1.0, NaN, 3.0]` will be replaced with `null` if the column is
nullable. If the vector column is non-nullable, then bad vectors will cause an
error
# Ingesting Data
Source: https://docs.lancedb.com/tables/create
Learn about different methods to ingest data into tables in LanceDB, including from various data sources and empty tables.
In LanceDB, tables store records with a defined schema that specifies column names and types. Across the SDKs, you can create tables from row-oriented data and Apache Arrow data structures. The Python SDK additionally supports:
* PyArrow schemas for explicit schema control
* `LanceModel` for Pydantic-based validation
## Create a table with data
Initialize a LanceDB connection and create a table
Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.
### From list of objects
You can provide a list of objects to create a table. The Python and TypeScript SDKs
support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
In Python, pass a list or other batch-like object; a single bare `dict` or single
`LanceModel` is rejected.
### Handle existing tables
By default, `create_table` raises an error if a table with the same name already exists.
You can change this behavior with two parameters that resolve the conflict in different ways:
* **Idempotent open**: return the existing table without modifying it. Use when your
code may run more than once (notebooks, retries, init scripts) and you want to reuse
the table on subsequent runs. The provided data is ignored, but the schema is
validated against the existing table and a mismatch raises an error.
* **Overwrite**: drop the existing table and create a new one with the provided data.
Use this for test fixtures or when you intentionally want to replace prior contents.
This permanently discards the old table's data.
`exist_ok` / `existOk` does not append the provided data to an existing table. Use
[`table.add()`](/tables/update) for that. If you need to ensure a table exists *and*
contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).
### From a custom schema
You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.
An explicit schema is also where you control nullability. If later writes omit a
non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
omitted or written with null values. Without an explicit schema, Python infers list-like vector
values as fixed-size `float32` vector fields from the observed dimension.
For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
`fill_value`, and `"null"` writes nulls.
### From an Arrow Table
You can also create LanceDB tables directly from Arrow tables.
Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.
### From a Pandas DataFrame
Python Only
Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.
### From a Polars DataFrame
Python Only
LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
is on the way.
### From Pydantic Models
Python Only
When you create an empty table without data, you must specify the table schema.
LanceDB supports creating tables by specifying a PyArrow schema or a specialized
Pydantic model called `LanceModel`.
For example, the following Content model specifies a table with 5 columns:
`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
pass the class as the value of the `schema` parameter to `create_table`.
The `vector` column is a `Vector` type, which is a specialized Pydantic type that
can be configured with the vector dimensions. It is also important to note that
LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
(which itself derives from `pydantic.BaseModel`).
#### Nested schemas
Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:
This can be used as the type of a LanceDB table column:
This creates a struct column called "document" that has two subfields
called "content" and "source":
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
In [28]: tbl.schema
Out[28]:
id: string not null
vector: fixed_size_list[1536] not null
child 0, item: float
document: struct not null
child 0, content: string not null
child 1, source: string not null
```
#### Validators
Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
for a `created_at` field.
When you run this code it, should raise the `ValidationError`.
### Loading Large Datasets
When ingesting large datasets, use `table.add()` on an existing table rather than
passing all data to `create_table()`. The `add()` method auto-parallelizes large
writes, while `create_table(name, data)` does not.
For best performance with large datasets, create an empty table first and then call
`table.add()`. This enables automatic write parallelism for materialized data sources.
#### From files (Parquet, CSV, etc.)
Python Only
For file-based data, pass a `pyarrow.dataset.Dataset` to `table.add()`. This streams
data from disk without loading the entire dataset into memory.
`pa.dataset()` input is currently Python-only. TypeScript and Rust support for
file-based dataset ingestion is tracked in
[lancedb#3173](https://github.com/lancedb/lancedb/issues/3173).
#### From iterators (custom batch generation)
When you need custom batch logic — generating embeddings on the fly, transforming
rows from an external source, etc. — use an iterator of `RecordBatch` objects.
Use this pattern when:
* Your source data already arrives in Arrow batches, readers, datasets, or streams.
* Materializing the entire ingest as one giant in-memory list or array would be too expensive.
* You want to control chunk size explicitly during ingestion.
Python can also consume iterators of other supported types like Pandas DataFrames or Python lists.
#### Write parallelism
For materialized data (`pa.Table`, `pd.DataFrame`, `pa.dataset()`), LanceDB
automatically parallelizes large writes — no configuration needed. Auto-parallelism
targets approximately 1M rows or 2GB per write partition.
For streaming sources (iterators, `RecordBatchReader`), LanceDB cannot determine
total size upfront. A `parallelism` parameter to control this manually is planned
but not yet exposed in Python or TypeScript
([tracking issue](https://github.com/lancedb/lancedb/issues/3173)).
#### Tracking ingestion progress
TypeScript Only
For long-running writes, pass a `progress` callback to `table.add()` to surface
per-batch progress in your UI, logs, or metrics pipeline. The callback fires
once per batch written and once more with `done: true` when the write completes.
Each invocation receives a `WriteProgress` object:
| Field | Description |
| :--------------- | :--------------------------------------------------------------------------------------- |
| `outputRows` | Rows written so far. |
| `outputBytes` | Bytes written so far. |
| `totalRows` | Expected total rows when the input source reports one. Always set on the final callback. |
| `elapsedSeconds` | Wall-clock seconds since the write started. |
| `activeTasks` | Parallel write tasks currently in flight. |
| `totalTasks` | Total parallel write tasks (the write parallelism). |
| `done` | `true` only for the final callback. |
A few things to know before you wire this up:
* Back-pressures the writer: callback invocations are serialized and run inline with each batch, so a slow callback will slow the write rather than drop updates. Every batch update is delivered, and the final `done: true` callback always fires (even on error or cancellation). Keep the callback cheap — offload heavy work to a queue you drain elsewhere.
* Errors swallowed: anything your callback throws is logged with `console.warn` and won't abort the write, so keep the callback side-effect-only and don't rely on it for control flow.
* Row totals: `totalRows` is only populated when the input source can report it up front (for example, a materialized `arrow.Table`). For streaming sources it stays `undefined` until the final callback, where it falls back to the actual rows written.
## Create empty table
You can create an empty table for scenarios where you want to add data to the table later.
An example would be when you want to collect data from a stream/external file and then add it to a table in
batches.
An empty table can be initialized via an Arrow schema.
Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
that has been extended to support LanceDB specific types like `Vector`.
Once the empty table has been created, you can append to it or modify its contents,
as explained in the [updating and modifying tables](/tables/update) section.
## Open an existing table
You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
If you forget the name of your table, you can always get a listing of all table names.
## Drop a table
Use the `drop_table()` method on the database to remove a table.
This permanently removes the table and is not recoverable, unlike deleting rows.
By default, if the table does not exist an exception is raised. To suppress this,
you can pass in `ignore_missing=True`.
# Basic Table Operations
Source: https://docs.lancedb.com/tables/index
Create tables, search vectors, and append data in LanceDB.
Now that you've completed the [LanceDB quickstart](/quickstart), you're ready to
explore some more table operations you'll typically need when working with LanceDB.
* **Ingest data into tables** from JSON data (and in Python, Pandas or Polars DataFrames)
* **Create empty tables** by defining explicit Arrow schemas
* **Vector similarity search** with filtering and projections
* **Filtered queries** that can operate on nested structs
* **Query Lance tables in DuckDB** via the Lance extension for SQL analytics (including joins)
This page uses **synchronous** Python snippets. If your Python app uses `asyncio`,
the same flow works with `connect_async(...)` and `await`-based table/query calls.
Use the example below as a template, and see [Quickstart](/quickstart#python-sync-and-async-apis)
for example snippets on both sync and async Python usage.
## Dataset
We'll work with this small dataset based on characters from the legends of Camelot. Note that
the `vector` column holds 4-dimensional embeddings, and the `stats` column is a nested struct
with several integer fields, indicating each character's attributes.
```json camelot.json icon="brackets-curly" expandable=true theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
[
{
"id": 1,
"name": "King Arthur",
"role": "King of Camelot",
"description": "The legendary ruler of Camelot, wielder of Excalibur, and leader of the Knights of the Round Table.",
"vector": [0.72, -0.28, 0.60, 0.86],
"stats": { "strength": 2, "courage": 5, "magic": 1, "wisdom": 4 }
},
{
"id": 2,
"name": "Merlin",
"role": "Wizard and Advisor",
"description": "A powerful wizard and prophet who mentors Arthur and shapes the destiny of Camelot through magic and foresight.",
"vector": [0.05, 0.88, 0.62, 0.85],
"stats": { "strength": 2, "courage": 4, "magic": 5, "wisdom": 5 }
},
{
"id": 3,
"name": "Queen Guinevere",
"role": "Queen of Camelot",
"description": "Arthur's queen, admired for her grace and diplomacy, whose romances and loyalties influence Camelot's fate.",
"vector": [0.22, -0.22, 0.42, 0.82],
"stats": { "strength": 1, "courage": 3, "magic": 1, "wisdom": 4 }
},
{
"id": 4,
"name": "Sir Lancelot",
"role": "Knight of the Round Table",
"description": "Arthur's most skilled knight, famed for unmatched combat prowess and his tragic love for Queen Guinevere.",
"vector": [0.86, -0.35, 0.38, 0.55],
"stats": { "strength": 5, "courage": 5, "magic": 1, "wisdom": 3 }
},
{
"id": 5,
"name": "Sir Gawain",
"role": "Knight of the Round Table",
"description": "A noble and honorable knight known for his courtesy and his encounter with the Green Knight.",
"vector": [0.82, -0.32, 0.52, 0.60],
"stats": { "strength": 4, "courage": 5, "magic": 1, "wisdom": 4 }
},
{
"id": 6,
"name": "Sir Galahad",
"role": "Knight of the Round Table",
"description": "The purest and most virtuous knight, chosen to achieve the Holy Grail due to his unwavering spiritual purity.",
"vector": [0.80, -0.20, 0.70, 0.78],
"stats": { "strength": 4, "courage": 5, "magic": 2, "wisdom": 5 }
},
{
"id": 7,
"name": "Sir Percival",
"role": "Knight of the Round Table",
"description": "A loyal and innocent knight whose bravery and sincerity make him one of the key seekers of the Holy Grail.",
"vector": [0.78, -0.36, 0.48, 0.52],
"stats": { "strength": 4, "courage": 4, "magic": 1, "wisdom": 3 }
},
{
"id": 8,
"name": "Mordred",
"role": "Traitor Knight",
"description": "Arthur's treacherous son or nephew who ultimately rebels against him, leading to Camelot's downfall.",
"vector": [0.68, -0.30, -0.65, 0.20],
"stats": { "strength": 4, "courage": 2, "magic": 1, "wisdom": 2 }
}
]
```
The `vector` arrays here are synthetic and for demonstration purposes only. In your real-world
applications, you'd generate these vectors from the raw text fields using a suitable embedding model.
## Connect to a database
### Option 1: Direct table access
We start by connecting to a LanceDB database path. The example below uses a local path in LanceDB OSS.
You can also connect LanceDB OSS directly to object storage. For credentials, endpoints, and provider-specific options, see
[Configuring storage](/storage/configuration).
### Option 2: Remote tables
If you're using LanceDB [Enterprise](/enterprise), you can connect using a `db://` URI,
along with any necessary credentials. Simply replace the local path with a remote `uri`
that points to where your data is stored, and you're ready to go.
* When you connect to a remote URI (Enterprise), `open_table(...)` returns a *remote* table.
Remote tables support core operations (ingest, search, update, delete), but some convenience
methods for bulk data export are not available.
* In the Python SDK, `table.to_arrow()` and `table.to_pandas()` are not implemented for remote tables.
To retrieve data, use search queries instead: `table.search(query).limit(n).to_arrow()`.
## Create a table and ingest data
### From JSON
LanceDB stores records in Lance tables. Each row is a record and each column
holds a field or related metadata. The simplest way to start is to obtain the source
data as a list of JSON records that includes a vector column and any metadata
fields you care about.
Load the data from the JSON file:
You can now create a LanceDB table from the loaded data. By default, creating a table with a name
that already exists raises an error. Use `mode="overwrite"` only when you intentionally want to
replace the existing table and its data, or use `exist_ok` / `existOk` when repeatable setup should
reuse the existing table instead of writing the supplied rows again.
If you want to avoid overwriting an existing table, omit the overwrite mode. For append-only
ingestion into a table that already exists, open the table and call `add(...)` instead of
`create_table(...)`. For repeatable setup with `exist_ok` / `existOk`, see
[Handle existing tables](/tables/create#handle-existing-tables).
For more ingestion patterns, including PyArrow tables, Python `pyarrow.dataset.Dataset` inputs,
empty tables, and Python `LanceModel` schemas with nested fields, see
[Ingesting data](/tables/create/).
### From Pandas DataFrames
Python Only
You can create LanceDB tables directly from [Pandas](https://pandas.pydata.org/) DataFrames. Simply
obtain the source data as a Pandas DataFrame, then create the table
and directly ingest to it.
### From Polars DataFrames
Python Only
You can also create LanceDB tables directly from [Polars](https://www.pola.rs/) DataFrames. Simply
obtain the source data as a Polars DataFrame, then create the table
and directly ingest to it.
### From an Arrow schema
If you want to create an *empty* table without any data -- say you want to
define the schema first and then incrementally add data later -- you can
do so by defining an Arrow schema explicitly.
Once the empty table is defined, LanceDB is ready to accept new data via
the `add` method, as shown in the next section.
LanceDB tables are type-aware, leveraging Apache Arrow under the hood.
You can display a given table's schema using the `schema` property or
method. For example, in Python, running `print(table.schema)` would show
something like the following:
```txt expandable=true theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
id: int64
name: string
role: string
description: string
vector: fixed_size_list[4]
child 0, item: float
stats: struct
child 0, courage: int64
child 1, magic: int64
child 2, strength: int64
child 3, wisdom: int64
```
## Append data to a table
LanceDB tables are mutable, and you can append new records to existing tables.
If you're starting with a fresh session, connect to the database and open the
existing table named `camelot`.
Prepare the new records to add. Here, we add two new magical characters
via the `add` method. For the Rust snippet, you can find the helper functions in the
[code](https://github.com/lancedb/docs/blob/main/tests/rs/basic_usage.rs).
We now have two new records in the table. Let's begin to query our data!
## Vector search
It's straightforward to run vector similarity search in LanceDB. Let's answer
some questions about the data using vector search with projections (returning only
the desired columns).
> Q1: *Who are the characters similar to "wizard"?*
| name | role | description |
| -------------------- | ------------------------- | ------------------------------- |
| Merlin | Wizard and Advisor | A powerful wizard and prophet |
| The Lady of the Lake | Mystical Guardian | A mysterious supernatural figu… |
| Morgan le Fay | Sorceress | A powerful enchantress, Arthur… |
| Queen Guinevere | Queen of Camelot | Arthur's queen, admired for he… |
| Sir Galahad | Knight of the Round Table | The purest and most virtuous k… |
We have Merlin, The Lady of the Lake, and Morgan le Fay in the top results, who
all have magical abilities.
Next, let's try to answer a different question that involves vector search while
filtering on a nested struct field. Filtering is done using the `where` method,
into which you can pass SQL-like expressions.
> Q2: *Who are the characters similar to "wizard" with high magic stats?*
| name | role | description |
| -------------------- | ------------------ | ------------------------------- |
| Merlin | Wizard and Advisor | A powerful wizard and prophet |
| The Lady of the Lake | Mystical Guardian | A mysterious supernatural figu… |
| Morgan le Fay | Sorceress | A powerful enchantress, Arthur… |
Only three characters have magical abilities greater than 3. Merlin is
clearly the most magical of them all!
## Filtered search
You can also run traditional analytics-style search queries that do not
involve vectors. For example, let's find the strongest characters in
the dataset. In the query below, we leave the `search` method empty to indicate
that we don't want to use any vector for similarity search (in TypeScript/Rust,
use `query()` instead), and use the `where` method to filter on the `strength` field.
> Q3: *Who are the strongest characters?*
| name | role | description |
| ------------ | ------------------------- | ------------------------------- |
| Sir Galahad | Knight of the Round Table | The purest and most virtuous k… |
| Sir Gawain | Knight of the Round Table | A noble and honorable knight k… |
| Sir Percival | Knight of the Round Table | A loyal and innocent knight wh… |
| Sir Lancelot | Knight of the Round Table | Arthur's most skilled knight, … |
| Mordred | Traitor Knight | Arthur's treacherous son or ne… |
Clearly, the strongest characters are all Knights of the Round Table!
Need SQL analytics like filters, aggregations, or joins on Lance tables? Use the DuckDB
Lance extension to query Lance tables directly with SQL. See the
[DuckDB integration guide](/integrations/data/duckdb).
## Add column
We can also add new columns to an existing LanceDB table using the `add_columns` method.
For this example, let's add a new float column named `power` that shows the average
of each character's strength, courage, magic, and wisdom stats.
The example above sums up the individual stats and divides by 4 to compute the average.
The resulting average total stats is cast to an Arrow float type under the hood for the
Lance table.
We can display the results of this column in descending order of power.
> Q4: *Who are the most powerful characters?*
Note that LanceDB's `where` only filters rows, but doesn't sort them by applying an `ORDER BY`
clause that you may be used to when working with SQL databases.
You can also sort the results after converting them to a Polars DataFrame.
In TypeScript/Rust, you can sort the
returned array in application code.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Sort Polars DataFrame by power in descending order
print(r1.sort("power", descending=True).limit(5))
```
| name | role | description | power |
| -------------------- | ------------------------- | ------------------------------- | ----- |
| Merlin | Wizard and Advisor | A powerful wizard and prophet … | 4.0 |
| Sir Galahad | Knight of the Round Table | The purest and most virtuous k… | 4.0 |
| The Lady of the Lake | Mystical Guardian | A mysterious supernatural figu… | 3.75 |
| Sir Lancelot | Knight of the Round Table | Arthur's most skilled knight, … | 3.5 |
| Sir Gawain | Knight of the Round Table | A noble and honorable knight k… | 3.5 |
Merlin and Sir Galahad are the most powerful characters when considering the average of
all their abilities! Sir Lancelot and the Lady of the Lake follow closely behind.
For column renames, type changes, nullability changes, and grouping multiple schema changes into
one operation, see [Schema and data evolution](/tables/schema/).
## Delete data
You can delete rows from a LanceDB table using the `delete` method with
a filtering expression.
Say we want to remove Mordred, the traitor knight, from our table.
This will delete the row(s) where the `role` value matches "Traitor Knight".
You can verify that the row has been deleted by running a search query again,
and confirming that Mordred no longer appears in the results.
## Drop column
If you want to remove or delete a column from an existing LanceDB table, you can use
the `drop_columns` method.
This will remove the `power` column we added earlier from the table schema.
## Drop table
If you want to delete an entire table from the database, you can use the
`drop_table` method.
This will delete the `camelot` table from the connected LanceDB database.
See the full code for these examples (including helper functions) in the
`basic_usage` file for the appropriate client language in the
[docs repo](https://github.com/lancedb/docs/tree/main/tests).
## What about vector indexes?
LanceDB supports vector indexes to speed up similarity search on large datasets.
For datasets up to a few hundred thousand vectors, LanceDB's highly efficient kNN
(brute-force) retrieval of nearest neighbors is often sufficient. As your dataset
grows larger, you can create vector indexes on your vector columns to accelerate
search. See the [indexing](/indexing/) documentation for details on how to create and use
vector indexes in LanceDB.
## What's next?
Now that you've learned the basics of creating tables, adding data, running
vector search, and modifying table schemas, you're ready to explore more
advanced features of LanceDB. Below are some suggested next pages.
Learn the different approaches to creating and ingesting data into LanceDB tables from various sources.
Learn how to update and modify existing LanceDB tables and their data.
Understand how to evolve your table schemas and data over time with LanceDB.
Explore LanceDB's built-in table versioning and time travel capabilities.
# Multimodal Data (Blobs)
Source: https://docs.lancedb.com/tables/multimodal
Learn how to store and query multimodal data (images, audio, video) directly in LanceDB using binary columns.
LanceDB handles multimodal data—images, audio, video, and PDF files—natively by storing the raw bytes in a binary column alongside your vectors and metadata. This approach simplifies your data infrastructure by keeping the raw assets and their embeddings in the same database, eliminating the need for separate object storage for many use cases.
This guide demonstrates how to ingest, store, and retrieve image data using standard binary columns, and also introduces the **Lance Blob API** for optimized handling of larger multimodal files.
## Store binary data
To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust).
### 1. Setup and imports
First, import the necessary libraries for LanceDB and Arrow in your SDK.
### 2. Prepare data
For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object.
### 3. Define the schema
When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list.
### 4. Ingest data
Now, create the table using the data and the defined schema.
## Retrieve and use blobs
When you search your LanceDB table, you can retrieve the binary column just like any other metadata.
### Convert bytes back to objects
Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer).
## Large Blobs (Blob API)
For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront.
### 1. Define a blob schema
To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata.
### 2. Ingest large blobs
You can then ingest data normally, and Lance will handle the optimized storage.
For more advanced usage, including random access and file-like reading of blobs, see the
Lance format's [blob API documentation](https://lance.org/guide/blob/).
### 3. Convert blob tables to pandas
When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`.
`blob_mode` accepts:
* `"lazy"` (default): returns blob columns as lazy `BlobFile` objects without eagerly materializing their payloads. Use this when you want to stream blob bytes on demand or only inspect a subset of rows. Namespace-backed local tables also use the Lance native blob-aware pandas conversion for lazy blobs; in-memory datasets fall back to the standard PyArrow `to_pandas()` path.
* `"bytes"`: eagerly materializes each blob as `bytes`. Use this when you need the raw payload in the DataFrame, for example to decode an image or audio clip in-process.
* `"descriptions"`: returns blob descriptors (offsets, sizes, and positions) instead of the data itself. Use this when you want to plan I/O without paying the cost of loading every blob.
`"bytes"` and `"descriptions"` require a filesystem-backed Lance dataset and are not supported on in-memory tables.
Extra keyword arguments are forwarded to the underlying PyArrow / Lance pandas conversion, so you can also pass options like `split_blocks` or `self_destruct`:
Query builders also accept `blob_mode` on their `to_pandas()` method:
* Plain scans support `"lazy"`, `"bytes"`, and `"descriptions"` with filters, projections, aliases, `limit`, and `offset`.
* Vector, FTS, hybrid, and ordered queries can't materialize blob columns through `to_pandas()`; omit blob columns from the projection for those query shapes.
* This works on both sync and async query builders. Extra PyArrow kwargs like `split_blocks` and `self_destruct` are still forwarded.
## Other modalities
The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data:
* **Audio:** Read `.wav` or `.mp3` files as bytes.
* **Video:** Store video transitions or full clips using the Blob API.
* **PDFs/Documents:** Store the raw file content for document search.
# Schema and Data Evolution
Source: https://docs.lancedb.com/tables/schema
Learn how to manage table schemas in LanceDB, including adding, altering, and dropping columns.
Schema evolution enables non-breaking modifications to a database table's structure — such as adding columns, altering data types, or dropping fields — to adapt to evolving data requirements without service interruptions.
LanceDB supports ACID-compliant schema evolution through granular operations (add/alter/drop columns), allowing you to:
* Iterate Safely: Modify schemas in production with versioned datasets and backward compatibility
* Scale Seamlessly: Handle ML model iterations, regulatory changes, or feature additions
* Optimize Continuously: Remove unused fields or enforce new constraints without downtime
## Schema evolution operations
LanceDB supports four primary schema evolution operations:
1. **Adding new columns**: Extend your table with additional attributes
2. **Altering existing columns**: Change column names, data types, or nullability
3. **Updating field metadata**: Attach or change per-column Arrow metadata
4. **Dropping columns**: Remove unnecessary columns from your schema
Schema evolution operations are applied immediately but do not typically require rewriting all data. However, data type changes may involve more substantial operations.
Each schema evolution operation commits a new table version and returns status metadata such as
the committed `version`. Run these operations from a mutable table handle; if you checked out an
older version for reads, call `checkout_latest` / `checkoutLatest` before modifying the schema.
## Add new columns
You can add new columns to a table with the [`add_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.add_columns)
method in Python, [`addColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#addcolumns) in TypeScript/JavaScript, or `add_columns` in Rust.
New columns are populated based on SQL expressions you provide.
### Set up the example table
First, let's create a sample table with product data to demonstrate schema evolution:
### Add derived columns
You can add new columns that are derived from existing data using SQL expressions.
For feature engineering on large existing tables, group related derived features into
one `add_columns` operation instead of running many separate writes. This creates one
new table version for the schema change and computes the new columns from the existing
rows, which avoids growing the table's version history with many small updates.
The same call can add multiple derived columns at once. For example, if you are
building several lightweight features from existing product fields, pass all of the
new column expressions together:
LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or
PyArrow `RecordBatch` iterators for populating new columns. New column values must be
defined with SQL expressions, or added as NULL columns from an Arrow field or schema.
If your transformation cannot be expressed in SQL, compute the values outside
`add_columns` before writing them back through another workflow.
### Add columns with default values
Add boolean columns with default values for status tracking:
### Add nullable columns
Add timestamp columns that can contain NULL values:
When adding columns that should contain NULL values, be sure to cast the NULL to the appropriate type, e.g., `cast(NULL as timestamp)`.
## Alter existing columns
You can alter columns using the [`alter_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.alter_columns)
method in Python, [`alterColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#altercolumns) in TypeScript/JavaScript, or `alter_columns` in Rust. This allows you to:
* Rename a column
* Change a column's data type
* Modify nullability (whether a column can contain NULL values)
### Set up the example table
Create a table with a custom schema to demonstrate column alterations:
### Rename columns
Change column names to better reflect their purpose:
### Change data types
Convert column data types for better performance or compatibility:
### Make columns nullable
You can alter columns to contain NULL values:
Changing a column to nullable affects future writes and merges too: missing values are accepted
only when the target column is nullable.
### Multiple changes at once
Apply several alterations in a single operation:
### Expression-based type changes
For transformations that are not simple casts (for example, converting `"$100"` to an integer), use a SQL-expression column add, then drop and rename:
### Alter embedding types and dimensions
It's quite common to need to change an embedding column's schema, in case a new model becomes available with a different embedding dimension.
* In Python, the example shows an in-place type update when the cast is compatible.
* In TypeScript and Rust, the example shows a dimension change (`384 -> 1024`), which cannot be cast in-place.
For dimension changes, use this 3-step pattern: add a new column with the target type, drop the old column, then rename the new column to the original name.
**`FixedSizeList` Dimension Changes in TypeScript and Rust**
`alterColumns` / `alter_columns` can cast between compatible types, but changing `FixedSizeList` dimensions (for example `384 -> 1024`) is not a compatible cast.
For such cases, use `addColumns` / `add_columns` (with `arrow_cast`), then `dropColumns` / `drop_columns`, then rename the replacement column.
Changing data types requires rewriting the column data and may be resource-intensive for large tables. Renaming columns or changing nullability is more efficient as it only updates metadata.
## Update field metadata
Each column in a LanceDB table can carry a small key/value map of Arrow field metadata — useful
for annotating columns with units, provenance, PII flags, embedding model versions, or any other
schema-level context your application needs.
Use [`update_field_metadata`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.update_field_metadata)
in Python, [`updateFieldMetadata`](https://lancedb.github.io/lancedb/js/classes/Table/#updatefieldmetadata)
in TypeScript/JavaScript, or `update_field_metadata` in Rust to add, change, or remove these
key/value pairs without rewriting the column data. Each call commits a new table version and returns
the new `version`.
Each update targets one field by **dot-path**: top-level columns are addressed by name (for
example `"embedding"`), and nested fields by their full path (for example `"address.zip"`). By
default, the keys you pass are **merged** into the field's existing metadata — keys you do not
mention are preserved, and passing `None` (Python) or `null` (TypeScript) deletes a key. Set
`replace: true` to swap the field's entire metadata map instead of merging.
To overwrite a field's metadata entirely instead of merging, set `replace` to `true`:
You can pass multiple updates in a single call to change metadata on several fields at once —
each call commits a single new table version.
## Drop columns
You can remove columns using the [`drop_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.drop_columns)
method in Python, [`dropColumns`](https://lancedb.github.io/lancedb/js/classes/Table/#dropcolumns) in TypeScript/JavaScript, or `drop_columns` in Rust.
### Set Up the example table
Create a table with temporary columns that we'll remove:
### Drop single columns
Remove individual columns that are no longer needed:
### Drop multiple columns
Remove several columns at once for efficiency:
Dropping columns cannot be undone. Make sure you have backups or are certain before removing columns.
# Updating and Modifying Table Data
Source: https://docs.lancedb.com/tables/update
Learn how to update, merge, and delete rows in a LanceDB table.
Updating or modifying data involves changing rows in an existing table.
LanceDB provides two families of write operations that can modify data in a table:
* `update(...)`: mutate existing rows that match a SQL filter.
* `merge_insert(...)`: compare incoming rows to existing rows by key, then choose what to do for each case.
The `update` method is simpler to use when you already know which rows you want to modify and you do not need to compare against an incoming dataset. The `merge_insert` method is more powerful when you have a new dataset that you want to merge into an existing table, and you want LanceDB to handle the logic of comparing against existing rows by key.
Let's look at an example that demonstrates these operations in practice.
## Connect to LanceDB
Connect to your local LanceDB instance:
Or, connect to LanceDB Enterprise:
In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data.
## Create the example table
We'll start by creating a simple table with `id`, `name`, and `login_count` columns. All examples below use the same table.
Expected table contents:
| id | name | login\_count |
| -- | ----- | ------------ |
| 1 | Alice | 10 |
| 2 | Bob | 20 |
The example above shows a PyArrow schema. You can just as well create the table using other
table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see the [ingestion](/tables/create/) guide for more details.
## Choose a write method
| Family | Method | Use this when... |
| -------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a SQL filter. |
| `merge_insert` | `.when_matched_update_all()` | You have incoming rows and want to update keys that already exist in the table. |
| `merge_insert` | `.when_not_matched_insert_all()` | You have incoming rows and want to insert keys that do not exist yet. |
| `merge_insert` | `.when_matched_update_all()` + `.when_not_matched_insert_all()` | You want both behaviors together (often called **upsert**: update existing keys **and** insert missing keys in the same operation). |
| `merge_insert` | `.when_not_matched_by_source_delete(...)` | You want to remove target rows that are missing from the incoming source set. |
Write operations return status metadata. For example, `update` reports updated row count and
committed `version`, `delete` reports deleted row count and `version`, and `merge_insert` reports
inserted, updated, deleted, retry-attempt, and `version` fields. Writes require a mutable table
handle; after checking out an older version for reads, call `checkout_latest` / `checkoutLatest`
before modifying data.
The committed `version` advances even for writes that affect zero rows, such as a delete predicate
that matches nothing.
## Update rows
Use `update` when you already know which target rows to modify and you do not need to compare against an incoming dataset.
Expected table contents:
| id | name | login\_count |
| -- | ----- | ------------ |
| 1 | Alice | 10 |
| 2 | Bobby | 20 |
Updating nested columns is not yet supported.
## Update rows with SQL expressions
Use `values_sql` when you want to use SQL-like expressions to update rows. This is useful for operations like incrementing a counter, or setting a column value based on another column.
Expected table contents:
| id | name | login\_count |
| -- | ----- | ------------ |
| 1 | Alice | 10 |
| 2 | Bob | 21 |
See the [SQL queries](/search/sql/) page for more information on the supported SQL syntax.
When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards.
## Merge incoming rows by key
Merging is different from updating because it involves comparing incoming rows to existing rows by key, and then choosing what to do based on whether the key exists in the target table or not.
The `merge_insert(""..."")` method lets you do this.
In merge operations, rows are split into three groups:
* **Matched**: key exists in both source and target.
* **Not matched**: key exists only in source.
* **Not matched by source**: key exists only in target.
Conditional merge clauses can compare old and new values. Use the `target.` prefix for the
existing table row and `source.` for the incoming row, for example
`target.last_update < source.last_update`.
**Use scalar indexes to speed up merge insert**
The merge insert command performs a join between the input data and the target table `on` the key you provide. This requires scanning that entire column, which can be expensive for large tables. To speed up this operation, create a scalar index on the join column, which will allow LanceDB to find matches without scanning the whole table.
Read more about scalar indices in the [Scalar Index](/indexing/scalar-index/) guide.
If you see this HTTP 400 error from `merge_insert`: `Bad request: Merge insert cannot be performed because the number of unindexed rows exceeds the maximum of 10000`. Verify that the scalar index on the join column is up to date before retrying.
**Rust: build merge predicates with DataFusion expressions**
In the Rust SDK, you can pass a `datafusion_expr::Expr` directly instead of a SQL string by using
`when_matched_update_all_expr` and `when_not_matched_by_source_delete_expr` on `MergeInsertBuilder`.
This is useful when you are constructing predicates programmatically and want to avoid building a SQL string.
These methods are only supported on local tables. Calling them against a remote table returns a
`NotSupported` error — use the SQL string variants (`when_matched_update_all` /
`when_not_matched_by_source_delete`) for remote tables.
Like the create table and add APIs, the merge insert API will automatically compute embeddings based on the [embedding registry](/embedding/index#embedding-registry) if the table has an embedding definition in its schema.
During `merge_insert`, if the input data doesn't contain the source column (i.e., the original field used to generate embeddings, such as text for a text embedding model or `image_uri` for an image model), or if a vector value is already provided, LanceDB skips embedding generation for that row. Embeddings are only auto-generated when that source field is present in the incoming data, **and** the vector field is empty.
Primary keys in LanceDB are metadata used by operations such as `merge_insert`; they are not
enforced as uniqueness constraints on ordinary writes. Keep using `merge_insert` or your own
deduplication logic when you need key-based upsert semantics.
### Update matched rows only
This updates keys that already exist in the target table. Source rows with new keys are ignored.
Expected table contents:
| id | name | login\_count |
| -- | ----- | ------------ |
| 1 | Alice | 10 |
| 2 | Bobby | 21 |
### Insert unmatched rows only
This inserts only brand-new keys from the source. Existing keys are left unchanged.
Expected table contents:
| id | name | login\_count |
| -- | ------- | ------------ |
| 1 | Alice | 10 |
| 2 | Bob | 20 |
| 3 | Charlie | 5 |
### Update matched rows and insert unmatched rows
Use both `when_matched_update_all()` and `when_not_matched_insert_all()` when you want to update existing keys and insert missing keys in one operation.
This is a conventional **upsert**.
Expected table contents:
| id | name | login\_count |
| -- | ------- | ------------ |
| 1 | Alice | 10 |
| 2 | Bobby | 21 |
| 3 | Charlie | 5 |
### Delete target rows that are missing from source
Use `when_not_matched_by_source_delete()` when you want to remove any target row that does not appear in the incoming source data.
Expected table contents:
| id | name | login\_count |
| -- | ------- | ------------ |
| 2 | Bobby | 21 |
| 3 | Charlie | 5 |
In the example above, LanceDB matches rows by `id`. Rows with `id=2` and `id=3` exist in both the table and incoming data, so they are updated. Row `id=1` exists only in the target, so it is deleted.
### Use partial columns in merge updates
Merge updates do not require you to provide values for all columns. You can provide only a subset of columns in source rows. For matched rows, only the provided columns are updated.
Expected table contents:
| id | name | login\_count |
| -- | ------- | ------------ |
| 1 | Alice | 10 |
| 2 | Bobby | 20 |
| 3 | Charlie | null |
Note that in the example above, when `merge_insert` creates a new row, any missing columns are written as `null`. If a missing column is non-nullable in your schema, the insert will fail.
## Delete rows
Delete operations **soft delete** rows that match a given condition.
The underlying data is not immediately removed, but is marked
for deletion (in the [deletion files](https://lance.org/format/table/#deletion-files) at the Lance format level) and excluded from query results.
Expected table contents:
| id | name | login\_count |
| -- | ----- | ------------ |
| 1 | Alice | 10 |
| 2 | Bob | 20 |
**Deleting rows removes them from the index**
When rows are deleted, those rows are also excluded from the index segments, so indexed queries will not return them either. If ALL the rows are deleted (i.e., the table is emptied), ensure that you recreate the index after ingesting new data.
To permanently remove deleted rows, you can optimize the table, which will run compaction and cleans up the soft-deleted rows, which frees up storage space.
* In LanceDB OSS, compaction and cleanup are manual. Run `table.optimize()` regularly to free up disk space.
* In LanceDB Enterprise, files aren't cleaned up by default. You can configure automatic compaction and cleanup behavior at cluster setup time to suit your organization's retention policy.
By default, table cleanup removes data up to 7 days ago. If you need to reclaim space from deleted rows more aggressively, manually call `table.optimize()` use a shorter retention window as follows:
# Versioning and Reproducibility
Source: https://docs.lancedb.com/tables/versioning
Learn how to implement versioning and ensure reproducibility in LanceDB. Includes version control, data snapshots, and audit trails.
This page shows the core table-versioning APIs used in the code snippets for Python, TypeScript, and Rust.
Each operation below maps directly to methods shown in the examples.
## Basic Versioning Example
Let's create a table with sample data to demonstrate LanceDB's versioning capabilities:
### Set Up the Table
First, let's create a table with some sample data:
### Check Initial Version
After creating the table, let's check the initial version information:
## Modify Data
When you modify data through operations like update or delete, LanceDB automatically creates new versions.
### Update Existing Data
Let's update some existing records to see versioning in action:
### Add New Data
Now let's add more records to the table:
### Check Version Changes
Let's see how the versions have changed after our modifications:
## Rollback to Previous Versions
LanceDB supports fast rollbacks to any previous version without data duplication.
### View All Versions
First, let's see all the versions we've created:
### Restore a Version Snapshot
Now let's restore a captured version snapshot:
## Tag-Based Versioning
Numeric table versions like `v3` or `v17` are precise but hard to remember. Tags
let you attach human-readable labels (e.g., `"prod"`, `"baseline"`,
`"q3-evaluation"`) to specific versions and check those out by name. They are
conceptually similar to git tags, and unlike numeric versions, **tagged versions
are preserved when old versions are pruned** (see the cleanup note at the bottom
of this page).
The tags API supports the standard CRUD operations — create, list, update, delete —
plus checking out by tag name.
Deleting a tag only removes the label, not the version it points to. After
deletion, the underlying table version becomes eligible for cleanup again.
## Branches
Beyond linear history, LanceDB also supports **branches** — isolated, writable
lines of history forked from `main` (or a specific version). Whereas tags and
`checkout` give you read-only views of existing versions, a branch has its own
writable history, making it ideal for experiments, backfills, and migrations
that you want to keep separate from production reads on `main`.
Branches are covered in their own guide: see [Branches](/tables/branching).
## Delete Data From the Table
Let's demonstrate how deletions also create new versions:
### Go Back to Latest Version
First, let's return to the latest version:
### Delete Data
Now let's delete some data to see how it affects versioning:
### Version History and Operations
On a fresh table, the snippets in this guide produce this version sequence:
1. `v1`: create table (`create_table` / `createTable` / `create_table`)
2. `v2`: update rows (`update`)
3. `v3`: add rows (`add`)
4. `v4`: restore snapshot (`restore`) from `version_after_mod`/`versionAfterMod`
5. `v5`: delete rows (`delete`)
Read-only and checkout operations shown here (`list_versions`/`listVersions`, `version`, `checkout`, `checkout_latest`/`checkoutLatest`) do not create new versions.
The version metadata fields can differ by backend. Direct table-backed version listing exposes a
timestamp, while namespace-backed listing may expose fields such as `manifest_path`,
`manifest_size`, `e_tag`, and `timestamp_millis`. In deployments that use managed versioning,
prefer the table version APIs exposed by LanceDB Enterprise or the namespace service instead of
mixing in lower-level Lance file operations.
**System Operations**
System operations like `optimize()`, index updates, and table compaction also increment table version numbers.
In LanceDB OSS and Enterprise, `optimize()` can prune older versions based on its retention setting (`cleanup_older_than`, 7 days by default),
which is when old-version files are removed and disk space is reclaimed.
**Tagged versions are exempt from cleanup.** A version with a tag pointing at it is
retained regardless of age, and its files are not removed by `optimize()`. To make
a tagged version eligible for pruning, [delete the tag](#tag-based-versioning) first.
# Loading Data for Model Training
Source: https://docs.lancedb.com/training/index
Stream, shuffle, transform, and resume model training data from LanceDB.
LanceDB makes an excellent data backend for training machine learning models. A `Table` can be used directly as input
to a data loader, but this is typically limited. A `Permutation` gives you control over which rows are accessed and in
what order. For a more complete solution, LanceDB also provides a streaming data loader through `StreamingDataset`.
This PyTorch `IterableDataset` adapts the lower-level `Permutation` API and adds prefetching, elastic determinism,
resumability, and multithreaded transformations.
## Basic data loading
Most model training frameworks iterate through data in batches and feed this data into the model. This process is
often referred to as **data loading**. The simplest way to load data into a model is to iterate a LanceDB table in
a loop and feed the data into the model.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
db = lancedb.connect("file://some/db/path")
table = db.open_table("some_table")
for batch in table:
print(batch.to_pydict())
```
In practice, this is too simplistic for effective training. We may not want to load all the data, or we may want
to load the data in a different order, or we may need to apply some sort of processing to the data before training.
To achieve this, we can use the `StreamingDataset`.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
import torch
from lancedb.streaming import StreamingDataset
db = lancedb.connect("file://some/db/path")
table = db.open_table("some_table")
dataset = StreamingDataset(table, shuffle_seed=42)
dataloader = torch.utils.data.DataLoader(
dataset,
batch_size=128,
num_workers=0,
)
for batch in dataloader:
train_step(batch)
```
`StreamingDataset` yields plain Python dictionaries by default. PyTorch's default collation function combines those
samples into batches. You can also iterate the dataset directly when you do not need collation.
`StreamingDataset` is built on the permutation API and works with both local LanceDB tables when using OSS, and
remote tables accessed through LanceDB Enterprise. The underlying table data can live on local disk or object storage.
Use the streaming data loader when the training data does not fit in memory, when you need deterministic global batches
across cluster sizes, or when you want filtering and prefetching to happen before PyTorch requests individual samples.
Use `Permutation` directly when you need map-style random access instead. Only one iterator can be active on a
`StreamingDataset` instance at a time; create a separate instance for each concurrent consumer.
## Advanced data loading
The `StreamingDataset` wraps a LanceDB `Table` and, by default, adds prefetching and conversion from Arrow to Python.
It can also handle more advanced scenarios. To explain these, consider a model trained with stochastic gradient
descent (SGD) and distributed data parallelism (DDP). In this example, we need to load batches onto multiple GPUs
across multiple servers. After each batch is processed, the GPUs exchange weights and the next batch is loaded. We
will use the following PyTorch terms:
* **World size** - The number of GPUs that we are loading. For example, if we have 2 servers and each server has 4
GPUs, the world size is 8.
* **Rank** - The identifier of the process loading data for a GPU. It is an integer in the range `[0, world_size)`.
Each rank gets its own portion of the data.
* **Global batch size** - The number of rows processed across all GPUs in each step of the SGD algorithm. For
example, if we have 8 GPUs and a global batch size of 1024, we load 128 rows onto each GPU for each step.
* **Batch size** - The number of rows processed by one GPU in each step. In the preceding example, the batch size
is 128.
Other concepts, such as read batch size and `num_workers`, are introduced in the relevant sections below.
### Prefetching
PyTorch datasets were originally built around in-memory structures like a Pandas DataFrame. When they are iterated,
they yield a single sample at a time. This makes sense for a simple in-memory structure, but accessing data on object
storage one row at a time introduces too much per-call overhead. To avoid this, `StreamingDataset` fetches and
transforms data in batches. The `read_batch_size` parameter controls how many rows are fetched from each split per
storage request and defaults to `64`.
In addition to batching requests, the prefetching mechanism reads ahead in the background. While one batch is being
transformed and processed by the GPU, `StreamingDataset` reads subsequent batches. The `prefetch_batches` parameter
controls how many batches are kept in flight per split and defaults to `4`. A larger value can provide more buffering
against jittery workloads, but it also increases memory use and I/O concurrency.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
ds = StreamingDataset(
table,
shuffle_seed=42,
read_batch_size=256, # rows fetched per LanceDB call per split
prefetch_batches=8, # batches to keep in flight per split
)
```
### Transformation
Many model training workloads require a transformation step between loading the data and training the model. For
example, we may need to decode images, tokenize text, or normalize data. A transformation function can be provided
using the `transform` parameter. Transformations can be expensive, so `StreamingDataset` applies them with a
`ThreadPoolExecutor` whose worker count equals the number of available CPUs.
Transformations are applied to batches, not individual samples, to amortize per-batch overhead. A transformation
function receives a PyArrow `RecordBatch` and must return an iterable with exactly one output sample for every input
row. The sample format should match what your data loader expects. For example, PyTorch's default collation function
accepts several sample types, with a Python dictionary being one of the most common. When no transform is provided,
the default transform converts the Arrow record batch into Python dictionaries without further processing.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
def normalize(batch: pa.RecordBatch) -> list[dict]:
# This pure-Python loop holds the GIL and is shown for illustration only.
# In practice, prefer a library like torchvision or numpy that releases the
# GIL so the ThreadPoolExecutor can run transforms in parallel.
rows = batch.to_pylist()
for row in rows:
row["image"] = [v / 255.0 for v in row["image"]]
return rows
ds = StreamingDataset(table, shuffle_seed=42, transform=normalize)
```
#### DataLoader workers
The thread-based transformation model that `StreamingDataset` uses by default is only effective when the transform
function releases the GIL. This is true for many Python scientific libraries, including NumPy, PyArrow, and
TorchVision, but not for pure-Python transforms. PyTorch can launch multiple DataLoader worker processes per rank,
and `StreamingDataset` uses `get_worker_info()` to give each worker a non-overlapping group of splits.
Multiprocessing adds pickling and transfer overhead and increases memory use. Start with `num_workers=0`, which keeps
loading in the rank's main process, and add workers only after confirming an unavoidable GIL bottleneck. If you use
workers, `num_splits` must be divisible by `world_size * num_workers`. Use the `forkserver` or `spawn` multiprocessing
context because LanceDB uses internal threads.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
dataloader = torch.utils.data.DataLoader(
dataset,
batch_size=batch_size,
num_workers=2,
multiprocessing_context="forkserver",
persistent_workers=True,
)
```
By default, `StreamingDataset` serializes enough table state to reopen the table in each worker. For custom connection
setup, pass a picklable `connection_factory` callable that accepts a table name and returns an open table. This avoids
serializing connection credentials into worker state.
### Observability & performance
Optimizing data loader performance is tricky because it can be difficult to locate the bottleneck. What is often
blamed on I/O can be a CPU bottleneck in the transform stage, or vice versa. To help distinguish them,
`StreamingDataset` exposes pipeline counters. `raw_queue_depth` is the number of loaded rows waiting to be transformed,
while `prefetch_queue_depth` is the number of transformed rows ready to be consumed. `unscanned_rows` and
`consumed_rows` show how far the current iterator has progressed.
If the `prefetch_queue_depth` is consistently zero but the `raw_queue_depth` is not then you have a CPU
transformation bottleneck. You should investigate GIL bottlenecks or look for ways to optimize your transformation.
This can often be done by batching the compute work. If both `prefetch_queue_depth` and `raw_queue_depth` are
consistently zero while the consumer is waiting, I/O is the likely bottleneck. A larger read batch size or clumped
shuffling could help.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import threading, time
ds = StreamingDataset(table, shuffle_seed=42)
def log_pipeline_health():
while True:
print(
f"unscanned={ds.unscanned_rows} "
f"raw={ds.raw_queue_depth} "
f"cooked={ds.prefetch_queue_depth} "
f"consumed={ds.consumed_rows}"
)
time.sleep(1.0)
monitor = threading.Thread(target=log_pipeline_health, daemon=True)
monitor.start()
for sample in ds:
train_step(sample)
print(
f"bytes loaded: {ds.bytes_loaded} "
f"fetch time: {ds.fetch_time:.2f}s "
f"transform time: {ds.transform_time:.2f}s"
)
```
`bytes_loaded` measures raw Arrow buffer bytes before transformation. `bytes_loaded`, `fetch_time`, and `transform_time`
are cumulative across iterations of the same dataset instance; because work runs concurrently, the summed stage times
can exceed wall-clock time. The queue depths report rows currently waiting in the pipeline, and the progress counters
reflect the latest iterator snapshot.
### Filtering data
By default, the streaming data loader includes all rows and columns. LanceDB is a columnar database that also supports
efficient random access. Reducing the number of columns you load has a direct impact on I/O performance. Reducing the
number of rows can also help, especially with a selective filter, large values, local data, or the LanceDB Enterprise
cache.
Use the `columns` parameter to specify which columns to load and the `filter` parameter to specify which rows to load.
The filter is evaluated once when the permutation is constructed, before the rows are divided into splits. Filtered-out
rows are not loaded from storage during iteration, and split sizes reflect the filtered row count.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
ds = StreamingDataset(
table,
shuffle_seed=42,
columns=["image", "label"], # skip all other columns
filter="category = 'train'", # only training rows
)
```
### Shuffling rows
By default, `StreamingDataset` sets `shuffle=True` and randomly assigns rows to splits. This helps prevent the model
from learning artifacts from storage order. Set `shuffle=False` to divide rows into splits sequentially, which is
useful for deterministic evaluation and debugging.
The effective shuffle seed combines `shuffle_seed` with `epoch`, so each epoch has a different ordering while runs
with the same inputs remain reproducible. Keep `epoch=0` if you want the same ordering across iterations. The default
`shuffle_seed` is `0`; set it to `None` to generate a random seed when the dataset is constructed.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Training loop: each epoch gets a different shuffled ordering
for epoch in range(num_epochs):
ds = StreamingDataset(
table,
shuffle_seed=42,
epoch=epoch, # changes the permutation each epoch
)
for sample in ds:
train_step(sample)
# Evaluation: deterministic sequential order, no shuffle
eval_ds = StreamingDataset(table, shuffle=False)
for sample in eval_ds:
eval_step(sample)
```
Shuffling can have significant impacts on I/O performance, especially if you are loading data from cloud storage.
In many cases the GPU pipeline is slow enough that this penalty will not be noticeable. However,
you can use the `shuffle_clump_size` parameter to shuffle the data in clumps (small contiguous batches that get
shuffled together). This will give some penalty to the randomness of the shuffle, but will significantly improve
I/O performance.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Clumped shuffle: groups of 16 contiguous rows are shuffled together,
# preserving read locality while still randomising the global ordering.
ds = StreamingDataset(
table,
shuffle_seed=42,
shuffle_clump_size=16,
)
```
### Data splits and elasticity
`StreamingDataset` partitions the permutation into a fixed number of equal-sized groups called splits. Each rank gets
a contiguous group of splits, and each DataLoader worker gets a contiguous subgroup of its rank's splits. Samples are
then yielded by round-robining over the assigned splits.
If `num_splits` is omitted, it defaults to `world_size`. This works when `num_workers=0`, but it ties the split layout
to the current number of ranks. A run resumed with a different world size would then construct a different layout and
would not see the same global batches.
To get **elastic determinism**, choose a fixed `num_splits` that is compatible with every topology you plan to use.
For example, a run with `num_splits=8` and 4 GPUs assigns 2 splits to each rank. The ranks pull from their splits in
round-robin order, producing the same global batches as a run with 8 GPUs and the same `num_splits`, `shuffle_seed`,
and `epoch`.
The following constraints apply:
* `num_splits` must be divisible by `world_size`.
* With DataLoader worker processes, `num_splits` must be divisible by `world_size * num_workers`.
* For the same samples to form each global training step across topologies, `global_batch_size` must be a multiple of
`num_splits`.
* The filtered row count must be at least `num_splits`. If it is not evenly divisible by `num_splits`, up to
`num_splits - 1` surplus rows are dropped so that every split has the same length.
For example, to switch between 8 and 6 GPUs, use a `num_splits` value divisible by both, such as 24. Highly composite
values can support several layouts: 48 supports 1, 2, 3, 4, 6, 8, 12, 16, 24, and 48 GPUs, while 60 supports 1, 2, 3,
4, 5, 6, 10, 12, 15, 20, 30, and 60 GPUs. Remember to include `num_workers` when checking divisibility.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import torch.distributed as dist
dist.init_process_group("nccl")
rank = dist.get_rank()
world_size = dist.get_world_size()
# num_splits=48 is divisible by 1, 2, 3, 4, 6, 8, 12, 16, 24, 48
# so this dataset works unchanged as you scale up or down GPUs.
ds = StreamingDataset(
table,
num_splits=48,
shuffle_seed=42,
epoch=current_epoch,
rank=rank,
world_size=world_size,
)
for sample in ds:
train_step(sample)
```
### Checkpointing and resumability
Model training is expensive, and failures can occur partway through a run. A model checkpoint is not enough for an
exact resume: the streaming data loader must also continue from the same position. `StreamingDataset.state_dict()`
captures the number of samples consumed from every split in a plain Python dictionary, and `load_state_dict()` restores
that position.
Save this state at a global-step boundary where every split has contributed equally. The easiest way to guarantee this
is to make `global_batch_size` a multiple of `num_splits`, as in the example below. If you checkpoint partway through a
round-robin cycle, the state reflects the preceding complete cycle and those partial-cycle samples can be replayed.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import torch
global_batch_size = 48 # one sample per split in each global step
batch_size = global_batch_size // world_size
dataset = StreamingDataset(
table,
num_splits=48,
shuffle_seed=42,
epoch=current_epoch,
rank=rank,
world_size=world_size,
)
dataloader = torch.utils.data.DataLoader(
dataset,
batch_size=batch_size,
num_workers=0,
)
for step, batch in enumerate(dataloader):
train_step(batch)
if (step + 1) % checkpoint_interval == 0:
torch.save(
{"model": model.state_dict(), "dataset": dataset.state_dict()},
f"checkpoint_{step + 1}.pt",
)
# --- resuming after a crash ---
checkpoint = torch.load("checkpoint_100.pt")
model.load_state_dict(checkpoint["model"])
dataset_state = checkpoint["dataset"]
dataset = StreamingDataset(
table,
num_splits=dataset_state["num_splits"],
shuffle_seed=dataset_state["shuffle_seed"],
epoch=dataset_state["epoch"],
rank=rank,
world_size=world_size, # may differ from the run that saved the checkpoint
)
dataset.load_state_dict(dataset_state)
dataloader = torch.utils.data.DataLoader(
dataset,
batch_size=global_batch_size // world_size,
num_workers=0,
)
for batch in dataloader:
train_step(batch)
```
`load_state_dict()` rejects a checkpoint whose `num_splits` or `shuffle_seed` differs from the new dataset. For an
exact resume, also use the same table snapshot, `epoch`, `shuffle`, `filter`, and other data-selection settings. The
`world_size` and number of DataLoader workers may change as long as the split and batch-size divisibility constraints
still hold.
## Permutations
In more complicated scenarios, you may want the flexibility to shuffle, split, and select data without using the full
iterable streaming data loader. In these cases, use `Permutation`, the lower-level class on which `StreamingDataset`
is built. A `Permutation` defines a custom ordering of the data and supports map-style access through `__getitem__()`
and batched access through `__getitems__()`.
# Object Detection for AV Perception
Source: https://docs.lancedb.com/training/object-detection
End-to-end fine-tuning of a Faster R-CNN object detector on curated dashcam slices, using LanceDB as the data backbone from raw frames to checkpoints.
This example walks through fine-tuning an autonomous vehicle (AV) perception model on targeted failure-mode slices of [BDD100K](https://www.bdd100k.com/) — riders, nighttime pedestrians, and distant pedestrians — using LanceDB as a single multimodal table from raw JPEG bytes through to the PyTorch training loop.
The full pipeline lives in the [lancedb/training](https://github.com/lancedb/training/tree/main/object-detection) repository. This page focuses on the parts most relevant to training: defining curated splits as materialized views, loading them through the [`Permutation`](/training/) API, and pinning checkpoints to an exact data version.
## What you get
Fine-tuning Faster R-CNN ResNet50 FPN v2 for 10 epochs on each curated slice (batch size 64, AMP, A100), starting from the same COCO-pretrained checkpoint and evaluating on the matching validation view:
| Failure mode | Metric | Baseline (COCO) | Fine-tuned | Δ% |
| ------------------------ | -------- | --------------- | ---------- | ---------- |
| **Nighttime pedestrian** | mAP\@0.5 | 0.4025 | **0.5192** | **+29.0%** |
| | Recall | 0.5923 | **0.7570** | **+27.8%** |
| **Rider** | mAP\@0.5 | 0.5563 | **0.6676** | **+20.0%** |
| | Recall | 0.6788 | **0.7847** | **+15.6%** |
| **Distant pedestrian** | mAP\@0.5 | 0.4746 | **0.5788** | **+22.0%** |
| | Recall | 0.6794 | **0.8024** | **+18.1%** |
No external data added — only training-distribution correction via SQL filters over a single Lance table. Each panel below shows the same frame with three overlaid predictions: **green** = ground truth · **red** = pretrained COCO baseline · **blue** = fine-tuned model.
![Rider detection — ground truth vs baseline vs fine-tuned]()
![Nighttime pedestrian detection — ground truth vs baseline vs fine-tuned]()
![Distant pedestrian detection — ground truth vs baseline vs fine-tuned]()
The rest of the page walks through the pipeline that produced these checkpoints.
## The failure modes
A perception model fine-tuned on a generic dataset typically misses the long-tail scenarios that matter most in deployment. Three common failure modes drive this example:
| Failure mode | Curation signal |
| -------------------------------------- | --------------------------------------------------- |
| **Riders** (person on bike/motorcycle) | `has_rider = true` |
| **Nighttime pedestrians** | `timeofday = 'night' AND has_person = true` |
| **Distant pedestrians** | `has_person = true AND person_bbox_area_pct < 30.0` |
Each curated slice becomes a [materialized view](/geneva/jobs/materialized-views) — a named, refreshable SQL filter over the source table — and the training script loads it by name. New footage flows in through `add()` → `backfill()` → `refresh()`; no manifests, no exports, no reshuffling on disk.
## 1. Schema
The source table holds raw image bytes alongside structured annotations. Bounding boxes are stored as a parallel list (one element per box) rather than a nested struct so they remain directly queryable with SQL.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
BDD_SCHEMA = pa.schema([
pa.field("image_id", pa.string()),
pa.field("split", pa.string()), # "train" | "val"
pa.field("image_bytes", pa.large_binary()), # raw JPEG
pa.field("width", pa.int32()),
pa.field("height", pa.int32()),
# scene metadata
pa.field("weather", pa.string()),
pa.field("scene", pa.string()),
pa.field("timeofday", pa.string()),
# annotations — parallel lists, one element per box
pa.field("ann_categories", pa.list_(pa.string())),
pa.field("ann_bboxes", pa.list_(pa.list_(pa.float32()))),
pa.field("ann_occluded", pa.list_(pa.bool_())),
])
```
Ingestion streams `pa.RecordBatch`es of raw frames + annotations directly into a Lance table — no intermediate preprocessing job. The table can live on local disk, S3, GCS, or Azure; everything downstream (backfills, views, the training loader) opens it in place via `lancedb.connect("s3://...")` with no local copy step.
## 2. Backfill curation features with Geneva
Curation signals are added as columns on the same table using [Geneva UDFs](/geneva/). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
from geneva.transformer import udf
# Tier 1 — CPU, derived from annotations alone
@udf(data_type=pa.bool_(), input_columns=["ann_categories"])
def has_rider(ann_categories: list[str]) -> bool:
return "rider" in (ann_categories or [])
# Tier 2 — GPU, runs a Faster R-CNN to find the largest detected person
# as a percentage of frame area. <30% = a distant or small pedestrian,
# the hard case we want to upweight in training.
@udf(data_type=pa.float32(),
input_columns=["image_bytes", "width", "height"],
cuda=True, num_gpus=1)
class PersonBboxAreaPct:
def __init__(self):
self._model = None
def __call__(self, image_bytes, width, height):
# lazy model load — runs once per Ray worker, then reused
...
```
Run the backfill against the live table:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import geneva
gconn = geneva.connect("data/bdd100k/lancedb")
tbl = gconn.open_table("bdd100k")
tbl.add_columns({"has_rider": has_rider})
tbl.add_columns({"person_bbox_area_pct": PersonBboxAreaPct()})
with gconn.local_ray_context():
tbl.backfill("has_rider")
tbl.backfill("person_bbox_area_pct")
```
Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See the [Geneva end-to-end example](/geneva/end-to-end) for more on the backfill pattern.
## 3. Define training splits as materialized views
A training split is a named SQL filter, not a CSV manifest. Each view stays in sync with the source table and bumps its `version` on every refresh — the link between a checkpoint and the exact data that produced it.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import geneva
gconn = geneva.connect("data/bdd100k/lancedb")
gtbl = gconn.open_table("bdd100k")
VIEWS = {
"bdd100k_rider_train":
"has_rider = true AND split = 'train'",
"bdd100k_rider_val":
"has_rider = true AND split = 'val'",
"bdd100k_nighttime_person_train":
"timeofday = 'night' AND has_person = true AND split = 'train'",
"bdd100k_nighttime_person_val":
"timeofday = 'night' AND has_person = true AND split = 'val'",
"bdd100k_distant_person_train":
"has_person = true AND person_bbox_area_pct < 30.0 AND split = 'train'",
"bdd100k_distant_person_val":
"has_person = true AND person_bbox_area_pct < 30.0 AND split = 'val'",
}
with gconn.local_ray_context():
for name, sql_filter in VIEWS.items():
query = gtbl.search().where(sql_filter)
mv = gconn.create_materialized_view(name, query)
mv.refresh()
print(f"[{name}] {mv.count_rows()} rows (version {mv.version})")
```
## 4. PyTorch DataLoader via the Permutation API
The training script doesn't know about the filter — it opens a view by name and reads through the [`Permutation`](/training/) API. Each DataLoader worker reopens its own connection lazily, reads Arrow batches directly from Lance (zero-copy, no intermediate file format), and the collate function decodes the whole batch in one pass. `Permutation` provides random-access indexing over the table, so shuffling is a cheap pointer rewrite rather than a full-dataset shuffle on disk.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
import torch
import torchvision.io as tio
from lancedb.permutation import Permutation
DETECTION_COLS = ["image_bytes", "ann_categories", "ann_bboxes"]
class LanceDetectionDataset(torch.utils.data.Dataset):
def __init__(self, uri: str, table_name: str):
self.uri, self.table_name = uri, table_name
self._perm = None
self.length = len(lancedb.connect(uri).open_table(table_name))
def __len__(self):
return self.length
def __getstate__(self):
# Permutation holds Rust async state — zero it so each worker reopens
state = self.__dict__.copy()
state["_perm"] = None
return state
def _ensure_open(self):
if self._perm is None:
tbl = lancedb.connect(self.uri).open_table(self.table_name)
self._perm = (
Permutation.identity(tbl)
.select_columns(DETECTION_COLS)
.with_format("arrow") # zero-copy
)
def __getitems__(self, indices: list[int]):
self._ensure_open()
return self._perm.__getitems__(indices)
```
The collate function decodes JPEG bytes and converts BDD category strings into COCO class IDs (so the comparison against the pretrained checkpoint is valid):
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
BDD_LABEL_MAP = {
"person": 1, "rider": 1, "bicycle": 2, "car": 3, "motorcycle": 4,
"bus": 6, "train": 7, "truck": 8, "traffic light": 10,
}
def detection_collate(batch):
images, targets = [], []
for raw, cats, bboxes in zip(
batch.column("image_bytes").to_pylist(),
batch.column("ann_categories").to_pylist(),
batch.column("ann_bboxes").to_pylist(),
):
buf = torch.frombuffer(bytearray(raw), dtype=torch.uint8)
images.append(tio.decode_image(buf, tio.ImageReadMode.RGB).float() / 255.0)
valid_boxes, valid_labels = [], []
for cat, box in zip(cats or [], bboxes or []):
lid = BDD_LABEL_MAP.get(cat)
if lid is None or box[2] <= box[0] or box[3] <= box[1]:
continue
valid_boxes.append(box)
valid_labels.append(lid)
targets.append({
"boxes": torch.tensor(valid_boxes or [], dtype=torch.float32).reshape(-1, 4),
"labels": torch.tensor(valid_labels or [], dtype=torch.int64),
})
return images, targets
```
Wire it into a standard `torch.utils.data.DataLoader`:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
def make_loader(uri, table_name, batch_size=64, num_workers=8, shuffle=False):
dataset = LanceDetectionDataset(uri, table_name)
sampler = torch.utils.data.RandomSampler(dataset) if shuffle else None
return torch.utils.data.DataLoader(
dataset,
batch_size=batch_size,
sampler=sampler,
num_workers=num_workers,
collate_fn=detection_collate,
pin_memory=torch.cuda.is_available(),
persistent_workers=(num_workers > 0),
multiprocessing_context="spawn" if num_workers > 0 else None,
)
```
`with_format("arrow")` keeps batches as zero-copy `pa.RecordBatch`es — no per-row Python boxing, no pickling between worker and main. Each DataLoader worker reopens its own `Permutation` after fork (the Rust async handle is cleared in `__getstate__`), so reads scale with `num_workers` and stream straight from the underlying object store. JPEG decode overlaps with GPU compute via `pin_memory` + `prefetch_factor`, which is what keeps the loader from becoming the bottleneck on a fast GPU.
## 5. Fine-tune Faster R-CNN
The training loop is plain PyTorch — the Lance integration ends at the loader. Mixed precision is enabled on CUDA for \~2× speedup on Ampere GPUs.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import time
import torch
from torchvision.models.detection import (
fasterrcnn_resnet50_fpn_v2, FasterRCNN_ResNet50_FPN_V2_Weights,
)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
use_amp = device.type == "cuda"
# COCO pretrained weights — head left intact since BDD uses a subset of COCO IDs
model = fasterrcnn_resnet50_fpn_v2(
weights=FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1
).to(device)
train_loader = make_loader("data/bdd100k/lancedb",
"bdd100k_rider_train",
batch_size=64, num_workers=14, shuffle=True)
val_loader = make_loader("data/bdd100k/lancedb",
"bdd100k_rider_val",
batch_size=64, num_workers=14)
optimizer = torch.optim.SGD(
[p for p in model.parameters() if p.requires_grad],
lr=0.04, momentum=0.9, weight_decay=1e-4,
)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)
scaler = torch.cuda.amp.GradScaler() if use_amp else None
for epoch in range(1, 11):
model.train()
t0 = time.time()
for images, targets in train_loader:
images = [img.to(device) for img in images]
targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
if all(t["labels"].numel() == 0 for t in targets):
continue
with torch.cuda.amp.autocast(enabled=use_amp):
losses = sum(model(images, targets).values())
optimizer.zero_grad()
if use_amp:
scaler.scale(losses).backward()
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), 10.0)
scaler.step(optimizer)
scaler.update()
else:
losses.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 10.0)
optimizer.step()
scheduler.step()
print(f"epoch {epoch} ({time.time() - t0:.1f}s)")
```
## 6. Pin the checkpoint to a data version
Every Lance table — including a materialized view — exposes a monotonically increasing `version`. Logging it next to the weights gives a permanent, deterministic link between a checkpoint and the exact data snapshot that produced it.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import json
from pathlib import Path
train_tbl = lancedb.connect("data/bdd100k/lancedb").open_table("bdd100k_rider_train")
out = Path("checkpoints/rider")
out.mkdir(parents=True, exist_ok=True)
torch.save(model.state_dict(), out / "fasterrcnn_bdd_finetuned.pt")
with open(out / "metadata.json", "w") as f:
json.dump({
"train_table": train_tbl.name,
"table_version": train_tbl.version,
"row_count": len(train_tbl),
}, f, indent=2)
```
To reproduce a run, [time-travel](/tables/versioning) the view to the recorded version before opening the loader:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
tbl = lancedb.connect("data/bdd100k/lancedb").open_table("bdd100k_rider_train")
tbl.checkout(version=7) # exact snapshot the checkpoint was trained on
```
## 7. Continuous updates
When new footage arrives, the same three calls update every downstream view — no view definitions change, no training-script edits required:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# 1. ingest the new footage into the source table
table.add(new_record_batches)
# 2. backfill computes only the new rows (incremental, checkpointed)
with gconn.local_ray_context():
tbl.backfill("has_rider")
tbl.backfill("person_bbox_area_pct")
# 3. refresh appends qualifying new rows to every materialized view
for view_name in gconn.table_names():
if view_name == "bdd100k":
continue
mv = gconn.open_table(view_name)
before = mv.count_rows()
mv.refresh()
print(f"[{view_name}] {before} → {mv.count_rows()} rows (version {mv.version})")
```
The next training run picks up the new data automatically — and pins itself to the new `version`.
## Full source
The complete code, including a synthetic-data mode for pipeline verification (`--synthetic 500`), GPU UDFs for CLIP embeddings and dHash deduplication, and the EDA notebook, is in this [GitHub repository](https://github.com/lancedb/training/tree/main/object-detection).
# PyTorch Integration
Source: https://docs.lancedb.com/training/torch
Learn how to use LanceDB with PyTorch for training and inference.
LanceDB provides a seamless integration with PyTorch for training and inference. This allows you to use LanceDB as a backend for your PyTorch models, and to use PyTorch for training and inference. You can use LanceDB to store your data, and PyTorch to train your models.
## Quickstart
The `Table` class in LanceDB implements a contract for a PyTorch
[Dataset](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.Dataset).
This means you can simply use a LanceDB table in a PyTorch dataloader directly.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
import torch
import pyarrow as pa
from lancedb.util import tbl_to_tensor
mem_db = lancedb.connect("memory://")
table = mem_db.create_table("test_table", pa.table({"a": range(1000)}))
# Any LanceDB table can be used as a PyTorch Dataset
dataloader = torch.utils.data.DataLoader(
table, batch_size=1024, shuffle=True, collate_fn=tbl_to_tensor
)
for batch in dataloader:
print(batch)
```
Although the `Table` class in LanceDB implements the `torch.utils.data.Dataset` interface, you may find that using
a table [Permutation](/training/) is more flexible.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.permutation import Permutation
permutation = Permutation.identity(table)
dataloader = torch.utils.data.DataLoader(permutation)
```
## Output Formats
By default, a `Table` data loader will emit Arrow data. `collate_fn` is PyTorch's batching hook: PyTorch calls it to
turn the fetched items into one batch. PyTorch's default collate function only knows how to combine tensors, NumPy
arrays, numbers, dicts, and lists, so it does not accept Arrow data directly. When using a `Table` directly, pass
LanceDB's `lancedb.util.tbl_to_tensor` helper as PyTorch's `collate_fn`; it converts numeric Arrow columns into a
column-major `torch.Tensor` with shape `(columns, rows)`.
`Permutation` works differently: its default output is a list of Python dicts, which PyTorch's default collate function
can batch into a dict of tensors. This is usually more convenient when you are getting started. However, there is a
significant performance penalty converting from Arrow, Lance's internal representation, to this default format. If you
want the default PyTorch dict-of-tensors behavior, use a `Permutation` as-is; if you want direct Arrow-to-tensor
conversion, either pass `lancedb.util.tbl_to_tensor` as `collate_fn` with a direct `Table` or configure a `Permutation`
with one of the transform formats described below.
To address this, the `Permutation` class provides a set of builtin transform functions that can be applied to map
the Arrow data in different ways. The `arrow` and `polars` formats will always avoid data copies. However, `numpy`,
`pandas`, and `torch_col` formats will also avoid data copies in most cases. The `python`, `python_col`, and
`torch` formats will all require at least one full copy of the data and are the slowest options.
### Using the torch\_col format with a torch data loader
The `torch_col` format is the most efficient way to convert from Arrow to a `torch.Tensor`. It will convert the
entire Arrow batch to a *column-major* `torch.Tensor`. In other words, given C columns and R rows, the resulting
Tensor will have shape `(C, R)`. However, this format generates an error if you are using a
`torch.utils.data.DataLoader` with the default collation function:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
TypeError: stack(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor
```
This error occurs because the default collation function does not currently expect a single two-dimensional tensor.
It expects a list of tensors which it will then stack. This is what is output by the `torch` format but that format
requires a data copy. To avoid this error, and avoid data copies, you will need to provide a custom collation function
in addition to specifying the `torch_col` format.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.permutation import Permutation
permutation = Permutation.identity(table).with_format("torch_col")
dataloader = torch.utils.data.DataLoader(permutation, collate_fn=lambda x: x)
```
This will now output a single two-dimensional tensor for each batch.
## Selecting columns
By default, the `Table` class will return all columns in the table when used as input to PyTorch. If you only need
a subset of columns, you can significantly reduce your I/O requirements by selecting only the columns you need. The
`Permutation` class provides a `select_columns` method that provides this functionality.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.permutation import Permutation
permutation = Permutation.identity(table).select_columns(["id", "prompt"])
dataloader = torch.utils.data.DataLoader(
permutation, batch_size=1024, shuffle=True
)
for batch in dataloader:
print(batch.schema)
```
## Using multiple DataLoader workers
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. LanceDB tables and `Permutation` objects are picklable, so each worker reopens the table after it starts.
Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and it is set to become the Python default. See [the performance guide](/performance) for more multiprocessing guidance.
`forkserver` is only available on POSIX systems (Linux and macOS). On Windows, use `spawn` instead — it is the only start method available there.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import torch
from lancedb.permutation import Permutation
permutation = Permutation.identity(table)
dataloader = torch.utils.data.DataLoader(
permutation,
batch_size=1024,
shuffle=True,
num_workers=4,
multiprocessing_context="forkserver",
persistent_workers=True,
)
```
### Remote tables in DataLoader workers
Remote LanceDB Enterprise tables (`db://...`) work the same way: workers reopen the table from the pickled connection state.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
import torch
from lancedb.util import tbl_to_tensor
db = lancedb.connect(
"db://my-database",
api_key="sk-...",
region="us-east-1",
)
table = db.open_table("my_table")
dataloader = torch.utils.data.DataLoader(
table,
batch_size=512,
num_workers=4,
multiprocessing_context="forkserver",
collate_fn=tbl_to_tensor,
)
```
This sends the connection state, including the API key, to each worker. Use a connection factory if credentials should be loaded inside the worker or your `client_config` contains a non-serializable `header_provider`.
### Providing a custom connection factory
`Permutation.with_connection_factory` lets each worker reopen the base table with custom logic. The factory takes the table name, returns a LanceDB table, and must be picklable.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import os
import lancedb
import torch
from lancedb.permutation import Permutation
def open_table(name: str):
db = lancedb.connect(
"db://my-database",
api_key=os.environ["LANCEDB_API_KEY"],
region="us-east-1",
)
return db.open_table(name)
table = open_table("my_table")
permutation = (
Permutation.identity(table)
.with_connection_factory(open_table)
)
dataloader = torch.utils.data.DataLoader(
permutation,
batch_size=512,
num_workers=4,
multiprocessing_context="forkserver",
)
```
# Fine-tuning a VLM on TextVQA
Source: https://docs.lancedb.com/training/vlm-finetuning
End-to-end fine-tuning of Qwen2.5-VL on a curated TextVQA slice, using LanceDB and Geneva to materialize expensive vision-language features once and train from cached columns.
This example walks through a vision-language model (VLM) fine-tuning pipeline for [TextVQA](https://textvqa.org/), where the task is to answer questions that require reasoning over text *inside an image*. The base model is `Qwen2.5-VL-3B-Instruct`, fine-tuned with the [QLoRA](https://arxiv.org/abs/2305.14314) method. The data backbone is one Lance table that evolves from raw multimodal rows into training-ready features.
The key idea is simple: in this QLoRA fine-tuning setup, we freeze the VLM's image encoder and train only a small adapter on the language-model side. We call that encoder the **vision tower** in this example: it is the part of the model that turns image pixels into visual hidden states before the language model reads them alongside the text prompt.
Because the vision tower's weights do not change during fine-tuning, its output for a given image also does not change. That means the pipeline can compute those visual hidden states once, store them as a fixed-size Lance column, and reuse them in every epoch instead of recomputing them in every training step. This also helps the run fit comfortably on a small GPU, because the training job does not need to keep the vision encoder active or pay for its forward pass on every batch.
Run the Colab-sized workflow on a free T4: download the pre-baked Lance subset, explore it, benchmark Lance vs Parquet, fine-tune with QLoRA, and evaluate base vs tuned answers.
Full demo repository with the notebook, Geneva UDFs, direct backfill fallback, dataloader, training loop, and evaluation scripts.
The Colab notebook uses a pre-baked subset of the TextVQA dataset: it downloads a curated Lance subset whose expensive feature columns have already been computed. This page explains the complete end-to-end pipeline that produced that subset, then shows how the notebook applies it to produce a fine-tuned model that improves performance on the TextVQA task.
## What you get
On the curated `text_dense` TextVQA slice, the demo fine-tunes `Qwen2.5-VL-3B-Instruct` with QLoRA and evaluates on held-out images:
| Setup | TextVQA accuracy |
| ---------------- | -------------------------- |
| Base model | 0.799 |
| LoRA-tuned model | **0.820** |
| Lift | **+2.1 percentage points** |
The larger point is not the absolute score, because you could just as well fine-tune a better base model on more data. The main takeaways are the workflow and quality-of-life improvements that you get when you combine LanceDB and Geneva:
1. **Add expensive features** as new columns without rewriting the raw dataset.
2. **Read fixed-size model features efficiently** for shuffled PyTorch batches.
3. **Iterate quickly** from feature idea to scalable CPU/GPU backfill, using Geneva UDFs.
## Why LanceDB fits this workflow
VLM fine-tuning pipelines spend a lot of time between "I have an experiment idea" and "I trained the model." LanceDB shortens that loop in three places.
Lance can append derived columns such as `ocr_token_count`, `dhash`, `vision_tower_hiddens`, and tokenized SFT prompts without rewriting the existing image/question/answer columns or managing sidecar files.
Lance is optimized for scans and random access over fixed-size lists, which are common in model training: embeddings, hidden states, token IDs, masks, and labels.
Geneva lets AI engineers express feature work as UDFs, run those UDFs across CPU or GPU workers, and materialize the results directly into the same Lance table.
In this pipeline, those three properties combine into the core optimization: compute the VLM vision features once, store them cheaply, then train by reading only the cached columns the model needs.
## Pipeline overview
The runnable demo uses the exact Colab subset hosted at [`lance-format/textvqa-lance-colab`](https://huggingface.co/datasets/lance-format/textvqa-lance-colab). It is derived from the Lance-formatted TextVQA corpus and stores inline JPEG bytes, questions, answers, OCR tokens, object classes, CLIP image/question embeddings, and the cached training features used by this example. The full demo pipeline adds three tiers of derived features on top.
Cheap CPU columns such as `question_length`, `answer_length`, `question_type`, and `ocr_token_count`.
Image-derived columns such as `dhash`, computed by decoding the JPEG once and storing a perceptual hash.
GPU-heavy columns: `vision_tower_hiddens` plus SFT token fields (`input_ids`, `attention_mask`, `labels`).
The Colab notebook's workflow starts after all three tiers have been computed. It downloads a small curated subset and runs the training/evaluation path without needing to run Geneva or the vision-tower backfill on the notebook GPU.
## 1. Start with a multimodal LanceDB table
The base schema comes from the TextVQA Lance dataset. One row contains the image bytes, natural-language question, reference answers, OCR tokens, scene tags, and retrieval embeddings.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import pyarrow as pa
BASE_SCHEMA = pa.schema([
pa.field("id", pa.int64()),
pa.field("image", pa.large_binary()),
pa.field("image_id", pa.string()),
pa.field("question_id", pa.string()),
pa.field("question", pa.string()),
pa.field("answers", pa.list_(pa.string())),
pa.field("answer", pa.string()),
pa.field("image_emb", pa.list_(pa.float32(), 512)),
pa.field("question_emb", pa.list_(pa.float32(), 512)),
pa.field("ocr_tokens", pa.list_(pa.string())),
pa.field("image_classes", pa.list_(pa.string())),
pa.field("set_name", pa.string()),
])
```
Because the raw image, text, OCR, and embedding features live together, the same table supports curation, retrieval, feature engineering, and training. For example, the notebook can run a text-to-image retrieval demo by searching `image_emb` with a question embedding that already exists in the row.
## 2. Add feature columns with Geneva
Geneva turns feature engineering into UDF definitions plus backfills. The UDFs can be simple text functions, image-processing functions, or stateful GPU model calls.
The Tier 1 features are ordinary CPU UDFs:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import re
import pyarrow as pa
from geneva.transformer import udf
_QUESTION_TYPE_PATTERNS = [
("how_many", re.compile(r"^\s*how\s+many\b", re.IGNORECASE)),
("what_brand", re.compile(r"^\s*what\s+(is\s+the\s+)?(brand|company|make)\b", re.IGNORECASE)),
("what", re.compile(r"^\s*what\b", re.IGNORECASE)),
]
@udf(data_type=pa.string(), input_columns=["question"])
def question_type(question: str) -> str:
for label, pattern in _QUESTION_TYPE_PATTERNS:
if pattern.search(question or ""):
return label
return "other"
@udf(data_type=pa.int32(), input_columns=["ocr_tokens"])
def ocr_token_count(ocr_tokens: list[str] | None) -> int:
return len(ocr_tokens) if ocr_tokens else 0
```
The Tier 3 feature is heavier: run Qwen2.5-VL's frozen vision tower once, then store the merged visual hidden states as a fixed-size fp16 list.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
IMAGE_PX = 560
LLM_TOKENS_PER_IMAGE = 400
VISION_HIDDEN = 2048
@udf(
data_type=pa.list_(pa.float16(), LLM_TOKENS_PER_IMAGE * VISION_HIDDEN),
input_columns=["image"],
)
class VisionTowerEmbedder:
def __init__(self):
self._model = None
self._processor = None
def _lazy_load(self):
if self._model is not None:
return
import torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
self._torch = torch
self._model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2.5-VL-3B-Instruct",
torch_dtype=torch.bfloat16,
device_map="cuda:0",
).model.visual.eval()
self._processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
def __call__(self, image: bytes) -> list[float]:
self._lazy_load()
# Decode image, resize to IMAGE_PX, run the frozen vision tower,
# and return fp16[400, 2048] flattened as one fixed-size list.
...
```
The fixed shape matters. With `IMAGE_PX = 560`, Qwen2.5-VL produces 400 merged visual tokens, each with hidden size 2048. That becomes one `fp16[400 * 2048]` column per row. Training can scan and randomly access that column without decoding images or running the vision tower in the hot loop, saving GPU compute at training time.
Run the tiered backfill with Geneva:
```bash bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
python -m vlm.backfill_geneva --tier 1 # CPU text columns
python -m vlm.backfill_geneva --tier 2 # image decode + dhash
python -m vlm.backfill_geneva --tier 3 # vision tower + SFT tokens
```
The same Tier 3 work can be done manually by creating PyArrow batches and calling Lance's column-evolution APIs directly. The demo repo includes [`backfill_direct.py`](https://github.com/lancedb/tmls-2026-demo/blob/main/vlm/backfill_direct.py) for that path. Geneva is the preferred abstraction when you want to scale the same feature code across CPU or GPU workers and keep backfills incremental.
See the full UDF registry in [`vlm/geneva_udfs.py`](https://github.com/lancedb/tmls-2026-demo/blob/main/vlm/geneva_udfs.py) and the backfill driver in [`vlm/backfill_geneva.py`](https://github.com/lancedb/tmls-2026-demo/blob/main/vlm/backfill_geneva.py).
## 3. Curate a training slice
The demo uses a `text_dense` slice: TextVQA examples whose images contain many OCR tokens. The slice was chosen empirically because it gave the clearest LoRA lift over the already-strong base model.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
TEXT_DENSE_OCR_THRESHOLD = 16
def matches_text_dense(row: dict) -> bool:
return len(row.get("ocr_tokens") or []) >= TEXT_DENSE_OCR_THRESHOLD
```
The Colab-ready bake ingests a small train split, backfills Tier 3 on that train table, ingests a held-out validation split, and optionally pushes the result to Hugging Face:
```bash bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
python -m vlm.colab_prepare \
--out data/colab \
--slice text_dense \
--train-rows 600 \
--val-rows 400 \
--hf-repo lance-format/textvqa-lance-colab \
--push
```
The train table contains cached Tier 3 columns because training reads them directly. The validation table keeps raw images because evaluation should run the full VLM on unseen images.
## 4. Explore the prepared table
Before training, it helps to look at the actual task. Each row pairs an image with a question whose answer is often visible as text in the image: a product label, phone screen, sign, book spine, or package.
**A:** TWA
**OCR:** 7h the Finest... 74 1E TWA 8 SALT REESE PEPPER
**A:** 12:39 am
**OCR:** AT\&T 12:39 AM TV CS WATCH P PANDORA YouTube Ustream
**A:** lego
**OCR:** LEGO CITY Ages/edades 5-12 POLICE B-403 4473 112 112 pcs
**A:** warning
**OCR:** WARNING Controlled Area Itis unlawf enter thisre without permission nstallation
The notebook downloads the public pre-baked subset:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from huggingface_hub import snapshot_download
import lancedb
import os
local = snapshot_download(
repo_id="lance-format/textvqa-lance-colab",
repo_type="dataset",
local_dir="data/colab",
)
def open_tbl(path: str):
name = os.path.basename(path).removesuffix(".lance")
return lancedb.connect(os.path.dirname(path)).open_table(name)
train_tbl = open_tbl(f"{local}/textvqa_colab_train.lance")
val_tbl = open_tbl(f"{local}/textvqa_colab_val.lance")
```
Because the table also ships CLIP embeddings, you can run cross-modal retrieval without loading a model:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import numpy as np
seed = (
train_tbl.search()
.select(["question", "question_emb"])
.limit(40)
.to_arrow()
.to_pylist()[11]
)
hits = (
train_tbl.search(
np.asarray(seed["question_emb"], dtype=np.float32),
vector_column_name="image_emb",
)
.select(["image", "question", "answer", "_distance"])
.limit(5)
.to_arrow()
.to_pylist()
)
```
This is the same table that later feeds training. There is no separate feature store, image directory, Parquet export, or manifest to keep synchronized.
## 5. Benchmark Lance vs Parquet-style reads
Many training pipelines start with Parquet. Parquet is excellent for columnar analytics, but training commonly needs shuffled batches and fixed-size tensor columns. The notebook compares Lance and Parquet on two access patterns:
| Column group | Why it matters |
| ----------------------------- | ----------------------------------------------------------------------------- |
| `image`, `question`, `answer` | Raw multimodal rows: the baseline "decode and tokenize during training" path. |
| `vision_tower_hiddens` | Cached fixed-size fp16 VLM features: the optimized training path. |
The notebook mirrors those column groups to uncompressed Parquet, then measures sequential scans and shuffled random batches:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
RAW = ["image", "question", "answer"]
VEC = ["vision_tower_hiddens"]
BATCH = 8
lance_ds = train_tbl.to_lance()
n = train_tbl.count_rows()
def seq(ds, cols):
t0 = time.time()
for _ in ds.to_batches(columns=cols, batch_size=BATCH):
pass
return n / (time.time() - t0)
def shuf(ds, cols, num_batches=20):
batches = [
sorted(rng.choice(n, BATCH, replace=False).tolist())
for _ in range(num_batches)
]
t0 = time.time()
for idx in batches:
ds.take(idx, columns=cols)
return (num_batches * BATCH) / (time.time() - t0)
```
One Colab run produced the following throughput:
| Throughput, rows/s | LanceDB | Parquet |
| ------------------------------------------- | ------: | ------: |
| `image` + `question` + `answer`, sequential | 2,603 | 8,311 |
| `image` + `question` + `answer`, shuffled | 2,613 | 352 |
| `vision_tower_hiddens` fp16, sequential | 1,452 | 90 |
| `vision_tower_hiddens` fp16, shuffled | 2,149 | -- |
The takeaways are workload-specific:
* For a traditional sequential scan over raw image/question/answer columns, Parquet is faster in this run: 8,311 rows/s vs 2,603 rows/s.
* For shuffled raw multimodal batches, Lance is faster because training reads scattered rows repeatedly instead of streaming the file once.
* For cached fp16 fixed-size arrays, Lance is about 16x faster than Parquet on the sequential scan. This is the training-relevant path in this example: the model reads `vision_tower_hiddens`, token IDs, masks, and labels as fixed-size columns.
* The benchmark intentionally skips the Parquet fp16 shuffled case. Parquet would re-decode whole row groups for each random batch, which is slow enough to distract from the real use case. The sequential fp16 row already shows the layout gap, while Lance shuffled reads remain fast.
The numbers shown above are central to the example. The Tier 3 feature is only useful if the storage format can read it efficiently in the way a trainer actually needs: projected columns, repeated scans, and shuffled batches. **Lance specializes in exactly that access pattern**, including fixed-size list columns stored on disk.
## 6. Load cached columns with the Permutation API
The training DataLoader projects only the columns needed by the cached training loop:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.permutation import Permutation
CACHED_COLS = [
"vision_tower_hiddens",
"input_ids",
"attention_mask",
"labels",
]
class LancePermutationDataset(torch.utils.data.Dataset):
def __init__(self, uri: str, table_name: str):
self.uri = uri
self.table_name = table_name
self._perm = None
self.length = len(lancedb.connect(uri).open_table(table_name))
def __len__(self):
return self.length
def __getstate__(self):
state = self.__dict__.copy()
state["_perm"] = None
return state
def _ensure_open(self):
if self._perm is None:
tbl = lancedb.connect(self.uri).open_table(self.table_name)
self._perm = (
Permutation.identity(tbl)
.select_columns(CACHED_COLS)
.with_format("arrow")
)
def __getitems__(self, indices: list[int]):
self._ensure_open()
return self._perm.__getitems__(indices)
```
Each worker opens its own `Permutation`, reads Arrow batches directly from Lance, and avoids per-row Python object conversion until the collate function converts arrays into tensors.
The training batch contains:
| Field | Shape |
| ---------------- | -------------------- |
| `vision_hiddens` | `fp16[B, 400, 2048]` |
| `input_ids` | `int64[B, 512]` |
| `attention_mask` | `int64[B, 512]` |
| `labels` | `int64[B, 512]` |
## 7. Fine-tune without loading the vision tower
The training process loads the language-model side of Qwen2.5-VL in 4-bit, deletes the vision tower, and wraps the LLM projections with LoRA adapters.
During the forward pass, the model embeds the token IDs, finds the `<|image_pad|>` positions, and inserts the cached visual hidden states into those positions:
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
def forward_cached(model, batch, image_pad_id: int):
base = model.get_base_model() if hasattr(model, "get_base_model") else model
inner = base.model
inputs_embeds = inner.get_input_embeddings()(batch.input_ids)
_, _, hidden_dim = inputs_embeds.shape
mask = (
(batch.input_ids == image_pad_id)
.unsqueeze(-1)
.expand_as(inputs_embeds)
)
vision_flat = batch.vision_hiddens.to(inputs_embeds.dtype).reshape(-1, hidden_dim)
inputs_embeds = inputs_embeds.masked_scatter(mask, vision_flat)
return model(
inputs_embeds=inputs_embeds,
attention_mask=batch.attention_mask,
labels=batch.labels,
).loss
```
At this point, the LanceDB integration is done. The rest is plain PyTorch: optimizer, gradient accumulation, checkpointing, and saving the LoRA adapter.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
loader = make_cached_loader(
"data/colab/textvqa_colab_train.lance",
batch_size=2,
shuffle=True,
)
for batch in loader:
batch = batch.to(device)
loss = forward_cached(model, batch, image_pad_id)
(loss / grad_accum).backward()
...
```
This produces a training log like the following. The loss falls as the adapter learns from the cached features, and peak VRAM stays at 5.3 GB because QLoRA trains without keeping the vision tower active:
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
step 10/300 loss=2.6694 5.9 samples/s
step 20/300 loss=2.3133 6.1 samples/s
.
.
.
step 290/300 loss=0.0359 6.3 samples/s
step 300/300 loss=0.4750 6.3 samples/s
saved adapter to runs/colab_lora/lora | peak VRAM 5.3 GB
```
The training loop pays zero per-step cost for image decode, vision-tower forward, or prompt tokenization. Those costs were moved into feature engineering, where LanceDB and Geneva make them durable, incremental, and reusable.
## 8. Evaluate on held-out images
Evaluation uses the held-out validation table and loads the full VLM, including the vision tower. That is intentional: inference should see raw unseen images, not the cached train features.
```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
rows = (
val_tbl.search()
.select(["image", "question", "answer", "answers"])
.limit(256)
.to_arrow()
.to_pylist()
)
base_model, processor = load_model(adapter_dir=None, load_4bit=True)
tuned_model, processor = load_model(adapter_dir="runs/colab_lora/lora", load_4bit=True)
base_score = score_textvqa(base_model, processor, rows)
tuned_score = score_textvqa(tuned_model, processor, rows)
```
In this end-to-end example, the held-out curated validation split produced:
| Model | TextVQA accuracy |
| ----------------------------- | -------------------------: |
| Base `Qwen2.5-VL-3B-Instruct` | 0.799 |
| QLoRA-tuned adapter | **0.820** |
| Lift | **+2.1 percentage points** |
The tuned adapter is not meant to be a state-of-the-art TextVQA checkpoint. It is the proof point for the pipeline: the same Lance table supports curation, feature engineering, efficient training reads, and evaluation on held-out raw images.
The notebook renders side-by-side examples: image, question, base answer, tuned answer, and ground truth. This closes the loop from feature idea to trained model while keeping the source data, derived features, training batches, and evaluation split in Lance.
## Full source
The complete demo implementation with helper scripts and usage instructions is in [this repo](https://github.com/lancedb/tmls-2026-demo).
The runnable Colab workflow: download, explore, benchmark, train, and evaluate.
Tier 1, Tier 2, and Tier 3 feature definitions.
Geneva-powered feature materialization.
QLoRA training from cached Lance columns.
# Why LanceDB for Training
Source: https://docs.lancedb.com/training/why-lancedb
Use LanceDB as the multimodal data layer for model training, fine-tuning, curation, and feature engineering workflows.
LanceDB is built for AI teams that need a practical data layer between raw multimodal datasets and model training.
Instead of moving data through separate systems for curation, feature engineering, search, manifests, and training,
you can keep the whole workflow attached to one versioned LanceDB table.
That table can hold images, video, audio, text, annotations, metadata, embeddings, tokenized fields, model outputs,
quality signals, and training-ready tensors. As the dataset evolves, LanceDB lets you add new columns, filter rows,
pin versions, and read batches without rewriting the original data.
![Training data lifecycle: Curation, Feature Engineering, Search and Retrieval, Training]()
LanceDB gives these stages one platform, so curation, feature engineering, retrieval, and training stay connected.
## A connected data lifecycle
Training pipelines usually need more than a pile of files. They need curation, derived features, reproducible splits,
fast random access, and a clean path into frameworks such as PyTorch. LanceDB keeps these pieces connected through
the same table model, whether you organize a workflow as one table or several related tables.
Use filters, vector search, full-text search, and retrieval workflows to find the examples that matter: hard negatives,
long-tail failure modes, duplicate clusters, low-quality samples, or targeted fine-tuning slices.
Add embeddings, detections, OCR output, labels, token IDs, hidden states, deduplication flags, or quality scores as
new columns. Lance's columnar layout and schema evolution avoid rewriting large raw media columns when you add features.
Build filtered splits and materialized views from the table instead of exporting CSV manifests. Data versions and tags
make it possible to tie a checkpoint back to the exact rows and features used for training.
Use fast random access and column projection to read only the columns a training step needs. LanceDB tables can be read
from local storage or object storage, and integrate with data loading patterns such as PyTorch datasets.
## Lance as the foundation
LanceDB is built on [Lance](https://lance.org/), an open-source lakehouse format designed for multimodal AI data.
The table below highlights the Lance features that enable the multimodal lakehouse on top.
| Capability | Why it matters for training |
| ------------------------ | ------------------------------------------------------------------------------------- |
| **Multimodal columns** | Store raw bytes, annotations, metadata, embeddings, and features together. |
| **Fast random access** | Support shuffled and sampled reads without reshuffling the dataset on disk. |
| **Column projection** | Read only images, tokens, labels, embeddings, or hidden states needed by a given run. |
| **Schema evolution** | Add new feature columns without rewriting existing media columns. |
| **Versioning** | Reproduce experiments against the same table snapshot, even as the dataset evolves. |
| **Search and filtering** | Find and materialize useful training slices directly from the table. |
## Search inside training workflows
Search is not limited to QA systems, agents, or production retrieval apps. It is also a practical way to inspect,
curate, and improve training data:
* Find visually similar examples when debugging model failures.
* Retrieve hard negatives or near-duplicates for contrastive training.
* Combine vector search, full-text search, and metadata filters to build targeted fine-tuning slices.
* Reuse the same table for both offline curation and production retrieval.
In LanceDB, retrieval and training workflows can operate over the same multimodal tables instead of forcing teams to
manage separate data systems for each stage.
## Projects using LanceDB for training workflows
A platform for reproducible world-model research built on a LanceDB data layer, reporting faster data loading on Push-T workloads.
A joint-embedding predictive world model from pixels, trained on the stable-worldmodel platform and its LanceDB data layer.
A drop-in LanceDB backend for Hugging Face LeRobot datasets with faster loading across robotics datasets.
In the world-model ecosystem, [stable-worldmodel](https://github.com/galilai-group/stable-worldmodel) reports
3-4x faster data loading on Push-T versus HDF5 / MP4 at a fraction of the disk footprint. Across these projects,
LanceDB and Lance provide the multimodal data layer that keeps raw observations, annotations, features, and training
access patterns in one format instead of scattering them across task-specific stores.
## Next steps
Learn how to use LanceDB permutations to select rows, project columns, split datasets, and shuffle training reads.
Use LanceDB tables and permutations with `torch.utils.data.DataLoader`.
Fine-tune an AV perception model on curated failure-mode slices backed by one LanceDB table.
Fine-tune a VLM on TextVQA using LanceDB and Geneva to cache expensive training features.
# Troubleshooting
Source: https://docs.lancedb.com/troubleshooting
Tips for troubleshooting basic LanceDB issues.
## Frequently-asked questions
For commonly asked questions about LanceDB, please refer to our [FAQ section](/faq).
## Getting technical support
If you're using LanceDB OSS, the best place to get help is in our
[Discord community](https://discord.gg/AUEWnJ7Txb),
under the relevant language channel for Python, TypeScript, or Rust.
By asking in the language-specific channel, you can get help from the community
and our engineering team.
If you are a LanceDB Enterprise user, please contact our support team at [support@lancedb.com](mailto:support@lancedb.com) for dedicated assistance.
## General issues
### Slow or unexpected query results
If you have slow queries or unexpected query results, it can be helpful to
print the resolved query plan.
LanceDB provides two powerful tools for query analysis and optimization: `explain_plan` and `analyze_plan`.
Read the full guide on [Query Optimization](/search/optimize-queries/).
### Python's multiprocessing module
Multiprocessing with `fork` is not supported. You should use `spawn` instead.
# RAG and Agents
Source: https://docs.lancedb.com/tutorials/agents/index
Explore a variety of RAG (Retrieval-Augmented Generation) and agent applications built with LanceDB.
The table below lists example notebooks we've prepared for a variety of RAG (Retrieval-Augmented Generation)
and agent applications built with LanceDB.
| Project | Description |
| :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Contextual RAG**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Contextual-RAG) | Improves retrieval by combatting the "lost in the middle" problem. This technique uses an LLM to generate succinct context for each document chunk, then prepends that context to the chunk before embedding, leading to more accurate retrieval. |
| **NVIDIA RAG Blueprint with LanceDB**
[Read the tutorial](/tutorials/agents/nvidia-rag-blueprint/)
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb) | Shows how to use LanceDB as the retrieval layer for NVIDIA RAG Blueprint with a Docker-first, retrieval-only integration path that includes hybrid search and pluggable rerankers. |
| **Matryoshka Embeddings**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/RAG-with_MatryoshkaEmbed-Llamaindex) | Demonstrates a RAG pipeline using Matryoshka Embeddings with LanceDB and LlamaIndex. This method allows for efficient storage and retrieval of nested, variable-sized embeddings. |
| **HyDE (Hypothetical Document Embeddings)**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Advance-RAG-with-HyDE) | An advanced RAG technique that uses an LLM to generate a "hypothetical" document in response to a query. This hypothetical document is then used to retrieve actual, similar documents, improving relevance. |
| **Late Chunking**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Advanced_RAG_Late_Chunking) | An advanced RAG method where documents are retrieved first, and then chunking is performed on the retrieved documents just before synthesis. This helps maintain context that might be lost with pre-chunking. |
| **Agentic RAG**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/Agentic_RAG) | This tutorial demonstrates how to build a RAG system where multiple AI agents collaborate to retrieve information and generate answers, leading to more robust and intelligent applications. |
# Multimodal Agent
Source: https://docs.lancedb.com/tutorials/agents/multimodal-agent/index
Build an AI agent that understands both text and images to help users find recipes using LanceDB and PydanticAI
[](https://colab.research.google.com/drive/1pxavAGoXa-KSh_4HxNpvP2AjHPcIRpbq?usp=sharing)
Ever wanted to combine the power of text and images in a single AI agent? In this tutorial,
you'll build an agent that can understand both text and images to help users discover recipes that are relevant to them. The approach shown combines LanceDB's multimodal capabilities with [Pydantic AI](https://ai.pydantic.dev/) for the agentic workflow.
## Key Technologies
* **LanceDB**: Embedded retrieval library and multimodal lakehouse for efficient storage and retrieval
* **PydanticAI**: Modern AI agent framework with type safety
* **Sentence Transformers**: Text embeddings for semantic search
* **CLIP**: Vision-language model for image understanding
* **Streamlit**: Interactive web application framework
## Tutorial Overview
### Option 1: Notebook
The notebook shows how to work through the steps and prepare a small sample recipe dataset, generate both text and image
embeddings, store everything efficiently in LanceDB, and then build a PydanticAI agent with custom tools to
query it. You'll finish by testing the agent against a few example questions to see the full multimodal flow
end to end.
This simple tutorial provides a step-by-step workflow with a small demo dataset of 4 examples.
No local setup required - just click and start learning about multimodal agents.
[](https://colab.research.google.com/drive/1pxavAGoXa-KSh_4HxNpvP2AjHPcIRpbq?usp=sharing)
### Option 2: Demo Application (Local Setup)
The demo application is the full codebase: you'll download and process a real recipe dataset with thousands
of items, run a Streamlit chat interface that supports image upload, and follow a structure that includes
production-minded touches like error handling, logging, and monitoring. Everything you need to deploy is
included.
Download the files for the full demo application here.
### Dataset Information
* **Source**: [Kaggle Recipe Dataset](https://www.kaggle.com/datasets/pes12017000148/food-ingredients-and-recipe-dataset-with-images)
* **Size**: Thousands of recipes with images
* **Format**: CSV file with recipe data and image references
### Setup
```bash bash icon="code" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# 1. Extract the downloaded files to a folder
# 2. Navigate to the folder in terminal
cd multimodal-recipe-agent
# 3. Install dependencies with uv
uv sync
# 4. Download the Kaggle dataset
# Visit: https://www.kaggle.com/datasets/pes12017000148/food-ingredients-and-recipe-dataset-with-images
# Extract recipes.csv to the data/ folder
# 5. Import the dataset
uv run python import.py
# 6. Run the complete Streamlit chat application
uv run streamlit run app.py
```
# NVIDIA RAG Blueprint with LanceDB
Source: https://docs.lancedb.com/tutorials/agents/nvidia-rag-blueprint/index
Use LanceDB as the retrieval layer for NVIDIA RAG Blueprint with a Docker-first, retrieval-only reference integration.
## What this tutorial shows
If you are using [NVIDIA RAG Blueprints](https://build.nvidia.com/blueprints) and want to evaluate LanceDB in that stack, this tutorial gives you a concrete starting point. It shows how to use LanceDB as the retrieval layer for a Docker-based NVIDIA RAG deployment with a small, script-driven reference integration where LanceDB OSS is embedded directly in the NVIDIA containers, the collection is prepared ahead of time, and the RAG server retrieves from it for search and generation. The example is intentionally retrieval-only, but it also includes hybrid search and reranker selection so you can see how LanceDB fits into a realistic NVIDIA retrieval workflow.
The runnable example for this tutorial lives in the
[VectorDB recipes repository](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb).
## How NVIDIA organizes vector databases
NVIDIA's [RAG Blueprint documentation](https://docs.nvidia.com/rag/latest/readme.html) effectively describes three different patterns for vector database support.
1. There are built-in backends such as Milvus, where NVIDIA already owns both ingestion and
retrieval.
2. There are built-in alternatives such as Elasticsearch, where NVIDIA still owns
the end-to-end flow but switches the backend through configuration.
3. Then, there is the custom vector database path, where you implement a `VDBRag` backend yourself and register it in NVIDIA's
factory.
The LanceDB example shown below fits into the third category. More specifically, it follows NVIDIA's
**retrieval-only** custom backend path: the data is prepared in LanceDB ahead of time, and NVIDIA
RAG Blueprint is then pointed at that existing collection for search and generation. It does not
yet teach NVIDIA's ingestor how to write new documents into LanceDB automatically.
## Deployment model
This reference integration uses **LanceDB OSS as an embedded retrieval library**, not as a separate
database service. In practice, `APP_VECTORSTORE_NAME` is set to `lancedb`, `APP_VECTORSTORE_URL`
points to a local filesystem path inside the NVIDIA containers, the LanceDB collection is prepared
ahead of time, and the NVIDIA RAG server loads the LanceDB adapter to retrieve directly from that
local dataset.
## What the recipe contains
The recipe at
[`examples/nvidia-rag-blueprint-lancedb`](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb)
is organized around a small number of practical pieces. The data-prep script builds a demo
LanceDB collection from scratch, generates embeddings through the LanceDB embedding registry, and
creates a full-text index so hybrid retrieval works immediately. The adapter file shows the
retrieval-only integration point for NVIDIA RAG Blueprint, while the Docker override and NVIDIA
change guide show the minimal configuration and source changes needed to run the example against
NVIDIA's containers.
## End-to-end flow
### 1. Prepare the LanceDB collection
From the [recipe directory](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb):
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
uv sync
uv run prepare_lancedb.py --embedder demo-keyword --reranker mrr
```
That script creates:
* a local LanceDB dataset under `data/`
* a collection named `nvidia_blueprint_demo`
* automatic embeddings generated at ingest time
* an FTS index for hybrid search
The default embedder is an offline demo embedder so the example stays easy to run. If you want a
more realistic setup, the same script can switch to a sentence-transformers embedder.
### 2. Patch the NVIDIA blueprint
Follow the instructions in the recipe's
[`nvidia_blueprint_changes.md`](https://github.com/lancedb/vectordb-recipes/tree/main/examples/nvidia-rag-blueprint-lancedb/nvidia_blueprint_changes.md).
The essential changes are:
* add LanceDB dependencies to the NVIDIA environment
* copy `lancedb_vdb.py` into the NVIDIA source tree
* register the `lancedb` branch in NVIDIA's VDB factory
NVIDIA's [RAG blueprint documentation](https://docs.nvidia.com/rag/latest/readme.html) and custom-VDB guide provide useful background if you want more context before applying the LanceDB-specific changes.
### 3. Start the Docker deployment
Set the absolute path to the recipe directory:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
export LANCEDB_RECIPE_DIR=/absolute/path/to/vectordb-recipes/examples/nvidia-rag-blueprint-lancedb
```
Then from the NVIDIA repo root:
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
docker compose \
-f deploy/compose/docker-compose-rag-server.yaml \
-f "$LANCEDB_RECIPE_DIR"/docker-compose.override.yml \
up -d --build
docker compose \
-f deploy/compose/docker-compose-ingestor-server.yaml \
-f "$LANCEDB_RECIPE_DIR"/docker-compose.override.yml \
up -d --build
```
The key environment values are:
* `APP_VECTORSTORE_NAME=lancedb`
* `APP_VECTORSTORE_URL=/opt/lancedb-recipe/data`
* `COLLECTION_NAME=nvidia_blueprint_demo`
* `APP_VECTORSTORE_SEARCHTYPE=hybrid`
* `LANCEDB_RERANKER=mrr`
## Verifying the integration
### Search
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl -X POST http://localhost:8081/v1/search \
-H 'Content-Type: application/json' \
-d '{
"query": "How do I replace Milvus in the NVIDIA RAG blueprint with LanceDB?",
"use_knowledge_base": true,
"collection_names": ["nvidia_blueprint_demo"],
"vdb_top_k": 3,
"reranker_top_k": 0
}'
```
### Generate
```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
curl -N -X POST http://localhost:8081/v1/generate \
-H 'Content-Type: application/json' \
-d '{
"messages": [{"role":"user","content":"Summarize the LanceDB integration approach."}],
"use_knowledge_base": true,
"collection_names": ["nvidia_blueprint_demo"],
"vdb_top_k": 3,
"reranker_top_k": 0
}'
```
## Hybrid retrieval and rerankers
This example is meant to prove more than a trivial vector lookup.
* LanceDB hybrid retrieval combines vector search with full-text search
* the recipe creates the FTS index as part of dataset prep
* the adapter supports `RRFReranker`, `MRRReranker`, and `CrossEncoderReranker`
* the default example uses `MRRReranker`, not a plain weighted linear combination
That matters for NVIDIA partner workloads because product names, storage platforms, and technical
jargon often need exact lexical matching as well as semantic retrieval.
## How this can be extended
The current example follows NVIDIA's **custom retrieval-only backend** path. In practice, that
means the LanceDB collection is created ahead of time and NVIDIA RAG Blueprint is then pointed at
that existing collection for search and generation. The sample data in `prepare_lancedb.py` exists
only to make that flow runnable end to end: it creates a small local collection, inserts a few
documents, generates embeddings, and builds an FTS index so the NVIDIA side has something real to
query.
A fuller integration is possible. NVIDIA's custom `VDBRag` interface also supports the pattern used
by built-in backends such as Milvus and Elasticsearch, where NVIDIA owns both ingestion and
retrieval. To make LanceDB work that way, a complete LanceDB backend would need to implement the
ingestion methods NVIDIA documents, especially `create_collection` and `write_to_index`, along with
the retrieval and collection-management methods expected by the rest of the stack.
The open work is in defining how NVIDIA's ingestor should write
records into LanceDB, how that storage is shared between the ingestor and the RAG server, and how
document and collection metadata should be exposed so the broader NVIDIA APIs behave correctly.
Until those pieces exist, this example should be read as: prepare LanceDB first, then let NVIDIA
retrieve from it.
# Time-Travel RAG with versioned data
Source: https://docs.lancedb.com/tutorials/agents/time-travel-rag/index
Learn how to build production-ready RAG systems with LanceDB's time-travel capabilities for regulatory compliance and audit trails.
All the scripts and code for this tutorial are available in the
[vectorDB recipes](https://github.com/lancedb/vectordb-recipes/tree/main/examples/time-travel-rag) repository.
## Use case: Financial services regulatory knowledge base
Imagine you're a major investment bank. Your team is tasked with building a critical Retrieval-Augmented Generation (RAG) system. This system must provide instant, accurate answers to compliance officers about ever-changing financial regulations. A wrong or out-of-date answer isn't just an inconvenience—it could lead to multi-million dollar fines, reputational damage, and regulatory audits.
Your knowledge base is a living entity, constantly evolving with:
* Daily regulatory updates from government bodies.
* New internal policy documents and interpretations.
* A/B testing of different embedding models and text chunking strategies to improve accuracy.
This dynamic environment creates a series of high-stakes challenges that traditional
vector databases are ill-equipped to handle.
## Pain points solved by LanceDB
1. "Our RAG gave different answers yesterday versus today. Which version was used in the official compliance report?" Without versioning, you can't prove what the AI knew at a specific point in time, making audits impossible.
2. "The new embedding model we deployed corrupted half the vectors. Can we instantly roll back our 10TB dataset?" With traditional systems, a rollback means a painful, hours-long (or days-long) process of re-indexing from a backup, leading to significant downtime.
3. "Regulators want to audit an AI-assisted decision from three months ago. How can we prove what data the model had access to at that exact moment?" Reproducibility is key for compliance. You must be able to reconstruct the exact state of the knowledge base for any historical query.
4. "We need to A/B test a new chunking strategy, but we can't disrupt the production system or duplicate the entire dataset." Experimentation is vital for improvement, but it can't come at the cost of production stability or a massive infrastructure bill.
LanceDB's [zero-cost data evolution](/tables/schema) and [time-travel capabilities](https://docs.lancedb.com/tables/versioning) directly address these critical enterprise pain points, providing the foundation for a reliable, auditable, and production-ready RAG system.
## Dataset: The U.S. Federal Register
To make this use case realistic, we'll use a perfect real-world dataset: The U.S. Federal Register, the official daily journal of the United States Government.
It contains all new rules, proposed rules, and notices from federal agencies. It is the canonical source for regulatory changes, and it's updated every business day. It even has a public API, allowing us to simulate the real-time ingestion of new documents.
An example output of the workflow defined in main.py
is shown below.
```bash main.py expandable theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
--- Initializing Database Environment ---
Removed old database at ./lancedb
Loading embedding model: all-MiniLM-L6-v2...
--- STEP 1: Initial Data Ingestion ---
Fetching 500 documents for publication date: 2024-08-19...
Successfully fetched 86 documents.
Embedding 86 documents...
Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7.12it/s]
Successfully embedded 86 documents.
Creating table 'federal_register'...
✅ Table 'federal_register' created. Version: 1, Rows: 86
--- STEP 2: Simulating Sequential Daily Updates ---
Fetching 500 documents for publication date: 2024-08-20...
Successfully fetched 102 documents.
Embedding 102 documents...
Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 10.66it/s]
Successfully embedded 102 documents.
✅ Data added to 'federal_register'. New Version: 2, Total Rows: 188
Fetching 500 documents for publication date: 2024-08-21...
Successfully fetched 114 documents.
Embedding 114 documents...
Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 10.19it/s]
Successfully embedded 114 documents.
✅ Data added to 'federal_register'. New Version: 3, Total Rows: 302
========================================================
= PART 1: AUDITING KNOWLEDGE BASE ACROSS TIME =
========================================================
Running audit for query: 'cybersecurity reporting requirements for public companies'
Attempting to open 'federal_register' and checkout version 1...
✅ Successfully checked out Version 1 of 'federal_register'. Total rows: 86
Querying table 'federal_register' (Version 1)...
--- Top Result for Version: V1 (all-MiniLM-L6-v2) ---
📄 Title: Public Company Accounting Oversight Board; Extension of Approval Periods for Proposed Rules on a Firm's System of Quality Control and Related Amendments to PCAOB Standards, Proposed Rules on Amendments Related to Aspects of Designing and Performing Audit Procedures That Involve Technology-Assisted Analysis of Information in Electronic Form, and Proposed Rules on Amendment to PCAOB Rule 3502 Governing Contributory Liability
🗓️ Date: 2024-08-19
📏 Distance: 1.1667
📝 Abstract:
[No abstract available for this document]
--------------------------------------
Attempting to open 'federal_register' and checkout version 2...
✅ Successfully checked out Version 2 of 'federal_register'. Total rows: 188
Querying table 'federal_register' (Version 2)...
--- Top Result for Version: V2 (all-MiniLM-L6-v2) ---
📄 Title: Information Collection Being Reviewed by the Federal Communications Commission Under Delegated Authority
🗓️ Date: 2024-08-20
📏 Distance: 1.1436
📝 Abstract:
As part of its continuing effort to reduce paperwork burdens, and as required by the Paperwork
Reduction Act (PRA) of 1995, the Federal Communications Commission (FCC or the Commission) invites
the general public and other Federal agencies to take this opportunity to comment on the following
information collection. Comments are requested concerning: whether the proposed collection of
information is necessary for the proper performance of the functions of the Commission, including
whether the information shall have practical utility; the accuracy of the Commission's burden
estimate; ways to enhance the quality, utility, and clarity of the information collected; ways to
minimize the burden of the collection of information on the respondents, including the use of
automated collection techniques or other forms of information technology; and ways to further reduce
the information collection burden on small business concerns with fewer than 25 employees. The FCC
may not conduct or sponsor a collection of information unless it displays a currently valid control
number. No person shall be subject to any penalty for failing to comply with a collection of
information subject to the PRA that does not display a valid Office of Management and Budget (OMB)
control number.
--------------------------------------
Attempting to open 'federal_register' and checkout version 3...
✅ Successfully checked out Version 3 of 'federal_register'. Total rows: 302
Querying table 'federal_register' (Version 3)...
--- Top Result for Version: V3 (all-MiniLM-L6-v2) ---
📄 Title: Equipment, Systems, and Network Information Security Protection
🗓️ Date: 2024-08-21
📏 Distance: 1.0942
📝 Abstract:
This proposed rulemaking would impose new design standards to address cybersecurity threats for
transport category airplanes, engines, and propellers. The intended effect of this proposed action
is to standardize the FAA's criteria for addressing cybersecurity threats, reducing certification
costs and time while maintaining the same level of safety provided by current special conditions.
--------------------------------------
✅ Date-based audit complete. Results show how knowledge evolves over time. This demonstrates LanceDB's powerful [versioning capabilities](/tutorials/tables/consistency#versioning) for maintaining audit trails.
=============================================================
= PART 2: A/B TESTING DIFFERENT EMBEDDING MODELS =
=============================================================
Loading embedding model: all-mpnet-base-v2...
Embedding 302 documents...
Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:03<00:00, 2.55it/s]
Successfully embedded 302 documents.
Creating table 'federal_register_experimental'...
✅ Table 'federal_register_experimental' created. Version: 1, Rows: 302
Comparing search results for the same data with different models:
Querying table 'federal_register' (Version 3)...
--- Top Result for Version: Latest Prod V3 (all-MiniLM-L6-v2) ---
📄 Title: Equipment, Systems, and Network Information Security Protection
🗓️ Date: 2024-08-21
📏 Distance: 1.0942
📝 Abstract:
This proposed rulemaking would impose new design standards to address cybersecurity threats for
transport category airplanes, engines, and propellers. The intended effect of this proposed action
is to standardize the FAA's criteria for addressing cybersecurity threats, reducing certification
costs and time while maintaining the same level of safety provided by current special conditions.
--------------------------------------
Querying table 'federal_register_experimental' (Version 1)...
--- Top Result for Version: Experimental (all-mpnet-base-v2) ---
📄 Title: Commission Information Collection Activities (FERC-725B); Comment Request; Extension
🗓️ Date: 2024-08-20
📏 Distance: 1.0827
📝 Abstract:
In compliance with the requirements of the Paperwork Reduction Act of 1995, the Federal Energy
Regulatory Commission (Commission or FERC) is soliciting public comment on the currently approved
information collection, FERC-725B, Mandatory Reliability Standards, Critical Infrastructure
Protection (CIP) (Update for CIP-012-1 to version CIP-012-02) Cyber Security--Communications between
Control Centers. The 60-day notice comment period ended on July 23, 2024, with no comments received.
--------------------------------------
✅ A/B test complete. Notice the difference in relevance (distance score) between models. This showcases how LanceDB enables [experimentation with different embedding models](/docs/embeddings/) without disrupting production systems.
```
# Feature Engineering
Source: https://docs.lancedb.com/tutorials/feature-engineering/index
Learn how to build features for your data in LanceDB.
| Example | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------- |
| **Feature Engineering 101**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/feature-engineering/feature-engineering-101.ipynb) | This example demonstrates how to use LanceDB's feature engineering platform to add new derived features. |
| **Materialized Views**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/feature-engineering/materialized-views.ipynb) | This example shows how to create materialized views: query results persisted as physical tables. |
## Read the docs
The relevant section of the documentation are listed below.
| Feature | Description |
| :---------------------------------------------------- | :------------------------------------------------------------- |
| [Feature Engineering](/geneva/) | Learn the fundamentals of feature engineering. |
| [Materialized Views](/geneva/jobs/materialized-views) | Learn more about creating materialized views. |
| [Contexts](/geneva/jobs/contexts) | Learn how to run your job on a Ray cluster for production use. |
# Tutorials
Source: https://docs.lancedb.com/tutorials/index
Step-by-step tutorials for building applications with LanceDB
Explore tutorials organized by use case:
| Tutorial | Description |
| :------------------------------------------------ | :-------------------------------------------------------------------------------------------- |
| [Search & advanced retrieval](/tutorials/search/) | Learn how to perform vector search and use advanced retrieval techniques. |
| [Agents](/tutorials/agents/) | Build Retrieval-Augmented Generation (RAG) applications and agents with LanceDB. |
| [Working with tables in LanceDB](/tables/) | Learn the basics of working with tables in LanceDB: creation, ingestion and schema evolution. |
## Recipes
If you're looking for ideas and hands-on code examples, we've worked on a collection of practical
projects in the repository linked below.
Check out past code examples and tutorials [here](https://github.com/lancedb/vectordb-recipes)
on GitHub.
# Search Tutorials
Source: https://docs.lancedb.com/tutorials/search/index
Learn how vector, full-text and hybrid search work in LanceDB.
The table below shows examples of applications built with LanceDB for search use cases.
| Example | Description |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Hybrid search & reranking on BEIR**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Inbuilt-Hybrid-Search) | This example demonstrates how to use LanceDB's built-in hybrid search feature, which combines the strengths of both semantic and full-text search. By using the BEIR dataset, it shows how to achieve more relevant results by searching for both the meaning of a query and the specific keywords it contains. |
| **Semantic search across videos**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/v-jepa-video-search) | Learn how to build a video search application using V-JEPA (Video Joint Embedding Predictive Architecture) and LanceDB. This example shows how to generate vector embeddings for videos and then use LanceDB to perform similarity searches, allowing you to find videos that are visually similar to a given query. |
| **Semantic result merging**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Vector-Arithmetic-with-LanceDB) | Explore the concept of vector arithmetic with LanceDB. This notebook demonstrates how you can manipulate vector embeddings to capture more complex relationships in your data. For instance, you can modify a search query by adding or subtracting vector representations of different concepts, enabling more nuanced and powerful semantic search. |
| **Reddit concept summarizer**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Reddit-summarization-and-search) | This project showcases a complete pipeline for acquiring text data from Reddit, transforming it into meaningful vector representations using embeddings, and then storing and managing those vectors in LanceDB. It demonstrates how to build applications on top of this data, such as summarization and powerful semantic search. |
| **NER-powered vector search**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/tutorials/NER-powered-Semantic-Search) | This example demonstrates how to use Named Entity Recognition (NER) to power vector search. By extracting entities (like people, places, and organizations) from text and creating vector embeddings of them, you can significantly improve the accuracy of your search results. |
| **Multi-vector search with XTR**
![Open In Colab]()
[View on GitHub](https://github.com/lancedb/vectordb-recipes/tree/main/examples/multivector_xtr) | This notebook dives into LanceDB's advanced multivector search capabilities, enhanced by the XTR (ConteXtualized Token Retriever) technique. It shows how to represent complex data with multiple vectors for more nuanced meaning and how XTR speeds up retrieval by prioritizing the most important tokens. |
| **Needle-in-a-haystack multi-vector search**
[Read the tutorial](/tutorials/search/multivector-needle-in-a-haystack/)
| This tutorial complements the XTR multi-vector example by comparing several retrieval strategies on a token-level "needle in a haystack" benchmark. It shows when full multi-vector search, pooling, and reranking help or hurt when the goal is to find an exact page rather than just a relevant document. |
## Read the docs
The relevant section of the documentation are listed below.
| Feature | Description |
| :------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| [Vector search](/search/vector-search/) | Learn the fundamentals of vector search, including how to perform similarity searches, use different distance metrics, and optimize performance. |
| [Hybrid search](/search/hybrid-search/) | Combine keyword-based search with vector search to improve retrieval accuracy and relevance. |
| [Full-text search](/search/full-text-search/) | Perform full-text search on your text data, and combine it with vector search for a powerful hybrid search experience. |
| [Reranking](/reranking/) | Refine your search results using reranking models to improve the relevance of the top-k results. |
| [Multi-vector search](/search/multivector-search/) | Use multiple vector embeddings per document to perform more nuanced and accurate searches. |
# Finding the Needle in a Haystack: Comparing Multi-vector Search Strategies
Source: https://docs.lancedb.com/tutorials/search/multivector-needle-in-a-haystack
Tutorial on token-level retrieval with LanceDB multivector search.
![]()
In the development of advanced search and retrieval systems, moving from keyword matching to semantic understanding is a critical step. However, a key distinction exists between finding a relevant document and locating a specific piece of information within that document with precision. While there are techniques that perform well for retrieving documents, most of them work by extracting summarized semantic meaning of the document.
This can be seen as these models trying to understand the "gist" of the documents. Both single-vector search and late-interaction approaches work well for these conditions with various tradeoffs involved. But there's another type of problem where the goal is not just to understand the overall topic of a document in general, but to also specifically account for the requested detail within the document. This "needle in a haystack" problem is a significant challenge, and addressing it is essential for building high-precision retrieval systems.
This guide provides a technical analysis of multivector search for high-precision information retrieval. We will examine various optimization strategies and analyze their performance. This guide should be seen as complementary to resources like [this blog post by Answer.AI](https://www.answer.ai/posts/colbert-pooling.html) on ColBERT pooling, which explains how pooling strategies can be effective for document-level retrieval. Here, we will demonstrate why those same techniques can be counterproductive when precision at an intra-document, token level is the primary objective.
To reproduce the work below, see the code [here](https://github.com/lancedb/research/tree/main/multivector-needle-haystack-bench).
## The Dataset
This task is different from benchmarks like BEIR, which focus on text-based doc retrieval, finding the most relevant documents from a large collection. Here, we want *intra-document localization*, where the goal is to find a precise piece of information within a single, dense document, in a multimodal setting.
### The Task: The Document Haystack Dataset
Our benchmark is built on the **[AmazonScience/document-haystack](https://huggingface.co/datasets/AmazonScience/document-haystack)** dataset, which contains 25 visually complex source documents (e.g., financial reports, academic papers). To create a rigorous test, our evaluation follows a per-document methodology:
* We process each of the 25 source documents independently.
* **Table Creation:** For a single source document (e.g., "AIG"), we ingest all pages from all of its page-length variants (from 5 to 200 pages long). This creates a temporary LanceDB table containing approximately 1,230 pages.
* **The Task:** We then query this table using a set of "needle" questions, where the goal is to retrieve the **exact page number** containing the answer. A successful retrieval means the correct page number is within the top K results.
* **Target metric:** We measure both retrieval accuracy (Hit\@K) and the average search latency for each query against this table.
**Dataset example**
The documents contain "text needles" like these
![]()
![]()
During evaluation, the queries processed are somewhat like this:
```
What is the secret currency in the document?
What is the secret object #3 in the document?
```
The intention of this task is to find the page which has the text needle that answers this questions
## Models and Architectures
Our testbed includes a baseline single-vector model and a family of advanced multivector models.
### Single-Vector (Bi-Encoder) Baseline: `openai/clip-vit-base-patch32`
A bi-encoder maps an entire piece of content (a query, a document page) to a *single* vector. The search process is simple: pre-compute one vector for every page, and at query time, find the page vector closest to the query vector.
* **Strength:** Speed and simplicity.
* **Weakness:** This creates an **information bottleneck**. All the nuanced details, keywords, and semantic relationships on a page must be compressed into a single, fixed-size vector. For finding a needle, this is like trying to describe a specific person's face using only one word.
### Multi-vector (Late-Interaction) Models
Multi-vector models, pioneered by ColBERT, take a different approach. Instead of one vector per page, they generate a *set of vectors* for each page—one for every token (or image patch).
* **Mechanism (MaxSim):** The search process is more sophisticated. For each token in the query, the system finds the most similar token on the page. These maximum similarity scores are then summed up to get the final relevance score. This "late-interaction" preserves fine-grained, token-level details.
* **The Models:** We used several vision-language models adapted for this architecture, including `ColPali`, `ColQwen2`, and `ColSmol`. While their underlying transformer backbones differ, they all share the ColBERT philosophy of representing documents as a bag of contextualized token embeddings.
![]()
## Different Retrieval Strategies Used
A full multivector search is powerful but computationally intensive. Here are five strategies for managing it, complete with LanceDB implementation details.
### 1. `base`: The Gold Standard (Full Multi-vector Search)
This is the pure, baseline late-interaction search. It offers the highest potential for accuracy by considering every token.
**LanceDB also integrates with ConteXtualized Token Retriever (XTR)** , an advanced retrieval model that prioritizes the most semantically important document tokens during search. This integration enhances the quality of search results by focusing on the most relevant token matches.
**LanceDB Implementation:**
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb
import pyarrow as pa
# Schema for multivector data
# Assumes embeddings are 128-dimensional
schema = pa.schema([
pa.field("page_num", pa.int32()),
pa.field("vector", pa.list_(pa.list_(pa.float32(), 128)))
])
db = lancedb.connect("./lancedb")
tbl = db.create_table("document_pages_base", schema=schema)
# Ingesting multi-token embeddings for a page
# multi_token_embeddings is a NumPy array of shape (num_tokens, 128)
tbl.add([{"page_num": 1, "vector": multi_token_embeddings.tolist()}])
# Searching with a multi-token query
# query_multi_vector is also shape (num_query_tokens, 128)
results = tbl.search(query_multi_vector).limit(5).to_list()
```
### 2. `flatten`: Mean Pooling
This strategy "flattens" the set of token vectors into a single vector by averaging them. This transforms the search into a standard, fast approximate nearest neighbor (ANN) search.
**LanceDB Implementation:**
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Schema for single-vector data
schema_flat = pa.schema([
pa.field("page_num", pa.int32()),
pa.field("vector", pa.list_(pa.float32(), 128))
])
tbl_flat = db.create_table("document_pages_flat", schema=schema_flat)
# Ingesting the mean-pooled vector
mean_vector = multi_token_embeddings.mean(axis=0)
tbl_flat.add([{"page_num": 1, "vector": mean_vector.tolist()}])
# Searching with a single averaged query vector
query_mean_vector = query_multi_vector.mean(axis=0)
results = tbl_flat.search(query_mean_vector).limit(5).to_list()
```
### 3. `max_pooling`
This is a variation of `flatten`. `max_pooling` takes the element-wise max across all token vectors instead of the mean. The implementation is identical to `flatten`, just with a different aggregation method (`.max(axis=0)`).
### 4. `flatten and multivector rerank`: The Hybrid "Optimization"
This two-stage strategy aims for the best of both worlds. First, use a fast, pooled-vector search to find a set of promising candidates. Then, run the full, accurate multivector search on *only* those candidates.
**LanceDB Implementation:**
This requires a table with two vector columns.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
# Schema with both flat and multivector columns
schema_rerank = pa.schema([
pa.field("page_num", pa.int32()),
pa.field("vector_flat", pa.list_(pa.float32(), 128)),
pa.field("vector_multi", pa.list_(pa.list_(pa.float32(), 128)))
])
tbl_rerank = db.create_table("document_pages_rerank", schema=schema_rerank)
# Ingest both vectors
tbl_rerank.add([
{
"page_num": 1,
"vector_flat": multi_token_embeddings.mean(axis=0).tolist(),
"vector_multi": multi_token_embeddings.tolist()
}
])
# --- Two-Stage Search ---
# Stage 1: Fast search on the flat vector
query_flat = query_multi_vector.mean(axis=0)
candidates = tbl_rerank.search(query_flat, vector_column_name="vector_flat") \
.limit(100) \
.with_row_id(True) \
.to_pandas()
# Stage 2: Precise multivector search on candidates
candidate_ids = tuple(candidates["_rowid"].to_list())
final_results = tbl_rerank.search(query_multi_vector, vector_column_name="vector_multi") \
.where(f"_rowid IN {candidate_ids}") \
.limit(5) \
.to_list()
```
### 5. `hierarchical token pooling`: Compressing the Haystack
This is an indexing-time strategy that aims to reduce the storage footprint and computational cost of multivector search by reducing the number of vectors per document. Instead of using every token vector, it clusters semantically similar tokens together and replaces them with a single, averaged vector.
* **Mechanism:** For each document, it computes the similarity between all token vectors, performs hierarchical clustering to group them, and then mean-pools the vectors within each cluster. This results in a smaller, more compact set of token vectors representing the document.
* **Goal:** To reduce memory and disk usage while attempting to preserve the most important semantic information, potentially offering a middle ground between the high accuracy of `base` search and the speed of pooled methods.
**LanceDB Implementation:**
The schema is identical to the `base` multivector search, but the data is pre-processed before ingestion.
```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from utils import pool_embeddings_hierarchical
import numpy as np
# Schema is the same as the base multivector schema
schema = pa.schema([
pa.field("page_num", pa.int32()),
pa.field("vector", pa.list_(pa.list_(pa.float32(), 128)))
])
tbl_hierarchical = db.create_table("document_pages_hierarchical", schema=schema)
# Pool the embeddings before ingestion
# multi_token_embeddings is a NumPy array of shape (num_tokens, 128)
pooled_embeddings = pool_embeddings_hierarchical(
multi_token_embeddings,
pool_factor=4 # Reduce vector count by a factor of 4
)
# Ingest the smaller set of multi-token embeddings
# pooled_embeddings is now shape (approx. num_tokens / 4, 128)
tbl_hierarchical.add([{"page_num": 1, "vector": pooled_embeddings.tolist()}])
# Search is identical to the base multivector search
results = tbl_hierarchical.search(query_multi_vector).limit(5).to_list()
```
## The Results
For a "needle in a haystack" task, retrieval accuracy is the primary metric of success. The benchmark results reveal a significant performance gap between the full multivector search strategy and common optimization techniques.
### Baseline Performance: Single-Vector Bi-Encoder
First, we establish a baseline using a standard single-vector bi-encoder model, `openai/clip-vit-base-patch32`. This represents a common approach to semantic search but, as the data shows, is ill-suited for this task's precision requirements.
| Model | Strategy | Hit\@1 | Hit\@5 | Hit\@20 | Avg. Latency (s) |
| :----------------------------- | :------- | :----- | :----- | :------ | :--------------- |
| `openai/clip-vit-base-patch32` | `base` | 1.6% | 4.7% | 11.8% | **0.008 s** |
With a Hit\@20 rate of just under 12%, the baseline model struggles to reliably locate the correct page. This performance level is insufficient for applications requiring high precision.
### Multi-vector Model Performance
We now examine the performance of multivector models using different strategies. The following table compares the `base` (full multivector), `flatten` (mean pooling), and `rerank` (hybrid) strategies across several late-interaction models.
| Model | Strategy | Hit\@1 | Hit\@5 | Hit\@20 | Avg. Latency (s) |
| :------------------------- | :------------------------------- | :-------- | :-------- | :-------- | :--------------- |
| `vidore/colqwen2-v1.0` | `flatten` | 1.9% | 5.5% | 11.9% | 0.010 s |
| `vidore/colqwen2-v1.0` | `flatten and multivector rerank` | 0.3% | 1.5% | 7.3% | 0.692 s |
| **`vidore/colqwen2-v1.0`** | **`hierarchical token pooling`** | **13.7%** | **60.5%** | **91.6%** | **0.144 s** |
| **`vidore/colqwen2-v1.0`** | **`base`** | **14.0%** | **65.4%** | **95.5%** | 0.668 s |
| `vidore/colpali-v1.3` | `flatten` | 1.7% | 4.5% | 9.3% | 0.008 s |
| `vidore/colpali-v1.3` | `flatten and multivector rerank` | 0.6% | 2.3% | 6.9% | 0.949 s |
| **`vidore/colpali-v1.3`** | **`hierarchical token pooling`** | **10.8%** | **41.7%** | **64.8%** | **0.189 s** |
| **`vidore/colpali-v1.3`** | **`base`** | **11.3%** | **42.3%** | **65.6%** | 0.936 s |
| `vidore/colSmol-256M` | `flatten` | 1.6% | 4.7% | 10.5% | 0.008 s |
| `vidore/colSmol-256M` | `flatten and multivector rerank` | 0.3% | 1.6% | 7.0% | 0.853 s |
| **`vidore/colSmol-256M`** | **`base`** | **14.4%** | **64.0%** | **91.7%** | 0.848 s |
The data shows a consistent pattern: the `base` strategy outperforms all other techniques. The flattned pooling and reranking strategies perform no better than the single-vector baseline. However, hierarchical token pooling seems like a decent alternative to base considering speed vs accuracy tradeoff. Let's look at the numbers in detail.
### In-Depth Analysis of Pooling Strategies
To further understand the failure of optimization techniques, we compared different methods for pooling token vectors into a single vector: `mean` (`flatten`), `max`.
| Model & Pooling Strategy | Hit\@1 | Hit\@5 | Hit\@20 | Avg. Latency (s) |
| :-------------------------------------- | :-------- | :-------- | :-------- | :--------------- |
| `vidore/colqwen2-v1.0` (`mean_pooling`) | 1.9% | 5.5% | 11.9% | 0.010 s |
| `vidore/colqwen2-v1.0` (`max_pooling`) | 1.4% | 4.2% | 11.2% | 0.011 s |
| **`vidore/colqwen2-v1.0` (`base`)** | **14.0%** | **65.4%** | **95.5%** | 0.668 s |
All flattened pooling methods perform poorly, confirming that the aggregation of token vectors into a single representation loses the fine-grained detail required for this task.
**Finding the Right Trade-Off**
![]()
1. **The Failure of Simple Pooling:** The `flatten` (mean pooling) and `max_pooling` strategies fail to improve upon the baseline. This is because their aggressive compression **destroys the essential localization signal**. The resulting single vector represents the *topic* of the page, not the *specific needle* on it.
2. **The Failure of `flatten and multivector rerank`:** This hybrid strategy is the *worst-performing* of all. The reason is a fundamental flaw in its design for this task: the first stage uses a simple pooled vector to retrieve candidates. Since this pooling eliminates the localization signal, the initial candidate set is effectively random.
3. **`hierarchical token pooling`:** By clustering and pooling tokens at indexing time, it reduces the number of vectors per page (in our case, by a factor of 4). This intelligently compresses the data, while preserving enough token-level detail in multivector setting. It achieves a **Hit\@20 of 91.6%**, only slightly behind the `base` strategy's 95.5%, but is significantly faster.
4. **`base` multivector Search:** The vanilla, un-optimized `base` multivector search remains the most accurate strategy. Preserving every token vector provides the highest guarantee of finding the needle, but this comes at the highest computational cost.
### Latency:
![]()
The "optimizations" are not all created equal. While simple pooling is fast, its inaccuracy makes it unusable. Hierarchical pooling, however, offers a compelling balance of speed and accuracy.
| Strategy (on `vidore/colqwen2-v1.0`) | Avg. Search Latency (s) | Hit\@20 Accuracy |
| :-------------------------------------------------------- | :---------------------- | :--------------- |
| `flatten` (Fast but Ineffective) | **0.010 s** | 11.9% |
| `flatten and multivector rerank` (Slower and Ineffective) | 0.692 s | 7.3% |
| **`hierarchical token pooling` (Accurate & Fast)** | **0.144 s** | **91.6%** |
| `base` (Most Accurate) | 0.668 s | **95.5%** |
| *Latency reported is as seen on NVIDIA H100 GPUs* | | |
## Practical Considerations
The accuracy of `base` multivector search is impressive, but its computational intensity has historically limited its use. `hierarchical token pooling` as a viable strategy creates a new, practical sweet spot on the accuracy-latency curve, making high-precision search accessible for a wider range of applications.
### Search Latency and Computational Complexity
As the benchmark data shows, the search latency for `base` multivector search is orders of magnitude higher than for single-vector (or pooled-vector) search. It's important to note that the reported \~670ms latency is an average from per-document evaluations. In this benchmark, each of the 25 documents is processed independently. All pages from a single document's variants (ranging from 5 to 200 pages) are ingested into a temporary table, resulting in a table size of approximately **1,230 rows (pages)** per evaluation. The search is performed on this table, and then the table is discarded. This highlights a significant performance cost even on a relatively small, per-document scale. This stems from a fundamental difference in computational complexity:
* **Modern ANN Search (for single vectors):** Algorithms like HNSW (Hierarchical Navigable Small World) provide sub-linear search times, often close to `O(log N)`, where `N` is the number of items in the index. This allows them to scale to billions of vectors with millisecond-level latency.
* **Late-Interaction Search (Multi-vector):** The search process is far more intensive. For each query, it must compute similarity scores between query tokens and the tokens of many candidate documents. The complexity is closer to `O(M * Q * D)`, where `M` is the number of candidate documents to score, `Q` is the number of query tokens, and `D` is the average number of tokens per document. `Hierarchical token pooling` directly attacks this problem by reducing `D`, leading to a significant reduction in search latency.
### When to Use Multi-Vector Search
Given these constraints, the choice of strategy depends on the specific requirements of the application.
* **For Maximum Precision (`base`):** In domains where the cost of missing the needle is extremely high, the full `base` search is the most reliable option.
* **For a Balance of Precision and Performance (`hierarchical token pooling`):** This is the ideal choice for many applications. It makes high-precision search practical for larger datasets and more interactive use cases where the sub-second latency of the `base` search may be too high. It significantly lowers the barrier to entry for adopting multivector search. It should still not be seen as a drop-in replacement for ANN, as it still requires more computational resources than single-vector search.
* **For General-Purpose Document Retrieval (`flatten` / single-vector):** For large-scale retrieval where understanding the "gist" is sufficient or where in cases where large-context text-based models suffice, single-vector search remains the most practical and scalable solution.
## Appendix: Full Benchmark Results
The full benchmark results are shown below.
```text theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
| name | _runtime | _step | _timestamp | _wandb | avg_inference_latency | avg_search_latency | hit_rates | model_name | strategy |
|:--------------------------------------------|-----------:|--------:|--------------:|:-------------------|------------------------:|---------------------:|:------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------|:---------------------|
| vidore/colqwen2-v0.1_base | 13410 | 0 | 1.75873e+09 | {'runtime': 13410} | 0.0418646 | 0.751151 | {'1': 0.1355151515151515, '10': 0.888, '20': 0.9597575757575758, '3': 0.3936969696969697, '5': 0.6349090909090909} | vidore/colqwen2-v0.1 | base |
| vidore/colqwen2-v1.0_rerank | 13008 | 0 | 1.75873e+09 | {'runtime': 13008} | 0.0426252 | 0.692482 | {'1': 0.003393939393939394, '10': 0.03296969696969697, '20': 0.07296969696969698, '3': 0.010666666666666666, '5': 0.015272727272727271} | vidore/colqwen2-v1.0 | rerank |
| vidore/colpali-v1.3_flatten | 2998 | 0 | 1.75872e+09 | {'runtime': 2998} | 0.0296511 | 0.00833224 | {'1': 0.017454545454545455, '10': 0.06448484848484848, '20': 0.09333333333333334, '3': 0.034666666666666665, '5': 0.04509090909090909} | vidore/colpali-v1.3 | flatten |
| vidore/colqwen2-v0.1_rerank | 13512 | 0 | 1.75873e+09 | {'runtime': 13512} | 0.0418855 | 0.662372 | {'1': 0.002909090909090909, '10': 0.026424242424242423, '20': 0.05987878787878788, '3': 0.0075151515151515155, '5': 0.014545454545454544} | vidore/colqwen2-v0.1 | rerank |
| vidore/colqwen2-v1.0_base | 12933 | 0 | 1.75873e+09 | {'runtime': 12933} | 0.0416839 | 0.667898 | {'1': 0.14012121212121212, '10': 0.8846060606060606, '20': 0.9553939393939394, '3': 0.4111515151515152, '5': 0.6538181818181819} | vidore/colqwen2-v1.0 | base |
| vidore/colpali-v1.3_rerank | 12090 | 0 | 1.75873e+09 | {'runtime': 12090} | 0.0310783 | 0.949047 | {'1': 0.006060606060606061, '10': 0.03878787878787879, '20': 0.06933333333333333, '3': 0.014545454545454544, '5': 0.022787878787878788} | vidore/colpali-v1.3 | rerank |
| vidore/colqwen2-v1.0_flatten | 4846 | 0 | 1.75874e+09 | {'runtime': 4846} | 0.0504031 | 0.010443 | {'1': 0.018666666666666668, '10': 0.07854545454545454, '20': 0.11903030303030304, '3': 0.041212121212121214, '5': 0.055030303030303034} | vidore/colqwen2-v1.0 | flatten |
| vidore/colqwen2-v0.1_base | 10838 | 0 | 1.75874e+09 | {'runtime': 10838} | 0.04627 | 0.692836 | {'1': 0.13575757575757577, '10': 0.8870303030303031, '20': 0.96, '3': 0.3941818181818182, '5': 0.6351515151515151} | vidore/colqwen2-v0.1 | base |
| vidore/colqwen2-v0.1_flatten | 4828 | 0 | 1.75874e+09 | {'runtime': 4828} | 0.0486335 | 0.0103028 | {'1': 0.018424242424242423, '10': 0.07224242424242425, '20': 0.10521212121212122, '3': 0.03903030303030303, '5': 0.05212121212121213} | vidore/colqwen2-v0.1 | flatten |
| vidore/colpali-v1.3_base | 10253 | 0 | 1.75874e+09 | {'runtime': 10253} | 0.0312334 | 0.93611 | {'1': 0.11272727272727272, '10': 0.551030303030303, '20': 0.6555151515151515, '3': 0.29987878787878786, '5': 0.4232727272727273} | vidore/colpali-v1.3 | base |
| vidore/colqwen2-v0.1_flatten | 4745 | 0 | 1.75874e+09 | {'runtime': 4745} | 0.0472825 | 0.00990508 | {'1': 0.018424242424242423, '10': 0.07296969696969698, '20': 0.10496969696969696, '3': 0.03951515151515152, '5': 0.05236363636363636} | vidore/colqwen2-v0.1 | flatten |
| vidore/colqwen2.5-v0.2_base | 17218 | 0 | 1.75875e+09 | {'runtime': 17218} | 0.0540356 | 0.694855 | {'1': 0.11903030303030304, '10': 0.7127272727272728, '20': 0.8366060606060606, '3': 0.336, '5': 0.5258181818181819} | vidore/colqwen2.5-v0.2 | base |
| vidore/colqwen2.5-v0.2_rerank | 16859 | 0 | 1.75875e+09 | {'runtime': 16859} | 0.0518383 | 0.71693 | {'1': 0.0026666666666666666, '10': 0.025212121212121213, '20': 0.060848484848484846, '3': 0.006787878787878788, '5': 0.01187878787878788} | vidore/colqwen2.5-v0.2 | rerank |
| vidore/colqwen2-v0.1_rerank | 9484 | 0 | 1.75875e+09 | {'runtime': 9484} | 0.0445599 | 0.691609 | {'1': 0.005333333333333333, '10': 0.030545454545454542, '20': 0.064, '3': 0.010666666666666666, '5': 0.017696969696969697} | vidore/colqwen2-v0.1 | rerank |
| vidore/colSmol-256M_flatten | 6822 | 0 | 1.75875e+09 | {'runtime': 6822} | 0.0404544 | 0.00822329 | {'1': 0.01575757575757576, '10': 0.06836363636363636, '20': 0.10496969696969696, '3': 0.03442424242424243, '5': 0.04678787878787879} | vidore/colSmol-256M | flatten |
| vidore/colSmol-500M_base | 12681 | 0 | 1.75875e+09 | {'runtime': 12681} | 0.0408026 | 0.850902 | {'1': 0.136, '10': 0.8029090909090909, '20': 0.8993939393939394, '3': 0.3806060606060606, '5': 0.5975757575757575} | vidore/colSmol-500M | base |
| vidore/colSmol-500M_rerank | 13066 | 0 | 1.75876e+09 | {'runtime': 13066} | 0.0440974 | 0.927632 | {'1': 0.003636363636363637, '10': 0.028606060606060607, '20': 0.07054545454545455, '3': 0.00896969696969697, '5': 0.015030303030303033} | vidore/colSmol-500M | rerank |
| vidore/colSmol-256M_rerank | 12646 | 0 | 1.75876e+09 | {'runtime': 12646} | 0.0391553 | 0.853279 | {'1': 0.003393939393939394, '10': 0.02909090909090909, '20': 0.07006060606060606, '3': 0.008727272727272728, '5': 0.015515151515151517} | vidore/colSmol-256M | rerank |
| vidore/colqwen2.5-v0.2_flatten | 6348 | 0 | 1.75876e+09 | {'runtime': 6348} | 0.0509971 | 0.00772653 | {'1': 0.017696969696969697, '10': 0.06545454545454546, '20': 0.09284848484848485, '3': 0.03515151515151515, '5': 0.045575757575757575} | vidore/colqwen2.5-v0.2 | flatten |
| vidore/colSmol-256M_base | 11554 | 0 | 1.75876e+09 | {'runtime': 11554} | 0.0366467 | 0.848463 | {'1': 0.1435151515151515, '10': 0.8426666666666667, '20': 0.9173333333333332, '3': 0.40824242424242424, '5': 0.6404848484848484} | vidore/colSmol-256M | base |
| vidore/colSmol-500M_flatten | 6395 | 0 | 1.75876e+09 | {'runtime': 6395} | 0.0384664 | 0.00716238 | {'1': 0.018424242424242423, '10': 0.07345454545454545, '20': 0.11393939393939394, '3': 0.03903030303030303, '5': 0.05090909090909091} | vidore/colSmol-500M | flatten |
| openai/clip-vit-base-patch32_base | 815 | 0 | 1.75876e+09 | {'runtime': 815} | 0.00533487 | 0.00794629 | {'1': 0.016, '10': 0.07636363636363637, '20': 0.11757575757575756, '3': 0.03296969696969697, '5': 0.04703030303030303} | openai/clip-vit-base-patch32 | base |
| vidore/colqwen2-v0.1_max_pooling | 8758 | 0 | 1.7595e+09 | {'runtime': 8758} | 0.0985753 | 0.0106716 | {'1': 0.015515151515151517, '10': 0.07296969696969698, '20': 0.11248484848484848, '3': 0.032484848484848484, '5': 0.04703030303030303} | vidore/colqwen2-v0.1 | max_pooling |
| vidore/colpali-v1.3_max_pooling | 4728 | 0 | 1.75949e+09 | {'runtime': 4728} | 0.0718573 | 0.0103053 | {'1': 0.011393939393939394, '10': 0.05672727272727273, '20': 0.08872727272727272, '3': 0.02666666666666667, '5': 0.03709090909090909} | vidore/colpali-v1.3 | max_pooling |
| vidore/colqwen2-v1.0_max_pooling | 8760 | 0 | 1.7595e+09 | {'runtime': 8760} | 0.0981102 | 0.0106316 | {'1': 0.013575757575757576, '10': 0.06933333333333333, '20': 0.112, '3': 0.02812121212121212, '5': 0.041939393939393936} | vidore/colqwen2-v1.0 | max_pooling |
| vidore/colqwen2-v0.1_max_pooling | 7696 | 0 | 1.7595e+09 | {'runtime': 7696} | 0.105203 | 0.011907 | {'1': 0.016242424242424242, '10': 0.07321212121212121, '20': 0.1132121212121212, '3': 0.03442424242424243, '5': 0.04945454545454545} | vidore/colqwen2-v0.1 | max_pooling |
| vidore/colSmol-256M_max_pooling | 13982 | 0 | 1.75951e+09 | {'runtime': 13982} | 0.0999708 | 0.0121211 | {'1': 0.00993939393939394, '10': 0.05818181818181818, '20': 0.09187878787878788, '3': 0.025212121212121213, '5': 0.037575757575757575} | vidore/colSmol-256M | max_pooling |
| vidore/colSmol-500M_max_pooling | 14191 | 0 | 1.75951e+09 | {'runtime': 14191} | 0.111858 | 0.0121703 | {'1': 0.012363636363636365, '10': 0.07539393939393939, '20': 0.13187878787878787, '3': 0.02787878787878788, '5': 0.0416969696969697} | vidore/colSmol-500M | max_pooling |
| vidore/colqwen2-v0.1_hierarchical_pooling | 4507 | 0 | 1.7599e+09 | {'runtime': 4507} | 0.0320023 | 0.133653 | {'1': 0.1296969696969697, '10': 0.8504242424242424, '20': 0.9343030303030304, '3': 0.37527272727272726, '5': 0.6041212121212122} | vidore/colqwen2-v0.1 | hierarchical_pooling |
| vidore/colpali-v1.3_hierarchical_pooling | 5816 | 0 | 1.7599e+09 | {'runtime': 5816} | 0.0217625 | 0.188727 | {'1': 0.10763636363636364, '10': 0.5372121212121213, '20': 0.6482424242424243, '3': 0.29333333333333333, '5': 0.4167272727272727} | vidore/colpali-v1.3 | hierarchical_pooling |
| vidore/colqwen2.5-v0.2_hierarchical_pooling | 8430 | 0 | 1.75991e+09 | {'runtime': 8430} | 0.043073 | 0.141276 | {'1': 0.11103030303030303, '10': 0.6892121212121212, '20': 0.8203636363636364, '3': 0.3185454545454545, '5': 0.4989090909090909} | vidore/colqwen2.5-v0.2 | hierarchical_pooling |
| vidore/colqwen2-v1.0_hierarchical_pooling | 5685 | 0 | 1.75991e+09 | {'runtime': 5685} | 0.0343824 | 0.144062 | {'1': 0.13745454545454547, '10': 0.822060606060606, '20': 0.9156363636363636, '3': 0.3856969696969697, '5': 0.6050909090909091} | vidore/colqwen2-v1.0 | hierarchical_pooling |
```