Vector Indexing

Applies to: Embedded ✅, Enterprise ✅ Vector indexes are robust tools in facilitating fast searches across large numeric datasets. LanceDB implements ANN (Approximate Nearest-Neighbor) queries with several techniques that provide benefits across a variety of use cases.

Do you need one?

Not always. LanceDB’s disk-based brute-force kNN search is fast even without an index: computing 100K pairs of 1000-dimension vectors takes under 20ms in benchmarks. For datasets around 100K rows or fewer, or applications that can accept ~100ms latency, a vector index is usually unnecessary overhead. Build one once a table passes roughly a million rows, dimensionality is high, or query latency matters more than that. See bypass_vector_index to run an exact search side-by-side and measure whether an index is worth it for your data.

Choosing the Right Index

LanceDB vector indexes are combined with several quantization techniques to admit efficient storage. The following table lists provided quantized vector indexes and their common use cases. You can specify index type manually in Lance with index_type.

If your priority is…	Use this index	Why	Approx. compression ratio	Python config class
Maximum compression	`IVF_RQ`	IVF indexing with RaBitQ quantization	Around `1/32` of raw size with `num_bits=1`	`IvfRq`
Higher accuracy at small dimensions (`dimension <= 256`)	`IVF_PQ`	IVF indexing with product quantization	Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`)	`IvfPq`
Best recall/latency trade-off	`IVF_HNSW_SQ`	IVF-HNSW indexing with scalar quantization	Typically a little larger than `1/4` of raw size	`IvfHnswSq`
Highest recall / no quantization	`IVF_HNSW_FLAT`	IVF-HNSW indexing with no quantization; embedded library only	Around raw vector size plus HNSW graph overhead	`IvfHnswFlat`

If your vector search frequently includes metadata filters (where(...)), use IVF_RQ or IVF_PQ. In filtered workloads, HNSW-backed IVF indexes such as IVF_HNSW_FLAT and IVF_HNSW_SQ can show higher latency variance.

Understanding Vector Indexes

LanceDB offers two vector indexes, which can be created on any numeric Lance dataset: Inverted File (IVF) and Hierarchical Navigable Small World (HNSW).

IVF

The Inverted File Index (IVF) accelerates ANN searches by drastically reducing the search space. The index consists of a small set of centroids corresponding to an approximate solution to the

k

-means clustering problem. Each vector remembers its nearest centroid, and each centroid remembers its associated set of vectors, called its partition.

IVF vector-space partitioning — Example partition of a geometric space, with colored marks denoting centroids.

At query time, we can compare the queried vector to the smaller set of centroids (as opposed to the entire dataset) for a closest match, then run a brute-force comparison against its resulting partition. This technique quickly prunes a large search space, giving an approximate ANN result. However, observe that a queried vector may lie near the boundary of

2

or more partitions; thus, the true nearest neighbors are scattered across several different partitions. To address this, LanceDB exposes the nprobes parameter, which specifies the number of partitions searched. A higher probe budget searches more partitions, which can improve recall at higher CPU and I/O cost.

HNSW

Hierarchical Navigable Small World (HNSW) constructs a layered graph hierarchy on the vector set, with edges representing distances. We can visualize an HNSW index as a vertical stack of graphs, with the top layer having very few edges and each other layer having a multiplicative factor more edges than the layer above it. High layers of a HNSW hierarchy represent sparse, higher-distance networks, and lower layers represent finer, lower-distance networks. To query a vector

q

, the index proceeds iteratively through layers, first finding

q

‘s nearest neighbor in the graph, then proceeding recursively through the induced subhierarchy until the lowest layer is reached.

HNSW layered graph hierarchy — HNSW hierarchy: sparse network at the top layer, fine network at the bottom.

To visualize this process, imagine that you must drive your car from San Francisco to a specific house in Boston. Initially, you must first drive thousands of miles on the interstate freeway I-90 E. Eventually, you merge onto the Massachusetts Turnpike, the center of the greater Boston highway system. From there, you use a series of increasingly smaller, narrower roads within the city (Charles River Bridge, St. Paul St, Thatcher St) before finally reaching the house. A key observation is that you must initially travel far distances through long-distance road networks (the interstate freeway system), before proceeding to finer and finer road networks (greater Boston highway system, central Brookline neighborhood connectors) before finally reaching your destination. This iterative series of road networks mimics the layered graph traversals performed by a HNSW query.

IVF + HNSWIn LanceDB, HNSW is not exposed as a top-level vector index. Instead, it’s available as a substructure which further indexes the selected vectors inside each IVF partition. This combines the scalability of IVF with the high recall of HNSW. LanceDB supports IVF-HNSW-based quantized indexes IVF_HNSW_FLAT, IVF_HNSW_PQ, and IVF_HNSW_SQ; a deployment builds only IVF_HNSW_SQ.

Using Vector Indexes

Learn how to configure, build, and search LanceDB vector indexes, including build and search time parameters, asynchronous objects, and several examples.

Configuration

Build-time Parameters

Parameter	Description
`distance_type`	Default is `l2`, others available are `cosine` and `dot`.
`num_partitions`	The number of IVF partitions constructed (corresponds to the $k$ in $k$ -means clustering). This takes precedence over `target_partition_size` if both are set.
`target_partition_size`	An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. `num_partitions` takes precedence over `target_partition_size` if both are set. If neither is set the number of partitions is automatically calculated based on the number of rows and the index type.
`num_sub_vectors`	Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if not a multiple of 16). Larger values produce better recall and slower search.
`max_iterations`	Maximum number of k-means training iterations, for every IVF/HNSW index type. Default `50`. Increase for larger datasets or to improve training quality.
`sample_rate`	Number of k-means training samples per partition, for every IVF/HNSW index type. Default `256`. Higher values increase both accuracy and training time.

Search-time Parameters

Parameter	Description
`limit`	Number of results to return (the `k` in `k-ANN`).
`nprobes`	Search a fixed number of partitions per index segment. Leave unset to use Auto.
`search_effort`	A value from `0` to `1` for tuning recall and latency. `0.5` uses Auto; higher values search more partitions. This is not a recall percentage.
`minimum_nprobes`	Minimum number of partitions to search per segment. Auto may search more from the start.
`maximum_nprobes`	Maximum number of partitions to search per segment. Leave unset to allow further searching when needed. A low cap can reduce recall or return fewer results than requested.
`ef`	HNSW search-time exploration factor. Start around `1.5 * k` and increase up to `10 * k` for higher recall.
`refine_factor`	Rerank candidates using full vectors to improve accuracy. A factor of `25` requests up to `25 * k` candidates. Higher values read more data; measure the recall gain and latency cost.
`distance_range(lower_bound, upper_bound)`	Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or “close-enough” matching.
`bypass_vector_index()`	Ignore the ANN index entirely and perform an exact (flat) scan. Can be used to measure ANN `recall@k`, or to query with a metric the index was not built for (e.g., a non-cosine query on a multivector column).

The distance_type can be specified at search time in addition to build time. If you do this, set it to the same value used to build the vector index. If the search metric does not match the index metric, LanceDB cannot use the index and falls back to a costly flat search. On LanceDB Cloud (legacy hosted service), the brute-force KNN safety limit can reject that search on a large table instead of executing it. An Enterprise deployment applies the same limit only when it is configured to.When the search metric is omitted, current SDKs use the index metric if an index is available. Some older SDK versions sent l2 when it was omitted, so explicitly set the matching metric or upgrade the SDK when querying a non-l2 index with an older client.

Recommended nprobes behavior by index type:

Index type	Guidance
`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`	Leave the probe settings unset, then tune `ef` first. Expect higher latency variance under filtered search.
`IVF_RQ`, `IVF_PQ`	Leave the probe settings unset and configure them only when recall is insufficient. If using a selective filter you may need to increase `maximum_nprobes` if you do not receive enough results.

Start with Auto, then measure recall and latency before changing probe settings. If you set both search_effort and nprobes, effort takes precedence. Explicit min/max bounds still apply; equal bounds fix the count. See tuning vector search for examples and guidance on choosing these controls.

For a filtered search, you can let Auto choose within a range. The maximum also limits further searching if too few matches are found:

# Let Auto choose between 10 and 50 partitions per segment, if available.
filtered = (
    table.search(query_vector)
    .minimum_nprobes(10)
    .maximum_nprobes(50)
    .where("id > 100")
    .limit(5)
    .to_arrow()
)

LanceDB also supports advanced search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints. Thresholding with distance_range:

# Only return results whose distance falls within [0.0, 0.5).
# Useful for near-duplicate detection or thresholded similarity search.
(
    table.search(np.random.random(128))
    .distance_range(lower_bound=0.0, upper_bound=0.5)
    .limit(10)
    .to_pandas()
)

Using bypass_vector_index: Use bypass_vector_index to compute an exact kNN result. Note that exact queries may be prohibitively slow on production scales.

query = np.random.random(128)
k = 10

# Ground truth: flat (exhaustive) scan, ignoring the ANN index.
truth = set(table.search(query).bypass_vector_index().limit(k).to_pandas()["id"])

# ANN results with the current nprobes setting.
ann = set(table.search(query).nprobes(20).limit(k).to_pandas()["id"])

recall_at_k = len(truth & ann) / k

Multivector indexing currently requires distance_type="cosine". Use bypass_vector_index() for non-cosine queries on a multivector column. See Multivector Search for the full rules.

Async API and Config Objects

Create vector indexes asynchronously with await table.create_index("vector", config=...). The config object admits the same build-time parameters described above. Pass an instance of a Python config class from the table in Choosing the Right Index:

import lancedb
import numpy as np
from lancedb.index import IvfPq

async def main():
    data = [
        {"id": i, "vector": np.random.random(8).astype(np.float32).tolist()}
        for i in range(512)
    ]

    db = await lancedb.connect_async("ex_lancedb")
    table = await db.create_table(
        "vector_index_async", data=data, mode="overwrite"
    )

    await table.create_index(
        "vector",
        config=IvfPq(
            distance_type="cosine",
            num_partitions=16,
            num_sub_vectors=4,
        ),
    )
    return await table.list_indices()

IVF Indexes

This example creates and queries an IVF_PQ index for a table of vectors with respect to cosine similarity. Specify vector_column_name if you have multiple vector columns or non-default names.

table_name = "vector-index-build-ivf"
table = db.open_table(table_name)
table.create_index(
    metric="cosine",
    vector_column_name="keywords_embeddings",
)
tbl = table
tbl.search(np.random.random((1536))).limit(2).nprobes(20).refine_factor(
    10
).to_pandas()

TypeScript currently doesn’t support IvfSq (IVF with Scalar Quantization).

For a vector field nested inside a struct, pass its full dotted path as vector_column_name (e.g. image.embedding). The same path is used at query time and is what list_indices() reports under columns. See Selecting the vector column for the full path syntax.

# The vector column `embedding` is nested inside the `image` struct.
# Pass its full dotted path as `vector_column_name`; the same path is used
# at query time and is what `list_indices()` reports under `columns`.
table.create_index(
    vector_column_name="image.embedding",
    num_partitions=1,
    num_sub_vectors=1,
    name="image_embedding_idx",
)

results = (
    table.search([0.0, 1.0], vector_column_name="image.embedding")
    .limit(1)
    .to_list()
)

IVF-HNSW Indexes

Beyond the general build-time parameters above, two additional parameters are specific to IVF-HNSW indexes:

Parameter	Description
`m`	The number of neighbors to select for each vector in the HNSW graph.
`ef_construction`	The number of candidates to evaluate during the construction of the HNSW graph. Start at `150`; increase for better recall, decrease for faster indexing.

Partition sizing follows the general num_partitions/target_partition_size guidance above. The snippet below builds and queries an IVF_HNSW_SQ index.

table.create_index(index_type="IVF_HNSW_SQ")
tbl = table
tbl.search(np.random.random((16))).limit(2).to_pandas()

Managing Vector Indexes

Enterprise-only In LanceDB Enterprise, vector indexes are managed automatically. The system asynchronously updates and optimizes indexes as a background process:

Automatically manages indexing parameters and storage
Can build an IVF_PQ index automatically on a vector column the deployment is configured to index
GPU-based IVF_PQ builds are available on request. Contact support to enable them for your deployment. Build time depends on dataset size, dimensions and GPU resources.

Open-Source LanceDB Embedded users can manually create vector indexes by calling table.create_index(). See the above sections for guidance on manually tuning index parameters as data changes.

In LanceDB Enterprise, create_index() returns immediately and the vector index builds asynchronously. To wait until all data is indexed, specify the wait_timeout parameter, or call wait_for_index(...) afterward. It waits for the named index to exist and for index_stats(...) to report num_unindexed_rows == 0. In the embedded library the index is built before create_index() returns.

Rows appended after an initial index build remain outside the index until refreshed manually (Embedded) or automatically (Enterprise). Normal search still checks those unindexed rows with a slower fallback path; fast_search() skips that fallback and searches only indexed rows.

Operational checksIn the embedded library, use optimize() after appends or other writes to fold new rows into existing indexes. A deployment does this in the background, and optimize() is not available there.

Check Index Status

In LanceDB Enterprise, vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status through the API or the LanceDB Enterprise UI. In the LanceDB Enterprise UI, navigate to your table page - the “Index” column reflects each column’s index status: it is blank when no index exists, shows an “in progress” label while the index is being built, and shows the index type once the build completes. To check status programmatically, use list_indices() and index_stats(). By default, the index name is formed by appending _idx to the column name (e.g., a keywords_embeddings column produces keywords_embeddings_idx). Note that list_indices() only returns information after the index is fully built. Each entry returned by list_indices() also carries detailed per-index metadata, so you can inspect an index without a follow-up index_stats() call. Node.js exposes the same fields in camelCase (num_indexed_rows → numIndexedRows):

Parameter	Description
`num_indexed_rows`, `num_unindexed_rows`	Index coverage over the table
`size_bytes`	Total size of the index files on disk
`num_segments`, `index_version`	On-disk layout and format version
`created_at`	Creation time (ms since the Unix epoch in Node.js)
`index_uuid`, `type_url`	Internal identifiers for the index segment
`index_details`	Type-specific details (e.g. IVF partition counts or quantization settings)

index_name = "keywords_embeddings_idx"
table.wait_for_index([index_name])
print(table.index_stats(index_name))

These fields are populated for LanceDB Embedded tables. In LanceDB Enterprise, list_indices() returns them except num_unindexed_rows; use index_stats() for index coverage.

Custom Index Names

The {column}_idx suffix is the default naming convention. Pass name=... to create_index() to override it. Once set, the custom name will be reflected in list_indices(), index_stats(name), and wait_for_index([name]).

# Override the default `{column}_idx` convention by passing `name=...`.
table.create_index(
    metric="cosine",
    vector_column_name="keywords_embeddings",
    name="my_custom_index",
)
table.wait_for_index(["my_custom_index"])
print(table.index_stats("my_custom_index"))

Quantization

Quantization is used in LanceDB to efficiently compress and store vector indexes. We discuss only the quantization techniques here; discussion of LanceDB vector indexes and quantized vector indexes can be found here.

Quantization Techniques

LanceDB provides

3

distinct quantization techniques: Product Quantization (PQ), RaBitQ Quantization (RQ), and Scalar Quantization (SQ). Recall that all quantizations perform lossy compression, in that they irreversibly lose some degree of precision to compactly store an index.

Product Quantization (PQ)

To visualize PQ, assume a vector dataset has

d

dimensions, with a chunk of a vector denoting a contiguous block of entries. Imagine that each vector has

m

disjoint chunks of

d/m

entries each, where the first chunk contains entries

0

through

d/m-1

, the second contains entries

d/m

through

2d/m-1

, and so on. Now, for each

i

, let

S_i

represent the set containing chunk

i

of each vector (entries

(i-1)d/m

id/m-1

). For each

S_i

independently, a small set of centroids is computed corresponding to an approximate solution to the

k

-means clustering problem. For each original vector in the dataset, every chunk is associated to its nearest centroid; thus, the vector itself is associated with the concatenation of

m

centroids. We then gather all our centroids (from all chunks) into a single lookup table, and for each vector, we store the concatenation of IDs of its corresponding centroids. At query time, we need only to compare each chunk of the queried vector with our set of centroids.

In the above example, the original vector is split into chunks (subvectors), each of which is associated to a centroid. The stored quantization code for this vector is the concatenation

2 || 1 || 4 || 3

. The original vector required

128

dimensions

\times 32

-bit floats

=4096

bits total, which has been compressed to

4

chunks

\times 8

-bit integers

= 32

bits of quantized storage.

RaBitQ Quantization (RQ)

RaBitQ is an advanced quantization technique that outperforms PQ in several ways. It needs no codebook to train, estimates distances very quickly at query-time, and can quantize each vector in (with some small overhead) as little as one bit per dimension! In practice, RaBitQ compresses a

1024

-dimensional float32 vector into just a few thousand bits, while maintaining good recall. The inner workings of RaBitQ quantization are rather mathematically dense. It generates a quantization codebook by applying a uniformly random, approximately distance-preserving orthogonal transformation of the vertices of the

d

-dimensional hypercube, where

d

is the dimensionality of the dataset. We defer the details, and an elegant theoretical error bound, to the original paper. Use RaBitQ quantization by selecting quantized index types ending in the suffix RQ. For example, import IvfRq from lancedb.index and call table.create_index("vector", config=IvfRq()). Note that when using IVF_RQ, the dimension of the dataset must be a multiple of 8. num_bits determines how many bits are used to quantize each dimension. By default, Python, Rust, and Node leave this unset and inherit Lance’s default, currently 5 bits. Higher values improve recall at the cost of additional storage and query-time compute. To retain the previous one-bit default and maximum compression, explicitly set num_bits=1 (for example, IvfRq(num_bits=1) in Python). Five-bit quantized vector codes use roughly five times the storage of one-bit codes; total index size also includes other overhead.

RaBitQ-quantized indexes computed with num_bits >= 2 use a newer on-disk layout and cannot be read by some older LanceDB versions. The new five-bit default therefore affects reader compatibility as well as index size. Upgrade readers to versions that support multi-bit IVF_RQ before creating indexes with the default settings, or explicitly set num_bits=1 when sharing indexes with readers that support only one-bit IVF_RQ.

See this blog post for further discussion and benchmarking of LanceDB’s RaBitQ implementation.

Scalar Quantization (SQ)

Scalar quantization quantizes each entry of a vector independently, by simply replacing it with the closest of a pre-defined set of values. In practice, it often uses

8

bits per dimension of a vector, and supports very fast encoding and decoding. For example, suppose we know all vector entries across our dataset lie in the range

[-128 \times 10^5, 127 \times 10^5]

. In this case, we could quantize a given value

v

as an

8

-bit representation of the integer

j

, where

j \times 10^5

is the closest value to

v

among all integral multiples

\{j \times 10^5: -128 \leq j \leq 127 \}

Quantization API Reference

Parameter	Description
`num_bits`	Bits per dimension for quantization. Only applies to `IVF_PQ`/`IVF_HNSW_PQ` (default `8`) and `IVF_RQ` (`IvfRq()` inherits Lance’s default, currently `5`, RaBitQ); not used by `IVF_FLAT`/`IVF_SQ`/`IVF_HNSW_FLAT`/`IVF_HNSW_SQ`. Higher values improve accuracy at the cost of additional storage.

max_iterations and sample_rate also affect quantizer training, but since they apply to every IVF/HNSW index type (not just quantized ones), they’re documented as general Build-time Parameters instead. All three are passed as keyword arguments to create_index, alongside index_type:

table.create_index(
    index_type="IVF_RQ",
    num_bits=2,
    max_iterations=100,
    sample_rate=512,
)

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Vector Indexing

Do you need one?

Choosing the Right Index

Understanding Vector Indexes

IVF

HNSW

Using Vector Indexes

Configuration

Build-time Parameters

Search-time Parameters

Async API and Config Objects

IVF Indexes

IVF-HNSW Indexes

Managing Vector Indexes

Check Index Status

Custom Index Names

Quantization

Quantization Techniques

Product Quantization (PQ)

RaBitQ Quantization (RQ)

Scalar Quantization (SQ)

Quantization API Reference

​Do you need one?

​Choosing the Right Index

​Understanding Vector Indexes

​IVF

​HNSW

​Using Vector Indexes

​Configuration

​Build-time Parameters

​Search-time Parameters

​Async API and Config Objects

​IVF Indexes

​IVF-HNSW Indexes

​Managing Vector Indexes

​Check Index Status

​Custom Index Names

​Quantization

​Quantization Techniques

​Product Quantization (PQ)

​RaBitQ Quantization (RQ)

​Scalar Quantization (SQ)

​Quantization API Reference

Do you need one?

Choosing the Right Index

Understanding Vector Indexes

IVF

HNSW

Using Vector Indexes

Configuration

Build-time Parameters

Search-time Parameters

Async API and Config Objects

IVF Indexes

IVF-HNSW Indexes

Managing Vector Indexes

Check Index Status

Custom Index Names

Quantization

Quantization Techniques

Product Quantization (PQ)

RaBitQ Quantization (RQ)

Scalar Quantization (SQ)

Quantization API Reference