Build an index

LanceDB provides a comprehensive suite of indexing strategies, vector, scalar, and full-text search, to optimize query performance across diverse workloads.

• Vector index: optimized for searching high-dimensional data (like images, audio, or text embeddings) by finding the most similar vectors efficiently.
• Scalar index: Used for filtering and sorting structured numeric or categorical data (e.g., timestamps, prices) efficiently.
• Full-text search index: speeds up text searches by indexing words and phrases, making queries like keyword searches much faster.

Compared to our open source version, LanceDB Cloud/Enterprise automates index optimization for all types of indices, when there are updates to the data. For vector indexing, our architecture revolutionizes traditional approaches through zero-config automation and elastic scalability:

• Automatic Column Detection: When a table contains a single vector column (e.g., vector_embedding), LanceDB:
  • Infers the vector column from the table schema
  • Creates optimized IVF-PQ/HNSW indexes without manual configuration
• Parameter Autotuning: LanceDB analyzes your data distribution to automatically configure indexing parameters.

​

Vector Index

LanceDB has implemented the state-of-art indexing algorithms (more about IVF-PQ and HNSW) with acceleration from our optimized infrastructure. We currently support the L2, Cosine , Dot and Hamming (only available for binary vectors) as distance calculation metrics. You can create multiple vector indices within a table.

import lancedb

# connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

# let's use the table created from quickstart
table_name = "lancedb-cloud-quickstart"
table = db.open_table(table_name)
# the vector column only needs to be specified when there are 
# multiple vector columns or the column is not named as "vector"
# L2 is used as the default distance metric
table.create_index(metric="cosine", vector_column_name="keywords_embeddings")

​

Check index status

Creating a vector index is fast. It takes only a few minutes to index 1 million vectors with 1536 dimensions. Index status can be checked on the dashboard or retrieved by calling APIs.

​

Option 1: From the dashboard

After navigating to the desired table page, the corresponding column will display the index once it has been created. The “Index” column will remain blank if no index exists or if index creation is still in progress.

​

Option 2: Calling APIs

We provide the list_indices() and index_stats() APIs to check index status. The index name is formed by appending “_idx” to the column name. Note that list_indices() will not return any information until the index has fully ingested and indexed all available data.

import time

def wait_for_index(table, index_name):
  POLL_INTERVAL = 10
  while True:
      indices = table.list_indices()

      if indices and any(index.name == index_name for index in indices):
          break
      print(f"⏳ Waiting for {index_name} to be ready...")
      time.sleep(POLL_INTERVAL)

  print(f"✅ {index_name} is ready!")

index_name = "keywords_embeddings_idx"
wait_for_index(table, index_name)
print(table.index_stats(index_name))
# IndexStatistics(num_indexed_rows=3000, num_unindexed_rows=0, index_type='IVF_PQ', 
#   distance_type='cosine', num_indices=None)

​

Index binary vectors

Binary vectors are useful for applications that use hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits.

When using binary vectors:

• Store them as fixed-size binary data - they are stored as uint8 arrays (every 8 bits are stored as a byte)
• Use Hamming distance for similarity search
• Pack binary vectors into bytes to save space

import lancedb
import numpy as np
import pyarrow as pa

# Connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

# Create schema with binary vector field
table_name = "test-hamming"

ndim = 256
schema = pa.schema(
    [
        pa.field("id", pa.int64()),
        # for dim=256, lance stores every 8 bits in a byte
        # so the vector field should be a list of 256 / 8 = 32 bytes
        pa.field("vector", pa.list_(pa.uint8(), 32)),
    ]
)

table = db.create_table(table_name, schema=schema, mode="overwrite")

data = []
for i in range(1024):
    vector = np.random.randint(0, 2, size=ndim)
    # pack the binary vector into bytes to save space
    # note: this is optional
    vector = np.packbits(vector)
    data.append(
        {
            "id": i,
            "vector": vector,
        }
    )
table.add(data)

# create the index with hamming distance
table.create_index(
    metric="hamming", vector_column_name="vector", index_type="IVF_FLAT"
)

# Create query vector and search
query = np.random.randint(0, 2, size=256)
query = np.packbits(query)
df = table.search(query).metric("hamming").limit(10).to_pandas()
df.vector = df.vector.apply(np.unpackbits)

​

Scalar Index

LanceDB Cloud and LanceDB Enterprise supports several types of Scalar indices to accelerate search over scalar columns.

• BTREE: The most common type is BTREE. This index is inspired by the btree data structure although only the first few layers of the btree are cached in memory. It will perform well on columns with a large number of unique values and few rows per value.
• BITMAP: this index stores a bitmap for each unique value in the column. This index is useful for columns with a finite number of unique values and many rows per value. For example, columns that represent “categories”, “labels”, or “tags”
• LABEL_LIST: a special index that can be used on List<T> columns to support queries with array_contains_all and array_contains_any using an underlying bitmap index. For example, a column that contains lists of tags (e.g. [“tag1”, “tag2”, “tag3”]) can be indexed with a LABEL_LIST index.

You can create multiple scalar indices within a table.

import lancedb

# connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

table_name = "lancedb-cloud-quickstart"
table = db.open_table(table_name)
# default is BTree
table.create_scalar_index("label", index_type="BITMAP")

You can check the scalar index status by using the two options above.

index_name = "label_idx"
wait_for_index(table, index_name)

​

Full-Text Search Index

We provide performant full-text search on LanceDB Cloud and LanceDB Enterprise, allowing you to incorporate keyword-based search (based on BM25) in your retrieval solutions.

import lancedb

# connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

table_name = "lancedb-cloud-quickstart"
table = db.open_table(table_name)
table.create_fts_index("text")

​

FTS Configuration Parameters

LanceDB supports the following configurable parameters for creating a full-text search index:

You can check the FTS index status by using the two options above.

index_name = "text_idx"
wait_for_index(table, index_name)

​

Update an Index

When new data is added to a table, LanceDB Cloud automatically updates indices in the background.

To see the status of an index, call the index_stats() API to view the number of rows that are currently unindexed. This will be zero when indices are fully up-to-date.

When there are unindexed rows in a table, queries will use brute force methods to search over the unindexed rows. This can mean a temporary increase in latency while the background indexing jobs are running. To avoid this, you can search only the data currently indexed by setting the fast_search query parameter to True.

​

GPU based Indexing

With GPU powered indexing, LanceDB is able to create a vector index with billions of rows in a few hours.