Build an index

LanceDB provides a comprehensive suite of indexing strategies, vector, scalar, and full-text search, to optimize query performance across diverse workloads.

• Vector index: optimized for searching high-dimensional data (like images, audio, or text embeddings) by finding the most similar vectors efficiently.
• Scalar index: Used for filtering and sorting structured numeric or categorical data (e.g., timestamps, prices) efficiently.
• Full-text search index: speeds up text searches by indexing words and phrases, making queries like keyword searches much faster.

Compared to our open source version, LanceDB Cloud/Enterprise automates index optimization for all types of indices, when there are updates to the data. For vector indexing, our architecture revolutionizes traditional approaches through zero-config automation and elastic scalability:

• Automatic Column Detection: When a table contains a single vector column (e.g., vector_embedding), LanceDB:
  • Infers the vector column from the table schema
  • Creates optimized IVF-PQ/HNSW indexes without manual configuration
• Parameter Autotuning: LanceDB analyzes your data distribution to automatically configure indexing parameters.

​

Vector Index

LanceDB has implemented the state-of-art indexing algorithms (more about IVF-PQ and HNSW) with acceleration from our optimized infrastructure. We currently support the L2, Cosine , Dot and Hamming (only available for binary vectors) as distance calculation metrics. You can create multiple vector indices within a table.

import lancedb

# connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

# let's use the table created from quickstart
table_name = "lancedb-cloud-quickstart"
table = db.open_table(table_name)
# the vector column only needs to be specified when there are 
# multiple vector columns or the column is not named as "vector"
# L2 is used as the default distance metric
table.create_index(metric="cosine", vector_column_name="keywords_embeddings")

​

Check index status

Creating a vector index is fast. It takes only a few minutes to index 1 million vectors with 1536 dimensions. Index status can be checked on the dashboard or retrieved by calling APIs.

​

Option 1: From the dashboard

After navigating to the desired table page, the corresponding column will display the index once it has been created. The “Index” column will remain blank if no index exists or if index creation is still in progress.

​

Option 2: Calling APIs

We provide the list_indices() and index_stats() APIs to check index status. The index name is formed by appending “_idx” to the column name. Note that list_indices() will not return any information until the index has fully ingested and indexed all available data.

import time

def wait_for_index(table, index_name):
  POLL_INTERVAL = 10
  while True:
      indices = table.list_indices()

      if indices and any(index.name == index_name for index in indices):
          break
      print(f"⏳ Waiting for {index_name} to be ready...")
      time.sleep(POLL_INTERVAL)

  print(f"✅ {index_name} is ready!")

index_name = "keywords_embeddings_idx"
wait_for_index(table, index_name)
print(table.index_stats(index_name))
# IndexStatistics(num_indexed_rows=3000, num_unindexed_rows=0, index_type='IVF_PQ', 
#   distance_type='cosine', num_indices=None)

​

Index binary vectors

Binary vectors are useful for applications that use hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits.

When using binary vectors:

• Store them as fixed-size binary data - they are stored as uint8 arrays (every 8 bits are stored as a byte)
• Use Hamming distance for similarity search
• Pack binary vectors into bytes to save space

import lancedb
import numpy as np
import pyarrow as pa

# Connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

# Create schema with binary vector field
table_name = "test-hamming"

ndim = 256
schema = pa.schema(
    [
        pa.field("id", pa.int64()),
        # for dim=256, lance stores every 8 bits in a byte
        # so the vector field should be a list of 256 / 8 = 32 bytes
        pa.field("vector", pa.list_(pa.uint8(), 32)),
    ]
)

table = db.create_table(table_name, schema=schema, mode="overwrite")

data = []
for i in range(1024):
    vector = np.random.randint(0, 2, size=ndim)
    # pack the binary vector into bytes to save space
    # note: this is optional
    vector = np.packbits(vector)
    data.append(
        {
            "id": i,
            "vector": vector,
        }
    )
table.add(data)

# create the index with hamming distance
table.create_index(
    metric="hamming", vector_column_name="vector", index_type="IVF_FLAT"
)

# Create query vector and search
query = np.random.randint(0, 2, size=256)
query = np.packbits(query)
df = table.search(query).metric("hamming").limit(10).to_pandas()
df.vector = df.vector.apply(np.unpackbits)

​

Scalar Index

LanceDB Cloud and LanceDB Enterprise supports several types of Scalar indices to accelerate search over scalar columns.

• BTREE: The most common type is BTREE. This index is inspired by the btree data structure although only the first few layers of the btree are cached in memory. It will perform well on columns with a large number of unique values and few rows per value.
• BITMAP: this index stores a bitmap for each unique value in the column. This index is useful for columns with a finite number of unique values and many rows per value. For example, columns that represent “categories”, “labels”, or “tags”
• LABEL_LIST: a special index that can be used on List<T> columns to support queries with array_contains_all and array_contains_any using an underlying bitmap index. For example, a column that contains lists of tags (e.g. [“tag1”, “tag2”, “tag3”]) can be indexed with a LABEL_LIST index.

You can create multiple scalar indices within a table.

import lancedb

# connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

table_name = "lancedb-cloud-quickstart"
table = db.open_table(table_name)
# default is BTree
table.create_scalar_index("label", index_type="BITMAP")

You can check the scalar index status by using the two options above.

index_name = "label_idx"
wait_for_index(table, index_name)

​

Build a scalar index on a UUID column

LanceDB supports building scalar indices on UUID columns, which are typically stored as FixedSizeBinary(16) data type. This enables efficient lookups and filtering operations on UUID-based primary keys or identifiers.

import lancedb
import uuid
import pyarrow as pa

class UuidType(pa.ExtensionType):
    def __init__(self):
        # Initialize with storage type and extension name
        super().__init__(pa.binary(16), "my.uuid")

    def __arrow_ext_serialize__(self):
        return b'uuid-metadata'

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        return UuidType()  # Return new instance
    
pa.register_extension_type(UuidType())

def generate_random_string(length=10):
    return ''.join(random.choices(string.ascii_letters + string.digits, k=length))

def generate_uuids(num_items):
    return [uuid.uuid4().bytes for _ in range(num_items)]

# Connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

n = 100
# Generate UUIDs and random strings
uuids = generate_uuids(n)
names = [generate_random_string() for _ in range(n)]

# Create arrays
uuid_array = pa.array(uuids, pa.binary(16))
name_array = pa.array(names, pa.string())

# Create extension array for UUIDs
extension_array = pa.ExtensionArray.from_storage(UuidType(), uuid_array)

# Create schema with both UUID and string fields
schema = pa.schema([
    pa.field('id', UuidType()),
    pa.field('name', pa.string())
])

# Create table with both columns
data_table = pa.Table.from_arrays([extension_array, name_array], schema=schema)
table_name = "index-on-uuid-test"
table = db.create_table(table_name, data=data_table, mode="overwrite")
table.create_scalar_index("id")
index_name = "id_idx"
wait_for_index(table, index_name)

new_users = [
    {"id": uuid.uuid4().bytes, "name": "Bobby"},  # Will update existing record
    {"id": uuid.uuid4().bytes, "name": "Charlie"},  # Will insert new record
]

# Upsert by id
table.merge_insert("id") \
    .when_matched_update_all() \
    .when_not_matched_insert_all() \
    .execute(new_users)

​

Full-Text Search Index

We provide performant full-text search on LanceDB Cloud and LanceDB Enterprise, allowing you to incorporate keyword-based search (based on BM25) in your retrieval solutions.

import lancedb

# connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

table_name = "lancedb-cloud-quickstart"
table = db.open_table(table_name)
table.create_fts_index("text")

You can check the FTS index status by using the two options above.

index_name = "text_idx"
wait_for_index(table, index_name)

​

FTS Configuration Parameters

LanceDB supports the following configurable parameters for creating a full-text search index:

​

Full-Text Search on Array Fields

LanceDB supports creating full-text search indices on string array columns, enabling efficient keyword-based search across multiple values within a single field. This is particularly useful for searching through tags, keywords, or any list of text items.

import lancedb

# connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

table_name = "fts-array-field-test"
schema = pa.schema([
    pa.field("id", pa.string()),
    pa.field("tags", pa.list_(pa.string())),  # List of strings for FTS
    pa.field("description", pa.string())  # Single string for comparison
])

# Generate sample data
data = {
    "id": [f"doc_{i}" for i in range(10)],
    "tags": [
        ["python", "machine learning", "data science"],
        ["deep learning", "neural networks", "AI"],
        ["database", "indexing", "search"],
        ["vector search", "embeddings", "AI"],
        ["full text search", "indexing", "database"],
        ["python", "web development", "flask"],
        ["machine learning", "deep learning", "pytorch"],
        ["database", "SQL", "postgresql"],
        ["search engine", "elasticsearch", "indexing"],
        ["AI", "transformers", "NLP"]
    ],
    "description": [
        "Python for data science projects",
        "Deep learning fundamentals",
        "Database indexing techniques",
        "Vector search implementations",
        "Full-text search guide",
        "Web development with Python",
        "Machine learning with PyTorch",
        "Database management systems",
        "Search engine optimization",
        "AI and NLP applications"
    ]
}

# Create table and add data
table = db.create_table(table_name, schema=schema, mode="overwrite")
table_data = pa.Table.from_pydict(data, schema=schema)
table.add(table_data)

# Create FTS index on tags
table.create_fts_index("tags")
wait_for_index(table, "tags_idx")

print("\nSearching for 'learning' in tags with a typo:")
result = table.search(MatchQuery("learnin", column="tags", fuzziness=1), query_type="fts").select(['id', 'tags', 'description']).to_arrow()
print("\nSearching for 'machine learning' in tags:")
result = table.search(PhraseQuery("machine learning", column="tags"), query_type="fts").select(['id', 'tags', 'description']).to_arrow()

​

Update an Index

When new data is added to a table, LanceDB Cloud automatically updates indices in the background.

To see the status of an index, call the index_stats() API to view the number of rows that are currently unindexed. This will be zero when indices are fully up-to-date.

When there are unindexed rows in a table, queries will use brute force methods to search over the unindexed rows. This can mean a temporary increase in latency while the background indexing jobs are running. To avoid this, you can search only the data currently indexed by setting the fast_search query parameter to True.

​

GPU based Indexing

With GPU powered indexing, LanceDB is able to create a vector index with billions of rows in a few hours.