Skip to main content
Applies to: Embedded ✅, Enterprise ✅ As you add new data to your LanceDB tables, your indexes may become outdated. Reindexing is the process of updating the index to account for new data. This applies to either a full-text search (FTS) index or a vector index. Reindexing is an important operation to run periodically as your data grows, as it has performance implications. As data is being added and a reindex operation is running, LanceDB will combine results from the existing index with exhaustive/flat search on the new data. This is done to ensure that you’re still retrieving results over all your data, but it does come at a performance cost. The more data that you add without reindexing, the impact on latency (due to exhaustive search) can be noticeable. Rather than dropping an existing index entirely and reindexing from scratch, LanceDB supports incremental indexing.
Enterprise job types: an index build runs as reindex_ followed by the algorithm, such as reindex_ivf_pq, reindex_btree or reindex_fts, and so does folding new rows into an existing index. A create-only build, one that must not replace an existing index, runs as create_ followed by the algorithm instead; Python’s create_scalar_index and create_fts_index ask for that by default. The name says which algorithm is building, not a separate feature.

​
Triggering an incremental update

In the embedded library, you trigger an incremental indexing operation on updated data with the optimize() method on a table. A deployment does this for you and does not support optimize(), as described below. Table optimization performs three maintenance operations:
  1. Compaction: merges small fragments into larger ones to improve read performance
  2. Pruning/Cleanup: removes files from versions older than a retention window (7 days by default) — see Cleanup for how retention works
  3. Index update: adds newly-ingested data to existing vector, scalar, and FTS indexes
Enterprise LanceDB Enterprise supports incremental reindexing through an automated background process. Once enough new rows have been added to a table, the system automatically adds them to its indexes. As the dataset grows, indexes are asynchronously updated in the background.
  • While indexes are being rebuilt, queries use brute force methods on unindexed rows, which may temporarily increase latency. If you can omit unindexed rows, set fast_search=True to search only indexed data. This can return incomplete results while indexing catches up.
  • Use index_stats() to view the number of unindexed rows. This will be zero when indexes are fully up-to-date. If you call wait_for_index(...), it polls the same status and can time out while continuous writes keep adding unindexed rows.
  • Rows below the deployment’s threshold stay unindexed until more arrive, so the count can stay above zero without further writes. FM indexes are not updated in the background.
The benefit of using LanceDB Enterprise is that it automates the reindexing process and operates continuously in the background, minimizing the impact on latency under high loads. In LanceDB Embedded, you must manually manage the reindexing cadence based on your data growth and performance needs. Opening a table in LanceDB Embedded logs a warning at most once per table per process when any index has at least 1,000,000 unindexed rows. Set LANCEDB_LOG=warn to surface this log if your logger hides warnings by default.

​
Index maintenance jobs

Two further Enterprise job types keep an index correct as the rows underneath it move, rather than adding new rows to it: See Advanced Conflict Resolution for why index_remap exists as a separate, deferred job rather than happening synchronously inside every compaction.

​
What to read next

Distributed Indexing

How an index build itself scales across the jobs fleet.

GPU Indexing

Building a vector index faster with GPU acceleration.