Skip to main content
Applies to: Embedded ✅, Enterprise ✅ Every write commits a new table version, and versions accumulate: small files from frequent appends, superseded files from updates and deletes, and rows added since an index was built. In the embedded library, optimize() addresses all three; a deployment handles them in the background and does not support optimize(). Run optimization as regular table maintenance, particularly after many small writes or a large ingest.

​
Optimize a table

Three operations, in one call: Compaction itself is only the first of the three: merging small fragments back into larger ones, which is what keeps a scan from opening more files than it needs to.
Enterprise job type: compact. Runs automatically as data accumulates, separately from cleanup. The two are distinct jobs even though optimize() bundles both in one LanceDB Embedded call.

​
Retraining indexes

Folding new rows into an index preserves the structure the index was trained on. When the data distribution has shifted enough that the index’s clusters are a poor fit, recreate the index with replace=True to train it on the current data. The deprecated retrain argument to optimize() does not rebuild indexes. This example assumes a populated table whose vector column has an IVF_FLAT index configured for L2 distance and four partitions:
Use your index’s column, type, distance metric, and training settings when rebuilding it. Recreating an index is more expensive than a normal optimize, and can help after a large ingest that changed the data distribution.

​
When to run it

In the embedded library there is no background compactor: optimize() runs when you call it. A deployment compacts a table automatically once enough small fragments accumulate, unless its operator turns this off. Frequent small appends are what make it necessary, because each one commits files. A table written in a few large batches needs it rarely; one taking continuous small writes needs it on a schedule. Opening a table in LanceDB Embedded logs a warning at most once per table per process when at least 100 fragments have fewer than 100,000 rows each. Set LANCEDB_LOG=warn to surface this log if your logger hides warnings by default.
Compaction rewrites files, which conflicts with a concurrent computed-column refresh or view refresh planned against the version it replaces. Sequence them rather than running both at once.
See Garbage Collection for pruning old versions, and Incremental Indexing for the index-folding half of optimize().

​
Performance

Two things accumulate on a long-lived table as more and more data gets added to it:
  • Many small fragments build up as you write, slowing down queries that have to scan across more files. Compaction merges them back into larger fragments.
  • Old versions build up as the table changes, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). Cleanup prunes versions older than a retention window.
In LanceDB Embedded, you run them yourself via optimize(), which bundles both into a single call:
The best practice is to run optimize() after large writes or on a schedule. It also bundles incremental index updates. See Incremental Indexing for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed. Update or rebuild the index after large update batches.