> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Compaction and cleanup

> Merge small files, prune old versions, and fold new rows into existing indexes.

**Applies to:** Embedded ✅, Enterprise ✅

Every write commits a new table version, and versions accumulate: small files
from frequent appends, superseded files from updates and deletes, and rows added
since an index was built. In the embedded library, `optimize()` addresses all
three; a deployment handles them in the background and does not support
`optimize()`.

Run optimization as regular table maintenance, particularly after many small
writes or a large ingest.

## Optimize a table

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
table.optimize()
```

Three operations, in one call:

| | |
| - | - |
| **Compaction** | merges small files into larger ones |
| **Prune** | removes versions older than the retention window (see [Garbage Collection](https://docs.lancedb.com/maintenance/cleanup.md)) |
| **Index** | folds newly added rows into existing indexes (see [Incremental Indexing](https://docs.lancedb.com/indexing/incremental-indexing.md)) |

Compaction itself is only the first of the three: merging small fragments back
into larger ones, which is what keeps a scan from opening more files than it
needs to.

> [!NOTE]
>
> **Enterprise job type:** `compact`. Runs automatically as data accumulates,
> separately from [`cleanup`](https://docs.lancedb.com/maintenance/cleanup). The two are distinct jobs
> even though `optimize()` bundles both in one LanceDB Embedded call.

## Retraining indexes

Folding new rows into an index preserves the structure the index was trained on.
When the data distribution has shifted enough that the index's clusters are a
poor fit, recreate the index with `replace=True` to train it on the current data.
The deprecated `retrain` argument to `optimize()` does not rebuild indexes.

This example assumes a populated table whose `vector` column has an IVF\_FLAT
index configured for L2 distance and four partitions:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb.index import IvfFlat

table.create_index(
    "vector",
    config=IvfFlat(distance_type="l2", num_partitions=4),
    replace=True,
)
```

Use your index's column, type, distance metric, and training settings when
rebuilding it. Recreating an index is more expensive than a normal optimize,
and can help after a large ingest that changed the data distribution.

## When to run it

In the embedded library there is no background compactor: `optimize()` runs
when you call it. A deployment compacts a table automatically once enough small
fragments accumulate, unless its operator turns this off.

Frequent small appends are what make it necessary, because each one commits
files. A table written in a few large batches needs it rarely; one taking
continuous small writes needs it on a schedule.

Opening a table in LanceDB Embedded logs a warning at most once per table per
process when at least 100 fragments have fewer than 100,000 rows each. Set
`LANCEDB_LOG=warn` to surface this log if your logger hides warnings by default.

> [!NOTE]
>
> Compaction rewrites files, which conflicts with a concurrent
> [computed-column refresh](https://docs.lancedb.com/feature-engineering/add-computed-column.md) or
> [view refresh](https://docs.lancedb.com/feature-engineering/refresh-materialized-view.md) planned against the
> version it replaces. Sequence them rather than running both at once.

See [Garbage Collection](https://docs.lancedb.com/maintenance/cleanup.md) for pruning old versions, and
[Incremental Indexing](https://docs.lancedb.com/indexing/incremental-indexing.md) for the index-folding half of
`optimize()`.

## Performance

Two things accumulate on a long-lived table as more and more data gets added to it:

* **Many small fragments build up as you write**, slowing down queries that have to scan across more files. **Compaction** merges them back into larger fragments.
* **Old versions build up as the table changes**, growing disk usage beyond the live data size (LanceDB retains them for time-travel and rollback). **Cleanup** prunes versions older than a retention window.

In LanceDB Embedded, you run them yourself via `optimize()`, which bundles both into a single call:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from datetime import timedelta
table.optimize()                                       # 7-day default retention
table.optimize(cleanup_older_than=timedelta(days=1))   # reclaim space sooner
```

The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates. See [Incremental Indexing](https://docs.lancedb.com/indexing/incremental-indexing.md) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed. Update or rebuild the index after large update batches.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.