Enterprise only
Distributed compaction splits a table’s fragments into independent,
fragment-disjoint tasks and fans them out across job workers in LanceDB
Enterprise. Unlike an index build, a compaction plan needs no bin-packing:
Lance’s own planner already produces disjoint tasks, so workers claim and
execute them directly, with no coordination needed to decide who does
what. See Architecture.
No task depends on another’s result, so a worker that finishes early
simply claims another task rather than waiting on its peers. Committing is
where the fleet does coordinate: finished tasks go through a shared
eligibility gate, and only the one worker that claims the commit actually
writes it, rather than each worker committing its own task independently.
Whether a failed task can still let the rest commit depends on the remap
mode. Inline remap is all-or-nothing: one failed task fails the whole job,
since the commit updates every affected index inline and can’t skip a
task’s share of that. Deferred remap can commit a subset — a failed
task’s fragments are abandoned and replanned in a later round, while the
rest of the job’s tasks commit without waiting on it. See
Advanced Conflict Resolution
for when compaction defers a remap at all.
Compaction interfaces
- Compaction for triggering and configuring a compaction.
- Garbage Collection for removing files compaction and other operations leave behind.
- Job Framework for the job types a deployment runs on its own schedule.