main (or from any
other branch). Anything you do on a branch (e.g., adding rows, changing the schema,
building an index) stays on that branch, so main keeps serving production
reads exactly as before. Branches are a natural fit when you want to:
- Experiment with a new index, schema change, or reprocessing step without
affecting live queries on
main. - Run a backfill or migration you’d like to validate before applying it to
main. - Hand a collaborator a frozen point-in-time fork while you keep writing to
main.
How branches relate to versions and tags
Every LanceDB table already tracks a linear history of versions, and you can tag a version orcheckout
one to read it. Branches add the missing piece: a separate, writable line of
history. Where a tag is a read-only label and checkout is a read-only view, a
branch forks from a point in history and then evolves on its own. Creating or
checking out a branch hands you a new table handle whose reads and writes are
scoped to that branch. The comparison table at
the end of this page lays out when to reach for each.
Branches are supported on local and namespace-backed tables in LanceDB Embedded,
as well as on LanceDB Enterprise tables.
Connect to a table
The branch API is identical no matter how you connect: only the connection itself differs between LanceDB Embedded and LanceDB Enterprise. Establish a connection (db) and open
a table (see Ingesting data), then use the same branch calls
in every example that follows.
LanceDB Embedded
Point LanceDB at a local directory (or an object-storage URI) to use it as an embedded library.LanceDB Enterprise
Enterprise Branching on LanceDB Enterprise works the same way as on LanceDB Embedded, apart from the differences noted below, but you connect to your Enterprise deployment with adb:// URI, an API key, and your
region and endpoint. Replace the placeholders below, including us-east-1 if your
deployment uses another region. In application code, load credentials from environment
variables or a secrets manager. Once connected, open a table and use the same branch
calls in every example that follows.
Work with branches
The lifecycle of a branch is short and predictable: fork it, write to it, reopen it whenever you need it, and delete it once you’re done. The examples below use a smallquotes table with three rows on main.
Create a branch
Forking frommain returns a table handle scoped to the new branch. main is
the reserved default source, so create needs only a name; to fork from
somewhere else, pass a branch name, a specific version, or both.
Write to a branch
Writes go through the branch handle and stay there: themain handle keeps
reporting its original row count. Listing branches returns a mapping of each
branch name to its metadata, including the version it was forked from.
Reopen a branch
A branch outlives the handle that created it. Reopen it later by name, either from an existing table handle or straight from the connection when you open the table. Both routes give you a writable handle tracking the branch’s latest state.Delete a branch
Deleting a branch removes it and its branch-local history;main is untouched.
A deployment refuses to delete a branch whose columns have been promoted onto
main, because main still depends on it.
Before deleting a branch, make sure you’ve retained any results you need. See
Apply branch-tested changes to main
below.
Apply branch-tested changes to main
How you apply a validated change depends on the type of work:
- Added columns (Enterprise): review the branch with
diff, then promote the new columns ontomainwithcherry_pick(cherryPickin TypeScript). See Compare and promote a branch intomain. - Backfill or transformation: rerun the validated job against
main. - Schema change: apply the same reviewed schema operation to
main. - Index change: build the index on
mainusing the configuration validated on the branch. - Selected row results: upsert those rows into
mainusing a stable unique key.
Upsert selected branch rows into main
If the result you want to retain is a set of inserted or updated rows, use
merge_insert to write them to
main. Despite its name, merge_insert is a row-ingestion operation: it
matches incoming rows by key and does not merge branch histories.
Read the rows you want from the reviewed branch, then upsert them into main on
your key column (id in this example). Here, candidate is the branch handle:
to_arrow(); read
the branch’s rows with a query instead. See
Avoid materializing the whole table.
This operation transfers only the inserted or updated rows in its input. It does
not transfer branch history, schema changes, indexes, or branch-local deletions,
and it does not determine which rows changed after the branch was created.
Reapplying an identical payload is idempotent on the key, but the operation is
not conflict-aware. If main has diverged, incoming branch rows can overwrite
newer values with the same key. Read and upsert the whole branch only when that
overwrite is intentional; otherwise, filter the branch read to the rows you
intend to apply.
Compare and promote a branch into main
Enterprise
On LanceDB Enterprise, branches expose two review-and-land calls: diff
inspects what a branch has changed relative to main, and cherry_pick
(cherryPick in TypeScript) promotes the branch’s new columns onto main in
place, without reissuing the branch’s writes.
diff and cherry_pick work on LanceDB Enterprise tables only. On an Embedded
table they fail as not supported: Python raises NotImplementedError, and
TypeScript and Rust return an error. A cherry-pick promotes added columns; use
the upsert or rerun patterns above for
row and index changes.Diff a branch
diff reads the branch and main without modifying either, and returns what
has changed: columns added, removed or changed, indexes added or removed, row
counts, and a list of errors (the reasons a cherry-pick of the branch could
not land now). An empty list means it could.
parentVersion,
mainVersion, branchVersion), rowCountMain and rowCountBranch, a
rowSummary, and removedColumns, changedColumns, addedIndexes and
removedIndexes. In Python the result is a dictionary with the same camelCase
keys.
The error codes are baseMoved (main has moved on since the version the
branch was forked from), rowCountMismatch, rowsChanged, columnRemoved,
columnChanged, nothingToApply, noColumnChanges, inputColumnDependency
and parentNotMain. A code the client does not know surfaces as unknown, so
older clients keep working.
Promote a branch’s columns
cherry_pick promotes the columns a branch added onto main. The server
re-evaluates the diff when the request arrives and either lands the change or
refuses it for the reasons diff would report. A refusal is not an exception:
the call returns a result whose status is "failed", with the reasons in
diff.errors, so you can decide what to do next. Cherry-picks are not retried.
Pass dry_run=True in Python, or true as the second argument in TypeScript,
to check without landing anything. preview.promotedColumns lists the columns
the cherry-pick would promote, or did.
status is ready (a dry run that would land), cherryPicked (it landed, and
mainVersionAfter is the new version of main), failed (refused, or for a
dry run, would be refused; see diff.errors), notImplemented, or unknown
for a status the client does not know. A branch that does not exist raises a
not-found error instead of returning a status.
In async Python, await the same calls on an AsyncTable:
await table.branches.diff("exp") and
await table.branches.cherry_pick("exp", dry_run=True). In Rust they are
Table::diff_branch and Table::cherry_pick.
Build indexes on a branch
One of the most useful things a branch buys you is a safe place to build and validate an index without affecting what’s in production on themain branch.
Fork a branch, create your vector (ANN) and full-text search (FTS)
indexes on it, and check recall and latency. Once you have selected a
configuration, build the corresponding index on main through your normal
deployment workflow. Because the indexes live on the branch, queries against
main never see a half-built index and are never slowed down by the branch’s
build.
This pattern is especially valuable on Enterprise deployments, where main is
typically serving production traffic while you tune an index configuration on
the side. A deployment builds the branch’s indexes in the background, so wait
for them before measuring.
main.
Branches vs. tags vs. versions
Now that you’re familiar with branches, you can see how they complement the other ways LanceDB give you to work with table history. Choose these approaches based on whether you need to label, read, or write data at a point in history:For linear version history (creating versions, listing them, rolling back, and
tagging), see the Versioning and Reproducibility guide.