Enterprise only
Refresh a computed column to fill
new rows and recompute values whose inputs changed. The example below continues
with the doubled column from the computed-column guide.
Computed columns require LanceDB Enterprise and the
Enterprise client version.
table.refresh_column(...) (sync) is also a local, blocking form, but it
raises on an Enterprise connection — use refresh_column_async, shown
above, or the SQL form instead. See
Run a refresh on Enterprise below for what the returned job
supports.The SQL form is
REFRESH COLUMN, with the same resource options.What a refresh recomputes
Without aWHERE predicate, a refresh fills unassigned rows and recomputes
values whose inputs changed since they were computed. Every other row is left
alone, so a second refresh with nothing changed does nothing. This supports
two common workflows.
Appending is cheap. Rows added since the last refresh are filled by the
next one; filled rows whose inputs are unchanged are skipped. Backfilling a
column and then ingesting more data does not mean recomputing the backfill.
A filled value catches up with its input at the next refresh. If doubled
derives from x and a row’s x is later updated, doubled keeps its old value
until the next refresh recomputes it.
That is what makes the call idempotent and safe to retry. However, unlike a
spreadsheet formula, nothing is recomputed until you refresh, so a derived
column is stale between an update and the next refresh.
Automatic refresh
A computed column can refresh itself on a schedule instead of waiting for an explicit call here. See Auto Refresh for enabling it, the SQL and SDK forms for changing the setting, and how catch-up works when it’s re-enabled after being off.Refreshing part of a table
Pythonwhere= requires the Enterprise client and an
Enterprise server with filtered refresh support. Stable 0.41.0 does not
accept this argument. The SQL form is available independently of that Python
argument on supporting servers.
Enterprise A refresh can be scoped to a predicate,
with where= in Python or WHERE in SQL, which is useful for filling a slice
first and checking it before committing to the whole table:
NULL. Use an unfiltered refresh
to let LanceDB track which outputs need updating.
Enterprise job type:
computed_refresh for a SQL-expression column,
function_refresh for one filled by a registered function.Re-score selected list elements
WHERE selects rows, not elements within a list. Refreshing a row that
contains ten answers passes the whole list to the Function. A mask can control
which elements your code computes, but the Function still returns one complete
list for that row. There is no element-level refresh or partial-list write API.
This example keeps sample_id as the link to an existing samples table with
a Blob column named media. It creates only answer metadata and scores; it
neither reads nor copies Blob payloads. Run it on a deployment supporting
@function, list signatures and Python where=, with samples already
containing sample IDs s1 and s2.
"maybe" is scored;
the other scores are copied from previous. The final equality checks Blob
descriptors on an otherwise unchanged source table, without fetching payloads.
Keep the source table unchanged during this comparison. Use the same model,
normalization and score type as the original computation in a real re-scoring
pass. For another pass, explicitly supply the intended prior scores and mask;
refresh does not merge the previous output into your Function’s inputs.
If answers need independent retries or versions, prefer one ordinary table row
per answer, with group_id, sample_id, a stable application answer_id,
answer text and score. Keep media in samples and join by sample_id when
needed. A row predicate can then select exact answer IDs, and Blob storage
stays in the source table. Do not use UNNEST to assume that nested Blob values
become supported Function inputs; see the
nested-Blob limits.
Run a refresh on Enterprise
Enterprise With an Enterprise cluster that enables computed columns, the refresh runs on the cluster.refresh_column raises
there; use refresh_column_async, which returns a job you can
wait on or poll instead of holding a connection open:
status(), resolve with wait(). status() is a point snapshot and
does not raise, which is what makes it safe in a loop; wait() raises
JobFailedError, JobCancelledError or TimeoutError.
A long backfill can therefore be started, left, and checked on from a different
process.
Admission and waiting
Submission does not reserve a supervisor or GPU immediately. A refresh can wait for source preparation/compaction, an earlier refresh, executor admission or worker capacity. A lifecyclerunning value from status() alone does not
prove that Python code has begun executing.
Use SHOW JOBS and the Console’s job detail to inspect
available phase and row progress. On SDK/server versions with job.progress(),
the snapshot can also include phase, queue_reason, timestamps and row
counters. These optional fields are version-dependent; missing values are
unknown, not zero progress or an ETA. creation_ms is submission time, not
worker start time. DESCRIBE JOB provides the finished run report, rather than
a live queue-position API.
Limits are deployment settings: the Agent dispatcher concurrency, executor
capacity, per-table automatic-refresh limit and Function-call
max_concurrency control different things. There is no universal five-refresh
limit per database. Ask your administrator for the effective settings and use
the Enterprise operations guide
for their configuration names. Repeated cancellation and resubmission does not
bypass those limits. Cancellation requests a stop, including while queued;
observe a terminal state before submitting replacement work. Committed work
is not rolled back.
Configure workers and resources
SQL exposes the worker budget directly, which is the lever when a backfill is too slow or too expensive:num_workers is only
accepted for SQL-expression columns; registered Function columns reject it.
In particular, gpu_count = 2 does not request two devices inside one Python
process or enable tensor parallelism. Inspect configured GPU classes and their
per-worker limits with db.list_gpu_classes() or SHOW GPU CLASSES. Choose a
model that fits the worker shape supported by your deployment; see
GPU placement and budgets.
The SDKs accept the same nested budget. CPU and memory are required; the GPU
count inside the budget defaults to zero, so a CPU-only Function can omit it:
REFRESH COLUMN ... WITH (...) form
still takes all three fields when CPU or memory is set; use gpu_count = 0
for a CPU-only Function. The SDK normalizes an omitted nested GPU count to a
numeric zero before sending the request.
The existing top-level gpu_count / gpuCount option remains the shorthand
for changing only the aggregate GPU budget. It cannot be combined with the
complete resource budget. Omitting both preserves the server’s existing
defaults.
These examples configure only the explicit refresh shown. A registered
Function declaration can separately set auto_refresh_resource_budget on
add_columns; that declaration controls the budget for its automatic
refreshes. SQL-expression columns reject the automatic resource-budget option
because they do not use Function workers.
To override the registered Function’s call concurrency for this logical
execution only:
max_concurrency caps user-code calls in flight for one Function within one
logical execution. It is not an account-wide quota and does not resize or
rewrite resource_budget. Omitting it uses the registration default; a
refresh value overrides that default for this execution only.
batch_size caps rows in one Function call; LanceDB adapts actual call and
work sizes. For Function columns, the refresh defaults to
error_policy = 'skip': failing rows get null outputs and are recorded in the
error store, subject to failure limits (5% of evaluated rows unless set). Set
error_policy = 'fail' to stop on a row error.
See the SQL refresh options.
For a column filled by a registered Function, the Python and Rust job APIs can
override the Function’s GPU placement for this refresh:
gpu_count is the aggregate GPU budget for the refresh’s Function workers,
not a request for that many GPUs per worker. It must be positive and preserves
the deployment’s CPU and memory defaults.
gpu_class must name a class returned by list_gpu_classes(); class names
are lowercase DNS labels. Omit either option to use the Function’s registered
class or the server’s configured budget. list_gpu_classes() requires System
Usage permission, and raises NotImplementedError when the server predates
GPU class discovery.
The Rust GPU discovery and refresh controls are not in the latest Rust
release. TypeScript does not expose GPU class discovery; pass
gpuClass
and gpuCount in the refreshColumnAsync options.publication_interval_rows in Python,
publicationIntervalRows in TypeScript, or publication_interval_rows() in Rust:
Number.MAX_SAFE_INTEGER. This option requires a supporting LanceDB Enterprise
server and Function outputs without Blob types; Blob inputs are allowed.
The job APIs also take the row filter and failure handling of the SQL
REFRESH COLUMN ... WHERE ... WITH (...) form, so a refresh can be narrowed
to new rows (see what a filter recomputes) and told to skip rows
whose Function call fails:
max_failed_rows, init_timeout_seconds, num_workers and
max_executor_processes_per_worker are accepted the same way; the
SQL refresh options describe each one.
The SDK rejects a blank where, a max_failed_fraction outside 0 to 1, a
timeout that is not a positive number and a zero worker or process count
before sending the request; the server validates everything else. A LanceDB
Embedded refresh rejects every option other than batch_size, gpu_class
and gpu_count; in particular, it rejects resource_budget rather than
silently ignoring it.
When a refresh fails
What a failure keeps depends on where the refresh ran. The local, blockingrefresh_column writes the column in one commit, so a failure leaves the
column as it was. A refresh job in LanceDB Enterprise — always
refresh_column_async or REFRESH COLUMN there, never the blocking form —
commits finished batches as it goes, so some values can remain after a
failure.
Either way, recover by fixing the cause and refreshing again. The new refresh
fills rows that are still empty or whose inputs changed, and normally skips
values that were kept. That is not guaranteed: a refresh can recompute values
that were already written (for example when the record of the inputs they
were computed from was not saved), so do not count on a retry costing only the
rows that did not complete.
For a column filled by a function, the usual cause is
the worker environment rather than the data. See
environment mismatches for the
symptoms, which rarely name their cause.
Related guides
Job Framework
Waiting on, inspecting, and cancelling server-side work.
Refresh a materialized view
The same idea, for a whole derived table.
Auto Refresh
Refresh on a schedule instead of calling this yourself.