Skip to main content
Enterprise only
Refresh a computed column to fill new rows and recompute values whose inputs changed. The example below continues with the doubled column from the computed-column guide.
Computed columns require LanceDB Enterprise and the Enterprise client version.
table.refresh_column(...) (sync) is also a local, blocking form, but it raises on an Enterprise connection — use refresh_column_async, shown above, or the SQL form instead. See Run a refresh on Enterprise below for what the returned job supports.
The refresh commits ordinary table versions (the values, then a metadata-only record of the inputs they were computed from), so versioning applies to it like any other write.
The SQL form is REFRESH COLUMN, with the same resource options.

What a refresh recomputes

Without a WHERE predicate, a refresh fills unassigned rows and recomputes values whose inputs changed since they were computed. Every other row is left alone, so a second refresh with nothing changed does nothing. This supports two common workflows. Appending is cheap. Rows added since the last refresh are filled by the next one; filled rows whose inputs are unchanged are skipped. Backfilling a column and then ingesting more data does not mean recomputing the backfill. A filled value catches up with its input at the next refresh. If doubled derives from x and a row’s x is later updated, doubled keeps its old value until the next refresh recomputes it. That is what makes the call idempotent and safe to retry. However, unlike a spreadsheet formula, nothing is recomputed until you refresh, so a derived column is stale between an update and the next refresh.

Automatic refresh

A computed column can refresh itself on a schedule instead of waiting for an explicit call here. See Auto Refresh for enabling it, the SQL and SDK forms for changing the setting, and how catch-up works when it’s re-enabled after being off.

Refreshing part of a table

Python where= requires the Enterprise client and an Enterprise server with filtered refresh support. Stable 0.41.0 does not accept this argument. The SQL form is available independently of that Python argument on supporting servers. Enterprise A refresh can be scoped to a predicate, with where= in Python or WHERE in SQL, which is useful for filling a slice first and checking it before committing to the whole table:
A filtered refresh recomputes every matching live row, including rows whose outputs are already current. Repeating the same predicate repeats those Function calls and their cost. To select only null outputs within a slice:
This skips non-null outputs even when their inputs changed, and can recompute outputs that were successfully computed as NULL. Use an unfiltered refresh to let LanceDB track which outputs need updating.
Enterprise job type: computed_refresh for a SQL-expression column, function_refresh for one filled by a registered function.

Re-score selected list elements

WHERE selects rows, not elements within a list. Refreshing a row that contains ten answers passes the whole list to the Function. A mask can control which elements your code computes, but the Function still returns one complete list for that row. There is no element-level refresh or partial-list write API. This example keeps sample_id as the link to an existing samples table with a Blob column named media. It creates only answer metadata and scores; it neither reads nor copies Blob payloads. Run it on a deployment supporting @function, list signatures and Python where=, with samples already containing sample IDs s1 and s2.
Here length is a deterministic stand-in for a scorer. Only "maybe" is scored; the other scores are copied from previous. The final equality checks Blob descriptors on an otherwise unchanged source table, without fetching payloads. Keep the source table unchanged during this comparison. Use the same model, normalization and score type as the original computation in a real re-scoring pass. For another pass, explicitly supply the intended prior scores and mask; refresh does not merge the previous output into your Function’s inputs. If answers need independent retries or versions, prefer one ordinary table row per answer, with group_id, sample_id, a stable application answer_id, answer text and score. Keep media in samples and join by sample_id when needed. A row predicate can then select exact answer IDs, and Blob storage stays in the source table. Do not use UNNEST to assume that nested Blob values become supported Function inputs; see the nested-Blob limits.

Run a refresh on Enterprise

Enterprise With an Enterprise cluster that enables computed columns, the refresh runs on the cluster. refresh_column raises there; use refresh_column_async, which returns a job you can wait on or poll instead of holding a connection open:
Poll with status(), resolve with wait(). status() is a point snapshot and does not raise, which is what makes it safe in a loop; wait() raises JobFailedError, JobCancelledError or TimeoutError. A long backfill can therefore be started, left, and checked on from a different process.

Admission and waiting

Submission does not reserve a supervisor or GPU immediately. A refresh can wait for source preparation/compaction, an earlier refresh, executor admission or worker capacity. A lifecycle running value from status() alone does not prove that Python code has begun executing. Use SHOW JOBS and the Console’s job detail to inspect available phase and row progress. On SDK/server versions with job.progress(), the snapshot can also include phase, queue_reason, timestamps and row counters. These optional fields are version-dependent; missing values are unknown, not zero progress or an ETA. creation_ms is submission time, not worker start time. DESCRIBE JOB provides the finished run report, rather than a live queue-position API. Limits are deployment settings: the Agent dispatcher concurrency, executor capacity, per-table automatic-refresh limit and Function-call max_concurrency control different things. There is no universal five-refresh limit per database. Ask your administrator for the effective settings and use the Enterprise operations guide for their configuration names. Repeated cancellation and resubmission does not bypass those limits. Cancellation requests a stop, including while queued; observe a terminal state before submitting replacement work. Committed work is not rolled back.

Configure workers and resources

SQL exposes the worker budget directly, which is the lever when a backfill is too slow or too expensive:
The resource values cap the Function workers together: this example allows up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the fleet. They are not per-worker allocations or a guaranteed worker count. num_workers is only accepted for SQL-expression columns; registered Function columns reject it. In particular, gpu_count = 2 does not request two devices inside one Python process or enable tensor parallelism. Inspect configured GPU classes and their per-worker limits with db.list_gpu_classes() or SHOW GPU CLASSES. Choose a model that fits the worker shape supported by your deployment; see GPU placement and budgets. The SDKs accept the same nested budget. CPU and memory are required; the GPU count inside the budget defaults to zero, so a CPU-only Function can omit it:
On current SQL servers, the equivalent REFRESH COLUMN ... WITH (...) form still takes all three fields when CPU or memory is set; use gpu_count = 0 for a CPU-only Function. The SDK normalizes an omitted nested GPU count to a numeric zero before sending the request. The existing top-level gpu_count / gpuCount option remains the shorthand for changing only the aggregate GPU budget. It cannot be combined with the complete resource budget. Omitting both preserves the server’s existing defaults. These examples configure only the explicit refresh shown. A registered Function declaration can separately set auto_refresh_resource_budget on add_columns; that declaration controls the budget for its automatic refreshes. SQL-expression columns reject the automatic resource-budget option because they do not use Function workers. To override the registered Function’s call concurrency for this logical execution only:
max_concurrency caps user-code calls in flight for one Function within one logical execution. It is not an account-wide quota and does not resize or rewrite resource_budget. Omitting it uses the registration default; a refresh value overrides that default for this execution only. batch_size caps rows in one Function call; LanceDB adapts actual call and work sizes. For Function columns, the refresh defaults to error_policy = 'skip': failing rows get null outputs and are recorded in the error store, subject to failure limits (5% of evaluated rows unless set). Set error_policy = 'fail' to stop on a row error. See the SQL refresh options. For a column filled by a registered Function, the Python and Rust job APIs can override the Function’s GPU placement for this refresh:
gpu_count is the aggregate GPU budget for the refresh’s Function workers, not a request for that many GPUs per worker. It must be positive and preserves the deployment’s CPU and memory defaults. gpu_class must name a class returned by list_gpu_classes(); class names are lowercase DNS labels. Omit either option to use the Function’s registered class or the server’s configured budget. list_gpu_classes() requires System Usage permission, and raises NotImplementedError when the server predates GPU class discovery.
The Rust GPU discovery and refresh controls are not in the latest Rust release. TypeScript does not expose GPU class discovery; pass gpuClass and gpuCount in the refreshColumnAsync options.
Completed Function outputs publish with their fragment by default. To opt into a row-count publication threshold, set publication_interval_rows in Python, publicationIntervalRows in TypeScript, or publication_interval_rows() in Rust:
The threshold is opt-in and must be greater than zero. The number of rows in an individual publication can differ from this value depending on completed work and Function batching; this is not an exact-N contract. Omit it to keep whole-fragment publication. In TypeScript, the threshold must be a positive safe integer, at most Number.MAX_SAFE_INTEGER. This option requires a supporting LanceDB Enterprise server and Function outputs without Blob types; Blob inputs are allowed. The job APIs also take the row filter and failure handling of the SQL REFRESH COLUMN ... WHERE ... WITH (...) form, so a refresh can be narrowed to new rows (see what a filter recomputes) and told to skip rows whose Function call fails:
max_failed_rows, init_timeout_seconds, num_workers and max_executor_processes_per_worker are accepted the same way; the SQL refresh options describe each one. The SDK rejects a blank where, a max_failed_fraction outside 0 to 1, a timeout that is not a positive number and a zero worker or process count before sending the request; the server validates everything else. A LanceDB Embedded refresh rejects every option other than batch_size, gpu_class and gpu_count; in particular, it rejects resource_budget rather than silently ignoring it.

When a refresh fails

What a failure keeps depends on where the refresh ran. The local, blocking refresh_column writes the column in one commit, so a failure leaves the column as it was. A refresh job in LanceDB Enterprise — always refresh_column_async or REFRESH COLUMN there, never the blocking form — commits finished batches as it goes, so some values can remain after a failure. Either way, recover by fixing the cause and refreshing again. The new refresh fills rows that are still empty or whose inputs changed, and normally skips values that were kept. That is not guaranteed: a refresh can recompute values that were already written (for example when the record of the inputs they were computed from was not saved), so do not count on a retry costing only the rows that did not complete. For a column filled by a function, the usual cause is the worker environment rather than the data. See environment mismatches for the symptoms, which rarely name their cause.

Related guides

Job Framework

Waiting on, inspecting, and cancelling server-side work.

Refresh a materialized view

The same idea, for a whole derived table.

Auto Refresh

Refresh on a schedule instead of calling this yourself.