Skip to main content
Applies to: Embedded ❌, Enterprise ✅ LanceDB extensions with no DataFusion equivalent. They are what makes feature engineering reachable from SQL: a column declared by DDL starts empty, and a refresh fills it.

​
REFRESH COLUMN

Without WHERE, a refresh fills unassigned rows and recomputes values whose inputs changed, leaving other current values alone. A retry after a failure can still repeat work that was not committed. With WHERE, a refresh recomputes every matching live row, including rows that already have current values. Repeating the same predicate can repeat the Function calls and their cost. To fill only null outputs within a slice, add the output column to the predicate:
This skips non-null outputs even if their inputs changed. It also selects outputs that were successfully computed as NULL; a null value does not necessarily mean the row has never been computed. Use an unfiltered refresh to let LanceDB track which outputs need updating. An initialized Function instance and its mutable state may survive across refresh jobs. Do not assume a new refresh resets instance state or calls __init__ again. Instances are not shared across workers. Refresh computed columns in dependency order. A refresh fails before writing if the rows it would compute read another computed or Function column that has not been refreshed for its current inputs. Refresh that input first. A computed input that is current but evaluates to NULL is a valid value, not an unfilled placeholder. WHERE scopes the refresh, which is how you fill a slice and check it before committing to the whole table. It does not narrow the per-fragment input readiness checks: if a selected fragment contains unfilled computed input rows anywhere, refresh that input first and then rerun the filtered child.

​
Previewing a refresh

EXPLAIN and EXPLAIN VERBOSE take the same options and validation as REFRESH COLUMN and return DataFusion’s plan_type, plan rows:
  • logical_plan: the RefreshColumn statement.
  • physical_plan: a RefreshJobExec root carrying what the job runs with (the snapshot, worker count or Function resource budget, error policy, batch size, row timeout, how many fragments have work, which fragment is shown) over the plan each worker executes for one fragment: a RefreshStageExec stages the replacement file from the stored values and the rows StaleRowsExec selects, computed by a ProjectionExec for an expression column or a PythonUdfExec for a Function column.
Nothing runs: the plan is built against a snapshot without submitting a job, starting a Function executor or taking a table lock. A refresh started later pins its own source version. Job metadata and the final result retain the original Job source_version. If a concurrent UPDATE or compaction invalidates staged fragments, a Function declaring result_stability=input_and_context and side_effects=none can re-plan the affected connected fragments for at most three job-wide rounds. Each successor request explicitly pins the newer snapshot it plans, executes, and finalizes against. These request-local snapshots do not replace the Job’s original source pin. Unaffected staged results commit without rerunning, while effectful or per-call Functions fail with the named concurrent-write error. EXPLAIN [VERBOSE] JOB '<job_id>' renders the same tree for a submitted job, with the registry’s view on the root: state, progress and the job’s options. The tree is rebuilt from the job’s spec against the table’s current version on its branch (the root’s source=), not the snapshot the job pinned when it started, so a representative fragment can differ from what a worker ran. A job whose work is not a plan, such as an index build or a compaction, renders its root alone. The predicate also accepts _rowaddr and _rowid. To retry specific rows reported by table.function_errors(), use each record’s row_id with _rowid when it is present. This stable ID continues to identify the same logical row after compaction or another rewrite:
For records without row_id, combine fragment_id and row_offset as (fragment_id << 32) | row_offset and select those physical addresses before compaction or another rewrite moves the rows:
Only matching live rows are recomputed. A successful Function refresh removes their earlier row errors from function_errors() and SHOW ERRORS when their physical locations have not changed; rows that fail again remain listed. After compaction or another rewrite moves a row, _rowid still selects it, but its earlier error may remain listed even when the retry succeeds. Error clearing currently matches physical locations, not stable row IDs. Older error records and tables without stable row IDs may omit row_id. Errors without an individual row location, including those whose per-row detail was capped, cannot all be selected individually.

​
Options

num_workers sets a fixed worker count for expression-backed refreshes. Registered Function columns reject num_workers, even without a resource budget. Their worker count adapts within the aggregate resource ceiling. The example above allows up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the Function workers together. It does not reserve those resources for each worker or guarantee that all of them will be used. gpu_count may be set alone to use the deployment’s default CPU and memory. When setting either cpu_millis or memory_bytes, provide all three budget fields. For a CPU-only Function, use gpu_count = 0 in the complete budget:
LanceDB adjusts the amount of work sent to each worker from observed execution time. batch_size limits rows in one Function call; it does not set the worker count or the size of a checkpoint. Small initial batches are expected while the refresh learns how long rows take. GPU class names are lowercase names of at most 63 letters, digits, or hyphens. The REST backfill_column body accepts gpu_class, gpu_count, and init_timeout_seconds as top-level options with the same validation and budget semantics as SQL.

​
Hugging Face model snapshots

A Function can declare immutable Hugging Face repositories and revisions during registration. The worker prepares each declared snapshot before it creates the user Function instance:
Each repository may be declared once, and its revision must be the full 40-character commit hash. A completed snapshot stores that revision in refs/main, so the original no-revision SentenceTransformer(repo) call resolves without contacting the Hub. Model-declaring Functions run with a private HF_HOME, HF_HUB_OFFLINE=1, and TRANSFORMERS_OFFLINE=1; those environment variables cannot be overridden through the Function env. The cache is scoped to the database, repository, and revision. A code-only Function version with the same declaration reuses the snapshot, while another revision or database uses a separate entry. With computedRefresh.execution.imageCache.hostPath, workers on one node share the cache. Without a node cache it is worker-local and is not a cluster-wide model artifact store. computedRefresh.execution.imageCache.maxBytes bounds the allocated bytes; a snapshot that does not fit is rejected before user code starts. Without models, the existing behavior is unchanged: the Hub client may still request metadata or missing files, so a cached model and pinned revision alone do not guarantee offline execution.

​
Discovering GPU classes

The result lists the legacy default class first, followed by named classes. Columns are name, resource_name, gpu_model, gpu_memory_bytes, max_gpu_count_per_worker, cpu_millis, memory_bytes, and ephemeral_storage_bytes. Optional values are NULL when the operator did not configure them. A non-NULL max_gpu_count_per_worker bounds adaptive growth for one worker; NULL leaves the refresh’s aggregate GPU budget as the ceiling. The authenticated REST endpoint GET /v1/gpu/classes returns the same catalog as gpu_classes. It reports configured hardware, not current free capacity.

​
Row errors

Set error_policy on the refresh, not on the Function definition. With skip, inspect the recorded failures with SHOW ERRORS or table.function_errors() and retry the affected rows after fixing the cause. Skipping row errors does not suppress initialization, infrastructure, or publication failures. Under error_policy = 'skip', a refresh defaults to failing when more than 50% of the first 1,000 evaluated rows in a fragment part fail. It also fails if the first 100 evaluated rows fail before any successful row. max_failed_fraction or max_failed_rows replaces that default fraction; when both are present, either limit can fail the Job. The counters are evaluated rows, not Function calls. Checks continue cumulatively after the 1,000-row sample and also run when a smaller part finishes. batch_size defines call boundaries: the worker checks the shared counts after every inner Function call, so a row-capped batch cannot continue past the ceiling. The part that crosses a limit is not published; fragments committed before it remain visible. LanceDB normally sizes calls from observed row latency and targets at most 25% of the row deadline. batch_size remains a hard upper bound, not a tuning requirement. For a Function whose image declares result_stability=input_and_context and side_effects=none, a timeout or resource failure can enter range recovery, which retries smaller physical-row ranges with a fresh executor. Under error_policy = 'skip', a row timeout can enter range recovery when the Function declares side_effects=none, including when its result stability is undeclared or per_call. Ordinary vectorized batch exceptions already use the same best-effort bisection under skip. Because a failed batch has no row results to salvage, healthy calls in that batch run again and a per-call Function may return different values. A single row that still times out is written as null and recorded as one RowTimeoutError. Functions with declared effects are not replayed automatically. A quarantined terminal row is an exception to the per-part failure limits. This applies to an OOM-poison row in a replay-safe Function and to a non-interruptible row timeout in a side_effects=none Function under skip. Quarantine writes one null row as a synthetic part, so applying the per-part fraction or initial-100-row guard there would reject every quarantine. Before bisecting, the supervisor retries the OOM-killed claim once by itself with one executor process on the largest eligible worker class; a row timeout reaches quarantine only after its execution range has been reduced to one row. The skipped row still counts toward the cumulative job-level failure limits, and the job fails if the quarantined rows cross either limit.

​
Long-running Functions

Source Functions default to a 300-second deadline per row. Set row_timeout_seconds high enough for the longest row, for example:
The batch watchdog accommodates the row deadline and the number of input rows, so a row allowed an hour can run past 15 minutes. In vectorized mode, the row deadline applies to each vectorized call. Automatic sizing starts with a small probe and adjusts later calls; an explicit batch_size only lowers the ceiling. A source Function can call heartbeat() from the runtime-provided lance_progress module inside __call__ to report liveness. Heartbeats do not extend the row deadline. init_timeout_seconds bounds only the interval from Initialize until the worker receives Initialized; image preparation, Secret resolution, and cleanup have their own bounded task handling. While a worker is waiting for Initialized, SHOW JOBS progress reports the initializing phase and initialized worker counts. It does not count endpoint readiness as initialization completion. If the budget expires, the refresh fails with an initialization-stage error and the bounded child stderr, even when error_policy = 'skip'. The row deadline does not extend the job’s execution limit. On AWS deployments that vend temporary storage credentials, a Function refresh attempt is capped at 11 hours 40 minutes (42,000 seconds). A shorter deployment or job limit still applies; increasing the deployment timeout cannot lift this credential limit. Split longer work into disjoint filtered refreshes, keeping in mind that each predicate recomputes all matching rows. Other deployments use their configured job limit. Automatic column refreshes also have a 24-hour execution limit; an explicit refresh is still subject to the deployment and credential limits.
num_workers is refused for a column filled by a registered function; a Function refresh’s worker count follows its own resource budget instead. error_policy (and the other Function-only options in the table above) is the reverse: it’s refused for a plain SQL-expression column, which has no per-row failures to skip, and accepted only for a Function column.

​
Automatic refresh timing

Automatic column refresh requires a supporting server release and an enabled deployment. A column’s auto_refresh_interval_seconds sets a minimum delay after its last successful refresh, not a completion deadline. The scheduler also waits for its computed inputs, available workers, and other refreshes. The deployment default allows one column refresh in flight per table. Failed automatic Function refreshes start with a 15-minute retry delay by default, then double it for consecutive failures: 15, 30, 60, 120 minutes, and so on. This is failure backoff, not a fixed refresh cadence. A manual refresh can run during that wait. Cancelling a Function refresh blocks automatic refresh for that binding on the same branch until a later explicit refresh completes or the binding is replaced. Check job history when automatic results stop appearing.

​
REFRESH MATERIALIZED VIEW

The syntax and options below apply to Enterprise deployments that support explicit full = false in SQL.
A refresh updates the view’s stored result. Refresh views in dependency order, starting with their source views. CASCADE is not supported.
Servers without SQL refresh option support require the basic statement without WITH. Older servers may accept only full = true; some also accept CASCADE. A local SDK’s refresh() options do not determine which SQL options a server accepts.

​
Watching a refresh

When a client requests an early job handle and the server can make the job registry available to it, the refresh returns a job_id with status submitted. The work then outlives the request. Track it with SHOW JOBS, and diagnose failures with SHOW ERRORS. The registry APIs require both System Usage and System Operate; refresh access scoped to a database or table is not enough. Submitting a Function refresh creates work; it does not reserve Function worker capacity or guarantee that a warm worker is available. A queued job reports a current admission observation when the dispatcher has one, but submission has no global queue position or start-time estimate. CREATE MATERIALIZED VIEW returns (operation, name, status, job_id). The original three columns keep their positions; job_id is appended and is empty when the statement ran natively without a registry job. When the caller lacks both system privileges above, the server omits the early job-handle acknowledgement and the query continues through ordinary adaptive polling. Continue polling the returned descriptor until it completes, then read the result; the final refresh success or error is delivered through that path. When the bounded retained-run store is full, the server runs the refresh synchronously and reports its actual outcome without announcing a change of mode. A client must use the early acknowledgement and status submitted, rather than the presence of job_id alone, to decide that it received an early registry handle. Published view results remain visible whether the refresh was polled or run through the synchronous capacity fallback. Use SHOW JOBS FOR JOB '<job_id>' to discover a refresh’s child jobs and inspect their phase, row progress, and attempt information. Pass a returned child ID to the same event command used for the parent:
Parent events do not include child events. Child diagnostics can expire at archival; the CLI reports unavailable history even when no rows remain. While a supervised refresh runs, SHOW JOBS includes a progress column containing a JSON sample: rows planned, computed, skipped and remaining, published and retained, failed rows, fragments planned and completed, active workers and active fragments, and the last row-progress timestamp when available. The same progress object is returned by POST /v1/jobs/describe and POST /v1/jobs/list. Samples update approximately every 30 seconds and after tracked progress changes. While at least one worker endpoint is ready and the latest sampled worker phase is preparing, initializing, or initialized, startup samples are published approximately every second so short initialization transitions are more likely to be observed. They are absent before the first sample and while the job is paused. Succeeded, failed, timed out, and canceled jobs keep the latest sample from the execution that ended, when one was recorded. sample_at_millis and execution_epoch identify that sample. A client may estimate remaining time from two observations only when the epoch and planned work are unchanged and accounted work advances; an estimate is unknown at startup, after a pause or epoch change, and when either sample is missing or stale. The Python SDK exposes a defensible estimate as estimated_remaining_time_millis and omits the field when the evidence is insufficient; the server does not persist or return a historical sampler. The optional worker_fleet object reports the Function-role worker cohorts from the supervisor’s latest completed status observation. desired is the policy target for Function workers and can exceed the cohorts that have been created. launched counts surviving workers whose cohort launch was confirmed, pending_workers counts observed active workers not yet ready to accept dispatch, including workers still initializing. ready counts distinct endpoints in that launched set that can accept dispatch after status and worker announcements are merged. When an observation does not provide a usable pending count, readers may derive it from active - ready; this fallback only describes currently visible endpoint evidence and is not a guarantee about workers that have not yet become visible. resource_classes lists the observed class names. Draining, reaping, and dead workers are excluded consistently. requested_gpus totals the GPU devices requested by the observed Function cohorts; ready_gpus totals devices on dispatchable endpoints. Neither number is free cluster capacity or a queue position. warm_reused is true when at least one observed cohort was proven to come from a reusable pool. It is false only when every observed cohort was proven cold, and absent when no cohort was observed or the observations are incomplete. A true value does not guarantee that every worker in the fleet avoided a cold start. worker_fleet is absent until a coherent Function-worker observation exists. first_launched_at_millis, first_ready_at_millis, and last_ready_at_millis record milestones for the current supervisor run. Readiness means a dispatchable endpoint, not that Python initialization or the first function call has completed. When no dispatchable endpoint exists, worker_fleet.wait_reason may carry a fixed substrate-derived wait reason; it is cleared once an endpoint is ready or the job reaches a terminal state. The top-level SHOW JOBS.phase reports queued, paused, starting, waiting_for_workers (no Function workers have launched), and starting_workers (workers have launched but no endpoint is ready). Once an endpoint is ready, it mirrors the nested progress.phase, falling back to running until the first worker sample. The nested phase reports the observed worker phase: preparing before the initialization handshake, initializing while the worker is inside the Initialize handshake, initialized after a successful Initialized response, running while processing rows, finalizing while publishing, and staged after local output is staged. The top-level phase becomes committing once all planned rows are accounted for. Terminal jobs report finished, failed, or cancelled when the outcome is known. workers_initializing and workers_initialized count distinct live workers at those handshake boundaries; endpoint readiness alone does not increment either counter. These are sampled observations, so a short phase may be missed, and neither phase claims that every worker is in the same state. queue_reason is populated only from a current substrate observation; in particular, before a Function supervisor starts, the dispatcher may report that the job is waiting for executor capacity and include the worker slots requested and currently available. Those values are a point-in-time admission observation. A GPU-capacity reason requires PodScheduled=False evidence naming the requested GPU resource. CPU pressure, taints, and image-pull failures are not relabeled as GPU capacity waits. The object does not report global queue position or an ETA. rows_computed counts evaluated rows, including work later recomputed. rows_published counts evaluated rows backed by accepted publication receipts at the sample. rows_retained counts completed staged output that no accepted receipt has published yet; it is a sampled lower bound, not an exact final cancellation count. rows_failed is present only when a successful terminal result carries a typed failed-row count; it is not inferred from in-flight or non-successful job state. active_workers counts owners of execute claims, including workers that have not reported their first batch yet; it is separate from worker_fleet.ready. Job listings return the most recently created jobs first across tables. SHOW JOB SPLITS '<job_id>' returns completed splits and, while the refresh runs, the latest task for each split that has not completed. state preserves the worker’s serialized phase, including phases introduced by a newer server; queued, assigned, initializing, running, failed, and completed do not share a closed mapping in older clients. Measurements that live evidence has not established are NULL, never a fabricated zero. error_count counts explicit failed execute completions for the current task identity in the current execution epoch, including failures that happened before a later retry or success. Expired leases are shown as queued; retries, expiry, success, replaced tasks, skipped rows, and row-level errors do not increment error_count, and they do not erase failures already recorded. Canceling a refresh leaves already published results visible. A later refresh of the same definition may reuse completed staged parts while its inputs and source provenance still match; parts affected by newer inputs or writes are recomputed or rejected under the concurrency rules above. Dropping a Function output column retires its binding and automatically cancels running and queued refreshes for that binding on the same table generation and branch. Failed drops and IF EXISTS drops that remove nothing leave refreshes running. Refreshes for other bindings are unaffected. SHOW JOB SPLITS '<job_id>' reports, per split: task, split and fragment identifiers, the worker that ran it, its row range, rows skipped, bytes read and staged, part and attempt counts, batch count, start and finish times, state, error_count, and the executor batch latency as p50/p95/p99/max in milliseconds. A split the supervisor recovered from a checkpoint rather than a worker report carries recovered = true. POST /v1/jobs/query_splits reports completed split records only — not the live, not-yet-completed rows the SQL form returns while the refresh is still running — and its records carry a different shape: no state or error_count, and a raw latency histogram in place of the SQL form’s precomputed quantiles.
DESCRIBE JOB '<job_id>' — also spelled SHOW JOB SUMMARY '<job_id>', or abbreviated DESC JOB — shows where a finished job spent its time, one line per row as EXPLAIN shows a plan: its totals, a timeline of its phases, and for a phase that runs on many workers at once, what their time went to. It works for any job; for a Function refresh the phases are queued, planning, starting workers, running splits and committing. A job still running returns no rows. FORMAT TABLE returns the same profile as typed rows, with unit saying whether millis is wall time or worker time, and FORMAT JSON as one JSON document.
Across refreshes, the system.public.function_refresh_metering view holds one row per succeeded refresh with the same summary, once the job events export has run. It needs system Usage and Select on the view. Compare a Function’s versions, or follow one over time:

​
See also

Refresh a computed column

The same operation from the SDKs, and what a refresh guarantees.

Table

Declaring the columns this fills.