Skip to main content
Enterprise only
LanceDB extensions with no DataFusion equivalent. They are what makes feature engineering reachable from SQL: a column declared by DDL starts empty, and a refresh fills it.

REFRESH COLUMN

Without WHERE, a refresh fills unassigned rows and recomputes values whose inputs changed, leaving other current values alone. A retry after a failure can still repeat work that was not committed. In the next Enterprise release, a successful filtered pilot records its rows as current. A later unfiltered refresh skips those successful rows while their inputs and Function binding remain unchanged. For example, a 40-row pilot with 39 successes on a 400-row table leaves 361 rows to evaluate in the full refresh. Repeating an explicit WHERE still forces the matching rows to run again. A completed part or calculated physical range is staged work, not a table commit by default. Without publication_interval_rows, results for a fragment become visible only after all split work selected for that fragment in this refresh is complete, including any planned physical ranges that return no rows, and the fragment is published. With the option set, completed durable parts publish in an intermediate table version once the oldest unpublished parts reach the row threshold, even while their split is still running. Cheap Functions cover many rows per wave, while a single slow call becomes readable after that call returns a durable part. The finalizer still publishes the completed fragment after all selected split work is complete, and no in-flight call is promised if it has not returned a durable part. Blob v2 Function outputs keep one final publication per fragment. An atomic binding rebind also keeps one final publication per fragment. With WHERE, a refresh recomputes every matching live row, including rows that already have current values. Repeating the same predicate can repeat the Function calls and their cost. To fill only null outputs within a slice, add the output column to the predicate:
This skips non-null outputs even if their inputs changed. It also selects outputs that were successfully computed as NULL; a null value does not necessarily mean the row has never been computed. Use an unfiltered refresh to let LanceDB track which outputs need updating. A separate WHERE request is a separate Job. It pays planning and Job startup overhead, can repeat Function image and Secret preparation, and pays commit overhead in addition to recomputing every matching row. Splitting work into predicates can isolate a slice or keep a Job bounded, but it is not a free checkpoint of the original Job. An initialized Function instance and its mutable state may survive across refresh jobs. Do not assume a new refresh resets instance state or calls __init__ again. Instances are not shared across workers. Refresh computed columns in dependency order. A SQL computed column’s refresh fails before writing if the rows it would compute read another computed or Function column that has not been refreshed for its current inputs. Refresh that input first. A computed input that is current but evaluates to NULL is a valid value, not an unfilled placeholder. WHERE scopes the refresh, which is how you fill a slice and check it before committing to the whole table. It does not narrow the per-fragment input readiness checks: if a selected fragment contains unfilled computed input rows anywhere, refresh that input first and then rerun the filtered child. A SQL computed column can opt into the same partial progress. Add WITH (ready_rows_only = true) to compute only rows whose computed inputs are assigned and current; rows still waiting on an unfilled input, a failed Function row, or a stale input fragment stay unchanged. A computed input that is current but evaluates to NULL is an assigned value and is computed, not deferred. The terminal result reports the deferred count in rows_remaining; refreshing the input and rerunning the child completes them. Without the option, the strict per-fragment readiness check above remains the default. A Function column’s refresh does not fail on an unfilled input. It defers each selected row whose computed input is NULL, or whose nullable input column is NULL for a non-nullable parameter, and computes the other rows. Deferred rows are neither assigned nor failed, and they stay retryable. The successful result counts them in rows_deferred and names the NULL inputs in deferred_inputs; see Watching a refresh. For a computed input, refresh that column first, then rerun this refresh. rows_deferred counts only rows the refresh selected: after a refresh, a rerun without WHERE does not select rows whose computed input is still NULL until that input changes.

Previewing a refresh

EXPLAIN and EXPLAIN VERBOSE take the same options and validation as REFRESH COLUMN and return DataFusion’s plan_type, plan rows:
  • logical_plan: the RefreshColumn statement.
  • physical_plan: a RefreshJobExec root carrying what the job runs with (the snapshot, worker count or Function resource budget, error policy, batch size, row timeout, how many fragments have work, which fragment is shown) over the plan each worker executes for one fragment: a RefreshStageExec stages the replacement file from the stored values and the rows StaleRowsExec selects, computed by a ProjectionExec for an expression column or a PythonUdfExec for a Function column.
Nothing runs: the plan is built against a snapshot without submitting a job, starting a Function executor or taking a table lock. A refresh started later pins its own source version. Job metadata and the final result retain the original Job source_version. If a concurrent compaction only moves rows without changing their values, the refresh can reuse complete, successfully staged non-Blob outputs through stable row IDs or a verified compaction row mapping. Rows outside the refresh selection keep their current values. This reuse makes no Function calls, so it also works for effectful or per-call Functions and does not consume the three-round recomputation limit. An input or output UPDATE invalidates reuse for the affected fragments. If reuse cannot be verified, staged coverage is incomplete, or outputs contain Blob values, recovery requires recomputation. Only Functions declaring result_stability=input_and_context and side_effects=none can recompute the affected fragments, for at most three job-wide compute rounds. Effectful or per-call Functions fail with the concurrent-write error when recomputation is required. Unaffected staged results commit without rerunning. Reuse keeps row mappings in supervisor memory for the affected publication wave. Large waves can require more supervisor memory; the refresh’s worker memory_bytes budget does not cap this memory. Each successor request pins the newer snapshot it plans, executes, and finalizes against. These request-local snapshots do not replace the Job’s original source pin. EXPLAIN [VERBOSE] JOB '<job_id>' renders the same tree for a submitted job, with the registry’s view on the root: state, progress and the job’s options. The tree is rebuilt from the job’s spec against the table’s current version on its branch (the root’s source=), not the snapshot the job pinned when it started, so a representative fragment can differ from what a worker ran. A job whose work is not a plan, such as an index build or a compaction, renders its root alone. The predicate also accepts _rowaddr and _rowid. To retry specific rows reported by table.function_errors(), use each record’s row_id with _rowid when it is present. This stable ID continues to identify the same logical row after compaction or another rewrite:
For records without row_id, combine fragment_id and row_offset as (fragment_id << 32) | row_offset and select those physical addresses before compaction or another rewrite moves the rows:
Only matching live rows are recomputed. A successful Function refresh removes their earlier row errors from function_errors() and SHOW ERRORS when their physical locations have not changed; rows that fail again remain listed. After compaction or another rewrite moves a row, _rowid still selects it, but its earlier error may remain listed even when the retry succeeds. Error clearing currently matches physical locations, not stable row IDs. Older error records and tables without stable row IDs may omit row_id. Errors without an individual row location, including those whose per-row detail was capped, cannot all be selected individually.

Checking column status

SHOW COLUMN STATUS reports one exact four-way partition for each Function-backed output:
Each row includes the pinned table version, Function identity and exact version, rows, fresh, stale, never_computed, failed, the latest matching refresh Job, and the automatic-refresh state. The four counts always add up to rows. A value that is legitimately NULL is still an assigned value; it is not counted as never computed unless the Function attempt skipped that row. This also applies to a nullable scalar output: a computed NULL is assigned and fresh, while an unrefreshed, deferred, or skipped row remains unassigned. The REST API exposes the same read-only result:
column, branch, and version are optional. Omit version to pin the latest version once for the request. Every status in a response describes that same table version. Use each result’s table_version as the authoritative snapshot version. The shared response middleware can observe a newer live handle, so the x-lancedb-version header need not equal that pinned version. If a Function publication is committed but its freshness sidecar is still finalizing, REST returns 503 with error code function_column_status_finalization_pending; clients such as wait_fresh retry this transient state. The API returns an explicit unavailable error when the stored evidence cannot prove the exact four-way partition. It does not substitute zeros or partial counts. The automatic-refresh projection includes fields the Query Node can prove. When the authoritative Lance Agent scheduler decision is unavailable, auto_refresh.state is unobservable, next_run_at_millis is null, and next_run_at_reason is authoritative_scheduler_projection_unavailable. The configured enabled and interval_seconds fields do not by themselves prove a next run.

Function recovery defaults

New Function refreshes default to error_policy = 'skip' with max_failed_fraction = 0.05. Successful output is committed before the job reports a threshold failure. Explicit fail retains terminal failure semantics; initialization, infrastructure and publication errors still fail. SHOW ERRORS ON table and function_errors('table', 'column') expose current unresolved failures with grouped causes, cumulative attempts and failure times. Successful retries clear current failures; SHOW ERRORS HISTORY ON table selects separately retained historical observations. Exact failed-row retries, version upgrades and quarantine require the matching Enterprise client. Quarantine is released by an admitted retry, so rejected requests preserve it. New materialized views use the same skip/5% default. Their successful output is published before threshold enforcement. Counts and current diagnostics belong to the committed compute attempt; abandoned replay work is excluded. Existing views retain their stored error policy.

Options

num_workers sets a fixed worker count for expression-backed refreshes. Registered Function columns reject num_workers, even without a resource budget. Their worker count adapts within the aggregate resource ceiling. The example above allows up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the Function workers together. It does not reserve those resources for each worker or guarantee that all of them will be used. Execution slots can run splits from different fragments concurrently. Global throughput across all slots is not a reliable estimate of how long one fragment will take: that fragment can use only the slots scheduled to its splits and competes with work on other fragments. Use per-fragment and split progress when estimating a particular fragment. A refresh also plans one shared tail wave of splits beyond the slots its budget admits, so a worker that finishes its share first can take remaining work instead of idling while the last splits run; the split floor and the job’s split budget still cap the count. gpu_count may be set alone to use the deployment’s default CPU and memory. Set cpu_millis and memory_bytes together to define an aggregate CPU and memory ceiling. When they are set and gpu_count is omitted, it defaults to zero, so a CPU-only refresh does not need to spell out the GPU limit:
Provide at least one GPU explicitly for a GPU Function. Supplying only one of cpu_millis and memory_bytes is rejected. A GPU Function also requires an explicit finite row_timeout_seconds on its definition or on the refresh. Without one, the refresh is rejected before a model call starts; LanceDB does not apply a hidden GPU default. CPU Functions keep the 300-second default. LanceDB adjusts the amount of work sent to each worker from observed execution time. batch_size limits rows in one Function call; it does not set the worker count or the publication_interval_rows threshold. Small initial batches are expected while the refresh learns how long rows take. batch_size does not by itself batch a scalar model call. A GPU Function whose __call__ accepts one row is invoked once per row, even when the refresh allows a larger call. That repeated Python/model-call boundary leaves a GPU idle between small operations and is substantially slower than a vectorized callable. Annotate every input as pyarrow.Array, or take one pyarrow.RecordBatch, and batch inside the callable. The MiniLM example shows the vectorized form. The Function/refresh batch_size and the model library’s own batch size are two different limits. The first bounds rows in one Function call. The second controls how many of those rows the model processes together. Set both when the model library supports its own batch size. publication_interval_rows opts into intermediate publications for Function-backed columns with non-Blob outputs. Blob inputs are allowed; Function outputs that contain Blob v2 are rejected. It is a row-count threshold over durable completed parts, not a time interval. A wave publishes the oldest unpublished parts once their accumulated rows reach the threshold; parts in a split may publish before that split completes. Unset keeps one final publication per fragment. The finalizer still publishes the remaining rows and completes the fragment. This SQL option is available in this server build. The Python client option is a separate SDK change and is not part of the current released client; SQL and SDK must match. GPU class names are lowercase names of at most 63 letters, digits, or hyphens. The REST backfill_column body accepts gpu_class, gpu_count, init_timeout_seconds, job_timeout_seconds, and publication_interval_rows as top-level options with the same validation and budget semantics as SQL.

CPU budgets inside Functions

os.cpu_count() and os.sched_getaffinity(0) can report the host node’s CPUs even when a Function worker has a smaller CPU quota. Pools created without an explicit size, including multiprocessing.Pool() and ProcessPoolExecutor(), can therefore start too many processes. LanceDB sets LANCE_FUNCTION_CPUS in each Function executor’s environment at startup. It is the worker’s available CPU count, accounting for its cgroup quota and affinity, divided by the number of executor processes sharing the worker, rounded down with a minimum of 1. For example, a 4-CPU worker with one executor receives 4; with four executors, each receives 1. This is a per-executor budget, separate from the aggregate refresh cpu_millis ceiling. Use this value to size pools and library worker counts explicitly:
Native thread pools default to the same budget through OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, and RAYON_NUM_THREADS; explicit Function settings for those variables take precedence. When combining a process pool with threaded libraries, also limit each child process’s native threads so their combined concurrency fits the budget.

Retrying temporary Function errors

Set retry_policy on a source Function at registration, or override it for one refresh. The same JSON object is accepted by CREATE FUNCTION ... WITH (retry_policy = '...') and by the registration and refresh REST APIs.
max_attempts includes the first call and accepts 1–100. An explicit {"max_attempts":1} disables user-code retries, including the built-in transient retry. Omitting the option preserves the Function default and existing behavior. Choose one or more matchers; matching any one allows retry: The Botocore transport exceptions are ReadTimeoutError, ConnectTimeoutError, and EndpointConnectionError from botocore.exceptions. Requests and HTTPX transport exceptions require explicit exception_types, such as ["requests.exceptions.ConnectionError", "httpx.TransportError"]. The runtime also checks wrapped causes. It does not infer a transient error from digits such as 429 appearing in an arbitrary message. An adapter may signal throttling with status_code or code, and retry_after_seconds. Message patterns must compile under both Rust’s regex validator and the Function environment’s Python re module. Retries use exponential backoff with full jitter. initial_backoff_seconds defaults to 0.2 and max_backoff_seconds to 30; both must be finite, between 0 and 3600, with the initial value no larger than the maximum. honor_retry_after defaults to true and honors HTTP Retry-After or the adapter’s retry_after_seconds. A requested delay above the configured maximum ends retry instead of calling the provider early. Cancellation interrupts backoff. A row Function has one retry budget per row. A batch Function can raise for the whole call, or it can return a list or tuple that contains one output value or one Exception for every input row. A returned exception is a row outcome: successful values are retained, and an explicit policy retries only rows whose marker matches. A row keeps one max_attempts budget whether its first failure was a whole-call exception or a later marker. An exception returned for a row does not repeat unrelated successful rows. With an explicit policy, a whole-call exception that returns no values follows the previous whole-call behavior and is not bisected after retries. skip retains existing outputs for every failed row in that call, leaves previously unassigned outputs unassigned, and records the error and attempts. A callable that raises before returning its complete outcome cannot provide the framework with successful values that never existed. Output conversion and partially consumed generator results are not retried. Output validation alone does not count as a call. Only opt in when repeating a partially failed call is safe. The runtime cannot undo effects a call performed before raising. Functions declaring external side effects are refused. User-code retries do not consume the worker’s infrastructure attempt budget; initialization, worker loss and OOM recovery keep their own policy. After exhaustion, fail fails the job and skip records the last error and count in SHOW ERRORS and the table errors API, subject to the normal failure limits. A Function image created before retry support must be registered as a new version before a refresh can override its policy.

Hugging Face model snapshots

A Function can declare immutable Hugging Face repositories and revisions during registration. The worker prepares each declared snapshot before it creates the user Function instance:
The Arrow annotations select vectorized execution. Each Function call receives at most the declared batch_size rows, possibly fewer while automatic sizing probes execution time, and passes its whole input list to encode once. The encode(batch_size=64) argument then batches the model’s own forward passes. Without vectorized annotations, the same class would call encode separately for every row. Build PyArrow types used by the callable inside the class method. Function packaging captures the callable source and does not accept PyArrow type objects as global values.
For GPU models, use a vectorized callable when throughput matters. Increasing only the Function or refresh batch_size does not merge scalar model calls. Row-at-a-time execution remains a compatibility mode; it is not the recommended pattern for embedding or other models that process batches efficiently.
Each repository may be declared once, and its revision must be the full 40-character commit hash. A completed snapshot stores that revision in refs/main, so the original no-revision SentenceTransformer(repo) call resolves without contacting the Hub. Model-declaring Functions run with a private HF_HOME, HF_HUB_OFFLINE=1, and TRANSFORMERS_OFFLINE=1; those environment variables cannot be overridden through the Function env. Before SQL execution or refresh worker admission, a shared preparation job imports the snapshot into database object storage. Concurrent consumers with the same database storage, pinned Secret scope, repository and revision reuse that attempt and its immutable snapshot. A code-only Function version can reuse the snapshot when these inputs are unchanged. Preparation failures are retained: waiting consumers do not automatically retry Hugging Face. An operator with System Operate privilege can retry the observed preparation job through POST /v1/jobs/retry_model_preparation, with {"job_id":"<observed preparation attempt>"}. After preparation succeeds, resubmit failed queries or refreshes; retrying the import does not restart them. Workers read prepared snapshots from database storage into a local model cache. Local cache sharing, capacity and preparation concurrency are configured by the deployment operator. Shared snapshots avoid repeated origin downloads; a cold worker still needs to read the model from database storage. Cached files do not mean that Python initialization or GPU model loading has already completed. Without models, the existing behavior is unchanged: the Hub client may still request metadata or missing files, so a cached model and pinned revision alone do not guarantee offline execution.

Discovering GPU classes

The result lists the legacy default class first, followed by named classes. Columns are name, resource_name, gpu_model, gpu_memory_bytes, max_gpu_count_per_worker, cpu_millis, memory_bytes, and ephemeral_storage_bytes. Optional values are NULL when the operator did not configure them. A non-NULL max_gpu_count_per_worker bounds adaptive growth for one worker; NULL leaves the refresh’s aggregate GPU budget as the ceiling. The authenticated REST endpoint GET /v1/gpu/classes returns the same catalog as gpu_classes. It reports configured hardware, not current free capacity.

Row errors

Set error_policy on the refresh, not on the Function definition. With skip, inspect the recorded failures with SHOW ERRORS or table.function_errors() and retry the affected rows after fixing the cause. Skipping row errors does not suppress initialization, infrastructure, or publication failures. For column refreshes, skipping a failed recompute retains the previous stored output. A row without an assigned output remains unassigned. Skipped and deferred rows remain out of date and retryable even when they retain non-null values; an unfiltered retry uses freshness state to select them. Successfully published pilot rows stay current and are not evaluated again unless their inputs or binding change or a WHERE explicitly selects them. table.function_errors(column="embedding") reads retained errors for that column across refresh jobs, including diagnostics from failed and timed-out jobs. A successful retry suppresses earlier row errors only when their physical locations still match, as described above; this is not a complete view of the rows currently failing. Pass job_id to restrict the listing to one refresh. The default limit is 10,000, and the maximum is 100,000. Row records and capped-fragment summaries share this budget. A column filter suppresses fragment summaries because those summaries do not identify a column. SHOW ERRORS ON documents [FOR JOB '<job_id>'] [LIMIT n] uses the same limits. Reaching exactly the result limit does not by itself set truncated. The flag is true when a matching entry was omitted, or when a scan budget left part of the history unexamined. The store scan visits at most limit error files; without job_id, footer diagnostics are limited to the 32 newest failed or timed-out refresh jobs. SQL reports truncation with a ListingTruncated marker row. If truncated, narrow by job_id or raise limit; there is currently no cursor for continuing the listing. The table error store covers published fragment work. A skipped row in a part that remains staged and unpublished is not written to function_errors() or SHOW ERRORS; its Job progress and split evidence can still record it. A failed Job can therefore report skipped rows with no table error records, and the error API is not a complete ledger of every attempted or staged row. When a Function worker dies or an executor is killed with SIGKILL, retained terminal records from POST /v1/table/{id}/errors can include attribution, worker_id, and task_id. Attribution describes the evidence for the death:
  • likely_cause: the task was the only executing task on an OOM-killed worker, with every claim’s activity known, or its executor was killed while the worker’s cgroup counted a new OOM kill during that session.
  • collateral: the task was not executing when its worker died.
  • unattributed: the evidence cannot identify a likely cause, for example because several tasks were executing, an attempt ledger was unreadable, or the worker died without OOM evidence.
A range identifies the inputs the task covered; it does not prove that every row caused the failure. A multi-row death record does not invent a failing row ID or Python traceback. Its row_range contains fragment-relative physical offsets, and row_address_range contains the corresponding _rowaddr bounds. Both ranges are half-open: include start and exclude end. SQL SHOW ERRORS keeps its existing columns and carries the attribution and range in the error message. Always read affected inputs at the record’s table_version. Physical row addresses can change after compaction, so applying the bounds to the latest version can retrieve different inputs. For example, to inspect up to 100 rows from a record with table_version = 12 and row_address_range from 30064771072 to 30064777700, send this body to POST /v1/table/documents/query:
The recorded version must still be retained. An absent attribution or range means that detail is unavailable; row timeout diagnostics continue to name their known row without a worker-death attribution. Jobs persisted by an earlier release keep their pinned skip policy. Under it, a refresh fails when more than 50% of the first 1,000 evaluated rows in a fragment part fail, or when the first 100 evaluated rows fail before any successful row. max_failed_fraction or max_failed_rows replaces that default fraction; when both are present, either limit can fail the Job. New refreshes default to 5% and publish successful output before enforcing their limits instead. The counters are evaluated rows, not Function calls. Checks continue cumulatively after the 1,000-row sample. A fragment part shorter than that sample does not apply the fraction: a part that retries reduced to a single row counts as one evaluated row, not as a 100% failure rate. The refresh applies the fraction to its completed splits as a whole at the end, so a tail part with one unprocessable row does not fail the rows evaluated before it. max_failed_rows applies to every part. batch_size defines call boundaries: the worker checks the cumulative counts after every inner Function call, so a row-capped batch cannot continue past the ceiling. With this per-part enforcement, the part that crosses a limit is not published and fragments committed before it remain visible; for new refreshes, successful output is published before the supervisor enforces the limit.

NaN and infinity

A Function refresh counts rows whose selected declared float result contains NaN or infinity in rows_with_nan and rows_with_inf. A row may appear in both counts. The scan includes float leaves inside vectors and structs and counts each input row once per category. The terminal Job summary carries the accepted-output total; SHOW JOB SPLITS exposes the corresponding per-split measurements. The default nan_policy = 'allow' preserves the Function’s IEEE values. Use nan_policy = 'error' with error_policy = 'skip' to write affected rows as null and record them as row failures:
nan_policy = 'null' requires the selected Function output field to be declared nullable. The computed column’s temporary nullable state while it is unassigned does not relax a non-nullable Function result. It writes affected rows as null without recording them as Function-call failures. The row counts remain available in the Job summary under either policy unless terminal recovery cannot attribute every accepted durable part; in that case both counts are NULL rather than a partial zero. LanceDB normally sizes calls from observed row latency and targets at most 25% of the row deadline. batch_size remains a hard upper bound, not a tuning requirement. For a Function whose image declares result_stability=input_and_context and side_effects=none, a timeout or resource failure can enter range recovery, which retries smaller physical-row ranges with a fresh executor. Under error_policy = 'skip', a row timeout can enter range recovery when the Function declares side_effects=none, including when its result stability is undeclared or per_call. Ordinary vectorized batch exceptions already use the same best-effort bisection under skip. Because a failed batch has no row results to salvage, healthy calls in that batch run again and a per-call Function may return different values. A single row that still times out retains its stored output or remains unassigned, stays retryable, and is recorded as one RowTimeoutError. Functions with declared effects are not replayed automatically. A quarantined terminal row is an exception to the per-part failure limits. Under skip, this applies to an OOM-poison row in a replay-safe Function, a typed OversizedRows failure for exactly one physical row after range recovery has reduced the unfinished work to that same row, and a non-interruptible row timeout in a side_effects=none Function. Oversized-row quarantine requires the executor evidence to name one physical row that matches the only unfinished row in the recovery range; multi-row or mismatched evidence does not become a skipped row. Before range bisection, OOM quarantine and oversized-row recovery require result_stability=input_and_context and side_effects=none. For a task with exactly one split whose execution range contains only that physical row, quarantine does not call the Function again. In this case, side_effects=none is sufficient even when undeclared stability resolves to per_call. Declared effects disable automatic replay. Under fail, a terminal oversized row fails the refresh without a skipped-row record. Quarantine publishes a synthetic skipped part that retains the stored output without reading the failing inputs. Previously unassigned outputs remain unassigned, and the row stays retryable. Applying the per-part fraction or initial-100-row guard there would reject every quarantine. Before bisecting, the supervisor retries the OOM-killed claim once by itself with one executor process on the largest eligible worker class; a row timeout reaches quarantine only after its execution range has been reduced to one row. The skipped row still counts toward the cumulative job-level failure limits, and the job fails if the quarantined rows cross either limit. Range recovery has a finite leaf and attempt budget. Under skip, when the failed range can no longer be narrowed, the rows recovery did not reach still get an outcome: only rows the refresh selected are recorded as skipped row errors, and they keep the failure class that exhausted recovery (RowTimeoutError for a row timeout, otherwise ResourceExhausted). Selected rows whose required input is unready stay deferred and are not recorded as failures. The fragment’s already computed parts publish with them, and each affected row retains its stored output or stays unassigned and retryable. The budget, the per-part failure limits and the fail behavior are unchanged.
Per-row Function failures can include a Python traceback, bounded to a UTF-8 prefix or omitted when the reported errors exceed the byte limit. The compact error message and skipped-row count remain complete. The persisted error file can retain a prefix of records in row order while rows_skipped covers the full count. If the compact skipped-row reports alone exceed the event limit, the refresh fails before publication. Skipped-row tracebacks require a newly baked source image.

Long-running Functions

CPU source Functions default to a 300-second deadline per row. Set row_timeout_seconds high enough for the longest row, for example:
The batch watchdog accommodates the row deadline and the number of input rows, so a row allowed an hour can run past 15 minutes. In vectorized mode, the row deadline applies to each vectorized call. Automatic sizing starts with a small probe and adjusts later calls; an explicit batch_size only lowers the ceiling. A source Function can call heartbeat() from the runtime-provided lance_progress module inside __call__ to report liveness. Heartbeats do not extend the row deadline. The 300-second fallback applies only to CPU Functions. A GPU Function must declare a finite positive row_timeout_seconds at registration or supply one on the refresh; otherwise the refresh is rejected before model execution. The Function definition reports effective_row_timeout_seconds and row_timeout_required, and an admitted Job carries the selected deadline in its spec. This makes the GPU choice visible instead of introducing a second deployment-wide default. Failed batch diagnostics report row_timeout_seconds and batch_timeout_seconds separately. These are the configured row limit and the outer batch watchdog budget, respectively. A child reporting a row timeout can exit before the batch watchdog expires; the larger batch budget does not extend the row limit. The diagnostic also reports row_timeout_source, which is default for the CPU fallback, definition for a Function declaration, or request for a refresh override, and elapsed_since_batch_start_seconds, which measures from the start of the affected input batch and excludes worker startup and queueing. When a non-interruptible multi-row batch fails as a unit, the terminal failure record names the batch boundary and leaves row_offset unset because the failing row is unknown. init_timeout_seconds bounds only the interval from Initialize until the worker receives Initialized; image preparation, Secret resolution, and cleanup have their own bounded task handling. While a worker is waiting for Initialized, SHOW JOBS progress reports the initializing phase and initialized worker counts. It does not count endpoint readiness as initialization completion. If the budget expires, the refresh fails with an initialization-stage error and the bounded child stderr, even when error_policy = 'skip'.

Inspecting a slow or stalled worker

Supporting server releases report no_progress warnings after 60 seconds without meaningful progress. A slow_split warning appears when a split has run for at least 60 seconds and five times the median execution cost per row, scaled to its row count. This comparison needs at least three completed samples and uses the most recent 128 samples. Warnings identify the worker, split when available, and last progress time. They appear in job progress and recorded events; they do not extend deadlines or restart work. An operator with the system operate privilege can request Python thread stacks from a selected worker:
Select worker_id from a warning or the job’s worker events. On supporting SDK releases, Python provides job.stack(worker_id) and await job.stack(worker_id) for asynchronous handles; Rust provides job.stack(worker_id).await?. Capture requires a supporting Linux worker runtime. The response reports captured, unavailable, or unsupported for each session, with captured_at_millis and truncated metadata. Missing frames do not prove that the worker is healthy or finished. A terminal or unreachable worker can be unavailable, and older runtimes can be unsupported. Each request samples at most eight executor sessions, retaining at most 8 KiB per session with a 250 ms capture budget. omitted_sessions counts sessions beyond that limit. Requests are limited to one per five seconds per job, and a recent sample may be reused for up to five seconds. Captures contain Python frames without source text, locals, or native frames. Timeout failures retain their reported exception traceback; sampled frames supply evidence when that traceback is missing. Inspect recorded failure events after the worker exits. The row deadline does not extend the job’s execution limit. A Function refresh runs under a whole-job deadline that defaults to the deployment’s job timeout (12 hours by default). Set job_timeout_seconds to change it for one refresh, for example:
The deadline counts from when execution starts, not from submission or from time spent queued behind another refresh. A refresh that records its own limit uses that limit; the deployment timeout is only the default when it records none. Automatic column refreshes record a 24-hour limit. Temporary table-scoped storage credentials renew before expiration, so a refresh can span multiple credential sessions without splitting the work. When the whole-job deadline fires, the Job state is timed_out and its failure names the job deadline and the value that applied. The same limit bounds the refresh’s Kubernetes Job, so the registry and the pods agree on when the job is dead.
num_workers is refused for a column filled by a registered function; a Function refresh’s worker count follows its own resource budget instead. error_policy (and the other Function-only options in the table above) is the reverse: it’s refused for a plain SQL-expression column, which has no per-row failures to skip, and accepted only for a Function column.

Functions whose memory keeps growing

A Function worker keeps each executor, and the state its __init__ built, for the parts it runs next. A Function that leaks memory therefore grows with every part until the worker is OOM-killed, and retrying the part leaks again. After each part that evaluated rows, the worker reads the executor’s resident memory other than file-backed pages, including the processes it started in its own process group. It closes the executor once the part completes when all of these hold:
  • the memory reached a new high in at least three of the executor’s last five parts
  • the memory grew by at least 1/16 of the executor’s memory budget since its first part, so a fresh executor frees a meaningful amount
  • one more part like the last could take the worker past 90% of its memory limit, counting the memory in use other than file cache
The next part on that slot starts a fresh executor, which runs __init__ again. The closed executor’s close() gets 30 seconds before it is killed. The completed part is kept, and a recycle is neither a failure nor a retry attempt. An executor whose memory holds steady is never recycled. Recycling needs a worker with a cgroup v2 memory limit; without one, executors are kept as before. Each recycle appends an executor_recycle event, shown by SHOW JOB EVENTS '<job_id>'. The successful job result reports the count as executor_recycles and up to 64 executor_recycle_details, with executor_recycles_truncated: true when some are omitted. Each detail has worker_id, session_slot, recycled_at_millis, the executor’s parts, its first_resident_bytes after its first part and resident_bytes after its last, and the worker’s worker_memory_bytes in use, excluding file cache, against its worker_memory_limit_bytes. These fields are absent when no executor was recycled. Repeated recycles of one Function point to a leak in its code.

Automatic refresh timing

Automatic column refresh requires a supporting server release and an enabled deployment. A column’s auto_refresh_interval_seconds sets a minimum delay after its last successful refresh, not a completion deadline. The scheduler also waits for its computed inputs, available workers, and other refreshes. The deployment default allows one column refresh in flight per table. Function-backed columns may also declare an auto_refresh_resource_budget when the column is added. An explicit declaration takes precedence over the deployment’s default resource budget, but it must remain within the deployment ceiling. CPU and memory must be specified together; when they are present and gpu_count is omitted, the GPU count defaults to zero. A GPU Function still requires at least one GPU in its declared budget. Without an explicit declaration, automatic refresh uses the deployment default. This aggregate budget limits the worker fleet; it is not a per-worker allocation or a direct user-call concurrency limit. Failed automatic Function refreshes start with a 15-minute retry delay by default, then double it for consecutive failures: 15, 30, 60, 120 minutes, and so on. This is failure backoff, not a fixed refresh cadence. A manual refresh can run during that wait. Cancelling a Function refresh blocks automatic refresh for that binding on the same branch until a later explicit refresh completes or the binding is replaced. Check job history when automatic results stop appearing.

REFRESH MATERIALIZED VIEW

The syntax and options below apply to Enterprise deployments that support explicit full = false in SQL.
A refresh updates the view’s stored result. Refresh views in dependency order, starting with their source views. CASCADE is not supported.
The example caps the SQL execution and its Function processes together at 8 CPU cores, 16 GiB, and 2 GPUs. These are aggregate ceilings, rather than allocations for each Function. CPU or memory requires all three budget fields; CPU and memory must be positive. A GPU cap alone is accepted. Use gpu_count = 0 for CPU-only views; GPU Functions and explicit GPU placement require a positive GPU budget when one is specified. gpu_class names follow the lowercase DNS-label rule and must identify an operator-configured class; use SHOW GPU CLASSES to discover classes. The override leaves registered Functions and saved MV placement unchanged. Omitted settings preserve deployment defaults and the MV’s existing placement. Explicit budgets and placement require an isolated execution backend that can enforce them. Unsupported backends reject constrained jobs. Resource options are retained across interruption recovery. A ceiling does not reserve its full GPU count or add parallel workers: current MV jobs use at most one GPU, shared by their GPU Function calls. These options affect this manual execution only; automatic-refresh settings remain unchanged.
Servers without SQL refresh option support require the basic statement without WITH. Older servers may accept only full = true; some also accept CASCADE. A local SDK’s refresh() options do not determine which SQL options a server accepts.

Analyze a materialized-view write

Analysis executes and commits the CREATE or REFRESH. It returns the executed physical plans and a statement summary. An unchanged incremental refresh returns a summary without recomputing rows. Per-Function metrics include successful executor-batch calls, input rows, summed time, and available batch p50/p95 latency. Runtime admission, preparation, input hydration and supervisor IPC transfer have separate measurements. Compute covers the adapter’s apply call, including conversion and lazy Blob reads. Batch latency includes child IPC and excludes admission; p50/p95 are histogram upper bounds. Missing measurements remain unavailable, including compute for older adapters. Function totals sum work across concurrent workers and differ from statement wall time. Writer time excludes upstream input waits. Completed SQL job profiles also retain per-Function aggregates in function_timings and the canonical run report.

Blob values in views and CTAS copies

CREATE MATERIALIZED VIEW ... AS SELECT and CREATE TABLE ... AS SELECT store the selected rows in a new table. Blob values stored by Lance, whether inline, packed, dedicated, or Managed, are copied as bytes into that table, including blob leaves inside structs and lists and values produced by UNNEST. Nulls and empty blobs keep their meaning. The destination storage version selects the stored encoding, so a 2.2 or newer destination writes Blob v2 even when the source uses the legacy blob encoding. The copy consumes storage for the selected payloads; SQL does not deduplicate payload objects across tables. To copy a Lance dataset on Hugging Face into your own storage, first register it with a shallow clone, then run this CTAS copy over the clone. See Import a Lance dataset with Blob values. For Managed Blob v2, the destination writes its own payload objects. It does not keep the source table’s managed object as a cross-table reference. After the statement commits, compacting, cleaning, or dropping the source does not affect the bytes the copy owns. Keep a source tag only when you also need to read the source snapshot. An external URI is not a copy of its object. SQL preserves a top-level external reference instead of fetching it; because SQL does not register source storage bases, a copy fails when Lance cannot store that reference. Nested external blobs are different: the copy reads and materializes them into destination-owned bytes, so the query must be able to access the external object. When a copy retains an external reference, the object remains owned by its own system: a tag does not pin its version, and a later read can fail or observe different bytes if the object changes.

Retaining a view or CTAS copy

A tag names a table version and keeps that version and the Lance-owned files it references, including Managed Blob objects, from being deleted by cleanup. Tag the destination, not the source, when the copy must stay readable. A tag on the source protects only the source and does not keep the destination copy. Tagging a version does not stop later refreshes or writes from advancing the table or view name. Read through the tag or check out the tagged version to get the frozen rows; the latest view name may already point to a newer refresh. If automatic refresh is enabled, disable it for a view that must remain unchanged, or accept that the latest name advances while the tag stays fixed. Compaction writes a new table version; it does not change an older tagged snapshot. Managed payload objects that the tagged version references remain protected until the tag or version is removed. Cleanup then reclaims only objects no retained snapshot references. Deleting the tag makes the version eligible for cleanup again. Source cleanup or compaction is safe for the bytes the copy owns, but not for future refreshes. A materialized view’s later refresh reads its source table again; keep the source available until refreshing is no longer required. Cleaning older source versions can force a rebuild, and dropping the source prevents refresh. A CTAS table has no such dependency. A branch is not a byte-for-byte copy. It shares the table’s history and payload files with its ancestors. A branch can keep inherited files readable while the branch exists, but it does not create a second copy, and deleting the branch or its remaining references can make inherited versions eligible for cleanup. Tag the relevant table version to retain a shared snapshot. Use a CTAS or MV result when you need destination-owned blob bytes. Creating a branch from a table version with Function-bound columns is also currently rejected; existing branches with those bindings remain readable and refreshable. To freeze a training set:
  1. Build the result with CREATE MATERIALIZED VIEW or CREATE TABLE ... AS SELECT, and verify its row count and representative bytes.
  2. Tag the destination table’s committed version. Keep the tag for as long as the training set must remain readable.
  3. If the result retains any external references rather than copied bytes, keep the external objects under their own retention policy. The table tag does not protect them.
  4. Source cleanup, compaction, or dropping the source is safe for the stored copy once it owns its bytes. A materialized view that will refresh again still needs its source table; keep it available until refreshing is no longer required. Cleaning older source versions may force a rebuild. Compacting the destination is also safe: it creates a new version and leaves the tagged version intact.
  5. Read the frozen rows by the tag or tagged version, not by the latest view name if the view continues to refresh.

Function row errors in materialized views

Choose the policy when creating the view:
New views default to error_policy = 'skip' with the 5% failure ceiling; existing views keep their stored policy. Under fail, a Function failure aborts the refresh and preserves the previously published result. With skip, a failing scalar Function input produces NULL; a failing table Function input produces no output rows. Planning, storage, worker, and commit failures propagate to normal job failure and recovery handling; they are not converted into skipped inputs. Custom Function images must report a failure as attributable to the input before it can be skipped. Errors from older images that do not distinguish callable failures from worker failures stop the refresh. Use SHOW ERRORS ON captions or the SDK’s table.function_errors() to inspect the diagnostics. SHOW ERRORS adds an input column containing JSON identifying the Function invocation and input offset. When a failed input can be traced to stored source rows, it also includes the source table, pinned version, and row address or stable row ID. Inputs computed by expressions or aggregations may have no single source row. Input offsets are not physical row offsets in the view. SDK error messages also include the source locator; use it instead of the view fragment fields for MV errors. Error records describe attempts and remain available after a later successful refresh. An incremental refresh over appended source rows leaves previously skipped inputs alone. When a change to existing source rows recomputes a block of the view, every input in that block runs again, skipped ones included. To retry them all, run:
A full refresh recomputes all inputs, including previously successful ones. The policy is fixed at creation; refresh does not accept a policy override.

Interrupted materialized-view jobs

Materialized-view creation and refresh check for overlapping work before submission and worker launch. These checks are best effort: concurrent requests can still reach storage together, where commit conflicts decide the outcome. A losing creation or manual refresh fails with a non-retryable concurrent-operation error. Automatic refresh skips an attempt when another refresh is active and checks eligibility again on a later scheduling pass. LanceDB can replay CREATE MATERIALIZED VIEW and supported REFRESH MATERIALIZED VIEW operations, including full = true, after an infrastructure interruption. Other SQL statements remain non-replayable. A replacement worker keeps the same Job, job handle, and original deadline. A counted infrastructure disruption can use one replacement worker. User cancellation and permanent SQL or Function errors terminate the Job, and a terminal Job is not restarted. Recovery can run Function code again when the data commit has not completed, so external side effects from that attempt can repeat. After the data commit, receipt recovery finishes state and catalog publication without computing the committed rows again, and it does not need to plan the original source or Function. Cancellation does not guarantee that an in-flight storage write stops; a data or catalog request that was already issued or sent may still complete. Recovery therefore does not guarantee exactly-once execution of external side effects.

Watching a refresh

When a client requests an early job handle and the server can make the job registry available to it, the refresh returns a job_id with status submitted. The work then outlives the request. Track it with SHOW JOBS, and diagnose failures with SHOW ERRORS. The registry APIs require both System Usage and System Operate; refresh access scoped to a database or table is not enough. A Function refresh that completes with failed output rows keeps the backend lifecycle state succeeded. The console displays Partially succeeded when terminal failure statistics are nonzero. Its terminal result and success event include summary.failure_stats; POST /v1/jobs/list returns the same object as failure_stats on each job row:
Split counts describe unique logical splits contributing to accepted publications. Failed attempts that recover, discarded output from replanning, and rows deferred for unready inputs do not count as failures. Percentages range from 0 to 100 and use the selected splits and rows as denominators; an empty job reports zero percent. Existing failure limits still determine whether a refresh fails outright. Older jobs or recovered records without sufficient publication provenance can omit these statistics; missing counts are unknown. A successful Function refresh that deferred rows because a required input was NULL reports them in its terminal result and success event:
rows_failed excludes deferred rows. Each deferred_inputs entry names an input column, the Function parameter bound to it, whether the column is computed, and the deferred rows on which it was NULL; a row with several NULL inputs counts under each. A refresh treats a computed input’s NULL as not yet computed, so refreshing that column first can fill those rows. Otherwise the column is nullable and the parameter is not. The job’s progress repeats both fields once the job succeeds. Both are absent when no row was deferred. A class Function’s close() failure preserves completed output and does not retry its rows or turn a successful refresh into a failure. The successful job result reports close_errors and close_warnings. Each warning has job scope, worker_id, session_slot, code, message, and an optional traceback. SHOW JOB EVENTS '<job_id>' includes a close_warning event for the failure. These fields are absent when no close errors were recorded. Warning messages and tracebacks retain at most 4,096 characters each. The result includes at most 64 warning details within a 64 KiB JSON array budget; close_errors still counts all recorded failures and close_warnings_truncated: true indicates omitted details. cleanup_warnings reports shutdown or diagnostic-collection problems, with the same count and byte budgets. cleanup_warning_count counts those problems, and cleanup_warnings_truncated: true indicates omitted details. Cleanup summaries retain the beginning and end of long messages, with an ellipsis between them. Workers scheduled for removal close their sessions before the successful result is returned. Warm workers can retain sessions for reuse across jobs, so close() can run later. A completed job result cannot include a close failure that has not happened yet. Close warnings are separate from the row-level errors returned by function_errors() and SHOW ERRORS; they do not change row failure statistics. Submitting a Function refresh creates work; it does not reserve Function worker capacity or guarantee that a warm worker is available. A queued job reports a current admission observation when the dispatcher has one, but submission has no global queue position or start-time estimate. CREATE MATERIALIZED VIEW returns (operation, name, status, job_id). The original three columns keep their positions; job_id is appended and is empty when the statement ran natively without a registry job. When the caller lacks both system privileges above, the server omits the early job-handle acknowledgement and the query continues through ordinary adaptive polling. Continue polling the returned descriptor until it completes, then read the result; the final refresh success or error is delivered through that path. When the bounded retained-run store is full, the server runs the refresh synchronously and reports its actual outcome without announcing a change of mode. A client must use the early acknowledgement and status submitted, rather than the presence of job_id alone, to decide that it received an early registry handle. Published view results remain visible whether the refresh was polled or run through the synchronous capacity fallback. Use SHOW JOBS FOR JOB '<job_id>' to discover a refresh’s child jobs and inspect their phase, row progress, and attempt information. Pass a returned child ID to the same event command used for the parent:
Parent events do not include child events. Child diagnostics can expire at archival; the CLI reports unavailable history even when no rows remain. To identify the server before retesting a deployed fix, use either SQL form:
version() identifies the server executable rather than the underlying SQL engine. SHOW SERVER VERSION returns its release, git_sha, and UTC RFC 3339 build_date. Unknown executable metadata is returned as NULL. With a supported SDK, db.server_info() (or await db.server_info() on an asynchronous connection) returns the responding query-node build, linked Lance and LanceDB versions, supported Function worker protocol, and configured query-node, job/CPU worker, SQL job, and GPU worker images. It requires System Usage permission and submits no query or Job. Older servers without GET /v1/server/info report that discovery is unsupported. During a rolling upgrade, requests can reach different query-node builds. Configured images do not establish which image an existing Job used. A completed Function refresh retains the finishing supervisor’s build in job.result["summary"]["server"] and its selected worker protocol in job.result["summary"]["worker_protocol_version"]. SQL Job results identify their worker in job.result["job_worker"]; this does not identify an overridden SQL child executable. Function refresh and SQL Job results also retain job.result["terminal_server"], the build of the process that wrote the terminal event. This is available for failures before execution and cancellation, when there may be no worker: it identifies the terminal writer, not a Function worker or SQL child. Historical results without these fields remain unknown when read by a newer server. Include these values and any server build information in an SDK exception when reporting a failure. While a supervised refresh runs, SHOW JOBS includes a progress column containing a JSON sample: rows planned, computed, skipped and remaining, published and retained, failed rows, fragments planned and completed, active workers and active fragments, and the last row-progress timestamp when available. The same progress object is returned by POST /v1/jobs/describe and POST /v1/jobs/list. Samples update approximately every 30 seconds and after tracked progress changes. While at least one worker endpoint is ready and the latest sampled worker phase is preparing, initializing, or initialized, startup samples are published approximately every second so short initialization transitions are more likely to be observed. They are absent before the first sample and while the job is paused. Succeeded, failed, timed out, and canceled jobs retain the latest row progress and timing from the execution that ended, when a sample was recorded. Terminal jobs zero live worker, GPU, and active fragment counts and clear resource_classes, warm_reused, and wait reasons. sample_at_millis and execution_epoch identify that sample. A client may estimate remaining time from two observations only when the epoch and planned work are unchanged and accounted work advances; an estimate is unknown at startup, after a pause or epoch change, and when either sample is missing or stale. The Python SDK exposes a defensible estimate as estimated_remaining_time_millis and omits the field when the evidence is insufficient; the server does not persist or return a historical sampler. The optional worker_fleet object reports the Function-role worker cohorts from the supervisor’s latest completed status observation. desired is the policy target for Function workers and can exceed the cohorts that have been created. launched counts surviving workers whose cohort launch was confirmed, pending_workers counts observed active workers not yet ready to accept dispatch, including workers still initializing. ready counts distinct endpoints in that launched set that can accept dispatch after status and worker announcements are merged. When an observation does not provide a usable pending count, readers may derive it from active - ready; this fallback only describes currently visible endpoint evidence and is not a guarantee about workers that have not yet become visible. resource_classes lists the observed class names. Draining, reaping, and dead workers are excluded consistently. requested_gpus totals the GPU devices requested by the observed Function cohorts; ready_gpus totals devices on dispatchable endpoints. Neither number is free cluster capacity or a queue position. warm_reused is true when at least one observed cohort was proven to come from a reusable pool. It is false only when every observed cohort was proven cold, and absent when no cohort was observed or the observations are incomplete. A true value does not guarantee that every worker in the fleet avoided a cold start. worker_fleet is absent until a coherent Function-worker observation exists. first_launched_at_millis, first_ready_at_millis, and last_ready_at_millis record milestones for the current supervisor run. Readiness means a dispatchable endpoint, not that Python initialization or the first function call has completed. When no dispatchable endpoint exists, worker_fleet.wait_reason may carry a fixed substrate-derived wait reason; it is cleared once an endpoint is ready or the job reaches a terminal state. The top-level SHOW JOBS.phase reports queued, paused, starting, waiting_for_workers (no Function workers have launched), and starting_workers (workers have launched but no endpoint is ready). Once an endpoint is ready, it mirrors the nested progress.phase, falling back to running until the first worker sample. The nested phase reports the observed worker phase: preparing before the initialization handshake, initializing while the worker is inside the Initialize handshake, initialized after a successful Initialized response, running while processing rows, finalizing while publishing, and staged after local output is staged. The top-level phase becomes committing once all planned rows are accounted for. Terminal jobs report finished, failed, or cancelled when the outcome is known. workers_initializing and workers_initialized count distinct live workers at those handshake boundaries; endpoint readiness alone does not increment either counter. These are sampled observations, so a short phase may be missed, and neither phase claims that every worker is in the same state. queue_reason is populated only from a current substrate observation; in particular, before a Function supervisor starts, the dispatcher may report that the job is waiting for executor capacity and include the worker slots requested and currently available. Those values are a point-in-time admission observation. A GPU-capacity reason requires PodScheduled=False evidence naming the requested GPU resource. CPU pressure, taints, and image-pull failures are not relabeled as GPU capacity waits. The object does not report global queue position or an ETA. rows_computed is the sampled number of evaluated rows, including work later retried or recomputed and work that has not been published. It is not an exact durable-output ledger, and it does not mean those rows were lost when a Job fails. rows_published counts logical evaluated rows backed by accepted fragment publication receipts at the sample; it is not the number of planned rows, nor a claim that every physical row in the fragment was recomputed. rows_retained counts the assigned rows of completed splits that no accepted publication receipt covers yet. Completed parts from a split that has not completed are not included, even if those parts are durably staged. It is a sampled lower bound and does not make rows table-visible. rows_failed is present only when a successful terminal result carries a typed failed-row count; it is not inferred from in-flight or non-successful job state. rows_deferred and deferred_inputs likewise come only from a successful terminal result, and are absent when no row was deferred. active_workers counts owners of execute claims, including workers that have not reported their first batch yet; it is separate from worker_fleet.ready. Job listings return the most recently created jobs first across tables. SHOW JOB SPLITS '<job_id>' returns completed splits and, while the refresh runs, the latest task for each split that has not completed. state preserves the worker’s serialized phase, including phases introduced by a newer server; queued, assigned, initializing, running, failed, and completed do not share a closed mapping in older clients. Measurements that live evidence has not established are NULL, never a fabricated zero. error_count counts explicit failed execute completions for the current task identity in the current execution epoch, including failures that happened before a later retry or success. Expired leases are shown as queued; retries, expiry, success, replaced tasks, skipped rows, and row-level errors do not increment error_count, and they do not erase failures already recorded. Canceling a refresh leaves already published results visible. A later refresh is not guaranteed to resume an earlier Job. It may reuse a completed staged part only when the runtime finds an existing compatible basis, source provenance, and selection, and the part remains within the bounded retention window. Otherwise the later refresh recomputes the work. Treat this as opportunistic reuse, not an automatic cross-Job checkpoint or a promise that unfinished rows carry forward. A Function output can be renamed without recomputing its rows. The column and its binding change in one commit, keeping existing values, refresh freshness, and error history.
Send Function output renames separately from type, nullability, virtual-column, or unrelated column changes. An output that another Function, computed column, or embedding function reads cannot be renamed. The new name cannot contain dots or backticks, be _rowid or _rowaddr, or start with __. Existing error records retain the column names recorded when their jobs ran; use the original column name when filtering that history. Dropping a Function output column retires its binding and automatically cancels running and queued refreshes for that binding on the same table generation and branch. Failed drops and IF EXISTS drops that remove nothing leave refreshes running. Refreshes for other bindings are unaffected. SHOW JOB SPLITS '<job_id>' reports, per split: task, split and fragment identifiers, the worker that ran it, its row range, rows skipped, rows with NaN or infinity, bytes read and staged, part and attempt counts, batch count, start and finish times, state, error_count, and the executor batch latency as p50/p95/p99/max in milliseconds. A split the supervisor recovered from a checkpoint rather than a worker report carries recovered = true. POST /v1/jobs/query_splits reports completed split records only — not the live, not-yet-completed rows the SQL form returns while the refresh is still running — and its records carry a different shape: no state or error_count, and a raw latency histogram in place of the SQL form’s precomputed quantiles.

Export a Function job timeline

On the console job page, open the Trace tab to explore the recording in the embedded viewer. It loads on demand and matches the console theme. Choose Refresh trace for a new snapshot of a running job. The bundled viewer runs in your browser and is served by the console, including in private deployments. Choose Download trace to save the recorded timeline as Chrome trace JSON. Open the file in Perfetto or Chrome tracing to inspect split spans grouped by worker, alongside timestamped planning, worker startup, and commit phases. Concurrent splits appear on separate display lanes under their worker; these lanes do not identify physical execution slots. With a compatible Python SDK and server, the same recording is available from an existing job handle:
The export reads the retained job records, including archived jobs, rather than the console’s displayed event window. A running job returns a partial snapshot without waiting for completion. Open work is marked as an observation with no invented finish time. Batch counts and latency statistics are attached to split details; individual batch timestamps and unpositioned worker-time totals are not drawn as spans. Missing or invalid timestamps remain explicit in the recording’s metadata. No new runtime recording is enabled. The console’s Stats tab shows the same server-rendered run report described below. It reads the retained terminal profile, not the console’s bounded event window. A running job shows that the report is not available yet; a terminal job without a readable profile says so explicitly rather than filling missing measurements with zero. POST /v1/jobs/stats exposes that report to clients. Its request is {"job_id": "<job_id>"}. A known job returns:
until it reaches a terminal state. At that point profile contains the typed profile and report contains the server-rendered text. Both remain null when a terminal job has no readable profile. Unknown jobs, and jobs the caller is not allowed to inspect, return 404. The route accepts failed, canceled, timed-out and archived terminal jobs, uses the same permission check as /v1/jobs/describe, and never submits, waits for, or cancels work. The text is rendered by the same server code used below for DESCRIBE JOB. With a compatible SDK, job.stats() returns that report verbatim, or None for the null case; clients must not reconstruct terminal timings from a partial event window. The structured profile is available for programmatic use and contains the Function-specific measurements described in Function refresh summary fields. DESCRIBE JOB '<job_id>' — also spelled SHOW JOB SUMMARY '<job_id>' or SHOW JOB '<job_id>', or abbreviated DESC JOB — shows where a finished job spent its time, one line per row as EXPLAIN shows a plan: its totals, a timeline of its phases, and for a phase that runs on many workers at once, what their time went to. It works for any job; for a Function refresh the phases are queued, planning, starting workers, running splits and committing. A job still running returns no rows. FORMAT TABLE returns its timeline and worker-time rows, with unit saying whether millis is wall time or worker time. FORMAT JSON returns one JSON document. For a Function job with GPU samples, text output adds a GPU utilization line and JSON includes gpu_utilization with the number of sampled tasks and device observations, the wall-time-weighted mean compute utilization, and peak device memory. CPU-only, unsampled, and non-Function jobs omit the field. It is derived from the existing once-per-second worker samples: it is not an instantaneous cluster metric, a peak compute utilization, or evidence that a GPU was resident for the whole Job. When Function workers reported their runtime, text output also includes a Function environments section. Identical environments are grouped across workers, while different hardware or dependencies remain separate. FORMAT JSON and the Stats route include function_environments with the full reports and worker/Function mappings. These reports are available for failed jobs too; they are omitted for older jobs without a recorded environment. See Inspect a Function worker’s environment. JSON and the Stats route also include function_refresh when the terminal Function summary exists. It carries the slot, split, batch and worker measurements below with their separate scopes; the field is absent for jobs without that summary.
The worker-slot block describes the supervisor run that finished the job. Split execution, split latency and batch latency include every split record in the terminal summary, including retries and attempts from earlier supervisor runs. They are reported separately because their scopes are not comparable as a compute-utilization ratio. Throughput is accepted output rows divided by the time from the first included split start to the last included split finish; the line is omitted when that wall time is unknown or zero.

Function refresh summary fields

A successful Function refresh stores these measurements in job.result["summary"]. /v1/jobs/describe fills job_created_at_millis from the registry’s actual creation record, not the query server’s clock or a later execution’s start. Older results can omit a field; an omitted measurement is unknown, not zero. Across refreshes, the system.public.function_refresh_metering view holds one row per succeeded refresh with the same summary, once the job events export has run. It needs system Usage and Select on the view. This grants access to refresh metering across all databases in the cluster; the connection’s database does not filter the view. Only grant this access to users who may read cluster-wide usage. SHOW JOBS instead lists jobs in the connected database. Choosing a database in the SDK does not narrow an API key’s permissions. For a recovered refresh, the terminal result’s rows_assigned is the logical count of accepted output rows. The summary’s rows_assigned uses that accepted total, including partial output accepted from an earlier attempt; individual completed split records can cover fewer rows. Use split records for split-level accounting, not as the output total. rows_skipped remains a completed-split subtotal. system.public.job_metering.rows_processed counts rows handled by completed Function execution splits: computed rows plus skipped rows, once per split completion. Planning scans and fragment finalization add no rows. Failed claims add no rows; completed split work can remain counted after cancellation or when a later recovery discards its output. This measures execution work, not the number of rows currently published in the table. For succeeded Function refreshes, job_metering also reports rows_computed from the terminal result’s rows_assigned, rows_skipped from the refresh summary, and rows_failed from the terminal result. rows_computed agrees with function_refresh_metering.rows_assigned, including accepted recovered output. rows_skipped includes deferred rows and row failures; rows_failed counts terminal row failures. These totals are NULL for jobs without the corresponding result, including failed or canceled jobs, and for other job types.
rows_reexecuted counts row inputs passed to later user-code invocations because of an explicit retry budget or legacy bisection inside their logical batch. bisection_rounds counts failed parent calls that were split into child calls. Completed split records include retry work whose output a later replan discards. A repeated receipt for the same task and split replaces its earlier record instead of adding it again. The terminal summary deduplicates replay work by durable part, so a runtime restart that rebuilds a completed split under a new task ID does not count it again, while durable parts from a discarded execution remain counted. Both values are absent for images or jobs that do not record the measurement, and an aggregate is absent if any contributing part is unknown. They measure Function row replay, not every external request a Function makes and not infrastructure retry or worker recovery. These counters are currently available in an individual refresh’s terminal summary and profile. The cluster-wide function_refresh_metering view described below does not project them. For a database-specific report, filter database_name explicitly, as below. This query filter is not an access control boundary. Compare a Function’s versions, or follow one over time:

See also

Refresh a computed column

The same operation from the SDKs, and what a refresh guarantees.

Table

Declaring the columns this fills.

Garbage Collection

How retained versions and Managed Blob objects are reclaimed.

Branches

Shared, writable lines of table history.