Enterprise only
LanceDB extensions with no DataFusion equivalent. They are what makes
feature engineering reachable from SQL: a column
declared by DDL starts empty, and a refresh fills it.
REFRESH COLUMN
WHERE, a refresh fills unassigned rows and recomputes values whose
inputs changed, leaving other current values alone. A retry after a failure
can still repeat work that was not committed.
In the next Enterprise release, a successful filtered pilot records its rows
as current. A later unfiltered refresh skips those successful rows while their
inputs and Function binding remain unchanged. For example, a 40-row pilot with
39 successes on a 400-row table leaves 361 rows to evaluate in the full refresh.
Repeating an explicit WHERE still forces the matching rows to run again.
A completed part or calculated physical range is staged work, not a table
commit by default. Without publication_interval_rows, results for a fragment
become visible only after all split work selected for that fragment in this
refresh is complete, including any planned physical ranges that return no rows,
and the fragment is published. With the option set, completed durable parts
publish in an intermediate table version once the oldest unpublished parts
reach the row threshold, even while their split is still running. Cheap
Functions cover many rows per wave, while a single slow call becomes readable
after that call returns a durable part. The finalizer still publishes the
completed fragment after all selected split work is complete, and no in-flight
call is promised if it has not returned a durable part. Blob v2 Function
outputs keep one final publication per fragment. An atomic binding rebind also
keeps one final publication per fragment.
With WHERE, a refresh recomputes every matching live row, including rows
that already have current values. Repeating the same predicate can repeat the
Function calls and their cost. To fill only null outputs within a slice, add
the output column to the predicate:
NULL; a null value does not
necessarily mean the row has never been computed. Use an unfiltered refresh
to let LanceDB track which outputs need updating.
A separate WHERE request is a separate Job. It pays planning and Job startup
overhead, can repeat Function image and Secret preparation, and pays commit
overhead in addition to recomputing every matching row. Splitting work into
predicates can isolate a slice or keep a Job bounded, but it is not a free
checkpoint of the original Job.
An initialized Function instance and its mutable state may survive across
refresh jobs. Do not assume a new refresh resets instance state or calls
__init__ again. Instances are not shared across workers.
Refresh computed columns in dependency order. A SQL computed column’s refresh
fails before writing if the rows it would compute read another computed or
Function column that has not been refreshed for its current inputs. Refresh
that input first. A computed input that is current but evaluates to NULL is
a valid value, not an unfilled placeholder.
WHERE scopes the refresh, which is how you fill a slice and check it before
committing to the whole table. It does not narrow the per-fragment input
readiness checks: if a selected fragment contains unfilled computed input rows
anywhere, refresh that input first and then rerun the filtered child.
A SQL computed column can opt into the same partial progress. Add
WITH (ready_rows_only = true) to compute only rows whose computed inputs are
assigned and current; rows still waiting on an unfilled input, a failed
Function row, or a stale input fragment stay unchanged. A computed input that is
current but evaluates to NULL is an assigned value and is computed, not
deferred. The terminal result reports the deferred count in rows_remaining;
refreshing the input and rerunning the child completes them. Without the
option, the strict per-fragment readiness check above remains the default.
A Function column’s refresh does not fail on an unfilled input. It defers each
selected row whose computed input is NULL, or whose nullable input column is
NULL for a non-nullable parameter, and computes the other rows. Deferred rows
are neither assigned nor failed, and they stay retryable. The successful result
counts them in rows_deferred and names the NULL inputs in
deferred_inputs; see Watching a refresh. For a computed input,
refresh that column first, then rerun this refresh. rows_deferred counts only
rows the refresh selected: after a refresh, a rerun without WHERE does not
select rows whose computed input is still NULL until that input changes.
Previewing a refresh
EXPLAIN and EXPLAIN VERBOSE take the same options and validation as
REFRESH COLUMN and return DataFusion’s plan_type, plan rows:
logical_plan: theRefreshColumnstatement.physical_plan: aRefreshJobExecroot carrying what the job runs with (the snapshot, worker count or Function resource budget, error policy, batch size, row timeout, how many fragments have work, which fragment is shown) over the plan each worker executes for one fragment: aRefreshStageExecstages the replacement file from the stored values and the rowsStaleRowsExecselects, computed by aProjectionExecfor an expression column or aPythonUdfExecfor a Function column.
source_version. If a
concurrent compaction only moves rows without changing their values, the refresh
can reuse complete, successfully staged non-Blob outputs through stable row IDs
or a verified compaction row mapping. Rows outside the refresh selection keep
their current values. This reuse makes no Function calls, so it also works for
effectful or per-call Functions and does not consume the three-round
recomputation limit.
An input or output UPDATE invalidates reuse for the affected fragments. If
reuse cannot be verified, staged coverage is incomplete, or outputs contain
Blob values, recovery requires recomputation. Only Functions declaring
result_stability=input_and_context and side_effects=none can recompute the
affected fragments, for at most three job-wide compute rounds. Effectful or
per-call Functions fail with the concurrent-write error when recomputation is
required. Unaffected staged results commit without rerunning.
Reuse keeps row mappings in supervisor memory for the affected publication wave.
Large waves can require more supervisor memory; the refresh’s worker
memory_bytes budget does not cap this memory.
Each successor request pins the newer snapshot it plans, executes, and finalizes
against. These request-local snapshots do not replace the Job’s original source
pin.
EXPLAIN [VERBOSE] JOB '<job_id>' renders the same tree for a submitted
job, with the registry’s view on the root: state, progress and the job’s
options. The tree is rebuilt from the job’s spec against the table’s
current version on its branch (the root’s source=), not the snapshot the
job pinned when it started, so a representative fragment can differ from
what a worker ran. A job whose work is not a plan, such as an index build
or a compaction, renders its root alone.
The predicate also accepts _rowaddr and _rowid. To retry specific rows
reported by table.function_errors(), use each record’s row_id with _rowid
when it is present. This stable ID continues to identify the same logical row
after compaction or another rewrite:
row_id, combine fragment_id and row_offset as
(fragment_id << 32) | row_offset and select those physical addresses before
compaction or another rewrite moves the rows:
function_errors() and SHOW ERRORS when their
physical locations have not changed; rows that fail again remain listed.
After compaction or another rewrite moves a row, _rowid still selects it,
but its earlier error may remain listed even when the retry succeeds. Error
clearing currently matches physical locations, not stable row IDs.
Older error records and tables without stable row IDs may omit row_id.
Errors without an individual row location, including those whose per-row
detail was capped, cannot all be selected individually.
Checking column status
SHOW COLUMN STATUS reports one exact four-way partition for each
Function-backed output:
rows, fresh, stale, never_computed, failed, the latest
matching refresh Job, and the automatic-refresh state. The four counts always
add up to rows. A value that is legitimately NULL is still an assigned
value; it is not counted as never computed unless the Function attempt skipped
that row. This also applies to a nullable scalar output: a computed NULL is
assigned and fresh, while an unrefreshed, deferred, or skipped row remains
unassigned.
The REST API exposes the same read-only result:
column, branch, and version are optional. Omit version to pin the
latest version once for the request. Every status in a response describes that
same table version. Use each result’s table_version as the authoritative
snapshot version. The shared response middleware can observe a newer live
handle, so the x-lancedb-version header need not equal that pinned version.
If a Function publication is committed but its freshness sidecar is still
finalizing, REST returns 503 with error code
function_column_status_finalization_pending; clients such as wait_fresh
retry this transient state. The API returns an explicit unavailable error when
the stored evidence cannot prove the exact four-way partition. It does not
substitute zeros or partial counts.
The automatic-refresh projection includes fields the Query Node can prove.
When the authoritative Lance Agent scheduler decision is unavailable,
auto_refresh.state is unobservable, next_run_at_millis is null, and
next_run_at_reason is
authoritative_scheduler_projection_unavailable. The configured enabled and
interval_seconds fields do not by themselves prove a next run.
Function recovery defaults
New Function refreshes default toerror_policy = 'skip'
with max_failed_fraction = 0.05. Successful output is committed before the
job reports a threshold failure. Explicit fail retains terminal failure
semantics; initialization, infrastructure and publication errors still fail.
SHOW ERRORS ON table and function_errors('table', 'column') expose current
unresolved failures with grouped causes, cumulative attempts and failure times.
Successful retries clear current failures; SHOW ERRORS HISTORY ON table
selects separately retained historical observations. Exact failed-row retries,
version upgrades and quarantine require the matching
Enterprise client.
Quarantine is released by an admitted retry, so rejected requests preserve it.
New materialized views use the same skip/5% default. Their successful output
is published before threshold enforcement. Counts and current diagnostics
belong to the committed compute attempt; abandoned replay work is excluded.
Existing views retain their stored error policy.
Options
num_workers sets a fixed worker count for expression-backed refreshes.
Registered Function columns reject num_workers, even without a resource
budget. Their worker count adapts within the aggregate resource ceiling. The
example above allows up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the
Function workers together. It does not reserve those resources for each worker
or guarantee that all of them will be used.
Execution slots can run splits from different fragments concurrently. Global
throughput across all slots is not a reliable estimate of how long one fragment
will take: that fragment can use only the slots scheduled to its splits and
competes with work on other fragments. Use per-fragment and split progress when
estimating a particular fragment. A refresh also plans one shared tail wave of
splits beyond the slots its budget admits, so a worker that finishes its share
first can take remaining work instead of idling while the last splits run; the
split floor and the job’s split budget still cap the count.
gpu_count may be set alone to use the deployment’s default CPU and memory.
Set cpu_millis and memory_bytes together to define an aggregate CPU and
memory ceiling. When they are set and gpu_count is omitted, it defaults to
zero, so a CPU-only refresh does not need to spell out the GPU limit:
cpu_millis and memory_bytes is rejected. A GPU Function also requires an
explicit finite row_timeout_seconds on its definition or on the refresh.
Without one, the refresh is rejected before a model call starts; LanceDB does
not apply a hidden GPU default. CPU Functions keep the 300-second default.
LanceDB adjusts the amount of work sent to each worker from observed execution
time. batch_size limits rows in one Function call; it does not set the worker
count or the publication_interval_rows threshold. Small initial batches are
expected while the refresh learns how long rows take.
batch_size does not by itself batch a scalar model call. A GPU Function whose
__call__ accepts one row is invoked once per row, even when the refresh allows
a larger call. That repeated Python/model-call boundary leaves a GPU idle between
small operations and is substantially slower than a vectorized callable. Annotate
every input as pyarrow.Array, or take one pyarrow.RecordBatch, and batch inside
the callable. The MiniLM example shows the
vectorized form.
The Function/refresh batch_size and the model library’s own batch size are two
different limits. The first bounds rows in one Function call. The second controls
how many of those rows the model processes together. Set both when the model
library supports its own batch size.
publication_interval_rows opts into intermediate publications for
Function-backed columns with non-Blob outputs. Blob inputs are allowed;
Function outputs that contain Blob v2 are rejected. It is a row-count threshold
over durable completed parts, not a time interval. A wave publishes the oldest
unpublished parts once their accumulated rows reach the threshold; parts in a
split may publish before that split completes. Unset keeps one final
publication per fragment. The finalizer still publishes the remaining rows and
completes the fragment. This SQL option is available in
this server build. The Python client option is a separate SDK change and is
not part of the current released client; SQL and SDK must match.
GPU class names are lowercase names of at most 63 letters, digits, or hyphens.
The REST backfill_column body accepts gpu_class, gpu_count,
init_timeout_seconds, job_timeout_seconds, and publication_interval_rows
as top-level options with the same validation and budget semantics as SQL.
CPU budgets inside Functions
os.cpu_count() and os.sched_getaffinity(0) can report the host node’s CPUs
even when a Function worker has a smaller CPU quota. Pools created without an
explicit size, including multiprocessing.Pool() and ProcessPoolExecutor(),
can therefore start too many processes.
LanceDB sets LANCE_FUNCTION_CPUS in each Function executor’s environment at
startup. It is the worker’s available CPU count, accounting for its cgroup
quota and affinity, divided by the number of executor processes sharing the
worker, rounded down with a minimum of 1. For example, a 4-CPU worker with one
executor receives 4; with four executors, each receives 1. This is a
per-executor budget, separate from the aggregate refresh cpu_millis ceiling.
Use this value to size pools and library worker counts explicitly:
OMP_NUM_THREADS,
MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, and RAYON_NUM_THREADS; explicit
Function settings for those variables take precedence. When combining a
process pool with threaded libraries, also limit each child process’s native
threads so their combined concurrency fits the budget.
Retrying temporary Function errors
Setretry_policy on a source Function at registration, or override it for one
refresh. The same JSON object is accepted by CREATE FUNCTION ... WITH (retry_policy = '...') and by the registration and refresh REST APIs.
max_attempts includes the first call and accepts 1–100. An explicit
{"max_attempts":1} disables user-code retries, including the built-in transient
retry. Omitting the option preserves the Function default and existing behavior.
Choose one or more matchers; matching any one allows retry:
The Botocore transport exceptions are
ReadTimeoutError, ConnectTimeoutError,
and EndpointConnectionError from botocore.exceptions. Requests and HTTPX
transport exceptions require explicit exception_types, such as
["requests.exceptions.ConnectionError", "httpx.TransportError"].
The runtime also checks wrapped causes. It does not infer a transient error
from digits such as 429 appearing in an arbitrary message. An adapter may
signal throttling with status_code or code, and retry_after_seconds.
Message patterns must compile under both Rust’s regex validator and the
Function environment’s Python re module.
Retries use exponential backoff with full jitter. initial_backoff_seconds
defaults to 0.2 and max_backoff_seconds to 30; both must be finite, between
0 and 3600, with the initial value no larger than the maximum.
honor_retry_after defaults to true and honors HTTP Retry-After or the
adapter’s retry_after_seconds. A requested delay above the configured maximum
ends retry instead of calling the provider early. Cancellation interrupts backoff.
A row Function has one retry budget per row. A batch Function can raise for the
whole call, or it can return a list or tuple that contains one output value or
one Exception for every input row. A returned exception is a row outcome:
successful values are retained, and an explicit policy retries only rows whose
marker matches. A row keeps one max_attempts budget whether its first failure
was a whole-call exception or a later marker. An exception returned for a row
does not repeat unrelated successful rows.
With an explicit policy, a whole-call exception that returns no values follows
the previous whole-call behavior and is not bisected after retries. skip
retains existing outputs for every failed row in that call,
leaves previously unassigned outputs unassigned, and records the error and
attempts. A callable
that raises before returning its complete outcome cannot provide the framework
with successful values that never existed. Output conversion and partially
consumed generator results are not retried. Output validation alone does not
count as a call.
Only opt in when repeating a partially failed call is safe. The runtime cannot
undo effects a call performed before raising. Functions declaring external side
effects are refused. User-code retries do not consume the worker’s infrastructure
attempt budget; initialization, worker loss and OOM recovery keep their own policy.
After exhaustion, fail fails the job and skip records the last error and count
in SHOW ERRORS and the table errors API, subject to the normal failure limits.
A Function image created before retry support must be registered as a new
version before a refresh can override its policy.
Hugging Face model snapshots
A Function can declare immutable Hugging Face repositories and revisions during registration. The worker prepares each declared snapshot before it creates the user Function instance:batch_size rows, possibly fewer while automatic sizing
probes execution time, and passes its whole input list to encode once. The
encode(batch_size=64) argument then batches the model’s own forward passes.
Without vectorized annotations, the same class would call encode separately for
every row.
Build PyArrow types used by the callable inside the class method. Function
packaging captures the callable source and does not accept PyArrow type objects
as global values.
Each repository may be declared once, and its revision must be the full
40-character commit hash. A completed snapshot stores that revision in
refs/main, so the original no-revision SentenceTransformer(repo) call
resolves without contacting the Hub. Model-declaring Functions run with a
private HF_HOME, HF_HUB_OFFLINE=1, and TRANSFORMERS_OFFLINE=1; those
environment variables cannot be overridden through the Function env.
Before SQL execution or refresh worker admission, a shared preparation job
imports the snapshot into database object storage. Concurrent consumers with
the same database storage, pinned Secret scope, repository and revision reuse
that attempt and its immutable snapshot. A code-only Function version can
reuse the snapshot when these inputs are unchanged.
Preparation failures are retained: waiting consumers do not automatically
retry Hugging Face. An operator with System Operate privilege can retry the
observed preparation job through POST /v1/jobs/retry_model_preparation, with
{"job_id":"<observed preparation attempt>"}. After preparation succeeds,
resubmit failed queries or refreshes; retrying the import does not restart them.
Workers read prepared snapshots from database storage into a local model cache.
Local cache sharing, capacity and preparation concurrency are configured by the
deployment operator. Shared snapshots avoid repeated origin downloads; a cold
worker still needs to read the model from database storage. Cached files do not
mean that Python initialization or GPU model loading has already completed.
Without models, the existing behavior is unchanged: the Hub client may still
request metadata or missing files, so a cached model and pinned revision alone
do not guarantee offline execution.
Discovering GPU classes
default class first, followed by named classes.
Columns are name, resource_name, gpu_model, gpu_memory_bytes,
max_gpu_count_per_worker, cpu_millis, memory_bytes, and
ephemeral_storage_bytes. Optional values are NULL when the operator did not
configure them. A non-NULL max_gpu_count_per_worker bounds adaptive growth
for one worker; NULL leaves the refresh’s aggregate GPU budget as the ceiling.
The authenticated REST endpoint GET /v1/gpu/classes returns the same catalog
as gpu_classes. It reports configured hardware, not current free capacity.
Row errors
Seterror_policy on the refresh, not on the Function definition. With skip,
inspect the recorded failures with SHOW ERRORS or table.function_errors()
and retry the affected rows after fixing the cause. Skipping row errors does
not suppress initialization, infrastructure, or publication failures.
For column refreshes, skipping a failed recompute retains
the previous stored output. A row without an assigned output remains
unassigned. Skipped and deferred rows remain out of date and retryable even
when they retain non-null values; an unfiltered retry uses freshness state to
select them. Successfully published pilot rows stay current and are not
evaluated again unless their inputs or binding change or a WHERE explicitly
selects them.
table.function_errors(column="embedding") reads retained errors for that
column across refresh jobs, including diagnostics from failed and timed-out
jobs. A successful retry suppresses earlier row errors only when their physical
locations still match, as described above; this is not a complete view of the
rows currently failing. Pass job_id to restrict the listing to one refresh.
The default limit is 10,000, and the maximum is 100,000. Row records
and capped-fragment summaries share this budget. A column filter suppresses
fragment summaries because those summaries do not identify a column.
SHOW ERRORS ON documents [FOR JOB '<job_id>'] [LIMIT n] uses the same limits.
Reaching exactly the result limit does not by itself set truncated. The flag
is true when a matching entry was omitted, or when a scan budget left part of
the history unexamined. The store scan visits at most limit error files;
without job_id, footer diagnostics are limited to the 32 newest failed or
timed-out refresh jobs. SQL reports truncation with a ListingTruncated marker
row. If truncated, narrow by job_id or raise limit; there is currently no
cursor for continuing the listing.
The table error store covers published fragment work. A skipped row in a part
that remains staged and unpublished is not written to function_errors() or
SHOW ERRORS; its Job progress and split evidence can still record it. A
failed Job can therefore report skipped rows with no table error records, and
the error API is not a complete ledger of every attempted or staged row.
When a Function worker dies or an executor is killed with SIGKILL, retained
terminal records from POST /v1/table/{id}/errors can include attribution,
worker_id, and task_id. Attribution describes the evidence for the death:
likely_cause: the task was the only executing task on an OOM-killed worker, with every claim’s activity known, or its executor was killed while the worker’s cgroup counted a new OOM kill during that session.collateral: the task was not executing when its worker died.unattributed: the evidence cannot identify a likely cause, for example because several tasks were executing, an attempt ledger was unreadable, or the worker died without OOM evidence.
row_range contains fragment-relative physical
offsets, and row_address_range contains the corresponding _rowaddr bounds.
Both ranges are half-open: include start and exclude end. SQL SHOW ERRORS
keeps its existing columns and carries the attribution and range in the error
message.
Always read affected inputs at the record’s table_version. Physical row
addresses can change after compaction, so applying the bounds to the latest
version can retrieve different inputs. For example, to inspect up to 100 rows
from a record with table_version = 12 and row_address_range from
30064771072 to 30064777700, send this body to
POST /v1/table/documents/query:
skip
policy. Under it, a refresh fails when more than 50% of the first 1,000
evaluated rows in a fragment part fail, or when the first 100 evaluated rows
fail before any successful row. max_failed_fraction or max_failed_rows
replaces that default fraction; when both are present, either limit can fail
the Job. New refreshes default to 5%
and publish successful output before enforcing their limits instead.
The counters are evaluated rows, not Function calls. Checks continue
cumulatively after the 1,000-row sample. A fragment part shorter than that
sample does not apply the fraction: a part that retries reduced to a single row
counts as one evaluated row, not as a 100% failure rate. The refresh applies
the fraction to its completed splits as a whole at the end, so a tail part with
one unprocessable row does not fail the rows evaluated before it.
max_failed_rows applies to every part. batch_size defines call boundaries:
the worker checks the cumulative counts after every inner Function call, so a
row-capped batch cannot continue past the ceiling. With this per-part
enforcement, the part that crosses a limit is not published and fragments
committed before it remain visible; for new refreshes, successful
output is published before the supervisor enforces the limit.
NaN and infinity
A Function refresh counts rows whose selected declared float result contains NaN or infinity inrows_with_nan and rows_with_inf. A row may appear in
both counts. The scan includes float leaves inside vectors and structs and
counts each input row once per category. The terminal Job summary carries the
accepted-output total; SHOW JOB SPLITS exposes the corresponding per-split
measurements.
The default nan_policy = 'allow' preserves the Function’s IEEE values. Use
nan_policy = 'error' with error_policy = 'skip' to write affected rows as
null and record them as row failures:
nan_policy = 'null' requires the selected Function output field to be
declared nullable. The computed column’s temporary nullable state while it is
unassigned does not relax a non-nullable Function result. It writes affected
rows as null without recording them as Function-call failures. The row counts
remain available in the Job summary under either policy unless terminal
recovery cannot attribute every accepted durable part; in that case both
counts are NULL rather than a partial zero.
LanceDB normally sizes calls from observed row latency and targets at most 25%
of the row deadline. batch_size remains a hard upper bound, not a tuning
requirement. For a Function whose image declares
result_stability=input_and_context and side_effects=none, a timeout or
resource failure can enter range recovery, which retries smaller physical-row
ranges with a fresh executor.
Under error_policy = 'skip', a row timeout can enter range recovery when the
Function declares side_effects=none, including when its result stability is
undeclared or per_call. Ordinary vectorized batch exceptions already use the
same best-effort bisection under skip. Because a failed batch has no row
results to salvage, healthy calls in that batch run again and a per-call
Function may return different values. A single row that
still times out retains its stored output or remains unassigned, stays
retryable, and is recorded as one RowTimeoutError. Functions with declared
effects are not replayed automatically.
A quarantined terminal row is an exception to the per-part failure limits.
Under skip, this applies to an OOM-poison row in a replay-safe Function, a
typed OversizedRows failure for exactly one physical row after range recovery
has reduced the unfinished work to that same row, and a non-interruptible row
timeout in a side_effects=none Function. Oversized-row quarantine requires
the executor evidence to name one physical row that matches the only
unfinished row in the recovery range; multi-row or mismatched evidence does not
become a skipped row. Before range bisection, OOM quarantine and oversized-row
recovery require result_stability=input_and_context and side_effects=none.
For a task with exactly one split whose execution range contains only that
physical row, quarantine does not call the Function again. In this case,
side_effects=none is sufficient even when undeclared stability resolves to
per_call. Declared effects disable automatic replay. Under fail,
a terminal oversized row fails the refresh without a skipped-row record.
Quarantine publishes a synthetic skipped part that retains
the stored output without reading the failing inputs. Previously unassigned
outputs remain unassigned, and the row stays retryable. Applying the per-part
fraction or initial-100-row guard there would reject every quarantine. Before
bisecting, the supervisor retries the OOM-killed claim once by itself with one
executor process on the largest eligible worker class; a row timeout reaches
quarantine only after its execution range has been reduced to one row. The
skipped row still counts toward the cumulative job-level failure limits, and
the job fails if the quarantined rows cross either limit.
Range recovery has a finite leaf and attempt budget. Under skip, when the
failed range can no longer be narrowed, the rows recovery did not reach still
get an outcome: only rows the refresh selected are recorded as skipped row
errors, and they keep the failure class that exhausted recovery
(RowTimeoutError for a row timeout, otherwise ResourceExhausted). Selected
rows whose required input is unready stay deferred and are not recorded as
failures. The fragment’s already computed parts publish with them, and each
affected row retains its stored output or stays unassigned and retryable. The
budget, the per-part failure limits and the fail behavior are
unchanged.
rows_skipped covers the full
count. If the compact skipped-row reports alone exceed the event limit, the
refresh fails before publication. Skipped-row tracebacks require a newly baked
source image.
Long-running Functions
CPU source Functions default to a 300-second deadline per row. Setrow_timeout_seconds high enough for the longest row, for example:
batch_size only lowers
the ceiling. A source Function can call
heartbeat() from the runtime-provided lance_progress module inside
__call__ to report liveness. Heartbeats do not extend the row deadline.
The 300-second fallback applies only to CPU Functions. A GPU Function must
declare a finite positive row_timeout_seconds at registration or supply one
on the refresh; otherwise the refresh is rejected before model execution. The
Function definition reports effective_row_timeout_seconds and
row_timeout_required, and an admitted Job carries the selected deadline in
its spec. This makes the GPU choice visible instead of introducing a second
deployment-wide default.
Failed batch diagnostics report row_timeout_seconds and
batch_timeout_seconds separately. These are the configured row limit and the
outer batch watchdog budget, respectively. A child reporting a row timeout can
exit before the batch watchdog expires; the larger batch budget does not extend
the row limit. The diagnostic also reports row_timeout_source, which is
default for the CPU fallback, definition for a Function declaration, or
request for a refresh override, and
elapsed_since_batch_start_seconds, which measures from the start of the
affected input batch and excludes worker startup and queueing. When a
non-interruptible multi-row batch fails as a unit, the terminal failure record
names the batch boundary and leaves row_offset unset because the failing row
is unknown.
init_timeout_seconds bounds only the interval from Initialize until the
worker receives Initialized; image preparation, Secret resolution, and
cleanup have their own bounded task handling. While a worker is waiting for
Initialized, SHOW JOBS progress reports the initializing phase and
initialized worker counts. It does not count endpoint readiness as
initialization completion. If the budget expires, the refresh fails with an
initialization-stage error and the bounded child stderr, even when
error_policy = 'skip'.
Inspecting a slow or stalled worker
Supporting server releases reportno_progress warnings after 60 seconds
without meaningful progress. A slow_split warning appears when a split has
run for at least 60 seconds and five times the median execution cost per row,
scaled to its row count. This comparison needs at least three completed
samples and uses the most recent 128 samples. Warnings identify the worker,
split when available, and last progress time. They appear in job progress and
recorded events; they do not extend deadlines or restart work.
An operator with the system operate privilege can request Python thread
stacks from a selected worker:
worker_id from a warning or the job’s worker events. On supporting
SDK releases, Python provides job.stack(worker_id) and
await job.stack(worker_id) for asynchronous handles; Rust provides
job.stack(worker_id).await?.
Capture requires a supporting Linux worker runtime. The response reports
captured, unavailable, or unsupported for each session, with
captured_at_millis and truncated metadata. Missing frames do not prove that
the worker is healthy or finished. A terminal or unreachable worker can be
unavailable, and older runtimes can be unsupported.
Each request samples at most eight executor sessions, retaining at most 8 KiB
per session with a 250 ms capture budget. omitted_sessions counts sessions
beyond that limit. Requests are limited to one per five seconds per job, and
a recent sample may be reused for up to five seconds. Captures contain Python
frames without source text, locals, or native frames. Timeout failures retain
their reported exception traceback; sampled frames supply evidence when that
traceback is missing. Inspect recorded failure events after the worker exits.
The row deadline does not extend the job’s execution limit. A Function refresh
runs under a whole-job deadline that defaults to the deployment’s job timeout
(12 hours by default). Set
job_timeout_seconds to change it for one refresh, for example:
timed_out and its
failure names the job deadline and the value that applied. The same limit
bounds the refresh’s Kubernetes Job, so the registry and the pods agree on when
the job is dead.
num_workers is refused for a column filled by a registered
function; a Function refresh’s worker count follows
its own resource budget instead. error_policy (and the other Function-only
options in the table above) is the reverse: it’s refused for a plain
SQL-expression column, which has no per-row failures to skip, and accepted
only for a Function column.Functions whose memory keeps growing
A Function worker keeps each executor, and the state its__init__ built, for
the parts it runs next. A Function that leaks memory therefore grows with every
part until the worker is OOM-killed, and retrying the part leaks again. After
each part that evaluated rows, the worker reads the executor’s resident memory
other than file-backed pages, including the processes it started in its own
process group. It closes the executor once the part completes when all of these
hold:
- the memory reached a new high in at least three of the executor’s last five parts
- the memory grew by at least 1/16 of the executor’s memory budget since its first part, so a fresh executor frees a meaningful amount
- one more part like the last could take the worker past 90% of its memory limit, counting the memory in use other than file cache
__init__
again. The closed executor’s close() gets 30 seconds before it is killed. The
completed part is kept, and a recycle is neither a failure nor a retry attempt.
An executor whose memory holds steady is never recycled. Recycling needs a
worker with a cgroup v2 memory limit; without one, executors are kept as
before.
Each recycle appends an executor_recycle event, shown by
SHOW JOB EVENTS '<job_id>'. The successful job result reports the count as
executor_recycles and up to 64 executor_recycle_details, with
executor_recycles_truncated: true when some are omitted. Each detail has
worker_id, session_slot, recycled_at_millis, the executor’s parts, its
first_resident_bytes after its first part and resident_bytes after its
last, and the worker’s worker_memory_bytes in use, excluding file cache,
against its worker_memory_limit_bytes. These fields are absent when no
executor was recycled. Repeated recycles of one Function point to a leak in its
code.
Automatic refresh timing
Automatic column refresh requires a supporting server release and an enabled deployment. A column’sauto_refresh_interval_seconds sets a minimum delay
after its last successful refresh, not a completion deadline. The scheduler
also waits for its computed inputs, available workers, and other refreshes.
The deployment default allows one column refresh in flight per table.
Function-backed columns may also declare an auto_refresh_resource_budget when
the column is added. An explicit declaration takes precedence over the
deployment’s default resource budget, but it must remain within the deployment
ceiling. CPU and memory must be specified together; when they are present and
gpu_count is omitted, the GPU count defaults to zero. A GPU Function still
requires at least one GPU in its declared budget. Without an explicit declaration,
automatic refresh uses the deployment default. This aggregate budget limits the
worker fleet; it is not a per-worker allocation or a direct user-call concurrency
limit.
Failed automatic Function refreshes start with a 15-minute retry delay by
default, then double it for consecutive failures: 15, 30, 60, 120 minutes, and
so on. This is failure backoff, not a fixed refresh cadence. A manual refresh
can run during that wait. Cancelling a Function refresh blocks automatic
refresh for that binding on the same branch until a later explicit refresh
completes or the binding is replaced. Check job history when automatic results
stop appearing.
REFRESH MATERIALIZED VIEW
The syntax and options below apply to Enterprise deployments that support explicitfull = false in SQL.
CASCADE is not supported.
gpu_count = 0 for CPU-only views; GPU Functions and explicit GPU placement
require a positive GPU budget when one is specified.
gpu_class names follow the lowercase DNS-label rule and must identify an
operator-configured class; use SHOW GPU CLASSES to discover classes. The
override leaves registered Functions and saved MV placement unchanged. Omitted
settings preserve deployment defaults and the MV’s existing placement.
Explicit budgets and placement require an isolated execution backend that can
enforce them. Unsupported backends reject constrained jobs. Resource options
are retained across interruption recovery. A ceiling does not reserve its full
GPU count or add parallel workers: current MV jobs use at most one GPU, shared
by their GPU Function calls. These options affect this manual execution only;
automatic-refresh settings remain unchanged.
Servers without SQL refresh option support require the basic statement without
WITH. Older servers may accept only full = true; some also accept CASCADE.
A local SDK’s refresh() options do not determine which SQL options a server
accepts.Analyze a materialized-view write
function_timings and the canonical run report.
Blob values in views and CTAS copies
CREATE MATERIALIZED VIEW ... AS SELECT and CREATE TABLE ... AS SELECT store
the selected rows in a new table. Blob values stored by Lance, whether inline,
packed, dedicated, or Managed, are copied as bytes into that table, including
blob leaves inside structs and lists and values produced by UNNEST. Nulls and
empty blobs keep their meaning. The destination storage version selects the
stored encoding, so a 2.2 or newer destination writes Blob v2 even when the
source uses the legacy blob encoding. The copy consumes storage for the selected
payloads; SQL does not deduplicate payload objects across tables.
To copy a Lance dataset on Hugging Face into your own storage, first register it
with a shallow clone, then run this CTAS copy over the clone. See
Import a Lance dataset with Blob values.
For Managed Blob v2, the destination writes its own payload objects. It does not
keep the source table’s managed object as a cross-table reference. After the
statement commits, compacting, cleaning, or dropping the source does not affect
the bytes the copy owns. Keep a source tag only when you also need to read the
source snapshot.
An external URI is not a copy of its object. SQL preserves a top-level external
reference instead of fetching it; because SQL does not register source storage
bases, a copy fails when Lance cannot store that reference. Nested external
blobs are different: the copy reads and materializes them into destination-owned
bytes, so the query must be able to access the external object. When a copy
retains an external reference, the object remains owned by its own system: a tag
does not pin its version, and a later read can fail or observe different bytes
if the object changes.
Retaining a view or CTAS copy
A tag names a table version and keeps that version and the Lance-owned files it references, including Managed Blob objects, from being deleted by cleanup. Tag the destination, not the source, when the copy must stay readable. A tag on the source protects only the source and does not keep the destination copy. Tagging a version does not stop later refreshes or writes from advancing the table or view name. Read through the tag or check out the tagged version to get the frozen rows; the latest view name may already point to a newer refresh. If automatic refresh is enabled, disable it for a view that must remain unchanged, or accept that the latest name advances while the tag stays fixed. Compaction writes a new table version; it does not change an older tagged snapshot. Managed payload objects that the tagged version references remain protected until the tag or version is removed. Cleanup then reclaims only objects no retained snapshot references. Deleting the tag makes the version eligible for cleanup again. Source cleanup or compaction is safe for the bytes the copy owns, but not for future refreshes. A materialized view’s later refresh reads its source table again; keep the source available until refreshing is no longer required. Cleaning older source versions can force a rebuild, and dropping the source prevents refresh. A CTAS table has no such dependency. A branch is not a byte-for-byte copy. It shares the table’s history and payload files with its ancestors. A branch can keep inherited files readable while the branch exists, but it does not create a second copy, and deleting the branch or its remaining references can make inherited versions eligible for cleanup. Tag the relevant table version to retain a shared snapshot. Use a CTAS or MV result when you need destination-owned blob bytes. Creating a branch from a table version with Function-bound columns is also currently rejected; existing branches with those bindings remain readable and refreshable. To freeze a training set:- Build the result with
CREATE MATERIALIZED VIEWorCREATE TABLE ... AS SELECT, and verify its row count and representative bytes. - Tag the destination table’s committed version. Keep the tag for as long as the training set must remain readable.
- If the result retains any external references rather than copied bytes, keep the external objects under their own retention policy. The table tag does not protect them.
- Source cleanup, compaction, or dropping the source is safe for the stored copy once it owns its bytes. A materialized view that will refresh again still needs its source table; keep it available until refreshing is no longer required. Cleaning older source versions may force a rebuild. Compacting the destination is also safe: it creates a new version and leaves the tagged version intact.
- Read the frozen rows by the tag or tagged version, not by the latest view name if the view continues to refresh.
Function row errors in materialized views
Choose the policy when creating the view:error_policy = 'skip' with the 5% failure ceiling;
existing views keep their stored policy. Under fail, a Function failure aborts the refresh and preserves the previously
published result. With skip, a failing scalar Function input produces NULL;
a failing table Function input produces no output rows. Planning, storage,
worker, and commit failures propagate to normal job failure and recovery
handling; they are not converted into skipped inputs.
Custom Function images must report a failure as attributable to the input before
it can be skipped. Errors from older images that do not distinguish callable
failures from worker failures stop the refresh.
Use SHOW ERRORS ON captions or the SDK’s table.function_errors() to inspect
the diagnostics. SHOW ERRORS adds an input column containing JSON identifying
the Function invocation and input offset. When a failed input can be traced to
stored source rows, it also includes the source table, pinned version, and row
address or stable row ID. Inputs computed by expressions or aggregations may
have no single source row. Input offsets are not physical row offsets in the
view. SDK error messages also include the source locator; use it instead of the
view fragment fields for MV errors. Error records describe attempts and remain
available after a later successful refresh.
An incremental refresh over appended source rows leaves previously skipped
inputs alone. When a change to existing source rows recomputes a block of the
view, every input in that block runs again, skipped ones included. To retry
them all, run:
Interrupted materialized-view jobs
Materialized-view creation and refresh check for overlapping work before submission and worker launch. These checks are best effort: concurrent requests can still reach storage together, where commit conflicts decide the outcome. A losing creation or manual refresh fails with a non-retryable concurrent-operation error. Automatic refresh skips an attempt when another refresh is active and checks eligibility again on a later scheduling pass. LanceDB can replayCREATE MATERIALIZED VIEW and supported
REFRESH MATERIALIZED VIEW operations, including full = true, after an
infrastructure interruption. Other SQL statements remain non-replayable.
A replacement worker keeps the same Job, job handle, and original deadline. A
counted infrastructure disruption can use one replacement worker. User
cancellation and permanent SQL or Function errors terminate the Job, and a
terminal Job is not restarted.
Recovery can run Function code again when the data commit has not completed, so
external side effects from that attempt can repeat. After the data commit,
receipt recovery finishes state and catalog publication without computing the
committed rows again, and it does not need to plan the original source or
Function. Cancellation does not guarantee that an in-flight storage write
stops; a data or catalog request that was already issued or sent may still
complete. Recovery therefore does not guarantee exactly-once execution of
external side effects.
Watching a refresh
When a client requests an early job handle and the server can make the job registry available to it, the refresh returns ajob_id with status
submitted. The work then outlives the request. Track it with
SHOW JOBS, and diagnose failures with
SHOW ERRORS. The registry APIs require both
System Usage and System Operate; refresh access scoped to a database or
table is not enough.
A Function refresh that completes with failed output rows keeps the backend
lifecycle state succeeded. The console displays Partially succeeded when
terminal failure statistics are nonzero. Its terminal result and success event
include summary.failure_stats; POST /v1/jobs/list returns the same object as
failure_stats on each job row:
NULL reports them in its terminal result and success event:
rows_failed excludes deferred rows. Each deferred_inputs entry names an
input column, the Function parameter bound to it, whether the column is
computed, and the deferred rows on which it was NULL; a row with several
NULL inputs counts under each. A refresh treats a computed input’s NULL as
not yet computed, so refreshing that column first can fill those rows.
Otherwise the column is nullable and the parameter is not. The job’s progress
repeats both fields once the job succeeds. Both are absent when no row was
deferred.
A class Function’s close() failure preserves completed output and does not
retry its rows or turn a successful refresh into a failure. The successful job
result reports close_errors and close_warnings. Each warning has job scope,
worker_id, session_slot, code, message, and an optional traceback.
SHOW JOB EVENTS '<job_id>' includes a close_warning event for the failure.
These fields are absent when no close errors were recorded.
Warning messages and tracebacks retain at most 4,096 characters each. The result
includes at most 64 warning details within a 64 KiB JSON array budget;
close_errors still counts all recorded failures and
close_warnings_truncated: true indicates omitted details. cleanup_warnings
reports shutdown or diagnostic-collection problems, with the same count and
byte budgets. cleanup_warning_count counts those problems, and
cleanup_warnings_truncated: true indicates omitted details. Cleanup summaries
retain the beginning and end of long messages, with an ellipsis between them.
Workers scheduled for removal close their sessions before the successful result
is returned. Warm workers can retain sessions for reuse across jobs, so close()
can run later. A completed job result cannot include a close failure that has not
happened yet. Close warnings are separate from the row-level errors returned by
function_errors() and SHOW ERRORS; they do not change row failure statistics.
Submitting a Function refresh creates work; it does not reserve Function worker
capacity or guarantee that a warm worker is available. A queued job reports a
current admission observation when the dispatcher has one, but submission has
no global queue position or start-time estimate.
CREATE MATERIALIZED VIEW returns (operation, name, status, job_id). The
original three columns keep their positions; job_id is appended and is empty
when the statement ran natively without a registry job.
When the caller lacks both system privileges above, the server omits the early
job-handle acknowledgement and the query continues through ordinary adaptive
polling. Continue polling the returned descriptor until it completes, then read
the result; the final refresh success or error is delivered through that path.
When the bounded retained-run store is full, the server runs the refresh
synchronously and reports its actual outcome without announcing a change of
mode. A client must use the early acknowledgement and status submitted, rather
than the presence of job_id alone, to decide that it received an early
registry handle. Published view results remain visible whether the refresh was
polled or run through the synchronous capacity fallback.
Use SHOW JOBS FOR JOB '<job_id>' to discover a refresh’s child jobs and
inspect their phase, row progress, and attempt information. Pass a returned
child ID to the same event command used for the parent:
version() identifies the server executable rather than the underlying SQL
engine. SHOW SERVER VERSION returns its release, git_sha, and UTC RFC 3339
build_date. Unknown executable metadata is returned as NULL.
With a supported SDK, db.server_info() (or await db.server_info() on an
asynchronous connection) returns the responding query-node build, linked Lance
and LanceDB versions, supported Function worker protocol, and configured
query-node, job/CPU worker, SQL job, and GPU worker images. It requires System
Usage permission and submits no query or Job. Older servers without
GET /v1/server/info report that discovery is unsupported.
During a rolling upgrade, requests can reach different query-node builds.
Configured images do not establish which image an existing Job used. A completed
Function refresh retains the finishing supervisor’s build in
job.result["summary"]["server"] and its selected worker protocol in
job.result["summary"]["worker_protocol_version"]. SQL Job results identify their
worker in job.result["job_worker"]; this does not identify an overridden SQL
child executable. Function refresh and SQL Job results also retain
job.result["terminal_server"], the build of the process that wrote the terminal
event. This is available for failures before execution and cancellation, when
there may be no worker: it identifies the terminal writer, not a Function worker
or SQL child. Historical results without these fields remain unknown when
read by a newer server. Include these values and any server build information in
an SDK exception when reporting a failure.
While a supervised refresh runs, SHOW JOBS includes a progress column
containing a JSON sample: rows planned, computed, skipped and remaining,
published and retained, failed rows, fragments planned and completed, active
workers and active fragments, and the last row-progress timestamp when
available. The same progress object is returned by
POST /v1/jobs/describe and POST /v1/jobs/list. Samples update
approximately every 30 seconds and after tracked progress changes. While at
least one worker endpoint is ready and the latest sampled worker phase is
preparing, initializing, or initialized, startup samples are published
approximately every second so short initialization transitions are more likely
to be observed. They are absent before the first sample and while the job is
paused. Succeeded, failed, timed out, and canceled jobs retain the latest row
progress and timing from the execution that ended, when a sample was recorded.
Terminal jobs zero live worker, GPU, and active fragment counts and clear
resource_classes, warm_reused, and wait reasons. sample_at_millis and
execution_epoch identify that sample. A client may estimate remaining time
from two observations only when the epoch and planned work are unchanged and
accounted work advances; an estimate is unknown at startup, after a pause or
epoch change, and when either sample is missing or stale. The Python SDK exposes
a defensible estimate as estimated_remaining_time_millis and omits the field
when the evidence is insufficient; the server does not persist or return a
historical sampler.
The optional worker_fleet object reports the Function-role worker cohorts
from the supervisor’s latest completed status observation. desired is the
policy target for Function workers and can exceed the cohorts that have been
created. launched counts surviving workers whose cohort launch was confirmed,
pending_workers counts observed active workers not yet ready to accept
dispatch, including workers still initializing. ready counts distinct
endpoints in that launched set that can accept dispatch after status and worker
announcements are merged. When an observation does not provide a usable
pending count, readers may derive it from active - ready; this fallback only
describes currently visible endpoint evidence and is not a guarantee about
workers that have not yet become visible.
resource_classes lists the observed class names. Draining, reaping, and dead
workers are excluded consistently. requested_gpus totals the GPU devices
requested by the observed Function cohorts; ready_gpus totals devices on
dispatchable endpoints. Neither number is free cluster capacity or a queue
position. warm_reused is true when at least one observed cohort was proven
to come from a reusable pool. It is false only when every observed cohort
was proven cold, and absent when no cohort was observed or the observations
are incomplete. A true value does not guarantee that every worker in the
fleet avoided a cold start. worker_fleet is absent until a coherent
Function-worker observation exists. first_launched_at_millis,
first_ready_at_millis, and last_ready_at_millis record milestones for the
current supervisor run. Readiness means a dispatchable endpoint, not that
Python initialization or the first function call has completed.
When no dispatchable endpoint exists, worker_fleet.wait_reason may carry a
fixed substrate-derived wait reason; it is cleared once an endpoint is ready
or the job reaches a terminal state.
The top-level SHOW JOBS.phase reports queued, paused, starting,
waiting_for_workers (no Function workers have launched), and
starting_workers (workers have launched but no endpoint is ready). Once an
endpoint is ready, it mirrors the nested progress.phase, falling back to
running until the first worker sample. The nested phase reports the observed
worker phase: preparing before the initialization handshake, initializing
while the worker is inside the Initialize handshake, initialized after a
successful Initialized response, running while processing rows,
finalizing while publishing, and staged after local output is staged. The
top-level phase becomes committing once all planned rows are accounted for.
Terminal jobs report finished, failed, or cancelled when the outcome is
known. workers_initializing and workers_initialized count distinct live
workers at those handshake boundaries; endpoint readiness alone does not
increment either counter. These are sampled observations, so a short phase may
be missed, and neither phase claims that every worker is in the same state.
queue_reason is populated only from a current substrate observation; in
particular, before a Function supervisor starts, the dispatcher may report that
the job is waiting for executor capacity and include the worker slots requested
and currently available. Those values are a point-in-time admission observation.
A GPU-capacity reason requires PodScheduled=False evidence naming the requested
GPU resource. CPU pressure, taints, and image-pull failures are not relabeled as
GPU capacity waits. The object does not report global queue position or an ETA.
rows_computed is the sampled number of evaluated rows, including work later
retried or recomputed and work that has not been published. It is not an exact
durable-output ledger, and it does not mean those rows were lost when a Job
fails. rows_published counts logical evaluated rows backed by accepted
fragment publication receipts at the sample; it is not the number of planned
rows, nor a claim that every physical row in the fragment was recomputed.
rows_retained counts the assigned rows of completed splits that no accepted
publication receipt covers yet. Completed parts from a split that has not
completed are not included, even if those parts are durably staged. It is a
sampled lower bound and does not make rows table-visible. rows_failed is
present only when a successful terminal result carries a typed failed-row count;
it is not inferred from in-flight or non-successful job state. rows_deferred
and deferred_inputs likewise come only from a successful terminal result, and
are absent when no row was deferred. active_workers
counts owners of execute claims, including workers that have not reported their
first batch yet; it is separate from worker_fleet.ready. Job listings return
the most recently created jobs first across tables.
SHOW JOB SPLITS '<job_id>' returns completed splits and, while the refresh
runs, the latest task for each split that has not completed. state preserves
the worker’s serialized phase, including phases introduced by a newer server;
queued, assigned, initializing, running, failed, and completed do
not share a closed mapping in older clients. Measurements that live evidence
has not established are NULL, never a fabricated zero. error_count counts
explicit failed execute completions for the current task identity in the
current execution epoch, including failures that happened before a later
retry or success. Expired leases are shown as queued; retries, expiry,
success, replaced tasks, skipped rows, and row-level errors do not increment
error_count, and they do not erase failures already recorded.
Canceling a refresh leaves already published results visible. A later refresh
is not guaranteed to resume an earlier Job. It may reuse a completed staged
part only when the runtime finds an existing compatible basis, source
provenance, and selection, and the part remains within the bounded retention
window. Otherwise the later refresh recomputes the work. Treat this as
opportunistic reuse, not an automatic cross-Job checkpoint or a promise that
unfinished rows carry forward.
A Function output can be renamed without recomputing its rows. The column and
its binding change in one commit, keeping existing values, refresh freshness,
and error history.
_rowid or _rowaddr, or start with __. Existing error records
retain the column names recorded when their jobs ran; use the original column
name when filtering that history.
Dropping a Function output column retires its binding and automatically cancels
running and queued refreshes for that binding on the same table generation and
branch. Failed drops and IF EXISTS drops that remove nothing leave refreshes
running. Refreshes for other bindings are unaffected.
SHOW JOB SPLITS '<job_id>' reports, per split: task, split and fragment
identifiers, the worker that ran it, its row range, rows skipped, rows with
NaN or infinity, bytes read and staged, part and attempt counts, batch count,
start and finish times, state, error_count, and the executor batch latency
as p50/p95/p99/max in milliseconds. A split the supervisor recovered from a
checkpoint rather than a worker report carries recovered = true.
POST /v1/jobs/query_splits
reports completed split records only — not the live, not-yet-completed rows
the SQL form returns while the refresh is still running — and its records
carry a different shape: no state or error_count, and a raw latency
histogram in place of the SQL form’s precomputed quantiles.
Export a Function job timeline
On the console job page, open the Trace tab to explore the recording in the embedded viewer. It loads on demand and matches the console theme. Choose Refresh trace for a new snapshot of a running job. The bundled viewer runs in your browser and is served by the console, including in private deployments. Choose Download trace to save the recorded timeline as Chrome trace JSON. Open the file in Perfetto or Chrome tracing to inspect split spans grouped by worker, alongside timestamped planning, worker startup, and commit phases. Concurrent splits appear on separate display lanes under their worker; these lanes do not identify physical execution slots. With a compatible Python SDK and server, the same recording is available from an existing job handle:POST /v1/jobs/stats exposes that report to clients. Its request is
{"job_id": "<job_id>"}. A known job returns:
profile contains the typed
profile and report contains the server-rendered text. Both remain null when
a terminal job has no readable profile. Unknown jobs, and jobs the caller is
not allowed to inspect, return 404. The route accepts failed, canceled,
timed-out and archived terminal jobs, uses the same permission check as
/v1/jobs/describe, and never submits, waits for, or cancels work.
The text is rendered by the same server code used below for DESCRIBE JOB.
With a compatible SDK, job.stats() returns that report verbatim, or None
for the null case; clients must not reconstruct terminal timings from a partial
event window. The structured profile is available for programmatic use and
contains the Function-specific measurements described in
Function refresh summary fields.
DESCRIBE JOB '<job_id>' — also spelled SHOW JOB SUMMARY '<job_id>' or
SHOW JOB '<job_id>', or abbreviated DESC JOB — shows where a finished job
spent its time, one
line per row as EXPLAIN shows a plan: its totals, a timeline of its phases,
and for a phase that runs on many workers at once, what their time went to.
It works for any job; for a Function refresh the phases are queued, planning,
starting workers, running splits and committing. A job still running returns
no rows. FORMAT TABLE returns its timeline and worker-time rows, with unit
saying whether millis is wall time or worker time. FORMAT JSON returns one
JSON document. For a Function job with GPU samples, text output adds a GPU
utilization line and JSON includes gpu_utilization with the number of sampled
tasks and device observations, the wall-time-weighted mean compute utilization,
and peak device memory. CPU-only, unsampled, and non-Function jobs omit the
field. It is derived from the existing once-per-second worker samples: it is not
an instantaneous cluster metric, a peak compute utilization, or evidence that a
GPU was resident for the whole Job.
When Function workers reported their runtime, text output also includes a
Function environments section. Identical environments are grouped across
workers, while different hardware or dependencies remain separate. FORMAT JSON and the Stats route include function_environments with the full reports
and worker/Function mappings. These reports are available for failed jobs too;
they are omitted for older jobs without a recorded environment. See
Inspect a Function worker’s environment.
JSON and the Stats route also include function_refresh when the terminal
Function summary exists. It carries the slot, split, batch and worker
measurements below with their separate scopes; the field is absent for jobs
without that summary.
Function refresh summary fields
A successful Function refresh stores these measurements injob.result["summary"]. /v1/jobs/describe fills
job_created_at_millis from the registry’s actual creation record, not the
query server’s clock or a later execution’s start. Older results can omit a
field; an omitted measurement is unknown, not zero.
Across refreshes, the
system.public.function_refresh_metering view holds one
row per succeeded refresh with the same summary, once the job events export has
run. It needs system Usage and Select on the view. This grants access to refresh
metering across all databases in the cluster; the connection’s database does not
filter the view. Only grant this access to users who may read cluster-wide usage.
SHOW JOBS instead lists jobs in the connected database. Choosing a database in
the SDK does not narrow an API key’s permissions.
For a recovered refresh, the terminal result’s rows_assigned is the logical
count of accepted output rows. The summary’s rows_assigned uses that accepted
total, including partial output accepted from an earlier attempt; individual
completed split records can cover fewer rows. Use split records for split-level
accounting, not as the output total. rows_skipped remains a completed-split
subtotal.
system.public.job_metering.rows_processed counts rows handled by completed
Function execution splits: computed rows plus skipped rows, once per split
completion. Planning scans and fragment finalization add no rows. Failed claims
add no rows; completed split work can remain counted after cancellation or when
a later recovery discards its output. This measures execution work, not the
number of rows currently published in the table.
For succeeded Function refreshes, job_metering also reports rows_computed
from the terminal result’s rows_assigned, rows_skipped from the refresh
summary, and rows_failed from the terminal result. rows_computed agrees with
function_refresh_metering.rows_assigned, including accepted recovered output.
rows_skipped includes deferred rows and row failures; rows_failed counts
terminal row failures. These totals are NULL for jobs without the corresponding
result, including failed or canceled jobs, and for other job types.
rows_reexecuted counts row inputs passed to later user-code invocations
because of an explicit retry budget or legacy bisection inside their logical
batch. bisection_rounds counts failed parent calls that were split into child
calls. Completed split records include retry work whose output a later replan
discards. A repeated receipt for the same task and split replaces its earlier
record instead of adding it again. The terminal summary deduplicates replay
work by durable part, so a runtime restart that rebuilds a completed split
under a new task ID does not count it again, while durable parts from a
discarded execution remain counted. Both values are absent for images or jobs
that do not record the measurement, and an aggregate is absent if any
contributing part is unknown. They measure Function row replay, not every
external request a Function makes and not infrastructure retry or worker
recovery.
These counters are currently available in an individual refresh’s terminal
summary and profile. The cluster-wide function_refresh_metering view
described below does not project them.
For a database-specific report, filter database_name explicitly, as below.
This query filter is not an access control boundary. Compare a Function’s
versions, or follow one over time:
See also
Refresh a computed column
The same operation from the SDKs, and what a refresh guarantees.
Table
Declaring the columns this fills.
Garbage Collection
How retained versions and Managed Blob objects are reclaimed.
Branches
Shared, writable lines of table history.