> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# REFRESH

> Filling computed columns and bringing materialized views up to date.

**Applies to:** Embedded ❌, Enterprise ✅

LanceDB extensions with no DataFusion equivalent. They are what makes
[feature engineering](https://docs.lancedb.com/oss/v0-41-0/feature-engineering.md) reachable from SQL: a column
declared by [DDL](https://docs.lancedb.com/oss/v0-41-0/sql/ddl/table.md#computed-column) starts empty, and a refresh fills it.

## REFRESH COLUMN

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN <column> ON [TABLE] <table>
  [WHERE <predicate>]
  [WITH (<option> = <value> [, ...])]
```

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN doubled ON my_table
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
```

Without `WHERE`, a refresh fills unassigned rows and recomputes values whose
inputs changed, leaving other current values alone. A retry after a failure
can still repeat work that was not committed.

With `WHERE`, a refresh **recomputes every matching live row**, including rows
that already have current values. Repeating the same predicate can repeat the
Function calls and their cost. To fill only null outputs within a slice, add
the output column to the predicate:

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN embedding ON documents
  WHERE created_at > '2026-01-01' AND embedding IS NULL
```

This skips non-null outputs even if their inputs changed. It also selects
outputs that were successfully computed as `NULL`; a null value does not
necessarily mean the row has never been computed. Use an unfiltered refresh
to let LanceDB track which outputs need updating.

An initialized Function instance and its mutable state may survive across
refresh jobs. Do not assume a new refresh resets instance state or calls
`__init__` again. Instances are not shared across workers.

Refresh computed columns in dependency order. A refresh fails before writing
if the rows it would compute read another computed or Function column that has
not been refreshed for its current inputs. Refresh that input first. A
computed input that is current but evaluates to `NULL` is a valid value, not
an unfilled placeholder.

`WHERE` scopes the refresh, which is how you fill a slice and check it before
committing to the whole table. It does not narrow the per-fragment input
readiness checks: if a selected fragment contains unfilled computed input rows
anywhere, refresh that input first and then rerun the filtered child.

### Previewing a refresh

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
EXPLAIN REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
```

`EXPLAIN` and `EXPLAIN VERBOSE` take the same options and validation as
`REFRESH COLUMN` and return DataFusion's `plan_type`, `plan` rows:

* `logical_plan`: the `RefreshColumn` statement.
* `physical_plan`: a `RefreshJobExec` root carrying what the job runs with
  (the snapshot, worker count or Function resource budget, error policy,
  batch size, row timeout, how many fragments have work, which fragment is
  shown) over the plan each worker executes for one fragment: a
  `RefreshStageExec` stages the replacement file from the stored values and
  the rows `StaleRowsExec` selects, computed by a `ProjectionExec` for an
  expression column or a `PythonUdfExec` for a Function column.

Nothing runs: the plan is built against a snapshot without submitting a job,
starting a Function executor or taking a table lock. A refresh started later
pins its own source version.

Job metadata and the final result retain the original Job `source_version`. If a
concurrent `UPDATE` or compaction invalidates staged fragments, a Function
declaring `result_stability=input_and_context` and `side_effects=none` can
re-plan the affected connected fragments for at most three job-wide rounds.
Each successor request explicitly pins the newer snapshot it plans, executes,
and finalizes against. These request-local snapshots do not replace the Job's
original source pin. Unaffected staged results commit without rerunning, while
effectful or per-call Functions fail with the named concurrent-write error.

`EXPLAIN [VERBOSE] JOB '<job_id>'` renders the same tree for a submitted
job, with the registry's view on the root: state, progress and the job's
options. The tree is rebuilt from the job's spec against the table's
current version on its branch (the root's `source=`), not the snapshot the
job pinned when it started, so a representative fragment can differ from
what a worker ran. A job whose work is not a plan, such as an index build
or a compaction, renders its root alone.

The predicate also accepts `_rowaddr` and `_rowid`. To retry specific rows
reported by `table.function_errors()`, use each record's `row_id` with `_rowid`
when it is present. This stable ID continues to identify the same logical row
after compaction or another rewrite:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN embedding ON documents WHERE _rowid IN (506, 507)
```

For records without `row_id`, combine `fragment_id` and `row_offset` as
`(fragment_id << 32) | row_offset` and select those physical addresses before
compaction or another rewrite moves the rows:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN embedding ON documents WHERE _rowaddr IN (0, 4294967297)
```

Only matching live rows are recomputed. A successful Function refresh removes
their earlier row errors from `function_errors()` and `SHOW ERRORS` when their
physical locations have not changed; rows that fail again remain listed.
After compaction or another rewrite moves a row, `_rowid` still selects it,
but its earlier error may remain listed even when the retry succeeds. Error
clearing currently matches physical locations, not stable row IDs.

Older error records and tables without stable row IDs may omit `row_id`.
Errors without an individual row location, including those whose per-row
detail was capped, cannot all be selected individually.

### Options

| Option | Meaning |
| - | - |
| `num_workers` | fixed worker count for SQL-expression columns; rejected for registered Function columns |
| `cpu_millis` | aggregate CPU ceiling across the refresh's Function workers, in thousandths of a CPU |
| `memory_bytes` | aggregate memory ceiling across the refresh's Function workers, in bytes |
| `gpu_class` | operator-defined GPU pool, such as `'a10g'` or `'h100'` |
| `gpu_count` | aggregate GPU ceiling across the refresh's Function workers |
| `max_executor_processes_per_worker` | cap on executor processes per worker |
| `batch_size` | hard maximum rows passed to one Function call; LanceDB normally chooses a smaller adaptive size |
| `error_policy` | Function columns only: `'fail'` (default) stops after configured retries; `'skip'` writes null outputs and records row errors, subject to failure limits |
| `retry_policy` | JSON policy for bounded retries of failed user-code calls; overrides the Function default |
| `max_failed_fraction` | with `'skip'`, fail after more than this fraction of evaluated rows fail; between `0` and `1` |
| `max_failed_rows` | with `'skip'`, fail after more than this many evaluated rows fail |
| `row_timeout_seconds` | maximum time per Function call in seconds; overrides the Function's declared deadline and informs adaptive call sizing for this refresh |
| `init_timeout_seconds` | maximum time for a Function `__init__` handshake in seconds; default `900`, separate from row execution and task preparation timeouts |

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN embedding ON documents
  WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
```

`num_workers` sets a fixed worker count for expression-backed refreshes.
Registered Function columns reject `num_workers`, even without a resource
budget. Their worker count adapts within the aggregate resource ceiling. The
example above allows up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the
Function workers together. It does not reserve those resources for each worker
or guarantee that all of them will be used.

`gpu_count` may be set alone to use the deployment's default CPU and memory.
When setting either `cpu_millis` or `memory_bytes`, provide all three budget
fields. For a CPU-only Function, use `gpu_count = 0` in the complete budget:

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN caption_length ON documents
  WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 0)
```

LanceDB adjusts the amount of work sent to each worker from observed execution
time. `batch_size` limits rows in one Function call; it does not set the worker
count or the size of a checkpoint. Small initial batches are expected while
the refresh learns how long rows take.

GPU class names are lowercase names of at most 63 letters, digits, or hyphens.
The REST `backfill_column` body accepts `gpu_class`, `gpu_count`, and
`init_timeout_seconds` as top-level options with the same validation and budget
semantics as SQL.

### Retrying temporary Function errors

Set `retry_policy` on a source Function at registration, or override it for one
refresh. The same JSON object is accepted by `CREATE FUNCTION ... WITH
(retry_policy = '...')` and by the registration and refresh REST APIs.

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN caption ON videos
WITH (
  retry_policy = '{"max_attempts":3,"transient":true,"initial_backoff_seconds":0.2,"max_backoff_seconds":30}',
  error_policy = 'skip'
)
```

`max_attempts` includes the first call and accepts 1–100. An explicit
`{"max_attempts":1}` disables user-code retries, including the built-in transient
retry. Omitting the option preserves the Function default and existing behavior.

Choose one or more matchers; matching any one allows retry:

| Field | Matches |
| - | - |
| `exception_types` | Exception class names, preferably qualified, such as `["builtins.ConnectionError"]`; subclasses also match |
| `message_pattern` | A regular expression in the rendered exception class and message |
| `error_codes` | Structured provider codes or HTTP status codes, such as `["SlowDown", "503"]` |
| `transient` | Built-in `ConnectionError`/`TimeoutError`, the Botocore transport exceptions listed below, HTTP 429/503, and provider codes `SlowDown`/`ServiceUnavailable` |

The Botocore transport exceptions are `ReadTimeoutError`, `ConnectTimeoutError`,
and `EndpointConnectionError` from `botocore.exceptions`. Requests and HTTPX
transport exceptions require explicit `exception_types`, such as
`["requests.exceptions.ConnectionError", "httpx.TransportError"]`.

The runtime also checks wrapped causes. It does not infer a transient error
from digits such as `429` appearing in an arbitrary message. An adapter may
signal throttling with `status_code` or `code`, and `retry_after_seconds`.
Message patterns must compile under both Rust's regex validator and the
Function environment's Python `re` module.

Retries use exponential backoff with full jitter. `initial_backoff_seconds`
defaults to 0.2 and `max_backoff_seconds` to 30; both must be finite, between
0 and 3600, with the initial value no larger than the maximum.
`honor_retry_after` defaults to `true` and honors HTTP `Retry-After` or the
adapter's `retry_after_seconds`. A requested delay above the configured maximum
ends retry instead of calling the provider early. Cancellation interrupts backoff.

The retry unit is one row for a row Function and one whole call for a batch
Function. Completed calls are not repeated. With an explicit policy, a failed
batch is not bisected after retries: `skip` nulls every row in that failed call
and records its error and `attempts`. Use a row Function when failures must be
isolated to individual rows. Output conversion and partially consumed generator
results are not retried.
Invalid values in a returned batch are isolated from that existing result,
without calling the Function again.

Only opt in when repeating a partially failed call is safe. The runtime cannot
undo effects a call performed before raising. Functions declaring external side
effects are refused. User-code retries do not consume the worker's infrastructure
attempt budget; initialization, worker loss and OOM recovery keep their own policy.
After exhaustion, `fail` fails the job and `skip` records the last error and count
in `SHOW ERRORS` and the table errors API, subject to the normal failure limits.

A Function image created before retry support must be registered as a new
version before a refresh can override its policy.

### Hugging Face model snapshots

A Function can declare immutable Hugging Face repositories and revisions during
registration. The worker prepares each declared snapshot before it creates the
user Function instance:

```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lancedb import HuggingFaceModel, udf

MODEL = HuggingFaceModel(
    repo="sentence-transformers/all-MiniLM-L6-v2",
    revision="1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
)


@udf(
    name="embed_text",
    pip=["sentence-transformers==3.3.1"],
    models=[MODEL],
)
class EmbedText:
    def __init__(self):
        from sentence_transformers import SentenceTransformer

        self.model = SentenceTransformer(
            "sentence-transformers/all-MiniLM-L6-v2",
        )

    def __call__(self, text: str) -> list[float]:
        return self.model.encode(text).tolist()
```

Each repository may be declared once, and its revision must be the full
40-character commit hash. A completed snapshot stores that revision in
`refs/main`, so the original no-revision `SentenceTransformer(repo)` call
resolves without contacting the Hub. Model-declaring Functions run with a
private `HF_HOME`, `HF_HUB_OFFLINE=1`, and `TRANSFORMERS_OFFLINE=1`; those
environment variables cannot be overridden through the Function `env`.

Before SQL execution or refresh worker admission, a shared preparation job
imports the snapshot into database object storage. Concurrent consumers with
the same database storage, pinned Secret scope, repository and revision reuse
that attempt and its immutable snapshot. A code-only Function version can
reuse the snapshot when these inputs are unchanged.

Preparation failures are retained: waiting consumers do not automatically
retry Hugging Face. An operator with System `Operate` privilege can retry the
observed preparation job through `POST /v1/jobs/retry_model_preparation`, with
`{"job_id":"<observed preparation attempt>"}`. After preparation succeeds,
resubmit failed queries or refreshes; retrying the import does not restart them.

Workers read prepared snapshots from database storage into a local model
cache. With `computedRefresh.execution.imageCache.hostPath`, workers on one
node share those local bytes. Without it, local materialization is private to
the worker, while the shared snapshot still avoids repeated origin downloads.
`computedRefresh.execution.modelCache.maxBytes` bounds local model bytes
independently of the image cache, with a default of 32 GiB. A snapshot that does
not fit is rejected before user code starts. For refreshes,
`computedRefresh.execution.modelCache.preparationConcurrency` limits concurrent
worker preparation requests to 16 by default. A request releases its slot when
the snapshot is ready, without waiting for Python initialization. This per-refresh window
does not impose a cluster-wide bandwidth quota.

Without `models`, the existing behavior is unchanged: the Hub client may still
request metadata or missing files, so a cached model and pinned revision alone
do not guarantee offline execution.

### Discovering GPU classes

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SHOW GPU CLASSES;
```

The result lists the legacy `default` class first, followed by named classes.
Columns are `name`, `resource_name`, `gpu_model`, `gpu_memory_bytes`,
`max_gpu_count_per_worker`, `cpu_millis`, `memory_bytes`, and
`ephemeral_storage_bytes`. Optional values are `NULL` when the operator did not
configure them. A non-NULL `max_gpu_count_per_worker` bounds adaptive growth
for one worker; NULL leaves the refresh's aggregate GPU budget as the ceiling.

The authenticated REST endpoint `GET /v1/gpu/classes` returns the same catalog
as `gpu_classes`. It reports configured hardware, not current free capacity.

### Row errors

Set `error_policy` on the refresh, not on the Function definition. With `skip`,
inspect the recorded failures with `SHOW ERRORS` or `table.function_errors()`
and retry the affected rows after fixing the cause. Skipping row errors does
not suppress initialization, infrastructure, or publication failures.

Under `error_policy = 'skip'`, a refresh defaults to failing when more than 50%
of the first 1,000 evaluated rows in a fragment part fail. It also fails if the
first 100 evaluated rows fail before any successful row. `max_failed_fraction`
or `max_failed_rows` replaces that default fraction; when both are present,
either limit can fail the Job.

The counters are evaluated rows, not Function calls. Checks continue
cumulatively after the 1,000-row sample and also run when a smaller part
finishes. `batch_size` defines call boundaries: the worker checks the shared
counts after every inner Function call, so a row-capped batch cannot continue
past the ceiling. The part that crosses a limit is not published; fragments
committed before it remain visible.

LanceDB normally sizes calls from observed row latency and targets at most 25%
of the row deadline. `batch_size` remains a hard upper bound, not a tuning
requirement. For a Function whose image declares
`result_stability=input_and_context` and `side_effects=none`, a timeout or
resource failure can enter range recovery, which retries smaller physical-row
ranges with a fresh executor.

Under `error_policy = 'skip'`, a row timeout can enter range recovery when the
Function declares `side_effects=none`, including when its result stability is
undeclared or `per_call`. Ordinary vectorized batch exceptions already use the
same best-effort bisection under skip. Because a failed batch has no row
results to salvage, healthy calls in that batch run again and a per-call
Function may return different values. A single row that still times out is
written as null and recorded as one `RowTimeoutError`. Functions with declared
effects are not replayed automatically.

A quarantined terminal row is an exception to the per-part failure limits.
Under `skip`, this applies to an OOM-poison row in a replay-safe Function, a
typed `OversizedRows` failure for exactly one physical row after range recovery
has reduced the unfinished work to that same row, and a non-interruptible row
timeout in a `side_effects=none` Function. Oversized-row quarantine requires
the executor evidence to name one physical row that matches the only
unfinished row in the recovery range; multi-row or mismatched evidence does not
become a skipped row. OOM and oversized-row quarantine require
`result_stability=input_and_context` and `side_effects=none`; declared effects
disable automatic replay. Under the default `fail`, a terminal oversized row
fails the refresh without a skipped-row record.

Quarantine writes one null row as a synthetic part, so applying the per-part
fraction or initial-100-row guard there would reject every quarantine. Before
bisecting, the supervisor retries the OOM-killed claim once by itself with one
executor process on the largest eligible worker class; a row timeout reaches
quarantine only after its execution range has been reduced to one row. The
skipped row still counts toward the cumulative job-level failure limits, and
the job fails if the quarantined rows cross either limit.

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN embedding ON documents
  WITH (error_policy = 'skip', max_failed_fraction = 0.05)
```

Per-row Function failures can include a Python traceback, bounded to a UTF-8
prefix or omitted when the reported errors exceed the byte limit. The compact
error message and skipped-row count remain complete. The persisted error file
can retain a prefix of records in row order while `rows_skipped` covers the full
count. If the compact skipped-row reports alone exceed the event limit, the
refresh fails before publication. Skipped-row tracebacks require a newly baked
source image.

### Long-running Functions

Source Functions default to a 300-second deadline per row. Set
`row_timeout_seconds` high enough for the longest row, for example:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH COLUMN analysis ON videos WITH (row_timeout_seconds = 3600)
```

The batch watchdog accommodates the row deadline and the number of input
rows, so a row allowed an hour can run past 15 minutes. In vectorized mode,
the row deadline applies to each vectorized call. Automatic sizing starts with
a small probe and adjusts later calls; an explicit `batch_size` only lowers
the ceiling. A source Function can call
`heartbeat()` from the runtime-provided `lance_progress` module inside
`__call__` to report liveness. Heartbeats do not extend the row deadline.

`init_timeout_seconds` bounds only the interval from `Initialize` until the
worker receives `Initialized`; image preparation, Secret resolution, and
cleanup have their own bounded task handling. While a worker is waiting for
`Initialized`, `SHOW JOBS` progress reports the `initializing` phase and
initialized worker counts. It does not count endpoint readiness as
initialization completion. If the budget expires, the refresh fails with an
initialization-stage error and the bounded child stderr, even when
`error_policy = 'skip'`.

The row deadline does not extend the job's execution limit. On AWS deployments
that vend temporary storage credentials, a Function refresh attempt is capped
at **11 hours 40 minutes (42,000 seconds)**. A shorter deployment or job limit
still applies; increasing the deployment timeout cannot lift this credential
limit. Split longer work into disjoint filtered refreshes, keeping in mind that
each predicate recomputes all matching rows.

Other deployments use their configured job limit. Automatic column refreshes
also have a 24-hour execution limit; an explicit refresh is still subject to
the deployment and credential limits.

> [!NOTE]
>
> `num_workers` is refused for a column filled by a registered
> [function](https://docs.lancedb.com/oss/v0-41-0/management/function.md); a Function refresh's worker count follows
> its own resource budget instead. `error_policy` (and the other Function-only
> options in the table above) is the reverse: it's refused for a plain
> SQL-expression column, which has no per-row failures to skip, and accepted
> only for a Function column.

### Automatic refresh timing

Automatic column refresh requires a supporting server release and an enabled
deployment. A column's `auto_refresh_interval_seconds` sets a minimum delay
after its last successful refresh, not a completion deadline. The scheduler
also waits for its computed inputs, available workers, and other refreshes.
The deployment default allows one column refresh in flight per table.

Failed automatic Function refreshes start with a 15-minute retry delay by
default, then double it for consecutive failures: 15, 30, 60, 120 minutes, and
so on. This is failure backoff, not a fixed refresh cadence. A manual refresh
can run during that wait. Cancelling a Function refresh blocks automatic
refresh for that binding on the same branch until a later explicit refresh
completes or the binding is replaced. Check job history when automatic results
stop appearing.

## REFRESH MATERIALIZED VIEW

The syntax and options below apply to Enterprise deployments that support
explicit `full = false` in SQL.

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH MATERIALIZED VIEW <name>
  [WITH (full = <boolean>)]
```

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH MATERIALIZED VIEW chunks
REFRESH MATERIALIZED VIEW chunks WITH (full = false)
REFRESH MATERIALIZED VIEW chunks WITH (full = true)
```

A refresh updates the view's stored result. Refresh views in dependency order,
starting with their source views. `CASCADE` is not supported.

| Option | Meaning |
| - | - |
| `full = false` (default) | Refresh incrementally when supported, or rebuild when incremental refresh is unavailable. An eligible view whose sources are unchanged completes without rewriting data. |
| `full = true` | Rebuild the complete stored result, even when the sources are unchanged. |

> [!NOTE]
>
> Servers without SQL refresh option support require the basic statement without
> `WITH`. Older servers may accept only `full = true`; some also accept `CASCADE`.
> A local SDK's `refresh()` options do not determine which SQL options a server
> accepts.

### Function row errors in materialized views

Choose the policy when creating the view:

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
CREATE MATERIALIZED VIEW captions
WITH (error_policy = 'skip') AS
SELECT id, caption(video) AS text FROM videos;
```

`error_policy = 'fail'` is the default, including for views created without this
option. A Function failure aborts the refresh and preserves the previously
published result. With `skip`, a failing scalar Function input produces NULL;
a failing table Function input produces no output rows. Planning, storage,
worker, and commit failures propagate to normal job failure and recovery
handling; they are not converted into skipped inputs.

Custom Function images must report a failure as attributable to the input before
it can be skipped. Errors from older images that do not distinguish callable
failures from worker failures stop the refresh.

Use `SHOW ERRORS ON captions` or the SDK's `table.function_errors()` to inspect
the diagnostics. `SHOW ERRORS` adds an `input` column containing JSON identifying
the Function invocation and input offset. When a failed input can be traced to
stored source rows, it also includes the source table, pinned version, and row
address or stable row ID. Inputs computed by expressions or aggregations may
have no single source row. Input offsets are not physical row offsets in the
view. SDK error messages also include the source locator; use it instead of the
view fragment fields for MV errors. Error records describe attempts and remain
available after a later successful refresh.

An incremental refresh over appended source rows leaves previously skipped
inputs alone. When a change to existing source rows recomputes a block of the
view, every input in that block runs again, skipped ones included. To retry
them all, run:

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
REFRESH MATERIALIZED VIEW captions WITH (full = true);
```

A full refresh recomputes all inputs, including previously successful ones.
The policy is fixed at creation; refresh does not accept a policy override.

### Interrupted materialized-view jobs

LanceDB can replay `CREATE MATERIALIZED VIEW` and supported
`REFRESH MATERIALIZED VIEW` operations, including `full = true`, after an
infrastructure interruption. Other SQL statements remain non-replayable.

A replacement worker keeps the same Job, job handle, and original deadline. A
counted infrastructure disruption can use one replacement worker. User
cancellation and permanent SQL or Function errors terminate the Job, and a
terminal Job is not restarted.

Recovery can run Function code again when the data commit has not completed, so
external side effects from that attempt can repeat. After the data commit,
receipt recovery finishes state and catalog publication without computing the
committed rows again, and it does not need to plan the original source or
Function. Cancellation does not guarantee that an in-flight storage write
stops; a data or catalog request that was already issued or sent may still
complete. Recovery therefore does not guarantee exactly-once execution of
external side effects.

## Watching a refresh

When a client requests an early job handle and the server can make the job
registry available to it, the refresh returns a `job_id` with status
`submitted`. The work then outlives the request. Track it with
[`SHOW JOBS`](https://docs.lancedb.com/oss/v0-41-0/sql/ddl/job#show), and diagnose failures with
[`SHOW ERRORS`](https://docs.lancedb.com/oss/v0-41-0/sql/ddl/job#errors). The registry APIs require both
`System Usage` and `System Operate`; refresh access scoped to a database or
table is not enough.

Submitting a Function refresh creates work; it does not reserve Function worker
capacity or guarantee that a warm worker is available. A queued job reports a
current admission observation when the dispatcher has one, but submission has
no global queue position or start-time estimate.

`CREATE MATERIALIZED VIEW` returns `(operation, name, status, job_id)`. The
original three columns keep their positions; `job_id` is appended and is empty
when the statement ran natively without a registry job.

When the caller lacks both system privileges above, the server omits the early
job-handle acknowledgement and the query continues through ordinary adaptive
polling. Continue polling the returned descriptor until it completes, then read
the result; the final refresh success or error is delivered through that path.
When the bounded retained-run store is full, the server runs the refresh
synchronously and reports its actual outcome without announcing a change of
mode. A client must use the early acknowledgement and status `submitted`, rather
than the presence of `job_id` alone, to decide that it received an early
registry handle. Published view results remain visible whether the refresh was
polled or run through the synchronous capacity fallback.

Use `SHOW JOBS FOR JOB '<job_id>'` to discover a refresh's child jobs and
inspect their phase, row progress, and attempt information. Pass a returned
child ID to the same event command used for the parent:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SHOW JOBS FOR TABLE frames;
SHOW JOBS FOR JOB '<parent_job_id>';
SHOW JOB EVENTS '<child_job_id>' LIMIT 100;
```

Parent events do not include child events. Child diagnostics can expire at
archival; the CLI reports unavailable history even when no rows remain.

While a supervised refresh runs, `SHOW JOBS` includes a `progress` column
containing a JSON sample: rows planned, computed, skipped and remaining,
published and retained, failed rows, fragments planned and completed, active
workers and active fragments, and the last row-progress timestamp when
available. The same `progress` object is returned by
`POST /v1/jobs/describe` and `POST /v1/jobs/list`. Samples update
approximately every 30 seconds and after tracked progress changes. While at
least one worker endpoint is ready and the latest sampled worker phase is
`preparing`, `initializing`, or `initialized`, startup samples are published
approximately every second so short initialization transitions are more likely
to be observed. They are absent before the first sample and while the job is
paused. Succeeded, failed, timed out, and canceled jobs retain the latest row
progress and timing from the execution that ended, when a sample was recorded.
Terminal jobs zero live worker, GPU, and active fragment counts and clear
`resource_classes`, `warm_reused`, and wait reasons. `sample_at_millis` and
`execution_epoch` identify that sample. A client may estimate remaining time
from two observations only when the epoch and planned work are unchanged and
accounted work advances; an estimate is unknown at startup, after a pause or
epoch change, and when either sample is missing or stale. The Python SDK exposes
a defensible estimate as `estimated_remaining_time_millis` and omits the field
when the evidence is insufficient; the server does not persist or return a
historical sampler.

The optional `worker_fleet` object reports the Function-role worker cohorts
from the supervisor's latest completed status observation. `desired` is the
policy target for Function workers and can exceed the cohorts that have been
created. `launched` counts surviving workers whose cohort launch was confirmed,
`pending_workers` counts observed active workers not yet ready to accept
dispatch, including workers still initializing. `ready` counts distinct
endpoints in that launched set that can accept dispatch after status and worker
announcements are merged. When an observation does not provide a usable
pending count, readers may derive it from `active - ready`; this fallback only
describes currently visible endpoint evidence and is not a guarantee about
workers that have not yet become visible.
`resource_classes` lists the observed class names. Draining, reaping, and dead
workers are excluded consistently. `requested_gpus` totals the GPU devices
requested by the observed Function cohorts; `ready_gpus` totals devices on
dispatchable endpoints. Neither number is free cluster capacity or a queue
position. `warm_reused` is `true` when at least one observed cohort was proven
to come from a reusable pool. It is `false` only when every observed cohort
was proven cold, and absent when no cohort was observed or the observations
are incomplete. A `true` value does not guarantee that every worker in the
fleet avoided a cold start. `worker_fleet` is absent until a coherent
Function-worker observation exists. `first_launched_at_millis`,
`first_ready_at_millis`, and `last_ready_at_millis` record milestones for the
current supervisor run. Readiness means a dispatchable endpoint, not that
Python initialization or the first function call has completed.
When no dispatchable endpoint exists, `worker_fleet.wait_reason` may carry a
fixed substrate-derived wait reason; it is cleared once an endpoint is ready
or the job reaches a terminal state.

The top-level `SHOW JOBS.phase` reports `queued`, `paused`, `starting`,
`waiting_for_workers` (no Function workers have launched), and
`starting_workers` (workers have launched but no endpoint is ready). Once an
endpoint is ready, it mirrors the nested `progress.phase`, falling back to
`running` until the first worker sample. The nested phase reports the observed
worker phase: `preparing` before the initialization handshake, `initializing`
while the worker is inside the `Initialize` handshake, `initialized` after a
successful `Initialized` response, `running` while processing rows,
`finalizing` while publishing, and `staged` after local output is staged. The
top-level phase becomes `committing` once all planned rows are accounted for.
Terminal jobs report `finished`, `failed`, or `cancelled` when the outcome is
known. `workers_initializing` and `workers_initialized` count distinct live
workers at those handshake boundaries; endpoint readiness alone does not
increment either counter. These are sampled observations, so a short phase may
be missed, and neither phase claims that every worker is in the same state.
`queue_reason` is populated only from a current substrate observation; in
particular, before a Function supervisor starts, the dispatcher may report that
the job is waiting for executor capacity and include the worker slots requested
and currently available. Those values are a point-in-time admission observation.
A GPU-capacity reason requires PodScheduled=False evidence naming the requested
GPU resource. CPU pressure, taints, and image-pull failures are not relabeled as
GPU capacity waits. The object does not report global queue position or an ETA.

`rows_computed` counts evaluated rows, including work later recomputed.
`rows_published` counts evaluated rows backed by accepted publication receipts
at the sample. `rows_retained` counts completed staged output that no accepted
receipt has published yet; it is a sampled lower bound, not an exact final
cancellation count. `rows_failed` is present only when a successful terminal
result carries a typed failed-row count; it is not inferred from in-flight or
non-successful job state. `active_workers` counts owners of execute claims, including
workers that have not reported their first batch yet; it is separate from
`worker_fleet.ready`. Job listings return the most recently created jobs first
across tables.
`SHOW JOB SPLITS '<job_id>'` returns completed splits and, while the refresh
runs, the latest task for each split that has not completed. `state` preserves
the worker's serialized phase, including phases introduced by a newer server;
`queued`, `assigned`, `initializing`, `running`, `failed`, and `completed` do
not share a closed mapping in older clients. Measurements that live evidence
has not established are `NULL`, never a fabricated zero. `error_count` counts
explicit failed execute completions for the current task identity in the
current execution epoch, including failures that happened before a later
retry or success. Expired leases are shown as `queued`; retries, expiry,
success, replaced tasks, skipped rows, and row-level errors do not increment
`error_count`, and they do not erase failures already recorded.

Canceling a refresh leaves already published results visible. A later refresh
of the same definition may reuse completed staged parts while its inputs and
source provenance still match; parts affected by newer inputs or writes are
recomputed or rejected under the concurrency rules above.

Dropping a Function output column retires its binding and automatically cancels
running and queued refreshes for that binding on the same table generation and
branch. Failed drops and `IF EXISTS` drops that remove nothing leave refreshes
running. Refreshes for other bindings are unaffected.

`SHOW JOB SPLITS '<job_id>'` reports, per split: task, split and fragment
identifiers, the worker that ran it, its row range, rows skipped, bytes read
and staged, part and attempt counts, batch count, start and finish times,
`state`, `error_count`, and the executor batch latency as p50/p95/p99/max
in milliseconds. A split the supervisor recovered from a checkpoint rather
than a worker report carries `recovered = true`. `POST /v1/jobs/query_splits`
reports completed split records only -- not the live, not-yet-completed rows
the SQL form returns while the refresh is still running -- and its records
carry a different shape: no `state` or `error_count`, and a raw latency
histogram in place of the SQL form's precomputed quantiles.

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SHOW JOB SPLITS 'j1-fr-0mvoq6qbb-3e172e8f1e334086979a5b2c0f1d5f7f-mydb:frames';
```

`DESCRIBE JOB '<job_id>'` -- also spelled `SHOW JOB SUMMARY '<job_id>'`, or
abbreviated `DESC JOB` -- shows where a finished job spent its time, one
line per row as `EXPLAIN` shows a plan: its totals, a timeline of its phases,
and for a phase that runs on many workers at once, what their time went to.
It works for any job; for a Function refresh the phases are queued, planning,
starting workers, running splits and committing. A job still running returns
no rows. `FORMAT TABLE` returns the same profile as typed rows, with `unit`
saying whether `millis` is wall time or worker time, and `FORMAT JSON` as one
JSON document.

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
DESCRIBE JOB 'j1-fr-0mvoq6qbb-3e172e8f1e334086979a5b2c0f1d5f7f-mydb:frames';
```

```
j1-fr-0mvoq6qbb-3e172e8f1e334086979a5b2c0f1d5f7f-mydb:frames
18.7 s · 6 fragments · 300,000 rows · 102 splits · 6 workers

Timeline                  at     took
  queued             +0.00 s   0.18 s
  planning           +0.18 s   0.13 s
  starting workers   +0.31 s   0.36 s   launch 0.12 s, ready 0.01 s, dispatch 0.23 s
  running splits     +0.67 s  16.45 s   88%
  committing        +17.12 s   1.47 s   publishing 1.40 s, final claim 0.06 s
  finishing         +18.59 s   0.11 s

Worker time, running splits (109 s)
  supervisor handoff    36%  39.6 s
  instance start        30%  33.0 s
  finalizing fragments  13%  14.6 s
  in the Function       10%  10.8 s
  idle                   7%   7.5 s
  reading and writing    1%   1.6 s
  other                  1%   1.5 s   reporting, slot waits, uploads
```

Across refreshes, the `system.public.function_refresh_metering` view holds one
row per succeeded refresh with the same summary, once the job events export has
run. It needs system Usage and Select on the view. This grants access to refresh
metering across all databases in the cluster; the connection's database does not
filter the view. Only grant this access to users who may read cluster-wide usage.
`SHOW JOBS` instead lists jobs in the connected database. Choosing a database in
the SDK does not narrow an API key's permissions.

For a database-specific report, filter `database_name` explicitly, as below.
This query filter is not an access control boundary. Compare a Function's
versions, or follow one over time:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT function_version,
       count(*) AS refreshes,
       avg(slot_occupancy) AS mean_occupancy,
       max(batch_latency_p95_millis) AS worst_p95_millis
FROM system.public.function_refresh_metering
WHERE database_name = 'my_database' AND function_id = 'embed'
GROUP BY function_version;

SELECT job_id, rows_assigned, splits,
       finished_at_millis - supervisor_started_at_millis AS millis
FROM system.public.function_refresh_metering
WHERE database_name = 'my_database'
  AND function_id = 'embed' AND ended_at_millis > 1793000000000
ORDER BY ended_at_millis DESC;
```

## See also

> [card] [Refresh a computed column](https://docs.lancedb.com/oss/v0-41-0/feature-engineering/refresh-column.md)
>
> The same operation from the SDKs, and what a refresh guarantees.

> [card] [Table](https://docs.lancedb.com/oss/v0-41-0/sql/ddl/table.md#alter)
>
> Declaring the columns this fills.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.