Enterprise only
Job handles
A job handle lets you track an operation, such as an index build or a computed column refresh. Use it to check progress, wait for completion and inspect failures. Methods such as Python’screate_index_async and TypeScript’s
createIndexAsync return job handles. In LanceDB Embedded, the work runs in your application
process; in Enterprise, it runs on server workers or through the SQL query
backend. A returned handle can represent an operation that has already finished,
so check its status before
waiting or requesting cancellation. See the operation’s API reference for when
the method returns.
Cached properties and live calls
Python job properties describe the last observation; reading them does not contact the server. Submission need not fetch the full registry record, so some properties can initially beNone.
For example, poll a registry handle and then retrieve its final details:
state alone. Structured progress methods and queue fields require matching
SDK/server support; see refresh admission.
Submit and wait for a job
- Submit the operation. Keep the returned job handle. For an Enterprise registry job, also keep its ID if another process needs to look it up later.
- Check progress. Read the current status while the operation runs.
- Wait for completion. Check that the job succeeded before using its result. If it failed or was cancelled, inspect the failure details and any results it committed before stopping.
job.status() returns the current state:
job.status() checks the current state without waiting for completion.
It returns failed or cancelled for unsuccessful jobs.
Use job.wait() to wait for completion and raise if the job fails or is
cancelled. Both Job.wait() and AsyncJob.wait() retry transient connection
and read errors with capped
exponential backoff within the caller’s timeout. Without a timeout, retries
continue indefinitely.
If the timeout expires while disconnected, the wait raises ConnectionError
with the job ID and instructions to resume waiting. A timeout while connected
raises TimeoutError. The job may still be running; timing out does not cancel it.
wait() again on the original handle. These jobs
cannot be reopened with db.open_job(); see SQL-backed job handles.
Inspect and cancel jobs
Reopening a job retrieves the existing operation; it does not restart failed
work. Treat the ID as an opaque identifier and pass it unchanged to the jobs API.
list_jobs yields JobInfo summaries, newest creation time first, rather than
live Job handles. Open an ID to inspect its full record. page_limit bounds
each server request, not the total yielded items, and page_token resumes a
listing. It is a lazy iterator: iteration performs requests and can raise
request errors. Use list(db.list_jobs(...)) to collect all remaining results;
there is no .to_list() method.
Job filters and synchronous
Listing indexing, slicing, and len() require
a Python SDK release that includes these APIs. Older SDKs expose only
page_token and page_limit; collect results with list(...) before indexing
or counting them. Filtering also requires a matching Enterprise server that
supports the requested filters.table, state, job_type
or job_types, and creation-time bounds. table names a table in
namespace_path (the root by default) and includes all its branches. Requests
stay scoped to the connected database. Creation times are Unix milliseconds:
created_at_gte_millis is inclusive and created_at_lt_millis is exclusive.
Choose either job_type or job_types.
The state filter uses registry states: in_progress, succeeded, failed,
canceled, and timed_out. These differ from the SDK lifecycle labels such
as running and finished returned in JobInfo.state.
jobs[:5] fetches only enough
pages to reach its stop. len(jobs) returns the remaining count by fetching
and caching every remaining result. Negative indexes and open-ended or reverse
slices also fetch and cache all remaining results. These operations can raise
request errors; later iteration drains their cache. AsyncListing remains an
async iterator: use async for to read it.
Listing.num_page_results() reports cached items available without another
request, including items retained by indexing, slicing, or counting.
Listing.page_token() reports the next request’s token. Drain all cached
items before saving that token, or resuming can skip them. Resume with the same
filters, for example db.list_jobs(table="events", state="in_progress", page_token=saved_token, page_limit=5). A None token after the final page can
coexist with unread cached items. A resumed listing is not a transactionally
frozen view of jobs submitted in the meantime.
events.schema describes the Arrow response. Use events.to_pylist() for
Python row dictionaries. Common event columns are event_seq (int64), state
(string), updated_at_millis (int64), updated_by (string), and nullable
emitted_from, emitted_by and claim_entity strings. Optional uint64 work
counters such as rows_processed and diagnostic details depend on job type
and server version. If a payload column is returned, its value is a nullable
JSON string, not an already decoded Python dictionary.
Empty results retain their schema. Events are newest first and belong to the
requested job; child events are separate. An event count is not a completed-row
count, and a running job can have no terminal result while producing events.
For a Function refresh, wait() returns a RefreshColumnResult such as
rows_assigned, rows_failed, rows_remaining, source_version and optional
published_version. Other job types have different results. Do not parse the
printed representation to recover a spec, result or progress measurement.
For object-shaped registry data, job.spec and job.result are decoded Python
dictionaries: access fields with keys, such as job.result.get("summary")
after refreshing a successful Function job. The separate typed object from
wait() uses attributes, such as result.rows_assigned.
Event responses have row limits. An explicit limit can
request more than the default, but it does not guarantee a complete event
history.
Embedded and Enterprise execution
The following table describes LanceDB Embedded operations and registry jobs on LanceDB Enterprise. SQL-backed job handles follow the query lifecycle described below.SQL-backed job handles
On Enterprise, several catalog operations return a job handle backed by the SQL query engine rather than the job registry:
All of these report
job.job_type as "query", with job.id holding the
connection-scoped SQL query UUID — use describe_query(job.id) to inspect
one. Unlike registry handles, they don’t expose event history, can’t be
reopened through open_job() or reattached after a client restart, and
cancellation depends on the SQL server’s support for it.
Object teardown
The “physical cleanup continues in the background” behavior above, for a dropped table, view, materialized view, Secret or Function, is a job type of its own:drop_object deletes the dropped object’s content, then clears it
from the database’s cleanup backlog. You trigger it indirectly, by dropping
one of those objects or a namespace — it runs as
this job rather than inline with the call. Whether a dropped table runs as
drop_object or as drop_table, which fences the
table’s directory with its own marker instead, depends on how the deployment
stores its catalog.
Learn more
- Incremental Indexing: submit index work.
- Refresh a computed column: populate derived values.
- SQL job commands: list jobs and inspect errors.
- SDKs: find the job API for your client.