Skip to main content
Enterprise only
Function registration, query execution and column refresh have different lifecycles. Match your SDK and Enterprise server release before relying on optional GPU discovery, progress fields or warm-worker reuse. The examples use the public @function decorator; see authoring guidance and Function signatures.

Query-time cold starts

A query that invokes a Python Function can pay for admission, worker placement, image preparation, Python startup, dependency import, model download and model initialization before producing its first row. These are separate kinds of warmth: Use a class Function to initialize expensive state once per instance and reuse it across that instance’s rows/batches. Each instance owns its own model copy. Supporting refresh deployments can reuse initialized instances across jobs; do not assume a new refresh resets mutable instance state. Query-time SQL execution has its own lifetime, and refresh-worker reuse does not promise a resident query-serving pool. A cached image is not a warm-latency guarantee. Measure time to first row separately from steady-state batch throughput, and record server/SDK versions, GPU class, model/dependencies and whether image, model files and initialized instances were retained. Do not extrapolate a single warm measurement to an independently submitted query or cold worker.

Latency-sensitive query embeddings

Keep document embeddings in a stored Function column and refresh them during ingestion or backfill. Compute each interactive query vector in an already initialized application model or a separately operated embedding service, then send the numeric vector to LanceDB. For example, with a long-lived SentenceTransformer instance in the application:
Construct encoder once at application startup. Use the same model revision, dimension, pooling and normalization as the stored document embeddings. A separate service needs its own capacity, authentication and timeout management; a thin HTTP Function does not make that service’s GPU part of LanceDB’s budget. SQL Function calls remain useful for batch transformations, exploratory queries and workloads that can tolerate startup. Precomputing document embeddings does not remove a query-time model cold start if each search still calls a remote GPU Function to embed the query string.

GPU placement and budgets

Discover the configured classes with Python db.list_gpu_classes(), SQL SHOW GPU CLASSES, or REST GET /v1/gpu/classes. These require supporting SDK/server versions and System Usage permission. The response describes the operator’s configured GPU resource name, optional model and memory per device, per-worker device ceiling and worker resources. It is configuration, not live inventory or a guarantee of currently available capacity. Missing metadata is unknown; a missing per-worker limit leaves the aggregate budget as the adaptive-growth ceiling. The deployment operator configures available classes and their per-worker limits. Choose a class supported by your deployment and a model that fits its worker shape. Choose a returned class when registering a Function with @function(gpu=True, gpu_class="h100"), or override the class for a refresh:
Here gpu_count caps devices across the Function worker fleet. It does not request two GPUs in one Python process, guarantee two workers, or configure your model library’s tensor parallelism. gpu=True requests GPU execution; it does not describe model memory requirements. A deployment’s adaptive worker shapes and maximum devices per worker are distinct from a supported explicit per-Function multi-device request. Do not assume a proposed registration option such as gpus_per_worker is available. Use a model that fits the supported worker shape, or a separately managed model service, until your release provides the required multi-device contract. Inspect the Function environment report for observed devices and libraries. Configured memory is per device; it does not establish that one model can allocate the sum of every device’s memory.

Refresh admission and concurrency

Submission creates a job, then source preparation, dispatcher admission and worker startup happen separately. Poor table layout can require compaction before a refresh; WHERE does not promise to limit that prerequisite to only the selected rows. An earlier column refresh can also block a later one. Dispatcher concurrency, cluster worker capacity and automatic-refresh admission are deployment-level limits. They are separate from a refresh’s aggregate cpu_millis, memory_bytes, gpu_count budget and a Function’s call-level max_concurrency. Ask your deployment operator about effective limits; defaults do not guarantee available capacity or a universal supervisor cap. GPU quotas, placement constraints and model startup can add waits after admission.

Distinguish waiting from execution

Use the Console’s job detail and SHOW JOBS. On matching SDK/server versions, job.progress() exposes optional phase, queue_reason and row/worker counters; see the detailed refresh progress reference. Older deployments can report only lifecycle running while work has not begun. Missing counters and timestamps are unknown. creation_ms is registry creation time, while supervisor_started_at_millis is execution start. In a finished job’s summary, their difference includes admission/queue time, not only Python startup. DESCRIBE JOB is a completed-run report, not a live queue ETA. Sampled worker readiness does not mean a model has initialized. Request cancellation with job.cancel() or CANCEL JOB, then observe the terminal state. Cancellation also applies while waiting; it does not roll back outputs already committed. A timeout in the client does not cancel the server job. Avoid cancelling and resubmitting merely to get a new ID; the replacement encounters the same admission constraints.

Related guidance