Enterprise only Early access
Some columns cannot be written as a SQL expression: an embedding, a model call,
an image decode. A function is a Python callable registered with the database,
versioned, and then applied as a column. The code runs where the data is
rather than pulling rows to your process.
Registering and running a function needs a LanceDB Enterprise deployment with
Functions enabled, which they are not by default, and the
Enterprise client version. On a
LanceDB Embedded connection create_function raises NotImplementedError.
With published Embedded packages, compute derived values in your application
and write them to the table. SQL computed-column declaration and refresh
require an experimental source build.
In SQL, a function is registered with
CREATE FUNCTION and listed with SHOW FUNCTIONS.Define a function
@function packages a scalar Python callable. Input and output types are inferred
from annotations, so double takes one non-nullable float64 parameter,
value, and returns a non-nullable float64:
Python
Python
input_schema
and output_schema together: one without the other is rejected. A scalar
output is non-nullable by default. Annotate the return as Optional[T] or
T | None, or pass a nullable output field, to declare a nullable scalar:
Nullable scalar outputs require an Enterprise server that supports them;
older servers reject the registration. On a supporting server, a computed
NULL counts as a filled result, independently of its stored value.Timestamps and durations
Baredatetime keeps its existing timestamp[us] default without a timezone.
For any other unit or a timezone, declare the Arrow type explicitly:
pa.timestamp accepts s, ms, us, and ns, and a timezone such as
UTC, America/New_York, or +08:00. The canonical signature uses
{"type":"timestamp:<unit>:<timezone>"}, with - for no timezone; the
microsecond/no-timezone case keeps the legacy {"type":"timestamp"} shape.
A timezone string of - is reserved for “no timezone” and is rejected.
pa.duration accepts the same four units and uses
{"type":"duration:<unit>"}; the microsecond case keeps the legacy
{"type":"duration"} shape. A bare timedelta has no unit and is rejected;
use Annotated[timedelta, pa.duration("s")] or an explicit schema.
Other units are never downgraded to microseconds. A reader that predates the
compound temporal representation cannot read those signatures. A deployment
must include the corresponding Lance Namespace and Sophon support; this
client-side declaration does not add server support.
Maps and decimals
Adict[str, T] annotation declares an Arrow map with a non-nullable utf8
key. A plain T value is non-nullable; annotate dict[str, Optional[T]] for
nullable values. An explicit pa.map_(...) schema preserves the value field’s
own nullable flag:
utf8; keys_sorted=True is not supported, and
Blob v2 values inside a map are not supported.
Decimal carries no precision or scale by itself, so a bare Decimal
annotation is rejected. Declare both explicitly with Annotated, or use an
explicit Arrow schema:
decimal128 and decimal256 keep the declared precision and scale in the
canonical signature; no default is chosen. Map and decimal signatures are sent
as canonical Arrow JSON. A deployment must support those types; an older
Enterprise server may reject the registration, and this client-side
declaration does not add server support.
Connect and create a table
The remaining steps run against your Enterprise deployment. Replace the database URI, API key, region and endpoint below with your deployment details. In application code, load credentials from environment variables or a secrets manager:Python
Optional[float];
a non-nullable column can bind to either. This example uses value: float,
so the table declares value non-nullable.
Computed-column inputs depend on the Enterprise server version. Where supported,
they bind to non-nullable parameters even though the physical column is
nullable: refresh skips rows whose computed input is still null. Check your
deployment’s support before using another computed column as an input.
Python
Register it
Registration returns a query-backed job; the immutable version exists once the SQL command finishes and the job succeeds. Itsjob_type is query, and its
single id is the SQL query UUID. Repeated waits reuse the completed result.
Python
create_function is the blocking equivalent. A registered version is fetched
with db.get_function(name, version=...). The version may be an integer or
its string form. Omit version to resolve and return the latest currently
visible revision.
Re-registering unchanged code reuses the published version and its resolved
environment lock. Pass refresh_lock=True to create_function or
create_function_async to request a fresh lock for this registration. It is a
registration action, not part of the Function’s source or runtime identity.
A deployment must support the field; an older server rejects the unknown
field rather than silently reusing the old lock.
The SQL command submits an internal
create_function image-build job and
waits for publication. The SDK returns the SQL query handle. Query handles
have no registry history and cannot be reopened with open_job().List and describe
list_functions pages through every published version in a namespace,
ordered by name then version. get_function accepts an integer or string
version, or resolves the latest visible revision when version is omitted.
Both take namespace_path, defaulting to the root namespace, the same as
create_function.
list_functions returns a lazy iterator: page_limit bounds each request,
not the total, and page_token resumes from a saved token. Child namespaces
are not included; list each namespace separately. A version’s
created_at_millis, metadata, disabled flag, and secret_bindings are
all readable this way without applying or calling the function.
Inspect the resolved environment
The version keeps the requested runtime separate from what the server actually installed. The typed catalog API is available in Python and Rust; TypeScript does not yet expose this readback API.version.runtime is the specification
sent at registration. version.resolved_environment is measured data from the
built image:
version.environment.requested and version.environment.resolved are
convenience members over the same canonical fields; they do not add another
wire shape. The canonical JSON exposes top-level runtime and
resolved_environment. A missing resolved_environment means the version or
environment cache predates measurement; it is unknown, not an empty installed
package list. A package’s source is present only when the submitted lock
retained it, and provenance fields are reported without client-side guessing.
For a multiarchitecture image, the scalar field is absent and
version.resolved_environments (also available as
version.environment_report) maps OCI os/architecture keys to the measured
environment for that platform. A missing platform key means its measurement is
unknown; an empty packages list inside a key is a measured empty environment
and is retained. The SDK never merges platforms or fabricates an aggregate
package list.
The corresponding canonical payload has this shape:
create_function_async(...) (or calling create_function) emits each server
warning from the scalar or per-platform measurements as a Python UserWarning;
get_function and list_functions read the same version but do not
replay those registration warnings.
Try it in SQL
A registered function is callable in SQL by its name, the same as any other scalar function. Before applying it as a column, call it directly over the table to check the registration succeeded, without committing a schema change:Equivalent SQL:
SELECT value, double(value) AS doubled FROM function_demo.
See Querying with SQL for execute_query and other languages.Apply it as a column
Calling the registered version binds each parameter, by name, to a column withcol(...). add_columns declares the result as a new column:
Python
rename(columns=...).
Bind more than one input, or fill more than one column
Parameters bind by name, so a function that takes several arguments binds several columns in one call.ratio divides two existing columns, so give
the table a second one first:
output_schema, then alias each field to the column
it fills with rename(columns=...) and pass the application itself, not a
mapping, so both columns come from one atomic binding:
input_schema and output_schema must be given together on v0.41.0. A
version after it accepts either alone, inferring the other from the
annotations.value_half and value_double together: there is no
state where one is filled and the other is still empty.
Fill the column
A refresh runs the function on the cluster as a job.wait()
blocks until the job finishes and returns its result, including how many rows
it filled:
Python
Python
Keep state in a class
A function that loads a model or opens a client should not do it for every row. Decorate a class instead: its__init__ does the expensive setup once,
and __call__ handles each row or batch.
Python
@function reads the signature of
__init__ and packages the source, so the client never loads the model. On
the cluster, each instance of the function calls __init__ once. It then calls
__call__ for every row or batch it processes, and close(), if defined, once
when it retires. Import heavy packages inside __init__, as above, so only the
worker needs them installed, and declare them in pip.
State belongs to one instance. An instance reuses it across every batch it
serves, but instances do not share it, so each one holds its own copy of the
model. Plan memory, or a GPU, for one copy per instance.
Register it the same way as a plain function:
Python
__init__ are the function’s initialization
fields. Their values come from each binding rather than from the function, so
one registered version can run with different settings on different columns:
Python
add_columns as above. Initialization values
are fixed for every row of a binding and take booleans, integers, floats,
strings, and lists or structs of them. A parameter with a default may be
omitted. Initialization values are stored with the binding, so keep
credentials out of them: pass secrets=[EnvVarSecret(...)] to
create_function and read the variable in __init__.
Calling the decorated class in your own process constructs it, the same as an
undecorated class, so local tests can create an instance and call it directly.
Blob v2 columns
A Function can bind a Blob v2 column as input and return another. In row mode, a top-level blob parameter arrives as Pythonbytes, or None for a nullable input, after the worker has read
the payload; annotate it BlobReader instead for seekable, incremental
reads that avoid materializing the whole payload first. Return bytes and let
the destination table store the Managed payload and descriptor —
output_schema=lancedb.blob(...) is what makes the result Blob v2 rather
than plain Arrow binary; a bare annotation persists as ordinary binary
instead, skipping the Managed storage path and fetch_blobs.
See Read and produce blobs for a full
example: extracting a frame from a stored video, chaining a second Function
onto its output, persisting both as columns or a materialized view, and
reading a blob input incrementally.
Declare the environment
The function runs on the Enterprise service, not in your application, so its dependencies are part of its definition:Symptoms of an environment mismatch
The packages worth pinning explicitly, because a mismatch in them produces the
errors above rather than a clear one: numpy, torch, pyarrow, attrs,
pydantic.
The client version defaults
python_version, so a function registered from a
3.11 environment runs on 3.11 unless you say otherwise. That default is usually
right and occasionally surprising. Pin it when the function is sensitive to it.Remote calls and credentials
A function that calls an external API needs a credential without putting it in the function’s source or itsenv map in plain text. A Secret is a
database-scoped named credential; db.create_secret stores the value once,
and an EnvVarSecret binds it to the environment variable the function
reads. Binding is a registration-time decision, passed to
create_function_async, not to @function. No API returns a Secret’s value,
and it never appears in the function’s registration: the function reads the
environment variable the way it always did, and the binding is what puts a
value there. A mistyped Secret name is caught at registration; a mistyped
environment variable name surfaces only on the first rows, since nothing
records which variables a function reads.
See Vend a Secret to a Function for the
full worked example.
Failure handling
For the full skip, inspect, grouped-error, stable-row-ID retry and quarantine workflow, see Handle Function failures. By default, a Function that raises fails the refresh job. Choose how the refresh handles row errors with its error policy:skip writes null outputs for failing rows and records their errors. Inspect
them with SHOW ERRORS or table.function_errors(). Failure limits still
apply, and initialization or infrastructure failures can still fail the job.
See the SQL refresh options. A Function
that calls a transiently failing external service can handle bounded retries
inside its body.
Letting the job fail is the other option: fix the cause and refresh again. The
next unfiltered refresh skips rows that are filled and current,
but how much of the failed job’s work was kept is not guaranteed, so do not
count on the retry costing only the rows that did not complete. A refresh with
WHERE recomputes every matching live row, including already-filled outputs.
What gets packaged
In practice this means a function should be self-contained: take its inputs as arguments, import what it needs at module level, and avoid capturing state from the surrounding scope.Supported global captures
The Python SDK’s source packager captures only these literal values:
For example, a module-level
OUT_TYPE = pa.list_(pa.float32(), 3) referenced
from the Function body is unsupported. Construct the type where it is used:
__init__ and keep it on self.
Later mutations of a client global do not update a registered version. Register
a new version to change captured literals or source. The diagnostic fix in
SDK #4938 names the unsupported
global and its type; it does not add Arrow/container capture support. There is
no supported arbitrary-object serialization or documented container-size limit
to rely on for those rejected types.
See the Function type matrix for
what can travel as row inputs and outputs rather than captured globals.
Pass code=[module, ...] to ship additional helper modules into the same
image, for logic the function imports but that does not belong inlined into
it — a model-loading helper shared by several Functions, for example.
Continuing the caption Function, with
its model-loading code factored into a sibling module instead of inlined:
code is captured and shipped the same way the function
itself is: as source, snapshotted at registration. Packages also include regular
data files such as JSON, YAML, prompts, and tokenizer files, including resources
inside vendored subpackages. Resource paths and bytes are preserved for
importlib.resources and package integrity checks. Changing a resource changes
the Function artifact’s identity. Caches, hidden paths, symlinks, and unsafe
paths are excluded.
Package resources require matching SDK and server support. Upgrade query nodes
and Function image-building workers before registering resource bundles; older
servers reject them. Older SDKs ship only Python source and warn when excluding
package data. On deployments without resource-bundle support, install the
package with pip= or conda= and omit it from code so the copied package does
not shadow the installed distribution. Use environment dependencies for
platform-specific packages and models= for large model snapshots.
Drop it
db.drop_function(name, version=...) removes only that published revision from
lookup and listing, preserving other revisions and their grants. It returns
False for a repeated drop or a name that is already gone. A revision never
published under an existing name is an error. The last visible revision’s drop
releases the name and schedules object cleanup, which can finish after the call
returns as a background job.
drop_function_async returns (dropped, job), where job is a query-backed
handle. Waiting confirms the revision’s catalog removal; physical cleanup can
continue afterward.
When a deployment reports dependency warnings for a boolean drop,
db.drop_function emits each warning as a UserWarning. This also applies to
await db.drop_function(...) on an async connection. The warnings are emitted
after catalog removal, and standard Python warnings filters apply. If warnings
are configured as errors, the call can raise after the revision has been
removed; the drop is not rolled back. Servers whose DROP response omits the
warning field remain silent.
A deployment refuses the drop unless it also allows changes to a function’s
lifecycle, which is a separate setting, off by default even where Functions are
enabled.
What to read next
Create a Function
Scalar, batch, stateful, GPU and remote-service examples for common curation tasks.
Computed columns
Refresh semantics, and the SQL alternative.
Embedding functions
Built-in embedding providers, which need no registration.