Functions

Applies to: Embedded ❌, Enterprise ✅ Early access Some columns cannot be written as a SQL expression: an embedding, a model call, an image decode. A function is a Python callable registered with the database, versioned, and then applied as a column. The code runs where the data is rather than pulling rows to your process. Registering and running a function needs a LanceDB Enterprise deployment with Functions enabled, which they are not by default. On a LanceDB Embedded connection create_function raises NotImplementedError; to derive a column there, declare a SQL computed column instead.

In SQL, a function is registered with CREATE FUNCTION and listed with SHOW FUNCTIONS.

Functions are the way to fill columns with Python in LanceDB Enterprise. If you already use Geneva, it remains available; see the Geneva docs for how its concepts map onto these. The steps below take a column of numbers to a filled column of their doubles.

Define a function

@function packages a scalar Python callable. Input and output types are inferred from annotations, so double takes one non-nullable float64 parameter, value, and returns a non-nullable float64:

Python

from lancedb import function


@function
def double(value: float) -> float:
    return value * 2

The decorated object stays callable. Calling it runs the original function in your process, without the database, so unit tests stay ordinary tests:

Python

print(double(1.5))
# 3.0

Where annotations cannot express the Arrow type precisely, pass input_schema and output_schema together: one without the other is rejected. A scalar output must be non-nullable, because a physical NULL is what marks a computed-column row as unfilled. A named-struct output may declare nullable fields: whether a row has a result at all is the struct’s own validity, which Enterprise keeps in a separate internal column when it expands the result into sibling columns, so a null field is a value rather than a missing row.

Connect and create a table

The remaining steps run against your Enterprise deployment. Replace the database URI, API key, region and endpoint below with your deployment details. In application code, load credentials from environment variables or a secrets manager:

Python

import lancedb

db = lancedb.connect(
    uri="db://your-database",
    api_key="your-api-key",
    region="us-east-1",
    host_override="https://your-endpoint",
    sql_host_override="grpc+tls://your-sql-endpoint:10026",
)

For an ordinary stored input column, the Arrow type must match the parameter. A nullable column requires a nullable parameter, such as Optional[float]; a non-nullable column can bind to either. This example uses value: float, so the table declares value non-nullable. Computed-column inputs depend on the Enterprise server version. Where supported, they bind to non-nullable parameters even though the physical column is nullable: refresh skips rows whose computed input is still null. Check your deployment’s support before using another computed column as an input.

Python

import pyarrow as pa

schema = pa.schema([pa.field("value", pa.float64(), nullable=False)])
table = db.create_table(
    "function_demo",
    [{"value": 1.0}, {"value": 2.0}, {"value": 3.0}],
    schema=schema,
)

Register it

Registration returns a query-backed job; the immutable version exists once the SQL command finishes and the job succeeds. Its job_type is query, and its single id is the SQL query UUID. Repeated waits reuse the completed result.

Python

job = db.create_function_async(double)
version = job.wait()

create_function is the blocking equivalent. A registered version is fetched with db.get_function(name, version=...).

The SQL command submits an internal create_function image-build job and waits for publication. The SDK returns the SQL query handle. Query handles have no registry history and cannot be reopened with open_job().

List and describe

list_functions pages through every published version in a namespace, ordered by name then version; get_function opens one exact version by name and version. Both take namespace_path, defaulting to the root namespace, the same as create_function.

for version in db.list_functions():
    print(version.name, version.version, version.created_at_millis)

version = db.get_function("double", version="1")

use futures::TryStreamExt;

# async fn list_and_get(connection: &lancedb::Connection) -> Result<(), Box<dyn std::error::Error>> {
let mut versions = connection.list_functions(&[], Default::default());
while let Some(version) = versions.try_next().await? {
    println!("{} {} {}", version.name(), version.version(), version.created_at_millis());
}

let version = connection.get_function("double", "1", &[]).await?;
# Ok(())
# }

SHOW FUNCTIONS
SHOW FUNCTIONS IN "analytics" WITH (name = 'double', limit = 100)

DESCRIBE FUNCTION double WITH (version = '1')

list_functions returns a lazy iterator: page_limit bounds each request, not the total, and page_token resumes from a saved token. Child namespaces are not included; list each namespace separately. A version’s created_at_millis, metadata, disabled flag, and secret_bindings are all readable this way without applying or calling the function.

Try it in SQL

A registered function is callable in SQL by its name, the same as any other scalar function. Before applying it as a column, call it directly over the table to check the registration succeeded, without committing a schema change:

reader = db.execute_query("SELECT value, double(value) AS doubled FROM function_demo")
print(reader.read_all().to_pylist())
# [{'value': 1.0, 'doubled': 2.0}, {'value': 2.0, 'doubled': 4.0}, {'value': 3.0, 'doubled': 6.0}]

Equivalent SQL: SELECT value, double(value) AS doubled FROM function_demo. See Querying with SQL for execute_query and other languages.

Apply it as a column

Calling the registered version binds each parameter, by name, to a column with col(...). add_columns declares the result as a new column:

Python

from lancedb import col

table.add_columns({"doubled": version(value=col("value"))})

The column is declared empty; the function has not run yet. A mapping with one function application keeps its scalar or named-struct result in the named column. A bare named-struct application expands its result fields as one atomic binding, with aliases from rename(columns=...).

Bind more than one input, or fill more than one column

Parameters bind by name, so a function that takes several arguments binds several columns in one call. ratio divides two existing columns, so give the table a second one first:

table.add_columns({"count": "value + 1"})

@function
def ratio(numerator: float, denominator: float) -> float:
    return numerator / denominator

version = db.create_function_async(ratio).wait()
table.add_columns({"ratio": version(numerator=col("value"), denominator=col("count"))})

A named-struct output fills more than one column from a single application. Declare the fields with output_schema, then alias each field to the column it fills with rename(columns=...) and pass the application itself, not a mapping, so both columns come from one atomic binding:

import pyarrow as pa

@function(
    input_schema=pa.schema([pa.field("value", pa.float64(), nullable=False)]),
    output_schema=pa.schema([
        pa.field("half", pa.float64(), nullable=False),
        pa.field("double", pa.float64(), nullable=False),
    ]),
)
def scale(value: float):
    return {"half": value / 2, "double": value * 2}

version = db.create_function_async(scale).wait()
application = version(value=col("value")).rename(
    columns={"half": "value_half", "double": "value_double"}
)
table.add_columns(application)

input_schema and output_schema must be given together on v0.41.0. A version after it accepts either alone, inferring the other from the annotations.

A single refresh fills value_half and value_double together: there is no state where one is filled and the other is still empty.

Fill the column

A refresh runs the function on the cluster as a job. wait() blocks until the job finishes and returns its result, including how many rows it filled:

Python

refresh = table.refresh_column_async("doubled")
result = refresh.wait()
print(result.rows_assigned)
# 3

The filled values are read like any other column:

Python

rows = table.search().select(["value", "doubled"]).to_list()
print(sorted((row["value"], row["doubled"]) for row in rows))
# [(1.0, 2.0), (2.0, 4.0), (3.0, 6.0)]

Filling then works as it does for a SQL computed column: a later refresh fills rows added since the last one and recomputes rows whose inputs changed, and leaves filled rows with unchanged inputs alone. See Refresh a column.

Keep state in a class

A function that loads a model or opens a client should not do it for every row. Decorate a class instead: its __init__ does the expensive setup once, and __call__ handles each row or batch.

Python

@function(pip=["sentence-transformers"])
class embed:
    def __init__(self, model: str = "all-MiniLM-L6-v2"):
        # Runs on the worker, once per instance: only the worker needs the package.
        from sentence_transformers import SentenceTransformer

        self.model = SentenceTransformer(model)

    def __call__(self, text: str) -> list[float]:
        return self.model.encode(text).tolist()

    def close(self):
        del self.model

Registering the class does not construct it. @function reads the signature of __init__ and packages the source, so the client never loads the model. On the cluster, each instance of the function calls __init__ once. It then calls __call__ for every row or batch it processes, and close(), if defined, once when it retires. Import heavy packages inside __init__, as above, so only the worker needs them installed, and declare them in pip. State belongs to one instance. An instance reuses it across every batch it serves, but instances do not share it, so each one holds its own copy of the model. Plan memory, or a GPU, for one copy per instance. Register it the same way as a plain function:

Python

embedder = db.create_function_async(embed).wait()

The annotated parameters of __init__ are the function’s initialization fields. Their values come from each binding rather than from the function, so one registered version can run with different settings on different columns:

Python

from lancedb import col

application = embedder(text=col("text"), model="all-mpnet-base-v2")

Pass the application to add_columns as above. Initialization values are fixed for every row of a binding and take booleans, integers, floats, strings, and lists or structs of them. A parameter with a default may be omitted. Initialization values are stored with the binding, so keep credentials out of them: pass secrets=[EnvVarSecret(...)] to create_function and read the variable in __init__. Calling the decorated class in your own process constructs it, the same as an undecorated class, so local tests can create an instance and call it directly.

Blob v2 columns

A Function can bind a Blob v2 column as input and return another. In row mode, a top-level blob parameter arrives as Python bytes, or None for a nullable input, after the worker has read the payload; annotate it BlobReader instead for seekable, incremental reads that avoid materializing the whole payload first. Return bytes and let the destination table store the Managed payload and descriptor — output_schema=lancedb.blob(...) is what makes the result Blob v2 rather than plain Arrow binary; a bare annotation persists as ordinary binary instead, skipping the Managed storage path and fetch_blobs. See Read and produce blobs for a full example: extracting a frame from a stored video, chaining a second Function onto its output, persisting both as columns or a materialized view, and reading a blob input incrementally.

Declare the environment

The function runs on the Enterprise service, not in your application, so its dependencies are part of its definition:


`pip`	requirements for the function’s execution environment
`conda`, `conda_channels`	conda packages instead of pip; channels in priority order
`env`	environment variables
`python_version`	Python major/minor for the execution environment, defaulting to yours
`gpu`	whether every execution requires a GPU

@function(gpu=True)
def extract_frame(video: BlobReader, ts: float) -> bytes:
    ...  # runs only on a GPU-capable worker

These are part of the immutable function version. Changing a dependency produces a new version rather than altering the one already registered, so a column filled by version 1 is unaffected by version 2. Declaring them is not optional bookkeeping. If the worker environment differs from the one the function was written against, the failures are among the hardest to diagnose in the system, because the error rarely names the cause.

Symptoms of an environment mismatch

Symptom	Likely cause
`TypeError: Enum.__new__() missing 1 required positional argument`	`attrs` version mismatch
`TypeError: Can't instantiate abstract class`	package structure differs between versions
`ArrowInvalid: cannot cast`, or serialization errors	NumPy 1.x against 2.x
`ModuleNotFoundError` on the worker	package installed locally but not declared
Model loading failures	PyTorch version mismatch
Permission denied	an API key present locally but absent from `env`

The packages worth pinning explicitly, because a mismatch in them produces the errors above rather than a clear one: numpy, torch, pyarrow, attrs, pydantic.

The client version defaults python_version, so a function registered from a 3.11 environment runs on 3.11 unless you say otherwise. That default is usually right and occasionally surprising. Pin it when the function is sensitive to it.

Remote calls and credentials

A function that calls an external API needs a credential without putting it in the function’s source or its env map in plain text. A Secret is a database-scoped named credential; db.create_secret stores the value once, and an EnvVarSecret binds it to the environment variable the function reads. Binding is a registration-time decision, passed to create_function_async, not to @function. No API returns a Secret’s value, and it never appears in the function’s registration: the function reads the environment variable the way it always did, and the binding is what puts a value there. A mistyped Secret name is caught at registration; a mistyped environment variable name surfaces only on the first rows, since nothing records which variables a function reads. See Vend a Secret to a Function for the full worked example.

Failure handling

By default, a Function that raises fails the refresh job. The Function definition has no error-handling parameter; choose the row-error policy on the refresh instead:

REFRESH COLUMN embedding ON documents WITH (error_policy = 'skip')

skip writes null outputs for failing rows and records their errors. Inspect them with SHOW ERRORS or table.function_errors(). Failure limits still apply, and initialization or infrastructure failures can still fail the job. See the SQL refresh options. A Function that calls a transiently failing external service can handle bounded retries inside its body. Letting the job fail is the other option: fix the cause and refresh again. The next unfiltered refresh skips rows that are filled and current, but how much of the failed job’s work was kept is not guaranteed, so do not count on the retry costing only the rows that did not complete. A refresh with WHERE recomputes every matching live row, including already-filled outputs.

What gets packaged

The artifact is a snapshot: the function source, plus exactly the module-level names it references (modules as imports, importable classes and functions as imports, literals inlined).Code that reaches the module namespace another way is rejected where it can be seen and unsupported where it cannot: globals(), eval, sys.modules, builtins. Closures and a non-standard __builtins__ are rejected outright.

In practice this means a function should be self-contained: take its inputs as arguments, import what it needs at module level, and avoid capturing state from the surrounding scope. Pass code=[module, ...] to ship additional helper modules into the same image, for logic the function imports but that does not belong inlined into it — a model-loading helper shared by several Functions, for example. Continuing the caption Function, with its model-loading code factored into a sibling module instead of inlined:

import caption_worker  # a sibling module with the model-loading logic

@function(gpu=True, code=[caption_worker])
def caption(jpeg: bytes) -> str:
    return caption_worker.run(jpeg)

Each module in code is captured and shipped the same way the function itself is: as source, snapshotted at registration. Only .py source files are included; package data such as JSON, YAML, prompt, or tokenizer files is not copied. @function warns before registration when it skips such files from a package in code, naming the affected paths. To use a package that includes the required resources, install it with pip= or conda= and remove that package from code; otherwise the Python-only copied package shadows the installed distribution.

Drop it

db.drop_function("double", version="1")

# async fn drop_it(connection: &lancedb::Connection) -> Result<(), Box<dyn std::error::Error>> {
let dropped = connection.drop_function("double", "1", &[]).await?;
# Ok(())
# }

DROP FUNCTION double WITH (version = '1')

db.drop_function(name, version=...) removes only that published revision from lookup and listing, preserving other revisions and their grants. It returns False for a repeated drop or a name that is already gone. A revision never published under an existing name is an error. The last visible revision’s drop releases the name and schedules object cleanup, which can finish after the call returns as a background job. drop_function_async returns (dropped, job), where job is a query-backed handle. Waiting confirms the revision’s catalog removal; physical cleanup can continue afterward. A deployment refuses the drop unless it also allows changes to a function’s lifecycle, which is a separate setting, off by default even where Functions are enabled.

Create a Function

Scalar, batch, stateful, GPU and remote-service examples for common curation tasks.

Computed columns

Refresh semantics, and the SQL alternative.

Embedding functions

Built-in embedding providers, which need no registration.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Define a function

Connect and create a table

Register it

List and describe

Try it in SQL

Apply it as a column

Bind more than one input, or fill more than one column

Fill the column

Keep state in a class

Blob v2 columns

Declare the environment

Symptoms of an environment mismatch

Remote calls and credentials

Failure handling

What gets packaged

Drop it

What to read next

Create a Function

Computed columns

Embedding functions

​Define a function

​Connect and create a table

​Register it

​List and describe

​Try it in SQL

​Apply it as a column

​Bind more than one input, or fill more than one column

​Fill the column

​Keep state in a class

​Blob v2 columns

​Declare the environment

​Symptoms of an environment mismatch

​Remote calls and credentials

​Failure handling

​What gets packaged

​Drop it

​What to read next

Create a Function

Computed columns

Embedding functions

Define a function

Connect and create a table

Register it

List and describe

Try it in SQL

Apply it as a column

Bind more than one input, or fill more than one column

Fill the column

Keep state in a class

Blob v2 columns

Declare the environment

Symptoms of an environment mismatch

Remote calls and credentials

Failure handling

What gets packaged

Drop it

What to read next