Feature engineering

Use feature engineering to turn raw data into features for search, analysis and model training. LanceDB stores derived columns and materialized views alongside the source data, so you can reuse the results across those workflows. Functions and materialized views are ordinary catalog objects. Create them through the SDK or SQL, then refresh the derived data when its inputs change.

In LanceDB Enterprise, every operation here also has a SQL form. See DDL to declare a column and REFRESH COLUMN to fill it.

Computed columns and materialized views

Computed columns and materialized views both keep derived data in the lakehouse, but they fill it differently, and the difference decides what a refresh costs:

	Computed column	Materialized view
Declaring or creating it	records the expression; no values are written	runs the query and stores its rows, unless created empty
A refresh	fills rows with no value and recomputes rows whose inputs changed	rebuilds the view completely from its source query
A refresh when nothing changed	fills nothing	still rebuilds the view and commits a new version
In LanceDB Enterprise	runs as a job	runs as a server-side materialization job

Computed columns fill incrementally

A computed column separates declaring from filling:

Declaring is cheap and immediate. It records the expression in the schema and commits a new table version, with no data written. It does not evaluate the expression over existing rows.
Fill the column with a separate refresh call that you can retry. In LanceDB Enterprise, submit the refresh as a job and track its progress.

Refresh new and changed rows. A refresh gives values to rows that lack them and recomputes rows whose inputs changed since they were computed. It leaves the rest alone, so calling it twice is not twice the work, and appending rows means the next refresh handles only the new ones. Computed values update when you refresh. If a column derives from price and a row’s price is later updated, the derived value keeps its old value until the next refresh recomputes it. Schedule refreshes after changing the source data when consumers need up-to-date features.

Materialized views store query results

A materialized view has no separate declaration step: creating one runs its query and stores the result, so the view is ready to read. A refresh rebuilds the stored result completely from the source query; see Refresh a materialized view for the full behavior, including automatic refresh. Enterprise runs materialization on remote compute, as a server-side job.

Choose a feature engineering workflow

	What it produces	Filled by	Available
View	a saved query, no stored rows of its own	nothing — it runs on every read	Enterprise
Computed column	a new column on an existing table	a SQL expression	Enterprise
Materialized view	a new table derived by a query	a refresh	Enterprise
Function	a computed column filled by Python	registered code	Enterprise
Geneva	columns filled by Python, for existing pipelines	the `geneva` package	Enterprise

A view has no refresh and no staleness window, because it stores the query rather than its result. Reach for one first, before a computed column or a materialized view, when the query is cheap enough to simply re-run on every read. Computed columns are the common case: a column declared with a SQL expression, filled by a refresh. Materialized views derive a whole table (a filtered, projected, or aggregated view of a source) and bring it up to date on each refresh. The result is an ordinary table: queryable, indexable, searchable. Functions run Python where the data is, for work SQL cannot express: embedding a column, calling a model, decoding an image. Registered once, versioned, then applied as a column. They are the way to do that on a deployment; Geneva remains available for existing pipelines. A materialized view can select computed columns. A computed column can be filled by a registered function.

Where the work runs

Enterprise Locally, a refresh runs in your process. With LanceDB Enterprise it runs across the cluster, and the API reflects that: refresh_column raises, and refresh_column_async returns a job handle instead. Computed columns are off by default in LanceDB Enterprise, and its administrator has to enable them before a column can be declared there.

job = table.refresh_column_async("power_score")
result = job.wait()
result.rows_assigned

A backfill over a large table can therefore be started, left, and checked on from a different process: the connection does not have to stay open. Enterprise manages the compute for the refresh. Declaring a computed column doesn’t refresh it on a schedule unless your deployment enables automatic refresh; see Auto Refresh. That is also why a function must declare the environment it needs.

Recovering from a failure

A function that raises fails the job. There is no per-row retry or skip policy in the function definition, so a function calling a flaky external service should handle that itself, inside the function body. When a refresh fails, fix the cause and run the refresh again. The new refresh fills what is still empty or stale. How much of the failed run was kept depends on where it ran (see Refresh a computed column), and a retry can recompute values that were already written, so do not count on it costing only the rows that did not complete.

Related guides

Create a view

A saved query, with no storage or refresh of its own.

Create a computed column

Declare a column from a SQL expression.

Refresh a computed column

Fill the rows that lack values, locally or as a job.

Create a materialized view

Store a query’s result as a table.

Refresh a materialized view

Bring a derived table up to date with its source.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation