Compress Function output columns

Use update_field_metadata to choose compression for a top-level column populated by a registered Function. Storage settings belong to the table column; keep compression metadata out of the Function’s output_schema. To apply compression to the first backfill, disable automatic refresh while adding the column, set its compression, then start a refresh. For an existing table with a blob_id column and a registered Function fetch_content that returns text:

from lancedb import col

table.add_columns(
    {"content": fetch_content(blob_id=col("blob_id"))},
    auto_refresh_enabled=False,
)
table.update_field_metadata(
    {
        "path": "content",
        "metadata": {
            "lance-encoding:compression": "zstd",
            "lance-encoding:compression-level": "22",
        },
    }
)
job = table.refresh_column_async("content")
job.wait()

Higher compression levels trade write speed for smaller storage. The settings apply to subsequent writes; changing metadata does not rewrite existing files. Set a key’s value to None to remove that setting. Use the default merge behavior of update_field_metadata to preserve the server-managed Function declaration. Compression updates are supported on Function outputs that no Function binding reads. If another Function will consume content, configure compression before binding that downstream Function: input contracts capture field metadata, so metadata changes on bound inputs remain rejected. Only lance-encoding:compression and lance-encoding:compression-level receive this exception; other metadata updates and metadata replacement remain rejected on Function-bound columns.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Compress Function output columns