Define Storage Integration

Applies to: Embedded ❌, Enterprise ✅ Before you can import data from external storage or attach an existing database, define a storage integration once per location. It is a namespace-scoped catalog object that stores the credential — neither operation names the integration directly, LanceDB matches the URI you read against whichever integration’s prefix covers it.

What each provider needs

Provider	Credential	Notes
AWS S3	An IAM role to assume, or a static access key pair	A role needs one extra step outside LanceDB: trust policy setup.
Google Cloud Storage	A service account’s JSON key	No role-based alternative.
Azure Blob Storage	A service principal’s tenant, client ID and client secret	All three required together.
HuggingFace	An access token	Always required to create an integration. For anonymous access at HuggingFace’s default rate limits, register no integration for that prefix instead.

Every credential option takes a Secret reference, never the value directly — create the Secret first.

Create one

AWS S3

Give either an IAM role to assume, or a static access key pair — mixing the two, or supplying only one half of the key pair, is refused.

CREATE STORAGE INTEGRATION raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-data/raw/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'

CREATE STORAGE INTEGRATION raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-data/raw/'
    STORAGE_AWS_ACCESS_KEY_ID_SECRET = aws_key_id
    STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = aws_key_secret

db.create_storage_integration(
    "raw", "s3://customer-data/raw/",
    storage_provider="S3",
    aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
)

db.create_storage_integration(
    "raw", "s3://customer-data/raw/",
    storage_provider="S3",
    aws_access_key_id_secret="aws_key_id",
    aws_secret_access_key_secret="aws_key_secret",
)

use lancedb::storage_integration::CreateStorageIntegrationRequest;

let info = db.create_storage_integration(
    "raw",
    CreateStorageIntegrationRequest::s3(
        "s3://customer-data/raw/",
        "arn:aws:iam::123456789012:role/lancedb-access",
    ),
    &[],
).await?;

Google Cloud Storage

CREATE STORAGE INTEGRATION gcs
    STORAGE_PROVIDER = 'GCP'
    STORAGE_PREFIX = 'gs://customer-data/raw/'
    GCP_CREDENTIALS_JSON_SECRET = gcp_service_account

db.create_storage_integration(
    "gcs", "gs://customer-data/raw/",
    storage_provider="GCP",
    gcp_credentials_json_secret="gcp_service_account",
)

The Secret holds the full service-account JSON key. GCP has no role-based alternative.

Azure Blob Storage

CREATE STORAGE INTEGRATION blobs
    STORAGE_PROVIDER = 'AZURE'
    STORAGE_PREFIX = 'az://customer-data/raw/'
    STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
    STORAGE_AZURE_TENANT_ID_SECRET = azure_tenant
    STORAGE_AZURE_CLIENT_ID_SECRET = azure_client
    STORAGE_AZURE_CLIENT_SECRET_SECRET = azure_credential

db.create_storage_integration(
    "blobs", "az://customer-data/raw/",
    storage_provider="AZURE",
    azure_storage_account="customeraccount",
    azure_tenant_id_secret="azure_tenant",
    azure_client_id_secret="azure_client",
    azure_client_secret_secret="azure_credential",
)

All four options are required together; there is no role-based alternative for Azure.

HuggingFace

CREATE STORAGE INTEGRATION hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = hf_token

db.create_storage_integration(
    "hub", "hf://datasets/owner/private/data/",
    storage_provider="HUGGINGFACE",
    huggingface_token_secret="hf_token",
)

HUGGINGFACE_TOKEN_SECRET is required to create this integration; omitting it is rejected, not treated as anonymous. For anonymous access at HuggingFace’s default rate limits, register no integration for that prefix at all.

Finish the AWS trust policy

An IAM role integration needs one step outside LanceDB: describing it returns the trust_policy and permissions_policy documents your role needs. Attach them in AWS, then describe the integration again and check access_verified to confirm LanceDB can assume the role. A static access key pair skips this step entirely. See Storage Integration for every provider’s complete option reference, S3’s two credential modes in full, and the output schema, or Storage Integration for the Python and SQL CRUD operations at a glance.

Import Datasets

Read or import through the integration once it exists.

Attach Database

Attach an existing database at a location the integration covers.

Storage Integration

The complete SQL reference: every provider’s options and output columns.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Define Storage Integration

What each provider needs

Create one

AWS S3

Google Cloud Storage

Azure Blob Storage

HuggingFace

Finish the AWS trust policy

What to read next

Import Datasets

Attach Database

Storage Integration

​What each provider needs

​Create one

​AWS S3

​Google Cloud Storage

​Azure Blob Storage

​HuggingFace

​Finish the AWS trust policy

​What to read next

Import Datasets

Attach Database

Storage Integration

What each provider needs

Create one

AWS S3

Google Cloud Storage

Azure Blob Storage

HuggingFace

Finish the AWS trust policy

What to read next