Storage Integration

Applies to: Embedded ❌, Enterprise ✅ A storage integration is a namespace-scoped catalog object, like a Secret, that names a credential for reading or importing from one external storage location: an S3, GCS or Azure prefix, or a HuggingFace dataset repository. A query or import statement that reads a matching URI uses the integration automatically — it is never named in the query itself.

Create and manage

Every provider takes a storage_prefix and storage_provider, plus provider-specific credential options. Credential options take a Secret reference, never a value directly — either a bare name (resolved in the caller’s own database, root namespace) or a SecretReference qualified with a database and namespace path:

CREATE STORAGE INTEGRATION hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = bootstrap.credentials.hf_token

DESCRIBE STORAGE INTEGRATION hub

SHOW STORAGE INTEGRATIONS

DROP STORAGE INTEGRATION hub

See Storage Integration for every provider’s exact options, S3’s two credential modes, and the full output schema — this page covers the CRUD operations above at a glance; that one is the complete reference.

Operation	Python (not yet released)	SQL	Notes
Create	`create_storage_integration`	`CREATE STORAGE INTEGRATION`	Fails if the name is taken, so a create never silently replaces one.
Describe	`describe_storage_integration`	`DESCRIBE` / `DESC`	One row: provider, prefix, credential references, and (for an AWS role integration) the generated trust policy.
List	`list_storage_integrations`	`SHOW` / `LIST STORAGE INTEGRATIONS`	Optionally scoped to one namespace.
Drop	`drop_storage_integration`	`DROP [IF EXISTS]`	Blocked while an attached database still depends on it.

The Python column names the methods landing in a future release (see the draft note above); use SQL today. There is no TypeScript or REST equivalent yet either. There is no ALTER. To rotate a credential, rotate the Secret it references — the integration’s reference does not change, so nothing that reads through it needs updating.

The S3 trust workflow

An S3 integration created with STORAGE_AWS_ROLE_ARN (instead of a static access key pair) needs one more step outside LanceDB: DESCRIBE returns trust_policy and permissions_policy, the IAM policy documents your role needs, built from aws_account_id, aws_iam_principal_arn and aws_external_id in the same row. Attach trust_policy as the role’s trust relationship and permissions_policy as its permissions, then check access_verified on a later DESCRIBE to confirm LanceDB can assume the role and read the prefix.

Privileges

A storage integration is its own object in the authorization model, a namespace child like a Secret: CREATE_STORAGE_INTEGRATION on the namespace to create one, USAGE on the integration for a query to read through it, OWNERSHIP for full control including dropping it. See Privileges for the privilege each operation requires and the full GRANT/REVOKE syntax.

Troubleshooting

A query does not use the integration you expected: integrations match by the most specific STORAGE_PREFIX covering the URI, not by name, and are scoped to native databases under the catalog root. Two integrations with equally specific prefixes make the query fail rather than guess; keep prefixes for the same bucket or dataset family unambiguous. 403 or access-denied reading a matching prefix: confirm USAGE on the integration, and for an S3 role integration, confirm access_verified is true on DESCRIBE — false usually means the trust policy from above is missing or misconfigured on the AWS side, not a LanceDB permission problem. DROP refuses with a dependent database: detach the database that still uses the integration first, then drop it.