Skip to main content
Applies to: Embedded ❌, Enterprise ✅ A storage integration is a namespace-scoped catalog object that lets a query read from, or import from, a storage location outside the deployment’s own tables — an S3, GCS or Azure prefix, or a HuggingFace dataset repository — without embedding a credential in the query itself. It is addressed and scoped the same way a Secret is: a 1-3 part identifier, name, namespace.name or database.namespace.name, with a nested namespace path $-joined into that middle part.

​
CREATE STORAGE INTEGRATION

STORAGE_PROVIDER and STORAGE_PREFIX are required for every provider. STORAGE_PREFIX is the URI prefix the integration covers — a query only matches an integration whose prefix is a prefix of the URI it reads. Options ending in _SECRET take a Secret identifier, not a quoted string: name, namespace.name or database.namespace.name, with a nested namespace path $-joined in the middle part (db.`team$prod`.key). Omitted parts default to the session’s current database and namespace. Creating the integration requires USAGE on each referenced Secret and on its database and namespace. Passing an option a provider does not accept, such as an Azure option on an S3 integration, is refused.

​
S3

Give either STORAGE_AWS_ROLE_ARN alone, or both key-pair options together — mixing the two, or supplying only one half of the key pair, is refused.

​
Azure

All four are required together; there is no role-based alternative for Azure.

​
GCP

GCP_CREDENTIALS_JSON_SECRET — note the option name has no STORAGE_ prefix, unlike every other provider’s options — names a Secret holding the full service-account JSON key. It is required; GCP has no role-based alternative here.

​
HuggingFace

HUGGINGFACE_TOKEN_SECRET is the only HuggingFace option, and it is required: creating an integration without it is refused. For anonymous access, at HuggingFace’s default anonymous rate limits, register no integration for that prefix at all. See Inspect Datasets for the full read and import workflow this integration unlocks.

​
The S3 trust policy

Creating an STORAGE_AWS_ROLE_ARN integration does not finish the AWS side of the setup. The result row’s trust_policy, permissions_policy and configuration_instructions columns — also visible from DESCRIBE — are the IAM policy documents your role needs: attach trust_policy as the role’s trust relationship, scoped to aws_account_id, aws_iam_principal_arn and aws_external_id from the same row, and attach permissions_policy to grant that role read access to the bucket. access_verified reports whether the deployment already confirmed it can assume the role and read the prefix; false typically means the trust policy has not been attached yet.

​
Reading and importing through the integration

A query or CREATE TABLE ... AS SELECT can reference a matching URI directly, once an integration covers it — the integration is never named in the statement. What the URI reads depends on the provider:
  • HuggingFace (hf://datasets/...) reads Parquet files from a dataset repository. See Inspect Datasets for the full workflow, including how a CTAS import distributes across plan executors.
  • S3, GCS and Azure (s3://, gs://, az://) read an existing Lance dataset at that prefix, the same way as a managed table:
A CTAS import distributes the same way regardless of provider: disjoint file and row-group ranges go to plan-executor workers, each writes uncommitted Lance files, and a coordinator commits the table once every worker finishes. See CREATE TABLE for CREATE TABLE AS SELECT syntax in general.

​
DESCRIBE STORAGE INTEGRATION

Returns one row with every column CREATE returns: the provider, prefix, the Secret references and plain-string options that apply to its provider (the rest are null), created_at_millis, access_verified, and — for an STORAGE_AWS_ROLE_ARN integration — the generated AWS policy documents from above. No call ever returns a Secret’s value, only the reference naming it.

​
SHOW STORAGE INTEGRATIONS

LIST is an alias for SHOW here. Returns one name column; IN narrows the listing to one namespace.

​
DROP STORAGE INTEGRATION

Returns dropped, true or false. There is no ALTER STORAGE INTEGRATION: to rotate a credential, rotate the Secret it references (see Secret) rather than dropping and recreating the integration. A storage integration an attached database still depends on blocks the drop; detach the database first.

​
GRANT and REVOKE

CREATE_STORAGE_INTEGRATION on the namespace creates one; USAGE on the integration itself lets a query match and read through it; OWNERSHIP grants full control, including dropping it. See Privileges for the full statement syntax, every grantee form, and how a query picks a matching integration without naming one.

​
What to read next

Inspect Datasets

Query and import HuggingFace or object-storage datasets, the most common use of a storage integration today.

Secret

The credential storage integration options reference.

Privileges

Every privilege, and the grant and revoke syntax that covers each one.