> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Import a Hugging Face dataset

> Store a Hugging Face token and import dataset rows into LanceDB Enterprise with SQL or Python.

Enterprise only

Import 100 movie reviews from the [IMDb dataset](https://huggingface.co/datasets/stanfordnlp/imdb)
into a LanceDB Enterprise table. The service reads the source Parquet files and
writes the table in your database storage. The Python client submits SQL and
receives results; it does not download and upload the dataset.

## Before you start

* Use an Enterprise deployment that supports Hugging Face SQL sources and
  storage integrations, with Secrets configured by your administrator.
* Select an existing native database with `database_manifest_enabled=true`.
  You need permission to create Secrets, storage integrations, and tables in
  its root namespace (`public`), plus usage permission on the database and
  namespace. Select that database and namespace in your SQL client before
  running the examples.
* For Python, install the LanceDB SDK with SQL support (`pip install -U lancedb`).
  Obtain your database name, API key, REST endpoint, and Flight SQL endpoint
  from your administrator. See [Connect with SQL](https://docs.lancedb.com/enterprise/v1-13-0/basics/sql.md).

The SQL and Python tabs below are alternative ways to run the same steps.
Use the same database and namespace throughout, and choose unused names for
`hf_token`, `imdb_hub`, and `imdb_sample`.

## 1. Obtain a Hugging Face token

Open [Hugging Face access tokens](https://huggingface.co/settings/tokens) and
create a token with read access to the dataset repository. For private or gated
datasets, the token's account must have access, including any required approval
or accepted dataset terms. A token does not grant access by itself.

For the Python examples, set `HF_TOKEN` in your client environment through your
secret manager, or enter it without echoing it:

```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import getpass
import os

os.environ["HF_TOKEN"] = getpass.getpass("Hugging Face read token: ")
```

IMDb is public and can also be read anonymously. This tutorial registers a token
so you can use the same procedure for repositories that require authentication.
Setting `HF_TOKEN` on your laptop or a server does not authenticate SQL workers.
The next step stores it in LanceDB and binds it to the repository.

## 2. Store the token and register the repository

Create a single-value [Secret](https://docs.lancedb.com/enterprise/v1-13-0/management/secret.md), then register a storage
integration that references its name. Use a repository or folder prefix without
wildcards; the integration is selected automatically for matching SQL sources.

> [tab] SQL
>
> The literal below shows the SQL syntax. For a real token, prefer the bound
> parameter in the Python tab to keep the value out of SQL text and history.
>
> ```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> CREATE SECRET hf_token
>     TYPE = GENERIC_STRING
>     SECRET_STRING = '<your-hugging-face-read-token>';
>
> CREATE STORAGE INTEGRATION imdb_hub
>     STORAGE_PROVIDER = 'HUGGINGFACE'
>     STORAGE_PREFIX = 'hf://datasets/stanfordnlp/imdb/'
>     HUGGINGFACE_TOKEN_SECRET = hf_token;
>
> DESCRIBE STORAGE INTEGRATION imdb_hub;
> ```

> [tab] Python
>
> Set `LANCEDB_DATABASE`, `LANCEDB_API_KEY`, `LANCEDB_REST_ENDPOINT`, and
> `LANCEDB_SQL_ENDPOINT` to the values for your deployment. Endpoints include their
> scheme and port, and the SQL endpoint may differ from the REST endpoint.
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> import os
>
> import lancedb
>
> db = lancedb.connect(
>     f"db://{os.environ['LANCEDB_DATABASE']}",
>     api_key=os.environ["LANCEDB_API_KEY"],
>     host_override=os.environ["LANCEDB_REST_ENDPOINT"],
>     sql_host_override=os.environ["LANCEDB_SQL_ENDPOINT"],
> )
>
> db.execute_query(
>     "CREATE SECRET hf_token TYPE = GENERIC_STRING SECRET_STRING = $1",
>     parameters=[os.environ["HF_TOKEN"]],
> ).read_all()
>
> db.execute_query("""
>     CREATE STORAGE INTEGRATION imdb_hub
>         STORAGE_PROVIDER = 'HUGGINGFACE'
>         STORAGE_PREFIX = 'hf://datasets/stanfordnlp/imdb/'
>         HUGGINGFACE_TOKEN_SECRET = hf_token
> """).read_all()
>
> print(db.execute_query("DESCRIBE STORAGE INTEGRATION imdb_hub").read_all())
> ```
>
> Consume each result reader with `read_all()` so the statement completes before
> you start the next step. This example uses the SDK's SQL API for both setup and
> import.

The description returns the integration configuration with a Secret reference.
Registration does not verify repository access; the next query checks that the
token and source path work together. Metadata does not return the token value.

## 3. Preview and import the training split

The dataset's `plain_text/train-*.parquet` path selects its training split.
First preview the `text` and `label` columns, then use `CREATE TABLE ... AS SELECT`
(CTAS) to import a small sample. The source URI is single-quoted in SQL.

> [tab] SQL
>
> ```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> SELECT text, label
> FROM 'hf://datasets/stanfordnlp/imdb/plain_text/train-*.parquet'
> LIMIT 3;
>
> CREATE TABLE imdb_sample AS
> SELECT text, label
> FROM 'hf://datasets/stanfordnlp/imdb/plain_text/train-*.parquet'
> LIMIT 100;
>
> SELECT COUNT(*) AS imported_rows FROM imdb_sample;
> SELECT text, label FROM imdb_sample LIMIT 3;
> ```

> [tab] Python
>
> Continue with the `db` connection from the previous step:
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> print(db.execute_query("""
>     SELECT text, label
>     FROM 'hf://datasets/stanfordnlp/imdb/plain_text/train-*.parquet'
>     LIMIT 3
> """).read_all())
>
> db.execute_query("""
>     CREATE TABLE imdb_sample AS
>     SELECT text, label
>     FROM 'hf://datasets/stanfordnlp/imdb/plain_text/train-*.parquet'
>     LIMIT 100
> """).read_all()
>
> print(db.execute_query(
>     "SELECT COUNT(*) AS imported_rows FROM imdb_sample"
> ).read_all().to_pylist())
> print(db.execute_query("SELECT text, label FROM imdb_sample LIMIT 3").read_all())
> ```

The count query returns `imported_rows = 100`; Python prints
`[{'imported_rows': 100}]`. The preview returns review text and integer labels
(`0` for negative, `1` for positive). Row order is unspecified, so the sampled
reviews may differ between runs.

After CTAS completes, `imdb_sample` contains its own copy of the selected rows.
Reading this table no longer requires Hugging Face access. Remove `LIMIT 100`
to import the full training split. Large imports can use distributed execution
when the deployment has distributed SQL enabled and a compatible executor fleet;
the planner may run this small sample locally on the service.

## 4. Clean up the tutorial objects

Keep `imdb_sample` if you want to explore it. Otherwise, remove the objects you
created, dropping the integration before its token Secret:

> [tab] SQL
>
> ```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> DROP TABLE imdb_sample;
> DROP STORAGE INTEGRATION imdb_hub;
> DROP SECRET hf_token;
> ```

> [tab] Python
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
> db.execute_query("DROP TABLE imdb_sample").read_all()
> db.execute_query("DROP STORAGE INTEGRATION imdb_hub").read_all()
> db.execute_query("DROP SECRET hf_token").read_all()
> ```

## Choose a different dataset or revision

Use a single-quoted Hugging Face dataset URI directly in a SQL `FROM` clause:

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT id, text
FROM 'hf://datasets/owner/repo/data/train-*.parquet'
WHERE text IS NOT NULL
LIMIT 100;
```

The URI format is `hf://datasets/<owner>/<repo>[@<revision>]/<path>`.
A path can identify a Parquet file, a directory, or a glob. A repository root
selects all Parquet files below it. Use `*` within one directory and `**` across
directories. Select a dataset configuration or split explicitly through its path.
An unmatched path returns an error; it never falls back to other splits.

The default revision is `main`. Each statement resolves the selected revision to
a commit before scanning, so workers read the same snapshot. Percent-encode `/`
in revision names, for example `@refs%2Fconvert%2Fparquet`. Files in a source must
have compatible schemas. Nested Parquet values are preserved; floating-point
lists are not automatically converted into fixed-size vectors.

## Repository access and permissions

Register a storage integration to use a token stored in a Secret for a repository
prefix. The Secret must already exist and contain a Hugging Face token with read
access to the repository:

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
CREATE STORAGE INTEGRATION private_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = bootstrap.credentials.hf_token;

SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10;
```

`SELECT` and CTAS automatically choose the most specific matching integration
across native databases under the catalog root; the query does not name an
integration. Equal-length matches use the alphabetically first fully qualified
integration name. The caller needs usage permission on the
selected integration and its database and namespace. Workers receive the
integration binding and resolve the configured Secret without embedding the
token in SQL or the query plan.

`HUGGINGFACE_TOKEN_SECRET` is required when creating an integration. Use
`STORAGE_PREFIX = 'hf://'` to provide a token for all dataset repositories,
with a longer matching prefix taking precedence.

When no integration matches, public repositories are read anonymously at
Hugging Face's anonymous rate limits. SQL queries and workers do not use process
`HF_TOKEN` or `HUGGINGFACE_TOKEN` values. Lookup, authorization, and selected-token
errors remain errors. Historical integrations without a token must be recreated
with a token Secret, or dropped to use anonymous access.

Hugging Face integrations support read-only sources. `ATTACH DATABASE` accepts
`s3://`, `gs://`, and `az://` storage locations and rejects `hf://` locations.

## Import behavior and limitations

```sql theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
CREATE TABLE imported AS
SELECT *
FROM 'hf://datasets/owner/repo/data/train-*.parquet';
```

HF CTAS requires a native database with `database_manifest_enabled=true`.
CTAS queries combining HF sources with WAL-backed tables are currently rejected;
they cannot publish a base-only read that omits unflushed data.
With distributed SQL enabled and an available plan-executor fleet, the planner
can assign disjoint file and row-group ranges to workers. Workers evaluate the
query and write uncommitted Lance files directly to the destination storage.
The query node collects transaction metadata and commits the completed table.
Queries with global limits, aggregations, or joins retain their required
coordination. Small queries can execute locally when the planner uses one task.
Without a fleet, local output partitions write concurrently.

All participating workers need access to the destination storage and source
repository through the selected integration or unmatched access mode described
above. Use matching query-node and executor builds
that support the HF source and fragment-write operators.

New-table CTAS prepares the table privately and publishes its name only after
the completed table commits. A failure or cancellation before publication leaves
the name free for a new statement. Retrying the same durable CTAS job reuses its
creation token to recognize a table it already published. If nothing was
published, the retry allocates a fresh private destination.

An explicit `DECLARE TABLE` retains its declaration when initialization fails.
Retry that initialization, or drop the declaration before starting a new import.
Uncommitted files are never included in the completed table.

This source reads Parquet files. It does not run Hugging Face dataset scripts or
automatically generate embeddings or indexes. Create indexes separately after
importing.

Dataset imports use `hf://datasets/...` SQL sources. A Function's
`HuggingFaceModel` declaration prepares a model snapshot for that Function's
runtime; it does not import dataset rows into a table. See
[Hugging Face model snapshots](https://docs.lancedb.com/enterprise/v1-13-0/sql/dml/refresh.md#hugging-face-model-snapshots)
for that workflow.

## Troubleshoot an import

| Symptom | What to check |
| - | - |
| Repository access fails | Confirm the token has read access and the account has accepted any gated dataset terms. Check that the integration prefix covers the source URI. |
| Integration registration succeeds but the preview fails | Registration stores configuration without contacting Hugging Face. Run the preview to check access, path, and Parquet schema compatibility. |
| A Secret or integration is missing, or usage is denied | Use the same database and namespace as setup, or qualify the object names. Ask the owner for usage permission on the selected integration and its parents. |
| No Parquet files match | Browse the repository files and select a real split/configuration path. Dataset scripts and non-Parquet files cannot be imported by this source. |
| `imdb_sample` already exists | Choose a new table name, or drop the tutorial table before repeating CTAS. |
| Secrets or HF CTAS are unavailable | Ask your administrator to check Secret configuration, native database manifests, and support for Hugging Face sources in the deployed query nodes and executors. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.