Skip to main content
Enterprise only
Import 100 movie reviews from the IMDb dataset into a LanceDB Enterprise table. The service reads the source Parquet files and writes the table in your database storage. The Python client submits SQL and receives results; it does not download and upload the dataset.

Before you start

  • Use an Enterprise deployment that supports Hugging Face SQL sources and storage integrations, with Secrets configured by your administrator.
  • Select an existing native database with database_manifest_enabled=true. You need permission to create Secrets, storage integrations, and tables in its root namespace (public), plus usage permission on the database and namespace. Select that database and namespace in your SQL client before running the examples.
  • For Python, install the LanceDB SDK with SQL support (pip install -U lancedb). Obtain your database name, API key, REST endpoint, and Flight SQL endpoint from your administrator. See Connect with SQL.
The SQL and Python tabs below are alternative ways to run the same steps. Use the same database and namespace throughout, and choose unused names for hf_token, imdb_hub, and imdb_sample.

1. Obtain a Hugging Face token

Open Hugging Face access tokens and create a token with read access to the dataset repository. For private or gated datasets, the token’s account must have access, including any required approval or accepted dataset terms. A token does not grant access by itself. For the Python examples, set HF_TOKEN in your client environment through your secret manager, or enter it without echoing it:
IMDb is public and can also be read anonymously. This tutorial registers a token so you can use the same procedure for repositories that require authentication. Setting HF_TOKEN on your laptop or a server does not authenticate SQL workers. The next step stores it in LanceDB and binds it to the repository.

2. Store the token and register the repository

Create a single-value Secret, then register a storage integration that references its name. Use a repository or folder prefix without wildcards; the integration is selected automatically for matching SQL sources.
The literal below shows the SQL syntax. For a real token, prefer the bound parameter in the Python tab to keep the value out of SQL text and history.
The description returns the integration configuration with a Secret reference. Registration does not verify repository access; the next query checks that the token and source path work together. Metadata does not return the token value.

3. Preview and import the training split

The dataset’s plain_text/train-*.parquet path selects its training split. First preview the text and label columns, then use CREATE TABLE ... AS SELECT (CTAS) to import a small sample. The source URI is single-quoted in SQL.
The count query returns imported_rows = 100; Python prints [{'imported_rows': 100}]. The preview returns review text and integer labels (0 for negative, 1 for positive). Row order is unspecified, so the sampled reviews may differ between runs. After CTAS completes, imdb_sample contains its own copy of the selected rows. Reading this table no longer requires Hugging Face access. Remove LIMIT 100 to import the full training split. Large imports can use distributed execution when the deployment has distributed SQL enabled and a compatible executor fleet; the planner may run this small sample locally on the service.

4. Clean up the tutorial objects

Keep imdb_sample if you want to explore it. Otherwise, remove the objects you created, dropping the integration before its token Secret:

Choose a different dataset or revision

Use a single-quoted Hugging Face dataset URI directly in a SQL FROM clause:
The URI format is hf://datasets/<owner>/<repo>[@<revision>]/<path>. A path can identify a Parquet file, a directory, or a glob. A repository root selects all Parquet files below it. Use * within one directory and ** across directories. Select a dataset configuration or split explicitly through its path. An unmatched path returns an error; it never falls back to other splits. The default revision is main. Each statement resolves the selected revision to a commit before scanning, so workers read the same snapshot. Percent-encode / in revision names, for example @refs%2Fconvert%2Fparquet. Files in a source must have compatible schemas. Nested Parquet values are preserved; floating-point lists are not automatically converted into fixed-size vectors.

Repository access and permissions

Register a storage integration to use a token stored in a Secret for a repository prefix. The Secret must already exist and contain a Hugging Face token with read access to the repository:
SELECT and CTAS automatically choose the most specific matching integration across native databases under the catalog root; the query does not name an integration. Equal-length matches use the alphabetically first fully qualified integration name. The caller needs usage permission on the selected integration and its database and namespace. Workers receive the integration binding and resolve the configured Secret without embedding the token in SQL or the query plan. HUGGINGFACE_TOKEN_SECRET is required when creating an integration. Use STORAGE_PREFIX = 'hf://' to provide a token for all dataset repositories, with a longer matching prefix taking precedence. When no integration matches, public repositories are read anonymously at Hugging Face’s anonymous rate limits. SQL queries and workers do not use process HF_TOKEN or HUGGINGFACE_TOKEN values. Lookup, authorization, and selected-token errors remain errors. Historical integrations without a token must be recreated with a token Secret, or dropped to use anonymous access. Hugging Face integrations support read-only sources. ATTACH DATABASE accepts s3://, gs://, and az:// storage locations and rejects hf:// locations.

Import behavior and limitations

HF CTAS requires a native database with database_manifest_enabled=true. CTAS queries combining HF sources with WAL-backed tables are currently rejected; they cannot publish a base-only read that omits unflushed data. With distributed SQL enabled and an available plan-executor fleet, the planner can assign disjoint file and row-group ranges to workers. Workers evaluate the query and write uncommitted Lance files directly to the destination storage. The query node collects transaction metadata and commits the completed table. Queries with global limits, aggregations, or joins retain their required coordination. Small queries can execute locally when the planner uses one task. Without a fleet, local output partitions write concurrently. All participating workers need access to the destination storage and source repository through the selected integration or unmatched access mode described above. Use matching query-node and executor builds that support the HF source and fragment-write operators. New-table CTAS prepares the table privately and publishes its name only after the completed table commits. A failure or cancellation before publication leaves the name free for a new statement. Retrying the same durable CTAS job reuses its creation token to recognize a table it already published. If nothing was published, the retry allocates a fresh private destination. An explicit DECLARE TABLE retains its declaration when initialization fails. Retry that initialization, or drop the declaration before starting a new import. Uncommitted files are never included in the completed table. This source reads Parquet files. It does not run Hugging Face dataset scripts or automatically generate embeddings or indexes. Create indexes separately after importing. Dataset imports use hf://datasets/... SQL sources. A Function’s HuggingFaceModel declaration prepares a model snapshot for that Function’s runtime; it does not import dataset rows into a table. See Hugging Face model snapshots for that workflow.

Troubleshoot an import