Enterprise only
Import 100 movie reviews from the IMDb dataset
into a LanceDB Enterprise table. The service reads the source Parquet files and
writes the table in your database storage. The Python client submits SQL and
receives results; it does not download and upload the dataset.
Before you start
- Use an Enterprise deployment that supports Hugging Face SQL sources and storage integrations, with Secrets configured by your administrator.
- Select an existing native database with
database_manifest_enabled=true. You need permission to create Secrets, storage integrations, and tables in its root namespace (public), plus usage permission on the database and namespace. Select that database and namespace in your SQL client before running the examples. - For Python, install the LanceDB SDK with SQL support (
pip install -U lancedb). Obtain your database name, API key, REST endpoint, and Flight SQL endpoint from your administrator. See Connect with SQL.
hf_token, imdb_hub, and imdb_sample.
1. Obtain a Hugging Face token
Open Hugging Face access tokens and create a token with read access to the dataset repository. For private or gated datasets, the token’s account must have access, including any required approval or accepted dataset terms. A token does not grant access by itself. For the Python examples, setHF_TOKEN in your client environment through your
secret manager, or enter it without echoing it:
HF_TOKEN on your laptop or a server does not authenticate SQL workers.
The next step stores it in LanceDB and binds it to the repository.
2. Store the token and register the repository
Create a single-value Secret, then register a storage integration that references its name. Use a repository or folder prefix without wildcards; the integration is selected automatically for matching SQL sources.- SQL
- Python
The literal below shows the SQL syntax. For a real token, prefer the bound
parameter in the Python tab to keep the value out of SQL text and history.
3. Preview and import the training split
The dataset’splain_text/train-*.parquet path selects its training split.
First preview the text and label columns, then use CREATE TABLE ... AS SELECT
(CTAS) to import a small sample. The source URI is single-quoted in SQL.
- SQL
- Python
imported_rows = 100; Python prints
[{'imported_rows': 100}]. The preview returns review text and integer labels
(0 for negative, 1 for positive). Row order is unspecified, so the sampled
reviews may differ between runs.
After CTAS completes, imdb_sample contains its own copy of the selected rows.
Reading this table no longer requires Hugging Face access. Remove LIMIT 100
to import the full training split. Large imports can use distributed execution
when the deployment has distributed SQL enabled and a compatible executor fleet;
the planner may run this small sample locally on the service.
4. Clean up the tutorial objects
Keepimdb_sample if you want to explore it. Otherwise, remove the objects you
created, dropping the integration before its token Secret:
- SQL
- Python
Choose a different dataset or revision
Use a single-quoted Hugging Face dataset URI directly in a SQLFROM clause:
hf://datasets/<owner>/<repo>[@<revision>]/<path>.
A path can identify a Parquet file, a directory, or a glob. A repository root
selects all Parquet files below it. Use * within one directory and ** across
directories. Select a dataset configuration or split explicitly through its path.
An unmatched path returns an error; it never falls back to other splits.
The default revision is main. Each statement resolves the selected revision to
a commit before scanning, so workers read the same snapshot. Percent-encode /
in revision names, for example @refs%2Fconvert%2Fparquet. Files in a source must
have compatible schemas. Nested Parquet values are preserved; floating-point
lists are not automatically converted into fixed-size vectors.
Repository access and permissions
Register a storage integration to use a token stored in a Secret for a repository prefix. The Secret must already exist and contain a Hugging Face token with read access to the repository:SELECT and CTAS automatically choose the most specific matching integration
across native databases under the catalog root; the query does not name an
integration. Equal-length matches use the alphabetically first fully qualified
integration name. The caller needs usage permission on the
selected integration and its database and namespace. Workers receive the
integration binding and resolve the configured Secret without embedding the
token in SQL or the query plan.
HUGGINGFACE_TOKEN_SECRET is required when creating an integration. Use
STORAGE_PREFIX = 'hf://' to provide a token for all dataset repositories,
with a longer matching prefix taking precedence.
When no integration matches, public repositories are read anonymously at
Hugging Face’s anonymous rate limits. SQL queries and workers do not use process
HF_TOKEN or HUGGINGFACE_TOKEN values. Lookup, authorization, and selected-token
errors remain errors. Historical integrations without a token must be recreated
with a token Secret, or dropped to use anonymous access.
Hugging Face integrations support read-only sources. ATTACH DATABASE accepts
s3://, gs://, and az:// storage locations and rejects hf:// locations.
Import behavior and limitations
database_manifest_enabled=true.
CTAS queries combining HF sources with WAL-backed tables are currently rejected;
they cannot publish a base-only read that omits unflushed data.
With distributed SQL enabled and an available plan-executor fleet, the planner
can assign disjoint file and row-group ranges to workers. Workers evaluate the
query and write uncommitted Lance files directly to the destination storage.
The query node collects transaction metadata and commits the completed table.
Queries with global limits, aggregations, or joins retain their required
coordination. Small queries can execute locally when the planner uses one task.
Without a fleet, local output partitions write concurrently.
All participating workers need access to the destination storage and source
repository through the selected integration or unmatched access mode described
above. Use matching query-node and executor builds
that support the HF source and fragment-write operators.
New-table CTAS prepares the table privately and publishes its name only after
the completed table commits. A failure or cancellation before publication leaves
the name free for a new statement. Retrying the same durable CTAS job reuses its
creation token to recognize a table it already published. If nothing was
published, the retry allocates a fresh private destination.
An explicit DECLARE TABLE retains its declaration when initialization fails.
Retry that initialization, or drop the declaration before starting a new import.
Uncommitted files are never included in the completed table.
This source reads Parquet files. It does not run Hugging Face dataset scripts or
automatically generate embeddings or indexes. Create indexes separately after
importing.
Dataset imports use hf://datasets/... SQL sources. A Function’s
HuggingFaceModel declaration prepares a model snapshot for that Function’s
runtime; it does not import dataset rows into a table. See
Hugging Face model snapshots
for that workflow.