Inspect Datasets

Applies to: Embedded ❌, Enterprise ✅ The same SELECT statement inspects three kinds of source: a table already in your database, a HuggingFace dataset, or a dataset in object storage — once a storage integration covers the latter two. Nothing is downloaded or imported until you choose to import it.

The basics: project, filter, limit

A table name, a HuggingFace URI, and an object-storage URI all work the same way after FROM:

SELECT id, text
FROM documents
WHERE text IS NOT NULL
LIMIT 100

Project — name the columns you need (id, text) instead of SELECT *, especially for a source with large or many columns.
Filter — WHERE narrows rows before they come back; combine predicates with AND/OR the same as any other table.
Limit — LIMIT n caps how many rows return. It bounds the result, not the read: a remote source is still fetched at file or row-group granularity, so a small LIMIT on a source with large files or row groups can still touch more data than the row count suggests.

The rest of this page swaps documents for a HuggingFace or object-storage source — the SELECT/WHERE/LIMIT syntax above doesn’t change.

Inspect a HuggingFace dataset

A single-quoted HuggingFace dataset URI works directly in a SQL FROM clause, the same as a table name — no separate download, no intermediate file on disk:

SELECT id, text
FROM 'hf://datasets/owner/repo/data/train-*.parquet'
WHERE text IS NOT NULL
LIMIT 100

reader = db.execute_query("""
    SELECT id, text
    FROM 'hf://datasets/owner/repo/data/train-*.parquet'
    WHERE text IS NOT NULL
    LIMIT 100
""")
sample = reader.read_all().to_pylist()

The URI is hf://datasets/<owner>/<repo>[@<revision>]/<path>. <path> can be one Parquet file, a directory, or a glob: * matches within one directory, ** matches across directories, and a repository root selects every Parquet file beneath it. Select a dataset configuration or split by including it in the path. The default revision is main; each statement resolves the revision to a commit once, so every worker reads the same snapshot. This source reads Parquet files — it does not run a dataset’s own generation scripts, and it does not compute embeddings or build indexes, so chunk, embed, or index the result as a separate step, the same as data from any other source.

Files matched by one statement must have compatible schemas, and an unmatched path is an error rather than a silent fallback to a different split. Nested Parquet values come through as-is; a floating-point list is not automatically converted into a fixed-size vector column.

Inspect a dataset in object storage

An s3://, gs:// or az:// prefix covered by a storage integration reads an existing Lance dataset at that prefix, the same way as a managed table:

SELECT id, text
FROM 's3://customer-data/raw/documents.lance'
WHERE text IS NOT NULL
LIMIT 100

This reads a dataset another process already wrote in Lance format, not an arbitrary file format — there is no Parquet-style reader for cloud storage the way there is for hf://. See Storage Integration for every provider’s setup.

Access a private repository or bucket

A private HuggingFace repository, one subject to HuggingFace’s authenticated rate limits, or any non-public bucket needs a storage integration covering it. For HuggingFace that integration is bound to a token Secret; for a non-public S3 bucket it can instead be an IAM role integration, which needs no Secret at all — see AWS S3.

db.create_secret("hf_token", "<your-hugging-face-token>")

# create_storage_integration is not in a released SDK yet; use the SQL
# form below until it ships.
db.execute_query("""
    CREATE STORAGE INTEGRATION private_hub
        STORAGE_PROVIDER = 'HUGGINGFACE'
        STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
        HUGGINGFACE_TOKEN_SECRET = hf_token
""").read_all()

reader = db.execute_query(
    "SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10"
)

CREATE SECRET hf_token TYPE = GENERIC_STRING SECRET_STRING = '<your-hugging-face-token>'

CREATE STORAGE INTEGRATION private_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = hf_token

SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10

A query never names the integration: the planner matches the most specific STORAGE_PREFIX that covers the URI being read, across every integration the caller has USAGE on. A HuggingFace integration always needs HUGGINGFACE_TOKEN_SECRET; creating one without it is rejected. For public access, register no integration for that prefix at all — an hf:// read that matches no integration is anonymous, with no environment-variable fallback. See Storage Integration for every provider and the full credential model.

Import into a table

Once you’ve inspected a source and it looks right, CREATE TABLE ... AS SELECT imports it:

CREATE TABLE imported AS
SELECT *
FROM 'hf://datasets/owner/repo/data/train-*.parquet'

db.execute_query("""
    CREATE TABLE imported AS
    SELECT * FROM 'hf://datasets/owner/repo/data/train-*.parquet'
""").read_all()
table = db.open_table("imported")

This CREATE TABLE AS SELECT runs as a distributed import when the deployment has an available plan-executor fleet: the planner assigns disjoint file and row-group ranges to workers, each worker writes uncommitted Lance files directly to the destination, and a coordinator commits the completed table from their transaction metadata once every worker finishes. A query with a global limit, aggregation or join keeps its usual coordination; a small import can run as one local task without a fleet. The table becomes queryable only after that final commit — nothing partial is ever visible, and a failed or cancelled import can resume the same durable job from its retained attempt rather than restarting from nothing. To start over after a failed one-off import instead, drop the partial table and rerun the statement.

Importing requires a native database with database_manifest_enabled. Combining a HuggingFace source with a WAL-backed destination table in one CTAS is rejected, since the import cannot publish a base-only read that omits the destination’s unflushed data.

Storage Integration

The credential object behind private repository access.

Create a Function

Add embeddings or derived columns to the imported data.

Chunk records into multiple rows

Split long text or media fields after importing.

Table

CREATE TABLE AS SELECT syntax and options in full.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Inspect Datasets

The basics: project, filter, limit

Inspect a HuggingFace dataset

Inspect a dataset in object storage

Access a private repository or bucket

Import into a table

What to read next

Storage Integration

Create a Function

Chunk records into multiple rows

Table

​The basics: project, filter, limit

​Inspect a HuggingFace dataset

​Inspect a dataset in object storage

​Access a private repository or bucket

​Import into a table

​What to read next

Storage Integration

Create a Function

Chunk records into multiple rows

Table

The basics: project, filter, limit

Inspect a HuggingFace dataset

Inspect a dataset in object storage

Access a private repository or bucket

Import into a table

What to read next