> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Inspect Datasets

> Query a table, a HuggingFace dataset, or a dataset in object storage the same way, with plain SELECT, WHERE, projection and LIMIT.

**Applies to:** Embedded ❌, Enterprise ✅

The same `SELECT` statement inspects three kinds of source: a table
already in your database, a HuggingFace dataset, or a dataset in object
storage -- once a [storage integration](https://docs.lancedb.com/management/storage-integration.md)
covers the latter two. Nothing is downloaded or imported until you choose
to import it.

## The basics: project, filter, limit

A table name, a HuggingFace URI, and an object-storage URI all work the
same way after `FROM`:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT id, text
FROM documents
WHERE text IS NOT NULL
LIMIT 100
```

* **Project** -- name the columns you need (`id, text`) instead of
  `SELECT *`, especially for a source with large or many columns.
* **Filter** -- `WHERE` narrows rows before they come back; combine
  predicates with `AND`/`OR` the same as any other table.
* **Limit** -- `LIMIT n` caps how many rows return. It bounds the result,
  not the read: a remote source is still fetched at file or row-group
  granularity, so a small `LIMIT` on a source with large files or row
  groups can still touch more data than the row count suggests.

The rest of this page swaps `documents` for a HuggingFace or
object-storage source -- the `SELECT`/`WHERE`/`LIMIT` syntax above doesn't
change.

## Inspect a HuggingFace dataset

A single-quoted HuggingFace dataset URI works directly in a SQL `FROM`
clause, the same as a table name -- no separate download, no intermediate
file on disk:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT id, text
FROM 'hf://datasets/owner/repo/data/train-*.parquet'
WHERE text IS NOT NULL
LIMIT 100
```

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
reader = db.execute_query("""
    SELECT id, text
    FROM 'hf://datasets/owner/repo/data/train-*.parquet'
    WHERE text IS NOT NULL
    LIMIT 100
""")
sample = reader.read_all().to_pylist()
```

The URI is `hf://datasets/<owner>/<repo>[@<revision>]/<path>`. `<path>` can
be one Parquet file, a directory, or a glob: `*` matches within one
directory, `**` matches across directories, and a repository root selects
every Parquet file beneath it. Select a dataset configuration or split by
including it in the path. The default revision is `main`; each statement
resolves the revision to a commit once, so every worker reads the same
snapshot. This source reads Parquet files -- it does not run a dataset's
own generation scripts, and it does not compute embeddings or build
indexes, so [chunk](https://docs.lancedb.com/curation/chunking.md), embed, or index the result as a
separate step, the same as data from any other source.

> [!NOTE]
>
> Files matched by one statement must have compatible schemas, and an
> unmatched path is an error rather than a silent fallback to a different
> split. Nested Parquet values come through as-is; a floating-point list is
> not automatically converted into a fixed-size vector column.

## Inspect a dataset in object storage

An `s3://`, `gs://` or `az://` prefix covered by a storage integration
reads an existing **Lance** dataset at that prefix, the same way as a
managed table:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
SELECT id, text
FROM 's3://customer-data/raw/documents.lance'
WHERE text IS NOT NULL
LIMIT 100
```

This reads a dataset another process already wrote in Lance format, not
an arbitrary file format -- there is no Parquet-style reader for cloud
storage the way there is for `hf://`. See
[Storage Integration](https://docs.lancedb.com/sql/ddl/storage-integration.md#read-import) for every
provider's setup.

## Access a private repository or bucket

A private HuggingFace repository, one subject to HuggingFace's
authenticated rate limits, or any non-public bucket needs a
[storage integration](https://docs.lancedb.com/management/storage-integration.md) covering it. For
HuggingFace that integration is bound to a token
[Secret](https://docs.lancedb.com/management/secret.md); for a non-public S3 bucket it can instead be
an IAM role integration, which needs no Secret at all -- see
[AWS S3](https://docs.lancedb.com/ingestion/define-storage-integration.md#aws).

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
db.create_secret("hf_token", "<your-hugging-face-token>")

# create_storage_integration is not in a released SDK yet; use the SQL
# form below until it ships.
db.execute_query("""
    CREATE STORAGE INTEGRATION private_hub
        STORAGE_PROVIDER = 'HUGGINGFACE'
        STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
        HUGGINGFACE_TOKEN_SECRET = hf_token
""").read_all()

reader = db.execute_query(
    "SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10"
)
```

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
CREATE SECRET hf_token TYPE = GENERIC_STRING SECRET_STRING = '<your-hugging-face-token>'

CREATE STORAGE INTEGRATION private_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = hf_token

SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10
```

A query never names the integration: the planner matches the most specific
`STORAGE_PREFIX` that covers the URI being read, across every integration
the caller has `USAGE` on. A HuggingFace integration always needs
`HUGGINGFACE_TOKEN_SECRET`; creating one without it is rejected. For public
access, register no integration for that prefix at all -- an `hf://` read
that matches no integration is anonymous, with no environment-variable
fallback. See
[Storage Integration](https://docs.lancedb.com/management/storage-integration.md) for every provider
and the full credential model.

## Import into a table

Once you've inspected a source and it looks right, `CREATE TABLE ... AS
SELECT` imports it:

```sql SQL icon="database" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
CREATE TABLE imported AS
SELECT *
FROM 'hf://datasets/owner/repo/data/train-*.parquet'
```

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
db.execute_query("""
    CREATE TABLE imported AS
    SELECT * FROM 'hf://datasets/owner/repo/data/train-*.parquet'
""").read_all()
table = db.open_table("imported")
```

This `CREATE TABLE AS SELECT` runs as a distributed import when the
deployment has an available plan-executor fleet: the planner assigns
disjoint file and row-group ranges to workers, each worker writes
uncommitted Lance files directly to the destination, and a coordinator
commits the completed table from their transaction metadata once every
worker finishes. A query with a global limit, aggregation or join keeps
its usual coordination; a small import can run as one local task without a
fleet. The table becomes queryable only after that final commit --
nothing partial is ever visible, and a failed or cancelled import can
resume the same durable job from its retained attempt rather than
restarting from nothing. To start over after a failed one-off import
instead, drop the partial table and rerun the statement.

> [!NOTE]
>
> Importing requires a native database with `database_manifest_enabled`.
> Combining a HuggingFace source with a WAL-backed destination table in one
> CTAS is rejected, since the import cannot publish a base-only read that
> omits the destination's unflushed data.

## What to read next

> [card] [Storage Integration](https://docs.lancedb.com/management/storage-integration.md)
>
> The credential object behind private repository access.

> [card] [Create a Function](https://docs.lancedb.com/curation/create-function.md)
>
> Add embeddings or derived columns to the imported data.

> [card] [Chunk records into multiple rows](https://docs.lancedb.com/curation/chunking.md)
>
> Split long text or media fields after importing.

> [card] [Table](https://docs.lancedb.com/sql/ddl/table.md#create)
>
> `CREATE TABLE AS SELECT` syntax and options in full.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.