> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LeRobotDataset

> Train LeRobot policies straight from object storage, and curate, search and visualize robotics datasets as Lance tables.

[LeRobot](https://huggingface.co/docs/lerobot/index) is HuggingFace's open-source robotics stack. Since [lerobot#4363](https://github.com/huggingface/lerobot/pull/4363) it reads Lance datasets natively, and `lerobot-lancedb` wraps that into a ready-made dataset class for video features: same items, same training pipeline. Image-backed features are not supported -- re-encode them as video first.

What you get:

* **Train from object storage while keeping your GPUs fed.** Lance reads random frames straight from S3, GCS, the HF Hub or HF Storage Buckets, fetching only the bytes each batch needs, so every batch is a global shuffle: any frame, from anywhere in the dataset. In the table below (samples/s, 8 workers, batch 64), Lance and upstream local both draw every batch from the whole dataset. Upstream streaming does a reservoir shuffle, reading the dataset in order and drawing each batch from a pool of recently seen frames, so those two columns are not the same job:

| dataset | window shuffle, Lance from object store | window shuffle, upstream streaming from the Hub | Lance (object store) vs upstream streaming | RAM, Lance | RAM, streaming | global shuffle, Lance from object store | global shuffle, upstream from local NVMe | Lance (object store) vs upstream (local NVMe) |
| - | - | - | - | - | - | - | - | - |
| DROID | 904 | 478 | 1.9x | 8 GB | 122 GB | 257 | 224 | 1.1x |
| roboturk | 778 | 152 | 5.1x | 6 GB | 9 GB | 1,074 | 983 | 1.1x |
| aloha\_mobile\_cabinet | 772 | 51 | 15.2x | 9 GB | 30 GB | 693 | 573 | 1.2x |
| koch\_pick\_place\_5\_lego | 701 | 54 | 13.0x | 8 GB | 15 GB | 876 | 526 | 1.7x |
| pusht | 5,340 | 648 | 8.2x | 3 GB | 7 GB | 5,508 | 4,224 | 1.3x |
| aloha\_static\_cups\_open | 597 | 45 | 13.2x | 10 GB | 38 GB | 575 | 483 | 1.2x |

The shuffle window is 0.05% of the dataset, at least 5,000 frames: 5,000 on the small sets, 13,815 on DROID. Streaming DROID at that window in upstream default needs 122 GB, so those two runs used a 256 GB machine with the same decode speed. The DROID pair is noisy: Lance 684 and 1,124, streaming 303 and 653.

Same batches: Lance from an object store is 1.9x faster than upstream streaming on DROID and 5x to 15x on the other datasets, with 8 GB of RAM against 122 GB on DROID. Global shuffle: Lance with nothing downloaded is 1.1x to 1.7x faster than upstream reading local NVMe after the download. berkeley\_rpt, toto and soarm101 are left out because their metadata lists frames the videos do not have; `lerobot-lance-doctor` reports that before you train.

* **Faster iteration, end to end.** 10,000 steps of SmolVLA on DROID, 8xH100, lerobot's default `num_workers=4`, same seed:

| | Lance, S3, nothing downloaded | upstream, local NVMe after a 384 GB download |
| - | - | - |
| wall clock | **1 h 27 m** | 2 h 00 m |
| steady rate | 495 samples/s | 361 samples/s |
| of each step spent waiting on data | 1.7% | 37.4% |
| final loss | 0.2380 | 0.2380 |

Same model, same loss, 33 minutes sooner, and no 384 GB download first. The difference is time the GPUs spend waiting on data, so it grows with the run: at 100k steps it is over five hours, and every re-run and sweep point pays it again.

* **The training table is also the index.** Parquet has no secondary indexes, so finding frames instead of scanning them usually means another system per question: a vector database for embeddings, a search service for instructions, a feature store for scores. With Lance those are columns and indexes on the same table the DataLoader reads. One query spans all of them, at a version the trainer can reopen.

## Install

```bash Bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
pip install "lerobot[lancedb]"   # the reader, inside lerobot itself
pip install lerobot-lancedb      # the converter
```

## Convert a dataset

Any LeRobot v3.0 dataset, from the Hub or a local directory:

```bash Bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
lerobot-lance-convert --repo-id lerobot/pusht --out ./pusht-lance
```

**Migration costs about as much as a file copy.** Videos are stored as blobs without re-encoding, so items are bit-identical to the source: 392,578 frames convert in 10 seconds, DROID's 27.6M in 34 minutes. Nothing about your recording or training code changes. The output keeps `meta/` as is and adds three Lance tables (`frames`, `videos`, `meta`). `lerobot-lance-doctor` audits the result and often finds problems in the source: DROID carries 44% orphaned rows, and three of eight public datasets we converted have frame-count defects that only random access exposes.

Host the result anywhere: an HF dataset repo (`hf upload`, tag it `v3.0`), an HF Storage Bucket (`hf buckets sync`), or any object store (`aws s3 sync`).

## Load and train

`lerobot-lancedb` provides one map-style dataset class for both image and
video features, reading from a local path or an `s3://`, `gs://` or `hf://`
URI with nothing downloaded up front. Pair it with `lance_mp_context()` so
multi-worker `DataLoader`s share Lance's own multiprocessing context instead
of the default one:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
from lerobot_lancedb import LanceDBDataset, lance_mp_context
from torch.utils.data import DataLoader

dataset = LanceDBDataset(root="hf://datasets/lance-format/pusht-lance")
sample = dataset[0]
print(sample["observation.state"].shape)
print(sample["action"].shape)

loader = DataLoader(
    dataset,
    batch_size=32,
    num_workers=8,
    multiprocessing_context=lance_mp_context(),
)
```

Items are bit-exact with `LeRobotDataset`, and `delta_timestamps` configures
temporal windows the same way.

Or from the CLI. Held-out evaluation on a remote dataset needs random access, so it only works on the Lance path:

```bash Bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
lerobot-train \
  --dataset.repo_id=my-org/my-bucket/pusht-lance \
  --dataset.repo_type=bucket \
  --dataset.eval_split=0.02 \
  --policy.type=act
```

## Visualize TB-scale datasets in Foxglove

LeRobot's dataset viewer can serve an episode to [Foxglove](https://foxglove.dev) over its WebSocket protocol ([lerobot#4542](https://github.com/huggingface/lerobot/pull/4542)), and with a Lance dataset it works wherever the data lives:

```bash Bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
lerobot-dataset-viz --repo-id lance-format/pusht-lance --episode-index 3 --display-mode foxglove
```

Open the Foxglove app, connect to `ws://localhost:8765`, and scrub. Cameras, joint state and actions show up as topics you can lay out however you like:

![Foxglove showing episode 7 of DROID served from S3: three camera panels and the joint state, scrubbed to 19.4 s](https://mintcdn.com/lancedb-bcbb4faf/06qyYZ1Az1OmWUp2/oss/v0-41-0/static/assets/images/integrations/lerobot-foxglove-droid.png?fit=max\&auto=format\&n=06qyYZ1Az1OmWUp2\&q=85\&s=4fc5a07ad42fc2009ecc4c74e19463eb)

Frames are fetched as you scrub, so looking at one episode of the 373 GB DROID from S3 moves a few MB, not the dataset. No MCAP export, no download, and nothing new to install on the dataset side: this is the same reader the trainer uses.

## Query, curate, enrich

The frames table is a LanceDB table like any other:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import lancedb

db = lancedb.connect("hf://datasets/lance-format/pusht-lance")
frames = db.open_table("frames")
videos = db.open_table("videos")

print(len(frames), len(videos))
print(frames.schema)
```

Filter with SQL over any feature, using the indexes built at conversion:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
frame_rows = (
    frames.search()
    .where("episode_index = 0 AND frame_index < 10", prefilter=True)
    .select(["episode_index", "frame_index", "timestamp", "action"])
    .limit(10)
    .to_list()
)

for row in frame_rows:
    print(row["episode_index"], row["frame_index"], row["timestamp"])
```

From here it is standard LanceDB. The examples below are from our DROID walkthrough, all on the table the trainer reads.

**Feature engineering costs nothing until you use it.** With [Geneva](https://docs.lancedb.com/oss/v0-41-0/geneva/udfs/udfs.md), `add_columns` registers a `@udf` function and its input columns as a new column and computes nothing; `backfill` computes and stores the values later, in parallel. No copy of the dataset, and the videos are never touched:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.float32(), input_columns=["action_joint_velocity"])
def jerk_score(v):                      # per-frame motion roughness
    ...

tbl.add_columns({"jerk_score": jerk_score})   # nothing computed yet
tbl.backfill("jerk_score", concurrency=8)     # 610,403 rows in 16.7 s
```

Embeddings work the same way, reading frames straight from the video blobs in the `videos` table:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
@udf(data_type=pa.list_(pa.float32(), 768), num_gpus=1,
     input_columns=["episode_index", "frame_index"])
class EmbedFromBlob:                    # SigLIP2 loads once per worker
    def __call__(self, episode_index, frame_index):
        blob = self.videos.fetch_blob_files("video_bytes", [video_row_id])[0]
        frame = VideoDecoder(blob).get_frames_at([pos]).data[0]
        ...
```

**Deduplication** removes near-static runs before they skew curation or
training. A stopped gripper holding one pose for a few seconds produces many
frames whose `emb_siglip2` vectors are nearly identical; LanceDB Enterprise's
native [`dedupe`](https://docs.lancedb.com/oss/v0-41-0/sql/dql/dedupe) collapses them into one
representative per cluster, directly over the index, without a pairwise scan:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
db.execute_query("""
    CREATE MATERIALIZED VIEW frames_deduped AS
    SELECT * FROM dedupe(
        'frames',
        'emb_siglip2',
        distance_threshold => 0.01
    )
""")
```

Every column survives the dedup unchanged -- `jerk_score`, the episode and
camera metadata, the video blob reference -- since `dedupe` returns full
rows, not just which ones matched. Wrap it in a materialized view, as above,
to keep the deduplicated set current as new episodes land; see
[Materialized views](https://docs.lancedb.com/oss/v0-41-0/feature-engineering/materialized-views.md).

**Search** is an index on those columns: a vector index over the embeddings built in 29.7 s across 27.6M rows, a full-text index over the instructions in 7.0 s, queries in 10 to 54 ms.

**Curation** is one query across all of it. Semantic search with a predicate on the derived score, no join, no second system:

```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
results = (
    tbl.search(vec("a gripper closing on an object"), vector_column_name="emb_siglip2")
    .where("jerk_score > 1.2875")  # the roughest 1% of frames
    .limit(4)
    .to_arrow()
)
```

**Mining** is the same query pointed at the rare cases. `tbl.search().where("jerk_score > 1.2875 AND success = false")` pulls the rough, failed frames out of 27.6M rows in milliseconds, ready to become a fine-tuning set or a labeling queue.

**EDA** is a scan of the tabular columns: a pass over all 27.6M rows reads 2.52 GB in 190 s and decodes no video.

Then drop what you found from training with `episodes=[...]` instead of rewriting files. The reader ignores columns it does not know, and the table is versioned, so curation and training share one copy of the data at a version you can reopen.

> [!NOTE]
>
> Lance datasets are read-only in LeRobot: record and edit in the default format, then convert.

Public examples to try everything above: [`lance-format/pusht-lance`](https://huggingface.co/datasets/lance-format/pusht-lance) and friends under the [`lance-format`](https://huggingface.co/lance-format) org.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.