LeRobot Native Integration

LeRobot is Hugging Face’s open-source robotics stack. LeRobot can read Lance datasets natively through the standard LeRobotDataset class and training pipeline. Setting "storage_format": "lance" in the dataset selects the Lance reader without changing how the rest of the workflow works. What you get:

Train from object storage while keeping your GPUs fed. Lance reads random frames straight from S3, GCS, the HF Hub or HF Storage Buckets, fetching only the bytes each batch needs. Every batch can be a global shuffle across the full dataset. In the table below (samples/s, 8 workers, batch 64), Lance and upstream local both draw every batch from the whole dataset. Upstream streaming does a reservoir shuffle, reading the dataset in order and drawing each batch from a pool of recently seen frames, so those two columns are not the same job:

dataset	frames	cameras	Lance, S3 (global shuffle, samples/s)	upstream, local (global shuffle, samples/s)	speedup vs local	upstream, streaming (reservoir shuffle, samples/s)	speedup vs streaming
DROID	27,630,375	3	2,258.5	346.7	6.5x	898.6	1.7x
berkeley_rpt	392,578	1	1,740.6	1,209.9	1.4x	274.5	6.3x
toto	325,699	1	1,097.0	936.3	1.2x	121.3	9.0x
roboturk	187,507	1	1,276.8	1,082.3	1.2x	128.2	10.0x
aloha_mobile_cabinet	127,500	3	815.7	294.2	2.8x	77.7	10.5x
koch_pick_place	37,972	2	854.7	274.9	3.1x	58.1	14.7x

The gap grows with how much video each sample carries: about 1.2x with one camera, 2.8x with three, 6.5x on DROID.

Faster iteration, end to end. 10,000 steps of SmolVLA on DROID, 8xH100, lerobot’s default num_workers=4, same seed:

	Lance, S3, nothing downloaded	upstream, local NVMe after a 384 GB download
wall clock	1 h 27 m	2 h 00 m
steady rate	495 samples/s	361 samples/s
of each step spent waiting on data	1.7%	37.4%
final loss	0.2380	0.2380

Training reached the same result, but finished 33 minutes faster and avoided a 384 GB download upfront. The savings come from reducing the time GPUs spend waiting on data, so they compound over longer runs. At 100k steps, that adds up to more than five hours saved, and every re-run and sweep benefits again.

The training table is also the index. Parquet has no secondary indexes, so finding frames instead of scanning them usually means another system per question, such as a vector database for embeddings, a search service for instructions, a feature store for scores. With Lance, those are columns and indexes on the same table the DataLoader reads. One query spans all of them, at a version the trainer can reopen.

Install

pip install "lerobot[lancedb]"   # the reader, inside lerobot itself
pip install "lerobot[lancedb-convert]"   # to convert LeRobot datasets to Lance

On macOS, use Python 3.12 or 3.13. Lance datasets with RGB video require TorchCodec, which does not fall back to PyAV in the Lance reader. If TorchCodec cannot load FFmpeg, install FFmpeg 8:

brew install ffmpeg@8
export PATH="$(brew --prefix ffmpeg@8)/bin:$PATH"
export DYLD_LIBRARY_PATH="$(brew --prefix ffmpeg@8)/lib:$DYLD_LIBRARY_PATH"

Verify with python -c "import torchcodec".

Convert a dataset

Any LeRobot v3.0 dataset, from the Hub or a local directory:

lerobot-lance-convert --repo-id lerobot/pusht --out ./pusht-lance

Migration takes about as long as copying the files. Videos are stored as blobs without re-encoding, so items are bit-identical to the source. For example, 392,578 frames convert in 10 seconds, DROID’s 27.6M in 34 minutes. Nothing about your recording or training code changes. The output keeps meta/ as is and adds three Lance tables (frames, videos, meta). lerobot-lance-doctor audits the result and often finds problems in the source. For example, DROID carries 44% orphaned rows, and three of eight public datasets we converted have frame-count defects that random access exposes. Host the result anywhere: an HF dataset repo (hf upload, tag it v3.0), an HF Storage Bucket (hf buckets sync), or any object store (aws s3 sync).

Load and train

An explicit root works for any object store: LeRobotDataset("lerobot/pusht", root="s3://my-bucket/pusht-lance"). Remote roots download only meta/. Data is fetched per batch. Temporal windows, DataLoader and the rest of the training stack work unchanged: Or from the CLI. Held-out evaluation on a remote dataset needs random access, so it only works on the Lance path:

lerobot-train \
  --dataset.repo_id=my-org/my-bucket/pusht-lance \
  --dataset.repo_type=bucket \
  --dataset.eval_split=0.02 \
  --policy.type=act

Visualize your training data directly in Foxglove

LeRobot’s dataset viewer can serve an episode to Foxglove over its WebSocket protocol (lerobot#4542), and with a Lance dataset it works wherever the data lives:

lerobot-dataset-viz --repo-id lance-format/pusht-lance --episode-index 3 --display-mode foxglove

Open the Foxglove app, connect to ws://localhost:8765, and scrub. Cameras, joint state and actions show up as topics you can lay out however you like: Foxglove loads only the frames you’re viewing, so opening one episode from the 373 GB DROID dataset transfers just a few MB from S3. You don’t need to export to MCAP or download the full dataset. Foxglove reads the same Lance data that training uses.

Query, curate, enrich

The frames table is a LanceDB table like any other: Filter with SQL over any feature, using the indexes built at conversion: From here it is standard LanceDB. The examples below are from our DROID walkthrough, all on the table the trainer reads. Derived columns are added without rewriting existing data. With Geneva, add_columns registers a Python function and its input columns as a new derived column. No values are computed at that point. backfill computes and stores the derived values later, in parallel, without rewriting the existing columns or video blobs:

@udf(data_type=pa.float32(), input_columns=["action_joint_velocity"])
def jerk_score(v):                      # per-frame motion roughness
    ...

tbl.add_columns({"jerk_score": jerk_score})   # nothing computed yet
tbl.backfill("jerk_score", concurrency=8)     # 610,403 rows in 16.7 s

Embeddings work the same way, reading frames straight from the video blobs in the same table:

@udf(data_type=pa.list_(pa.float32(), 768), num_gpus=1,
     input_columns=["episode_index", "frame_index"])
class EmbedFromBlob:                    # SigLIP2 loads once per worker
    def __call__(self, episode_index, frame_index):
        blob = self.videos.take_blobs("video_bytes", indices=[vrow])[0]
        frame = VideoDecoder(blob).get_frames_at([pos]).data[0]
        ...

Search is an index on those columns: a vector index over the embeddings built in 29.7 s across 27.6M rows, a full-text index over the instructions in 7.0 s, queries in 10 to 54 ms. Curation is one query across all of it. Semantic search with a predicate on the derived score without a join:

tbl.search(vec("a gripper closing on an object"), vector_column_name="emb_siglip2") \
   .where("jerk_score > 1.2875")        # the roughest 1% of frames
   .limit(4)

Mining is the same query pointed at the rare cases. tbl.search().where("jerk_score > 1.2875 AND success = false") pulls the rough, failed frames out of 27.6M rows in milliseconds, ready to become a fine-tuning set or a labeling queue. EDA is a scan of the tabular columns: a pass over all 27.6M rows reads 2.52 GB in 190 s and decodes no video. Then drop what you found from training with episodes=[...] instead of rewriting files. The reader ignores columns it does not know, and the table is versioned, so curation and training share one copy of the data at a version you can reopen.

Lance datasets are read-only in LeRobot: record and edit in the default format, then convert.

You can try the examples above with lance-format/pusht-lance and other datasets in the lance-format organization.

Get started

Model training

Guides

Feature Engineering (Geneva)

Support

LeRobot Native Integration

Install

Convert a dataset

Load and train

Visualize your training data directly in Foxglove

Query, curate, enrich

​Install

​Convert a dataset

​Load and train

​Visualize your training data directly in Foxglove

​Query, curate, enrich

Install

Convert a dataset

Load and train

Visualize your training data directly in Foxglove

Query, curate, enrich