Permutation

Applies to: Embedded ✅, Enterprise ✅ Most model training frameworks feed data into a model by sequencing through successive batches, a process often referred to as data loading. The simplest way to perform data loading is to read a LanceDB table in batches, in the order its rows are stored, through an identity Permutation. A table itself is not iterable.

from lancedb.permutation import Permutation

db = lancedb.connect("file://some/db/path")
table = db.open_table("some_table")

for batch in Permutation.identity(table).with_format("arrow").iter(batch_size=1024):
    print(batch.to_pydict())

In specific use cases, you may need the flexibility to shuffle, split, and select data without using the full StreamingDataset object. In these cases, use Permutation, the lower-level functionality StreamingDataset is built on. A Permutation defines a customizable row ordering, and supports map-style access through __getitem__() and batched access through __getitems__().

Version Pinning

A Permutation is pinned to the version of its base table at the time it was constructed, and every read (including future reads from a StreamingDataset that wraps it, and every DataLoader worker after it starts) resolves against that pinned version. This makes iteration deterministic across compactions and worker restarts: the permutation only addresses rows that existed when it was built, and rows appended to the base table afterwards are not visible through the existing permutation. To include newly appended rows, build a new Permutation (and any StreamingDataset that depends on it) against the updated table. Permutations written before this behavior shipped continue to read as they did before.

Output formats

By default, a Table data loader emits Arrow data. collate_fn is PyTorch’s batching hook: PyTorch calls it to turn the fetched items into one batch. PyTorch’s default collate function only knows how to combine tensors, NumPy arrays, numbers, dicts, and lists, so it does not accept Arrow data directly. When using a Table directly, pass LanceDB’s lancedb.util.tbl_to_tensor helper as PyTorch’s collate_fn; it converts numeric Arrow columns into a column-major torch.Tensor with shape (columns, rows). Permutation works differently: its default output is a list of Python dicts, which PyTorch’s default collate function can batch into a dict of tensors. This is usually more convenient when you are getting started. However, there is a significant performance penalty converting from Arrow, Lance’s internal representation, to this default format. If you want the default PyTorch dict-of-tensors behavior, use a Permutation as-is; if you want direct Arrow-to-tensor conversion, either pass lancedb.util.tbl_to_tensor as collate_fn with a direct Table or configure a Permutation with one of the transform formats described below. To address this, the Permutation class provides a set of builtin transform functions that can be applied to map the Arrow data in different ways. The arrow and polars formats will always avoid data copies. However, numpy, pandas, and torch_col formats will also avoid data copies in most cases. The python, python_col, and torch formats will all require at least one full copy of the data and are the slowest options.

Using the torch column format with a torch data loader

The torch_col format is the most efficient way to convert from Arrow to a torch.Tensor. It will convert the entire Arrow batch to a column-major torch.Tensor. In other words, given C columns and R rows, the resulting Tensor will have shape (C, R). However, this format generates an error if you are using a torch.utils.data.DataLoader with the default collation function:

TypeError: stack(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor

This error occurs because the default collation function does not currently expect a single two-dimensional tensor. It expects a list of tensors which it will then stack. This is what is output by the torch format but that format requires a data copy. To avoid this error, and avoid data copies, you will need to provide a custom collation function in addition to specifying the torch_col format.

from lancedb.permutation import Permutation

permutation = Permutation.identity(table).with_format("torch_col")
dataloader = torch.utils.data.DataLoader(permutation, collate_fn=lambda x: x)

This will now output a single two-dimensional tensor for each batch.

Selecting columns

By default, the Table class will return all columns in the table when used as input to PyTorch. If you only need a subset of columns, you can significantly reduce your I/O requirements by selecting only the columns you need. The Permutation class provides a select_columns method that provides this functionality.

import pyarrow as pa
import torch
from lancedb.permutation import Permutation

db = lancedb.connect("memory://")
prompts = db.create_table(
    "prompts",
    pa.table(
        {
            "id": range(4),
            "prompt": ["north", "south", "east", "west"],
            "embedding": [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8]],
        }
    ),
)

permutation = Permutation.identity(prompts).select_columns(["id", "prompt"])
dataloader = torch.utils.data.DataLoader(
    permutation, batch_size=1024, shuffle=True
)

for batch in dataloader:
    print(sorted(batch))  # ['id', 'prompt']: the embedding is never read

Each batch is a dict of the selected columns, the default Permutation format described above.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Version Pinning

Output formats

Using the torch column format with a torch data loader

Selecting columns

​Version Pinning

​Output formats

​Using the torch column format with a torch data loader

​Selecting columns

Version Pinning

Output formats

Using the torch column format with a torch data loader

Selecting columns