Pydantic - LanceDB

Pydantic is a data validation library in Python. LanceDB integrates with Pydantic for schema inference, data ingestion, and query result casting. Using lancedb.pydantic.LanceModel, users can seamlessly integrate Pydantic with the rest of the LanceDB APIs. First, import the necessary LanceDB and Pydantic modules:

Python

import tempfile
from pathlib import Path

import lancedb
from lancedb.pydantic import LanceModel, Vector

Next, define your Pydantic model by inheriting from LanceModel and specifying your fields including a vector field:

Python

class LanceDocs(LanceModel):
text: str
vector: Vector(2)

Set the database connection URL:

Python

db = lancedb.connect(str(Path(tempfile.mkdtemp()) / "pydantic-docs"))

Now you can create a table, add data, and perform vector search operations:

Python

table = db.create_table("docs", schema=LanceDocs, mode="overwrite")
table.add(
[
{"text": "hello world", "vector": [1.0, 0.0]},
{"text": "goodbye world", "vector": [0.0, 1.0]},
]
)
results = table.search("hello world").limit(1).to_pydantic(LanceDocs)
print(results[0].text)

Vector Field

LanceDB provides a lancedb.pydantic.Vector method to define a vector Field in a Pydantic Model.

Python

import pyarrow as pa
import pydantic
from lancedb.pydantic import Vector, pydantic_to_schema

class MyModel(pydantic.BaseModel):
id: int
url: str
embeddings: Vector(768)

schema = pydantic_to_schema(MyModel)
assert schema == pa.schema(
[
pa.field("id", pa.int64(), False),
pa.field("url", pa.utf8(), False),
pa.field("embeddings", pa.list_(pa.float32(), 768)),
]
)

This example demonstrates how LanceDB automatically converts Pydantic field types to their corresponding Apache Arrow data types. The pydantic_to_schema() function takes a Pydantic model and generates an Arrow schema where:

int fields become pa.int64() (64-bit integers)
str fields become pa.utf8() (UTF-8 encoded strings)
Vector(768) becomes pa.list_(pa.float32(), 768) (fixed-size list of 768 float32 values)
The False parameter indicates that the fields are not nullable

Type Conversion

LanceDB automatically convert Pydantic fields to Apache Arrow DataType. Current supported type conversions:

Pydantic Field Type	PyArrow Data Type
`int`	`pyarrow.int64`
`float`	`pyarrow.float64`
`bool`	`pyarrow.bool`
`str`	`pyarrow.utf8()`
`list`	`pyarrow.List`
`BaseModel`	`pyarrow.Struct`
`Vector(n)`	`pyarrow.FixedSizeList(float32, n)`

LanceDB supports to create Apache Arrow Schema from a pydantic.BaseModel via lancedb.pydantic.pydantic_to_schema method.

Python

from typing import List, Optional

import pyarrow as pa
import pydantic
from lancedb.pydantic import Vector, pydantic_to_schema

class FooModel(pydantic.BaseModel):
id: int
s: str
vec: Vector(1536)  # fixed_size_list<item: float32>[1536]
li: List[int]

schema = pydantic_to_schema(FooModel)
assert schema == pa.schema(
[
pa.field("id", pa.int64(), False),
pa.field("s", pa.utf8(), False),
pa.field("vec", pa.list_(pa.float32(), 1536)),
pa.field("li", pa.list_(pa.int64()), False),
]
)

This example shows a more complex Pydantic model with various field types and demonstrates how LanceDB handles:

Basic types: int and str fields
Vector fields: Vector(1536) creates a fixed-size list of 1536 float32 values
List fields: List[int] becomes a variable-length list of int64 values
Schema generation: The pydantic_to_schema() function automatically converts all these types to their Arrow equivalents

​Vector Field

​Type Conversion

Vector Field

Type Conversion