---
name: lancedb
description: Use when building vector search applications, managing multimodal
  data, creating RAG systems, performing semantic search, or working with
  embeddings. LanceDB works with both embedded local databases and enterprise
  deployments for production AI/ML workloads.
metadata:
  mintlify-proj: lancedb
  version: "1.0"
---

# LanceDB Skill

## Product Summary

LanceDB is a vector database for AI applications that stores and searches embeddings alongside structured data and multimodal content (images, audio, video). It runs embedded in your application (local disk or object storage) or on an enterprise cluster. Use it to build semantic search, RAG systems, and feature engineering pipelines. Key entry points: `lancedb.connect(uri)` for local/cloud databases, `db.create_table()` to ingest data, `table.search(vector)` for vector queries, and `db.execute_query()` for SQL on enterprise deployments. SDKs available for Python, TypeScript, and Rust. Primary docs: https://docs.lancedb.com

## When to Use

Reach for this skill when:
- Building vector search or semantic search features (search by meaning, not keywords)
- Creating RAG (Retrieval-Augmented Generation) applications that need to retrieve context
- Storing and querying embeddings from OpenAI, Cohere, HuggingFace, or other embedding models
- Managing multimodal data (text + images + audio) in a single versioned table
- Performing feature engineering with computed columns or materialized views
- Needing to index large datasets (1M+ rows) for fast similarity search
- Building production AI applications that require distributed query execution
- Combining vector search with metadata filtering or full-text search

## Quick Reference

### Connection Patterns

| Use Case | Code |
|----------|------|
| Local embedded database | `db = lancedb.connect("./data")` |
| Object storage (S3/GCS/Azure) | `db = lancedb.connect("s3://bucket/path")` |
| Enterprise deployment | `db = lancedb.connect("db://database", api_key="...", host_override="...")` |

### Core Operations

| Task | Python | TypeScript | Rust |
|------|--------|-----------|------|
| Create table | `db.create_table("name", data)` | `await db.createTable("name", data)` | `db.create_table("name", reader).execute().await` |
| Open table | `db.open_table("name")` | `await db.openTable("name")` | `db.open_table("name").execute().await` |
| Vector search | `table.search(vector).limit(k).to_list()` | `await table.search(vector).limit(k).toArray()` | `table.vector_search(&vec)?.limit(k).execute().await` |
| Add rows | `table.add(data)` | `await table.add(data)` | `table.add(reader).execute().await` |
| Update rows | `table.update(where="id=1", values={...})` | `await table.update({where: "id=1", values: {...}})` | `table.update().only_if("id=1").column("col", "val").execute().await` |
| Delete rows | `table.delete("id=1")` | `await table.delete("id=1")` | `table.delete("id=1").await` |
| SQL query (Enterprise) | `db.execute_query("SELECT ...").read_all()` | Use Flight SQL client | `db.execute_query_async("SELECT ...").await` |

### Vector Search Configuration

| Parameter | Purpose | Example |
|-----------|---------|---------|
| `distance_type` | Metric for similarity (l2, cosine, dot, hamming) | `.distance_type("cosine")` |
| `limit` | Number of results to return | `.limit(10)` |
| `where` | Metadata filter (prefilter by default) | `.where("category='news'")` |
| `nprobes` | Partitions to search in IVF index (leave unset for Auto) | `.nprobes(20)` |
| `refine_factor` | Rerank candidates on full vectors | `.refine_factor(10)` |
| `select` | Columns to return | `.select(["id", "text", "_distance"])` |

### Indexing

| Index Type | When to Use | Compression |
|------------|------------|-------------|
| `IVF_PQ` | General-purpose, good accuracy at small dims (≤256) | 1/64 of raw size |
| `IVF_RQ` | Maximum compression, high-dim vectors | 1/32 with num_bits=1 |
| `IVF_HNSW_SQ` | Best recall/latency trade-off | ~1/4 of raw size |
| `IVF_FLAT` | Binary vectors (hamming distance) | Raw size + overhead |

Create index: `table.create_index("vector_col", config=IvfPq(num_partitions=16))`

### Data Types

| Category | Types |
|----------|-------|
| Vectors | `fixed_size_list<float32, dim>` (Arrow type) |
| Blobs | `blob()` for images, audio, video (v2) or `large_binary()` with metadata (v1) |
| Scalars | int8-int64, float16/32/64, utf8, binary, date, timestamp |
| Nested | struct, list, large_list, map |

## Decision Guidance

### When to Use Vector Index vs Brute-Force Search

| Condition | Decision |
|-----------|----------|
| Table < 100K rows | Skip index; brute-force is fast enough |
| Table 100K-1M rows | Measure latency; index if queries exceed 100ms |
| Table > 1M rows | Build index; brute-force becomes prohibitive |
| High-dimensional vectors (>1000 dims) | Build index earlier; brute-force cost grows with dimension |
| Frequent metadata filters | Use IVF_PQ or IVF_RQ; HNSW has higher latency variance under filters |

### When to Use Prefilter vs Postfilter

| Scenario | Approach |
|----------|----------|
| Want nearest neighbors *within* matching subset | Prefilter (default): `.where("category='news'")` |
| Want top-k results *then* filter | Postfilter: `.where("category='news'", prefilter=False)` |
| Selective filter (few matches) | Postfilter may return fewer results than requested |
| Broad filter (many matches) | Prefilter is more efficient |

### When to Use SQL vs SDK Operations

| Task | Use SQL | Use SDK |
|------|---------|---------|
| Vector search | SDK (simpler) | ✓ |
| Joins across tables | SQL | ✓ |
| Aggregations (GROUP BY, SUM) | SQL | ✓ |
| Window functions | SQL | ✓ |
| Merge insert (upsert by key) | SDK | ✓ |
| Create/drop tables | Both | ✓ |
| Metadata filters + vector search | SDK (prefilter) | ✓ |
| Complex analytical queries | SQL | ✓ |

### Embedded vs Enterprise

| Feature | Embedded | Enterprise |
|---------|----------|-----------|
| Deployment | In-process, local disk or object storage | Managed cluster |
| SQL support | No | Yes (Flight SQL) |
| Distributed queries | No | Yes |
| Automatic indexing | No | Yes (background) |
| Real-time write engine | No | Yes (optional) |
| Materialized views | No | Yes |
| Functions (UDFs) | No | Yes |
| Best for | Development, single-machine workloads | Production, multi-user, large scale |

## Workflow

### 1. Set Up Connection
- Determine deployment type: local directory, object storage, or enterprise
- Load credentials from environment variables (never hardcode)
- Connect: `db = lancedb.connect(uri, api_key=..., region=...)`

### 2. Prepare Data and Schema
- Decide on table schema: column names, types, vector dimensions
- Ensure vector columns are `fixed_size_list<float32, dim>` (Arrow type)
- For embeddings: use embedding model consistently (same model for ingestion and queries)
- For multimodal: use `lancedb.blob()` for images/audio/video

### 3. Create and Ingest Table
- Create from data: `table = db.create_table("name", data=records)`
- Or create empty: `table = db.create_table("name", schema=schema)`
- Append rows: `table.add(new_records)`
- Upsert by key: `table.merge_insert("id").when_matched_update_all().when_not_matched_insert_all().execute(records)`

### 4. Build Indexes (if needed)
- Measure brute-force latency first on representative queries
- For tables > 1M rows or latency > 100ms: `table.create_index("vector_col", config=IvfPq())`
- Wait for build: `table.wait_for_index(["vector_col_idx"])`
- Check status: `table.index_stats("vector_col_idx")`

### 5. Run Searches
- Vector search: `results = table.search(query_vector).limit(10).to_list()`
- With filters: `table.search(query_vector).where("category='news'").limit(10).to_list()`
- Full-text search: `table.search("keyword", query_type="fts").limit(10).to_list()`
- Hybrid: `table.search("query text", query_type="hybrid").limit(10).to_list()`

### 6. Verify and Optimize
- Check query plan: `query.explain_plan()` (shows operations)
- Measure execution: `query.analyze_plan()` (shows timing and I/O)
- Adjust index parameters if recall is low: increase `nprobes` or `refine_factor`
- Monitor index coverage: `table.index_stats()` reports `num_unindexed_rows`

## Common Gotchas

- **Vector dimension mismatch**: Query vector must match table vector dimension exactly. Embedding model change requires table recreation.
- **Distance metric mismatch**: Index built with `cosine` must be queried with `cosine`. Mismatched metric falls back to brute-force scan.
- **Unindexed rows after append**: New rows added after index build are not indexed until refresh. Normal search checks them with fallback; `fast_search()` skips fallback.
- **Embedding model consistency**: Use same embedding model for both document and query vectors. Different models produce incomparable embeddings.
- **Null vectors in search**: Queries with null vectors or all-null columns fail. Ensure embeddings are generated for all rows.
- **Large batch ingestion**: For datasets > 1GB, use batches or file-backed input to avoid memory overflow. See `ingestion/insert` for large-load patterns.
- **Shallow clone dependency**: Shallow clones reference source data files. If source is compacted past retention, clone becomes invalid.
- **SQL only on Enterprise**: `db.execute_query()` fails on embedded databases. Use SDK operations instead.
- **Blob v2 compatibility**: Managed blobs require upgraded readers/writers. Older clients cannot safely maintain tables with managed blobs.
- **Prefilter vs postfilter confusion**: Prefilter (default) returns k nearest *within* filter; postfilter returns k nearest *then* filters (may return < k results).

## Verification Checklist

Before submitting work with LanceDB:

- [ ] Connection credentials loaded from environment, not hardcoded
- [ ] Vector dimension matches embedding model output (e.g., OpenAI ada-002 = 1536)
- [ ] Distance metric specified at index build and query time matches (cosine, l2, dot, hamming)
- [ ] Embedding model is consistent across ingestion and queries
- [ ] Table schema includes required fields and correct nullability
- [ ] For large tables (>1M rows): index built and `index_stats()` shows `num_unindexed_rows == 0`
- [ ] Query plan inspected with `explain_plan()` to confirm index usage
- [ ] Recall measured on representative queries (compare indexed vs brute-force with `bypass_vector_index()`)
- [ ] Batch size appropriate for memory constraints (test with `to_batches()`)
- [ ] Multimodal data (blobs) fetched with correct `blob_mode` (bytes, descriptions, or lazy)
- [ ] SQL queries tested on Enterprise deployment (not embedded)
- [ ] Materialized views refreshed after source table updates (Enterprise only)

## Resources

**Comprehensive navigation:** https://docs.lancedb.com/llms.txt

**Critical pages:**
- [Quickstart](https://docs.lancedb.com/quickstart) — Create table, add data, run first search
- [Vector Search](https://docs.lancedb.com/search/vector-search) — Distance metrics, indexing, tuning nprobes
- [Ingestion](https://docs.lancedb.com/ingestion/index) — Create, insert, update, delete, merge operations

---

> For additional documentation and navigation, see: https://docs.lancedb.com/llms.txt