DEDUPE

Applies to: Embedded ❌, Enterprise ✅ dedupe scans a vector column’s index partitions for rows closer than a distance threshold, resolves the overlapping pairs it finds into duplicate groups, and returns one representative row per group — without a full pairwise scan or a shuffle. See duplicate_pairs to inspect the pairs themselves before committing to this resolution.

SELECT * FROM dedupe(
    <table>,
    <vector_column>,
    distance_threshold => <float>
    [, resolution => 'connected_components' | 'direct_representative']
)

SELECT * FROM dedupe(
    'frames',
    'vec',
    distance_threshold => 0.02
)

Earlier releases spelled this vector_dedup; that name is rejected on current deployments.

Input

Argument	Type	Required	Meaning
table name	string literal	yes	the table to scan
vector column	string literal	yes	the indexed vector column to compare
`distance_threshold`	float	yes	maximum distance for a pair to count as close
`resolution`	`'connected_components'` (default) \| `'direct_representative'`	no	how overlapping pairs group together — see Resolution

dedupe always scans every partition and returns original rows — sampling a subset of partitions and id_column are pair-query-only options, accepted by duplicate_pairs but refused here.

Output

One row per duplicate group, in the table’s own schema — every column of <table>, including a BlobV2 column, comes through unchanged on the representative row Lance selected for that group. Unlike duplicate_pairs, there is no fixed dedupe-specific schema to document: the output schema is the source table’s schema. Because the result is already shaped like the table, it composes directly into a new table or a materialized view:

CREATE MATERIALIZED VIEW frames_deduped AS
SELECT * FROM dedupe(
    'frames',
    'vec',
    distance_threshold => 0.02
)

Resolution

resolution chooses how overlapping pairs group together:

`resolution` value	How pairs group
`connected_components` (default)	groups transitively: if a~~b and b~~c are both close, all three resolve together
`direct_representative`	only pairs within the threshold of each other resolve together

Estimating on a sample first

dedupe itself always scans every partition. To estimate a duplicate rate on a large table before committing to a full scan, sample partitions with duplicate_pairs instead, which accepts selection, count and seed for exactly this.

DUPLICATE_PAIRS

Inspect the close pairs themselves instead of the resolved groups.

Deduplicate near-duplicates

When to reach for this instead of exact-key deduplication.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Input

Output

Resolution

Estimating on a sample first

What to read next

DUPLICATE_PAIRS

Deduplicate near-duplicates

​Input

​Output

​Resolution

​Estimating on a sample first

​What to read next

DUPLICATE_PAIRS

Deduplicate near-duplicates

Input

Output

Resolution

Estimating on a sample first

What to read next