Skip to main content
Enterprise only
dedupe scans a vector column’s index partitions for rows closer than a distance threshold, resolves the overlapping pairs it finds into duplicate groups, and returns one representative row per group — without a full pairwise scan or a shuffle. See duplicate_pairs to inspect the pairs themselves before committing to this resolution.
Earlier releases spelled this vector_dedup; that name is rejected on current deployments.

Input

dedupe always scans every partition and returns original rows — sampling a subset of partitions and id_column are pair-query-only options, accepted by duplicate_pairs but refused here.

Index and snapshot requirements

Use one compatible, current vector index covering the source snapshot. Full scans reject unindexed fragments. Comparisons stay within physical IVF partitions and index segments; this does not enumerate cross-partition or cross-segment duplicates. Quantized-index distances use encoded vector representations. See recall and scale limits before treating the result as globally exact deduplication.

Output

One row per duplicate group, in the table’s own schema — every column of <table>, including a BlobV2 column, comes through unchanged on the representative row Lance selected for that group. Unlike duplicate_pairs, there is no fixed dedupe-specific schema to document: the output schema is the source table’s schema. Because the result is already shaped like the table, it composes directly into a new table or a materialized view:

Resolution

resolution chooses how overlapping pairs group together:

Estimating on a sample first

dedupe itself always scans every partition. To estimate a duplicate rate on a large table before committing to a full scan, sample partitions with duplicate_pairs instead, which accepts selection, count and seed for exactly this.

What to read next

DUPLICATE_PAIRS

Inspect the close pairs themselves instead of the resolved groups.

Deduplicate near-duplicates

When to reach for this instead of exact-key deduplication.