Data curation

Data curation turns a collection of records into a dataset suited to a task. You might select labeled examples for training, prepare a fixed evaluation set, or remove duplicate documents from a search corpus. Start with the result you need. Then inspect the source data, choose which records belong, and save the selected data or the rules used to produce it.

Choose a curation task

Task	What it helps you decide
Create a Function	How to run your own Python — on CPU, on GPU, or against a remote service — as part of SQL.
Vend a Secret to a Function	How a bound Secret’s value actually reaches a running Function.
Inspect Datasets	How to query a table, a HuggingFace dataset, or an object-storage dataset the same way, and import without a separate download step.
Chunk records	How to split one record into several — passages, clips, or tiles.
Scope to what changed	How to process only the rows added since your last run.
Profile a dataset	Whether coverage, missing values and class balance suit the task.
Statistical Sampling	Which random fraction to inspect or use for an experiment.
Selective Sampling	How to re-fetch specific rows by ID, address or offset.
Join datasets	How to attach labels, metadata or scores stored elsewhere.
Rank and deduplicate	Which records to keep when several share an explicit key.
Cluster by index partition	How to group records by vector proximity without training a new clustering model.
Deduplicate near-duplicates	Which records to collapse when they only look alike, with no shared key.

The analytical examples in this section use Enterprise SQL. Connect with the SQL guide. For a random subset through the Python SDK in LanceDB Embedded or LanceDB Enterprise, see Permutation.

Prepare a labeled document dataset

Suppose you need one labeled record per document for a training experiment. Inspect the source records, choose the latest version of each document, then join its label. Finally, check the class balance of the selected records.

Example data

Connect with an Enterprise SQL client. The examples use a small document table and a separate label table. Expand the setup below to create them.

Create the example tables

curation_documents has six records describing four documents. curation_labels has one label for each of three documents. row_id identifies a record; doc_id identifies the document it describes.Run each statement separately in a scratch database or namespace. Create these tables once for the whole section, using fresh names if they already exist.

CREATE TABLE curation_documents (
    row_id BIGINT, doc_id BIGINT, version BIGINT,
    source VARCHAR, caption VARCHAR, score DOUBLE
);
INSERT INTO curation_documents VALUES
    (1, 10, 1, 'web', NULL, 0.80),
    (2, 10, 2, 'web', 'A cat on a chair', 0.95),
    (3, 20, 1, 'books', 'A dog in a field', 0.90),
    (4, 30, 1, 'web', NULL, 0.70),
    (5, 30, 1, 'web', 'A cat by a window', 0.85),
    (6, 40, 1, 'books', 'A bird in a tree', 0.60);

CREATE TABLE curation_labels (doc_id BIGINT, label VARCHAR);
INSERT INTO curation_labels VALUES
    (10, 'cat'), (20, 'dog'), (30, 'cat');

score is a sample quality score, not a search result score. Document 10 has two versions. Document 30 has two records at the same version. Document 40 has no label. These cases let you check the effect of each operation.

The following query chooses the latest record for each document, then keeps only documents with a label:

WITH ranked AS (
    SELECT *, row_number() OVER (
        PARTITION BY doc_id ORDER BY version DESC, row_id DESC
    ) AS rn
    FROM curation_documents
)
SELECT d.row_id, d.doc_id, d.caption, l.label
FROM ranked AS d
JOIN curation_labels AS l ON l.doc_id = d.doc_id
WHERE d.rn = 1
ORDER BY d.doc_id;

The result contains documents 10, 20 and 30, with labels cat, dog and cat. Document 40 is excluded because this particular task requires a label. The row_id tie-breaker makes the choice for document 30 deterministic; choose a tie-breaker that fits your own data. The example assumes one label per document. Duplicate keys in the label table would multiply the result rows. The joining guide shows how to check that assumption.

Check the selected dataset

Compare the result with the task’s requirements: are the expected documents present, are required fields populated, and is the class distribution suitable? This example has two cats and one dog. That distribution may be appropriate for exploration, but it does not establish a representative evaluation set. Profiling provides the counts to inspect. Statistical Sampling and ranking help you adjust the selection without changing the original records.

Save and reuse the result

Choose the output according to how you will use it:

A fixed dataset: save the query result in a new table with CREATE TABLE AS SELECT. Keep that output unchanged when comparing experiments.
A reusable selection: save the query as a view. A view stores the query definition, so it is not a frozen evaluation set.
Stored results that you refresh: use a materialized view when its supported query and refresh behavior fit the workflow.

Record the source versions and selection rules alongside the output. Continue with data loading to use the curated data in a training pipeline.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Data curation

Choose a curation task

Prepare a labeled document dataset

Example data

Check the selected dataset

Save and reuse the result

​Choose a curation task

​Prepare a labeled document dataset

​Example data

​Check the selected dataset

​Save and reuse the result

Choose a curation task

Prepare a labeled document dataset

Example data

Check the selected dataset

Save and reuse the result