Scope curation to what changed

Applies to: Embedded ❌, Enterprise ✅ A curation task you run repeatedly — chunking, deduplication, profiling — does not need to rescan rows it already processed. CHANGES reads only the rows appended to a table since an earlier version, so a recurring job can scope itself to what is new.

Example data

curation_batches gets a batch of new documents appended periodically, with no updates or deletes between appends:

CREATE TABLE curation_batches (doc_id BIGINT, source VARCHAR, text VARCHAR);
INSERT INTO curation_batches VALUES
    (1, 'web', 'A cat on a chair'),
    (2, 'web', 'A dog in a field');
-- note the version after this insert, for example 5
INSERT INTO curation_batches VALUES
    (3, 'books', 'A bird in a tree'),
    (4, 'books', 'A cat by a window');
-- the table is now at a later version, for example 7

Read only what is new

CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5) reads the rows appended after version 5 — documents 3 and 4 — without rescanning 1 and 2:

SELECT * FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5)

Record the table’s version after each curation run, then start the next run’s CHANGES from it. See Versioning for the APIs that read a table’s current version.

Chain it into a curation task

CHANGES is a relation, like any other table reference, so it composes with the rest of this page: chunk, dedupe, or join only the new rows, instead of the whole table. Materializing the result keeps a running, append-only table of work already done:

INSERT INTO curated_batches
SELECT doc_id, source, text
FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5)
WHERE source = 'books'

A chunker Function or a dedupe call works the same way: put the CHANGES relation where the source table would otherwise go, and only the new rows flow through it.

Strictly append-only

The clause is named for what it requires, not just what it does. If anything other than a pure append happened to the table between the starting version and the current one — an update, a delete, a merge — the query is refused rather than silently returning an incomplete or wrong result:

-- refused if any row in curation_batches was updated or deleted after version 5
SELECT * FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5)

When that happens, either re-run from a later starting version that is itself after the non-append write, or read the table in full instead of incrementally for that run. See Changes since a version for the complete set of refused forms.

CHANGES

The full SQL syntax, placement rules, and unsupported forms.

Table

How table versions work, and how to find one to start from.

Introduction

Getting Started

Concepts

Data Ingestion

Search

Data Curation

Feature Engineering

Training

Management

Maintenance

Security

Deployment

SQL Reference

Scope curation to what changed

Example data

Read only what is new

Chain it into a curation task

Strictly append-only

What to read next

CHANGES

Table

​Example data

​Read only what is new

​Chain it into a curation task

​Strictly append-only

​What to read next

CHANGES

Table

Example data

Read only what is new

Chain it into a curation task

Strictly append-only

What to read next