add() to append records to an existing table. It does not compare incoming
keys with stored keys. Choose merge insert when the
same record may arrive again and should update an existing row.
Append a first batch
Install the Python SDK withpip install lancedb. This complete example starts
with Alice, appends Bob and Charlie, and prints all three records.
append_rows.py
INSERT, including INSERT ... SELECT.
Prepare the input
Match the table’s schema, including required fields and vector dimensions. Supported input formats differ by SDK; see Create a table. Inspect conversion errors before retrying, and verify the final row count and representative values.Choose input for a large load
Avoid building one large in-memory list when records already live in files or arrive as batches. Create a table with the intended schema, then append through the appropriate input path.Read from files
Python accepts apyarrow.dataset.Dataset. This example writes a small Parquet
file locally, creates an empty table with its schema, and appends the two records.
For your own data, point the dataset reader at the input files instead.
load_parquet.py
Produce batches during ingestion
Use batches when processing the input as it arrives. The Python example creates an empty table and appends ten records produced by a batch iterator.Control write parallelism
Python’sadd() can estimate parallelism for materialized inputs. A streaming
reader may not expose its total size or support replay. Use write_parallelism
when controlling the number of concurrent partitions; increasing it can also
increase memory use. TypeScript does not expose that manual option.
Measure with realistic row sizes and storage. Avoid one write per row when a
batch will do, and avoid a large worker count for a small input.
Track completion and progress
Python, TypeScript and Rust support progress callbacks when adding data. These examples start with Alice, append Bob and Charlie, and print the number of rows written as progress updates arrive. After the write succeeds, they query the table to show all three records. Each program uses a temporary local database and removes it when finished.- Python: install
lancedb, save the code asinsert_progress.py, and runpython insert_progress.py. - TypeScript: install
@lancedb/lancedb, save the code asinsert_progress.mts, and runnode insert_progress.mtswith Node.js 22.18 or later. - Rust: download the Rust examples,
extract the project, and run
cargo run --locked --bin progress.
tqdm and pass progress=True instead
of a callback.
Read progress updates
Updates include row and byte counts, elapsed time, and parallel write activity. The field names differ by language:
Python and TypeScript report elapsed time in seconds; Rust returns a
Duration.
The expected row count may be unknown for streaming input. On the final update,
it falls back to the number of rows written.
For remote multipart writes, both rows and IPC bytes count acknowledged parts.
Both start at zero; a retried part is counted once, after acknowledgement.
Scanning input or sending a rejected part does not advance these counts.
Acknowledged parts are staged and do not imply a successful commit.
Keep callbacks short and use the write operation’s result to determine success.
The final done update means progress reporting has ended; it can also occur
after a write fails or is cancelled. Input errors before writing starts may
produce no updates.
TypeScript delivers callbacks asynchronously. Slow callbacks can queue up,
and the add() promise may resolve before every callback has run. Exceptions
from a TypeScript callback are logged and do not abort the write.
Verify writes and handle retries
Wait for the write operation’s result, then check the expected records. Progress callbacks describe activity; they do not establish that a write succeeded. If a request fails after the server may have committed it, blindly repeating an append can duplicate rows. Current clients do not automatically replay non-idempotent writes after500, 502, 503, or 504, or after a
response is lost. A plain 503 does not prove that a write was rejected before
commit. Connection failures before sending, 409 transaction conflicts, and
explicit 429 rate-limit rejections may be retried within RetryConfig limits. Idempotent multipart
parts have a separate retry policy described below.
If multipart completion times out, loses its response, or receives a 5xx, the
error reports commit outcome unknown. The client neither aborts the upload
nor resends completion or the input under
a new upload ID: aborting at this point could delete already committed data.
A request ID helps trace the attempt. It does not make an append safe to replay.
Keep the batch pending until you know the original attempt’s terminal result or
can prevent it from committing later. An immediate read with missing IDs does
not establish that boundary. Then, with writes for those IDs serialized, check a
fresh snapshot against the expected record IDs and Blob content. Retry only
confirmed missing rows and stop on duplicate IDs or conflicting content.
Configure connection-wide limits with client_config.retry_config in Python
(lancedb.remote.RetryConfig) or clientConfig.retryConfig in TypeScript.
Override the budget for one add() with Python table.add(data, retries=0)
or TypeScript table.add(data, { retries: 0 }). The value must be an integer
between 0 and 255. Rust write builders also provide a per-call override, for
example table.add(data).retries(0).execute().await?,
table.update().retries(0), and table.merge_insert(&["id"]).retries(0).
This disables transport retries for that operation, including multipart part
retries; it does not change local transaction conflict handling. Increasing a
retry budget never makes ambiguous commits safe to replay.
Configure large uploads
Enterprise This section applies only to LanceDB Enterprise tables (db:// connections). LanceDB Embedded writes are unaffected.
Large measured inputs, unknown-length streams, and explicit parallel uploads use
multipart writes when the server supports them. The client slices large batches
before IPC encoding and buffers each part for replay. A disconnected request,
read timeout, or configured retryable staging status retries exactly the same
upload ID, part ID, and IPC bytes. RetryConfig.statuses controls status retries;
an empty list disables them. The default includes 500, 502, 503, and 504.
The server replaces that part’s staged transaction, so losing
its response does not append the rows twice. Exhausting the part retry budget
aborts the uncommitted session; it does not restart the entire upload.
A complete request rejected with a transaction conflict (409) or rate limit
(429) also triggers best-effort cleanup. Cleanup uses the call’s retry budget;
its failure does not replace the original error. Ambiguous completion outcomes
never trigger cleanup or replay.
Part size still affects memory and HTTP timeouts. A single blob row cannot be
split across Arrow batches; a very large row may require a larger timeout and
more memory. Size caller batches by Blob payload bytes and allow for Arrow
encoding overhead and live buffers. Blob storage packing thresholds are separate
from upload budgets. No connection-pool or keepalive setting guarantees that a
write response will arrive.
Two environment variables control how a multipart write’s parts are cut. Both are picked up automatically by the Python and TypeScript clients, for clients that support these settings:
A part is cut after an encoding slice reaches either limit. Input batches are
sliced to approximately 8 MiB before encoding, preserving row order. Peak
additional memory scales with active writers and the part budget. Lower
write_parallelism or the byte budget to reduce memory use.
add() after a timeout.
Remote create_table(name, data) remains one atomic request, including overwrite
and exist_ok modes. The client encodes into an automatically removed temporary
file before sending, rejects payloads larger than 256 MiB of compressed IPC, and
never silently creates an empty table as a fallback. Encoding and input failures
therefore issue no create request. Temporary disk usage is bounded by this budget;
individual oversized rows can still need substantial encoding memory.
For larger imports, create the schema first and upload with progress reporting.
The empty table becomes visible immediately; add() commits its rows atomically.
The following examples use an existing connection db, an Arrow schema, and
input data matching that schema (an Arrow reader in Rust):
Complete write workflow
Complete examples: create, append, update, upsert and delete
Complete examples: create, append, update, upsert and delete
Each program creates a temporary local database, performs the five operations,
prints the result, and removes its temporary data when finished.After the operations, Alice has 10 logins, Bobby has 21, and Charlie has 6.
Dana is selected for deletion and is absent from the final result.
- Python: install
lancedb, save the code asingestion.py, and runpython ingestion.py. - TypeScript: install
@lancedb/lancedb, save the code asingestion.mts, and runnode ingestion.mtswith Node.js 22.18 or later. - Rust: download the complete Rust project,
extract it, and run
cargo run --locked. It includessrc/main.rs, the Cargo manifest and a lockfile for the example’s dependencies. The code is shown below.