Skip to main content
Use add() to append records to an existing table. It does not compare incoming keys with stored keys. Choose merge insert when the same record may arrive again and should update an existing row.

Append a first batch

Install the Python SDK with pip install lancedb. This complete example starts with Alice, appends Bob and Charlie, and prints all three records.
append_rows.py
The program uses temporary local storage. For a complete TypeScript or Rust example, see Complete write workflow. Enterprise SQL offers INSERT, including INSERT ... SELECT.

Prepare the input

Match the table’s schema, including required fields and vector dimensions. Supported input formats differ by SDK; see Create a table. Inspect conversion errors before retrying, and verify the final row count and representative values.

Choose input for a large load

Avoid building one large in-memory list when records already live in files or arrive as batches. Create a table with the intended schema, then append through the appropriate input path.

Read from files

Python accepts a pyarrow.dataset.Dataset. This example writes a small Parquet file locally, creates an empty table with its schema, and appends the two records. For your own data, point the dataset reader at the input files instead.
load_parquet.py
A file-backed source can be scanned without first loading the whole dataset into application memory. This Python input API is distinct from converting files to Arrow batches yourself in another language.

Produce batches during ingestion

Use batches when processing the input as it arrives. The Python example creates an empty table and appends ten records produced by a batch iterator.
The language APIs differ: the TypeScript example appends each constructed Arrow batch, while the Rust example supplies an Arrow reader when creating the table. These are input-format examples, not equivalent parallel-write benchmarks.

Control write parallelism

Python’s add() can estimate parallelism for materialized inputs. A streaming reader may not expose its total size or support replay. Use write_parallelism when controlling the number of concurrent partitions; increasing it can also increase memory use. TypeScript does not expose that manual option. Measure with realistic row sizes and storage. Avoid one write per row when a batch will do, and avoid a large worker count for a small input.

Track completion and progress

Python, TypeScript and Rust support progress callbacks when adding data. These examples start with Alice, append Bob and Charlie, and print the number of rows written as progress updates arrive. After the write succeeds, they query the table to show all three records. Each program uses a temporary local database and removes it when finished.
  • Python: install lancedb, save the code as insert_progress.py, and run python insert_progress.py.
  • TypeScript: install @lancedb/lancedb, save the code as insert_progress.mts, and run node insert_progress.mts with Node.js 22.18 or later.
  • Rust: download the Rust examples, extract the project, and run cargo run --locked --bin progress.
For a progress bar in Python, install tqdm and pass progress=True instead of a callback.

Read progress updates

Updates include row and byte counts, elapsed time, and parallel write activity. The field names differ by language: Python and TypeScript report elapsed time in seconds; Rust returns a Duration. The expected row count may be unknown for streaming input. On the final update, it falls back to the number of rows written. For remote multipart writes, both rows and IPC bytes count acknowledged parts. Both start at zero; a retried part is counted once, after acknowledgement. Scanning input or sending a rejected part does not advance these counts. Acknowledged parts are staged and do not imply a successful commit. Keep callbacks short and use the write operation’s result to determine success. The final done update means progress reporting has ended; it can also occur after a write fails or is cancelled. Input errors before writing starts may produce no updates. TypeScript delivers callbacks asynchronously. Slow callbacks can queue up, and the add() promise may resolve before every callback has run. Exceptions from a TypeScript callback are logged and do not abort the write.

Verify writes and handle retries

Wait for the write operation’s result, then check the expected records. Progress callbacks describe activity; they do not establish that a write succeeded. If a request fails after the server may have committed it, blindly repeating an append can duplicate rows. Current clients do not automatically replay non-idempotent writes after 500, 502, 503, or 504, or after a response is lost. A plain 503 does not prove that a write was rejected before commit. Connection failures before sending, 409 transaction conflicts, and explicit 429 rate-limit rejections may be retried within RetryConfig limits. Idempotent multipart parts have a separate retry policy described below. If multipart completion times out, loses its response, or receives a 5xx, the error reports commit outcome unknown. The client neither aborts the upload nor resends completion or the input under a new upload ID: aborting at this point could delete already committed data. A request ID helps trace the attempt. It does not make an append safe to replay. Keep the batch pending until you know the original attempt’s terminal result or can prevent it from committing later. An immediate read with missing IDs does not establish that boundary. Then, with writes for those IDs serialized, check a fresh snapshot against the expected record IDs and Blob content. Retry only confirmed missing rows and stop on duplicate IDs or conflicting content. Configure connection-wide limits with client_config.retry_config in Python (lancedb.remote.RetryConfig) or clientConfig.retryConfig in TypeScript. Override the budget for one add() with Python table.add(data, retries=0) or TypeScript table.add(data, { retries: 0 }). The value must be an integer between 0 and 255. Rust write builders also provide a per-call override, for example table.add(data).retries(0).execute().await?, table.update().retries(0), and table.merge_insert(&["id"]).retries(0). This disables transport retries for that operation, including multipart part retries; it does not change local transaction conflict handling. Increasing a retry budget never makes ambiguous commits safe to replay.

Configure large uploads

Enterprise This section applies only to LanceDB Enterprise tables (db:// connections). LanceDB Embedded writes are unaffected. Large measured inputs, unknown-length streams, and explicit parallel uploads use multipart writes when the server supports them. The client slices large batches before IPC encoding and buffers each part for replay. A disconnected request, read timeout, or configured retryable staging status retries exactly the same upload ID, part ID, and IPC bytes. RetryConfig.statuses controls status retries; an empty list disables them. The default includes 500, 502, 503, and 504. The server replaces that part’s staged transaction, so losing its response does not append the rows twice. Exhausting the part retry budget aborts the uncommitted session; it does not restart the entire upload. A complete request rejected with a transaction conflict (409) or rate limit (429) also triggers best-effort cleanup. Cleanup uses the call’s retry budget; its failure does not replace the original error. Ambiguous completion outcomes never trigger cleanup or replay. Part size still affects memory and HTTP timeouts. A single blob row cannot be split across Arrow batches; a very large row may require a larger timeout and more memory. Size caller batches by Blob payload bytes and allow for Arrow encoding overhead and live buffers. Blob storage packing thresholds are separate from upload budgets. No connection-pool or keepalive setting guarantees that a write response will arrive. Two environment variables control how a multipart write’s parts are cut. Both are picked up automatically by the Python and TypeScript clients, for clients that support these settings: A part is cut after an encoding slice reaches either limit. Input batches are sliced to approximately 8 MiB before encoding, preserving row order. Peak additional memory scales with active writers and the part budget. Lower write_parallelism or the byte budget to reduce memory use.
Splitting into more parts does not change the final table. The server stages the parts under their shared upload ID before committing the write. This upload protocol is separate from repeating an application-level add() after a timeout. Remote create_table(name, data) remains one atomic request, including overwrite and exist_ok modes. The client encodes into an automatically removed temporary file before sending, rejects payloads larger than 256 MiB of compressed IPC, and never silently creates an empty table as a fallback. Encoding and input failures therefore issue no create request. Temporary disk usage is bounded by this budget; individual oversized rows can still need substantial encoding memory. For larger imports, create the schema first and upload with progress reporting. The empty table becomes visible immediately; add() commits its rows atomically. The following examples use an existing connection db, an Arrow schema, and input data matching that schema (an Arrow reader in Rust):

Complete write workflow

Each program creates a temporary local database, performs the five operations, prints the result, and removes its temporary data when finished.
  • Python: install lancedb, save the code as ingestion.py, and run python ingestion.py.
  • TypeScript: install @lancedb/lancedb, save the code as ingestion.mts, and run node ingestion.mts with Node.js 22.18 or later.
  • Rust: download the complete Rust project, extract it, and run cargo run --locked. It includes src/main.rs, the Cargo manifest and a lockfile for the example’s dependencies. The code is shown below.
After the operations, Alice has 10 logins, Bobby has 21, and Charlie has 6. Dana is selected for deletion and is absent from the final result.