Skip to main content
Applies to: Embedded ✅, Enterprise ✅

​
What is the real time engine?

Every write to a LanceDB table is normally a commit to the underlying Lance dataset. That is ideal for bulk loads, but a stream of small upserts becomes a stream of small commits, and each write is only visible once its commit lands. The real time engine (RTE) puts an LSM-style write tier in front of the table. It turns many small writes into a few large commits, and makes each write searchable within milliseconds of being acknowledged. It is enabled per table, and nothing about how you query that table changes.

​
The Lance MemWAL format

RTE is built on MemWAL, a log-structured-merge (LSM) format that Lance itself defines — the structure Enterprise serves is part of the open Lance format, not a layer bolted on beside it. A table with RTE enabled is divided into one or more shards (see RTE writes for enabling and sizing them), and each shard holds:
  • A memtable: the active, in-memory table for that shard’s newest rows, queryable as soon as a write lands in it.
  • A WAL buffer: the same writes, batched and flushed to object storage before a write is acknowledged, so an unflushed memtable is never the only copy of a row.
  • Its own in-memory indexes (vector, full-text, BTree, and other scalar types), kept current as rows land, independently of the base table’s indexes.
A write commits in two stages, both before it is acknowledged: the WAL buffer append makes it durable, then it lands in the shard’s memtable. When a memtable crosses a size or row-count threshold, it freezes and flushes as an immutable SSTable — an ordinary Lance data file — and a fresh memtable takes over. RTE Compaction later folds SSTables into the base table, catching its indexes up as it goes.

​
WAL nodes serve MemWAL shards

This section describes LanceDB Enterprise’s serving layer. LanceDB Embedded runs the same MemWAL mechanics directly against the local dataset, in one process — there is no separate node to route to or scale; see RTE writes and RTE reads for what that looks like. RTE’s Enterprise serving layer is a dedicated component, separate from the query nodes and plan executors that serve ordinary queries: A query node routes every request for an RTE-enabled table — reads and writes alike — to the WAL node that owns that table, by a consistent hash of the database and table name. WAL node pods share nothing but the underlying object storage; no pod talks to another one directly. One WAL node owns every shard of a given table. Splitting a table into more shards parallelizes that table’s own writes into more independent append streams on the node that already owns it; adding more WAL node replicas instead spreads ownership of more tables across more pods. Shard a large table to raise its own ingest ceiling, and add WAL node replicas to raise the cluster’s overall capacity. If a WAL node pod is replaced, its successor must claim ownership of a shard, by a version number that only ever increases, before writing to it — so two pods are never both appending to the same shard’s WAL at once.

​
Reading a real time table

Every query runs against the base table and the fresh tier — every shard’s memtable and not-yet-compacted SSTables — in parallel. Fresh rows supersede base rows with the same primary key, and the two are merged into one result. See RTE reads for what this means for query results.

​
Next steps