> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Real Time engine (RTE)

> Absorb small, frequent writes cheaply and make them queryable within milliseconds.

**Applies to:** Embedded ✅, Enterprise ✅

## What is the real time engine?

Every write to a LanceDB table is normally a commit to the underlying Lance
dataset. That is ideal for bulk loads, but a stream of small upserts becomes a
stream of small commits, and each write is only visible once its commit lands.

The real time engine (RTE) puts an LSM-style write tier in front of the table.
It turns many small writes into a few large commits, and makes each write
**searchable within milliseconds** of being acknowledged. It is enabled per
table, and nothing about how you query that table changes.

## The Lance MemWAL format

RTE is built on **MemWAL**, a log-structured-merge (LSM) format that Lance
itself defines -- the structure Enterprise serves is part of the open Lance
format, not a layer bolted on beside it. A table with RTE enabled is divided
into one or more **shards** (see [RTE writes](https://docs.lancedb.com/ingestion/realtime-write.md#primary-key)
for enabling and sizing them), and each shard holds:

* A **memtable**: the active, in-memory table for that shard's newest rows,
  queryable as soon as a write lands in it.
* A **WAL buffer**: the same writes, batched and flushed to object storage
  before a write is acknowledged, so an unflushed memtable is never the only
  copy of a row.
* Its own **in-memory indexes** (vector, full-text, BTree, and other scalar
  types), kept current as rows land, independently of the base table's
  indexes.

A write commits in two stages, both before it is acknowledged: the WAL buffer
append makes it durable, then it lands in the shard's memtable. When a
memtable crosses a size or row-count threshold, it freezes and flushes as an
immutable **SSTable** -- an ordinary Lance data file -- and a fresh memtable
takes over. [RTE Compaction](https://docs.lancedb.com/enterprise/realtime-engine-compaction.md)
later folds SSTables into the base table, catching its indexes up as it goes.

## WAL nodes serve MemWAL shards

This section describes LanceDB Enterprise's serving layer. LanceDB Embedded
runs the same MemWAL mechanics directly against the local dataset, in one
process -- there is no separate node to route to or scale; see
[RTE writes](https://docs.lancedb.com/ingestion/realtime-write.md#getting-started) and
[RTE reads](https://docs.lancedb.com/search/realtime-read.md#embedded) for what that looks like.

RTE's Enterprise serving layer is a dedicated component, separate from the
[query nodes and plan executors](https://docs.lancedb.com/enterprise/distributed-search.md)
that serve ordinary queries:

| Component | Responsibility |
| - | - |
| WAL node | Own one or more tables' shards; append their WAL, maintain their memtables and in-memory indexes, and serve reads and writes against them. |

A query node routes every request for an RTE-enabled table -- reads and
writes alike -- to the WAL node that owns that table, by a consistent hash of
the database and table name. WAL node pods share nothing but the underlying
object storage; no pod talks to another one directly.

One WAL node owns every shard of a given table. Splitting a table into more
shards parallelizes that table's own writes into more independent append
streams on the node that already owns it; adding more WAL node replicas
instead spreads ownership of more *tables* across more pods. Shard a large
table to raise its own ingest ceiling, and add WAL node replicas to raise the
cluster's overall capacity.

If a WAL node pod is replaced, its successor must claim ownership of a shard,
by a version number that only ever increases, before writing to it -- so two
pods are never both appending to the same shard's WAL at once.

## Reading a real time table

Every query runs against the base table and the fresh tier -- every shard's
memtable and not-yet-compacted SSTables -- in parallel. Fresh rows supersede
base rows with the same primary key, and the two are merged into one result.
See [RTE reads](https://docs.lancedb.com/search/realtime-read.md) for what this means for query
results.

## Next steps

* [RTE writes](https://docs.lancedb.com/ingestion/realtime-write.md) — enable RTE and write to a table
* [RTE reads](https://docs.lancedb.com/search/realtime-read.md) — query an RTE table
* [Distributed RTE Write](https://docs.lancedb.com/enterprise/distributed-real-time-write.md) — how Enterprise shards and scales writes
* [Distributed RTE Read](https://docs.lancedb.com/enterprise/distributed-real-time-read.md) — how Enterprise combines reads across machines
* [RTE Compaction](https://docs.lancedb.com/enterprise/realtime-engine-compaction.md) — how Enterprise folds shards into the base table


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.