> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Distributed Shuffle

> How a SQL query's intermediate results move between plan executors for a join, aggregate or sort.

**Applies to:** Embedded ❌, Enterprise ✅

A distributed SQL query plans as a sequence of **stages**, each with its
own set of partitions, connected by an **exchange**: a streaming gRPC
service that carries one stage's output to the next stage's plan
executors. [Distributed Search](https://docs.lancedb.com/enterprise/distributed-search.md) covers
how a query gets placed on the fleet in the first place; this page covers
what happens between placement and the final result, for a query whose
plan needs data repartitioned across machines -- a hash join or a grouped
aggregate, for example, where rows with the same key must land on the
same worker regardless of which plan executor originally read them.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
%%{init: {"theme": "base", "themeVariables": {
  "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
  "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
  "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
  "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
    subgraph Stage1["Stage: scan"]
    direction TB
    PE1["Plan executor<br/>reads its partition"]
    PE2["Plan executor<br/>reads its partition"]
    end

    EX["Exchange<br/>streaming gRPC,<br/>repartitions by join/group key,<br/>backpressure via a byte budget"]

    subgraph Stage2["Stage: join / aggregate"]
    direction TB
    PE3["Plan executor<br/>probes its share of keys"]
    PE4["Plan executor<br/>probes its share of keys"]
    end

    PE1 --> EX
    PE2 --> EX
    EX --> PE3
    EX --> PE4
    classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
    classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
    class EX accent
```

## Pipelined, not batched

The exchange moves data as a gRPC stream of row batches, not a file a
later stage waits for: a consuming plan executor pulls batches as they
become available, and a slow consumer throttles its producer through a
byte budget on unread messages rather than letting memory grow without
bound. A partition that stops being read -- for example the probe side of
a hash join whose build side turned out empty -- is drained rather than
left to stall the budget for every other partition sharing the same
connection.

## What to read next

> [card] [Distributed Search](https://docs.lancedb.com/enterprise/distributed-search.md)
>
> How a query gets placed on the query-serving fleet in the first place.

> [card] [Query plans](https://docs.lancedb.com/basics/sql.md#query-plans)
>
> Inspect a specific query's plan and its actual execution metrics.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.