> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Pipeline counters for diagnosing StreamingDataset bottlenecks.

**Applies to:** Embedded ✅, Enterprise ✅

Optimizing data loader performance can be tricky, as runtime bottlenecks are often difficult to locate across various stages of the I/O and CPU pipelines. To help distinguish bottlenecks, [`StreamingDataset`](https://docs.lancedb.com/training/data-loading) exposes several pipeline counters: `raw_queue_depth` shows the number of loaded rows waiting to be transformed,
and `prefetch_queue_depth` is the number of transformed rows ready to be consumed. `unscanned_rows` and
`consumed_rows` show how far the current iterator has progressed.

If the `prefetch_queue_depth` is consistently zero but the `raw_queue_depth` is not, then you likely have a CPU
transformation bottleneck. You should investigate GIL bottlenecks or look for ways to optimize your transformation.
This can often be done by batching the compute work. If both `prefetch_queue_depth` and `raw_queue_depth` are
consistently zero while the consumer is waiting, I/O is the likely bottleneck. A larger read batch size or clumped
shuffling could help.

```py Python icon=Python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
import threading, time

from lancedb.streaming import StreamingDataset

ds = StreamingDataset(table, shuffle_seed=42)

def log_pipeline_health():
    while True:
        print(
            f"unscanned={ds.unscanned_rows} "
            f"raw={ds.raw_queue_depth} "
            f"cooked={ds.prefetch_queue_depth} "
            f"consumed={ds.consumed_rows}"
        )
        time.sleep(1.0)

monitor = threading.Thread(target=log_pipeline_health, daemon=True)
monitor.start()

for sample in ds:
    train_step(sample)

print(
    f"bytes loaded: {ds.bytes_loaded} "
    f"fetch time: {ds.fetch_time:.2f}s "
    f"transform time: {ds.transform_time:.2f}s"
)
```

`bytes_loaded` measures raw Arrow buffer bytes before transformation. `bytes_loaded`, `fetch_time`, and `transform_time`
are cumulative across iterations of the same dataset instance; because work runs concurrently, the summed stage times
can exceed wall-clock time. The queue depths report rows currently waiting in the pipeline, and the progress counters
reflect the latest iterator snapshot.

## Next steps

> [card] [PyTorch Data Loader](https://docs.lancedb.com/training/data-loading.md)
>
> Stream, shuffle and transform training data with StreamingDataset.

> [card] [Checkpointing](https://docs.lancedb.com/training/checkpointing.md)
>
> Save progress so a failed run can pick back up.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.