Skip to content

Streaming query performance

These benchmarks compare Session.sql_reader(...) with the fully materialized Session.sql(...) result path.

Test setup

The generated dataset contains about 10.5 million rows. Each result is the median of three measured Linux runs after one warmup, using a local SSD and one process.

Time to first batch

Query sql_reader(...) Session.sql(...) Improvement
SELECT * FROM prices 370.7 ms 2,312 ms 84% earlier
SELECT * FROM prices ORDER BY ts 2,489 ms 13,182 ms 81% earlier

sql_reader(...) yields batches as the engine produces them. Session.sql(...) must collect the complete result before returning.

Peak process memory

These measurements process each batch immediately rather than retaining the full result.

Query sql_reader(...) Session.sql(...) and iterate Reduction
SELECT * FROM prices 2.30 GiB 3.60 GiB 36% lower
SELECT * FROM prices ORDER BY ts 3.66 GiB 4.84 GiB 24% lower

The streaming path can discard each processed batch instead of retaining a materialized table.

Calling sql_reader(...).read_all() removes this memory advantage. Its performance is in the same range as Session.sql(...) when both paths collect the complete result.

Wide binary rows

Streaming does not impose a byte limit on decoded batches or parallel execution. The wide-row benchmark scans 32,768 rows in four segments, each with one row group and eight 2,048-byte binary columns. The payload alone is 512 MiB when decoded, while the Parquet files total 11,271,319 bytes. The values are intentionally compressible.

The following results were measured on October 6, 2026, using the published timeseries-table-format==0.8.0 wheel, CPython 3.12.12, PyArrow 25.0.0, Linux x64, and 16 logical CPUs. Each value is the median of three fresh-process samples after one discarded warmup for that configuration. Preparation runs in a separate process so its allocations do not inflate reader peak memory. The filesystem cache is not cleared. DataFusion's effective defaults were batch_size=8192 and target_partitions=16.

Settings Batches Largest batch Peak RSS First batch Total scan Rows/s
Defaults 4 128.9 MiB 697.8 MiB 79.9 ms 109.1 ms 300,438
batch_size=1024 32 16.1 MiB 528.2 MiB 67.4 ms 126.8 ms 258,474
target_partitions=1 4 128.9 MiB 275.3 MiB 64.0 ms 283.0 ms 115,802
Both settings 32 16.1 MiB 162.8 MiB 18.4 ms 235.1 ms 139,391

All full scans returned 32,768 rows and 540,672,000 Arrow bytes, consuming and discarding each batch without read_all(). Smaller batches and fewer partitions reduced peak memory, with a throughput cost on this workload. Batch size counts rows, so even the smaller batches held 16.1 MiB each.

Peak RSS includes the interpreter, runtime, and other process allocations. It does not isolate a particular queue or prove a memory ceiling. These are local measurements, not portable performance thresholds. The raw results include pre-query process peaks, effective DataFusion batch size and target partition count, and every measured sample. The default target partition count can be lower than the machine's logical CPU count, for example under CPU affinity limits.

Early close

For every configuration, the benchmark also consumes and discards one batch, closes the reader, verifies that another close succeeds and another read fails, then runs SELECT 1 in the same session. Each worker has a 60-second timeout to detect hangs.

All early-close checks passed. Median close() time was 9.9 ms with defaults and 0.013 ms with both settings. Their process peaks were 692.0 MiB and 151.2 MiB, respectively. Parallel scans can decode substantial data before the first batch reaches Python, so closing early is not a guarantee of low peak memory or immediate memory return to the operating system.

Reproduce the benchmark

From the repository root:

cd crates/timeseries-table-python
uv pip install -p .venv/bin/python numpy
uv run -p .venv/bin/python maturin develop --features test-utils
.venv/bin/python bench/sql_conversion.py \
    --target-ipc-gb 2 \
    --warmups 1 \
    --runs 3 \
    --include-streaming \
    --summary

Increase --target-ipc-gb on machines with more memory. Use --json path/to/out.json to save the raw results.

To run only the wide-row scan and early-close matrix from the repository root, using the same package versions as the measurements above:

uv run --no-project --python 3.12.12 \
    --with timeseries-table-format==0.8.0 --with pyarrow==25.0.0 \
    python crates/timeseries-table-python/bench/sql_conversion.py \
    --wide-streaming --warmups 1 --runs 3 --summary \
    --json sql-reader-wide.json

This mode uses the public API and does not require NumPy or test-utils. After building a local extension, run the same script with its Python environment to measure that build. For a small correctness check, use --wide-rows-per-segment 1025 --warmups 0 --runs 1. The JSON records time to first batch, total scan time, rows per second, largest batch nbytes, close time, process peaks, and effective DataFusion settings. The worker reads the settings after recording scan timing and peak memory. On Windows, peak_rss_bytes reports peak working set. The additional sampled isolated_peak_rss_bytes is available on Linux and includes the worker's post-scan diagnostic queries.

For application code, see Stream query results.