Selective reads on wide tables

We compared Delta Arrow Reader (DAR), delta-rs, DuckDB, Polars, and single-machine Spark on queries that return about 900 rows from Delta tables containing about 60 million rows. The 416-column tables return 69 selected columns; the 90-column tables return 71.

Matching rows are either grouped together or spread out within row groups. Each layout has a snapshot with and without deletion vectors (DVs), which mark deleted rows without rewriting data files. These combinations give eight cases. The inputs use public TPC-H lineitem data at scale factor 10, extended with synthetic numeric columns. These are custom scans, not standard TPC-H queries or a TPC-H score.

Query an initialized table

The README chart shows the first complete query after table initialization. All five readers use this boundary: query planning, reading, and consuming every result are timed. Process startup, Python imports, Spark session/JVM startup, and table initialization are excluded. No query result cache is used.

DAR was rerun from merged commit cd3064e with its default automatic warmup and partial-page policy. The other four readers retain their original five samples per case. The same input hashes, queries, hardware, CPU and memory limits, MinIO binary, and network proxy settings were verified before the update. This comparison combines the new DAR measurements with the earlier competitor measurements.

Every value is the median of five independent processes, in seconds; lower is faster. Each process initializes its table once and runs two queries. Only its first query contributes to this table. Every case passed exact value checks before timing began.

All readers had four physical CPU cores and an 8 GiB memory limit. Storage used 200 ms request latency (+/-20 ms jitter) and a shared 150 Mbps bandwidth limit. These conditions emulate a network connection, not a particular cloud service.

Initialization and repeated queries

Initialization is paid once per loaded table. DAR's default initialization prepares metadata and samples the connection before deciding whether smaller reads should help. That work is included in the initialization figures below. The query chart describes applications that keep a table object and query it. Use the combined figures below to include its first-use cost.

Spark consumes the full result through DataFrame.toArrow(), so it has no comparable streaming first-batch timestamp. The raw timings retain initialization, both query clocks, cleanup, and process measurements. Blank fields mean unavailable, not zero.

Requests and downloaded data

The following values come from separate diagnostic runs with request tracing enabled. Tracing is disabled during timing. Each cell shows response-body MiB / request count for initialization and both queries together, including Delta logs, listings, Parquet, deletion vectors, and DAR's connection sampling. One MiB is 1,048,576 bytes. The byte count is what the proxy accepted for delivery to the reader, including bytes from interrupted responses.

File statistics identify five of 130 Parquet files for the 416-column query and six of 60 for the 90-column query. The predicates and matching file sets are unchanged by DVs. Whole-session diagnostics can touch additional files during initialization, so their file counts also include preparation work.

DAR can request parts of eligible pages when the page layout and measured connection cost make that useful. Fewer downloaded bytes can come with more requests; latency, bandwidth, and concurrency determine whether that trade helps. The I/O CSV separates traffic by object type. DAR chooses ranges using its connection measurements, so its request pattern can vary between invocations. These records do not provide comparable decoded-page counters for all five readers.

Tables and queries

Each base table contains all 59,986,052 SF10 source rows. Parquet divides a file into row groups, then divides each group's column data into pages. In the grouped layout, matches sit close together within each row group. The scattered layout reorders rows within each group, preserving values and file/group membership. A DV snapshot shares its base table's Parquet bytes and deletes one of the 895 matching rows.

Both queries apply this WHERE filter (the predicate) and read all selected columns and matching rows, without COUNT or LIMIT:

WHERE l_shipdate = DATE '1995-03-15'
  AND l_shipmode = 'AIR'
  AND l_linenumber IN (1)

The 416-column query returns five source columns and 64 nullable numeric payloads. The 90-column query also returns l_suppkey and l_quantity. The provenance data contains the full SQL, projection lists, fixture identities, file sizes, and writer settings.

The writer targets 512 MiB files with Zstd level 3, plain encoding, and no dictionaries. Row groups contain at most 131,072 rows: four groups per file for the 416-column table and eight for the 90-column table. Page settings are 20,000 rows, 1 MiB, and 1,024-row write batches. Checking the row limit at batch boundaries can produce pages with 20,480 rows.

Test conditions

The reader allocation uses both SMT siblings of four physical cores. MinIO has CPUs 4 and 5 and 4 GiB; the observer uses CPU 6 and the proxy CPU 7. Spark runs with local[8] and a 4 GiB JVM heap within the common memory limit. Other runner settings retain the original benchmark's fixed resource budget; using DAR's default read policy does not mean every runner setting is a library default. The measured DAR build includes changes after the 0.6.2 release.

The scattered 416-column cases use a separate storage warmup. The other six use their exact validation run as storage warmup. The update preserves this per-case choice. Storage warmup happens in a separate process and is distinct from the table initialization measured in every invocation.

The transport is emulated. Hardware, cache state, query selectivity, file encoding, and storage conditions can change the results. These timings do not predict performance on a particular S3 service or every Delta table.

Data and reproduction

The current comparison contains 200 timed invocations: 40 new DAR sessions and 160 retained competitor sessions. No samples were removed or replaced within a case's five-run set.

• Current timings: every displayed sample, including separate initialization and first- and second-query clocks.
• Current I/O: complete-session diagnostic traffic.
• Current provenance: build and input identities, exact-value certificates, source-record hashes, and export checksums.

The original report and its 400 timing samples, summary, I/O, and provenance remain unchanged. They include the older DAR build and measurements that open a fresh table for each query. Those opening times are not used in the current chart.

The evidence release contains the original campaign archive. The default-policy update is a separate archive, selective-read-defaults-cd3064e.tar.gz, with its raw runs, reference results, exact exports, diagnostics, frozen sources, and replay script. Bulk tables and installed engines are excluded.

The reproduction guide explains archive verification and the pinned fixture and reader setup.