Command-line reader

dar inspect prints a Delta table's snapshot version and complete logical Arrow schema as JSON. It reads Delta log metadata without scanning rows or reading Parquet data or deletion-vector files. A table with an unsupported scan protocol can still be inspected if its snapshot and schema can be loaded.

dar scan reads one snapshot and writes its rows as an Arrow IPC stream. Use it with redirected stdout or a subprocess pipe.

Use the CLI from a Python wheel

The delta-arrow-reader Python package includes a native command with the same name. Install a validated CI wheel with pip to get the Python API and CLI in one environment, or use uv tool/uvx for a separate tool environment. Follow the Python installation instructions for download and installation commands; PyPI publication is planned.

The wheels support Linux x86_64 with glibc 2.28 or newer, Windows x86_64, and macOS arm64 with macOS 12.0 or newer. They require CPython 3.10 or newer and install PyArrow even when you only need the CLI. The installation guide lists the tested Python versions and runtime requirements. No Rust toolchain is needed to install a wheel.

Use the full command name for wheel installations:

delta-arrow-reader --version
delta-arrow-reader inspect /data/orders
delta-arrow-reader scan --column id --limit 100 /data/orders > sample.arrow

The examples below use the standalone name dar. Substitute delta-arrow-reader when using a wheel. Both commands accept the same arguments and have the same output and exit statuses; wheels do not install a dar alias. The Linux archives and containers below have their own platform and runtime requirements.

Use a Linux binary

CLI releases provide validated archives for these platforms:

Both targets are built on Debian 12. The same binaries are tested on Debian 12, Ubuntu 24.04, and Fedora 44. They use glibc and libgcc shared libraries. On Debian or Ubuntu, install libc6, libgcc-s1, and ca-certificates; on Fedora, install glibc, libgcc, and ca-certificates. The certificate bundle is needed for HTTPS storage. The archives do not support Alpine Linux's musl runtime or a scratch container. Other platforms can build from source.

Follow the version-pinned installation and checksum steps to download the two archives, cli-artifacts.json, and SHA256SUMS from a completed GitHub Release. Extract the archive for your CPU after verification. Keep LICENSE and NOTICE alongside dar, including when distributing it with a service. The executable works outside a source checkout and does not require Python or Rust.

Use a container or a Go service

The CLI container packages the same standalone binary with its runtime libraries and CA certificates. Copy the index digest from the completed Release's CLI section and replace RELEASE_INDEX_HASH below. Docker selects the matching architecture from the index:

dar_image='ghcr.io/mag1cfrog/delta-arrow-reader@sha256:RELEASE_INDEX_HASH'
docker run --rm --read-only "$dar_image" --version
docker run --rm --read-only \
  --mount type=bind,src=/absolute/path/to/table,dst=/table,readonly \
  "$dar_image" inspect /table
docker run --rm --read-only \
  --mount type=bind,src=/absolute/path/to/table,dst=/table,readonly \
  "$dar_image" scan --column id --limit 100 /table > sample.arrow

The image runs dar directly as UID/GID 65532:65532. Omitting arguments prints help. Each invocation exits when the command finishes; the image does not run a Reader server. Mounted files and their parent directories must be readable by that user. Supply storage credentials and configuration at runtime. Keep HTTPS certificate verification enabled and provide a readable custom trust bundle when required by your storage endpoint. See HTTPS configuration.

When dar is the container's PID 1, SIGINT and SIGTERM stop it with exit status 130 and 143, respectively. The CLI does not flush buffered output or finish an IPC stream after either signal. Treat that result as an interrupted scan.

A Go service can invoke the binary with exec.CommandContext and decode stdout with the official Arrow Go streaming IPC reader. Pass each CLI argument separately, drain stderr separately, and require both successful IPC decoding and a zero child exit status. An early stop or decoding failure must cancel, close, and reap the child before returning.

The runnable Go example includes bounded stderr capture, cancellation tests, and a consumer Dockerfile. Its COPY --from stages copy /usr/local/bin/dar and /usr/share/doc/dar/ from an explicit CLI image into a Debian 12 service image. The service still needs glibc 2.36 or newer, the binary manifest's shared libraries, and CA certificates. Copying the executable alone does not support Alpine or scratch. The Go application needs no cgo or Rust toolchain at runtime.

Build from source

Install the Rust build prerequisites, then run these commands from a checkout containing the CLI crate:

cargo build --locked -p delta-arrow-reader-cli
./target/debug/dar --version
./target/debug/dar --help

To install the binary into Cargo's binary directory, usually ~/.cargo/bin:

cargo install --locked --path crates/delta-arrow-reader-cli
dar inspect --help

The CLI package is built from source and is not published to crates.io. It shares the workspace version, and dar --version prints dar <version> plus a newline. Help and version work outside the checkout without opening files, loading a table, starting a runtime, or contacting storage.

If the executable is named delta-arrow-reader (delta-arrow-reader.exe on Windows), help and version display delta-arrow-reader. Both names accept the same arguments and produce the same inspection JSON, Arrow streams, diagnostics, and exit statuses.

-h is short for --help, and -V is short for --version. For inspection help, use dar inspect --help or dar help inspect. Scan help is available through dar scan --help or dar help scan.

Inspect a snapshot

dar inspect /data/orders
dar inspect --table-version 0 /data/orders
dar inspect --storage-options-file ./storage.json s3://example-bucket/orders
dar inspect -- './-table with spaces, and commas'

The command accepts exactly one UTF-8 local path or URL. Relative paths use the process working directory. Quote paths and option values that contain spaces; each must arrive as one command-line argument. -- ends option parsing and allows a table name beginning with -.

dar inspect [--table-version N] [--storage-options-file PATH] [--] TABLE

Both options may appear once. Omit --table-version to select the latest snapshot, or pass unsigned decimal digits from 0 through 18446744073709551615. Signs, whitespace, fractions, exponents, and overflow are rejected. Missing values, unknown options, and extra arguments are errors. Running dar without a subcommand exits with status 2.

The parser accepts the full u64 range. The current Kernel cannot load version 18446744073709551615; requesting it returns a snapshot error (status 1).

Explicit versions use a forward listing of retained log entries up to the requested version. Kernel validates the commits and checkpoints in that listing. Discovery does not search numeric version gaps.

Scan rows

dar scan /data/orders > orders.arrow
dar scan --column customer_id --column total --limit 100 /data/orders > sample.arrow
dar scan --no-columns /data/orders > row-counts.arrow
dar scan --table-version 0 /data/orders > original.arrow

The output uses Arrow IPC stream framing. Open it with a streaming IPC reader. Stdout must be a file or pipe; a terminal is rejected with status 2 before table loading, including for --limit 0.

dar scan [--table-version N] [--storage-options-file PATH]
         [--predicate-file PATH] [--column NAME ... | --no-columns]
         [--execution-options-file PATH] [--target-partitions N] [--limit N] [--] TABLE

Table paths, snapshot selection, and storage options follow the inspection rules. Each invocation reads one immutable snapshot.

Omit projection flags to read all columns. Repeat --column once per logical column, in the desired output order. A name containing commas or dots remains one name. The reader validates names and rejects duplicates. --no-columns selects zero columns and preserves the number of live rows in each batch. It conflicts with --column and may appear once.

--limit accepts unsigned decimal digits in the platform's usize range and may appear once. Zero is valid; omitting it reads all rows. Limits count live rows after deletion vectors and predicate filtering. The command does not promise a global row order. Empty results, including --limit 0, contain the selected schema in a valid IPC stream.

The CLI completes scan planning before writing the schema, then writes and flushes each batch before requesting another. It preserves logical types, nullability, nested values, and schema and field metadata. It does not collect the whole result. Memory use includes the core's read buffers, the current batch, and IPC encoding state; variable-sized batches have no fixed byte bound.

Require both successful Arrow decoding and a successful child exit status. IPC readers can accept EOF at a message boundary, so readable output can still be incomplete. Scan or encoding errors exit 1 without writing the IPC end marker. Stdout failures, including an early pipe close, exit 3. Status 0 means the end marker was written and stdout was flushed. Shell redirects can leave partial files after failure; discard those files.

Filter rows

dar scan --predicate-file PATH reads a typed JSON predicate from a local UTF-8 file. The option may appear once; omitting it applies no predicate. Paths resolve against the working directory, and - names a literal file. The CLI never reads a predicate from stdin.

For a table with an Int64 column named id, this selects IDs greater than or equal to 10 and returns only value:

cat > predicate.json <<'JSON'
{"op":"ge","column":"id","value":{"type":"int64","value":"10"}}
JSON
dar scan --predicate-file predicate.json --column value /data/orders > filtered.arrow

Filter columns need not appear in the output projection. Filtering uses the native reader and happens before --limit counts output rows. For example, the following file combines a comparison with a null check:

cat > predicate.json <<'JSON'
{
  "op": "and",
  "args": [
    {"op": "ge", "column": "id", "value": {"type": "int64", "value": "10"}},
    {"op": "is_not_null", "column": "region"}
  ]
}
JSON
dar scan --predicate-file predicate.json --limit 100 /data/orders > sample.arrow

Predicate objects

The file must contain exactly one predicate object. Only the fields shown for each operator are accepted:

Comparison operators are eq (equal), ne (not equal), lt (less than), le (less than or equal), gt (greater than), and ge (greater than or equal). PREDICATE in the table means another predicate object. Empty AND arguments evaluate to true; empty OR arguments evaluate to false.

Only rows where the predicate evaluates to true are returned. Comparisons with null values produce unknown results, including under NOT. Use the null check operators to test for nulls explicitly.

Column names are passed unchanged. They must identify an unambiguous top-level logical field; dots are not interpreted as nested access. There is no SQL parser or schema-based scalar type inference.

Scalar objects

Comparison values must be non-null objects with a type tag and the fields listed below. Types must exactly match the column's Arrow type, including timestamp timezone and decimal precision and scale. No widening or string coercion is performed.

Canonical integer strings are "0" or an optional minus sign followed by a nonzero digit and further digits. "-0", leading zeros, plus signs, whitespace, fractions, exponents, and overflow are rejected. Integers are parsed directly at their declared width without passing through floating point.

Bytes, size, precision, and scale use integer JSON tokens, so 1.0 and 1e0 are invalid for those fields. Float32 conversion may round or underflow to zero. Dates and timestamps accept negative epoch offsets without calendar parsing or timezone conversion. timezone is required even when null. Decimals are not rescaled or rounded; negative scales are allowed.

The file limit is 1 MiB (1,048,576 bytes). The tree limit is 1024 predicate nodes and 32 levels, counting the root as one node at level 1. Duplicate keys, unknown fields or tags, incorrect shapes or types, unexpected nulls, invalid UTF-8, and trailing non-whitespace content are rejected before table loading. These failures produce status 2 with configuration/invalid_input_json; file open or read failures use configuration/input_file_io.

Column resolution, scalar/schema compatibility, and decimal precision, scale, and value compatibility are validated by the core during scan planning. Failures there produce status 1 with scan_planning/unsupported_predicate and leave stdout empty.

Execution options

--target-partitions N sets the execution partition target for one scan. It may appear once and accepts positive unsigned decimal digits fitting the executable's usize range. Zero, signs, whitespace, fractions, exponents, and overflow are rejected before table loading with status 2 and configuration/invalid_cli_argument. Leading zeros are accepted.

Omit the flag to keep the core's automatic partition selection. The built scan may have fewer partitions than requested, for example when there are fewer files. This flag does not change the table's partition columns. inspect rejects it.

dar scan --execution-options-file PATH reads execution settings from a local UTF-8 JSON object. The option may appear once and is accepted only by scan. Omitting the file or supplying {} uses the native reader's defaults. Each invocation starts with fresh settings.

cat > execution.json <<'JSON'
{
  "parquet_backend": "direct",
  "max_concurrent_file_reads_per_scan": 1,
  "max_concurrent_file_reads_per_partition": 1,
  "output_buffer_batches_per_partition": 1,
  "prefetch_files_per_partition": 0,
  "parquet_metadata_size_hint_bytes": null,
  "parquet_full_file_read_threshold_bytes": 1048576
}
JSON
dar scan --target-partitions 2 --execution-options-file execution.json \
  --limit 100 /data/orders > sample.arrow

Only these fields are accepted. Omitted fields retain their current native defaults:

Numeric values must be JSON integer tokens fitting the executable's usize range. Booleans, strings, negative values, fractions, and exponent forms such as 1e0 are rejected. The three capacity fields are also capped at tokio::sync::Semaphore::MAX_PERMITS (2305843009213693951 on 64-bit builds). This capacity bound does not apply to byte sizes or prefetch counts.

Explicit null is accepted only for the three nullable fields in the table. For parquet_metadata_size_hint_bytes, omission keeps the default hint while null disables it. Null on max_concurrent_file_reads_per_scan selects derived capacity; null on parquet_full_file_read_threshold_bytes disables full-file buffering. Zero is accepted only for prefetch_files_per_partition, where it disables preparation of future files.

Both backends use the concurrency and output-buffer limits. Prefetch, metadata hints, and full-file buffering apply to the direct backend. See the execution options reference for their effects. Execution settings compose with predicates, projection, and row limits.

File paths resolve against the working directory; - names a literal file. The limit is 1 MiB (1,048,576 bytes). Invalid UTF-8, duplicate or unknown keys, non-object roots, wrong types, and trailing non-whitespace content are rejected before table loading. These failures use status 2 with configuration/invalid_input_json; file access failures use configuration/input_file_io.

The native setters validate positive values and capacity limits before table loading. Their failures also exit 2, preserving the core's redacted configuration/invalid_configuration diagnostic and leaving stdout empty.

Storage options file

--storage-options-file accepts a local UTF-8 JSON file containing an object of string keys and string values. For example, to access an HTTP endpoint:

{"allow_http":"true"}

The CLI forwards keys and values unchanged, including case, spaces, and commas. The core reader interprets provider options. Omitting the file supplies an empty explicit map and keeps the core's environment and provider credential behavior. An empty object is also valid.

The file limit is 1 MiB (1,048,576 bytes), measured before JSON parsing. Reading stops when that limit is exceeded. Invalid UTF-8, malformed JSON, trailing non-whitespace content, duplicate keys (including equivalent escaped keys), and non-string values are rejected before table loading. File paths resolve against the working directory. A path of - names a local file; it does not read standard input.

Inspection JSON

A successful inspection writes one UTF-8 JSON object followed by a newline to stdout and leaves stderr empty. This example represents an empty schema:

{"format_version":1,"table_version":"0","schema":{"fields":[],"metadata":{}}}

table_version is a decimal string so JSON consumers preserve the full u64 range. schema uses the upstream arrow_schema::Schema Serde representation at the locked workspace Arrow version (currently 58.4.0). It retains field order, nested types, nullability, field metadata, and schema metadata.

For example, a nullable string field is encoded as:

{
  "name": "city",
  "data_type": "Utf8",
  "nullable": true,
  "dict_id": 0,
  "dict_is_ordered": false,
  "metadata": {}
}

Parameterized types use Serde's enum encoding, such as {"Decimal128":[20,4]} or {"Struct":[...]}. Rust consumers using the matching Arrow version with its serde feature can deserialize the schema value directly into arrow_schema::Schema. JSON object key order is not significant. Incompatible changes to the envelope or schema encoding require a new format_version.

The envelope does not include table locations or storage options. Requested schema metadata is returned as table data, including any sensitive strings the table author put there.

Errors and process status

An ordinary failure writes one JSON object plus newline to stderr. Its only fields are the strings phase, code, and message:

{"phase":"configuration","code":"invalid_cli_argument","message":"Invalid command-line arguments."}

Diagnostics omit argument values, file paths, URLs, configuration contents, credential names and values, and underlying dependency messages. Reader failures retain the core reader's phase, code, and redacted Display message. The CLI does not start a tracing logger or mix progress or usage text into errors. Use --help for usage.

Provider configuration rejected during table loading is a reader failure (status 1), even if the option came from a valid local JSON file.

The complete inspection JSON is built before writing. Failures before output leave stdout empty. Consumers must check the process status and discard output from any unsuccessful invocation, including partial output after a write failure. Status 3 is preserved even when stderr cannot be written.

Each command runs in one process. The CLI drives its asynchronous runtime on the main thread and ignores TOKIO_WORKER_THREADS. Scans use a separate output thread so a blocked stdout pipe does not stall network reads. The output thread finishes writing and flushing each batch before requesting another. The core uses background threads for blocking work. Runtime shutdown does not wait for unfinished blocking work, including core engine cleanup.

An unrecoverable panic during reader execution, including failure to start a runtime thread, terminates the process with status 1 and a redacted runtime_initialization diagnostic. Panic payloads and backtraces are omitted. Outside PID 1, Linux signals keep their normal termination behavior: a killed process is unsuccessful and need not emit an error object. This applies while loading, waiting for a batch, or writing to a blocked pipe. A parent that cancels the child must also wait for it to exit and reap it.