ScanExecutionOptions

Leave these settings at their defaults for your first query. When you need a file-read limit or different buffering, Configuration explains which setting to try and its tradeoff.

Pass an options object to DeltaTable for direct-scan defaults, to to_reader() for one scan, or to Session.register_table() for SQL. A supplied object replaces the complete set of options at that scope. The reference below lists exact names, types, defaults, and validation rules.

ScanExecutionOptions

ScanExecutionOptions(
    *,
    parquet_backend: Literal["direct", "delta_kernel"] = "direct",
    max_concurrent_file_reads_per_scan: int | None = None,
    max_concurrent_file_reads_per_partition: int = 3,
    output_buffer_batches_per_partition: int = 1,
    prefetch_files_per_partition: int = 2,
    parquet_metadata_size_hint_bytes: int | None = 65536,
    parquet_full_file_read_threshold_bytes: int | None = None,
)

Control how a scan reads files and buffers results.

Start with the defaults. If a query uses too many resources, try lowering max_concurrent_file_reads_per_scan. It caps files read at once in one scan, not network requests or total memory. A lower limit may slow the query.

Examples:

from delta_arrow_reader import ScanExecutionOptions

options = ScanExecutionOptions(max_concurrent_file_reads_per_scan=4)
with table.to_reader(execution_options=options, limit=10) as reader:
    print(reader.read_all())

Pass the options to DeltaTable(...) for direct-scan defaults, to scan() or to_reader() for one scan, or to Session.register_table() for SQL. An override replaces the entire options object. Instances are immutable; create a new one to change settings. These resource limits do not change the connection check's budget during table initialization.

Parameters:

Name	Type	Description	Default
`parquet_backend`	`Literal['direct', 'delta_kernel']`	`"direct"` uses the reader's asynchronous Parquet backend. `"delta_kernel"` delegates data-file reads to Delta Kernel.	`'direct'`
`max_concurrent_file_reads_per_scan`	`int \| None`	File-read limit across all partitions of one scan. `None` derives it from the partition target times the per-partition limit, capped at the concurrency capacity below.	`None`
`max_concurrent_file_reads_per_partition`	`int`	File-read limit for each group of files processed together. Both this and the scan limit apply.	`3`
`output_buffer_batches_per_partition`	`int`	Batches each partition can queue while waiting for the consumer. Larger buffers use more memory.	`1`
`prefetch_files_per_partition`	`int`	Upcoming files to prepare ahead of time per partition. `0` disables prefetch. Direct backend only.	`2`
`parquet_metadata_size_hint_bytes`	`int \| None`	Bytes to request initially from each file's end when loading Parquet metadata. `None` disables the hint. If the metadata is larger, the reader makes another request. Direct backend only.	`65536`
`parquet_full_file_read_threshold_bytes`	`int \| None`	Read a file entirely into memory when its size is at or below this threshold. `None` disables this. This can reduce requests for small files but reads unneeded data and uses more memory. Direct backend only.	`None`

When set, the file-read limits and output buffer size must be positive integers no greater than sys.maxsize >> 2. Prefetch accepts nonnegative integers. Byte settings accept positive integers or None. All integers must fit 2 * sys.maxsize + 1; booleans are not accepted.

Raises:

Type	Description
`TypeError`	A setting has the wrong type, including a Boolean for an integer.
`ValueError`	A setting is out of range or the backend name is unknown.
`OverflowError`	An integer exceeds `2 * sys.maxsize + 1`.

parquet_backend `property`

parquet_backend: Literal['direct', 'delta_kernel']

Selected Parquet backend: "direct" or "delta_kernel".

max_concurrent_file_reads_per_scan `property`

max_concurrent_file_reads_per_scan: int | None

File-read limit for a whole scan, or None to derive it automatically.

max_concurrent_file_reads_per_partition `property`

max_concurrent_file_reads_per_partition: int

Maximum number of files each scan partition can read at once.

output_buffer_batches_per_partition `property`

output_buffer_batches_per_partition: int

Maximum number of output batches queued per partition.

prefetch_files_per_partition `property`

prefetch_files_per_partition: int

Number of upcoming files prepared per partition; 0 disables prefetch.

parquet_metadata_size_hint_bytes `property`

parquet_metadata_size_hint_bytes: int | None

Initial Parquet metadata request size in bytes, or None for no hint.

parquet_full_file_read_threshold_bytes `property`

parquet_full_file_read_threshold_bytes: int | None

Largest file to buffer entirely, in bytes, or None to disable this.

ScanExecutionOptions