Configuration

Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your environment. When using temporary credentials, also set AWS_SESSION_TOKEN. Replace the bucket, table prefix, and region in this example:

import os

from delta_arrow_reader import DeltaTable

options = {
    "aws_region": "<region>",
    "aws_access_key_id": os.environ["AWS_ACCESS_KEY_ID"],
    "aws_secret_access_key": os.environ["AWS_SECRET_ACCESS_KEY"],
}
if token := os.environ.get("AWS_SESSION_TOKEN"):
    options["aws_session_token"] = token

table = DeltaTable("s3://<bucket>/<table-prefix>", storage_options=options)

Keep credentials out of source code. storage_options takes string keys and values; the available settings depend on the storage backend.

Delta tables store their rows in data files. Opening a supported S3 table prepares two things that later queries can reuse:

1. It loads information about the table's files and keeps it in memory. Queries use this to decide which files they need.
2. It reads parts of data files to check the connection speed. This helps it decide when downloading less data will make a query faster.

The saved information stays in memory; it does not cache query results or persist on disk.

An incomplete check does not by itself fail table loading. The first query that can use the measurements retries once, with the same five-second budget. Concurrent queries share the retry. If it also fails, queries use normal reads without another attempt.

The check makes real requests and downloads data. It requires the default direct backend and a supported S3 connection.

If a query uses too many resources, try a limit on concurrent file reads. This example allows at most four files to be read at once in one scan:

from delta_arrow_reader import ScanExecutionOptions

options = ScanExecutionOptions(max_concurrent_file_reads_per_scan=4)
with table.to_reader(execution_options=options, limit=10) as reader:
    print(reader.read_all())

Lowering this limit can reduce resource use, but may slow the query. It applies to each scan separately, does not cap network requests or total memory, and does not change the connection check during table loading.

Pass execution_options to DeltaTable(...) to set defaults for its direct scans. Passing it to to_reader() or scan() replaces the full options object for that one scan.

For controls over buffering and reading ahead, see ScanExecutionOptions. For the number of groups of files processed in parallel, see target_partitions in to_reader().

Register the table you already opened. SQL reuses its file information and any connection measurements. Registration does not repeat the preparation.

SQL registrations have their own default read settings. To give SQL a custom file-read limit, pass it when registering the table:

from delta_arrow_reader import ScanExecutionOptions, Session

options = ScanExecutionOptions(max_concurrent_file_reads_per_scan=4)
session = Session()
session.register_table("events", table, execution_options=options)

Omit execution_options to use SQL's defaults. Custom read settings passed to DeltaTable do not automatically carry over to SQL. The SQL quickstart shows how to run the query afterward.

Choose the same behavior with with_warmup. This example skips the connection check and keeps the file information for later queries:

use delta_arrow_reader::{DeltaReaderError, DeltaTable, DeltaTableBuilder, WarmupMode};

async fn open_table() -> Result<DeltaTable, DeltaReaderError> {
    DeltaTableBuilder::new("s3://bucket/table")
        .with_warmup(WarmupMode::QueryPlanning)
        .load_table()
        .await
}

Call this function inside your application's Tokio runtime. Use WarmupMode::Automatic for the default or WarmupMode::None to leave preparation to queries.

Rust also lets you shorten or extend the connection check. To give it two seconds, replace with_warmup above with:

.with_warmup(WarmupMode::Network {
    max_duration: std::time::Duration::from_secs(2),
})

That limit applies to the check and its possible retry, after file information has loaded. Zero skips the check but still prepares file information. Python currently supports the three named modes without a custom time limit.

For Rust read limits, use DeltaScanExecutionOptions on the table builder or one scan. SQL uses datafusion::ScanOptions.execution_options when registering the table. Changing read settings later does not undo preparation already performed.