Delta metadata lifecycle
Every Delta query needs to know which Parquet files belong to the loaded table version. It can then narrow that list using the query predicate, partition values, and file statistics. The Delta log or checkpoint supplies the file list. The list and the information used to prune it are the table's scan metadata.
WarmupMode controls how much reusable metadata load_table prepares. The
default None mode waits until each scan is built. QueryPlanning prepares the
reusable Delta metadata while the table loads.
Each query still gets its own file, row-group, and page selection. The cache does not reuse one query's selection for another query or change query results.
What happens before rows are read
A query against remote storage can involve three rounds of I/O:
- The reader uses the Delta log or checkpoint to find the active files. Delta Kernel applies the query predicate to their partition values and file statistics, removing files that cannot match.
- The reader fetches the remaining Parquet footers. Row-group statistics may narrow the read again.
- The reader fetches the selected Parquet row groups and produces Arrow batches.
The first round mixes work that can be reused with work that belongs to one
query. Replaying the Delta history and reconciling the active files can be
reused for a loaded table version. Applying a predicate cannot.
WarmupMode::QueryPlanning moves only that reusable work to table
initialization. Parquet metadata and data remain part of each query.
This difference matters most for a selective query. Statistics may reduce a large table to a few Parquet row groups, making the data read small, while Delta replay remains a fixed planning cost. On remote storage, that planning cost can take longer than reading the result. Reusing active-file metadata removes the repeated replay without changing the later pruning or Parquet I/O.
No warmup
DeltaTableBuilder::load_table() uses WarmupMode::None by default:
table load -> table version and schema
query 1 -> Delta replay -> query pruning -> Parquet footer/data
query 2 -> Delta replay -> query pruning -> Parquet footer/data
query 3 -> Delta replay -> query pruning -> Parquet footer/data
Loading the table selects a version and reads its schema. The reader waits until a scan is built to assemble the active-file metadata and ask Delta Kernel to prune it for that scan.
This keeps table loading quick and avoids retaining an active-file cache. The tradeoff is that every scan against the loaded table repeats the Delta replay. No warmup usually fits tables that are queried once, queried occasionally, or loaded in a process with tight memory limits.
Query-planning warmup
DeltaTableBuilder::with_warmup(WarmupMode::QueryPlanning) prepares the same
reusable metadata before load_table() returns:
table initialization -> Delta replay -> metadata cache
|
query 1 -----------------------------------+-> pruning -> Parquet footer/data
query 2 -----------------------------------+-> pruning -> Parquet footer/data
query 3 -----------------------------------+-> pruning -> Parquet footer/data
The method returns after the cache is ready, so table initialization takes
longer. The cache contains the reconciled Delta add metadata for the current
active files, including available file statistics. It does not contain the raw
commit history.
When a scan is built, the reader gives this metadata back to Delta Kernel
through Scan::scan_metadata_from. Delta Kernel applies that scan's predicate
and statistics pruning. Different queries can therefore select different
files while sharing the result of Delta replay.
The cache belongs to one loaded table. If the same location is loaded twice,
the two table objects have separate caches. Each table stays fixed at its
loaded version. Calling refresh creates another table instead of changing the
existing one.
Which mode should you use?
This choice applies to the loaded table, whether you query it through the streaming API or DataFusion SQL. The warmup behavior is the same through both APIs.
| Consideration | No warmup | Query planning |
|---|---|---|
| Typical use | One query, occasional queries, or tight memory limits | Repeated queries where Delta replay is costly |
| Table loading | Returns without building a reusable scan cache | Waits for Delta replay and cache creation |
| Memory while loaded | No reusable active-file cache | Keeps active-file metadata and statistics |
| Each later scan | Replays Delta metadata, then prunes | Reuses Delta metadata, then prunes |
| Seeing a newer version | refresh returns a new lazy table |
refresh updates or reuses the retained cache |
| Parquet metadata and data reads | Per query | Per query |
Use no warmup unless you know the table will serve repeated queries and the saved planning time justifies slower initialization and higher memory use. No single query count is a reliable break-even point. The result depends on the table history, checkpoint shape, file count, available statistics, storage latency, query selectivity, and memory pressure.
Results from a real S3 workload
On August 27, 2026, a real-S3 benchmark loaded two production-shaped tables and ran six queries:
| Measurement | No warmup | Query-planning warmup |
|---|---|---|
| Table initialization | 1.196 s | 4.089 s |
| HIP physical planning | 2.097 s | 5.1 ms |
| Schedule physical planning | 756 ms | 1.1 ms |
| Complete six-query session | 50.205 s | 42.451 s |
| Resident memory after initialization | 38.5 MiB | 68.7 MiB |
| Full-session peak resident memory | 233.9 MiB | 248.9 MiB |
The warmed run spent more time and memory during initialization. After that, physical planning fell from seconds to milliseconds, and the six-query session finished sooner. Both runs returned the same results and performed the same Parquet I/O.
These numbers come from one workload. They are not a performance guarantee or a general break-even point. The benchmark methodology, environment, and limitations describe how the measurements were collected and what can affect them.
Version and refresh behavior
Every loaded table and cache represents one exact Delta version. Commits
written later do not change that table or any scan already built from it.
DeltaTable::refresh checks for the latest version and returns another
immutable table. If no newer version exists, it reuses the existing snapshot,
schema, and query-planning cache.
With no warmup, the returned table has no retained active-file cache. Its scans continue to assemble scan metadata when they are built. With query-planning warmup, refresh passes the old version and retained active-file metadata to Delta Kernel. Delta Kernel reconciles commits written after that version. If a newer checkpoint prevents incremental reconciliation, it performs a full metadata replay from that checkpoint instead.
Refresh returns an error rather than a partially updated table when the latest snapshot, schema, or retained query-planning metadata cannot be loaded. The caller can keep using the original table after such an error. A lazy table may load a newer unsupported protocol for inspection, but its scans still reject that protocol. Refreshing a warmed table fails if the newer protocol cannot be used to rebuild its query-planning cache.
DeltaTableProvider::refresh applies the same table refresh and rebuilds its
DataFusion schema when needed. It returns a new provider but does not replace a
provider already registered in a SessionContext.
The in-memory cache does not provide:
- background polling;
- TTL or eviction;
- persistence across processes;
- sharing between independently loaded tables;
- Parquet data-page or decoded-batch caching; or
- automatic selection of which tables to warm.