Python log records and handlers

Use this reference when formatting structured records or writing a logging handler. It also defines TRACE mapping, event ownership, and delivery behavior. To turn on ordinary application logs, start with Enable Python logging.

Configure structured records

This example sends INFO and higher levels to stderr, including each event's structured fields:

import logging
from delta_arrow_reader import DeltaTable

logger = logging.getLogger("delta_arrow_reader")
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(levelname)s %(message)s %(dar_fields)s"))
logger.addHandler(handler)
logger.setLevel(logging.INFO)
logger.propagate = False

table = DeltaTable("path/to/table")
latest = table.refresh()

If your application already configures logging, use its handlers. The package adds only a NullHandler to its own logger, so unconfigured applications remain quiet even when they catch reader exceptions. It does not configure the root logger or add console or file output. Records follow normal Python propagation. Changes to logger and handler levels apply to later events, including those from existing tables, sessions, and partially consumed readers.

Level mapping

To include TRACE, set the logger's level to 5 and allow level 5 in its handlers. The package does not register a global name for that numeric level.

Read a log record

Each record has dar_target, the Rust event's target, and dar_fields, a dictionary of the event's fields. Numbers and booleans keep their types. Dictionaries contain at most 64 fields; text values are limited to 1,024 characters.

Core lifecycle records identify the operation and available snapshot context. Failures include error_phase, error_code, error_reason, and, when available, error_source_kind. The tracing reference defines these fields, event ownership, levels, and redaction rules.

Execution records start when batches are requested. A completed DataFusion partition does not establish whole-query success, and independent partitions can each fail. Check the result of the Python call or Arrow read. Converting a core failure into an exception, or replaying a failed Arrow read, does not add another failure record.

Exceptions and diagnostic events

For the exceptions to catch in application code, see Handle Python errors.

Before row consumption, the binding converts core errors into Python exceptions with the core's phase, code, reason, and safe source_kind attributes. source_kind is None when the core error has no underlying source. These attributes remain available when logging is disabled. Core scan-option validation keeps its ValueError type and carries the same attributes. Binding-owned DeltaReaderError exceptions, such as SQL planning and schema export failures, have reason = None and source_kind = None; their diagnostic events carry the available details.

Failures reported by the binding

The binding forwards core events and reports failures it owns on the delta_arrow_reader_python target, through the same Python logger. Its records omit SQL, identifiers, argument values, schema contents, paths, and dependency messages at every level.

Binding events

Each failure in this table emits one ERROR. Typed core failures are forwarded without another ERROR. SQL planning success means a reader is ready; execution begins when the consumer requests batches.

SQL source classification uses multiple_errors for collections, unknown for unrecognized sources, and source_chain_limit after 16 links. An I/O failure can also have an unknown category; its OS code can still distinguish causes such as exhausted file descriptors.

Invalid or duplicate SQL table names use the core code datafusion_adapter in both the exception and event. The exception's reason distinguishes an empty name, an invalid name, and failed registration.

Schema export keeps the existing exception's phase=schema, code=schema_conversion, and message. Runtime creation keeps its RuntimeError message.

Exceptions without another event

Requested-schema import errors use the fixed ValueError message invalid requested schema. Dependency text and detailed Arrow causes are not exposed. The consumer can inspect or replace the requested schema and retry.

Write a logging handler

Handlers run synchronously on the calling thread or a reader worker. Keep them short: do not start reader work or wait for another thread to finish reader work. The bridge uses a scoped Rust subscriber and preserves other native extensions' global tracing configuration.

Exceptions and recursive calls

Calls that start reader work or close/export a RecordBatchStream raise RuntimeError inside a handler. Reading an exported Arrow reader raises a PyArrow error and leaves the reader usable. Pure validation and schema inspection preserve their exceptions without recursively forwarding logs.

An Arrow consumer's close() can release its stream from a handler. If a read is active, cleanup waits until its native callback returns, and the current batch may still be returned. Later reads see a closed stream. Follow the consumer's threading rules; this behavior does not permit concurrent reads and closes.

Callbacks use PyO3's best-effort interpreter shutdown check. Delivery during finalization is not guaranteed.