API Reference

Use this page to look up public entry points and exact Python signatures. For examples, start with the Python quickstart or Rust quickstart.

Find an Entry Point

Rust

The published Rust API reference is docs.rs/delta-funnel.

Build the reference for the checked-out version with:

cargo doc -p delta-funnel --open

Python

The package and import name are deltafunnel. The typed public surface is recorded in deltafunnel.pyi.

Values and Type Aliases

__version__: str

LoadMode: TypeAlias = Literal["append_existing", "create_and_load", "replace"]
WriteAllCacheMode: TypeAlias = Literal["auto", "explicit", "disabled"]
Report: TypeAlias = dict[str, object]
Options: TypeAlias = Mapping[str, object]

class WriteAllExecutionOptions(TypedDict, total=False):
    cache_mode: WriteAllCacheMode
    cache_aliases: Sequence[str]

class FileRepartitioning:
    FILL_MISSING_PARALLELISM: ClassVar[FileRepartitioning]
    REBALANCE: ClassVar[FileRepartitioning]

class ProviderScanOptions:
    def __init__(
        self,
        *,
        max_concurrent_file_reads_per_scan: int | None = None,
        max_concurrent_file_reads_per_partition: int = 3,
        output_buffer_capacity_per_partition: int = 1,
        native_async_prefetch_file_count_per_partition: int = 2,
        parquet_metadata_size_hint: int | None = 65_536,
        parquet_full_file_read_threshold: int | None = None,
        intra_file_repartitioning: FileRepartitioning = FileRepartitioning.FILL_MISSING_PARALLELISM,
        use_view_types: bool = False,
    ) -> None: ...

Reports are JSON-compatible Python dictionaries. See Dry runs and reports for report interpretation.

Functions

init_logging

def init_logging(
    filter: str | None = None,
    logger: str = "deltafunnel",
) -> bool

Installs the Python logging bridge for Rust tracing events. It returns True when it installs the process-wide subscriber and False when a subscriber is already set. See Python logging for filters, record fields, and setup.

init_perfetto_diagnostics

def init_perfetto_diagnostics(
    filter: str | None = None,
    logger: str = "deltafunnel",
    wait_timeout_seconds: float = 10.0,
) -> bool

Installs the Python logging bridge and Perfetto profiling layer, then waits for a capture to attach. The function is available only in builds compiled with Perfetto diagnostics. Use it for an advanced whole-process capture. RankedProfileConfig manages activation automatically for operation-scoped HTML reports. See Perfetto diagnostics for build and capture steps.

Exceptions

DeltaFunnelError

class DeltaFunnelError(Exception):
    phase: str
    kind: str
    message: str
    context: object | None

Delta Funnel raises this exception for configuration, planning, and workflow failures. phase identifies the failed operation area, kind is a stable machine-readable category, message is safe for display, and context contains structured details when available.

Errors raised after an operation finishes can also expose deltafunnel_operation_status, deltafunnel_operation_error, or deltafunnel_operation_report. Inspect these fields before retrying a write. See Troubleshoot a failed run.

Classes

RankedProfileConfig

class RankedProfileConfig:
    report_path: Path
    sample_hz: Literal[100, 1000]
    artifact_path: Path | None
    max_operator_activity_spans: int

    def __init__(
        self,
        report_path: str | PathLike[str],
        *,
        sample_hz: Literal[100, 1000] = 1000,
        artifact_path: str | PathLike[str] | None = None,
        max_operator_activity_spans: int = 100_000,
    ) -> None

Configures one operation-scoped ranked HTML report. Set artifact_path to retain the same model as a .dfprofile file for later HTML or terminal inspection. The configuration is immutable. Delta Funnel creates missing parent directories. Use 1000 Hz for short operations and 100 Hz for longer captures. max_operator_activity_spans is the positive per-operation limit for detailed DataFusion activity spans. Increase it from its 100,000 default only when capture health reports an activity truncation marker.

Only one ranked profile can be active in a process. The type is importable from every wheel, but using it requires a diagnostics-enabled Linux build. It does not enable the exact profile, and dry runs reject it. See Generate a ranked HTML report.

Session

class Session:
    def __init__(
        self,
        *,
        default_mssql_connection_string: str | None = None,
        target_partitions: int | None = None,
        output_batch_size: int | None = None,
        provider_scan_options: ProviderScanOptions | None = None,
        validation_options: Options | None = None,
        schema_options: Options | None = None,
    ) -> None

A session owns registered sources, lazy SQL tables, runtime configuration, and SQL Server defaults.

Session options

Dictionary-backed validation and schema options reject unknown keys.

Use the typed options object for editor completion and checked policy values:

from deltafunnel import FileRepartitioning, ProviderScanOptions, Session

session = Session(
    provider_scan_options=ProviderScanOptions(
        intra_file_repartitioning=FileRepartitioning.REBALANCE,
        use_view_types=True,
    )
)

ProviderScanOptions accepts these keyword arguments:

Delta Funnel defaults to ordinary Arrow Utf8 and Binary arrays because they perform better in its transformation-heavy workflows. Set use_view_types to True for scan-heavy workloads that benefit from view arrays. Partition string and binary columns remain dictionary encoded in either mode.

For an eligible file, the native async reader performs one full object-store GET and serves that file's later Parquet metadata and data range reads from a temporary in-memory store. The store is released when the file stream finishes or is dropped. Choose the threshold from measured file sizes and query access patterns. A higher threshold can reduce remote requests for small files that are read broadly, but it can transfer unnecessary bytes for narrow projections.

For example, to buffer Parquet files up to 2 MiB for one measured workload:

session = Session(
    provider_scan_options=ProviderScanOptions(
        parquet_full_file_read_threshold=2 * 1024 * 1024,
    )
)

See the Delta Arrow Reader execution options for the execution boundaries controlled by these values.

validation_options accepts:

See Dry runs and reports for the behavior and cost of the validation and scan-summary modes.

schema_options accepts:

The bounded string and binary forms choose an explicit SQL Server length; the observed_* forms infer a bounded length from observed values. normalize_utc_datetime2 converts timezone-aware values to UTC before using the timezone-free timestamp target. The checked_* policies accept values only when the target representation can hold them. Normalization, rounding, and truncation policies perform the conversion named by the value.

Session.delta_lake

@overload
def delta_lake(
    self,
    source_uri: str,
    *,
    version: int | None = None,
    storage_options: Mapping[str, str] | None = None,
    name: str,
    progress: bool | None = None,
) -> Table

@overload
def delta_lake(
    self,
    source_uri: str,
    *,
    version: int | None = None,
    storage_options: Mapping[str, str] | None = None,
    name: None = None,
    progress: bool | None = None,
) -> PendingDeltaSource

With name, loads and registers the Delta source immediately. Without name, returns an unregistered PendingDeltaSource; call its alias method to load and register it. version selects a Delta snapshot. storage_options must map string keys to string values. See Private S3 sources for documented AWS keys.

Session.table_from_sql

def table_from_sql(self, sql: str) -> Table

Builds a lazy table from one read-only DataFusion SQL statement. It plans the query but does not read rows.

Session.write_all

def write_all(
    self,
    outputs: Sequence[MssqlOutputSpec],
    *,
    options: WriteAllExecutionOptions | None = None,
    dry_run: bool | None = None,
    progress: bool | None = None,
    execution_profile: bool = False,
    ranked_profile: RankedProfileConfig | None = None,
) -> Report

Plans or executes several SQL Server output specs in order and returns one report. Every spec must come from the same session.

• dry_run=True plans without writing and rejects options, execution_profile, and ranked_profile.
• options={"cache_mode": "auto"} enables eligible shared-work caching.
• options={"cache_mode": "explicit", "cache_aliases": ["prepared", "export"]} caches exactly those registered derived aliases in dependency order. The two keys must be supplied together, and the selection must not skip a registered derived alias between a selected alias and a later selected alias or output.
• options={"cache_mode": "disabled"} disables shared-work caching.
• execution_profile=True attaches exact profiles to attempted outputs and executed cache aliases.
• ranked_profile writes an operation-scoped ranked HTML report.

See Multiple outputs and shared caching for workflow examples and Inspect write-all profiles for profile ownership.

Session.write_all_for_stream_benchmark

def write_all_for_stream_benchmark(
    self,
    outputs: Sequence[MssqlOutputSpec],
    *,
    options: WriteAllExecutionOptions | None = None,
    execution_profile: bool = False,
    ranked_profile: RankedProfileConfig | None = None,
) -> Report

Executes and fully drains several output streams without contacting SQL Server. The method preserves normal output planning, shared-cache work, DataFusion execution, row counting, and batch schema validation. It skips SQL Server lifecycle work, bulk writes, target validation, and cleanup. Output specs must still resolve valid target and connection configuration for planning. Produced counts are available through output_row_count and batch_shaping; write_stats remains zero because the benchmark writes no rows.

Use this method only to benchmark query and stream execution without SQL Server I/O. It does not predict end-to-end write performance.

PendingDeltaSource

Returned by Session.delta_lake(...) when name is omitted. The source is not available to SQL until it is registered.

PendingDeltaSource.alias

def alias(
    self,
    name: str,
    *,
    progress: bool | None = None,
) -> Table

Loads and registers the pending source under name, then returns a Table. The progress value applies to this registration call.

Table

A lazy Delta source or SQL-derived query associated with its owning session.

Table.alias

def alias(self, name: str) -> Table

Registers the lazy SQL-derived table under name so later SQL can reference it, then returns the registered table.

Table.preview

def preview(
    self,
    limit: int = 20,
    *,
    progress: bool | None = None,
    execution_profile: bool = False,
    ranked_profile: RankedProfileConfig | None = None,
) -> Preview

Executes a bounded query and returns a rendered Preview. Phase timings are always included. execution_profile=True enables the returned exact profile. ranked_profile writes a sampled native CPU report without implicitly enabling the exact profile. The method reads rows but does not contact SQL Server.

Table.show

def show(
    self,
    limit: int = 20,
    *,
    progress: bool | None = None,
) -> None

Executes the same bounded query as preview and prints its text form to Python stdout. It does not retain the Preview or enable exact execution profiling.

Table.to_mssql

def to_mssql(
    self,
    *,
    schema: str,
    table: str,
    load_mode: LoadMode,
    name: str | None = None,
    connection_string: str | None = None,
) -> MssqlOutputSpec

Builds an output specification without planning or writing rows. name defaults to the target table name and identifies the output in write_all reports. connection_string overrides the session default for this output.

Table.write_to_mssql

def write_to_mssql(
    self,
    *,
    schema: str,
    table: str,
    load_mode: LoadMode,
    dry_run: bool | None = None,
    name: str | None = None,
    connection_string: str | None = None,
    progress: bool | None = None,
    execution_profile: bool = False,
    ranked_profile: RankedProfileConfig | None = None,
) -> Report

Plans or executes one SQL Server output and returns a report.

• dry_run=True plans without writing and rejects execution_profile and ranked_profile.
• execution_profile=True attaches execution_profile to an execute report.
• ranked_profile writes an operation-scoped ranked HTML report.
• connection_string overrides the session default for this call.

If ranked report export fails after SQL Server succeeds, the raised exception reports deltafunnel_operation_status="completed" and contains the sanitized report in deltafunnel_operation_report. Do not treat the export error as evidence that a write is safe to retry.

See SQL Server writes for load modes and Inspect returned SQL Server output diagnostics for profile details.

Preview

class Preview:
    text: str
    html: str
    phase_timings: list[dict[str, object]]
    execution_profile: dict[str, object] | None

    def __str__(self) -> str
    def _repr_html_(self) -> str

text is the plain-text table. html backs notebook display. phase_timings is always populated. execution_profile is populated when Table.preview receives execution_profile=True.

MssqlOutputSpec

An opaque output specification created by Table.to_mssql(...) and consumed by Session.write_all(...). It retains its owning session, lazy table, output name, target, load mode, and optional connection override.

Related Reference

• Execution profile reference defines the returned profile schema, metric mapping, labels, and redaction rules.
• Diagnostics reference defines tracing events, operation phase timings, stream outcomes, and cache lifecycle fields.
• Progress displays defines automatic, forced, and disabled progress behavior.

Moved Reference Sections

These anchors preserve links to sections that moved out of this page.

Execution profile model

Profile schema

Raw and aggregated metrics

Labels and redaction

Delta provider snapshots