Export and Inspect Execution Profiles

Delta Funnel provides two operation-scoped profiling models:

Both options work with Table.preview, Table.write_to_mssql, Session.write_all, and Session.write_all_for_stream_benchmark. You can enable either one or both for the same operation. Neither is enabled by default.

Exact execution profiling works in normal wheels. Ranked profiling requires a diagnostics-enabled Linux wheel and the setup in Set up Perfetto diagnostics for Python.

Generate an operation-scoped ranked HTML report

Use 1000 Hz for a short investigation and 100 Hz for a longer or lower-volume capture:

from deltafunnel import RankedProfileConfig

preview = table.preview(
    limit=100_000,
    ranked_profile=RankedProfileConfig(
        "target/profiles/preview.profile.html",
        sample_hz=1000,
        artifact_path="target/profiles/preview.dfprofile",
    ),
)

The report is scoped to this operation. Open the self-contained HTML file in a browser.

artifact_path is optional. Set it to keep the validated ranked model used by both HTML and the terminal inspector. The larger raw Perfetto capture is discarded:

delta-funnel-perfetto inspect target/profiles/preview.dfprofile
delta-funnel-perfetto report target/profiles/preview.dfprofile \
  --output target/profiles/preview-again.profile.html

Capture health reports a truncation marker when an operation exceeds the default 100,000 detailed operator activity spans. Raise the limit for that workload without changing its DataFusion execution configuration:

ranked_profile = RankedProfileConfig(
    "target/profiles/full-source.profile.html",
    sample_hz=100,
    artifact_path="target/profiles/full-source.dfprofile",
    max_operator_activity_spans=1_000_000,
)

Higher limits increase trace volume, memory use, and profiling overhead. Delta Funnel scales the semantic trace buffer up to 1 GiB with this value. Rerun the capture and inspect the artifact. A healthy inspection omits the capture-health warning block. If the block appears, do not rely on complete semantic attribution until capture_complete and semantic_complete are true and truncation_marker_count is zero.

Use the same configuration for a single SQL Server write:

report = table.write_to_mssql(
    schema="dbo",
    table="orders",
    load_mode="append_existing",
    ranked_profile=RankedProfileConfig(
        "target/profiles/orders-write.profile.html",
        sample_hz=100,
    ),
)

Or for a complete multi-output workflow:

report = session.write_all(
    outputs,
    options={"cache_mode": "auto"},
    ranked_profile=RankedProfileConfig(
        "target/profiles/write-all.profile.html",
        sample_hz=100,
    ),
)

Exact durations and sampled CPU counts use different units. Function self and inclusive values are statistical on-CPU samples, not wall-clock durations. See Choose a Delta Funnel profiling method for measurement guidance and screenshots.

Only one ranked profile can be active in a Python process. Ranked profiling is not available for dry runs. If an operation fails, Delta Funnel preserves that error while still attempting to finish the report.

Inspect returned preview diagnostics

Use exact execution profiling when you need DataFusion operator metrics alongside the always-available phase timings, without native CPU sampling:

preview = table.preview(limit=20, execution_profile=True)

for timing in preview.phase_timings:
    print(timing["phase_name"], timing["status"], timing["elapsed_micros"])

profile = preview.execution_profile

phase_timings is available for every executed preview. The returned execution_profile is populated only when you pass execution_profile=True. A failure before physical planning completes cannot produce an operator profile.

The returned profile follows the execution profile schema. Operator metrics are cumulative and may overlap, so do not add them together and call the result wall time.

Inspect returned SQL Server output diagnostics

Enable exact profiling on a single write:

report = table.write_to_mssql(
    schema="dbo",
    table="daily_orders",
    load_mode="create_and_load",
    execution_profile=True,
)

profile = report["execution_profile"]
phase_timings = report["phase_timings"]

The profile describes DataFusion query execution. phase_timings describes the SQL Server planning, stream consumption, writer finalization, validation, swap, and cleanup phases. Each timing has an elapsed duration but no start offset. A failed call can retain a completed query profile when the failure happened later in the SQL Server lifecycle.

Inspect write-all profiles

Enable exact profiling for every attempted output and executed cache alias:

report = session.write_all(
    outputs,
    options={"cache_mode": "auto"},
    execution_profile=True,
)

Profiles remain attached to the output or cache alias that produced them. An operation that fails before its profile observer is installed has no profile. Skipped outputs are not attempted and also have no profile. See Multiple outputs and shared caching for report navigation.

Profile write-all without SQL Server I/O

Use the stream benchmark path to execute full output queries and shared-cache work without opening SQL Server connections:

report = session.write_all_for_stream_benchmark(
    outputs,
    options={"cache_mode": "auto"},
    execution_profile=True,
    ranked_profile=RankedProfileConfig(
        "target/profiles/write-all-stream.profile.html",
        sample_hz=1000,
        artifact_path="target/profiles/write-all-stream.dfprofile",
    ),
)

The benchmark drains every output batch and retains row counts, schema checks, cache materialization, exact profiles, and the operation-scoped ranked profile. It skips SQL Server lifecycle work, target validation, bulk encoding, writes, and cleanup. Produced counts are reported through output_row_count and batch_shaping; write_stats remains zero because no rows are written. Use regular write_all when measuring end-to-end behavior.

Combine exact and ranked profiling

Pass both configurations when you need returned operator metrics and ranked native CPU functions from the same operation:

preview = table.preview(
    limit=100_000,
    execution_profile=True,
    ranked_profile=RankedProfileConfig(
        "target/profiles/preview.profile.html",
        sample_hz=1000,
        artifact_path="target/profiles/preview.dfprofile",
    ),
)

The ranked report and artifact paths must name different files. This also applies when symlinks or hard links make them refer to the same file. Ranked profiling does not enable the returned exact execution profile.

Migrate from the earlier Python API

The profiling API changed before 1.0. Replace earlier calls as follows:

The old names are not compatibility aliases. Python rejects them instead of silently selecting a different profiling model.

Related reference

• Python API reference defines exact signatures and configuration fields.
• Execution profile reference defines returned operator metrics and redaction.
• Diagnostics reference defines operation phase timings and terminal events.
• Perfetto diagnostics runbook covers diagnostics-wheel setup, symbols, CLI inspection, and advanced raw capture.