Delta Funnel

Move Delta Lake data into SQL Server without Spark or an ODBC driver. Transform rows with DataFusion SQL, then load them through native TDS bulk writes.

Project status

Delta Funnel is early project code. The Rust crate is available on crates.io, and the Python package is available on PyPI.

Start here

Follow these steps in order:

Install Delta Funnel.
Choose one language:
- Python quickstart
- Rust quickstart
Core concepts: understand sessions, sources, tables, outputs, and reports.

Common tasks

SQL Server writes: configure connections and load modes.
Dry runs and reports: validate workflows and inspect structured results.
Multiple outputs and shared caching: write related outputs without repeating common upstream work.
Progress displays: configure and interpret live Python progress.
Private S3 sources: configure credentials and troubleshoot source access.

Troubleshoot a run

Troubleshoot a failed run: inspect failure reports and collect safe troubleshooting information.
Python logging: route diagnostic events through standard-library logging.

Profile performance

Choose a profiling method: select exact operator metrics, Samply, or Perfetto based on the question you need to answer.
Export execution profiles: inspect preview and SQL Server operations with returned metrics or a ranked report.

Reference

API reference: find the Rust and Python API entry points.
Diagnostics reference: look up tracing events, operation phases, stream outcomes, and cache lifecycle fields.
Execution profile reference: look up the returned profile schema, metrics, labels, and redaction rules.

About Delta Funnel

Observed: 13.4M rows in ~14 minutes vs. a ~2 hour Spark/JDBC path.

Project links: GitHub, PyPI, crates.io, docs.rs, and release notes.

Why I wrote this

People like to have finalized golden-layer data ported into a relational database such as SQL Server. I work at an on-prem Microsoft shop, which made the practical deployment target a Windows VM. I had to set up WSL and Spark just to do that job, then deal with slow JDBC writes because sql-spark-connector is no longer maintained.

I built a native solution on top of delta-kernel-rs, tiberius, and datafusion without the overhead of the JVM or JDBC/ODBC.