Skip to content

Session reference

Session registers data sources and runs DataFusion SQL queries.

Register data sources

  • register_tstable(...) registers every committed segment in a managed table root.
  • register_parquet(...) registers an unmanaged Parquet file or directory.

Both sources use the same SQL query API.

A directory passed to register_parquet(...) must contain at least one Parquet file so DataFusion can infer its schema. Registering an empty directory raises DataFusionError.

Choose a result type

Method Return type Use when
sql(...) pyarrow.Table The complete result fits in memory
sql_reader(...) pyarrow.RecordBatchReader You want to process large results in batches

See Stream query results for usage and Streaming query performance for benchmarks. For integer-index query rules, see Integer ordered indexes.

Session.sql(...) returns a normal pyarrow.Table, so it can be passed to other Arrow-compatible libraries. For example, after installing Polars:

import polars as pl

frame = pl.from_arrow(session.sql("SELECT * FROM prices"))

SQL result rows have no guaranteed order unless the query includes an ORDER BY clause.

Queries use DataFusion SQL. See the DataFusion SQL reference for supported syntax and functions.

API

__init__() -> None

Create a new DataFusion-backed SQL session.

The session runs async Rust internals on an internal Tokio runtime and releases the GIL while executing queries.

deregister(name: str) -> None

Deregister a previously registered table name.

Raises:

Type Description
ValueError:

If name is empty.
KeyError:

If name is not registered.

register_parquet(name: str, path: str) -> None

Register a Parquet file or directory under a name for SQL queries.

Parameters:

Name Type Description Default
name str

SQL table name to register under.

 required
path str

Path to a Parquet file or a directory of Parquet files.

 required
Notes

If name is already registered, it is replaced atomically (with rollback on failure).

register_tstable(name: str, table_root: str) -> None

Register a time-series table under a name for SQL queries.

Parameters:

Name Type Description Default
name str

SQL table name to register under.

 required
table_root str

Filesystem directory containing the table.

 required
Notes

If name is already registered, it is replaced atomically (with rollback on failure).

sql(query: str, *, params: object | None = None) -> pyarrow.Table

Run a SQL query and return the results as a pyarrow.Table.

Parameters:

Name Type Description Default
query str

SQL query string.

 required
params object | None

Optional query parameter values for DataFusion SQL placeholders:

  • Positional: pass a list/tuple to bind $1, $2, ... Example: sess.sql("select * from t where x = $1", params=[1])
  • Named: pass a dict to bind $name placeholders (keys may optionally start with $). Example: sess.sql("select * from t where x = $a", params={"a": 1})

Supported Python value types: None, bool, int (i64 range), float, str, bytes.

 None
Notes

DataFusion infers placeholder types from context when possible (e.g. in WHERE clauses). If you use placeholders in a SELECT projection without type context, you may need an explicit cast, e.g. SELECT CAST($1 AS BIGINT) AS x.

sql_reader(query: str, *, params: object | None = None) -> pyarrow.RecordBatchReader

Run a SQL query and return a streaming pyarrow.RecordBatchReader.

Parameters:

Name Type Description Default
query str

SQL query string.

 required
params object | None

Optional query parameter values for DataFusion SQL placeholders.

 None
Notes

Unlike Session.sql(...), this does not materialize the full result eagerly. Iterate batches incrementally or call reader.read_all() if you want a pyarrow.Table.

tables() -> list[str]

Return the list of currently registered table names (sorted).