Add nullable columns to an existing table¶

Use table.add_columns(pyarrow.Schema) to introduce new payload fields after the first successful append establishes the table's schema. The fields must be nullable and top-level. Existing types, nullability, names, and keys cannot change. This operation adds fields; it does not compute or backfill historical values. Use a subsequent keyed row update to assign externally computed values to those columns.

Run the example¶

The example creates a temporary table, appends historical rows, adds two fields, re-registers its SQL table, then appends values and omissions. It prints six rows; historical and omitted values are null. The same script runs in the documentation tests.

from pathlib import Path
from tempfile import TemporaryDirectory

import pyarrow as pa

import timeseries_table_format as ttf


def run(*, table_root: Path) -> pa.Table:
    table = ttf.TimeSeriesTable.create(
        table_root=str(table_root),
        index_column="tick",
        index_type="int64",
        index_granularity=1,
        entity_columns=["device"],
    )
    # The first append establishes the canonical schema.
    table.append(
        pa.table({"tick": [0, 0], "device": ["A", "B"], "reading": [1.0, 2.0]})
    )
    session = ttf.Session()
    session.register_tstable("readings", str(table_root))

    version = table.add_columns(
        pa.schema(
            [
                pa.field("quality", pa.float64(), nullable=True),
                pa.field("reviewed", pa.bool_(), nullable=True),
            ]
        )
    )
    assert version == table.version() == 3
    # Replace the existing registration to expose the new schema.
    session.register_tstable("readings", str(table_root))
    assert session.sql("SELECT quality FROM readings")["quality"].to_pylist() == [
        None,
        None,
    ]

    table.append(
        pa.table(
            {
                "tick": [1, 1],
                "device": ["A", "B"],
                "reading": [3.0, 4.0],
                "quality": [0.9, 0.8],
                "reviewed": [True, False],
            }
        )
    )
    # Evolved tables fill omitted nullable payloads with null. All keys are required.
    table.append(pa.table({"tick": [2, 2], "device": ["A", "B"]}))
    return session.sql("SELECT * FROM readings ORDER BY tick, device")


if __name__ == "__main__":
    with TemporaryDirectory() as directory:
        print(run(table_root=Path(directory) / "readings"))

Preserve schema annotations¶

Pass a schema describing only the new fields. Schema and field metadata, including metadata on nested children, is rejected because the logical table model cannot persist it. Complete supported structs, lists, and maps may be added as nullable top-level fields. Dots in names are literal; they do not select children of an existing field.

Later appends must match the canonical types and nullable annotations of every provided field, subject to the existing lossless scalar widenings. An array without null values can still have a nullable field; keep nullable=True for an added field even when that batch supplies every value. After the first addition, any nullable payload may be omitted and becomes null. Every key remains required. A table without an explicit addition retains baseline strict missing-field behavior.

Re-register queries and handle conflicts¶

Call Session.register_tstable again with the same name and root after each addition. It replaces the registration and exposes the latest schema. Already built query plans keep their snapshots; newly planned scans through a stale registration report a schema change. A query naming a new field may instead fail earlier during name resolution.

add_columns uses the current handle's version and does not refresh or retry. On ConflictError, reopen the table, inspect the current state, and reconcile the request before retrying. If a TimeseriesTableError reports an ambiguous commit, neither success nor rollback is guaranteed; reopen and reconcile the log before taking further mutation steps.

Adding fields requires a compatible client. The first successful addition declares a reader feature atomically, so older clients reject the evolved table. Installing this release and ordinary appends do not activate that feature. See Table protocol compatibility for the persistent contract and TimeSeriesTable reference for argument and error details.