io.SourceBatch

What a versioned source has between a recorded version and its latest

Usage

io.SourceBatch(
    start,
    end,
    snapshot_facts,
    changes=None,
    facts=list(),
    trusted="",
    kind="change_feed",
    snapshot=None,
    fallback=None,
    schema=None
)

one: what VersionedSource.changes returns.

Either changes from the format’s change feed (the net change, with facts about the inserted rows and the trusted assumption it rests on), or the new snapshot alone with the reason the feed is unavailable (fallback). Build one with unchanged, from_changes or from_snapshot.

Attributes

start: int: The version the batch starts from (since).
end: int: The version it reaches.
snapshot_facts: list[Fact]: What the metadata says about every row at end.
changes: Changes | None: The change, or None with snapshot.
facts: list[Fact]: What the metadata says about the inserted rows.
trusted: str: What the change rests on, for the refresh report.
kind: Literal["change_feed", "snapshot_diff"]: "change_feed" (trusted) or "snapshot_diff" (exact).
snapshot: pa.Table | None: The table at end, when there is no change.
fallback: str | None: Why there is no change, with snapshot.
schema: pa.Schema | None: The table’s schema at end; None: the one recorded.

Methods

Name	Description
from_changes()	The change from `since` to `until`, as rows inserted and deleted.
from_snapshot()	No change is available from `since`: the whole table at `until`
unchanged()	No change: the table is still at `version`.

from_changes()

The change from since to until, as rows inserted and deleted.

Usage

Source

from_changes(
    since,
    until,
    *,
    inserts=(),
    deletes=(),
    trusted,
    facts=(),
    snapshot_facts=(),
    kind="change_feed",
    schema=None
)

Parameters

since: int: The version the change starts from.
until: int: The version it reaches.
inserts: pa.Table | Sequence[Row] = (): Rows added: an Arrow table, or tuples in column order.
deletes: pa.Table | Sequence[Row] = (): Rows removed, one copy each, in the same forms.
trusted: str: What the change rests on, shown in the refresh report, such as "Δ trusted as coming from Delta CDF v3→v5".
facts: Sequence[Fact] = (): What the metadata says about the inserted rows. Default: none (preconditions are checked on the rows).
snapshot_facts: Sequence[Fact] = (): What the metadata says about every row at until. Default: none.
kind: Literal["change_feed", "snapshot_diff"] = "change_feed": "change_feed" (the change is trusted) or "snapshot_diff" (it was computed from two snapshots, so it is exact).
schema: pa.Schema | None = None: The table’s schema at until, which the rows have. Default: the one recorded.

Returns

SourceBatch: The batch.

from_snapshot()

No change is available from since: the whole table at until

Usage

Source

from_snapshot(since, until, table, *, reason, snapshot_facts=(), schema=None)

instead. The refresh diffs it against the state, and recomputes the nodes reading it.

Parameters

since: int: The version the change would start from.
until: int: The version of table.
table: pa.Table: The table at until.
reason: str: Why there is no change, shown in the refresh report.
snapshot_facts: Sequence[Fact] = (): What the metadata says about every row of table. Default: none.
schema: pa.Schema | None = None: The table’s schema at until. Default: table’s.

Returns

SourceBatch: The batch.

unchanged()

No change: the table is still at version.

Usage

Source

unchanged(version, facts=(), schema=None)

Parameters

version: int: The version asked for as since, and still the latest.
facts: Sequence[Fact] = (): What the metadata says about every row. Default: none.
schema: pa.Schema | None = None: The table’s schema. Default: the one recorded.

Returns

SourceBatch: A batch from version to version.