io.IcebergSource
An Iceberg table of a catalog, read by snapshot sequence number, as
Usage
io.IcebergSource(
catalog, identifier, *, where=None, name=None, on_violation=None
)the input of transforms. name is the table’s name in the plans that read it (default: the last part of identifier).
A refresh reads the files that appends added since the version a node’s state was computed from; after row-level deletes or overwrites it diffs the data files that changed. A table dropped and created again under identifier (a new table UUID) is read in full. Tables with equality deletes cannot be read.
With where (a SQL condition on the columns), the source is the rows on which it is TRUE: snapshots read only the data files whose statistics do not rule it out, and changes are filtered.
Parameters
catalog: Catalog-
A pyiceberg catalog.
identifier: str-
The table’s identifier in it, e.g.
"db.readings". where: str | None = None-
A SQL condition on the columns. The source is the rows on which it is TRUE; data files are skipped only where Lean proves from their statistics that it is TRUE on none of their rows (
Rel.eval_prunedinSubstraitLean/Pruning.lean). Default: None, every row. name: str | None = None-
The table’s name in the plans that read it. Default: the last part of
identifier. on_violation: _quarantine.Policy | None = None-
What a refresh does with a batch that breaks a precondition on this source, for transforms that set no
on_violationof their own:"error","quarantine"or"hold"(see sverify.transform). Default: None,"error".
Attributes
catalog-
The catalog.
identifier-
The table’s identifier.
where-
The SQL condition, or None.
name-
The table’s name in the plans that read it.
skipped: list[str]- The data files the last snapshot did not read.
Methods
| Name | Description |
|---|---|
| changes() |
The change from sequence number since to until, as a refresh
|
| contract() |
The contract the table property sverify.contract records.
|
| latest() | The current snapshot’s sequence number (0 before the first). |
| schema() | The table’s current schema, as Arrow. |
| snapshot() | The table’s rows at a snapshot, with what its metadata says about |
| stored() | The node state the current snapshot holds, if an IcebergSink |
| table() |
The table as the catalog loads it now (a pyiceberg.table.Table).
|
changes()
The change from sequence number since to until, as a refresh
Usage
changes(since, until=None)reads it.
Parameters
since: int-
The version the node’s state was computed from.
until: int | None = None- A snapshot sequence number to read up to. Default: the current snapshot.
Returns
SourceBatch-
The rows of the files appends added (trusted), or the diff of
the data files that changed (exact); or, where snapshot
is gone or unreadable, the snapshot atsinceuntiland the reason.
Raises
ValueError-
No ancestor has sequence number
until.
contract()
The contract the table property sverify.contract records.
Usage
contract(version=None)Parameters
version: int | None = None- A snapshot’s sequence number: the contract the table recorded while that snapshot was current, read from the metadata files the table’s metadata log keeps. Default: the current one.
Returns
Contract | None-
The contract, or None if the table had none then (or its
metadata file is gone).
latest()
The current snapshot’s sequence number (0 before the first).
Usage
latest()schema()
The table’s current schema, as Arrow.
Usage
schema()snapshot()
The table’s rows at a snapshot, with what its metadata says about
Usage
snapshot(version=None)every row (required fields, file statistics, and the contract of an IcebergSink that wrote every snapshot).
Parameters
version: int | None = None- A snapshot sequence number of the current snapshot’s ancestry; 0 is the empty table before the first snapshot. Default: the current snapshot.
Returns
Snapshot- The version, the rows and the facts.
Raises
ValueError- No ancestor has that sequence number.
stored()
The node state the current snapshot holds, if an IcebergSink
Usage
stored()wrote it under the table’s contract: the current snapshot is the sink’s, or the snapshots after the sink’s last one only rewrite files (replace, as compaction does). A restarted job’s first refresh resumes the sink’s node from it, when it was stored by the same plan from the same sources.
Returns
Stored | None-
The stored state, or None if a snapshot after the sink’s last
one changed rows.
table()
The table as the catalog loads it now (a pyiceberg.table.Table).
Usage
table()