Skip to main content

io.Batch

What a Source read: new rows and the cursor to read from next time.

Usage

Source

io.Batch(
    table,
    cursor=None,
    kind="append",
    deletes=None,
)

Attributes

table: Any

The rows, as an Arrow table, a Polars or pandas DataFrame, a Polars LazyFrame, or anything else pyarrow.table accepts. What they are depends on kind.

cursor: Any

Where the next read starts: read(since=cursor). Any value, such as the greatest timestamp read or the files read so far. Default: None.

kind: Kind

"append": table is rows added since the last read (the default). "snapshot": table is the whole table; the source diffs it against the last read. "changes": table is rows added and deletes rows removed (one copy each).

deletes: Any
With kind="changes", the rows removed, in the same forms as table. Default: None.