io.from_files()

A source that reads the files dropped in a directory: each refresh

Usage

io.from_files(
    path,
    *,
    format="parquet",
    name,
    schema=None,
    filesystem=None,
)

reads the files that are new since the last, and appends their rows. The cursor is the list of files read.

This assumes files are only added: a file changed or removed after it was read is not noticed, except by refresh(audit=True), which reads the files read so far again and compares.

Parameters

path: str: The directory, as a path or a URI such as s3://bucket/drop.
format: str = "parquet": The files’ format and suffix, as pyarrow.dataset names it: "parquet" (the default), "csv", "feather", "json".
name: str: The table’s name in the plans that read it.
schema: pa.Schema | None = None: The table’s schema, if the directory may start empty. Default: that of the first files.
filesystem: Any = None: A pyarrow.fs.FileSystem to read path with, such as one with credentials. Default: the one path’s URI names, with its default credentials (environment, profile, role).

Returns

Source: The source.