io.Storage
Options for the object store tables live in (credentials, region,
Usage
io.Storage(options=None)endpoint), given once and passed to every source and sink that needs them: DeltaSource(uri, storage_options=s3).
options is either a fixed mapping or a function returning a fresh one, called every time a table is opened, so a long-running job can renew an expiring token. sverify keeps no result of that function, and repr shows option names, never values. Prefer what the store’s default chain finds (environment variables, ~/.aws profiles, instance or pod roles) or a function that fetches short-lived credentials over literal keys, which travel with any object holding them.
The options are delta-rs’s (deltalake’s storage_options), such as AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, GOOGLE_SERVICE_ACCOUNT or AZURE_STORAGE_ACCOUNT_NAME. A source or sink of your own can take a Storage too and call options().
Parameters
options: Mapping[str, str] | Callable[[], Mapping[str, str]] | None = None- The options, or a function returning them. Default: none, the store’s default chain.
Examples
import boto3
from sverify.io import DeltaSink, DeltaSource, Storage
def credentials():
c = boto3.Session().get_credentials().get_frozen_credentials()
return {
"AWS_REGION": "eu-central-1",
"AWS_ACCESS_KEY_ID": c.access_key,
"AWS_SECRET_ACCESS_KEY": c.secret_key,
"AWS_SESSION_TOKEN": c.token or "",
}
s3 = Storage(credentials)
orders = DeltaSource("s3://lake/orders", storage_options=s3)
sink = DeltaSink("s3://lake/paid", storage_options=s3)Methods
| Name | Description |
|---|---|
| options() | The options now: the mapping, or what the function returns. |
options()
The options now: the mapping, or what the function returns.
Usage
options()Returns
dict[str, str]- A new dict of options.