Skip to main content

Polars facts

103 facts: what the engine does, with provenance

Each fact records what the engine does at a pinned version, what the reference semantics does, and the checks behind it, from characterization/ (the format). Evidence runs from docs to source, engine_test, probe, fuzz and proof.

integers

polars-int-abs: abs of an integer: abs(MIN) is MIN

Engine: polars py-2.1.0; area: integers; status: open; evidence: probe.

Verdict: wraps at MIN; the model’s abs takes overflow: ERROR only

abs is wrapping_abs: abs(MIN) is MIN at every width. Every other value is exact.

The docs are silent on abs(MIN): a docs gap.

The reference: abs with overflow: ERROR, the only value the model accepts for it (SILENT and SATURATE are refused), so abs(MIN) is an overflow issue.

Options: overflow.

Rows of the choice tables: int.abs overflow (i8, i16, i32, i64).

Constructs: abs.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe Int8.abs() in characterization/polars/probes-streaming.json - probe Int16.abs() in characterization/polars/probes-streaming.json - probe Int32.abs() in characterization/polars/probes-streaming.json - probe Int64.abs() in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.abs.html silent on abs(MIN) - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_abs.py#L80-L83 test_abs_overflow_wrapping: Int8 abs(-128) = -128 (eager, in memory)

polars-int-arith: Integer + - *, unary - and **: wrap at the column’s width

Engine: polars py-2.1.0; area: integers; status: open; evidence: probe.

Verdict: matches overflow: SILENT at every width

Integer arithmetic wraps at the operands’ width (two’s complement, Rust’s wrapping_add, wrapping_sub, wrapping_mul, wrapping_neg): Int8 127 + 1 is -128, -MIN is MIN. No setting makes it raise or saturate. Polars keeps the width of the operands; a literal takes the column’s type.

** on integers wraps too (2 ** 63 is -2⁶³ in Int64); a negative integer exponent raises (InvalidOperationError, the exponent is cast to u32).

The docs are silent on integer overflow: a docs gap.

The reference: add, subtract, multiply, negate with overflow: ERROR when the plan omits it (#202); SILENT when the plan asks. Polars gives SILENT natively; ERROR needs a guard, at each width.

Options: overflow.

Rows of the choice tables: int.arith overflow (i8, i16, i32, i64); int.pow overflow (i64); int.pow overflow (i8, i16, i32; evidence source).

Constructs: +, -, *, unary -, **.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-11; issues #202.

Provenance:

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L72-L106 pow_to_uint_dtype: an exponent literal of 2 to 10 multiplies the column by itself, any other calls Pow::pow. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L108-L150 pow_on_series: an integer base with an integer exponent, for every width by with_match_physical_integer_type!. - https://github.com/rust-num/num-traits/blob/num-traits-0.2.19/src/pow.rs#L73-L99 num-traits 0.2.19 (Polars’s Cargo.lock): Pow<u32> for i8, i16, i32, i64 is the primitive pow. - https://github.com/pola-rs/polars/blob/py-2.1.0/Cargo.toml#L210-L238 Release profiles: none turns overflow-checks on. - https://github.com/pola-rs/polars/blob/py-2.1.0/.github/workflows/release-python.yml#L293 Wheels build with --profile dist-release. - 2026-11-04, engine-test, by characterization pipeline round 3 (integers): The tests pin wrapping + - * and negation at Int8. For ** they pin that the result keeps the base’s width at every signed width, but no input overflows, so the int.pow overflow rows for i8 to i32 stay at the source reading.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L109-L118 test_list_arithmetic_values (Int8): -5 - 127 is 124 and -5 * 127 is -123 (wrapped), on columns, broadcasts and lists, eager and lazy - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_neg.py#L49-L52 test_neg_overflow_wrapping: -(-128) in Int8 is -128 (eager DataFrame.select, in memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L696-L761 test_power_series: Int8 ** UInt8, Int16 ** UInt8, Int32 ** UInt8 and Int64 ** UInt8 keep the base’s type (1, 4); Int64 ** UInt64(2**33) raises conversion from u64 to u32 failed (eager) - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.add.html add/sub/mul/neg docstrings silent on integer overflow - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/expr.py#L6963-L7000 Expr.pow: an integer to a positive integer is an integer; cast to float for a negative exponent

polars-int-div-domain: Integer // has no domain error

Engine: polars py-2.1.0; area: integers; status: no-finding; evidence: engine_test.

Verdict: every input is in the domain; zero divisors and MIN // -1 are other options

Integer floor division gives a quotient for every pair of operands but two kinds: a zero divisor is NULL (on_division_by_zero) and MIN // -1 wraps to MIN (overflow), both in polars-int-divide. No input raises or gives a value of its own. The kernels are one macro for Int8 to Int64. / on integers divides doubles (polars-int-true-divide).

The reference: divide on integers: on_domain_error never applies (int.div in docs/core-semantics.md, Choice.any).

Options: on_domain_error.

Rows of the choice tables: int.div on_domain_error (i8); int.div on_domain_error (i16, i32, i64; evidence source).

Constructs: //, floordiv.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, source-reading, by characterization pipeline round 2 (integers): One macro instantiates the integer kernels for i8 to i64, so the reading covers every width. Each path handles a zero divisor and MIN // -1 and returns a quotient for every other input.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L35-L45 Column by column: a zero divisor is masked to NULL, the rest is wrapping_floor_div_mod. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L103-L140 A literal divisor: 0 gives NULL, -1 negates (wrapping), others divide the absolute values. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L39-L68 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L230-L234 - 2026-11-04, engine-test, by characterization pipeline round 3 (integers): The Rust test shows the floor-division kernel gives a value for every Int8 pair (the zero divisor’s (0, 0) is masked to NULL by the caller), so no Int8 input raises. test_floordiv_mod pins NULL for a zero divisor at Int8 and Int32. i16 and i64 rest on the source reading of the shared macro.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L86-L109 Rust test_signed_wrapping_div_mod: for every i8 pair, wrapping_floor_div_mod equals the floor quotient and remainder of the doubles, wrapped to i8 ((x as i32) as i8), and (0, 0) for a zero divisor. So -128 // -1 is -128, -128 % -1 is 0, and no pair panics. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/expr/test_literal.py#L142-L188 test_floordiv_mod: lit(1, Int8) // lit(0, Int8) and % are NULL, as for the default Int32 literals; 10 // 3 is 3. Eager pl.select, in memory - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L247-L259 test_truncdiv_int (Int8, lazy collect()): truncdiv gives -128 for -128 / -1 and NULL for 5 / 0; its comment says floordiv does the same

polars-int-divide: Integer // floors, truncdiv truncates; NULL for a zero divisor

Engine: polars py-2.1.0; area: integers; status: open; evidence: probe.

Verdict: // floors, truncdiv truncates; x // 0 is NULL; MIN // -1 wraps

// on integers is floor division (wrapping_floor_div_mod): -7 // 2 is -4, 1 // -2 is -1. A zero divisor gives NULL (the result’s validity masks it), and MIN // -1 wraps to MIN.

Expr.truncdiv divides rounding toward zero (Operator::TruncDivide, the kernel prim_wrapping_trunc_div), on both engines: -7 truncdiv 2 is -3. It shares the rest with //: a zero divisor gives NULL and MIN truncdiv -1 wraps to MIN. The result type is that of //.

So sverify could translate Substrait’s integer divide to truncdiv in place of its floor-division shim (int_div_trunc in _polars_shims.py), keeping the guards for a zero divisor and MIN / -1. / on integers is a division of doubles (polars-int-true-divide).

The docs are silent on a zero divisor giving NULL and on MIN // -1 wrapping: a docs gap.

The reference: divide on integers truncates toward zero, with on_division_by_zero: ERROR when the plan omits it (NULL when asked) and overflow: ERROR at MIN / -1. Polars’s // differs on every quotient of operands of opposite signs that is not exact (26 of the 121 rows of the NULL candidate).

Options: division_type, on_division_by_zero, overflow.

Rows of the choice tables: int.div overflow (i8, i16, i32, i64); int.div on_division_by_zero (i8, i16, i32, i64); int.div division_type (i8, i16, i32, i64).

Constructs: //, floordiv, truncdiv.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, source-reading, by session brave-borg-41f7cc: Read at tag py-2.1.0 (commit 7f902bd3).
  • 2026-11-04, engine-test, by characterization pipeline round 3 (integers): Contradicts the description’s second paragraph: Polars 2.1.0 has a truncating integer division in Python, Expr.truncdiv (Operator::TruncDivide), on both engines. Its tests pin truncation toward zero, MIN truncdiv -1 wrapping to MIN and NULL for a zero divisor, which matches Substrait’s divide on integers except for the overflow and division-by-zero options. The description is left as it is for review.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L247-L259 test_truncdiv_int: Int8 a.truncdiv(b) of 7, -7, 7, -7, -128, -128, 5, NULL by 2, 2, -2, -2, 2, -1, 0, 1 is 3, -3, -3, 3, -64, -128, NULL, NULL (lazy collect()) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L262-L269 test_truncdiv_signed_min: Int64 MIN truncdiv 2 and 3 truncate toward zero - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/expr.py#L6756-L6795 Expr.truncdiv: divides, rounding toward zero; engine support in-memory, streaming, distributed; same result type as floordiv - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/expr/test_literal.py#L142-L188 test_floordiv_mod: // and % of Int8 and Int32 literals, a zero divisor gives NULL (eager, in memory) - 2026-11-04, manual-repro, by characterization pipeline round 4 (streaming probes): Corrected the description, title, verdict, constructs and settings. Round 1 said the Python API has no truncating integer division; Polars 2.1.0 has Expr.truncdiv (Operator::TruncDivide), documented for in-memory, streaming and distributed, and its tests (test_truncdiv_int, test_truncdiv_signed_min, round 3) pin truncation toward zero, MIN truncdiv -1 wrapping to MIN and NULL for a zero divisor. The repro above confirms both engines, with a column and a literal divisor. The rows are unchanged: // and truncdiv share the overflow and division-by-zero behaviour.

Polars 2.1.0 (Python), macOS arm64. - pl.LazyFrame({'a': [7, -7, 7, -7, -128, 5, None], 'b': [2, 2, -2, -2, -1, 0, 1]}, schema={'a': pl.Int8, 'b': pl.Int8}).select(pl.col('a').truncdiv(pl.col('b')), pl.col('a').truncdiv(2).alias('lit')).collect(engine=e).rows() for e in ('in-memory', 'streaming') gives both engines: [(3, 3), (-3, -3), (-3, 3), (3, -3), (-128, -64), (None, 2), (None, None)] - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L247-L269 test_truncdiv_int and test_truncdiv_signed_min - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/expr.py#L6756-L6795 Expr.truncdiv - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/expressions/binary.rs#L160-L186 FloorDivide/TruncDivide dispatch to floor_div_series/trunc_div_series - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/int_divide.rs#L32-L98 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L39-L68 wrapping_div then -1 on a nonzero remainder of the other sign - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.truncdiv.html states floordiv rounds down by contrast; silent on a zero divisor (NULL) and MIN // -1 - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L86-L109 #[test] test_signed_wrapping_div_mod: all i8 pairs, floored and wrapped (-128 // -1 = -128) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L121-L122 Int8 -5 // 3 = -2 (eager and lazy, both engines; slow) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L238-L243 Int32 // 5 matches numpy’s floor (eager)

polars-int-mod-overflow: Integer % never overflows

Engine: polars py-2.1.0; area: integers; status: no-finding; evidence: engine_test.

Verdict: no overflow at any width; MIN % -1 is 0

% on integers has no overflowing input. The kernel takes wrapping_rem and, when the remainder is nonzero and its sign differs from the divisor’s, adds the divisor: the sum lies strictly between 0 and the divisor, so it fits. MIN % -1 is 0 (wrapping_rem), and a literal divisor of ±1 fills 0. One macro instantiates the kernels for Int8 to Int64, so every width behaves the same. The floor remainder and NULL for a zero divisor are in polars-int-modulus.

The reference: modulus on integers is the truncated remainder (Int.tmod), which never overflows either; MIN % -1 is 0 (min_mod_neg_one).

Options: overflow.

Rows of the choice tables: int.mod overflow (i8); int.mod overflow (i16, i32, i64; evidence source).

Constructs: %, mod.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, source-reading, by characterization pipeline round 2 (integers): The kernels are one generic macro, so the reading covers i8 to i64. The adjustment mod_ += other only runs when the remainder and the divisor have opposite signs, where the sum cannot overflow. The wheels build without overflow-checks (polars-int-arith), so the wrapping operations cannot panic either.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L39-L68 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L76-L80 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L60-L70 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L174-L208 A literal divisor: the remainder of the absolute values, below |divisor|, with the divisor’s sign. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L230-L234 impl_signed_arith_kernel! for i8, i16, i32, i64, i128. - 2026-11-04, engine-test, by characterization pipeline round 3 (integers): The Rust unit test runs wrapping_floor_div_mod, the kernel of % and // between two integer columns, on all 65 536 Int8 pairs: MIN % -1 is 0 and every remainder fits. It is a test of the kernel, below either engine; both engines call it. It covers i8 only (its comment says the result should transfer to other widths), so i16 to i64 stay at the source reading of the macro. The path of a literal divisor (prim_wrapping_mod_scalar, strength reduction) is not covered by it.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L86-L109 Rust test_signed_wrapping_div_mod: for every i8 pair, wrapping_floor_div_mod equals the floor quotient and remainder of the doubles, wrapped to i8 ((x as i32) as i8), and (0, 0) for a zero divisor. So -128 // -1 is -128, -128 % -1 is 0, and no pair panics. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L109-L124 test_list_arithmetic_values (Int8): -5 // 3 is -2, -5 % 3 is 1, through the column, broadcast and list paths

polars-int-modulus: Integer %: the floor remainder, NULL for a zero divisor

Engine: polars py-2.1.0; area: integers; status: open; evidence: probe.

Verdict: division_type: FLOOR; x % 0 is NULL; MIN % -1 is 0

% on integers is the remainder of floor division, with the sign of the divisor (-1 % 2 is 1, 1 % -7 is -6). A zero divisor gives NULL, and MIN % -1 is 0.

The docs are silent on the remainder’s sign, a zero divisor and MIN % -1: a docs gap.

The reference: modulus with division_type: TRUNCATE (the remainder has the dividend’s sign); FLOOR is refused by the model. on_division_by_zero: ERROR when omitted. So the reference matches Polars only where the operands have the same sign or the remainder is 0.

Options: division_type, on_division_by_zero, min_mod_neg_one.

Rows of the choice tables: int.mod on_division_by_zero (i8, i16, i32, i64); int.mod division_type (i8, i16, i32, i64); int.mod min_mod_neg_one (i8, i16, i32, i64).

Constructs: %, mod.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L39-L68 wrapping_floor_div_mod: wrapping_div and wrapping_rem, then the divisor is added to a nonzero remainder whose sign differs from it. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L76-L80 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L60-L70 Column by column: a zero divisor is masked to NULL. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L174-L208 A literal divisor: 0 gives an all-NULL column, ±1 fills 0, any other a strength-reduced unsigned remainder with the divisor’s sign. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/signed.rs#L230-L234 - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.mod.html silent on the remainder’s sign, x % 0 (NULL) and MIN % -1 - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L86-L109 #[test] test_signed_wrapping_div_mod: all i8 pairs, remainder with the divisor’s sign, -128 % -1 = 0 - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L124-L125 Int8 -5 % 3 = 1 (eager and lazy, both engines; slow) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/expr/test_literal.py#L142-L188 test_floordiv_mod: 1 % 0 is NULL (eager, in memory)

polars-int-pow-exponent: Integer **: a negative exponent fails the query

Engine: polars py-2.1.0; area: integers; status: no-finding; evidence: probe.

Verdict: raises InvalidOperationError for a negative exponent

With an integer base and an integer exponent, ** first casts the whole exponent column to UInt32 with strict_cast. A negative exponent anywhere in it, or one above 2³² − 1, fails the query with InvalidOperationError and a hint to cast to float first. The cast does not depend on the base’s width. Every other input gives a value (0 ** 0 is 1); overflow wraps (polars-int-arith).

A float exponent turns the base into the exponent’s float type, and the result is that type.

The reference: power on integers: a negative exponent raises (negExponent in Safety.lean), and on_domain_error never applies (int.pow in docs/core-semantics.md). An exponent above 64 with |base| > 1 is an overflow, which raises; with |base| ≤ 1 the reference computes it, where Polars raises from 2³² on.

Options: on_domain_error.

Rows of the choice tables: int.pow on_domain_error (i64); int.pow on_domain_error (i8, i16, i32; evidence source).

Constructs: **, pow.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): Int64 base, ** -1: InvalidOperationError: conversion from i32 to u32 failed.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (integers): The exponent’s cast runs before the base’s type is looked at (with_match_physical_integer_type! dispatches the base, for every width), so the Int64 probe holds for Int8, Int16 and Int32.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L108-L150 An integer exponent is strict_cast to UInt32, with the hint in the error. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L72-L106 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L909-L918 pow_dtype: an integer base keeps its type unless the exponent is a float. - 2026-11-04, engine-test, by characterization pipeline round 3 (integers): The test pins the strict cast of the exponent to UInt32 for an Int64 base only, and has no negative exponent. The i8 to i32 rows stay at the source reading.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L696-L761 test_power_series: Int64 ** UInt64 with 2**33 raises InvalidOperationError conversion from u64 to u32 failed; an exponent that fits succeeds; a ** lit(None) raises (null exponent dtype). Eager Series

polars-int-shift: No bit-shift expressions

Engine: polars py-2.1.0; area: integers; status: open; evidence: probe.

Verdict: no << or >> on expressions

Polars’s Python expressions have no bit shift: pl.Expr has no __lshift__ or __rshift__, and its bitwise_* methods are and, or, xor and bit counts. shift moves rows.

The reference: shift_left and shift_right (functions_arithmetic.yaml), with the range open (ST-1). A plan with them is refused on Polars or written as a multiplication or floor division by a power of two.

Options: shift_range.

Rows of the choice tables: int.shift shift_range (i8, i16, i32, i64; evidence docs).

Constructs: shift_left, shift_right.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-1.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (integers): The API reference’s expression operators list &, |, ^ and ~ as the bitwise operators, with no shift; the bitwise_* methods are and, or, xor and bit counts, and Expr.shift moves rows. The same holds for every integer type, so the shift rows are settled: Polars has no shift to characterize.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observation reads the Python API (dir(pl.Expr)), which both engines share.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe int-shift in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/dsl/expr/mod.rs#L747-L773 Operator: no shift; polars-sql has none either

polars-int-true-divide: Integer /: a division of doubles

Engine: polars py-2.1.0; area: integers; status: open; evidence: probe.

Verdict: Int64 / Int64 is Float64; / 0 is ±inf or NaN; a literal divisor multiplies by its reciprocal

/ on integers casts both operands to Float64 and divides: 7 / 2 is 3.5, x / 0 is ±inf, 0 / 0 is NaN. Integers above 2⁵³ round on the way.

The cast happens before the division (binary.rs#L151-L158), so the float kernels run, and a literal or one-row divisor multiplies by its reciprocal (polars-double-scalar-divisor, PL-3): on a frame of two rows or more 49 / 49 is 0.9999999999999999 and 5 / 3 is 1.6666666666666665, and so is x / x.max() for x = 49.

The docs are silent on the rounding above 2⁵³: a docs gap. The truediv docs call the division IEEE 754, which the reciprocal contradicts: PL-8.

No upstream test pins quotients above 2⁵³ or an inexact quotient by a literal; they may change between versions.

The reference: Substrait’s divide on integers returns the integer type and truncates. A plan that divides integers to a double casts first, and the double division’s options apply (polars-double-divide).

Options: implicit_casts, on_division_by_zero.

Constructs: /, truediv.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-8.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, source-reading, by session brave-borg-41f7cc: Read at tag py-2.1.0 (commit 7f902bd3).
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe int-true-divide in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass: Corrected the route: an earlier check cited prim_true_div of signed.rs, which Expr / does not reach. binary.rs#L151-L158 casts both sides to Float64 first, so a literal divisor takes the float reciprocal kernel; added to the description and the verdict. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/expressions/binary.rs#L151-L158 integer / integer: both cast to Float64, then the float kernel; signed.rs’s prim_true_div is not on this route - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L109-L119 a length-1 divisor multiplies by its reciprocal - 2026-11-04, manual-repro, by characterization pipeline reverse pass - pl.DataFrame({'a': [49, 5, 0]}).select(pl.col('a') / 49, pl.col('a') / 3, pl.col('a') / 0) gives 49/49 = 0.9999999999999999, 5/3 = 1.6666666666666665, 49/0 = inf, 0/0 = nan (Polars 2.1.0, macOS arm64) - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L190-L196 UInt8 1 / 0 = inf, 0 / 0 = NaN (eager and lazy, both engines; slow) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/series/test_series.py#L1218-L1229 [1, 2] / 2 = [0.5, 1.0] (eager)

decimals

polars-decimal-arith: Decimals: 38 digits, the larger scale, half to even

Engine: polars py-2.1.0; area: decimals; status: shimmed; evidence: probe.

Verdict: Decimal(38, max(s₁, s₂)); rounds half to even; raises on overflow and on / 0

Arithmetic on decimals returns Decimal(38, max(s₁, s₂)) for +, -, * and / (an integer operand counts as scale 0), and sum returns Decimal(38, s): the result’s precision is always 38, so Decimal(6, 2) * 1000 gives 9999990.00 and overflows only past 38 digits, where it raises (ComputeError).

A result with more digits than the scale rounds half to even: 0.05 * 0.5 is 0.02, 1.00 / 8 is 0.12, a cast from scale 3 to scale 1 makes 1.050 into 1.0 and 1.150 into 1.2, and a double cast to a decimal rounds half to even on its exact value (0.625 is 0.62). Division by zero raises. mean of decimals is a Float64.

The docs are silent on the result’s precision and scale and on the half-to-even rounding: a docs gap.

The reference: functions_arithmetic_decimal.yaml gives the result types (multiply: scale s₁ + s₂, divide: max(6, s₁ + p₂ + 1), precision capped at 38 by borrowing scale) and no rounding; the reference rounds half away from zero (decimal_rescale). Decimal division is refused until #27. ST-8 records the open rounding. The translation computes a product on the unscaled values at scale s₁ + s₂ and checks each result against the plan’s precision (fit_decimal); AVG and sverify_decimal_div divide half away from zero (decimal_div_half_away, Polars.decimal_div_half_away); an empty SUM is NULL (sum_empty_null).

Options: overflow, decimal_rescale, decimal_result_type, raise_set.

Rows of the choice tables: dec.arith overflow (decimal); dec.arith decimal_rescale (decimal).

Constructs: +, -, *, /, sum, mean, cast.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-8; issues #27.

Provenance:

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe decimal-result-types in characterization/polars/probes-streaming.json - probe decimal-rounding in characterization/polars/probes-streaming.json - probe decimal-overflow in characterization/polars/probes-streaming.json - probe decimal-precision-38 in characterization/polars/probes-streaming.json - probe decimal-divide-by-zero in characterization/polars/probes-streaming.json - probe decimal-mean in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/sum.rs#L134-L215 streaming decimal sum: i128 checked_add with a sticky overflow sentinel; finish() raises ‘overflow in decimal addition in sum’ past 38 digits. - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_decimal.py#L333-L357 test_decimal_arithmetic: Decimal(20, 10) + - * / all give Decimal(38, 10); 0.1 / 20.1 == 0.0049751244, 10.1 * 10.19 == 102.919; eager, in memory. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_decimal.py#L1221-L1230 test_decimal_mul_rounds_half_even: [0.25, 0.35, -0.25, -0.35] * 0.5 == [0.12, 0.18, -0.12, -0.18] as Decimal(38, 2), eager. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_decimal.py#L784-L794 test_decimal_arithmetic_schema_int: an integer operand counts as scale 0, result Decimal(38, 6); s / 3 == 0.333333. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_decimal.py#L1104-L1135 sum to Decimal(38, s), overflow past 38 digits raises ComputeError, partial sums past 38 digits allowed; parametrized engine in [‘streaming’, ‘in-memory’]. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_decimal.py#L1438-L1446 test_decimal_div_by_zero_mixed_scale: 1.5 / 0.000 raises ComputeError ‘division by zero’. - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/api/polars.datatypes.Decimal.html Precision 38 is the default and the maximum (SRC py-polars/src/polars/datatypes/classes.py L447-L459); silent on result precision and scale of arithmetic and on its rounding. - 2026-11-05, source-reading, by session polars-decimals: Read at tag py-2.1.0 (commit 7f902bd3). tests/test_polars_decimal.py runs each decimal operation’s written plan on Polars and DuckDB against Rel.eval. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/decimal.rs#L813-L831 dec128_mul_scaled: the product of the unscaled values on 256 bits (i64 × i64 on 128), then rounded to the result’s scale half to even: no intermediate overflow. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/decimal.rs#L489-L493 dec128_fits: the precision check against 10^p.

polars-decimal-casts: Decimal casts: half to even, correctly rounded to double, truncated to integer

Engine: polars py-2.1.0; area: decimals; status: shimmed; evidence: probe.

Verdict: a smaller scale and a double round half to even; to a double correctly rounded; to an integer truncates; past the precision raises

A decimal cast to a smaller scale rounds half to even (1.050 at scale 1 is 1.0, 1.150 is 1.2). A double cast to Decimal(p, s) is x · 10ˢ in doubles, rounded half to even (0.625 is 0.62, 0.125 is 0.12, -0.375 is -0.38), with the double 10ˢ exact only up to s = 22; NaN and ±∞ raise.

A decimal cast to Float64 is correctly rounded: one division where the unscaled value and 10ˢ are exact doubles, else the decimal’s text through Rust’s float parser. A decimal cast to an integer truncates toward zero, and raises outside the integer’s range (strict) or is NULL (strict=False).

A cast to a smaller precision raises where the value does not fit (strict) or is NULL. to_physical gives the unscaled value as an Int128 (1.25 at scale 2 is 125), and Int128 arithmetic wraps silently (2¹²⁶ · 2 is -2¹²⁷).

The reference: decimal_rescale: TIE_AWAY_FROM_ZERO (roundDiv10, UnFn.dround); a double to a decimal is x · 10ˢ rounded once to a double, then half away from zero (FloatCode.toDecimal?); a decimal to a double is correctly rounded (FloatCode.ofDecimal); to an integer truncates (UnFn.dtrunc). The translation shims the halves (decimal_round_half_away, round_half_away on the double product) and refuses a double cast to more than 22 decimal places.

Options: decimal_rescale, float_to_int, cast_failure.

Rows of the choice tables: dec.arith decimal_rescale (decimal).

Constructs: cast, to_physical.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #27.

Provenance:

polars-decimal-compare: Decimal comparisons: exact across scales and with integers

Engine: polars py-2.1.0; area: decimals; status: no-finding; evidence: probe.

Verdict: exact

== and < compare decimals of different scales, and a decimal with an integer, by value: 12.35 at scale 2 equals 12.350 at scale 3, and -0.05 < -0.051 is false. A rescale that would overflow decides by the sign instead.

sum of no decimals, or only NULLs, is 0 at the column’s scale (as for integers, polars-sum-int); min of only NULLs is NULL.

The reference: Comparisons at the larger scale, exactly (docs/reference-semantics.md, Decimals); SUM of no values is NULL (empty_group_aggregates), which sum_empty_null gives.

Constructs: ==, <, >, min, max, sum.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #27.

Provenance:

doubles

polars-atan2: arctan2(0, 0) is 0

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: 0, where the reference raises

pl.arctan2(y, x) is Rust’s atan2: atan2(0, 0) is 0 and the infinities give the IEEE angles. It is the only edge case that differs.

The docs are silent on atan2(0, 0) and the infinities: a docs gap.

No upstream test pins this; it may change between versions.

The reference: atan2 with on_domain_error: ERROR, which raises at (0, 0); Substrait leaves its domain open (ST-2).

Options: on_domain_error.

Rows of the choice tables: fp.elem on_domain_error (fp64); fp.elem on_domain_error (fp32; evidence source).

Constructs: arctan2.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-2.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): atan2 is generic over the float type: a Float32 y computes f32::atan2 and returns Float32, so atan2(0, 0) is 0 there too.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/trigonometry.rs#L52-L88 arctan2_on_floats: x is cast to y’s float type, then yv.atan2(xv). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L69-L71 - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.arctan2.html silent on atan2(0, 0) and the infinities - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/functions/lazy.py#L1774-L1809 the docstring at the tag - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/sql/test_trigonometric.py#L9-L30 test_arctan2: only (±√2/2, ±√2/2) → ±45°, ±135° (eager, in memory); no upstream test pins (0, 0) or ±inf

polars-double-arith: Double + - *: IEEE, NaN for inf - inf and 0 * inf

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: IEEE: NaN where the reference raises

Double arithmetic is IEEE 754: inf - inf, inf + (-inf) and 0 * inf are NaN. Every other edge case agrees with the reference.

An operation on finite values whose result overflows gives ±inf without raising, in Float64 and in Float32: 1e308 + 1e308 and 1e308 * 10 are inf, -1e308 - 1e308 is -inf; Float32 overflows at its own range (3e38 + 3e38 is inf). Same on both engines.

A literal ±0.0 addend or subtrahend returns the column unchanged, and 0.0 - x is -x (the literal kernels’ shortcuts, float.rs#L45-L65): -0.0 + 0.0 is -0.0 and 0.0 - 0.0 is -0.0, where IEEE gives +0.0. Two columns give +0.0, and so does a one-row morsel of the streaming engine (polars-zero-sign-streaming). The NaN cases above are unaffected.

No upstream test pins inf - inf, inf + (-inf), 0 * inf or the ±0.0 shortcuts; they may change between versions.

The reference: add, subtract, multiply raise a domain issue for inf - inf and 0 · inf (nan_creation: raise, #257).

Options: nan_creation, float_overflow.

Rows of the choice tables: fp.arith nan_creation (fp64); fp.arith nan_creation (fp32; evidence source); fp.arith float_overflow (fp32, fp64).

Constructs: +, -, *.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-5, ST-9; issues #257.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (doubles): Polars’s user guide says it generally follows IEEE 754 for Float32 and Float64, with two exceptions: any NaN equals any other NaN and is greater than every non-NaN value, and operations guarantee nothing about the sign of zero or NaN or a NaN’s payload (a sort or group by may make every zero +0 and every NaN a positive NaN). It promises no error bound unless a function says otherwise. The user guide is not versioned; the section reads the same at tag py-2.1.0. So inf - inf and 0 * inf are NaN in Float32 as in Float64 (IEEE 754’s invalid operation). Settles the Float32 row at docs level; the probes cover Float64.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): impl_float_arith_kernel! is one macro instantiated for pf16, f32 and f64, so Float32 and Float64 run the same operations. The streaming engine evaluates expressions through polars-expr on each morsel (StreamExpr in crates/polars-stream/src/expression.rs), so it shares these kernels with the in-memory engine. + - * are the IEEE operations, so inf - inf and 0 * inf are NaN in Float32 as in Float64. The literal shortcuts (x + 0, x * 1) return x, which changes no NaN.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L8-L31 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L45-L76 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L124-L126 - 2026-11-04, probe, by characterization pipeline session (float_overflow rows), polars 2.1.0 - lf = pl.LazyFrame({‘a’: [big, -big], ‘b’: [big, big], ‘c’: [10.0, small]}) with all three Float64 (big 1e308, small 1e-10) and Float32 (big 3e38, small 1e-3); select a + b, a - b, a * c, a / c; pl.LazyFrame(…).collect(engine=e) for e in (‘in-memory’, ‘streaming’), polars 2.1.0 → Float64: [(inf, 0.0, inf, 1e+307), (0.0, -inf, -1.0000000000000001e+298, -inf)]; Float32: [(inf, 0.0, inf, 3.000000106909804e+37), (0.0, -inf, -3.0000001227554363e+35, -inf)]; streaming the same - 2026-11-04, source-reading, by characterization pipeline reverse pass: Added to the description: the ±0 literal shortcuts (-0.0 + 0.0 is -0.0), which polars-zero-sign-streaming recorded and this fact lacked. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L45-L65 a ±0 literal addend or subtrahend returns lhs unchanged, and 0 - x is neg(x): -0.0 + 0.0 and 0.0 - 0.0 give -0.0 (IEEE: +0.0) - 2026-11-04, manual-repro, by characterization pipeline reverse pass - pl.DataFrame({'z': [-0.0, 0.0, -0.0]}).select(a=pl.col('z') + 0.0, b=pl.col('z') - (-0.0), c=0.0 - pl.col('z'), d=pl.col('z') + pl.lit(pl.Series([0.0, 0.0, 0.0]))) gives signs a: -,+,-; b: -,+,-; c: +,-,+; d: +,+,+ (Polars 2.1.0, macOS arm64) - 2026-11-04, engine-test, by characterization pipeline reverse pass - no upstream test pins inf - inf, inf + (-inf) or 0 * inf (searched py-polars/tests/unit/operations/arithmetic, expr, series, dataframe at py-2.1.0)

polars-double-divide: Double / between columns: IEEE

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: on_division_by_zero: IEEE, but ±inf / ±inf is NaN

/ between two double columns is IEEE 754: x / 0 is ±inf, 0 / 0 is NaN, ±inf / ±inf is NaN. No setting changes it. With the IEEE candidate, only the four ±inf / ±inf rows differ, as for DuckDB (duckdb-R15-double-arith).

A length-1 divisor against a longer column (a literal on a frame of two rows or more) is a multiplication by its reciprocal. Zero and infinite divisors give the same results there, but a subnormal divisor’s reciprocal is inf, so 0.0 / 1e-310 is NaN and 1e-300 / 1e-310 is inf (polars-double-scalar-divisor, PL-3).

A quotient of finite values too large for the type is ±inf: -1e308 / 1e-10 is -inf in Float64, -3e38 / 1e-3 is -inf in Float32, on both engines.

On the streaming engine the kernel is chosen per morsel, so a morsel of one row divides exactly; which rows land in one depends on the thread count and POLARS_IDEAL_MORSEL_SIZE (polars-double-scalar-divisor). A division by an aggregate (x / x.max()) has a length-1 divisor too.

The truediv docs and the user guide call float division IEEE 754, which the reciprocal kernel contradicts: PL-8.

The reference: divide with on_division_by_zero: ERROR when omitted; IEEE, NULL, LIMIT when asked. ±inf / ±inf is a domain issue (#257).

Options: on_division_by_zero, nan_creation, on_domain_error, float_overflow.

Rows of the choice tables: fp.div on_division_by_zero (fp64); fp.div on_division_by_zero (fp32; evidence engine_test); fp.div nan_creation (fp64); fp.div nan_creation (fp32; evidence engine_test); fp.div on_domain_error (fp64); fp.div on_domain_error (fp32; evidence source); fp.div float_overflow (fp32, fp64).

Constructs: /.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-3, PL-8; issues #204, #257.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (doubles): Expr.truediv’s notes say zero division follows IEEE-754: 0/0 is an invalid operation and returns NaN, n/0 on finite operands is ±infinity. The note is not tied to a float type, and the user guide says Float32 and Float64 both follow IEEE 754, so Float32 division behaves the same (and ±inf / ±inf is NaN, IEEE’s invalid operation). Settles the Float32 rows and on_domain_error (NaN for 0/0 and ±inf / ±inf) at docs level; the probes cover Float64.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): impl_float_arith_kernel! is one macro instantiated for pf16, f32 and f64, so Float32 and Float64 run the same operations. The streaming engine evaluates expressions through polars-expr on each morsel (StreamExpr in crates/polars-stream/src/expression.rs), so it shares these kernels with the in-memory engine. Column by column / is l / r, so x / 0 is ±inf and 0 / 0 and inf / inf are NaN in Float32; with a literal divisor x * (1 / c) gives the same at c = ±0 and c = ±inf.

The literal path creates NaN where IEEE does not: 0.0 / c for a subnormal c is 0 * inf, NaN (polars-double-scalar-divisor). So the nan_creation row’s “IEEE” holds column by column only, and its Float32 entry stays at the docs.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L109-L126 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/mod.rs#L40-L63 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/arithmetic/numeric.rs#L105-L163 + - * // % on two ChunkedArrays: equal lengths take the array kernel, a length-1 side the scalar kernel. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/arity.rs#L894-L942 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/column/mod.rs#L1816-L1839 A literal is a scalar column, passed to the operation as a one-value Series. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/expression.rs#L10-L38 The streaming engine evaluates each expression with polars-expr’s PhysicalExpr on every morsel, so it runs the same kernels. - 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): Corrects the description: the IEEE claim now covers / between two columns, and a second paragraph gives the literal path’s NaN and inf for a subnormal divisor, with links to polars-double-scalar-divisor and PL-3. The nan_creation rows hold column by column; the Float32 entry stays at the docs.

Read at tag py-2.1.0 (commit 7f902bd3). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L109-L119 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/arithmetic/numeric.rs#L105-L163 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/arity.rs#L894-L942 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/expression.rs#L10-L38 - 2026-11-04, engine-test, by characterization pipeline round 3 (doubles): Pins the Float32 results of a zero divisor: x / 0 is inf and 0 / 0 NaN. The test runs each case on one-row operands and, through BROADCAST_SERIES_COMBINATIONS, with the left side broadcast to three rows against a one-row right side, which takes the reciprocal-divisor kernel (PL-3); EXEC_OP_COMBINATIONS runs it on Series (in memory) and as a lazy select with collect(). These inputs give the same results on the IEEE and the reciprocal path. Marked slow, which CI still runs. ±inf / ±inf is not tested, so the on_domain_error row for fp32 stays at the source.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L198-L218 test_list_arithmetic_values, Float32: 1 // 0 is inf, 1 % 0 NaN, 1 / 0 inf, 0 // 0 NaN, 0 % 0 NaN, 0 / 0 NaN - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L153-L172 test_list_arithmetic_values, Float32 (picked so nothing upcasts to Float64): 7 // 3 is 2, -5 % 3 is 1, 2 / 128 is 0.015625 - 2026-11-04, probe, by characterization pipeline session (float_overflow rows), polars 2.1.0 - pl.LazyFrame({‘a’: [big, -big], ‘c’: [10.0, small]}) with Float64 (big 1e308, small 1e-10) and Float32 (big 3e38, small 1e-3); select a / c; pl.LazyFrame(…).collect(engine=e) for e in (‘in-memory’, ‘streaming’), polars 2.1.0 → Float64: 1e+307, -inf; Float32: 3.000000106909804e+37, -inf; streaming the same - 2026-11-04, source-reading, by characterization pipeline reverse pass: Added to the description: the streaming engine picks the kernel by morsel length, and x / x.max() takes the reciprocal; the docs contradiction is PL-8. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/in_memory_source.rs#L118-L122 streaming: morsels of len.div_ceil(k * num_pipelines) rows; a frame shorter than the thread count streams as one-row morsels, which take the equal-length IEEE kernel - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/arity.rs#L894-L942 equal lengths: array kernel; a length-1 side: scalar (reciprocal) kernel - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/dataframe/test_df.py#L2982-L3016 test_floordiv_truediv: DataFrame / scalar equals Python exactly (eager, in memory); DataFrame._div materializes the scalar to full height, so this pins the column kernel, not pl.col / literal - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/dataframe/frame.py#L1121-L1138

polars-double-modulus: Double %: x - y·floor(x / y)

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: floor remainder; NaN for x % 0 and for finite x % ±inf

% on doubles computes x - y * floor(x / y): the remainder has the divisor’s sign (5.5 % -2 is -0.5), x % 0 is NaN, and a finite x % ±inf is NaN too (x / inf is 0, and inf * 0 is NaN), where C’s fmod and Python’s % give x. The formula also loses the exactness of fmod for large quotients.

// on doubles is floor(x / y).

A zero remainder is always 0.0: x - y * q with equal terms is +0.0 under round to nearest, and for x = -0.0 the product y * q is -0.0, so -0.0 % y is 0.0 too (fmod gives -0.0 for -4 % 2 and -0.0 % y).

The docs are silent on % of doubles (the floored remainder, x % 0, x % ±inf): a docs gap.

No upstream test pins x % ±inf; it may change between versions.

The reference: modulus on doubles (sverify’s sverify_fmod): the truncated remainder (fmod), with a zero divisor an error as on integers, and fmod(x, ±inf) = x. 60 of the 144 rows differ.

Options: division_type, on_division_by_zero, nan_creation, zero_sign.

Rows of the choice tables: fp.mod division_type (fp64); fp.mod on_division_by_zero (fp64); fp.mod nan_creation (fp64); fp.mod division_type (fp32; evidence engine_test); fp.mod on_division_by_zero (fp32; evidence engine_test); fp.mod nan_creation (fp32; evidence engine_test); fp.mod zero_sign (fp32, fp64; evidence source).

Constructs: %, //.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-1, PL-3.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, source-reading, by session brave-borg-41f7cc: Read at tag py-2.1.0 (commit 7f902bd3).
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): impl_float_arith_kernel! is one macro instantiated for pf16, f32 and f64, so Float32 and Float64 run the same operations. The streaming engine evaluates expressions through polars-expr on each morsel (StreamExpr in crates/polars-stream/src/expression.rs), so it shares these kernels with the in-memory engine. So the floor remainder, NaN for a zero divisor and NaN for a finite x % ±inf hold for Float32 as the Float64 probes show.

With a literal divisor on a frame of two rows or more, the kernel multiplies by the reciprocal: where x * (1 / y) rounds below an integer that x / y reaches, the floor is one less and the remainder is y itself (49.0 % 49.0 is 49.0), and a subnormal divisor makes the reciprocal infinite. That path is polars-double-scalar-divisor.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L41-L43 Column by column: l - r * (l / r).floor(). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L96-L103 A literal divisor: x - rhs * (x * (1 / rhs)).floor(); a literal dividend: lhs - x * (lhs / x).floor(). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L124-L126 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L11-L22 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/arithmetic/numeric.rs#L105-L163 + - * // % on two ChunkedArrays: equal lengths take the array kernel, a length-1 side the scalar kernel. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/arity.rs#L894-L942 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/column/mod.rs#L1816-L1839 A literal is a scalar column, passed to the operation as a one-value Series. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/expression.rs#L10-L38 The streaming engine evaluates each expression with polars-expr’s PhysicalExpr on every morsel, so it runs the same kernels. - 2026-11-04, engine-test, by characterization pipeline round 3 (doubles): Pins the Float32 floor remainder (-5 % 3 is 1, the divisor’s sign) and NaN for a zero divisor (1 % 0, 0 % 0). The test runs each case on one-row operands and, through BROADCAST_SERIES_COMBINATIONS, with the left side broadcast to three rows against a one-row right side, which takes the reciprocal-divisor kernel (PL-3); EXEC_OP_COMBINATIONS runs it on Series (in memory) and as a lazy select with collect(). These inputs give the same results on the IEEE and the reciprocal path. Marked slow, which CI still runs. No test checks the sign of a zero remainder, so zero_sign stays at the source.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L198-L218 test_list_arithmetic_values, Float32: 1 // 0 is inf, 1 % 0 NaN, 1 / 0 inf, 0 // 0 NaN, 0 % 0 NaN, 0 / 0 NaN - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L153-L172 test_list_arithmetic_values, Float32 (picked so nothing upcasts to Float64): 7 // 3 is 2, -5 % 3 is 1, 2 / 128 is 0.015625 - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.mod.html silent on float semantics (floored remainder, x % 0, x % inf) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/expr.py#L6797-L6825 - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/sql/test_numeric.py#L777-L829 SQL % truncates (-7.5 % 2 = -1.5); the comment says the expression API floors instead (eager, in memory). No upstream test of x % ±inf - 2026-11-04, source-reading, by characterization pipeline reverse pass: From the reverse pass’s check of PL-1, which this fact records. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/floor_divmod.rs#L4-L5 source comment: the same division / modulo combination as Python - 2026-11-04, engine-test, by characterization pipeline reverse pass: From the reverse pass’s check of PL-1, which this fact records. - no upstream test of x % ±inf at py-2.1.0; test_list.py#L169-L170 and #L210-L217 pin only finite and zero divisors

polars-double-scalar-divisor: Dividing by a literal multiplies by its reciprocal

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: x / c is x * (1 / c): not correctly rounded, NaN or inf for a subnormal c; 49.0 % 49.0 is 49.0

When one operand of /, // or % on floats has length 1 and the other more (a literal, pl.lit, or a one-row column against a longer one), Polars computes the reciprocal of the divisor once and multiplies: x / c is x * (1 / c), x // c is floor(x * (1 / c)), x % c is x - c * floor(x * (1 / c)). Two roundings replace one, so x / c can miss the IEEE quotient by an ulp: 49.0 / 49.0 is 0.9999999999999999, and then 49.0 // 49.0 is 0.0 and 49.0 % 49.0 is 49.0. Recorded as PL-3.

For a subnormal c the reciprocal overflows to inf: 0.0 / 1e-310 is NaN where IEEE gives 0.0, and 1e-300 / 1e-310 is inf where IEEE gives 1e10 (% gives -inf). The same holds for Float32 below about 2.9e-39.

On a one-row frame both sides have length 1 and the IEEE kernel runs, so the one-row probes do not show it. Two columns of equal length use l / r. Both engines share the kernels, but the streaming engine picks one per morsel: its in-memory source cuts a frame into morsels of len.div_ceil(k · threads) rows, so a frame shorter than the thread count streams as one-row morsels and divides exactly, and a frame a little longer mixes both results in one column. With POLARS_MAX_THREADS=10, 49.0 / 49.0 on 4 or 10 rows gave 1.0, on 11 rows both 1.0 and 0.9999999999999999, on 40 rows 0.9999999999999999. With one thread a 4-row frame gives 0.9999999999999999, and % 49.0 gives 49.0. So the result depends on POLARS_MAX_THREADS and POLARS_IDEAL_MORSEL_SIZE.

polars-scalar-divisor records the same finding from #590’s probe and is the id the engine tables cite.

Integer / casts both sides to Float64 and runs the same kernel, so a literal integer divisor takes the reciprocal too: Int64 49 / 49 is 0.9999999999999999 and 5 / 3 is 1.6666666666666665 (polars-int-true-divide). A division by an aggregate (x / x.max()) has a length-1 divisor as well. DataFrame / 3 repeats the scalar to the frame’s height and divides exactly.

The truediv docs and the user guide call float division IEEE 754, which this contradicts: PL-8. No upstream test pins this; it may change between versions.

The reference: divide on doubles is the IEEE quotient, rounded once (rounding: TIE_TO_EVEN), and creates no NaN from non-NaN inputs other than 0/0 and inf/inf, which raise. sverify_fmod is the truncated remainder, smaller than the divisor in magnitude.

Options: rounding, nan_creation, division_type.

Rows of the choice tables: fp.div rounding (fp64); fp.div rounding (fp32; evidence source); fp.div nan_creation (fp64); fp.div nan_creation (fp32; evidence source); fp.mod division_type (fp64); fp.mod division_type (fp32; evidence source).

Constructs: /, //, %.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-3, PL-8.

Provenance:

  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): impl_float_arith_kernel! is one macro instantiated for pf16, f32 and f64, so Float32 and Float64 run the same operations. The streaming engine evaluates expressions through polars-expr on each morsel (StreamExpr in crates/polars-stream/src/expression.rs), so it shares these kernels with the in-memory engine. The broadcast in apply_binary_kernel_broadcast_owned sends a length-1 right side to the scalar kernel, and a literal reaches it as a one-value Series.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L78-L103 floor_div, trunc_div and mod with a literal divisor use inv = 1 / rhs. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L109-L119 prim_true_div_scalar is prim_wrapping_mul_scalar(lhs, 1 / rhs). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L124-L126 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/mod.rs#L52-L63 legacy_div_scalar, which ChunkedArray / ChunkedArray reaches for floats, is true_div_scalar. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/expressions/binary.rs#L134-L144 / on floats is left / right on the columns. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/arithmetic/numeric.rs#L105-L163 + - * // % on two ChunkedArrays: equal lengths take the array kernel, a length-1 side the scalar kernel. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/arity.rs#L894-L942 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/column/mod.rs#L1816-L1839 A literal is a scalar column, passed to the operation as a one-value Series. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/expression.rs#L10-L38 The streaming engine evaluates each expression with polars-expr’s PhysicalExpr on every morsel, so it runs the same kernels. - 2026-11-04, manual-repro, by characterization pipeline round 2 (doubles): Polars 2.1.0, macOS arm64. Streaming splits a small frame into one-row morsels, where both sides have length 1 and the IEEE kernel runs; morsels of two rows or more take the literal path, as the in-memory engine does. - pl.DataFrame({‘x’: [49.0, 3.0, -0.0, 0.0, 0.5]}).select(pl.col(‘x’) / 49.0, pl.col(‘x’) % 49.0, pl.col(‘x’) // 49.0): 0.9999999999999999, 49.0 and 0.0 for x = 49.0. The same on pl.DataFrame({‘x’: [49.0]}): 1.0 and 0.0. - pl.DataFrame({‘x’: [0.0, 1e-300, 2.0]}).select(pl.col(‘x’) / 1e-310, pl.col(‘x’) % 1e-310, pl.col(‘x’) // 1e-310): [nan, inf, inf], [nan, -inf, -inf], [nan, inf, inf]. On pl.DataFrame({‘x’: [0.0]}): 0.0. - pl.DataFrame({‘x’: [0.0, 1.0, 0.5]}, schema={‘x’: pl.Float32}).select(pl.col(‘x’) / pl.lit(1e-40, dtype=pl.Float32)): [nan, inf, inf]. - pl.LazyFrame({‘x’: [49.0] * 5000 + [0.0]}).select(pl.col(‘x’) / 49.0, pl.col(‘x’) % 49.0).collect(engine=‘streaming’): 0.9999999999999999 and 49.0 for x = 49.0. The same query on a 4-row LazyFrame gave the IEEE results (1.0, 0.0, and 1e-300 / 1e-310 = 10000000000.00003) under streaming and the deviations under engine=‘in-memory’. - 2026-11-04, engine-test, by characterization pipeline round 3 (doubles): The broadcast cases of the test run x / c, x // c and x % c with a one-row c against three rows, the reciprocal path, in Float32: 1 / 0 is inf, 0 / 0 NaN, 7 // 3 is 2, -5 % 3 is 1. Each of these agrees with the IEEE quotient too, so the test neither confirms nor contradicts PL-3; no Polars test at py-2.1.0 asserts an inexact quotient of a broadcast divisor (searched py-polars/tests/unit for float divisions by literals and scalar aggregates). The rows stay at the source.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L198-L218 test_list_arithmetic_values, Float32: 1 // 0 is inf, 1 % 0 NaN, 1 / 0 inf, 0 // 0 NaN, 0 % 0 NaN, 0 / 0 NaN - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_list.py#L153-L172 test_list_arithmetic_values, Float32 (picked so nothing upcasts to Float64): 7 // 3 is 2, -5 % 3 is 1, 2 / 128 is 0.015625 - 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): Streaming agrees with memory on 9 000 rows: 49.0 % 49.0 is 49.0, 49.0 // 49.0 0.0, 0.0 / 1e-310 NaN, 1e-300 / 1e-310 inf. So PL-3 holds on the streaming engine when the morsels hold more than one row.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-scalar-divisor in characterization/polars/probes-streaming.json - 2026-11-04, probe, by characterization pipeline (merging duplicates after #636): polars-scalar-divisor records the same finding from #590’s probe and is the id the engine tables cite. - probe s-scalar-divisor in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass: Corrected the description: it said a small frame under streaming gives the IEEE results; that holds only while the frame is shorter than the thread count (in_memory_source.rs#L118-L122). Added integer / and division by an aggregate, which take the reciprocal too. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/in_memory_source.rs#L118-L122 morsel length = len.div_ceil(k * num_pipelines): which kernel runs depends on threads and POLARS_IDEAL_MORSEL_SIZE - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/morsel.rs#L12-L14 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/expressions/binary.rs#L151-L158 integer / integer casts both sides to Float64 first, so a literal integer divisor takes the float reciprocal kernel too - 2026-11-04, manual-repro, by characterization pipeline reverse pass - POLARS_MAX_THREADS=10 POLARS_IDEAL_MORSEL_SIZE=64: pl.LazyFrame({'x': [49.0] * n}).select(pl.col('x') / 49.0).collect(engine='streaming') for n in (4, 10, 11, 40) gives 4: [1.0]; 10: [1.0]; 11: [0.9999999999999999, 1.0]; 40: [0.9999999999999999] (Polars 2.1.0, macOS arm64) - POLARS_MAX_THREADS=1 vs 10: pl.LazyFrame({'x': [49.0, 0.0, 1e-300, 49.0]}).select(pl.col('x') / 49.0, pl.col('x') % 49.0).collect(engine='streaming') gives 1 thread: (0.9999999999999999, 49.0) for x = 49.0; 10 threads: (1.0, 0.0); in-memory: (0.9999999999999999, 49.0) either way - pl.DataFrame({'a': [49, 5, 0]}).select(pl.col('a') / 49, pl.col('a') / 3, pl.col('a') / pl.col('a').max()) gives 49/49 = 0.9999999999999999, 5/3 = 1.6666666666666665 (IEEE 1.6666666666666667), a / a.max() = 0.9999999999999999 for a = 49 - pl.DataFrame({'y': [5.0, -7.0]}) / 3 vs .select(pl.col('y') / 3) gives DataFrame / 3: 1.6666666666666667, -2.3333333333333335 (IEEE); pl.col('y') / 3: 1.6666666666666665, -2.333333333333333 - 2026-11-04, engine-test, by characterization pipeline reverse pass - no upstream test pins pl.col(x) / literal on two rows or more; test_df.py#L2982-L3016 checks DataFrame / scalar, which materializes the scalar and runs the column kernel - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/dataframe/test_df.py#L2982-L3016 - 2026-11-04, manual-repro, by characterization pipeline reverse pass: From the reverse pass’s check of PL-3, which this fact records. - POLARS_MAX_THREADS=10 POLARS_IDEAL_MORSEL_SIZE=64: pl.LazyFrame({'x': [49.0] * n}).select(pl.col('x') / 49.0).collect(engine='streaming') gives n=4: [1.0]; n=11: [0.9999999999999999, 1.0]; n=40: [0.9999999999999999]

polars-elem-rounding: Elementary functions: Rust’s libm calls, in the column’s float type

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: sqrt correctly rounded; the rest is the platform’s libm; x ** n for a literal n from 2 to 9 multiplies

sqrt, exp, sin, cos, tan, arcsin, arccos, arctan, arctan2 and ** are generic over the float type and call Rust’s f32 or f64 method: Float32 computes in single precision and returns Float32, Float64 in double. sqrt is IEEE’s correctly rounded square root; the others go to the platform’s libm (sinf, exp, pow, …), which promises no correct rounding, so their last bits can differ between macOS and Linux. Integer inputs are cast to Float64 first.

x ** y with a literal exponent takes shortcuts: 1 returns x, 0.5 is sqrt(x), and an integer from 2 to 9 multiplies x by itself that many times, rounding after each product, where powf rounds once. With a column exponent it is powf, except that a one-row exponent of 2 is x * x.

The docs are silent on the rounding of these functions and on Float32 staying single precision: a docs gap.

No upstream test pins their last bits or Float32’s precision; they may change between versions.

The reference: The reference computes these with Lean’s Float (the platform’s libm); only NULL, the sign and the domain are exact, the last bits are each engine’s (#54).

Options: rounding.

Rows of the choice tables: fp.elem rounding (fp32, fp64).

Constructs: sqrt, exp, sin, cos, tan, arcsin, arccos, arctan, arctan2, **.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): Each function matches Float32 and Float64 and calls one generic body with num_traits::Float, so Float32 never goes through f64. The output type is the input’s float type (map_to_float_dtype, pow_dtype). Rust’s f32::sqrt and f64::sqrt compile to the hardware instruction; the others call the platform’s libm.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/trigonometry.rs#L7-L32 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/trigonometry.rs#L90-L178 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/trigonometry.rs#L52-L88 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L37-L70 pow_on_floats: the literal shortcuts, then Pow::pow (powf). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L11-L35 pow_on_chunked_arrays: a one-row exponent of 2 is base * base. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L191-L221 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/log.rs#L102-L123 exp on Float32 is f32::exp. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L41-L44 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L69-L71 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L329-L331 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L721-L728 - 2026-11-04, manual-repro, by characterization pipeline round 2 (doubles): Float32 stays in single precision. - pl.DataFrame({‘x’: [49.0, 1.0, 0.5, 2.0, -0.0]}, schema={‘x’: pl.Float32}).select(sqrt, sin, exp, ** 0.3): every column Float32; sqrt(0.5) = 0.70710677, sin(1) = 0.84147096, exp(1) = 2.7182817, the correctly rounded single-precision values (Polars 2.1.0, macOS arm64). - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/user-guide/concepts/data-types-and-structures/#floating-point-numbers no error bound promised; silent on libm and Float32 precision - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/expr.py#L6963-L7000 Expr.pow: the result dtype rule only - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/expr/test_exprs.py#L270-L297 test_exp_log1p: Float32 → Float32, integers → Float64 (lazy, both engines); values compared with tolerance - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/series/test_series.py#L1872-L1900 test_trigonometric against numpy with tolerance (eager)

polars-fp-ieee: Float32 and Float64 arithmetic: IEEE 754 between columns

Engine: polars py-2.1.0; area: doubles; status: no-finding; evidence: source.

Verdict: IEEE 754 rounding for + - *, and for / between two columns

Polars’s user guide says Float32 and Float64 generally follow IEEE 754, with exceptions for NaN equality and order and for the sign of zero and NaN. The kernels for + - * are the IEEE operations in both float types, so they round as IEEE 754 binary32 and binary64 do, to nearest with ties to even. / between two columns of equal length is the IEEE quotient too.

A length-1 divisor against a longer column (a literal on a frame of two rows or more) is divided by multiplying with its reciprocal, two roundings: polars-double-scalar-divisor and PL-3.

The guide promises no error bound for other computations unless a function says so, so it does not settle the rounding of the elementary functions or logarithms (polars-elem-rounding, polars-log-base).

The reference: rounding: TIE_TO_EVEN for + - × / on doubles (native Float). fp32 arithmetic is refused in every profile until #417’s stage 3.

Options: rounding.

Rows of the choice tables: fp.arith rounding (fp32, fp64); fp.div rounding (fp32, fp64).

Constructs: +, -, *, /.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-3.

Provenance:

  • 2026-11-04, docs, by characterization pipeline round 1 (doubles): Polars’s user guide says it generally follows IEEE 754 for Float32 and Float64, with two exceptions: any NaN equals any other NaN and is greater than every non-NaN value, and operations guarantee nothing about the sign of zero or NaN or a NaN’s payload (a sort or group by may make every zero +0 and every NaN a positive NaN). It promises no error bound unless a function says otherwise. The user guide is not versioned; the section reads the same at tag py-2.1.0.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): Raises fp.arith rounding to source: + - * are the IEEE operations for f32 and f64, column by column and with a literal (whose shortcuts return x exactly). The streaming engine runs the same polars-expr kernels.

Corrects the / claim: IEEE only when both sides are columns of equal length (or both have length 1). With a length-1 divisor against a longer side, x / c is x * (1 / c), which can miss by an ulp (49.0 / 49.0 is 0.9999999999999999). Title, verdict and description now say so and link polars-double-scalar-divisor (PL-3).

Read at tag py-2.1.0 (commit 7f902bd3). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L8-L31 Column kernels: l + r, l - r, l * r. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L45-L76 Literal kernels: x + c, x * c, with shortcuts that return x for c = 0 (add, subtract) and c = 1, exact. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L109-L126 prim_true_div is l / r; prim_true_div_scalar multiplies by 1 / rhs. One macro for pf16, f32, f64. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/arithmetic/numeric.rs#L105-L163 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/arity.rs#L894-L942 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/expression.rs#L10-L38

polars-fp32: Float32 arithmetic stays in Float32, and pulls in a double literal

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: native single precision; Float32 + 0.1 is Float32

Arithmetic on Float32 is single precision (3e38 + 3e38 is inf, 16777216 + 1 is 16777216), and sum and mean of Float32 are Float32. sum adds in Float32 on both engines; the streaming mean sums in f64 and rounds the quotient to Float32 once (polars-mean-float-streaming). A Python float or int literal takes the column’s type, so Float32 + 0.1 adds 0.1 rounded to a float. Casts between the formats round to nearest (polars-cast-to-double).

The docs are silent on a literal taking a Float32 column’s type: a docs gap.

No upstream test pins single-precision overflow or rounding (3e38 + 3e38, 16777216 + 1); they may change between versions.

The reference: fp32 arithmetic is refused in every profile until #417’s stage 3.

Options: fp32, implicit_casts.

Rows of the choice tables: fp32 (fp32).

Constructs: +, sum, mean.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #417.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe fp32-arith in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass: Added to the description: the streaming mean of Float32 sums in f64 (reduce/mean.rs#L30-L52); the result type stays Float32. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L124-L126 the f32 kernels - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/utils/supertype.rs#L519-L520 a dynamic int or float literal takes a float column’s type - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L40-L47 SumCast: f32 sums as f32 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mean.rs#L30-L52 streaming mean of Float32: f64 sum, quotient cast to f32 - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_datatypes.py#L296-L315 literals with a Float32 column stay Float32 (lazy, both engines) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L1307-L1319 Float32 / 2 stays Float32

polars-int-float-compare: An integer compared with a double: both as doubles

Engine: polars py-2.1.0; area: doubles; status: no-finding; evidence: probe.

Verdict: Int64 == Float64 compares in Float64

Comparing an Int64 with a Float64 casts the integer to Float64 first: 2⁵³ + 1 == 2.0⁵³ is true.

The reference: Substrait’s comparisons take two arguments of one type; a plan casts first, and the cast’s rounding applies.

Options: implicit_casts.

Constructs: ==, <.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe int-float-compare in characterization/polars/probes-streaming.json

polars-log-base: log10 and log(base) divide natural logarithms

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: ln(x) / ln(base): last-bit differences from log10, log2

log10() is log(10.0), and log(base) is Rust’s f64::log, ln(x) / ln(base): two roundings, so it can miss the correctly rounded result by an ulp. log10(0.5) is -0.30102999566398114 where log10 gives -0.3010299956639812, and log2(1e308) is 1023.1538532253077 where log2 gives 1023.1538532253076.

On Float32 the same division runs in single precision, and the default base math.e becomes the float nearest e, whose ln rounds to 0.99999994. So the natural log() of a Float32 misses by an ulp: log(0.5) is -0.69314724 where the correctly rounded value is -0.6931472, and log10(0.5) is -0.30102998 where it is -0.30103001. Recorded as PL-4.

The docs are silent on log10 and log dividing two logarithms: a docs gap. The user guide promises no error bound.

No upstream test pins this; it may change between versions.

The reference: log10 and log2 are the platform’s log10 and log2; only NULL, the sign and the domain are exact in the reference, the last bits are each engine’s (#54).

Options: rounding.

Rows of the choice tables: fp.log rounding (fp64); fp.log rounding (fp32).

Constructs: log10, log.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-4; issues #54.

Provenance:

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/log.rs#L49-L77 Equal float types: x.log(base) in that type; a float input casts the base to its type. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/expr.py#L12229-L12256 Expr.log(base=math.e). - https://doc.rust-lang.org/1.90.0/std/primitive.f32.html#method.log f32::log may not be correctly rounded; it is ln(self) / ln(base). - 2026-11-04, manual-repro, by characterization pipeline round 2 (doubles): The correctly rounded Float32 values of ln(0.5) and log10(0.5) are -0.693147182464599609375 and -0.30103000998497009277. - pl.DataFrame({‘x’: [0.5, 2.0, 49.0]}, schema={‘x’: pl.Float32}).select(pl.col(‘x’).log(), pl.col(‘x’).log10()): log gave -0.6931472420692444, 0.6931472420692444, 3.8918204307556152, log10 gave -0.3010299801826477, … (Polars 2.1.0, macOS arm64). - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.log10.html ‘Compute the base 10 logarithm of the input array, element-wise’; no accuracy note. The body is self.log(10.0) (expr.py L914-L935). - https://docs.pola.rs/user-guide/concepts/data-types-and-structures/#floating-point-numbers Polars ‘does not provide guarantees on the error unless mentioned otherwise’ (SRC docs/source/user-guide/concepts/data-types-and-structures.md L186-L189). - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/expr/test_exprs.py#L230-L268 test_log: [1, 3, 9, 27, 81].log(base [3, 3, 9, 3, 9]) == [0, 1, 1, 3, 2] for Float16/32/64 and mixed types, lazy collect() (both engines in CI), rel_tol 1e-5: pins result types and values, not the last bits. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L1132-L1162 test_log_exp: log10 of [1, 100, 1000] == [0, 2, 3], log vs np.log, rel_tol 1e-5. No upstream test pins the one-ulp misses of ln(x)/ln(base). - 2026-11-04, source-reading, by characterization pipeline reverse pass: From the reverse pass’s check of PL-4, which this fact records. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/log.rs#L60-L72 A float column with a base of another type casts the base to the column’s type (math.e to 2.7182817 for Float32), then x.log(base) in f32.

polars-math-domain: Elementary functions: NaN outside the domain, -inf for log(0)

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: on_domain_error: NAN; on_log_zero: MINUS_INFINITY

sqrt, the logarithms, arcsin, arccos and ** give NaN outside their domain (sqrt(-1), log(-1), arcsin(2.5), (-1) ** 0.5), log(0) is -inf, and sin, cos, tan of ±inf are NaN. Inside the domain they are Rust’s f64 functions (the platform’s libm).

The docs are silent on results outside a function’s domain: a docs gap.

No upstream test pins log(0), log(-1), (-1) ** 0.5 or sin(±inf); they may change between versions.

The reference: on_domain_error: ERROR and on_log_zero: ERROR (#215); sin(±inf) is a domain issue.

Options: on_domain_error, on_log_zero.

Rows of the choice tables: fp.elem on_domain_error (fp64); fp.log on_domain_error (fp64); fp.log on_log_zero (fp64); fp.elem on_domain_error (fp32; evidence source); fp.log on_domain_error (fp32; evidence source); fp.log on_log_zero (fp32).

Constructs: sqrt, log, log10, sin, cos, tan, arcsin, arccos, **.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-16; issues #215.

Provenance:

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/trigonometry.rs#L7-L32 Float32 stays Float32; integers are cast to Float64. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/trigonometry.rs#L90-L178 Generic over T: PolarsFloatType with num_traits::Float: sin, cos, tan, asin, acos, atan. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L37-L70 pow_on_floats, generic; a literal 0.5 is sqrt. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L191-L221 sqrt on Float32 is f32::sqrt. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/log.rs#L49-L77 log(base): x.log(base) in the column’s float type. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L721-L728 map_to_float_dtype: Float32 maps to Float32. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L920-L931 log_dtype: a float input keeps its type. - 2026-11-04, manual-repro, by characterization pipeline round 2 (doubles): on_log_zero on Float32. - pl.DataFrame({‘x’: [-0.0]}, schema={‘x’: pl.Float32}).select(pl.col(‘x’).log(), pl.col(‘x’).log10(), pl.col(‘x’).log(2)): each -inf, dtype Float32 (Polars 2.1.0, macOS arm64). - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L115-L124 test_sqrt_neg_inf: sqrt([-inf, -9, 0, 9, inf]) == [nan, nan, 0.0, 3.0, inf] (compared as strings), eager, in memory. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/series/test_series.py#L1872-L1900 test_trigonometric: sin, cos, tan, arcsin, arccos, arctan and the hyperbolics of [0.0, pi, None, nan] equal numpy’s (arcsin(pi), arccos(pi) NaN), eager, in memory. No upstream test of log(0) = -inf, log(-1), (-1) ** 0.5 or sin(inf). - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/user-guide/concepts/data-types-and-structures/#floating-point-numbers ‘Polars generally follows the IEEE 754 floating point standard’ (data-types-and-structures.md L178-L179); the function docstrings say nothing on domain errors.

polars-nan-equality: NaN equals NaN: in ==, grouping, unique and joins

Engine: polars py-2.1.0; area: doubles; status: no-finding; evidence: probe.

Verdict: nan_equality: NAN_IS_NAN natively

NaN == NaN is true and NaN != NaN false; is_in([nan]) finds NaN; group_by, unique, n_unique put every NaN in one group, and a join matches NaN keys. NULL compares as NULL (eq_missing makes NULL equal NULL).

The reference: NAN_IS_NAN (#89, an override of IEEE equality) for =, grouping, DISTINCT and joins.

Options: nan_equality, nan_zero_grouping.

Rows of the choice tables: cmp nan_equality (fp64); cmp nan_equality (fp32; evidence engine_test).

Constructs: ==, is_in, group_by, unique, join.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-4; issues #89.

Provenance:

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L317-L372 impl_float_eq_ord! for pf16, f32, f64: NaN equals NaN and is the largest value. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/comparisons/simd.rs#L171-L286 impl_float_total_ord_kernel! for f32 and f64: the column comparison kernels. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L24-L48 canonical_f32 and canonical_f64: x + 0.0, every NaN one quiet NaN, for hashing (group by, unique, joins). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L384-L402 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/sort/mod.rs#L109-L115 Sorting uses TotalOrd::tot_cmp. - 2026-11-04, engine-test, by characterization pipeline round 3 (doubles): Pins NaN equality for Float32: NaN == NaN and -NaN == NaN are true in comparisons, is_in and index_of.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_comparison.py#L313-L474 test_total_ordering_float_series (slow): every pair of 0.0, -0.0, ±1.0, ±NaN, ±inf, NULL as Float32 against Float32 and Float64 against Float32, column against column and against a broadcast first value or literal; ==, !=, <, <=, >, >=, eq_missing, ne_missing must follow reference_ordering_propagating: NaN equals NaN (also -NaN), NaN is above every number including inf, 0.0 equals -0.0, NULL propagates. Eager DataFrame.select, in memory - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_is_in.py#L376-L381 test_is_in_float, Float32 and Float64: NaN is_in([-0.0, -NaN]) and 0.0 is_in([-0.0, -NaN]) are both true (eager) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_index_of.py#L61-L90 test_float, Float32 and Float64: index_of(-NaN) finds NaN and index_of(-0.0) finds 0.0, on the series, its ascending and descending sorts and a chunked copy (eager and lazy collect()) - 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64.

join-nan-zero-null’s recorded differ 1 is a tie order: sort('y') ties the rows 0.0 and -0.0 (both y = 2), and the engines list them in different orders. The join results are the same set of rows on both engines; it is no engine difference. - probe Float64 == Float64 in characterization/polars/probes-streaming.json - probe nan-compare in characterization/polars/probes-streaming.json - probe nan-is-in in characterization/polars/probes-streaming.json - probe group-nan-zero in characterization/polars/probes-streaming.json - probe unique-nan-zero in characterization/polars/probes-streaming.json - probe join-nan-zero-null in characterization/polars/probes-streaming.json

polars-nan-order: NaN sorts above +inf, in comparisons and in sorts

Engine: polars py-2.1.0; area: doubles; status: no-finding; evidence: probe.

Verdict: nan_order: GREATEST natively, ascending and descending

NaN > x is true for every number x, inf < NaN is true, and sort puts NaN after +inf ascending and first (after the NULLs) descending. rank, median and quantile use the same order, so median(1, 2, NaN) is 2.

The reference: GREATEST: NaN above +inf in sorts and in <.

Options: nan_order.

Rows of the choice tables: cmp nan_order (fp64); cmp nan_order (fp32; evidence engine_test); agg.quant nan_order (fp64).

Constructs: <, >, sort, rank, median, quantile.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-4.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (doubles): Polars’s user guide says it generally follows IEEE 754 for Float32 and Float64, with two exceptions: any NaN equals any other NaN and is greater than every non-NaN value, and operations guarantee nothing about the sign of zero or NaN or a NaN’s payload (a sort or group by may make every zero +0 and every NaN a positive NaN). It promises no error bound unless a function says otherwise. The user guide is not versioned; the section reads the same at tag py-2.1.0. NaN above every non-NaN value is stated for both float types, so it settles cmp’s Float32 row; the probes cover Float64.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): Equality, order, sorting and hashing of floats are one generic implementation for f32 and f64 (and pf16), so Float32 behaves as the Float64 probes show.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L317-L372 impl_float_eq_ord! for pf16, f32, f64: NaN equals NaN and is the largest value. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/comparisons/simd.rs#L171-L286 impl_float_total_ord_kernel! for f32 and f64: the column comparison kernels. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L24-L48 canonical_f32 and canonical_f64: x + 0.0, every NaN one quiet NaN, for hashing (group by, unique, joins). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L384-L402 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/sort/mod.rs#L109-L115 Sorting uses TotalOrd::tot_cmp. - 2026-11-04, engine-test, by characterization pipeline round 3 (doubles): Pins the NaN order of Float32 comparisons: NaN is greater than +inf and every number, and equal to NaN. The streaming sort test pins that the streaming engine places NaN, the infinities and the zeros of a Float32 key as the in-memory sort does; it compares the engines and states no order of its own.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_comparison.py#L313-L474 test_total_ordering_float_series (slow): every pair of 0.0, -0.0, ±1.0, ±NaN, ±inf, NULL as Float32 against Float32 and Float64 against Float32, column against column and against a broadcast first value or literal; ==, !=, <, <=, >, >=, eq_missing, ne_missing must follow reference_ordering_propagating: NaN equals NaN (also -NaN), NaN is above every number including inf, 0.0 equals -0.0, NULL propagates. Eager DataFrame.select, in memory - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_sort.py#L280-L395 test_sort_direct_key_dtypes with the f32 key (517 rows with NaN, ±inf, -0.0, NULL), ascending and descending, maintain_order=True: the streaming sort must equal the in-memory sort exactly, with the in-memory and the partitioned (POLARS_SORT_PARTITION_THRESHOLD_BYTES=1, morsels of 7) sort modes - 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe Float64 < Float64 in characterization/polars/probes-streaming.json - probe sort-doubles in characterization/polars/probes-streaming.json - probe rank-descending-nan in characterization/polars/probes-streaming.json - probe median-nan in characterization/polars/probes-streaming.json

polars-nan-sign: A NaN’s sign survives

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: -NaN keeps its sign bit

Negation flips a NaN’s sign bit (-nan is 0xfff8000000000000), abs clears it and sqrt keeps it. The sign of a NaN that arithmetic creates (inf - inf) is the platform’s: positive on arm64, negative on x86.

No upstream test pins this; it may change between versions.

The reference: One NaN (nan_kind: ONE_NAN).

Options: nan_kind.

Rows of the choice tables: nan_kind (fp64); nan_kind (fp32; evidence source).

Constructs: unary -, abs, sqrt.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-4.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (doubles): Polars’s user guide says it generally follows IEEE 754 for Float32 and Float64, with two exceptions: any NaN equals any other NaN and is greater than every non-NaN value, and operations guarantee nothing about the sign of zero or NaN or a NaN’s payload (a sort or group by may make every zero +0 and every NaN a positive NaN). It promises no error bound unless a function says otherwise. The user guide is not versioned; the section reads the same at tag py-2.1.0. So the sign the probes show surviving is what Polars 2.1.0 happens to do: the docs do not promise it, and a sort or group by may make every NaN positive. The Float32 row stays open.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): impl_float_arith_kernel! is one macro instantiated for pf16, f32 and f64, so Float32 and Float64 run the same operations. The streaming engine evaluates expressions through polars-expr on each morsel (StreamExpr in crates/polars-stream/src/expression.rs), so it shares these kernels with the in-memory engine. -x flips the sign bit and abs clears it for f32 as for f64, and sqrt is the hardware square root, which keeps a NaN’s sign. A NaN that arithmetic creates takes the platform’s default NaN.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L13-L19 abs is x.abs(), negation -x. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L124-L126 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/dispatch/pow.rs#L214-L221 - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_float.py#L131-L146 Closest test, test_hash: hash(-nan) == hash(nan) (eager). No upstream test reads the sign bit of a NaN from negation, abs, sqrt or inf - inf.

polars-round: round(): half to even by default

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: rounding: TIE_TO_EVEN; TIE_AWAY_FROM_ZERO with mode='half_away_from_zero'

round() rounds half to even (round_ties_even): 0.5 is 0, 2.5 is 2, -2.5 is -2. round(mode='half_away_from_zero') is Rust’s f64::round, and mode='to_zero' truncates. floor and ceil agree with the reference on every edge case.

The reference: round with rounding: TIE_AWAY_FROM_ZERO (#335), the only value the model accepts. So the writer needs mode='half_away_from_zero', which matches on every row.

Options: rounding.

Rows of the choice tables: fp.round rounding (fp64); fp.round rounding (fp32; evidence engine_test).

Constructs: round.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #335.

Provenance:

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/round.rs#L141-L153 The default mode is f32::round_ties_even or f64::round_ties_even. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/round.rs#L75-L78 Float32 with 0 decimals applies the f32 operation. - 2026-11-04, engine-test, by characterization pipeline round 3 (doubles): The ties at zero digits are exact in Float32, so the test pins round half to even (and half away from zero by option) for fp32 as for fp64.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/lazyframe/test_lazyframe.py#L611-L640 test_round_mode, for Float16, Float32 and Float64 (lazy collect()): half_to_even gives 0.5 → 0.0, 1.5 → 2.0, 2.5 → 2.0, -0.5 → -0.0, -1.5 → -2.0, 2.25 → 2.2 and 2.75 → 2.8 at one digit; half_away_from_zero gives 1.0, 2.0, 3.0, -1.0, -2.0, 2.3, 2.8 - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/expr.py#L2001 round(decimals=0, mode='half_to_even'): half to even is the default - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_decimal.py#L1497-L1525 test_decimal_rounding_widens_precision: Decimal(2, 1) [9.5, -9.5, 0.5].round(0) == [10.0, -10.0, 0.0] (half to even), mode=‘half_away_from_zero’ == [10.0, -10.0, 1.0], eager, in memory. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/round.rs#L384-L390 #[test] test_round_series: [1.003, 2.23222, 3.4352].round(2, default) gives 1.0 first (assert_eq!, exact).

polars-round-digits: round(d): op(x · 10ᵈ) / 10ᵈ, no negative d

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: matches the reference on the edge cases; d < 0 refused

round(d) multiplies by 10ᵈ, rounds (half to even unless mode says otherwise) and divides back; from d = 300 it splits the multiplier into powers of 2 and 5, from d = 326 it returns the input, and a non-finite intermediate returns the input. The two roundings make ties of the decimal text round either way: round(2.675, 2) is 2.68 though the double is below 2.675, and round(0.285, 2) is 0.28.

A negative d is refused by the Python API (OverflowError, decimals is a u32).

The round docs say the half_ modes round to the nearest rounded value, which round(2.675, 2) = 2.68 contradicts: PL-9. The docs are silent on a negative decimals and on d ≥ 326 returning the input: a docs gap.

No upstream test pins this; it may change between versions.

The reference: round to d digits (#335), TIE_AWAY_FROM_ZERO, with negative digits. On the probe’s edge cases at d = 1, 2, 16, 308, 309, 400 Polars and the reference agree.

Options: rounding.

Rows of the choice tables: fp.round rounding (fp64).

Constructs: round.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-9; issues #335.

Provenance:

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe Float64.round(1) in characterization/polars/probes-streaming.json - probe Float64.round(2) in characterization/polars/probes-streaming.json - probe Float64.round(16) in characterization/polars/probes-streaming.json - probe Float64.round(308) in characterization/polars/probes-streaming.json - probe Float64.round(309) in characterization/polars/probes-streaming.json - probe Float64.round(400) in characterization/polars/probes-streaming.json - probe round-digits-ties in characterization/polars/probes-streaming.json - probe round-digits-negative in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.round.html ‘half_’ strategies ‘round all values to the nearest rounded value, only using the strategy to break ties when a value falls exactly between two rounded values’ (SRC py-polars/src/polars/expr/expr.py L2001-L2025). Contradicted: round(2.675, 2) is 2.68 though the double 2.67499999999999982… is nearer 2.67. Drafted as a docs entry (PL-D2). - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/lazyframe/test_lazyframe.py#L558-L584 test_round: lazy collect(), Float32/Float64, (1835.665, 2) expected 1835.67, (1.005, 2) 1.0, (1e-20, 20) 1e-20, rel_tol 1e-5. Py-2.1.0 returns 1835.66 for the default mode and the test passes on tolerance: it does not pin the x * 10^d rounding. No test for d >= 300. - 2026-11-04, probe, by characterization pipeline reverse pass - pl.LazyFrame({‘x’: [2.675, 1835.665, 0.285, 1.005]}).select(e=pl.col(‘x’).round(2), a=pl.col(‘x’).round(2, mode=‘half_away_from_zero’)).collect(engine=…) with POLARS_IDEAL_MORSEL_SIZE=64: in-memory and streaming both e [2.68, 1835.66, 0.28, 1.0], a [2.68, 1835.67, 0.28, 1.0]; Python round gives [2.67, 1835.66, 0.28, 1.0] (Polars 2.1.0, macOS arm64).

polars-scalar-divisor: Double /, //, % by a scalar: a multiplication by the reciprocal

Engine: polars py-2.1.0; area: doubles; status: decided; evidence: probe.

Verdict: divide of doubles by a scalar rounds twice (2u, approx) and overflows below the smallest normal divisor; // and % give wrong results

With a length-1 divisor (a literal) against a column of two or more rows, Polars’s in-memory engine computes x / c as x * (1 / c): 49.0 / 49.0 is 0.9999999999999999, and 0.0 / 1e-310 is NaN (1 / 1e-310 is inf, and 0 * inf NaN). // and % divide the same way. A column divisor of the same length divides as IEEE does. Here the streaming engine gave the IEEE quotient on four rows, because a frame shorter than the thread count streams as one-row morsels; on longer frames, or on one thread, streaming takes the reciprocal too (polars-double-scalar-divisor, PL-3).

The same finding as polars-double-scalar-divisor (PL-3), which carries the source reading, the streaming probes (the reciprocal path also runs on streaming once a morsel holds two or more rows: s-scalar-divisor, 9,000 rows) and the ulp bound (polars-ulp-arith). This fact keeps its id for the engine tables (SubstraitLean/EngineTables.lean) that cite it.

The reference: divide of doubles rounds the exact quotient once (TIE_TO_EVEN), and 0.0 / c is 0.0 for a finite nonzero c. Decided on #444: Polars’s / is approx with 2u per division, under the precondition that a divisor is 0 or not subnormal; // and % floor the quotient (49.0 % 49.0 is 49.0), so their translation broadcasts the divisor to a column (column_divisor).

Options: rounding, nan_creation.

Constructs: /, //, %.

Provenance:

  • 2026-11-04, probe, by Claude session frosty-bassi-b2cf7d (#444 step 3a, PR #590): Polars 2.1.0 (Python) on macOS arm64.
    • pl.DataFrame({'x': [49.0, 0.0, 7.0, -0.0]}).select(pl.col('x') / pl.lit(49.0)) gives [0.9999999999999999, 0.0, 0.14285714285714285, -0.0]
    • pl.DataFrame({'x': [49.0, 0.0, 7.0, -0.0]}).select(pl.col('x') / pl.lit(1e-310)) gives [inf, nan, inf, nan]
    • pl.DataFrame({'x': [49.0, 0.0, 7.0, -0.0]}).lazy().select(pl.col('x') / pl.lit(49.0)).collect(engine='streaming') gives [1.0, 0.0, 0.14285714285714285, -0.0]
    • pl.DataFrame({'x': [1e-300, 49.0, 0.0]}).select(pl.col('x') / pl.lit(1e-310)) gives [inf, inf, nan]
    • pl.DataFrame({'x': [1e-300, 49.0, 0.0]}).select(pl.col('x') % pl.lit(49.0)) gives [1e-300, 49.0, 0.0]
  • 2026-11-04, probe, by characterization pipeline (merging duplicates after #636): The same finding as polars-double-scalar-divisor (PL-3), which carries the source reading, the streaming probes (the reciprocal path also runs on streaming once a morsel holds two or more rows: s-scalar-divisor, 9,000 rows) and the ulp bound (polars-ulp-arith). This fact keeps its id for the engine tables (SubstraitLean/EngineTables.lean) that cite it.
  • 2026-11-04, source-reading, by characterization pipeline reverse pass: Corrected the description: the four-row streaming result is a thread-count effect; with POLARS_MAX_THREADS=1 the same frame gives 0.9999999999999999 (polars-double-scalar-divisor’s reverse-pass repro).

polars-ulp-arith: + - * / round once; / by a literal and x ** n round more

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: column ops ≤ 0.5 ulp; x / c < 1.5 ulp (1.2 seen); x ** n for literal n in 3..9 up to n − 1 ulp (3.8 seen)

+, -, * and / of two columns are one IEEE operation in the column’s type (Float32 stays binary32): correctly rounded, ≤ 0.5 ulp, on every platform.

x / c with a literal (or any length-1) divisor is x * (1 / c) (PL-3, polars-double-scalar-divisor): two roundings. Bound: 1 / c is off by at most 2^-53 relative (2^-24 for Float32), which is at most 1 ulp of the quotient, plus the product’s 0.5, so < 1.5 ulp. Observed, fp64 / fp32: c = 49: 1.20 / 0.84 ulp; c = 7.3: 1.20 / 0.81; c = 0.1: 1.00 / 0.75; c = 10: 0.80 / 0.60; c = 3: 0.67 / 0.67. A third to a half of the quotients miss the correctly rounded value.

x ** n with a literal integer n from 2 to 9 multiplies x by itself n − 1 times (polars-elem-rounding), rounding after each product: n − 1 roundings, so up to about n − 1 ulp. x ** 2 is one product, correctly rounded. Observed max, fp64 / fp32: n = 3: 1.24 / 1.24; 4: 1.55 / 1.63; 5: 2.16 / 2.27; 6: 2.51 / 2.44; 7: 3.43 / 2.96; 8: 3.17 / 3.50; 9: 3.59 / 3.78 ulp, with 25 to 55 % of results off. DuckDB’s pow(x, n) on the same inputs stays within 0.51 ulp (libm pow). These are hardware operations: the bounds hold on every platform.

No upstream test pins this; it may change between versions.

The reference: add, subtract, multiply and divide round once (rounding: TIE_TO_EVEN); power is one libm pow call in Lean’s Float. sverify’s ε (#85) for Polars has to take 1.5 ulp for a division by a literal and n − 1 ulp for x ** n, or shim them (divide by a column, pow with a non-literal exponent).

Options: rounding, fp32.

Rows of the choice tables: fp.arith rounding (fp32, fp64); fp.div rounding (fp32, fp64); fp.elem rounding (fp32, fp64).

Constructs: +, -, *, /, **.

Settings: {"engine": "streaming", "platform": "macOS arm64"}.

Provenance:

  • 2026-11-04, source-reading, by characterization pipeline round 5 (ulp bounds)
  • 2026-11-04, probe, by characterization pipeline round 5 (ulp bounds): Probe script ulp_probe.py (session scratchpad, not in the repository), run from python/ as uv run --with mpmath python ulp_probe.py --n 3000 --seed 1 --out ulp.json: about 3 000 inputs per operation and type, random and hard cases, against mpmath at 256 bits. ‘bad’ counts results that differ from the correctly rounded value; the max is in ulps of the result type at the exact value’s binade. DuckDB runs on a table loaded from Parquet, Polars on a LazyFrame with collect(engine=‘streaming’), so the column kernels run. DuckDB 2.0.0 and Polars 2.1.0 (Python wheels), macOS 26.6 arm64, Python 3.12.11, mpmath 1.3.0.

Elementwise kernels are shared by the streaming and in-memory engines (polars-elem-rounding, polars-double-scalar-divisor); the probe collects with engine=‘streaming’ on 3 000-row frames, so morsels hold more than one row and the literal paths run. - pl.LazyFrame({‘a’: X, ‘b’: Y}).select(a + b, a - b, a * b, a / b).collect(engine=‘streaming’), Float64 and Float32, 3 000 rows: max 0.5000, 0.5000, 0.4994, 0.4999 ulp (fp64), 0.5 each (fp32); 0 off the correctly rounded value. - select(pl.col(‘a’) / pl.lit(c, dtype=…)) for c in 49, 3, 10, 0.1, 7.3: fp64 max 1.2041 (1 490 of 3 000 off), 0.6667 (1 040), 0.8000 (1 018), 0.9999 (975), 1.2006 (1 550); fp32 0.8367 (755), 0.6667 (1 044), 0.6000 (606), 0.7500 (450), 0.8073 (670). Worst fp64 case: -0.8922618801772502 / 7.3. - select(pl.col(‘x’) ** n) for n = 2 … 9, x in ±[0.5, 2]: fp64 max 0.49999, 1.2436, 1.5533, 2.1604, 2.5084, 3.4332, 3.1737, 3.5886 ulp (0, 772, 1 084, 1 264, 1 383, 1 520, 1 589, 1 656 of 3 000 off); fp32 0.4998, 1.2412, 1.6263, 2.2720, 2.4416, 2.9587, 3.5025, 3.7812. Worst fp64: 1.7975193975534105 ** 7. - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L21-L31 column kernels: add, sub, mul are one IEEE operation per pair (prim_binary_values). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L109-L112 prim_true_div: l / r, one rounding. - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/user-guide/concepts/data-types-and-structures/#floating-point-numbers No error guarantee ‘unless mentioned otherwise’; no operator documents its rounding. No upstream test pins the ulp error of / by a literal or of x ** n.

polars-ulp-libm: Elementary functions and logarithms: the platform libm per type; logs divide

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: fp64 as the libm (tan 2.3 ulp on macOS); log10, log(b) 1.5 to 2.1 ulp; Float32 in single precision, log() 1.5 ulp

Each function is Rust’s f64 or f32 method, so Float64 calls the libm’s double function and Float32 its float one (sinf, expf, …) and returns Float32 (polars-elem-rounding). sqrt is the hardware instruction, correctly rounded. log(base) is Rust’s x.log(base), ln(x) / ln(base) in the column’s type (polars-log-base, PL-4); log10() is log(10), and there is no log2, so log(2) serves. For Float64 the default base e rounds to a double whose ln is 1.0, so log() is ln.

Max error in ulps, measured on macOS arm64 (3 000 inputs each), fp64 / fp32: sqrt 0.5 / 0.5; exp 0.502 / 0.511; log() 0.500 / 1.500 (528 of 746 fp32 results off, PL-4); log10() 2.09 / 1.53; log(2) 1.46 / 1.15; log(3) 2.05 / 1.53; log(b) with a column base 2.13 / 1.63; ** with a column exponent (powf) 0.504 / 0.510; sin 0.77 / 0.77; cos 0.83 / 0.72; tan 2.30 / 0.63; arcsin 0.61 / 0.63; arccos 0.89 / 0.63; arctan 0.72 / 0.65; arctan2 0.56 / 0.68.

The fp64 functions return the same values as DuckDB’s on every probed input (same libm). Unlike DuckDB, log10 and log(2) divide two logarithms where DuckDB calls log10 and log2 (≤ 0.502 ulp), and Float32 computes in single precision where DuckDB widens to double: the macOS float functions stay below 0.8 ulp but miss the correctly rounded value up to 30 % of the time (arctan 893 of 3 000).

Linux x86_64 (glibc, version unknown; the glibc 2.39 manual’s bounds): double exp, log, sin, cos, asin, acos, atan, pow 1 ulp, tan and atan2 no known error; float expf, logf, sinf, cosf, tanf, asinf, acosf, atanf, powf 1 ulp, atan2f 2 ulp. Polars’s log10 and log(b) add about 1.5 ulp to the libm’s log error by their formula. To be measured on homelab.

Linux x86_64 (glibc 2.39), same 3000 inputs per function: most functions are within about 0.5 ulp where macOS is not (sin 0.50 against 0.77, acos 0.50 against 0.89, tan at moderate arguments 0.51 against 2.4), but cos and tan of DOUBLE reach 7.95 and 14.36 ulp at x = 5.319372648326541e+255, near a zero of cos (glibc gives cos = -4.68716592425462e-19, the exact value is -4.6871659242546276e-19; macOS is within 0.05 ulp there, checked with mpmath at 3000 bits). log10 reaches 1.35 ulp on Linux (0.50 on macOS) and log(b, x) 1.47 to 1.56. So no libm-backed function has a platform-independent bound below glibc’s: take the larger of the two columns, and more for huge arguments of cos and tan.

No upstream test pins this; it may change between versions.

The reference: The reference computes these with Lean’s Float (the platform’s libm, in double) and fixes rounding: TIE_TO_EVEN; the last bits are each engine’s (#54). sverify’s ε (#85) per function, type and platform comes from this table; Float32 would round to binary32 in the reference.

Options: rounding, fp32.

Rows of the choice tables: fp.elem rounding (fp32, fp64); fp.log rounding (fp32, fp64).

Constructs: sqrt, exp, log, log10, sin, cos, tan, arcsin, arccos, arctan, arctan2, **.

Settings: {"engine": "streaming", "platform": ["macOS arm64", "Linux x86_64"]}.

Provenance:

Elementwise kernels are shared by the streaming and in-memory engines (polars-elem-rounding, polars-double-scalar-divisor); the probe collects with engine=‘streaming’ on 3 000-row frames, so morsels hold more than one row and the literal paths run. - pl.LazyFrame({‘x’: xs}, schema={‘x’: pl.Float64}).select(f(pl.col(‘x’))).collect(engine=‘streaming’): max ulp (off of 3 000): sqrt 0.49998 (0), exp 0.5015 (5), log() 0.5000 (0), log10() 2.0931 (1 735), log(2) 1.4574 (983), sin 0.7687 (108), cos 0.8319 (118), tan 2.3017 (1 252), arcsin 0.6068 (121), arccos 0.8857 (274), arctan 0.7217 (13), arctan2 0.5561 (10), a ** b 0.5044 (4), x.log(b) with a column b 2.1326 (1 200), log(10) literal 2.1326 (1 787), log(3) literal 2.0527 (1 625). Worst log10: x = 2.208308239418851e-256. - The same with schema Float32 (results Float32, ulps of binary32): sqrt 0.49996 (0), exp 0.5113 (5 of 1 697), log() 1.4999 (528 of 746), log10() 1.5296 (166), log(2) 1.1541 (167), sin 0.7691 (45 of 2 323), cos 0.7175 (35), tan 0.6323 (126), arcsin 0.6333 (571 of 3 000), arccos 0.6333 (294), arctan 0.6526 (893), arctan2 0.6828 (312), a ** b 0.5105 (5 of 1 149), x.log(b) 1.6333 (631 of 2 115), log(10) literal 1.5169, log(3) literal 1.5264. - 2026-11-04, probe, by characterization pipeline platform run (homelab, Linux x86_64, glibc 2.39-0ubuntu8.9; scratchpad ulp_probe.py –n 3000 –seed 1, the same inputs as the macOS run): Same seed and inputs as the macOS measurement. - ulp_probe.py output on homelab: cos fp64 max 7.954598 ulp, tan fp64 max 14.360645 ulp at 5.319372648326541e+255; sin 0.501170, asin 0.501869, acos 0.501590, atan 0.499671, atan2 0.499917, log10 1.347590 (DuckDB), log(b, x) 1.470205 (DuckDB), 2.132601 (Polars) - python3 -c 'import math; x=5.319372648326541e+255; print(repr(math.cos(x)), repr(math.tan(x)))' gives -4.68716592425462e-19 -2.1334853857537075e+18 - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/rust-toolchain.toml#L1-L2 Polars py-2.1.0 builds with nightly-2026-09-01, so the std docs to read are the nightly ones, not 1.90.0. - 2026-11-04, docs, by characterization pipeline reverse pass - https://doc.rust-lang.org/nightly/std/primitive.f64.html#method.sin Rust std: ‘Unspecified precision’: varies by platform, Rust version and invocation; corresponds to the libc function. The same note is on exp, ln, log, powf, tan, asin, acos, atan, atan2. - https://docs.pola.rs/user-guide/concepts/data-types-and-structures/#floating-point-numbers Polars gives no error guarantee ‘unless mentioned otherwise’. - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/series/test_series.py#L1872-L1900 test_trigonometric compares with numpy at rel_tol 1e-5; no upstream test pins any function’s ulp error.

polars-ulp-round: round(d): one rounding after the decimal pick

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: result ≤ 0.5 ulp from the decimal it picks; the pick follows x·10^d, off the exact decimal on near-ties

round(d) for Float64 is round_half_even(x · 10^d) / 10^d (10^d by powi, exact up to 10^22), for Float32 the same in f64 and cast to f32 (polars-round-digits). The division rounds n / 10^d once.

Bounds: the fp64 result is within 0.5 ulp of n / 10^d (observed max 0.4999 for d up to 12); the Float32 result within 0.5 ulp of binary32 in the probes, after two roundings. The pick n differs from the exact decimal rounding of x where x · 10^d rounds onto or across a tie: 0.6 to 3.9 % of random decimals with d to d + 4 digits (6 to 39 of 1 000), about half of decimal-text ties (round(39.55, 1) is 39.6 though the double is below 39.55). Float32 inputs gave no wrong pick. Hardware operations only (and powi for d > 22): platform-independent.

The round docs’ “nearest rounded value” is contradicted by the picks: PL-9. No upstream test pins this; it may change between versions.

The reference: The reference rounds with DuckDB’s formula, TIE_AWAY_FROM_ZERO (#335); Polars’s default mode is half to even (polars-round, polars-round-digits). Only the final division’s 0.5 ulp is rounding error in the #85 sense.

Options: rounding, fp32.

Rows of the choice tables: fp.round rounding (fp32, fp64).

Constructs: round.

Settings: {"engine": "streaming", "platform": "macOS arm64"}.

See: upstream ledger PL-9.

Provenance:

  • 2026-11-04, source-reading, by characterization pipeline round 5 (ulp bounds)
  • 2026-11-04, probe, by characterization pipeline round 5 (ulp bounds): Probe script ulp_probe.py (session scratchpad, not in the repository), run from python/ as uv run --with mpmath python ulp_probe.py --n 3000 --seed 1 --out ulp.json: about 3 000 inputs per operation and type, random and hard cases, against mpmath at 256 bits. ‘bad’ counts results that differ from the correctly rounded value; the max is in ulps of the result type at the exact value’s binade. DuckDB runs on a table loaded from Parquet, Polars on a LazyFrame with collect(engine=‘streaming’), so the column kernels run. DuckDB 2.0.0 and Polars 2.1.0 (Python wheels), macOS 26.6 arm64, Python 3.12.11, mpmath 1.3.0.

Elementwise kernels are shared by the streaming and in-memory engines (polars-elem-rounding, polars-double-scalar-divisor); the probe collects with engine=‘streaming’ on 3 000-row frames, so morsels hold more than one row and the literal paths run. - select(pl.col(‘x’).round(d)) for d in 0, 1, 2, 3, 5, 8, 12 on the inputs of duckdb-ulp-round (Float64): final rounding ≤ 0.4000, 0.4800, 0.4960, 0.4998, 0.4998, 0.4999 ulp (d = 1 … 12); wrong pick 0, 697, 827, 864, 831, 771, 940 of 3 000 (0, 7, 12, 16, 10, 6, 39 of the 1 000 random decimals). - The same on Float32: ≤ 0.5 ulp of binary32 from the exact decimal rounding, 0 wrong picks for every d. - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/lazyframe/test_lazyframe.py#L558-L584 test_round uses rel_tol 1e-5: (1835.665, 2) expects 1835.67 and py-2.1.0 returns 1835.66 in the default mode; the wrong-pick rate is not pinned by any upstream test. - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.round.html ‘nearest rounded value’ wording contradicted by the x * 10^d pick (PL-D2).

polars-zero-add: x + 0.0 and 0.0 - x keep or flip a zero’s sign

Engine: polars py-2.1.0; area: doubles; status: shimmed; evidence: probe.

Verdict: zero_sign: not IEEE either, so up to one zero (canon)

-0.0 + 0.0 is -0.0 and 0.0 - 0.0 (as 0.0 - x) is -0.0, where IEEE 754 gives +0.0 for both under round-to-nearest; -0.0 - 0.0 is -0.0, as IEEE gives. The sign of a zero result of + and - is therefore Polars’s own (polars-zero-sign).

The same finding as polars-zero-sign-streaming (which supersedes polars-zero-sign): the literal shortcuts x + 0.0 → x and 0.0 - x → -x run on both engines once a morsel holds more than one row; on streaming frames under about 100 rows (one-row morsels) the IEEE zero comes back (s-zero-sign-literal). This fact keeps its id for the engine tables that cite it.

The reference: One zero (zero_sign: ONE_ZERO, #89): -0.0 is 0.0 everywhere, so the translation canonicalises (one_zero) where a result’s zeros are observed.

Options: zero_sign.

Constructs: +, -.

Provenance:

  • 2026-11-04, probe, by Claude session frosty-bassi-b2cf7d (#444 step 3a, PR #590): Polars 2.1.0 (Python) on macOS arm64.
    • pl.DataFrame({'x': [-0.0, -0.0]}).select(pl.col('x') + 0.0) gives [-0.0, -0.0]
    • pl.DataFrame({'x': [0.0, 0.0]}).select(pl.lit(0.0) - pl.col('x')) gives [-0.0, -0.0]
    • pl.DataFrame({'x': [-0.0, -0.0]}).select(pl.col('x') - 0.0) gives [-0.0, -0.0]
  • 2026-11-04, probe, by characterization pipeline (merging duplicates after #636): The same finding as polars-zero-sign-streaming (which supersedes polars-zero-sign): the literal shortcuts x + 0.0 → x and 0.0 - x → -x run on both engines once a morsel holds more than one row; on streaming frames under about 100 rows (one-row morsels) the IEEE zero comes back (s-zero-sign-literal). This fact keeps its id for the engine tables that cite it.

polars-zero-order: -0.0 equals 0.0 in comparisons, sorts, grouping and joins

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: one zero for equality and order; which zero a group or unique keeps depends on maintain_order

0.0 == -0.0 is true, sort treats them as equal (with maintain_order the input order stays), and group_by, unique and joins put them together. Without maintain_order, the group’s key and the surviving row of unique carry the zero seen first, which can be -0.0; with group_by(maintain_order=True) the key was 0.0 in a probe where -0.0 came first. Polars promises neither. Arithmetic still tells the zeros apart (polars-zero-sign).

No upstream test pins which zero a group key or unique keeps, or zero grouping on the streaming engine; it may change between versions.

The reference: ONE_ZERO (#89): -0.0 is 0.0 everywhere, so a group key is never -0.0.

Options: zero_sign, nan_zero_grouping.

Rows of the choice tables: cmp zero_sign (fp64); cmp zero_sign (fp32; evidence engine_test).

Constructs: ==, sort, group_by, unique, join.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-4; issues #89.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (doubles): Polars’s user guide says it generally follows IEEE 754 for Float32 and Float64, with two exceptions: any NaN equals any other NaN and is greater than every non-NaN value, and operations guarantee nothing about the sign of zero or NaN or a NaN’s payload (a sort or group by may make every zero +0 and every NaN a positive NaN). It promises no error bound unless a function says otherwise. The user guide is not versioned; the section reads the same at tag py-2.1.0. Comparisons are not among the exceptions, so -0.0 == 0.0 holds in Float32 as IEEE 754 compares them, and a sort or group by may make every zero +0, which puts them together. Settles cmp’s Float32 row at docs level. The docs do not promise which zero a group key carries.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): Equality, order, sorting and hashing of floats are one generic implementation for f32 and f64 (and pf16), so Float32 behaves as the Float64 probes show.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L317-L372 impl_float_eq_ord! for pf16, f32, f64: NaN equals NaN and is the largest value. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/comparisons/simd.rs#L171-L286 impl_float_total_ord_kernel! for f32 and f64: the column comparison kernels. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L24-L48 canonical_f32 and canonical_f64: x + 0.0, every NaN one quiet NaN, for hashing (group by, unique, joins). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/total_ord.rs#L384-L402 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/sort/mod.rs#L109-L115 Sorting uses TotalOrd::tot_cmp. - 2026-11-04, engine-test, by characterization pipeline round 3 (doubles): Pins 0.0 == -0.0 (and neither < nor >) for Float32, in comparisons, is_in and index_of; the streaming sort of a key with both zeros equals the in-memory one with maintain_order=True.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_comparison.py#L313-L474 test_total_ordering_float_series (slow): every pair of 0.0, -0.0, ±1.0, ±NaN, ±inf, NULL as Float32 against Float32 and Float64 against Float32, column against column and against a broadcast first value or literal; ==, !=, <, <=, >, >=, eq_missing, ne_missing must follow reference_ordering_propagating: NaN equals NaN (also -NaN), NaN is above every number including inf, 0.0 equals -0.0, NULL propagates. Eager DataFrame.select, in memory - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_is_in.py#L376-L381 test_is_in_float, Float32 and Float64: NaN is_in([-0.0, -NaN]) and 0.0 is_in([-0.0, -NaN]) are both true (eager) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_index_of.py#L61-L90 test_float, Float32 and Float64: index_of(-NaN) finds NaN and index_of(-0.0) finds 0.0, on the series, its ascending and descending sorts and a chunked copy (eager and lazy collect()) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_sort.py#L280-L395 test_sort_direct_key_dtypes with the f32 key (517 rows with NaN, ±inf, -0.0, NULL), ascending and descending, maintain_order=True: the streaming sort must equal the in-memory sort exactly, with the in-memory and the partitioned (POLARS_SORT_PARTITION_THRESHOLD_BYTES=1, morsels of 7) sort modes - 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

s-zero-key-first groups and dedups 100 000 zeros, -0.0 first, lazily on each engine: group_by without maintain_order, unique() and unique(maintain_order=True) keep -0.0, the zero seen first; group_by(maintain_order=True) gives the key 0.0, on both engines. That is not the zero seen first, which the description says; reported for review, description unchanged.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64.

join-nan-zero-null’s recorded differ 1 is a tie order: sort('y') ties the rows 0.0 and -0.0 (both y = 2), and the engines list them in different orders. The join results are the same set of rows on both engines; it is no engine difference. - probe nan-compare in characterization/polars/probes-streaming.json - probe sort-zero-sign in characterization/polars/probes-streaming.json - probe group-nan-zero in characterization/polars/probes-streaming.json - probe unique-nan-zero in characterization/polars/probes-streaming.json - probe join-nan-zero-null in characterization/polars/probes-streaming.json - probe s-zero-key-first in characterization/polars/probes-streaming.json - 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Corrects the verdict and description after s-zero-key-first, on both engines: group_by without maintain_order and unique (with and without maintain_order) keep -0.0, the zero seen first, but group_by(maintain_order=True) gives the key 0.0. The description had said the key is always the zero seen first. - probe s-zero-key-first in characterization/polars/probes-streaming.json - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_float.py#L149-L193 test_group_by_float: -0.0 and 0.0 land in one group (and -nan with nan), keys x and (x, a), maintain_order True and False, eager, in memory. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_float.py#L67-L128 test_unique, test_unique_counts: [-0.0, -nan, 0.0, None, 1.0, nan] has 4 unique values, counts [2, 2, 1, 1]; the expected 0.0 does not pin the sign (assert_series_equal holds -0.0 == 0.0). - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_float.py#L196 test_joins: -0.0 joins 0.0, NaN joins NaN, eager. No upstream test pins which zero a group key or unique row keeps, nor zero grouping on the streaming engine.

polars-zero-sign: -0.0 in arithmetic: two zeros

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: IEEE: 1 / -0.0 is -inf

Arithmetic keeps the sign of zero: 1.0 / -0.0 is -inf, -(0.0) and 0.0 * -1 are -0.0, sqrt(-0.0) is -0.0. Comparison, grouping and sorting are in polars-zero-order.

With a literal operand on a frame of two rows or more, the kernels take shortcuts that move the sign of a zero away from IEEE: x + 0.0 and x - 0.0 return x, so -0.0 + 0.0 and -0.0 - (-0.0) are -0.0 (IEEE gives 0.0), and 0.0 - x is -x, so 0.0 - 0.0 is -0.0. Two columns use the IEEE operations.

The reference: One zero (zero_sign: ONE_ZERO, #89): -0.0 is 0.0 everywhere.

Options: zero_sign.

Rows of the choice tables: fp.arith zero_sign (fp64); fp.neg zero_sign (fp64); fp.div zero_sign (fp64); fp.arith zero_sign (fp32; evidence source); fp.neg zero_sign (fp32; evidence source); fp.div zero_sign (fp32; evidence source).

Constructs: /, unary -, *, sqrt.

Settings: {"engine": "in-memory"}.

See: upstream ledger ST-4; issues #89; superseded by polars-zero-sign-streaming.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (doubles): Polars’s user guide says it generally follows IEEE 754 for Float32 and Float64, with two exceptions: any NaN equals any other NaN and is greater than every non-NaN value, and operations guarantee nothing about the sign of zero or NaN or a NaN’s payload (a sort or group by may make every zero +0 and every NaN a positive NaN). It promises no error bound unless a function says otherwise. The user guide is not versioned; the section reads the same at tag py-2.1.0. So the two zeros the probes show are what Polars 2.1.0 happens to do: the docs do not promise them, and a sort or group by may turn -0.0 into 0.0. The Float32 rows stay open.
  • 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): impl_float_arith_kernel! is one macro instantiated for pf16, f32 and f64, so Float32 and Float64 run the same operations. The streaming engine evaluates expressions through polars-expr on each morsel (StreamExpr in crates/polars-stream/src/expression.rs), so it shares these kernels with the in-memory engine. The column kernels are the IEEE operations, so Float32 keeps two zeros as Float64 does.

The kernels for a literal operand take shortcuts that move the sign of a zero away from IEEE, at both widths: x + 0.0 and x - 0.0 return x unchanged, so -0.0 + 0.0 and -0.0 - (-0.0) are -0.0 (IEEE gives 0.0), and 0.0 - x is -x, so 0.0 - 0.0 is -0.0. A literal is a length-1 column, so the shortcut runs on frames of two rows or more; on one row both sides have length 1 and the column kernel runs.

Read at tag py-2.1.0 (commit 7f902bd3). The streaming engine evaluates expressions through polars-expr on each morsel (crates/polars-stream/src/expression.rs), so these kernels serve both engines. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L8-L43 The column kernels: -x, l + r, l - r, l * r, l / r. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L45-L76 The literal kernels: x + 0 and x - 0 return x, 0 - x is -x, x * ±1 is x or -x. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L109-L126 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/arithmetic/numeric.rs#L105-L163 + - * // % on two ChunkedArrays: equal lengths take the array kernel, a length-1 side the scalar kernel. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/arity.rs#L894-L942 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/column/mod.rs#L1816-L1839 A literal is a scalar column, passed to the operation as a one-value Series. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/expression.rs#L10-L38 The streaming engine evaluates each expression with polars-expr’s PhysicalExpr on every morsel, so it runs the same kernels. - 2026-11-04, manual-repro, by characterization pipeline round 2 (doubles): Confirms the literal shortcuts on Float64. - pl.DataFrame({‘x’: [49.0, 3.0, -0.0, 0.0, 0.5]}).select(a=pl.col(‘x’) + 0.0, s=pl.col(‘x’) - (-0.0), l=pl.lit(0.0) - pl.col(‘x’)) gave a = -0.0 and s = -0.0 for x = -0.0, and l = -0.0 for x = 0.0 (Polars 2.1.0, macOS arm64). - 2026-11-04, source-reading, by characterization pipeline round 2 (doubles): Adds the second paragraph of the description: the literal shortcuts for x + 0, x - 0 and 0 - x, read in the source and seen in the manual repro of this round.

Read at tag py-2.1.0 (commit 7f902bd3). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L45-L65 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/arithmetic/numeric.rs#L105-L163 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/arity.rs#L894-L942 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/expression.rs#L10-L38 - 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): In memory, s-zero-sign-literal shows the shortcuts on 4 rows and on 6000. Streaming takes them only in morsels of two rows or more: polars-zero-sign-streaming, which supersedes this fact for the streaming engine. - probe s-zero-sign-literal in characterization/polars/probes-streaming.json

polars-zero-sign-streaming: -0.0 in arithmetic on the streaming engine: two zeros, literal shortcuts per morsel

Engine: polars py-2.1.0; area: doubles; status: open; evidence: probe.

Verdict: IEEE: 1 / -0.0 is -inf; with a literal operand the sign depends on the morsel’s length

Arithmetic keeps the sign of zero: 1.0 / -0.0 is -inf, -(0.0) and 0.0 * -1 are -0.0, sqrt(-0.0) is -0.0. Comparison, grouping and sorting are in polars-zero-order.

The streaming engine runs the in-memory kernels on each morsel. With a literal operand, a morsel of two rows or more takes the literal kernels’ shortcuts: x + 0.0 and x - 0.0 return x, so -0.0 + 0.0 and -0.0 - (-0.0) are -0.0, and 0.0 - x is -x, so 0.0 - 0.0 is -0.0. A morsel of one row takes the IEEE operations (0.0 in those three cases).

Which rows land in a one-row morsel depends on the frame’s size and the machine’s thread count: on 10 cores every row of a frame of up to 10 rows did, so -0.0 + 0.0 was 0.0 there and -0.0 in memory; from about 100 rows none did. The in-memory engine’s behaviour is in polars-zero-sign.

polars-zero-add records the literal-shortcut part of this finding from #590’s probe and is the id the engine tables cite.

No upstream test pins this; it may change between versions.

The reference: One zero (zero_sign: ONE_ZERO, #89): -0.0 is 0.0 everywhere.

Options: zero_sign.

Rows of the choice tables: fp.arith zero_sign (fp64); fp.neg zero_sign (fp64); fp.div zero_sign (fp64); fp.arith zero_sign (fp32; evidence source); fp.neg zero_sign (fp32; evidence source); fp.div zero_sign (fp32; evidence source).

Constructs: /, unary -, *, sqrt, +, -.

Settings: {"engine": "streaming"}.

See: issues #89; supersedes polars-zero-sign.

Provenance:

  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): zero-sign-arith (1 / z, -z, z * -1, sqrt on -0.0 and 0.0) gives the same on both engines; for these the shortcuts and IEEE agree.

s-zero-sign-literal counts the negative zeros of x + 0.0, x - (-0.0), x - 0.0 and 0.0 - x over x = -0.0, 0.0, … : on 6000 rows 3000 in each column on both engines (the shortcuts); on 4 rows in memory 2 in each (the shortcuts), streaming 0, 0, 2, 0 (IEEE).

A manual repro of x + 0.0 on n copies of -0.0 under streaming gave IEEE’s 0.0 in every row for n up to 10 and the shortcut’s -0.0 in all rows from n = 100; between them one row of some sizes was 0.0 (n = 11: 10 of 11 rows -0.0). In memory every size gave -0.0. Streaming splits a small frame into several morsels on this 10-core machine, and a one-row morsel takes the column kernel, as for the scalar divisor (polars-double-scalar-divisor, PL-3).

python -m sverify_dev.probe_polars --diff --write collects each s- observation’s lazy query with engine='in-memory' and engine='streaming'. Polars 2.1.0 (Python), macOS arm64. - probe zero-sign-arith in characterization/polars/probes-streaming.json - probe s-zero-sign-literal in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline round 5 (streaming engine): Carried over from polars-zero-sign’s round-2 reading; the morsel’s length decides between the kernels, which the probe confirms. impl_float_arith_kernel! serves f32 and f64 alike, so the Float32 rows rest on the source. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/arithmetic/float.rs#L45-L76 The literal kernels: x + 0 and x - 0 return x, 0 - x is -x, x * ±1 is x or -x. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/arithmetic/numeric.rs#L105-L163 Equal lengths take the array kernel, a length-1 side the scalar kernel. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/expression.rs#L10-L38 The streaming engine evaluates each expression with polars-expr’s PhysicalExpr on every morsel. - 2026-11-04, probe, by characterization pipeline (merging duplicates after #636): polars-zero-add records the literal-shortcut part of this finding from #590’s probe and is the id the engine tables cite. - probe s-zero-sign-literal in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/user-guide/concepts/data-types-and-structures/#floating-point-numbers ‘Operations do not guarantee any particular behavior on the sign of zero’ (SRC docs/source/user-guide/concepts/data-types-and-structures.md L180-L184). No upstream test of the literal kernels’ zero shortcuts or of the sign of a zero result.

strings

polars-case: to_uppercase, to_lowercase: Rust’s Unicode mappings

Engine: polars py-2.1.0; area: strings; status: no-finding; evidence: probe.

Verdict: char_set: UTF8, Rust’s str::to_uppercase

Case mapping is Rust’s str::to_uppercase and to_lowercase: full Unicode mappings, which can change the length (ß is SS, İ lowercases to i̇), with a final sigma (ΣΑΣ lowercases to σας). On ASCII they agree with the reference on every probe row.

The reference: upper, lower with char_set: UTF8, a black box defined as Rust’s str::to_uppercase (Text.upper); Lean evaluates them on ASCII only.

Options: char_set.

Rows of the choice tables: str.case char_set (string).

Constructs: str.to_uppercase, str.to_lowercase.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-10.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe String.str.to_uppercase() in characterization/polars/probes-streaming.json - probe String.str.to_lowercase() in characterization/polars/probes-streaming.json - probe case-unicode in characterization/polars/probes-streaming.json

polars-concat: concat_str: NULL in, NULL out; ignore_nulls=True skips them

Engine: polars py-2.1.0; area: strings; status: no-finding; evidence: probe.

Verdict: null_handling: ACCEPT_NULLS by default; IGNORE_NULLS with the flag

pl.concat_str(a, b) and a + b are NULL when an input is NULL; concat_str(..., ignore_nulls=True) skips NULLs.

The reference: str_concat with null_handling: ACCEPT_NULLS (NULL in, NULL out); IGNORE_NULLS refused.

Options: null_handling.

Rows of the choice tables: str.concat null_handling (string).

Constructs: concat_str, +.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe pl.concat_str(String, String) in characterization/polars/probes-streaming.json - probe concat-nulls in characterization/polars/probes-streaming.json

polars-length: len_chars counts code points; len_bytes bytes

Engine: polars py-2.1.0; area: strings; status: no-finding; evidence: probe.

Verdict: character_length natively

len_chars counts code points (e with a combining accent is 2), as character_length does; len_bytes counts UTF-8 bytes. Both return UInt32.

The reference: character_length counts code points.

Options: char_set.

Constructs: str.len_chars, str.len_bytes.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe String.str.len_chars() in characterization/polars/probes-streaming.json - probe len-chars-bytes in characterization/polars/probes-streaming.json

polars-match-functions: starts_with, ends_with, literal replace_all

Engine: polars py-2.1.0; area: strings; status: no-finding; evidence: probe.

Verdict: native, case-sensitive

starts_with, ends_with, contains(literal=True) and replace_all(literal=True) are case-sensitive and agree with the reference on every probe row (the empty prefix matches everything). Case-insensitive matching is a regex flag, (?i).

The reference: starts_with, ends_with, replace with case_sensitivity: CASE_SENSITIVE.

Options: case_sensitivity.

Rows of the choice tables: str.match case_sensitivity (string).

Constructs: str.starts_with, str.ends_with, str.contains, str.replace_all.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe String.str.starts_with('a') in characterization/polars/probes-streaming.json - probe String.str.starts_with('') in characterization/polars/probes-streaming.json - probe String.str.ends_with('a') in characterization/polars/probes-streaming.json - probe String.str.ends_with('') in characterization/polars/probes-streaming.json - probe String.str.replace_all('a', 'xy', literal=True) in characterization/polars/probes-streaming.json

polars-regex: Regexes are Rust’s regex crate: no backreferences or lookaround, Unicode classes

Engine: polars py-2.1.0; area: strings; status: open; evidence: probe.

Verdict: regex crate syntax; extract is NULL without a match; $1 in replacements

Patterns are the Rust regex crate’s: no lookaround and no backreferences (both raise ComputeError when the query runs), \d and \w are Unicode (\d matches ٣), POSIX classes are ASCII. An invalid pattern raises.

str.extract(p, g) is NULL where the pattern does not match or the group did not take part; DuckDB’s regexp_extract gives '' there. replace_all replaces every match, including empty ones (b* in aaa gives -a-a-a-), and its replacement names groups as $1 or ${1}; \1 is literal. replace replaces the first match.

On ASCII, replace_all agrees with regexp_replace(…, 'g') on the probe’s patterns, and extract with regexp_extract wherever the pattern matches.

No upstream test pins the refusal of lookaround and backreferences, Unicode \d or empty-match replacement; they may change between versions.

The reference: regexp_extract, regexp_replace with a constant pattern (#50), RE2’s syntax, which has no backreferences or lookaround either; a pattern that does not match extracts ''. Lean evaluates the classes on ASCII only (Text.classesDiffer).

Options: regex_dialect, case_sensitivity.

Constructs: str.contains, str.extract, str.replace, str.replace_all.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #50.

Provenance:

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe String.str.extract('^(a+)(b)?', 2) in characterization/polars/probes-streaming.json - probe String.str.extract('(a|ab)(c|bcd)?', 0) in characterization/polars/probes-streaming.json - probe String.str.extract('(a*)*', 1) in characterization/polars/probes-streaming.json - probe String.str.extract('x(?:a)(y)', 1) in characterization/polars/probes-streaming.json - probe String.str.extract('.$', 0) in characterization/polars/probes-streaming.json - probe String.str.extract('(b)', 1) in characterization/polars/probes-streaming.json - probe String.str.replace_all('a+', '-') in characterization/polars/probes-streaming.json - probe String.str.replace_all('b*', '-') in characterization/polars/probes-streaming.json - probe String.str.replace_all('$', '!') in characterization/polars/probes-streaming.json - probe regex-syntax in characterization/polars/probes-streaming.json - probe regex-invalid in characterization/polars/probes-streaming.json - probe regex-replace-refs in characterization/polars/probes-streaming.json - probe regex-extract-nomatch in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/regex_cache.rs#L39-L51 patterns compile with regex::RegexBuilder (regex 1.13.1) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/chunked_array/strings/extract.rs#L72-L92 extract: NULL on no match or a group that did not take part - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/chunked_array/strings/namespace.rs#L652-L654 replace_all: regex_replacen with n=0 ($1 / ${1} groups, empty matches replaced) - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.str.extract.html regex crate syntax; null if the regex captures nothing - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.str.replace_all.html $ names a capture group; $$ for a literal $ - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/string/test_string.py#L1039-L1071 extract: no match → None (in-memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/string/test_string.py#L1328-L1384 replace_all: $1 substitution; invalid regex raises ComputeError (in-memory)

polars-reverse: str.reverse reverses grapheme clusters

Engine: polars py-2.1.0; area: strings; status: open; evidence: probe.

Verdict: graphemes, where the reference reverses code points

str.reverse reverses extended grapheme clusters (unicode_reverse): a letter keeps its combining accent (e\u0301x becomes xe\u0301) and a flag’s two regional indicators stay together. It agrees with the reference on ASCII and on precomposed text.

The reference: reverse reverses the code points (String.toList.reverse), so a combining accent moves to the other side of its letter.

Options: char_set.

Constructs: str.reverse.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, source-reading, by session brave-borg-41f7cc: Read at tag py-2.1.0 (commit 7f902bd3).
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe String.str.reverse() in characterization/polars/probes-streaming.json - probe reverse-graphemes in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.str.reverse.html example: ‘man303ana’ reverses to ‘anañam’, the combining tilde stays on its n - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/string/test_string.py#L2164-L2181 test_string_reverse: ‘man303ana’ → ‘anan303am’ (grapheme clusters, in-memory)

polars-string-functions-missing: No levenshtein, digests, repeat, ascii or LIKE on expressions

Engine: polars py-2.1.0; area: strings; status: open; evidence: probe.

Verdict: missing from pl.Expr.str

pl.Expr.str has no levenshtein, md5, sha256, repeat, ascii, like or ilike. LIKE exists in Polars’s SQL interface; through expressions it is a regex (str.contains) with the pattern translated. hash is Polars’s own hash, unrelated to the digests.

The docs are silent on SQL LIKE running as a regex: a docs gap.

The reference: Each is a Substrait function (or sverify’s black box) the reference evaluates.

Options: digests.

Constructs: levenshtein, md5, sha256, repeat, ascii, like, ilike.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observation reads the Python API (hasattr(pl.Expr.str, …)), which both engines share.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe string-functions-missing in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/string.py the str namespace defines no levenshtein, md5, sha256, repeat, ascii, like or ilike - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-sql/src/sql_expr.rs#L719-L766 SQL LIKE/ILIKE: literal fast path, else regex (?s)/(?is) with % → ., _ → ., via str.contains - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/sql/test_strings.py#L210-L249 test_string_like: LIKE/ILIKE// patterns, regex metacharacters literal

polars-string-order: Strings sort by UTF-8 bytes; False before True

Engine: polars py-2.1.0; area: strings; status: no-finding; evidence: probe.

Verdict: code point order, no collation

Strings compare and sort by their UTF-8 bytes, which is code point order: '' < B < Z < a < b < é. Booleans sort False before True. NULLs first by default (polars-null-order).

The reference: Strings in code point order (Lean’s String order); booleans false < true.

Constructs: sort, <.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe sort-strings-bools in characterization/polars/probes-streaming.json

polars-strpos: str.find: a 0-based byte offset, NULL when absent

Engine: polars py-2.1.0; area: strings; status: open; evidence: probe.

Verdict: byte offset from 0; NULL where strpos is 0

str.find(t) returns the byte offset of the first match, from 0, or NULL when there is none: find('héllo', 'l') is 3 (the character position is 2), and find('', 'a') is NULL. The empty pattern is found at 0.

No upstream test pins byte offsets on non-ASCII text; it may change between versions.

The reference: strpos(s, t) is the 1-based character position of the first match, 0 when there is none (#209). A writer adds 1 and maps NULL to 0, and on text that is not ASCII it needs the character position (the bytes before the match are not the characters).

Options: negative_start, string_positions.

Rows of the choice tables: str.match string_positions (string).

Constructs: str.find.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe String.str.find('a', literal=True) in characterization/polars/probes-streaming.json - probe String.str.find('b', literal=True) in characterization/polars/probes-streaming.json - probe String.str.find('', literal=True) in characterization/polars/probes-streaming.json - probe find-bytes in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.str.find.html ‘Return the bytes offset of the first substring matching a pattern’; None when not found - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/string/test_string.py#L335-L373 test_str_find: 0-based offsets and None on no match, ASCII only (in-memory); bytes vs characters not pinned - 2026-11-04, probe, by characterization pipeline reverse pass - LazyFrame({‘s’:[‘héllo’]}).select(str.find(‘l’, literal=True)) on in-memory and streaming: 3

polars-substring: str.slice: 0-based, negative offsets from the end

Engine: polars py-2.1.0; area: strings; status: no-finding; evidence: probe.

Verdict: substring with negative_start: WRAP_FROM_END, offset - 1

str.slice(offset, length) counts characters (code points) from 0; a negative offset counts from the end, and the part before the start is cut off (slice(-10, 7) of hello is he). An offset past the end gives ''. A length of NULL takes the rest; a negative length is refused (InvalidOperationError, cast to u64).

With the offset shifted by one for positive starts, it matches Substrait’s substring with WRAP_FROM_END on every probe row.

The reference: substring(s, start, length), 1-based, negative_start: WRAP_FROM_END (#209); offsets outside [-2³², 2³²) raise (#336).

Options: negative_start, string_positions.

Rows of the choice tables: str.sub negative_start (string); str.sub string_positions (string).

Constructs: str.slice, str.head, str.tail.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #209, #336.

Provenance:

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe String.str.slice(0, 2) in characterization/polars/probes-streaming.json - probe String.str.slice(1, 2) in characterization/polars/probes-streaming.json - probe String.str.slice(-2, 5) in characterization/polars/probes-streaming.json - probe String.str.slice(-10, 3) in characterization/polars/probes-streaming.json - probe String.str.slice(5, 1) in characterization/polars/probes-streaming.json - probe slice in characterization/polars/probes-streaming.json - probe slice-negative-length in characterization/polars/probes-streaming.json

polars-substring-range: str.slice takes any 64-bit offset

Engine: polars py-2.1.0; area: strings; status: open; evidence: probe.

Verdict: computes for offsets past 32 bits; a negative length raises

No offset makes str.slice raise: an offset of 2⁴⁰ gives '', and slice(-2⁴⁰, 2⁴¹) the whole string. A negative length raises (InvalidOperationError): the length is cast to UInt64 strictly.

The docs are silent on out-of-range offsets and a negative length raising: a docs gap.

No upstream test pins offsets as large as 2⁴⁰; it may change between versions.

The reference: substring_range: offsets outside [-2³², 2³²) raise (#336), as DuckDB does (duckdb-substring-range).

Options: substring_range.

Rows of the choice tables: str.sub substring_range (string).

Constructs: str.slice.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #336.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe substring-huge in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass: Corrected the description: it said no offset or length makes str.slice raise. A negative length raises (namespace.rs#L915-L922, pinned by test_string.py#L59-L61); the claim holds for offsets. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/chunked_array/strings/namespace.rs#L915-L922 offset cast to Int64; length strict-cast to UInt64, so a negative length raises - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/chunked_array/strings/substring.rs#L172-L206 offset >= len gives ’‘; a negative offset before the start is clamped with saturating arithmetic; never raises - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/string/test_string.py#L30-L61 test_str_slice_expr: negative offsets; str.slice(0, -1) raises InvalidOperationError (in-memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/string/test_string.py#L225-L238 test_str_slice_multibyte: offsets outside the string clip (in-memory) - 2026-11-04, probe, by characterization pipeline reverse pass - LazyFrame({’s’:[‘abc’]}).select(str.slice(0, -1)) raises InvalidOperationError (conversion from i32 to u64 failed) on in-memory and streaming

polars-trim: strip_chars() trims Unicode whitespace; with characters, those

Engine: polars py-2.1.0; area: strings; status: open; evidence: probe.

Verdict: btrim(s, ' ') natively; the default trims all whitespace

strip_chars() with no argument is Rust’s str::trim, every Unicode whitespace character (tab, newline, no-break space, em space). With characters it trims those: strip_chars(' ') is btrim(s, ' ') on every probe row.

The docs are silent on whitespace meaning Unicode’s White_Space: a docs gap.

No upstream test pins non-ASCII whitespace; it may change between versions.

The reference: btrim(s) trims spaces only; btrim(s, chars) the given characters.

Constructs: str.strip_chars, str.strip_chars_start, str.strip_chars_end.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, source-reading, by session brave-borg-41f7cc: Read at tag py-2.1.0 (commit 7f902bd3).
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe String.str.strip_chars(' ') in characterization/polars/probes-streaming.json - probe strip-whitespace in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.str.strip_chars.html None strips ‘all leading and trailing whitespace’; silent on Unicode whitespace (docs gap) - https://doc.rust-lang.org/std/primitive.str.html#method.trim str::trim: Unicode White_Space - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/string/test_string.py#L699-L708 test_str_strip_chars: tab and space stripped; character sets (ASCII only, in-memory)

temporal

polars-date-parts: Date parts: ISO weeks, weekday 1–7, floor before 1970

Engine: polars py-2.1.0; area: temporal; status: no-finding; evidence: probe.

Verdict: ISO week and year; Monday is 1; parts and epoch floor

week is the ISO week and iso_year its year (2024-12-30 is week 1 of 2025; 2021-01-03 is week 53 of 2020); weekday runs from Monday 1 to Sunday 7; millisecond and microsecond are the fraction of the second in those units.

Before 1970 parts floor: 1969-12-31 23:59:59.5 has second 59, microsecond 500000, epoch('s') -1, and truncate('1s') goes down to 23:59:59. truncate('1w') goes to Monday, '1mo', '1q', '1y' to the first day.

The reference: date_part and date_trunc (Time.lean): ISO weeks, Monday-based isodow, flooring before the epoch; their range is open in the reference.

Options: week_numbering.

Constructs: dt.year, dt.iso_year, dt.week, dt.weekday, dt.ordinal_day, dt.quarter, dt.epoch, dt.second, dt.truncate.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe date-parts in characterization/polars/probes-streaming.json - probe before-epoch in characterization/polars/probes-streaming.json - probe truncate-units in characterization/polars/probes-streaming.json - probe date-diff in characterization/polars/probes-streaming.json

polars-date-range: Dates past 9999 and before year 1; out-of-range days panic

Engine: polars py-2.1.0; area: temporal; status: open; evidence: probe.

Verdict: no date_range limit; Int32 days beyond chrono’s range panic

Date arithmetic is not limited to 0001-01-01 … 9999-12-31: one day before 0001-01-01 is 0000-12-31 and one after 9999-12-31 is +10000-01-01; timestamps (microseconds) reach year 12736 and beyond.

A date is an Int32 of days, wider than chrono’s NaiveDate (about ±262 000 years). Casting such a day count to Date succeeds, dt.year() of it is NULL, and turning it into text or a Python value panics (PanicException: out-of-range date, PL-2).

A Datetime with a time zone adds a Duration on its physical Int64 (UTC) and keeps the zone, the same addition as a naive timestamp, so it has no date_range limit either: Europe/Berlin 9999-12-31 12:00 plus a day is +10000-01-01 12:00 CET, and 0001-01-01 12:00 minus a day is 0000-12-31 12:00 LMT. dt.offset_by with months works in local time and crosses both bounds the same way.

The docs are silent on chrono’s range and the panic: a docs gap.

No upstream test pins the panic (PL-2) or 0000-12-31; it may change between versions.

The reference: date_range: date arithmetic raises outside 0001-01-01 … 9999-12-31 (#251).

Options: date_range, raise_set.

Rows of the choice tables: time.arith date_range (date, timestamp, timestamp_tz).

Constructs: +, -, dt.offset_by, cast.

Settings: {"engine": ["streaming", "in-memory"]}.

See: upstream ledger PL-2; issues #251.

Provenance:

polars-month-arith: offset_by('1mo') clamps to the month’s end

Engine: polars py-2.1.0; area: temporal; status: no-finding; evidence: probe.

Verdict: end-of-month clamping

Adding months keeps the day where it exists and otherwise takes the month’s last day: 2024-01-31 plus a month is 2024-02-29, 2024-02-29 plus a year is 2025-02-28, 2023-03-31 minus a month is 2023-02-28. pl.duration has no months; calendar steps go through offset_by.

The reference: Adding months keeps the day of month, or takes the month’s last day where it is shorter (Time.addMonths: 2024-01-31 + 1 month is 2024-02-29).

Options: date_range.

Constructs: dt.offset_by.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe date-month-arith in characterization/polars/probes-streaming.json

polars-strftime: strftime is chrono’s format

Engine: polars py-2.1.0; area: temporal; status: open; evidence: probe.

Verdict: chrono’s specifiers: %.f milliseconds here, %U Sunday weeks

dt.strftime uses chrono’s specifiers: %j the day of the year, %a the short weekday, %U the week with weeks starting Sunday, %V the ISO week; %.f printed .600 for 600 000 µs (the shortest of 3, 6 or 9 digits).

No upstream test pins this; it may change between versions.

The reference: strftime is opaque in the reference (time.part); DuckDB’s specifiers are its own.

Options: cast_formatting.

Constructs: dt.strftime.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe strftime in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/temporal/datetime.rs#L47-L81 strftime: chrono StrftimeItems + format_with_items (chrono 0.4.45) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/temporal/date.rs#L35-L49 Date: the same, via date32_to_date - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.strftime.html refers to chrono’s strftime table - https://docs.rs/chrono/0.4.45/chrono/format/strftime/index.html %j, %a, %U, %V; %.f chooses 3, 6 or 9 digits

polars-temporal-errors: TIME + duration unsupported; DST gaps and invalid dates raise

Engine: polars py-2.1.0; area: temporal; status: open; evidence: probe.

Verdict: raises where the reference refuses or computes

time + duration is refused (SchemaError, no supertype). replace_time_zone raises on a local time in a DST gap (non_existent='null' gives NULL). str.to_date raises on an invalid date such as 2024-02-30 and, with strict=False, gives NULL; it accepts 2024-2-3 for %Y-%m-%d.

Months only exist in dt.offset_by (pl.duration has none), and offset_by accepts Date and Datetime: on a Time column it raises InvalidOperationError (expected Date or Datetime, got time). Time ± Duration finds no supertype, so TIME ± INTERVAL with or without months is refused when the plan is resolved.

The docs are silent on time + duration being refused and 2024-2-3 parsing: a docs gap.

No upstream test pins time + duration, offset_by on Time, 2024-02-30 or 2024-2-3; they may change between versions.

The reference: TIME ± INTERVAL is refused (time_interval_months); time zones are outside the model.

Options: time_interval_months, time_zones, cast_parsing.

Rows of the choice tables: time.arith time_interval_months (time).

Constructs: +, dt.replace_time_zone, str.to_date.

Settings: {"engine": ["streaming", "in-memory"]}.

Provenance:

polars-timestamp-unit: Timestamps are microseconds by default; nanoseconds reach 2262

Engine: polars py-2.1.0; area: temporal; status: no-finding; evidence: probe.

Verdict: Datetime('us'); ns casts fail past 2262-04-11

A Python datetime becomes Datetime('us'). Datetime('ns') holds 1677-09-21 … 2262-04-11; a cast to it outside that range raises (InvalidOperationError) unless strict=False.

The reference: precision_timestamp at 0, 3, 6, 9 (#323); the reference’s timestamps are microseconds.

Options: timestamp_precision.

Constructs: Datetime, cast.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #323.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The default unit is the frame’s, set when it is built, before any engine runs; the cast to Datetime('ns') and its overflow ran on both.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe timestamp-unit in characterization/polars/probes-streaming.json

casts

polars-cast-double-to-int: Double to integer: truncates; out of range raises or is NULL

Engine: polars py-2.1.0; area: casts; status: no-finding; evidence: probe.

Verdict: float_to_int: TRUNCATE; strict raises, strict=False NULL; wrap_numerical saturates

A cast from Float64 to an integer truncates toward zero (2.7 is 2, -2.7 is -2). A value out of the target’s range, ±inf or NaN raises with strict=True (the default, InvalidOperationError) and is NULL with strict=False.

wrap_numerical=True saturates for doubles (1e19 is 127 in Int8, -1e19 is -128, NaN is 0, Rust’s as), where it wraps for integers (polars-cast-int).

The reference: float_to_int: TRUNCATE (#209); cast_failure as the cast states, THROW_EXCEPTION for UNSPECIFIED.

Options: float_to_int, cast_failure.

Rows of the choice tables: float_to_int (any); cast_failure (any).

Constructs: cast.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-6; issues #209.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe cast-double-to-int in characterization/polars/probes-streaming.json - probe cast-double-to-int-strict in characterization/polars/probes-streaming.json - probe cast-double-to-int-wrap in characterization/polars/probes-streaming.json - probe cast-decimal-to-int in characterization/polars/probes-streaming.json

polars-cast-int: Integer to a narrower integer or decimal: raises, NULL, or wraps

Engine: polars py-2.1.0; area: casts; status: no-finding; evidence: probe.

Verdict: strict raises, strict=False NULL, wrap_numerical wraps

A cast to a narrower integer that does not fit raises with strict=True, is NULL with strict=False, and wraps with wrap_numerical=True (300 is 44 in Int8). A cast to a decimal too narrow for it raises or is NULL the same way. A decimal to an integer truncates.

The reference: cast_failure: THROW_EXCEPTION or RETURN_NULL, as the cast states.

Options: cast_failure.

Rows of the choice tables: cast_failure (any).

Constructs: cast.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe cast-int-narrow in characterization/polars/probes-streaming.json - probe cast-int-narrow-strict in characterization/polars/probes-streaming.json - probe cast-int-to-decimal-overflow in characterization/polars/probes-streaming.json - probe cast-decimal-to-int in characterization/polars/probes-streaming.json

polars-cast-string: Casts from and to strings

Engine: polars py-2.1.0; area: casts; status: open; evidence: probe.

Verdict: strict parsing; shortest round-trip for doubles; no string → bool or date

Text to an integer accepts an optional sign and digits only (2, 3.0, 1e2, 0x10 fail); text to a double accepts nan, inf, -Infinity, .5, 5. and rounds 1e400 to inf. Failures raise with strict=True and are NULL with strict=False.

A double becomes its shortest round-trip text, with an exponent at 1e16 (1e+16) and at 1e-7 (1e-7), without one at 1e15 and 1e-5, -0.0, inf and NaN. Booleans become true and false; integers and doubles become booleans by != 0 (NaN is true).

Text to Boolean or to Date is refused (InvalidOperationError); str.to_date parses dates.

The docs are silent on the text forms that parse, how a double prints and text to Boolean being refused: a docs gap.

No upstream test pins the rejected forms, nan/inf parsing or the double’s text format; they may change between versions.

The reference: Casts to and from strings follow the reference’s cast rules; cast_failure as stated.

Options: cast_parsing, cast_formatting, cast_failure.

Rows of the choice tables: cast_failure (any).

Constructs: cast.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe cast-string-to-int in characterization/polars/probes-streaming.json - probe cast-string-to-double in characterization/polars/probes-streaming.json - probe cast-double-to-string in characterization/polars/probes-streaming.json - probe cast-bool in characterization/polars/probes-streaming.json - probe cast-string-to-bool in characterization/polars/probes-streaming.json - probe cast-string-to-date in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/cast/binary_to.rs#L20-L68 integers: atoi_simd::parse::<_, true, true> (0.18.1), sign and digits; doubles: fast_float2::parse (0.2.4) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/cast/binview_to.rs#L66-L84 unparsable text becomes NULL; strict mode raises on the new NULLs - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/cast/primitive_to.rs#L64-L86 double to text: zmij 1.0.23 shortest round-trip - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/cast/boolean_to.rs#L39-L43 Boolean to text: true/false (the doc comments at L38 and mod.rs L379 say 1/0) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/cast/mod.rs#L555-L601 text to Date bails (removed in 2.0); number to Boolean is != 0; no text arm for Boolean, so it bails - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/releases/upgrade/2/ ‘Remove casts from string to temporal types’: cast(pl.Date) of text raises, use str.to_date (docs/source/releases/upgrade/2.md L1357-L1390) - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.cast.html states strict/non-strict only; silent on accepted text formats, float text format and text to Boolean - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_cast.py#L39-L88 test_leading_plus_zero_int/uint/float: sign and leading zeros accepted, ‘-.5’, ‘0.’, ‘+.5’ parse; eager Series.cast (in-memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_temporal.py#L2676-L2698 test_string_to_temporal_cast_unsupported: text to Date/Datetime/Time raises InvalidOperationError (in-memory)

polars-cast-to-double: Integer to double and double to float round to nearest

Engine: polars py-2.1.0; area: casts; status: no-finding; evidence: probe.

Verdict: round to nearest even; overflow to ±inf

Int64 to Float64 rounds to the nearest double (2⁵³ + 1 is 2⁵³). Float64 to Float32 rounds to the nearest float, overflows to inf and underflows to 0.

The reference: The conversions between integers and doubles are opaque constants with proved properties (SubstraitLean/Conversions.lean), the same in every profile; fp32 arithmetic is refused until #417.

Options: fp32.

Rows of the choice tables: fp32 (fp32).

Constructs: cast.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #417.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe cast-int-to-double in characterization/polars/probes-streaming.json - probe cast-double-to-float in characterization/polars/probes-streaming.json

polars-ulp-casts: Integer and decimal to Float64 and Float32: correctly rounded

Engine: polars py-2.1.0; area: casts; status: no-finding; evidence: probe.

Verdict: ≤ 0.5 ulp: Int64, Int128 and Decimal to Float64 and Float32

Int64 and Int128 to Float64 and Float32 are Rust as casts (through num_traits), which round to nearest, ties to even: ≤ 0.5 ulp, also above 2^53 and for 128-bit values.

Decimal to Float64 divides once when the unscaled value is at most 2^53 and the scale at most 22, and otherwise formats <unscaled>e-<scale> and parses it with Rust’s correctly rounded parser; Decimal to Float32 rounds the Float64 again unless it sits exactly on a binary32 midpoint, then parses. Both correctly rounded.

Observed, 3 000 values each: 0 off the correctly rounded value for Int64, Int128 (to ±2^126) and Decimal(18, 6), (18, 17), (38, 10), (38, 30), (38, 2), to Float64 and Float32. Platform-independent. DuckDB’s HUGEINT cast misses by up to 0.625 ulp where Polars’s Int128 cast is exact on the same values (duckdb-ulp-casts).

The reference: Casts of integers and decimals to doubles round once to nearest, ties to even (Lean’s Float.ofInt, Float.ofScientific).

Options: fp32.

Constructs: cast(pl.Float64), cast(pl.Float32).

Settings: {"engine": "streaming", "platform": "macOS arm64"}.

Provenance:

Elementwise kernels are shared by the streaming and in-memory engines (polars-elem-rounding, polars-double-scalar-divisor); the probe collects with engine=‘streaming’ on 3 000-row frames, so morsels hold more than one row and the literal paths run. - pl.LazyFrame({‘x’: ints}, schema={‘x’: pl.Int64}).select(pl.col(‘x’).cast(pl.Float64), …cast(pl.Float32)).collect(engine=‘streaming’): max 0.5000 / 0.49999 ulp, 0 of 3 000 off. The same for Int128 (±2^126): 0.5000 / 0.49998, 0 off. - pl.Series(decimals, dtype=pl.Decimal(p, s)).cast(…) for (18,6), (18,17), (38,10), (38,30), (38,2): max ≤ 0.49997 (Float64), ≤ 0.5000 (Float32) ulp, 0 off for each.

nulls

polars-kleene: Boolean logic is Kleene’s; all and any skip NULLs by default

Engine: polars py-2.1.0; area: nulls; status: no-finding; evidence: probe.

Verdict: three-valued &, |, ~; all(ignore_nulls=False) is Kleene’s

&, | and ~ follow three-valued logic (False & NULL is False, True | NULL is True, True & NULL is NULL). all() and any() skip NULLs (all(True, NULL) is True); with ignore_nulls=False they are Kleene’s (all(True, NULL) is NULL).

The reference: and, or, not are Kleene’s; bool_and and bool_or skip NULLs as aggregates do.

Options: three_valued_logic.

Constructs: &, |, ~, all, any.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe kleene in characterization/polars/probes-streaming.json - probe any-all-nulls in characterization/polars/probes-streaming.json

polars-null-keys: NULL keys: one group; joins skip them by default

Engine: polars py-2.1.0; area: nulls; status: no-finding; evidence: probe.

Verdict: NULLs group together; nulls_equal=True joins them

group_by and unique put the NULLs in one group. A join does not match NULL keys unless nulls_equal=True.

The reference: Grouping and DISTINCT put NULLs together; an equi-join’s = does not match NULL (and IS NOT DISTINCT FROM keys do).

Options: null_join_keys.

Rows of the choice tables: null_join_keys (any).

Constructs: group_by, unique, join.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64.

join-nan-zero-null’s recorded differ 1 is a tie order: sort('y') ties the rows 0.0 and -0.0 (both y = 2), and the engines list them in different orders. The join results are the same set of rows on both engines; it is no engine difference. - probe group-nan-zero in characterization/polars/probes-streaming.json - probe unique-nan-zero in characterization/polars/probes-streaming.json - probe join-nan-zero-null in characterization/polars/probes-streaming.json

nested

polars-explode: explode keeps a row for a NULL list and drops an empty list

Engine: polars py-2.1.0; area: nested; status: open; evidence: probe.

Verdict: a NULL list gives a NULL row, [] no row; keep_nulls=False drops the NULL row too

explode gives one row per element, one NULL row for a NULL list and no row for an empty list (the defaults empty_as_null=False, keep_nulls=True). With keep_nulls=False a NULL list gives no row either, which is SQL’s UNNEST; empty_as_null=True gives an empty list a NULL row.

The reference: UNNEST: one row per element, none for an empty list or NULL.

Options: unnest_null_empty.

Constructs: explode.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

explode calls the eager DataFrame.explode, in memory; s-explode explodes 6000 rows lazily on each engine, with and without empty_as_null=False, keep_nulls=False: the same rows in the same order (6000 and 4500).

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe explode in characterization/polars/probes-streaming.json - probe s-explode in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/explode_and_offsets.rs#L59-L175 empty list: NULL row only if empty_as_null; NULL list: NULL row only if keep_nulls; else dropped - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/lazyframe/frame.py#L7840-L7846 defaults empty_as_null=False, keep_nulls=True - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_ir.rs#L645-L666 streaming: Explode is streamable and runs as a Map node calling the in-memory evaluate_function_ir per morsel - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/lazyframe/api/polars.LazyFrame.explode.html empty_as_null and keep_nulls documented; defaults False and True - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_explode.py#L620-L640 test_explode_params: [[1,2,3], None, [4,5,6], []] under all four flag pairs (in-memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_explode.py#L526-L548 test_explode_parametric: Hypothesis over the flags - 2026-11-04, probe, by characterization pipeline reverse pass: Corrected the description: it said the default gives a NULL row for an empty list. At 2.1.0 the default empty_as_null=False gives no row for an empty list (the probe explode shows it too) and one NULL row for a NULL list. - LazyFrame({‘i’:[1,2,3,4],‘l’:[[1,2],[],None,[None]]}).explode(‘l’) on in-memory and streaming (morsel 64): [(1,1),(1,2),(3,None),(4,None)]; the empty list gives no row by default

polars-list-index: list.get: 0-based, negative from the end, out of range raises

Engine: polars py-2.1.0; area: nested; status: open; evidence: probe.

Verdict: 0-based; raises out of range unless null_on_oob=True

list.get(i) counts from 0, a negative index from the end (-1 is the last element). An index out of range raises (ComputeError: get index is out of bounds) by default and is NULL with null_on_oob=True; a NULL list gives NULL.

The list.get docs say an out-of-range index returns None, and the default raises: PL-12.

No upstream test pins the default null_on_oob=False; it may change between versions.

The reference: list_extract(l, i) (reference-semantics.md, Nested) counts from 1, as DuckDB’s does.

Options: list_indexing, raise_set.

Constructs: list.get, list.slice.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-12.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe list-get in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/chunked_array/list/namespace.rs#L320-L338 lst_get: bails ‘get index is out of bounds’ unless null_on_oob; negative index from the end - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/chunked_array/list/get.rs#L9-L75 per-row index: start+idx or end+idx, bail or NULL; NULL list or index gives NULL - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.list.get.html docs bug (draft PL-D5): says an out-of-bounds index returns None, but the default null_on_oob=False raises - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/list/test_list.py#L24-L92 test_list_arr_get: get(0)→[1,4,6], get(-1)→[3,5,9], get(3)/get(-3) with null_on_oob=False raise (in-memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/namespaces/list/test_list.py#L95-L140 test_list_arr_get_null_on_oob: out of range → None

polars-list-order: Lists sort with NULL elements lowest; no < or min on lists

Engine: polars py-2.1.0; area: nested; status: open; evidence: probe.

Verdict: list_null_order: LEAST in sorts; list comparisons refused

sort orders lists lexicographically, a prefix first, and a NULL element below every value: [] < [NULL] < [1] < [1, NULL] < [1, 2] ascending, reversed descending, NULL lists first by default. < between lists and min, max of lists raise (InvalidOperationError).

The docs are silent on how lists sort: a docs gap.

No upstream test pins the prefix order; it may change between versions.

The reference: list_null_order: NULL where a NULL element decides <, and GREATEST in ORDER BY (#207). DuckDB puts the NULL element above every value (duckdb-R11).

Options: list_null_order.

Rows of the choice tables: list.order list_null_order (list); agg.order list_null_order (list).

Constructs: sort, <, min.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #207.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (nested): The docs are silent on the order of lists and on min/max of lists; the rows rest on the 2026-11-04 probes (list-sort, list-compare, list-min).

The docs at stable are those of Polars 2.1.0: their [source] links point at tag py-2.1.0. - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.list.sort.html list.sort(descending=False, nulls_last=False) sorts the elements inside each list; it says nothing about how lists compare - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.sort.html sort(descending, nulls_last); no word on list keys - 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe list-sort in characterization/polars/probes-streaming.json - probe list-compare in characterization/polars/probes-streaming.json - probe list-min in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/sort/mod.rs#L692-L725 list sort: arg_sort over row-encoded keys - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-row/src/encode.rs#L788-L866 list encoding: continuation token per element, termination token after; prefix sorts first - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-row/src/row.rs#L98-L129 into_nested: element NULLS_LAST = DESCENDING, so a NULL element is lowest ascending (#22557) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_expr.rs#L3236-L3264 streaming sort row-encodes a list key the same way - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/series/comparison.rs#L248 < between lists bails InvalidOperation - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_row_encoding.py#L407-L431 test_nested_sorting_22557: [[‘A’], None, [None], [‘B’]] sorts to [None, [None], [‘A’], [‘B’]] (in-memory)

polars-nested-equality: Lists and structs: NULL elements and fields compare equal

Engine: polars py-2.1.0; area: nested; status: no-finding; evidence: probe.

Verdict: structural equality, NULL inside equals NULL

== on lists and on structs treats a NULL element or field as equal to NULL: [1, NULL] == [1, NULL] and {a: 1, b: NULL} == {a: 1, b: NULL} are true; a NULL list or struct gives NULL (eq_missing makes it equal). Grouping puts equal structs together.

Structs sort field by field, a NULL field first.

The reference: Equality is structural, and list equality treats NULLs as equal, as grouping does (reference-semantics.md, “Equality for grouping” and Nested).

Options: struct_equality.

Constructs: ==, group_by, sort.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe list-equality in characterization/polars/probes-streaming.json - probe struct-equality in characterization/polars/probes-streaming.json - probe struct-group-sort in characterization/polars/probes-streaming.json

aggregates

polars-arg-min-max: arg_min, arg_max: the first index of a tie

Engine: polars py-2.1.0; area: aggregates; status: no-finding; evidence: probe.

Verdict: the first of tied rows

arg_max over 1, 3, 3, 1 is 1 and arg_min is 0: the first of the tied rows. NaN is skipped (polars-min-max-nan).

The reference: tie_pick: any row of the tie.

Options: tie_pick.

Rows of the choice tables: agg.pick tie_pick (any).

Constructs: arg_min, arg_max.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe arg-min-max-ties in characterization/polars/probes-streaming.json - probe min-max-nan in characterization/polars/probes-streaming.json

polars-avg-int: mean and median of integers: Float64

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: a double: the Float64 sum over the count

mean of integers is a Float64 (the values cast to doubles and summed, polars-sum-float), so it rounds above 2⁵³ (mean(2⁶², 2⁶², 1) is 3.0744573456182584e18). median of integers is a Float64 too. Both are NULL for no values.

Every integer width takes one generic path (with_match_physical_numeric_polars_type, T: PolarsIntegerType): each value becomes an f64, exactly for 8 to 32 bits, and the sum is an f64: pairwise ungrouped, Kahan per group in memory, value by value per group when streaming. A sum below 2⁵³ is exact in any order, so for Int8 to Int32 the mean is the exact quotient rounded once on either engine. Above 2⁵³ (Int64) the engines can round the sum differently.

The docs are silent on the rounding above 2⁵³: a docs gap.

No upstream test pins the rounding above 2⁵³; it may change between versions.

The reference: avg of i64 returns i64 in Substrait; the reference follows the declared output type (avg_int_result, #209).

Options: avg_int_result, empty_group_aggregates.

Rows of the choice tables: agg.isum avg_int_result (i64); agg.isum avg_int_result (i8, i16, i32; evidence engine_test); agg.isum empty_group_aggregates (i64); agg.isum empty_group_aggregates (i8, i16, i32; evidence source).

Constructs: mean, median.

Settings: {"engine": ["streaming", "in-memory"]}.

See: issues #209.

Provenance:

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_group_by.py#L104-L199 test_group_by_mean_by_dtype: grouped mean of 1, 2, 3 | 4 is Float64 2.0, 4.0 for Int8, Int16, Int32 (and the unsigned widths); Float32 stays Float32. Lazy collect(), schema and values checked - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_group_by.py#L907-L918 test_overflow_mean_partitioned_group_by_5194 (slow): grouped mean of 100 000 Int32 values of 10⁷ is 10000000.0, past the Int32 range of the sum (eager, in memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/sql/test_window_functions.py#L1369-L1386 test_window_avg_of_large_integers: SQL AVG of Int32 2·10⁹ over three window frames matches DuckDB, on engines=['in-memory', 'streaming'] - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.mean.html the example shows mean of an integer column as f64; silent on rounding above 2^53 - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/expr/expr.py#L3801-L3820

polars-corr: corr: shared state; NaN for one pair or zero variance

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: dp_xy / sqrt(dp_xx · dp_yy); NaN where the reference is NULL; no clamp

PearsonState returns dp_xy / sqrt(dp_xx * dp_yy) if dp_xx * dp_yy > 0, else NaN: one pair or a zero variance is NaN. Not clamped to [-1, 1]. dp_xx * dp_yy can overflow or underflow before the square root.

No pairs: null in the streaming reducer; the in-memory path (polars-ops cov.rs) returns NaN.

Integers become doubles (as f64) and decimals are cast to Float64 in memory, so the degenerate cases are those of doubles: one pair or a constant side is NaN for Int64 and Decimal too. Above 2⁵³ distinct integers can round to one double: corr of 2⁶², 2⁶² + 1, 2⁶² + 2 with 1, 2, 3 is NaN.

The streaming reduction accepts primitive numerics only: corr of a Decimal column raises InvalidOperationError on the default lazy engine and succeeds in memory. No pairs is null when streaming and NaN in memory (above).

The docs are silent on one pair, a constant side and no pairs, the types corr takes and the engines: a docs gap.

No upstream test pins no pairs or a Decimal input; they may change between versions.

The reference: AggFn.corr (corrOf): NULL for fewer than two pairs, or for a side whose values are all equal; else the co-deviation sum by the square root of the product of the two squared-deviation sums. It is not clamped to [-1, 1]. Engines may differ in the last bits (#54) until #85 bounds the error.

Where the reference is NULL (one pair, a constant side), Polars gives NaN.

Options: stat_algo, stat_degenerate.

Rows of the choice tables: agg.stat stat_algo (i64, fp64, decimal); agg.stat stat_degenerate (i64, fp64, decimal).

Constructs: corr, pearson_corr.

Settings: {"engine": ["streaming", "in-memory"]}.

See: upstream ledger PL-5, ST-3; issues #54, #85.

Provenance:

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L320-L323 test_pearson_corr_zero_column_29315: corr of a constant 0.0 column with 0..n-1 is NaN for n = 117 and 245 (eager, in memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L261-L267 test_corr_cov_lit_produces_zero_nan_26633: corr(lit(1), a) is NaN and cov 0.0 (eager) - 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): Confirms PL-5 for corr: the same query (with cov and corr of Decimal columns) raises on the streaming engine and gives 0.9707253433941508 in memory.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-cov-decimal in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.corr.html silent on degenerate inputs (one pair, a constant side, no pairs), on accepted dtypes and on the engines; no engine-support directive - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L40-L42 test_corr_nan: a = [1.0, 1.0], b = [1.0, 2.0] gives NaN (eager, in memory). No upstream test pins no-pairs (null streaming, NaN in memory) or Decimal input - 2026-11-04, engine-test, by characterization pipeline reverse pass: From the reverse pass’s check of PL-5, which this fact records. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L269-L293 no test of cov/corr on Decimal; skew’s streaming-vs-in-memory hypothesis test includes pl.Decimal in its dtypes, so the suite expects decimal parity for moment reductions - 2026-11-04, docs, by characterization pipeline reverse pass: From the reverse pass’s check of PL-5, which this fact records. - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.cov.html no dtypes and no engine-support directive; pl.corr likewise

polars-count-empty: count skips NULLs; aggregates of nothing

Engine: polars py-2.1.0; area: aggregates; status: no-finding; evidence: probe.

Verdict: count skips NULLs, len counts rows; min, first of NULLs are NULL

count counts the non-NULL values and len (pl.len()) the rows. Over NULLs only, min, max and first are NULL, count is 0 and mode is [NULL]. sum gives 0 there (polars-sum-int, polars-sum-float).

The reference: count(x) skips NULLs and count(*) counts rows; other aggregates of no values are NULL.

Options: empty_group_aggregates.

Constructs: count, len, min, first, mode.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe count-nulls in characterization/polars/probes-streaming.json - probe aggregates-empty in characterization/polars/probes-streaming.json

polars-cov: cov: two-pass blocks with Chan merge; Welford when grouped on streaming

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: two-pass per block + Chan (in memory; streaming ungrouped); Welford (streaming grouped); no clamp; Decimal raises on streaming (PL-5)

CovState follows var (polars-var-std): two-pass per block of 128 merged by Chan’s combine in memory, Welford’s update in the streaming reducer. A weight at most ddof is null. CovState does not clamp.

Integers become doubles (as f64), so values above 2⁵³ round; decimals are cast to Float64 in memory. The streaming engine checks for a primitive numeric type before it builds the reduction, and a decimal is not one: cov of a Decimal column raises InvalidOperationError on the default lazy engine and succeeds in memory.

Streaming, ungrouped, merges moment::cov per chunk, the block algorithm of the in-memory path; the per-value Welford update is the grouped streaming path. In memory a column of length 1 is taken as a broadcast constant, so the covariance of one pair is 0.0 with ddof=0 and null with ddof=1 without reaching CovState.

The docs are silent on a weight at most ddof giving NULL, the algorithm, the types cov takes and the engines: a docs gap.

No upstream test pins a Decimal input; it may change between versions.

The reference: AggFn.cov (covOf) is two-pass: the two means, then the products of the deviations added in list order from +0.0, by n or n - 1. No pairs is NULL, and so is a sample of one pair. Engines may differ in the last bits (#54) until #85 bounds the error.

Options: stat_algo.

Rows of the choice tables: agg.stat stat_algo (i64, fp64, decimal).

Constructs: cov.

Settings: {"engine": ["streaming", "in-memory"]}.

See: upstream ledger PL-5; issues #54, #85.

Provenance:

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L108-L123 test_cov_one_row, engine in-memory and streaming: grouped cov(ddof=1) of one pair is NULL and cov(ddof=0) 0.0; of (2, 3), (3, 2) -0.5 and -0.25; ungrouped head(1) cov(ddof=1) is NULL - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L74-L96 test_cov_corr_mixed_float_schema, both engines: cov and corr of Float32 with Float64 are Float64, grouped and not - 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): Confirms PL-5 on the streaming engine: cov of two Decimal(10, 2) columns raises InvalidOperationError (cov operation not supported for dtype decimal[10,2]), where memory gives 2.625 (and corr 0.9707253433941508).

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-cov-decimal in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.cov.html states ddof as the divisor N - ddof; silent on a weight at most ddof (null), the algorithm, dtypes and engines - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L98-L104 test_cov: cov(A, B) of fruits_cars is -2.5, Float64 (lazy collect, default engine). No upstream test of Decimal input - 2026-11-04, engine-test, by characterization pipeline reverse pass: From the reverse pass’s check of PL-5, which this fact records. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L269-L293 no test of cov/corr on Decimal; skew’s streaming-vs-in-memory hypothesis test includes pl.Decimal in its dtypes, so the suite expects decimal parity for moment reductions

polars-first-last: first, last: the first and last row in input order, NULL included

Engine: polars py-2.1.0; area: aggregates; status: no-finding; evidence: probe.

Verdict: input order, deterministic; NULLs taken unless ignore_nulls=True

first and last take the first and last row of the group in input order, also when it is NULL; ignore_nulls=True skips NULLs. Over 200 000 rows in 10 groups, in memory and streaming, first was the least row index, last the greatest, and the list a group collects (pl.col(x) in agg, implode) kept input order.

str.join concatenates in that order and skips NULLs.

Streaming, first and last are reductions keyed by the morsel’s sequence number: first keeps the value of the lowest sequence number (the first row of a morsel, or the earliest row of a group in it), last the highest (a later row of the same morsel replaces it), and pipelines merge by the same rule. So they follow input order as far as morsel numbers do. implode with maintain_order (the list agg(pl.col(x)) builds) has no streaming lowering: a group_by with it falls back to the in-memory engine, and ungrouped it runs in the in-memory fallback. implode(maintain_order=False) streams and keeps no order.

The reference: tie_pick: any row of the group (each measure may take its own); tie_order: any order for string_agg and array_agg. Polars’s choice is allowed, and deterministic where DuckDB’s depends on parallelism (duckdb-mode-ties).

Options: tie_pick, tie_order.

Rows of the choice tables: agg.pick tie_pick (any); agg.seq tie_order (any).

Constructs: first, last, implode, str.join, group_by.

Settings: {"engine": ["streaming", "in-memory"]}.

Provenance:

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_group_by.py#L786-L808 test_streaming_group_by_first_last_stored, engine='streaming' with morsels of 32, hot tables of 4 and stored partitions: over k = i % 50, v = i if i >= 50 else NULL, first is NULL, last 950 + k, first(ignore_nulls=True) 50 + k, last(ignore_nulls=True) 950 + k - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_group_by.py#L43-L125 test_streaming_group_by_types, streaming: first and last of strings, booleans and dates follow input order within the group - 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): Streaming agrees with memory: over 100 000 rows in 7 groups, first and last are the group’s least and greatest row index, and the list agg(pl.col('i')) builds is in input order.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-first-last-grouped in characterization/polars/probes-streaming.json - probe s-implode-order in characterization/polars/probes-streaming.json

polars-mean-float: mean of floats: the sum by the count

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: sum / n

mean is the sum (polars-sum-float, pairwise or Kahan by path) divided by the count.

The mean sums in f64 whatever the input: _sum_as_f64 (pairwise, f32 lanes widened to f64) ungrouped, KahanSum<f64> per index group in memory. A Float32 mean is that f64 quotient rounded to f32.

Streaming: NumMeanReducer keeps an f64 sum and a count; a morsel adds its _sum_as_f64, a grouped row adds itself (reduce_one), and pipelines merge by adding sums and counts. So a grouped mean is Kahan in memory and a plain left-to-right f64 sum when streaming, as for sum (polars-sum-float).

Order class (in memory): (a) ungrouped, _sum_as_f64: the pairwise kernel per chunk (Float32 lanes widened exactly to f64), chunks added in turn, then one division by the count; index-grouped, Kahan in f64, then the division; a Float32 mean rounds the f64 quotient to f32. (b) Pairwise: γ_{h+1} Σ|x_i| / n with h as in polars-sum-float (Higham §4.2); Kahan: (2u + O(nu²)) Σ|x_i| / n plus the quotient’s u (§4.3); u = 2⁻⁵³; the Float32 result adds one rounding at 2⁻²⁴. (c) The source below; s-group-mean-cancel (0.5 in memory). (d) Ungrouped Float64: ANY. Grouped (Kahan): no Lean order. Float32: the sum refines ANY on the widened values, but the final rounding to f32 is outside AggFn.avg, which returns the double quotient.

The reference: AggFn.avg is floatSum / n, NULL without values.

Options: sum_order, rounding.

Rows of the choice tables: agg.fsum sum_order (fp32, fp64); agg.fsum rounding (fp32, fp64; evidence source).

Constructs: mean, avg.

Settings: {"engine": "in-memory"}.

See: issues #54, #85; superseded by polars-mean-float-streaming.

Provenance:

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-group-mean-cancel in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline round 5 (summation order classes): Read at tag py-2.1.0 (commit 7f902bd3). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L135-L137 _sum_as_f64 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L240-L246 mean: _sum_as_f64 / count - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/group_by/aggregations/mod.rs#L952-L990 agg_mean: KahanSum of f64 per index group - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L254-L265 the first n mod 128 values added in turn, then mainsum + restsum - https://epubs.siam.org/doi/10.1137/1.9780898718027.ch4 Higham, Accuracy and Stability of Numerical Algorithms (2nd ed., SIAM 2002), ch. 4 Summation: §4.2 recursive, pairwise and any-order bounds; §4.3 compensated summation

polars-mean-float-streaming: mean of floats on the streaming engine: varies between runs when grouped

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: f64 sum / n; grouped, the order of the additions varies between runs

On the streaming engine mean is NumMeanReducer: an f64 sum and a count. A morsel adds its _sum_as_f64, a grouped row adds itself, and pipelines merge by adding sums and counts. A Float32 mean is the f64 quotient rounded to f32.

The grouping of the additions follows the scheduling of morsels, so a grouped mean can change between runs: one Float64 group of 1e16, 1, 1, -1e16 gave 0.5, 0.25 and 0.0 on different runs (in memory, Kahan: 0.5). A frame of 2 to 4 rows streams as one-row morsels on a 10-thread machine, so which pipelines add which rows, and in what order their partial sums merge, changes between runs. Its sum takes the same additions as polars-sum-float-streaming’s (2.0, 1.0 or 0.0 over 200 runs; always 0.0 with POLARS_MAX_THREADS=1 or 2), so the mean is that sum divided by 4; the mean’s own counts were not recorded. The recorded s-group-mean-cancel (differ 0, streaming 0.5) is one outcome among several. The in-memory engine’s algorithms are in polars-mean-float.

Order class (streaming): (a) NumMeanReducer keeps an f64 sum: ungrouped, each morsel adds its _sum_as_f64 (pairwise per chunk) and pipelines merge in turn; grouped, each row is added in turn per pipeline and the pipelines’ sums merge; one division by the count, rounded to f32 for Float32. (b) Any tree: γ_{h+1} Σ|x_i| / n, h as in polars-sum-float-streaming, at most n − 1 (Higham §4.2), u = 2⁻⁵³; Float32 adds one rounding at 2⁻²⁴. (c) The source below; s-group-mean-cancel (0.25 and 0.0 on different runs). (d) Float64: ANY. Float32: the f64 sum of the widened values refines ANY, and the final rounding to f32 is outside AggFn.avg.

The docs are silent on the summation order, its variation between runs and the f64 sum of a Float32 mean: a docs gap.

No upstream test pins the streaming summation order; it may change between versions.

The reference: AggFn.avg is floatSum / n, NULL without values.

Options: sum_order, rounding, empty_group_aggregates.

Rows of the choice tables: agg.fsum sum_order (fp64); agg.fsum sum_order (fp32; evidence source); agg.fsum rounding (fp64); agg.fsum rounding (fp32; evidence source); agg.fsum empty_group_aggregates (fp32, fp64; evidence source).

Constructs: mean, avg.

Settings: {"engine": "streaming"}.

See: issues #54, #85; supersedes polars-mean-float.

Provenance:

  • 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): s-group-mean-cancel: 0.5 in memory; streaming 0.25 in the recorded run and 0.0 in another, so the result depends on the run.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64.

Corrected (thread-count runs): s-group-mean-cancel’s recorded differ 0 (streaming 0.5) is one outcome among several; on 10 threads the streaming mean varies by run, and one or two threads give the sum 0.0. - probe s-group-mean-cancel in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline round 4 (streaming probes): Carried over from polars-mean-float’s round-2 reading. The Float32 row rests on it: the probes ran Float64 only. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mean.rs#L29-L52 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mean.rs#L80-L124 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/reduce.rs#L42-L104 - 2026-11-04, source-reading, by characterization pipeline round 5 (summation order classes): Read at tag py-2.1.0 (commit 7f902bd3). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mean.rs#L29-L52 finish: f64 sum / count, cast to f32 for Float32 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mean.rs#L73-L124 NumMeanReducer: reduce_ca adds _sum_as_f64, reduce_one and combine += - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/reduce.rs#L42-L104 ungrouped: one state per pipeline, merged in pipeline order - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mod.rs#L457-L528 grouped: reduce_one per row; combine_subset merges the pipelines’ states - https://epubs.siam.org/doi/10.1137/1.9780898718027.ch4 Higham, Accuracy and Stability of Numerical Algorithms (2nd ed., SIAM 2002), ch. 4 Summation: §4.2 recursive, pairwise and any-order bounds; §4.3 compensated summation - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.mean.html silent on the summation algorithm, run-to-run variation and Float32 handling; engine-support in-memory, streaming, distributed - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_scalar_window.py#L84 streaming mean().over() compared with in memory under check_exact=False: the test suite tolerates rounding differences. No test pins the streaming order - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/aggregation/test_aggregations.py#L737-L740 test_mean_overflow: Int64 mean of [9223372036854775800, 100] is a float ≈ 4.611686018427388e18 (eager, in memory) - 2026-11-04, probe, by regression-test runs on macOS arm64, 10 threads (reported to the characterization pipeline): Corrected the description: the 4-row grouped mean is not fixed on streaming. The recorded differ 0 is one outcome; the regression-test runs saw it vary on 10 threads, as the sum does (2.0 ×92, 1.0 ×75, 0.0 ×33 over 200 runs; always 0.0 on 1 or 2 threads). - probe s-group-mean-cancel in characterization/polars/probes-streaming.json

polars-min-max-nan: min, max and max_horizontal skip NaN

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: NaN is ignored unless every value is NaN; nan_max propagates it

max over 1, NaN, -inf, inf is inf and arg_max points at inf, though sort puts NaN last: the aggregates skip NaN, and give NaN only when there is nothing else (max(NaN, NULL) is NaN). pl.max_horizontal(x, NaN) is x. nan_max and nan_min propagate NaN.

min and min_horizontal agree with NaN-as-greatest, since a NaN is never the least of a group with a number.

Float32 and Float64 share the code, and the streaming reducers call the in-memory kernel per morsel and merge with the same min_ignore_nan, so both widths and both engines skip NaN.

The nan_min docstring names numpy’s nanmax and gets numpy’s NaN handling backwards: PL-10, a text fix.

No upstream test pins NaN in min and max on the streaming engine; it may change between versions.

The reference: MAX, greatest follow nan_order: GREATEST, so max(1, NaN) is NaN. The Polars translation writes the shim max_nan_greatest for MAX of doubles (#697) and needs a NaN check for greatest; 20 of 144 rows of greatest differ.

Options: nan_order.

Rows of the choice tables: agg.order nan_order (fp32, fp64); cmp nan_order (fp64).

Constructs: min, max, arg_max, max_horizontal, min_horizontal, nan_max, nan_min.

Settings: {"engine": ["streaming", "in-memory"]}.

See: upstream ledger PL-10, ST-4.

Provenance:

polars-min-max-zero: min and max of ±0: the sign is the platform’s

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: f64::min/simd_min pick the zero; -0.0 for min on arm64

min and max of doubles combine values with Rust’s f32::min/f64::min (max) and the SIMD kernel’s simd_min/simd_max, which give either zero when the operands are 0.0 and -0.0. On macOS arm64 min over {0.0, -0.0} is -0.0 and max is 0.0 in either input order, for Float32 and Float64, grouped or not, on both engines. On Linux x86_64 max of [0.0, -0.0] is -0.0 too.

With the column’s sorted flag set, min returns the first non-null value and max the last, so it returns the zero the sort put first, and the sort holds the two zeros equal (polars-zero-order).

No upstream test pins this; it may change between versions.

The reference: One zero (zero_sign: ONE_ZERO, #89): -0.0 is 0.0 everywhere, so MIN and MAX over zeros are 0.0.

Options: zero_sign.

Rows of the choice tables: agg.order zero_sign (fp32, fp64).

Constructs: min, max.

Settings: {"engine": ["streaming", "in-memory"]}.

See: issues #89.

Provenance:

polars-mode: mode returns every most frequent value

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: a list of all modes, in no stated order

mode returns all the values tied for most frequent, as several rows (a list per group); with maintain_order=True in order of first occurrence, otherwise in an unspecified order. NULL counts as a value ([None, None, True].mode() is [None]), on both engines. mode().first() picks one.

The docs are silent on the default order and on NULL counting as a value: a docs gap.

No upstream test pins the streaming engine’s tie order; it may change between versions.

The reference: MODE returns one value, any of the tie (tie_pick). A writer takes one element of Polars’s list.

Options: tie_pick.

Constructs: mode.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe mode-ties in characterization/polars/probes-streaming.json - probe aggregates-empty in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/chunked_array/mode.rs#L4-L37 in memory: group_tuples(parallel, maintain_order), keep every group of maximal size in group order; NULL is a group - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_expr.rs#L988-L1052 streaming ungrouped: group_by(value).agg(len), filter(count == max(count)); tie order from the streaming group_by unless maintain_order - 2026-11-04, docs, by characterization pipeline reverse pass: Corrected the description: it said the order of the modes is not documented; the docs offer maintain_order and are silent on the default. Added NULL counting as a value. - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.mode.html may return several values; maintain_order parameter; silent on the default order and on NULL counting as a value - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_mode.py#L7-L37 test_mode: ties [“B”, “C”] (order checked only with maintain_order), [1, 2, 3] all tied, [None, None, True] → [None] (eager, in memory) - 2026-11-04, probe, by characterization pipeline reverse pass - pl.LazyFrame({‘x’: [None, None, True]}).select(pl.col(‘x’).mode()).collect(engine=‘streaming’) is [None]: NULL counts as a value on the streaming engine too (polars 2.1.0)

polars-nonfinite-aggregates: Aggregates over ±inf: NaN

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: sum(inf, -inf) and std over inf are NaN

A sum or mean over both infinities is NaN, and std, var over a group with an infinity are NaN.

Float32 takes the same generic code in single precision, so a Float32 sum of inf and -inf is NaN too, on both engines, grouped and not. Nothing checks for the infinities; the Kahan step keeps its error term only while it is finite, so it adds no NaN of its own.

A sum of finite values that overflows is ±inf without raising: sum of 1e308 and 1e308 (Float64) and of 3e38 and 3e38 (Float32, summed in Float32) is inf, grouped and not, on both engines. mean of the Float64 pair is inf; mean of the Float32 pair sums in f64 and is the finite 3.0000000054977558e+38.

The docs are silent on infinities and NaN in sum, mean, var and std: a docs gap.

No upstream test pins inf + (-inf) or var and std with an infinity; they may change between versions.

The reference: nan_creation: raise (#257); a statistical aggregate of ±inf raises (stat_degenerate). ST-3 and ST-5 record that Substrait leaves both open.

Options: nonfinite_aggregates, nan_creation, float_overflow.

Rows of the choice tables: agg.fsum nan_creation (fp32, fp64); agg.fsum float_overflow (fp32, fp64).

Constructs: sum, mean, std, var.

Settings: {"engine": ["streaming", "in-memory"]}.

See: upstream ledger ST-3, ST-5; issues #257.

Provenance:

polars-quantile: quantile: nearest by default, six methods

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: nearest by default; linear interpolates

quantile(q) takes the nearest value by default (index round(q · (n - 1))); the other methods are lower, higher, midpoint, linear and equiprobable. median is quantile(0.5, 'linear'). At q = 0.4 over 1, 2, 3, 4: nearest 2, lower 2, higher 3, midpoint 2.5, linear 2.2, equiprobable 2. NaN sorts above every number (polars-nan-order).

The position is f = q · (n - 1) over the non-null values; nearest takes index f.round() (halves away from zero), linear gives (f - ⌊f⌋) · (hi - lo) + lo in f64, midpoint (lo + hi) / 2, and equal neighbours give lo. For neighbours of opposite sign near the double’s range hi - lo overflows to inf, and the result leaves the interval.

Integers go through the same kernel (QuantileDispatcher for every PolarsIntegerType), each value read to_f64, so the result is Float64. Decimals: ungrouped, the kernel runs on the scaled Int128 values, then divides by 10^scale (two roundings); grouped, the column is first cast to Float64 (correctly rounded). The result is Float64 in both cases.

The result type is moment_dtype: integers, decimals and Float64 give Float64, Float32 (and Float16) keep their type, and Date gives Datetime(us). The docs say a single quantile is f64: PL-11.

The streaming engine has no quantile reduction: ungrouped, quantile runs as a columnar function on the collected column and median through the in-memory fallback; grouped, the query falls back to the in-memory group_by. So both engines run this kernel.

The reference: quantile_disc takes a value of the input; quantile_cont any double between the neighbours (contAllowed).

Options: quantile_interp.

Rows of the choice tables: agg.quant quantile_interp (i64, fp64, decimal).

Constructs: quantile, median.

Settings: {"engine": ["streaming", "in-memory"]}.

See: upstream ledger PL-11.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (aggregates): The examples run on an Int64 column, so the methods and the Float64 result hold for integers. They agree with the probe: nearest takes index round(0.3 · 5) = 2. The docs do not mention decimal input.

The docs at stable are those of Polars 2.1.0: their [source] links point at tag py-2.1.0. - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.quantile.html quantile(q, interpolation='nearest'), methods nearest, higher, lower, midpoint, linear, equiprobable; a single quantile is one f64. Over the Int64 column 0..5 at q = 0.3: nearest 2.0, higher 2.0, lower 1.0, midpoint 1.5, linear 1.5 - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.median.html median uses linear interpolation; f64 result - 2026-11-04, source-reading, by characterization pipeline round 2 (aggregates): Read at tag py-2.1.0 (commit 7f902bd3). Confirms the docs-level i64 row. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/quantile.rs#L21-L112 quantile_idx and the interpolations - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/quantile.rs#L266-L268 median is quantile(0.5, Linear) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/group_by/aggregations/mod.rs#L271-L283 QuantileDispatcher for every integer type, f64 result - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/series/implementations/decimal.rs#L35-L47 apply_scale: the f64 result divided by the scale factor - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/series/implementations/decimal.rs#L508-L517 decimal median: on the physical Int128, then scaled - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/series/implementations/decimal.rs#L535-L540 decimal quantile_reduce: on the physical Int128, then scaled - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/group_by/aggregations/dispatch.rs#L428-L428 grouped decimal median: cast to Float64 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/group_by/aggregations/dispatch.rs#L467-L492 grouped quantile: decimals cast to Float64 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/convert.rs#L59-L120 the streaming reductions: no quantile; Median is todo!() - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_expr.rs#L2276-L2280 ungrouped median: in-memory fallback - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_expr.rs#L2784-L2837 other functions (quantile): a ColumnarFunction node over the collected input - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_group_by.rs#L534-L536 grouped: only elementwise functions lower; others make the group_by fall back - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_group_by.rs#L709-L713 grouped median: no streaming lowering - 2026-11-04, manual-repro, by characterization pipeline round 2 (aggregates): Polars 2.1.0 (Python) from the project’s environment, macOS arm64, POLARS_MAX_THREADS=2; each LazyFrame collected with engine="in-memory" and engine="streaming". - Decimal(10, 2) column [1.10, 2.20, 3.35]: quantile(0.4, ‘linear’) 1.98 and median() 2.2, Float64, on both engines - 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): Streaming agrees with memory: grouped median and quantile(0.3, 'linear') over 30 000 rows give 50.0 and 30.0 per group on both engines.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-quantile-median in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass: Corrected: the description implied the result is always Float64; Float32 stays Float32 and Date gives Datetime(us) (moment_dtype). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L689-L704 moment_dtype: Float32 stays Float32, Float16 stays Float16, other numerics Float64, Date → Datetime(us) - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L933-L1010 test_quantile (crate): each method on UInt32 [2, 1, null, 3, 5, null, 4] (in memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L151-L165 test_quantile: nearest 2, lower 1, higher 3, midpoint 1.5, linear 1.96 on fruits_cars A (eager and lazy) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_statistics.py#L168-L216 test_quantile_list_schema_29030, test_quantile_schema_dtypes_29030: result types Float64, List(Float64), Datetime for Date - 2026-11-04, probe, by characterization pipeline reverse pass - Float32 [1, 2, 3, 4]: quantile(0.4, ‘linear’) and median() are Float32 on both engines; Date quantile(0.5) is Datetime(us). polars 2.1.0, macOS arm64; the docstring says f64

polars-sum-float: sum of floats: pairwise, Kahan when grouped

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: pairwise (full); Kahan (grouped, in memory); in turn (grouped, streaming); in the input’s width

A full aggregation sums pairwise: blocks of 128 in 16 SIMD stripes, combined pairwise, the remainder added in turn. Groups of indices use Kahan summation with a finiteness guard. Streaming merges partial sums plainly.

The sum of no values (or only nulls) is 0 in the Series API.

The sum has the input’s width. A Float32 column adds in single precision on every path: sum_arr_as_f32 with f32 lanes, KahanSum<f32> per group, NumSumReducer<Float32Type> (SumCast maps f32 to f32). One Float32 group of 16777216, 1, 1 sums to 16777216 ungrouped and grouped when streaming, and to 16777218 grouped in memory (Kahan). Each step is one IEEE addition, rounded to nearest with ties to even; the paths differ in the order of the additions. The float-sum kernel uses plain +, not the reassociable alg_add_f64.

Streaming engine (a lazy collect() by default, polars-exec-engine): ungrouped, each morsel is summed by the in-memory kernel (ChunkAgg::sum, pairwise per chunk), added to its pipeline’s running sum, and the pipelines’ sums are added in pipeline order. Which morsels a pipeline receives depends on scheduling, so the grouping of a large sum can change between runs. Grouped, each value is added in turn to its group’s sum (reduce_one, +=) and the pipelines’ partial sums are added on merge. So grouped sums differ between the engines: Kahan in memory, plain left-to-right when streaming. One Float64 group of 1e16, 1, 1, -1e16 sums to 2.0 in memory and 0.0 streaming.

Order class (in memory): (a) ungrouped, pairwise with blocks of 128: in a block each of 16 lanes adds its 8 values in turn and the lanes reduce in a tree of depth 4, the blocks combine pairwise, and the first n mod 128 values are added in turn and then to the rest; several chunks add their sums in turn. Grouped by index groups, Kahan’s compensated sum in the input’s width with the error kept only while finite; grouped by slices (sorted keys), the ungrouped kernel per slice. (b) Any tree: |ŝ − s| ≤ γ_h Σ|x_i| (Higham §4.2), h ≤ max(12 + ⌈log₂ B⌉, r) for one array of n = 128B + r values (7 lane additions, 4 lane levels, ⌈log₂ B⌉ pairwise levels, the final addition; r for the leading remainder); for n ≥ 2¹⁶ that is about ⌈log₂ n⌉ + 5, below the sequential n − 1. Kahan: (2u + O(nu²)) Σ|x_i| (§4.3). u = 2⁻⁵³ for Float64, 2⁻²⁴ for Float32, which sums in f32 throughout. (c) The source below; polars-sum-float’s manual repros (16777218 and 2.0 grouped in memory). (d) Ungrouped and slice-grouped: ANY (a fixed tree), never SEQUENTIAL past 128 values. Index-grouped Kahan: no Lean order (duckdb-kahan-sum has a four-value Kahan result outside every tree sum). Float32: no Lean order either, as SumOrder rounds every step to a double; the class is ANY at u = 2⁻²⁴. The in-memory planner can also hand a group_by with at most two plain column keys and sum/mean/min/max/first/last/count aggregations to the streaming group-by (partitionable_gb, when it has a streaming builder); the probes saw Kahan in memory, so that branch was not taken there.

The reference: AggFn.fsum (floatSum) adds the doubles one by one in list order from +0.0; the writers that need that order exactly use Dialect.float_sum. NULL without values.

Options: sum_order, empty_group_aggregates, rounding.

Rows of the choice tables: agg.fsum sum_order (fp32, fp64); agg.fsum rounding (fp32, fp64; evidence source).

Constructs: sum.

Settings: {"POLARS_IDEAL_MORSEL_SIZE": ["100000 (default)", "64"], "POLARS_SORT_PARTITION_THRESHOLD_BYTES": ["33554432 (default)", "1024"], "engine": "in-memory"}.

See: issues #54, #85; superseded by polars-sum-float-streaming.

Provenance:

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-group-fsum-cancel in characterization/polars/probes-streaming.json - probe s-group-fsum-f32 in characterization/polars/probes-streaming.json - probe s-fsum-runs in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline round 5 (summation order classes): Read at tag py-2.1.0 (commit 7f902bd3). With the simd feature the block uses Simd<F, 16> lanes; without it the same 16-lane loop, so the tree is the same. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L13-L14 STRIPE = 16, PAIRWISE_RECURSION_LIMIT = 128 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L45-L64 horizontal sum: 16 → 8 → 4 lanes, then (v0 + v2) + (v1 + v3) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L72-L104 a block of 128: 16 lanes, each adding 8 values in turn - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L194-L216 pairwise over whole blocks, split at half the blocks - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L254-L265 the first n mod 128 values added in turn, then mainsum + restsum - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L127-L137 ChunkAgg::sum folds the chunks’ sums left; _sum_as_f64 likewise - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/group_by/aggregations/mod.rs#L867-L936 agg_sum: KahanSum per index group; per slice ChunkAgg::sum or the rolling kernel for overlapping slices - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/kahan_sum.rs#L38-L50 Kahan step; the error kept only while finite - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-mem-engine/src/planner/lp.rs#L19-L50 partitionable_gb - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-mem-engine/src/planner/lp.rs#L674-L706 a partitionable group_by goes to GroupByStreamingExec when a streaming builder is given - https://epubs.siam.org/doi/10.1137/1.9780898718027.ch4 Higham, Accuracy and Stability of Numerical Algorithms (2nd ed., SIAM 2002), ch. 4 Summation: §4.2 recursive, pairwise and any-order bounds; §4.3 compensated summation - 2026-11-04, probe, by characterization pipeline round 5 (probe_polars –diff, small morsels): Rerun on small inputs (3,000 or 5,000 rows) in a subprocess with POLARS_IDEAL_MORSEL_SIZE=64 and POLARS_SORT_PARTITION_THRESHOLD_BYTES=1024, which Polars reads at start-up: the same behaviour as the large inputs at the defaults (the unstable streaming sort, sums and var varying by run), so the probes stay cheap. The variables only move the thresholds; the large-input results at the defaults stand. - probe s-fsum-runs in characterization/polars/probes-streaming.json

polars-sum-float-streaming: sum of floats on the streaming engine: in turn per group, scheduling-dependent

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: grouped: plain left-to-right; ungrouped: pairwise per morsel, grouping varies between runs

On the streaming engine a grouped float sum adds each value in turn to its group’s sum (NumSumReducer, reduce_one, +=), in the input’s width, and adds the pipelines’ partial sums on merge. With one pipeline the additions run in input order: one Float64 group of 1e16, 1, 1, -1e16 sums to 0.0 (in memory, Kahan: 2.0), and one Float32 group of 16777216, 1, 1 to 16777216 (in memory 16777218). A frame of 2 to 4 rows streams as one-row morsels on a 10-thread machine, so which pipelines add which rows, and in what order their partial sums merge, changes between runs. Over 200 runs on 10 threads the Float64 group gave 2.0 92 times, 1.0 75 times and 0.0 33 times; with POLARS_MAX_THREADS=1 or 2 it is always 0.0. The Float32 group varies the same way (counts not recorded). The recorded results of s-group-fsum-cancel (0.0) and s-group-fsum-f32 (16777216) are one outcome among several. Each step is one IEEE addition rounded to nearest, ties to even.

Ungrouped, each morsel is summed pairwise by the in-memory kernel and added to its pipeline’s running sum; which morsels a pipeline gets depends on scheduling. So the same query can give different sums from run to run: five runs of a 300 000-value sum gave 151234.90000035526 and 151234.9000003553 (in memory always the first). A grouped sum over 300 000 rows also varied between runs, on both engines (polars-exec-row-order).

The sum of no values (or only nulls) is 0. The in-memory engine’s algorithms are in polars-sum-float.

Order class (streaming): (a) ungrouped, each morsel is summed by the in-memory kernel (pairwise with blocks of 128: in a block each of 16 lanes adds its 8 values in turn and the lanes reduce in a tree of depth 4, the blocks combine pairwise, and the first n mod 128 values are added in turn and then to the rest), added in turn to its pipeline’s running sum, and the pipelines’ sums are added in pipeline order; the morsels each pipeline gets depend on scheduling, so the tree’s shape changes between runs. Grouped, each pipeline adds each row in turn to its group’s sum and the pipelines’ partial sums are added on merge: sequential leaves combined in an order of unknown shape. (b) Any tree: γ_h Σ|x_i| (Higham §4.2). Ungrouped, h ≤ h_morsel + M + P with h_morsel as in polars-sum-float (h ≤ max(12 + ⌈log₂ B⌉, r) for one array of n = 128B + r values (7 lane additions, 4 lane levels, ⌈log₂ B⌉ pairwise levels, the final addition; r for the leading remainder) per morsel), M the morsels of one pipeline and P the pipelines; grouped, h ≤ m − 1 + P with m the largest pipeline’s row count for the group; at most n − 1 in both. u = 2⁻⁵³ for Float64, 2⁻²⁴ for Float32 (summed in f32). (c) The source below; s-fsum-runs (two results over five runs), s-group-fsum-runs, s-group-fsum-cancel (0.0) and s-group-fsum-f32 (16777216). (d) Float64: ANY. Grouped with one pipeline, SEQUENTIAL in the order rows arrive; in general ANY. Float32: the class of ANY at u = 2⁻²⁴, which SumOrder (double steps) cannot express.

The docs are silent on the float summation order and its variation between runs: a docs gap.

No upstream test pins the streaming summation order; it may change between versions.

The reference: AggFn.fsum (floatSum) adds the doubles one by one in list order from +0.0; the writers that need that order exactly use Dialect.float_sum. NULL without values.

Options: sum_order, empty_group_aggregates, rounding.

Rows of the choice tables: agg.fsum sum_order (fp32, fp64); agg.fsum rounding (fp32, fp64); agg.fsum empty_group_aggregates (fp32, fp64; evidence source).

Constructs: sum.

Settings: {"POLARS_IDEAL_MORSEL_SIZE": ["100000 (default)", "64"], "POLARS_SORT_PARTITION_THRESHOLD_BYTES": ["33554432 (default)", "1024"], "engine": "streaming"}.

See: issues #54, #85; supersedes polars-sum-float.

Provenance:

  • 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): s-group-fsum-cancel: 2.0 in memory, 0.0 streaming; s-group-fsum-f32: 16777218 in memory, 16777216 streaming; s-fsum-runs: streaming gave two values over five runs, in memory one; s-group-fsum-runs varied on both engines.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-group-fsum-cancel in characterization/polars/probes-streaming.json - probe s-group-fsum-f32 in characterization/polars/probes-streaming.json - probe s-fsum-runs in characterization/polars/probes-streaming.json - probe s-group-fsum-runs in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline round 4 (streaming probes): Carried over from polars-sum-float’s round-2 reading of the streaming paths, which the probes confirm. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/sum.rs NumSumReducer: reduce_one adds a value to its group’s sum; combine adds partial sums - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/reduce.rs#L42-L104 ungrouped reduce: one state per pipeline, merged in turn - 2026-11-04, source-reading, by characterization pipeline round 5 (summation order classes): Read at tag py-2.1.0 (commit 7f902bd3). - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L13-L14 STRIPE = 16, PAIRWISE_RECURSION_LIMIT = 128 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L45-L64 horizontal sum: 16 → 8 → 4 lanes, then (v0 + v2) + (v1 + v3) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L72-L104 a block of 128: 16 lanes, each adding 8 values in turn - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L194-L216 pairwise over whole blocks, split at half the blocks - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/float_sum.rs#L254-L265 the first n mod 128 values added in turn, then mainsum + restsum - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L127-L137 ChunkAgg::sum folds the chunks’ sums left; _sum_as_f64 likewise - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/sum.rs#L63-L115 NumSumReducer: reduce_ca adds the morsel’s ChunkAgg::sum; reduce_one and combine += - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/reduce.rs#L42-L104 ungrouped: one state per pipeline, merged in pipeline order - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mod.rs#L457-L528 grouped: reduce_one per row; combine_subset merges the pipelines’ states - https://epubs.siam.org/doi/10.1137/1.9780898718027.ch4 Higham, Accuracy and Stability of Numerical Algorithms (2nd ed., SIAM 2002), ch. 4 Summation: §4.2 recursive, pairwise and any-order bounds; §4.3 compensated summation - 2026-11-04, probe, by characterization pipeline round 5 (probe_polars –diff, small morsels): Rerun on small inputs (3,000 or 5,000 rows) in a subprocess with POLARS_IDEAL_MORSEL_SIZE=64 and POLARS_SORT_PARTITION_THRESHOLD_BYTES=1024, which Polars reads at start-up: the same behaviour as the large inputs at the defaults (the unstable streaming sort, sums and var varying by run), so the probes stay cheap. The variables only move the thresholds; the large-input results at the defaults stand. - probe s-fsum-runs in characterization/polars/probes-streaming.json - probe s-group-fsum-runs in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.sum.html states the Int8/UInt8/Int16/UInt16 → Int64 cast and 0 for no non-null values; silent on float summation order and run-to-run variation - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_scalar_window.py#L84 streaming sum().over() compared with in memory under check_exact=False - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/aggregation/test_aggregations.py#L872-L884 test_sum_empty_and_null_set: Float32 empty and all-null sums are 0, select and group_by (eager, in memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/aggregation/test_aggregations.py#L1747-L1750 test_sum_inf_not_nan_25849: grouped sum with one inf is inf (eager, in memory) - 2026-11-04, probe, by regression-test runs on macOS arm64, 10 threads (reported to the characterization pipeline): Corrected the description: it gave the 4-row grouped sums as fixed. On 10 threads they vary by run (s-group-fsum-cancel over 200 runs: 2.0 ×92, 1.0 ×75, 0.0 ×33); POLARS_MAX_THREADS=1 or 2 always gives 0.0. The recorded differ of both probes is one run’s outcome. - probe s-group-fsum-cancel in characterization/polars/probes-streaming.json - probe s-group-fsum-f32 in characterization/polars/probes-streaming.json

polars-sum-int: sum of integers: widened below 32 bits, wraps at 32 and 64; 0 for no values

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: Int8/Int16 sum to Int64; Int32/Int64 wrap; empty is 0

sum of Int8, Int16, UInt8 and UInt16 returns Int64; Int32 and Int64 keep their type and wrap, ungrouped and grouped (Int32 2³¹-1 + 1 is -2³¹).

The sum of no values, or of NULLs only, is 0, also per group: Substrait’s sum0. product of no values is 1.

Both engines wrap, by different code: in memory wrapping_sum_arr calls wrapping_add; the streaming NumSumReducer adds with +=, which wraps because Polars’s release builds leave overflow checks off.

The docs are silent on Int32 and Int64 sums wrapping: a docs gap.

No upstream test pins the wrap; it may change between versions.

The reference: sum of integers returns the input type, with overflow: ERROR (an inferred precondition checked on the data, #316, #437), and NULL for no values (functions_arithmetic.yaml: “The sum of zero elements yields null”).

Options: overflow, empty_group_aggregates.

Rows of the choice tables: agg.isum overflow (i8, i16, i32, i64); agg.isum empty_group_aggregates (i64).

Constructs: sum, product.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #316, #437.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, source-reading, by session brave-borg-41f7cc: Read at tag py-2.1.0 (commit 7f902bd3).
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe int-sum-types in characterization/polars/probes-streaming.json - probe int-sum-grouped in characterization/polars/probes-streaming.json - probe int-sum-empty in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass: Added to the description: the in-memory and streaming kernels and why both wrap. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L27-L44 SumCast: u8, u16, i8, i16 sum as i64; u32, u64, i32, i64 keep their type - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/sum.rs#L178-L203 in memory: wrapping_sum_arr / wrapping_sum_arr_upcast, wrapping_add - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/sum.rs#L16-L60 streaming new_sum_reduction: NumSumReducer per physical type; null_on_empty uses VecMaskGroupedReduction - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/sum.rs#L80-L113 streaming: init zero; reduce_one, reduce_ca and combine use +=, which wraps in release builds - https://github.com/pola-rs/polars/blob/py-2.1.0/Cargo.toml#L210-L212 release profile: lto and debug only, overflow-checks stay off - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.sum.html states Int8/UInt8/Int16/UInt16 → Int64 and 0 for no non-null values; silent on Int32/Int64 wrapping - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_list.py#L316-L358 test_list_sum_and_dtypes: sum dtypes Int8/Int16/UInt8/UInt16 → Int64, Int32 → Int32, Int64 → Int64 (eager) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/lazyframe/test_optimizations.py#L178-L211 test_count_guarded_sum, in-memory and streaming: when(count > 0).then(sum) becomes sum(null_on_empty=true), same result. No upstream test of integer overflow wrapping - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_group_by.py#L313-L337 streaming grouped sums over every integer dtype match the eager total - 2026-11-04, probe, by characterization pipeline reverse pass - Int32 [2**31 - 1, 1]: sum() is -2147483648 with engine in-memory and streaming (POLARS_IDEAL_MORSEL_SIZE=64), polars 2.1.0, macOS arm64

polars-var-std: var and std: two-pass blocks of 128, or Welford, by path

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: two-pass per 128-block with Chan merge (ungrouped); Welford (grouped); clamps below 0

Ungrouped: two-pass per block of 128 values (mean = alg_sum(x) / n, then dp = alg_sum((x - mean)^2); alg_sum is a plain left fold, reassociated only on nightly Rust), blocks merged in turn by Chan’s combine. The chunks of a ChunkedArray merge the same way.

Grouped, in memory: groups of indices use Welford’s online variance; slice groups use the block algorithm; overlapping slices use rolling VarState windows. The streaming reducers insert one value at a time (Welford) or combine per chunk. So grouped and ungrouped results come from different algorithms.

A weight at most ddof is null: no values is null, the sample variance of one value is null, the population variance of one value is 0. VarState::finalize clamps a negative variance to 0 and propagates NaN; the Welford path does not clamp.

Integers and decimals become doubles first, on both engines: an integer by as f64 (round to nearest, so 2⁶², 2⁶² + 1, 2⁶² + 2 all become 2⁶² and their variance is 0.0), a decimal by the cast to Float64 (dec128_to_f64, correctly rounded; VarStdReducer with needs_cast, Decimal’s var and agg_var). The result is Float64 (Float32 for Float32 input). ddof is a u8 and the divisor n - ddof: Substrait’s SAMPLE is ddof=1, POPULATION ddof=0.

The Python wheels build with the nightly feature, so alg_add_f64 is Rust’s algebraic_add, which lets the compiler reassociate and fuse the additions of a block’s two passes: their rounding order is the compiler’s, fixed per build. Welford’s updates (online_variance, VarState::insert_one) and Chan’s merge use plain IEEE operations.

Streaming runs VarState too. Ungrouped, each morsel’s chunks go through the 128-value blocks and merge by Chan’s formula into their pipeline’s state, and the pipelines’ states merge in pipeline order: block boundaries follow the morsels. Grouped, each value goes through insert_one (Welford) and the pipelines merge by Chan’s formula. In memory, groups of indices use online_variance (Welford, no clamp). The two engines agree on NULL, the clamp of VarState and the type, and can differ in the last bits.

The reference: AggFn.var (varOf, SubstraitLean/Aggregate.lean) is two-pass with native doubles in list order: the mean (floatSum / n), then the squared deviations from it, added from +0.0, by n or n - 1, and never below 0. A sample of one value is NULL. Only NULL and the sign are exact; engines may differ in the last bits (#54) until #85 bounds the error.

Options: stat_algo, stat_degenerate, rounding.

Rows of the choice tables: agg.var stat_algo (i64, fp64, decimal); agg.var stat_degenerate (i64, fp64, decimal); agg.var distribution (i64, fp64, decimal); agg.var rounding (i64); agg.var rounding (fp64, decimal; evidence source).

Constructs: var, std.

Settings: {"POLARS_IDEAL_MORSEL_SIZE": ["100000 (default)", "64"], "POLARS_SORT_PARTITION_THRESHOLD_BYTES": ["33554432 (default)", "1024"], "engine": "in-memory"}.

See: issues #54, #85; superseded by polars-var-std-streaming.

Provenance:

The docs at stable are those of Polars 2.1.0: their [source] links point at tag py-2.1.0. - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.var.html var(ddof=1): the divisor is N - ddof; var of the Int64 column -1, 0, 1 is the Float64 1.0 - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.std.html std(ddof=1), the same divisor; std of the Int64 column -1, 0, 1 is 1.0 - 2026-11-04, source-reading, by characterization pipeline round 2 (aggregates): Read at tag py-2.1.0 (commit 7f902bd3). Confirms the docs-level distribution row: ddof is the divisor’s offset on every path. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/moment.rs#L70-L134 VarState: two-pass block (alg_sum_f64), insert_one, Chan’s combine, finalize (null at weight ≤ ddof, clamp below 0) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/moment.rs#L584-L644 var: values as_() f64 in blocks of 128 (CHUNK_SIZE), merged in turn - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/var.rs#L12-L28 in-memory var: chunks merged by combine - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-arrow/src/legacy/kernels/take_agg/var.rs#L11-L63 online_variance for groups of indices, values to_f64 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/frame/group_by/aggregations/mod.rs#L1311-L1360 in-memory grouped var of integers: online_variance, or the Float64 cast for rolling slices - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/series/implementations/decimal.rs#L182-L195 grouped var/std of decimals: cast to Float64 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/series/implementations/decimal.rs#L519-L533 ungrouped var/std of decimals: cast to Float64 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/decimal.rs#L557-L565 dec128_to_f64: the nearest f64 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/var_std.rs#L9-L47 streaming: VarStdReducer per physical numeric type; decimals through Float64Type with needs_cast - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/var_std.rs#L67-L142 reduce_one is insert_one, reduce_ca merges moment::var per chunk, finish is finalize(ddof) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/runtime/polars-runtime-64/Cargo.toml#L47-L48 nightly feature, which also turns on simd - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/runtime/polars-runtime-64/Cargo.toml#L114 default = ["full", "nightly"]: the wheels build with it - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/rust-toolchain.toml toolchain nightly-2026-09-01 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/algebraic_ops.rs#L1-L29 alg_add_f64 is algebraic_add under nightly, a plain + otherwise; alg_sum_f64 folds from -0.0 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/reduce.rs#L42-L104 ungrouped streaming reduce: each pipeline folds its morsels into a local state (update_group), the local states merge in pipeline order (combine_subset) - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mod.rs#L396-L455 VecGroupedReduction: update_group calls reduce_ca on a morsel; update_groups_subset calls reduce_one per row - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/mod.rs#L457-L528 update_groups_while_evicting (per row reduce_one) and combine_subset (combine) - 2026-11-04, manual-repro, by characterization pipeline round 2 (aggregates): Polars 2.1.0 (Python) from the project’s environment, macOS arm64, POLARS_MAX_THREADS=2; each LazyFrame collected with engine="in-memory" and engine="streaming". - Int64 column [5]: var() null, var(ddof=0) 0.0, std() null, on both engines; Decimal(10, 2) column [5]: the same - Int64 column [262, 262 + 1, 2**62 + 2]: var() 0.0 on both engines (the exact sample variance is 1) - Decimal(10, 2) column [1.10, 2.20, 3.35]: var() 1.2658333333333334 on both engines - 2026-11-04, engine-test, by characterization pipeline round 3 (aggregates): Supports the decimal path (cast to Float64) and the agreement of the engines on degenerate input; the decimal test compares within a tolerance, so it does not pin the rounding. The rounding rows stay where they are.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/aggregation/test_aggregations.py#L1424-L1518 test_agg_invalid_same_engines_behavior: var and std of one value, Decimal, Float32, Float64, Int32 and others, grouped and not: the streaming engine errors exactly when the in-memory engine does, and otherwise returns the same frame - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L727-L753 Rust test_var: Int32 5, 8, 9, 5, 0 (and with NULLs) var(1) is exactly 12.3 - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_decimal.py#L1036-L1064 test_decimal_agg: var and std of a Decimal(scale=3) column equal those of the Int64 column, grouped and not, within assert_frame_equal’s default tolerance (eager) - 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): Narrowed to the in-memory engine; the streaming engine is in polars-var-std-streaming. Its ungrouped variance changes between runs (three values in five runs), which this fact did not say; grouped it agrees with the in-memory engine on the probe.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-var-runs in characterization/polars/probes-streaming.json - probe s-group-var in characterization/polars/probes-streaming.json - 2026-11-04, probe, by characterization pipeline round 5 (probe_polars –diff, small morsels): Rerun on small inputs (3,000 or 5,000 rows) in a subprocess with POLARS_IDEAL_MORSEL_SIZE=64 and POLARS_SORT_PARTITION_THRESHOLD_BYTES=1024, which Polars reads at start-up: the same behaviour as the large inputs at the defaults (the unstable streaming sort, sums and var varying by run), so the probes stay cheap. The variables only move the thresholds; the large-input results at the defaults stand. - probe s-var-runs in characterization/polars/probes-streaming.json

polars-var-std-streaming: var/std on the streaming engine: ungrouped result varies between runs

Engine: polars py-2.1.0; area: aggregates; status: open; evidence: probe.

Verdict: Chan merge of per-morsel blocks (ungrouped, run-dependent); Welford per value (grouped)

On the streaming engine var and std run VarState. Ungrouped, each morsel’s chunks go through the 128-value two-pass blocks and merge by Chan’s formula into their pipeline’s state, and the pipelines’ states merge in turn. Block boundaries follow the morsels and the merge order follows scheduling, so the result can change in the last bits between runs: five runs over 300 000 values gave 4.440337823592621, …622 and …623 (in memory always 4.440337823592626).

Grouped, each value goes through insert_one (Welford) and pipelines merge by Chan’s formula; on four values near 1e8 the recorded run agrees with the in-memory engine (1.6666666666666667). That is one outcome among several: the four rows stream as one-row morsels on a 10-thread machine, so which pipelines’ states merge, and in what order, changes between runs, and the result varies by run (the counts were not recorded; the one-thread value was not recorded either).

Integers and decimals become doubles first (as f64, dec128_to_f64); the result is Float64 (Float32 for Float32 input). ddof is the divisor’s offset: SAMPLE is ddof=1, POPULATION ddof=0; a weight at most ddof is NULL, and VarState::finalize clamps a negative variance to 0. The in-memory engine’s algorithms are in polars-var-std.

Algorithm and error (sum_order does not apply; stat_algo): ungrouped, each run of up to 128 values (chunk_as_float, values widened to f64) is a two-pass block: the mean by alg_sum_f64, then the sum of squared deviations by alg_sum_f64, and blocks merge by Schubert and Gertz’s update (Chan’s pairwise formula), chained per pipeline and then across pipelines. alg_sum_f64 folds with algebraic_add in the nightly build the wheels use (default = ["full", "nightly"]), which lets the compiler reassociate, so the order inside a block is unspecified. Grouped, Welford’s update per value and the same merge. Error: the two-pass formula has a relative error of order n·u + n²u²κ² for the condition number κ (Higham §1.9), with n ≤ 128 per block here; the chained merges add errors of order u·κ per merge (Higham §1.9 for the updating formulas), unbounded in the number of merges; no tighter bound is stated. For the Lean side this is a WELFORD_CHAN-like algorithm, which StatAlgo treats as GENERIC: the generic properties hold, and no error bound is used.

The docs are silent on the algorithm, the variation between runs, the clamp to 0 and NULL for a weight at most ddof: a docs gap.

No upstream test pins the streaming order, the clamp or that NULL; they may change between versions.

The reference: Substrait’s variance and std_dev with distribution; the reference states the generic properties (non-negative, NULL for a weight at most ddof) and leaves the algorithm to the engine profile.

Options: stat_algo, stat_degenerate, rounding.

Rows of the choice tables: agg.var stat_algo (fp64); agg.var stat_algo (i64, decimal; evidence source); agg.var stat_degenerate (fp64); agg.var stat_degenerate (i64, decimal; evidence source); agg.var distribution (fp64); agg.var distribution (i64, decimal; evidence source); agg.var rounding (fp64); agg.var rounding (i64, decimal; evidence source).

Constructs: var, std.

Settings: {"POLARS_IDEAL_MORSEL_SIZE": ["100000 (default)", "64"], "POLARS_SORT_PARTITION_THRESHOLD_BYTES": ["33554432 (default)", "1024"], "engine": "streaming"}.

See: supersedes polars-var-std.

Provenance:

  • 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): s-var-runs: streaming gave three values over five runs, in memory one; s-group-var: both engines 1.6666666666666667.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64.

Corrected (thread-count runs): s-group-var’s recorded agreement (differ 0) is one outcome among several; the regression-test runs on 10 threads saw it vary by run. - probe s-var-runs in characterization/polars/probes-streaming.json - probe s-group-var in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline round 4 (streaming probes): Carried over from polars-var-std’s round-2 reading of the streaming paths (VarStdReducer with needs_cast for integers and decimals). The i64 and decimal rows rest on it: the streaming probes ran Float64. Polars’s test_agg_invalid_same_engines_behavior (round 3) checks the engines agree on one-value input, Decimal included. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/moment.rs VarState: blocks, insert_one, Chan’s combine, finalize clamp - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/var_std.rs VarStdReducer, needs_cast for integers and decimals - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/reduce.rs#L42-L104 - 2026-11-04, source-reading, by characterization pipeline round 5 (summation order classes): Read at tag py-2.1.0 (commit 7f902bd3). That the nightly feature reaches polars-utils through polars-python was not traced link by link. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/moment.rs#L37-L133 CHUNK_SIZE = 128; VarState::new (two-pass), insert_one (Welford), combine, finalize - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/moment.rs#L584-L602 chunk_as_float: runs of 128 widened values - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-compute/src/moment.rs#L629-L642 var: VarState::new per chunk, combined - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-utils/src/algebraic_ops.rs#L1-L29 alg_add_f64: algebraic_add with nightly; alg_sum_f64 folds from -0.0 - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/runtime/polars-runtime-64/Cargo.toml#L48 nightly feature - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/runtime/polars-runtime-64/Cargo.toml#L114 default = [“full”, “nightly”] - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-expr/src/reduce/var_std.rs#L80-L100 VarStdReducer: combine, insert_one per row, var(arr) per morsel - https://epubs.siam.org/doi/10.1137/1.9780898718027.ch1 Higham, ch. 1, §1.9: error of the two-pass and updating formulas for the sample variance - 2026-11-04, probe, by characterization pipeline round 5 (probe_polars –diff, small morsels): Rerun on small inputs (3,000 or 5,000 rows) in a subprocess with POLARS_IDEAL_MORSEL_SIZE=64 and POLARS_SORT_PARTITION_THRESHOLD_BYTES=1024, which Polars reads at start-up: the same behaviour as the large inputs at the defaults (the unstable streaming sort, sums and var varying by run), so the probes stay cheap. The variables only move the thresholds; the large-input results at the defaults stand. - probe s-var-runs in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.var.html ddof is the divisor’s offset N - ddof; silent on algorithm, variation, clamp and NULL for a weight at most ddof - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.std.html same as var - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/aggregate/mod.rs#L729-L751 test_var (crate): Int32 [5, 8, 9, 5, 0] with and without nulls, var(1) = 12.3 exactly, std(1) within 1e-9 of 3.5071355833500366 (in memory) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_scalar_window.py#L84 streaming std/var over windows compared with in memory under check_exact=False. No test pins the streaming bits - 2026-11-04, probe, by regression-test runs on macOS arm64, 10 threads (reported to the characterization pipeline): Corrected the description: it gave the 4-row grouped variance as matching in memory. On 10 threads it varies by run, as the grouped sum and mean do; the recorded differ 0 is one outcome. - probe s-group-var in characterization/polars/probes-streaming.json

windows

polars-rank: rank: average by default; NULL ranks are NULL

Engine: polars py-2.1.0; area: windows; status: open; evidence: probe.

Verdict: rank('min') is RANK, 'dense' is DENSE_RANK, 'ordinal' is ROW_NUMBER

rank() gives tied rows their average rank as a double; the methods min, max, dense and ordinal give integers. A NULL’s rank is NULL (SQL ranks NULLs with the other rows), and descending=True reverses the order with NaN first. There is no ntile, percent_rank or cume_dist expression.

rank sorts with a stable arg_sort (NULLs last, then dropped), so ordinal numbers tied rows in input order; min, max, dense and average give a run of equal values one rank. NULL rows keep a NULL rank (the input’s validity).

The expression API has no ntile, percent_rank or cume_dist; polars-sql (pl.sql, SQLContext) has NTILE, PERCENT_RANK and CUME_DIST, built from expressions. Its NTILE(k) over n rows gives the first n % k buckets one row more, the SQL standard’s spread. The order of tied rows within it follows over’s order_by sort, unchecked here.

The reference: rank, dense_rank, row_number over the window’s order, NULLs placed by the key’s direction; percent_rank, cume_dist, ntile as SQL.

Options: default_window_frame, ntile_spread, tie_order.

Rows of the choice tables: win.rank tie_order (any); win.ntile ntile_spread (any); win.ntile tie_order (any; evidence docs).

Constructs: rank.

Settings: {"engine": ["streaming", "in-memory"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, docs, by characterization pipeline round 1 (windows): rank('ordinal'), Polars’s ROW_NUMBER, breaks ties by input order, so over with a pre-sorted frame keeps a stable sort’s order of ties. rank('min') and rank('dense') give tied rows one rank. There is no NTILE, so ntile_spread and its tie order have no Polars counterpart: a plan with ntile is refused.

The docs at stable are those of Polars 2.1.0: their [source] links point at tag py-2.1.0. - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rank.html ‘ordinal’: every value gets a distinct rank, in the order the values occur in the Series; ‘random’: ties ranked independently of that order (seeded by seed); Notes: only non-null values are ranked and nulls are kept - https://docs.pola.rs/api/python/stable/reference/expressions/window.html the window functions listed are over and rolling - https://docs.pola.rs/api/python/stable/genindex.html the API index has no ntile, percent_rank or cume_dist - 2026-11-04, source-reading, by characterization pipeline round 2 (windows): Read at tag py-2.1.0 (commit 7f902bd3). Confirms the docs-level win.rank row and, through polars-sql, ntile_spread; the tie order of NTILE stays at the docs level. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/rank.rs#L29-L155 rank: arg_sort with nulls_last, Ordinal in sorted order, the other methods per run of equal values; the input’s validity kept - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/sort/arg_sort.rs#L53-L110 descending: a stable reverse - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/chunked_array/ops/sort/arg_sort.rs#L270-L285 sort_by/par_sort_by: stable sorts - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-sql/src/functions.rs#L910-L930 polars-sql’s PERCENT_RANK, CUME_DIST, NTILE - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-sql/src/functions.rs#L2098-L2127 NTILE: q = n / k, the first n % k buckets of q + 1 rows - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-sql/src/functions.rs#L3127-L3130 window_row_index: int_range(0, len) within the window - 2026-11-04, engine-test, by characterization pipeline round 3 (windows): Pins the bucket spread of polars-sql’s NTILE against DuckDB on both engines. Both orderings of test_ntile are total (ties broken by id), so the order of tied rows within NTILE stays unpinned and the win.ntile tie_order row stays at the docs. The ranking functions match DuckDB on ties and NULLs.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/sql/test_rank_functions.py#L219-L235 test_ntile: SQL NTILE(k) for k = 1, 2, 3, 5, 9 over ORDER BY id and PARTITION BY g ORDER BY x DESC, id matches DuckDB, on engines=['in-memory', 'streaming'], dtypes checked - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/sql/test_rank_functions.py#L163-L198 test_rank_funcs_nulls_and_ties: RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST with tied and NULL keys under eight orderings, with and without partitions, match DuckDB on both engines - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_expr.rs#L2784-L2800 streaming: rank has no lowering of its own; the generic fallback runs the in-memory kernel on the collected column - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/rank.rs#L170-L273 test_rank (crate): ordinal [2, 3, 6, 4, 5, 7, 1], dense [2, 3, 4, 3, 3, 4, 1] for [1, 2, 3, 2, 2, 3, 0] - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-ops/src/series/ops/rank.rs#L300-L310 test_rank_reverse: dense descending keeps NULL ranks NULL - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/test_rank.py#L35-L50 test_rank_df: average Float64 [1.5, 1.5, 3.5, 3.5, 5.0], max index type [2, 2, 4, 4, 5] (eager)

polars-window-values: Windows: over keeps the input order inside a partition

Engine: polars py-2.1.0; area: windows; status: no-finding; evidence: probe.

Verdict: first().over, shift(1).over, cum_sum().over in input order

An expression .over(g) runs per partition in input order: first().over('g') is the partition’s first row, shift(1).over('g') the previous row (NULL at the start), cum_sum().over('g') the running sum. A window order is a sort before (or order_by= in over).

The reference: lag, lead, first_value and windowed aggregates over the window’s order and frame; ties in any order (tie_order).

Options: default_window_frame, tie_order.

Rows of the choice tables: win.value tie_order (any).

Constructs: over, shift, first, cum_sum.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

window-lag-first has three rows; s-window-order checks shift(1), first() and cum_count() .over('g') on 100 000 rows in 7 partitions, lazily on each engine: input order inside each partition on both.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe window-lag-first in characterization/polars/probes-streaming.json - probe s-window-order in characterization/polars/probes-streaming.json

relations

polars-clustered: No clustering-only sort; sort clusters

Engine: polars py-2.1.0; area: relations; status: no-finding; evidence: probe.

Verdict: a sort on the keys

Polars has no operation that only brings equal keys together; sort on the keys does (with maintain_order=True the rows of a key keep their order).

The reference: SORT_DIRECTION_CLUSTERED: equal keys adjacent, groups in any order; the parser refuses it for now.

Options: clustered.

Rows of the choice tables: clustered (any).

Constructs: sort.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The streaming sort with maintain_order=True keeps ties in input order above its partition threshold too (polars-sort-stability-streaming).

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe clustered in characterization/polars/probes-streaming.json

polars-null-order: sort: NULLs first, ascending and descending

Engine: polars py-2.1.0; area: relations; status: no-finding; evidence: probe.

Verdict: NULLS_FIRST by default in both directions

sort puts NULLs first by default, ascending and descending; nulls_last=True puts them last, per key when given a list. So Polars’s default is the same NULL placement in both directions, where SQL’s convention flips it with the direction.

The reference: The key’s direction states the NULL placement; SORT_DIRECTION_UNSPECIFIED is NULLS_LAST (D8). Polars is native for every stated direction with nulls_last; its default differs from DuckDB’s NULLS_LAST (duckdb-F2).

Options: null_order.

Rows of the choice tables: null_order (any).

Constructs: sort, top_k, rank.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger ST-7.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observations’ frames hold a few rows, which streaming runs as one-row morsels; the probes’ 6000-row columns cover morsels of many rows.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe sort-doubles in characterization/polars/probes-streaming.json - probe sort-null-default in characterization/polars/probes-streaming.json

polars-set-operations: Set operations are concatenation, unique and semi/anti joins

Engine: polars py-2.1.0; area: relations; status: no-finding; evidence: probe.

Verdict: UNION ALL is concat; INTERSECT, EXCEPT need nulls_equal=True and unique

Polars has no set-operation node: pl.concat is UNION ALL, and unique after it is UNION. INTERSECT and EXCEPT are a semi and an anti join on every column with nulls_equal=True, then unique: without nulls_equal NULL rows never match (a semi join drops them, an anti join keeps them).

INTERSECT ALL and EXCEPT ALL have no direct form; the reference’s numbering of equal rows (a cum_count over every column) would write them.

The reference: UNION ALL is bag union and UNION DISTINCT its distinct; INTERSECT and EXCEPT are semi and anti joins with NULLs equal, then DISTINCT, and the ALL forms number equal rows (#344).

Options: set_operations, null_join_keys.

Constructs: concat, unique, join.

Settings: {"engine": ["in-memory", "streaming"]}.

See: issues #344.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

pl.concat of two frames runs eagerly, in memory; s-set-operations runs the union, intersect and except lazily on each engine, on 6000 and 6000 rows. UNION ALL keeps both inputs in order on both engines, and the results hold the same rows. The intersect’s two rows came out in another order (in memory NULL first, streaming 1 first), which the fact does not pin.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe set-operations in characterization/polars/probes-streaming.json - probe s-set-operations in characterization/polars/probes-streaming.json

polars-sort-stability: sort is stable on the probe, guaranteed only with maintain_order

Engine: polars py-2.1.0; area: relations; status: no-finding; evidence: probe.

Verdict: ties keep input order (guaranteed with maintain_order=True)

Sorting 200 000 rows on a key with 7 values kept the input order within each key, in memory and streaming, with and without maintain_order. Polars documents order among ties only for maintain_order=True.

top_k keeps 2 of 3 tied rows; which ones is not documented.

The streaming sort buffers its input. Up to POLARS_SORT_PARTITION_THRESHOLD_BYTES (32 MiB, at most half the out-of-core memory budget) it concatenates the morsels in sequence order and sorts them with the in-memory sort. Above it, it splits the rows into key-range buckets: with maintain_order each task reads a contiguous run of frames, which keeps input order within a bucket; without it the frames are dealt to the tasks in turn, and the source does not keep input order among ties. The 200 000-row probe stayed below the threshold. With the threshold lowered to 100 000 bytes, a sort of 200 000 rows on 7 keys still kept input order among ties, with and without maintain_order, so the unordered path is unconfirmed.

The reference: tie_order: any order of the ties, and any rows of a tie for a fetch. Every behaviour is allowed.

Options: tie_order.

Rows of the choice tables: tie_order (any).

Constructs: sort, top_k.

Settings: {"POLARS_IDEAL_MORSEL_SIZE": ["100000 (default)", "64"], "POLARS_SORT_PARTITION_THRESHOLD_BYTES": ["33554432 (default)", "1024"], "engine": "in-memory"}.

See: superseded by polars-sort-stability-streaming.

Provenance:

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_sort.py#L462-L468 test_sort_stability_with_duplicate_keys: sort('k', maintain_order=True) over 517 rows with keys i % 5 returns idx in (key, input index) order on the streaming engine, equal to the in-memory sort, in the in-memory, partitioned (morsels of 7, 1-byte partition threshold) and spilled sort modes - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_sort.py#L280-L395 test_sort_direct_key_dtypes with the f32 key (517 rows with NaN, ±inf, -0.0, NULL), ascending and descending, maintain_order=True: the streaming sort must equal the in-memory sort exactly, with the in-memory and the partitioned (POLARS_SORT_PARTITION_THRESHOLD_BYTES=1, morsels of 7) sort modes - 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): Narrowed to the in-memory engine; the streaming engine is in polars-sort-stability-streaming. Above the partition threshold its sort without maintain_order does not keep input order among ties, which this fact left unconfirmed.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-sort-ties-large in characterization/polars/probes-streaming.json - 2026-11-04, probe, by characterization pipeline round 5 (probe_polars –diff, small morsels): Rerun on small inputs (3,000 or 5,000 rows) in a subprocess with POLARS_IDEAL_MORSEL_SIZE=64 and POLARS_SORT_PARTITION_THRESHOLD_BYTES=1024, which Polars reads at start-up: the same behaviour as the large inputs at the defaults (the unstable streaming sort, sums and var varying by run), so the probes stay cheap. The variables only move the thresholds; the large-input results at the defaults stand. - probe s-sort-ties-large in characterization/polars/probes-streaming.json

polars-sort-stability-streaming: Streaming sort is unstable above the partition threshold without maintain_order

Engine: polars py-2.1.0; area: relations; status: no-finding; evidence: probe.

Verdict: ties keep input order with maintain_order=True; without it, not above 32 MiB

The streaming sort concatenates its morsels in order and runs the in-memory sort up to POLARS_SORT_PARTITION_THRESHOLD_BYTES (32 MiB). Above it, it splits rows into key-range buckets. With maintain_order=True each task reads a contiguous run of frames and ties keep input order (Polars’s test_sort_stability_with_duplicate_keys checks it, also partitioned and spilled). Without it the frames are dealt to the tasks in turn, and ties come out in another order: a sort of 3 000 000 rows on i % 3 returned the rows of key 1 out of input order (in memory in order).

Below the threshold ties kept input order on every probe, with and without maintain_order. Polars documents order among ties only for maintain_order=True.

The reference: tie_order: any order of the ties, and any rows of a tie for a fetch. Every behaviour is allowed.

Options: tie_order.

Rows of the choice tables: tie_order (any).

Constructs: sort, top_k.

Settings: {"POLARS_IDEAL_MORSEL_SIZE": ["100000 (default)", "64"], "POLARS_SORT_PARTITION_THRESHOLD_BYTES": ["33554432 (default)", "1024"], "engine": "streaming", "maintain_order": [true, false]}.

See: supersedes polars-sort-stability.

Provenance:

  • 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): s-sort-ties-large: sort('k') without maintain_order over 3 000 000 rows; the input indices of key 1 are increasing in memory (True) and not when streaming (False).

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-sort-ties-large in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline round 4 (streaming probes): Carried over from polars-sort-stability’s round-2 reading, which the probe now confirms for the unordered path. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/sort/mod.rs#L121-L165 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/sort/mod.rs#L493-L535 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/nodes/sort/tuning.rs#L5-L64 partition threshold - 2026-11-04, probe, by characterization pipeline round 5 (probe_polars –diff, small morsels): Rerun on small inputs (3,000 or 5,000 rows) in a subprocess with POLARS_IDEAL_MORSEL_SIZE=64 and POLARS_SORT_PARTITION_THRESHOLD_BYTES=1024, which Polars reads at start-up: the same behaviour as the large inputs at the defaults (the unstable streaming sort, sums and var varying by run), so the probes stay cheap. The variables only move the thresholds; the large-input results at the defaults stand. - probe s-sort-ties-large in characterization/polars/probes-streaming.json

execution

polars-exec-cse-hoist: CSE hoists a cast shared by when/then branches onto every row

Engine: polars 2.1.0; area: execution; status: deviation; evidence: probe.

Verdict: deviation: a guarded strict cast raises when two branches share it (PL-6)

A lazy query collects with common-subexpression elimination on. When two expressions share a subexpression, CSE computes it once in a WITH_COLUMNS below the projection, on every row. That includes a strict cast inside when(c).then(...): the cast now runs on the rows c excludes and raises there, although when/then alone masks the branch (polars-exec-when-eager). Two columns when(x == '1').then(x.cast(Int64) + 1) and when(x == '1').then(x.cast(Int64) + 2) over x = ['1', 'a'] raise InvalidOperationError on both engines; with comm_subexpr_elim=False they give [(2, 3), (None, None)], and so does an eager DataFrame.select, which skips CSE.

So whether a query raises depends on the optimizer. PL-6 records it upstream.

The docs are silent on common-subexpression elimination: the lazy optimizations guide lists common subplan elimination and leaves it out, and comm_subexpr_elim says only that it elides duplicate expressions: a docs gap. The when docstring says every branch runs on every row, which documents the CSE outcome; the masking CSE undoes is the undocumented part (PL-13).

No upstream test pins this; it may change between versions.

The reference: The reference evaluates a CASE branch only on the rows that reach it, so a plan whose casts are guarded by their conditions is error-free; sverify’s Polars route collects lazily with CSE on, so such a plan can raise (ST-12, #604).

Options: evaluation_order.

Constructs: when/then/otherwise, cast(strict=True), comm_subexpr_elim.

Settings: {"comm_subexpr_elim": [true, false], "engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-6, PL-13; issues #604.

Provenance:

polars-exec-engine: Engine choice: a lazy collect() runs on the streaming engine by default

Engine: polars py-2.1.0; area: execution; status: open; evidence: engine_test.

Verdict: auto is streaming for lazy queries, in memory for eager ones; four switches change it

collect(engine="auto"), the default, uses the engine set by Config.set_engine_affinity or POLARS_ENGINE_AFFINITY, and otherwise the streaming engine (docstring at py-2.1.0). The Rust side resolves Engine::Auto to the in-memory engine only for eager operations and to streaming for every lazy query (polars-lazy/src/frame/mod.rs); eager DataFrame methods always run in memory (_eager_engine). With engine="streaming" there is no whole-query fallback: collect_with_engine runs polars_stream::run_query and returns its error. The fallback is per node: the lowering hands operations it cannot stream to the in-memory engine inside the streaming plan (polars-exec-streaming-fallback). The only whole-query switch is POLARS_AUTO_STREAMING=1, which tries streaming first for a non-streaming engine.

sverify targets the streaming engine (#605 pins collect(engine="streaming")), so the settings that matter are those that move a query off it: POLARS_ENGINE_AFFINITY and Config.set_engine_affinity no longer apply to an explicit engine, while POLARS_FORCE_STREAMING only forces streaming. The risk is the per-node fallback: a node run in memory takes the in-memory engine’s algorithms (its float sums, its row order) on a materialized frame. The probes behind polars-sum-float, polars-var-std, polars-cov and polars-mean-float called eager DataFrame methods, which run in memory, and need rerunning under engine="streaming".

Switches that change the engine: Config.set_engine_affinity(name), which writes POLARS_ENGINE_AFFINITY; Config.set_engine_affinity(Engine object), which clears the variable and keeps the engine in a Python-side override; POLARS_FORCE_STREAMING=1, which forces streaming whatever the affinity; POLARS_AUTO_STREAMING=1, which tries streaming first for a non-streaming engine. POLARS_MAX_THREADS, read once at import, sets the thread pool. pl.Config’s other settings change display only.

Next round: probe the engine-dependent facts under engine="streaming", and the result of collect() with each switch.

The docs are silent on POLARS_FORCE_STREAMING and POLARS_AUTO_STREAMING: a docs gap.

No upstream test pins that a lazy collect() with engine="auto" and no affinity runs the streaming engine; it may change between versions.

The reference: The reference has no engine choice. A profile’s facts hold under the settings they record (sverify._engine_settings, #444); the Polars row accepts the affinity auto and in-memory.

Options: settings.

Constructs: LazyFrame.collect, Config.set_engine_affinity.

Settings: {"POLARS_AUTO_STREAMING": "unset", "POLARS_ENGINE_AFFINITY": "auto", "POLARS_FORCE_STREAMING": "unset", "POLARS_MAX_THREADS": "the host's core count", "engine": ["in-memory", "streaming"]}.

Provenance:

polars-exec-platform: Platform dependence: undocumented

Engine: polars py-2.1.0; area: execution; status: open; evidence: probe.

Verdict: elementary functions use Rust’s f64 methods, so the platform’s libm; integer powers below 10 by repeated multiplication

The Polars docs say nothing about platform-dependent results.

Source at py-2.1.0: the trigonometric, hyperbolic, exp, log and pow kernels call num_traits::Float methods (v.sin(), v.exp(), v.log(base), Pow::pow), which for f64 are Rust’s standard methods and lower to the platform’s libm. The libm crate is a dependency of polars-ops but only for scalbn in round, which is exact. So the last bits of the elementary functions may differ between macOS arm64 (probes) and Linux x86_64 (CI).

pow with a scalar exponent takes shortcuts: 1 returns the base, 0.5 calls sqrt (so (-inf) ** 0.5 is NaN and (-0.0) ** 0.5 is -0.0, where IEEE pow gives inf and +0), and an integral exponent from 2 to 9 multiplies the base by itself, which rounds after each product. A column exponent calls pow for every row. These are platform-independent but can differ from pow in the last bits.

Next round: rerun characterization/polars/probes.json on homelab and diff.

Platform: on Linux x86_64 a NaN that an operation creates is negative, on macOS arm64 positive. DuckDB 2.0.0 gives signbit(sin('inf'::DOUBLE)) and signbit('inf'::DOUBLE - 'inf'::DOUBLE) true and prints -nan on Linux, false and nan on macOS; Polars 2.1.0’s sin(inf) has the bits fff8000000000000 on Linux and 7ff8000000000000 on macOS, on both engines. The sign comes from the CPU’s default NaN (x86’s is negative) and libm. The reference has one NaN, so only nan_kind and anything that shows a NaN’s sign (signbit, text) depend on the platform.

No upstream test pins this; it may change between versions.

The reference: The reference treats the elementary functions as black boxes with stated laws, so a last-bit difference is allowed.

Options: platform.

Constructs: sin, pow, exp, log, sqrt.

Settings: {"engine": ["in-memory", "streaming"], "platform": ["macOS arm64", "Linux x86_64"]}.

Provenance:

The reference gives IEEE +inf with no domain error: nanCase for fpow (SubstraitLean/Safety.lean) raises only for a finite negative base to a finite non-integer exponent, and the -inf base is excluded; an infinity from the data passes through (docs/reference-semantics.md, the rows on creating NaN and ±∞). So a literal 0.5 exponent on Polars gives NaN where the reference gives +inf: the writer should not emit ** 0.5 for a base that may be -inf, or should guard it.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe Float64 ** Float64 in characterization/polars/probes-streaming.json row [-inf, 0.5]: NaN with a literal exponent (the 0.5 → sqrt shortcut), inf with a column exponent (pow), on the in-memory and the streaming engine alike - 2026-11-04, probe, by characterization pipeline platform run (homelab, Linux x86_64, glibc 2.39-0ubuntu8.9; scratchpad ulp_probe.py –n 3000 –seed 1, the same inputs as the macOS run): scratchpad platform_repros.py, run on both machines. Also: integer AVG of (2^53+1, 0, 0) is 3002399751580330.5 and BIGINT var_samp/var_pop of 2^53+1..3 are 6.0/4.0 on both platforms (no long double difference seen); tan(1.0391439199447632) is 1.7002787929535204 on macOS and the correctly rounded 1.7002787929535208 on Linux. - SELECT signbit(sin('inf'::DOUBLE)), sin('inf'::DOUBLE)::VARCHAR, signbit(('inf'::DOUBLE) - ('inf'::DOUBLE)) gives Linux x86_64: true, -nan, true; macOS arm64: false, nan, false - Polars: pl.DataFrame({‘x’: [inf, -inf, 1.039…] * 2000}).lazy().select(pl.col(‘x’).sin()).collect(engine=e) for in-memory and streaming: bits fff8000000000000 on Linux, 7ff8000000000000 on macOS - 2026-11-04, docs, by characterization pipeline reverse pass - https://doc.rust-lang.org/nightly/std/primitive.f64.html#method.sin Rust std: the precision of sin (and the other libm-backed methods) ‘varies by platform, Rust version’ and between invocations; it corresponds to the libc function. - https://github.com/pola-rs/polars/blob/py-2.1.0/rust-toolchain.toml#L1-L2 Polars py-2.1.0 builds with nightly-2026-09-01. - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/.github/workflows/test-python.yml#L168-L179 The suite runs per OS with float comparisons at rel_tol 1e-5; no upstream test pins platform-specific bits of sin, exp, log or pow.

polars-exec-predicate-pushdown: Strict casts raise on the rows left after predicate pushdown

Engine: polars py-2.1.0; area: execution; status: open; evidence: probe.

Verdict: which rows a raising expression sees depends on the optimizer; sverify turns predicate pushdown off

The cast docstring says strict raises if the cast is invalid on the rows after predicates are pushed down. So the rows a strict cast sees are those the optimized plan reaches, which the optimizer decides: predicate pushdown applies filters as early as possible, at the scan (lazy optimizations guide). Neither page says whether a pushed filter can move above or below a raising expression, or whether the conditions of one filter are evaluated in order.

sverify collects every translated plan with predicate_pushdown=False (sverify.polars_engine.collect), because pushdown would move its guards and filters below the filters before them, onto rows the plan never evaluates there. The other optimizations (simplify expressions, common subexpressions, slice pushdown) stay on.

Round 2 (source): with pushdown on, Polars classifies each conjunct (ExprPushdownGroup): strict casts of a non-literal, list.get/arr.get with null_on_oob=False, list.gather, replace_strict, decimal and truncating arithmetic and strict strptime are fallible; non-elementwise expressions are barriers. A filter(a & b) is split into one filter per infallible conjunct below one filter of the fallible ones, so the fallible ones see fewer rows. A fallible predicate is not pushed past a filter, a join or any node that removes rows (ir_removes_rows); it may pass a projection that keeps every row. A fallible projection (with_columns(x.cast(t))) does not stop other predicates from moving below it, so the cast then raises on fewer rows; POLARS_PUSHDOWN_OPT_MAINTAIN_ERRORS=1 keeps them above. The optimizer is shared by both engines; the streaming engine applies the optimized plan’s filters per morsel. Gap: the list is closed, so raising functions outside it count as pushable, for example str.to_integer(strict=True) (a FunctionOptions::elementwise() string function). Such a predicate can move below a filter onto rows the plan removes there. sverify’s predicate_pushdown=False avoids all of this; a probe of filter(s != 'x').filter(s.str.to_integer() == 1) with pushdown on goes to the next round.

Polars’s fallible list (is_fallible_top_level) holds strict casts and strict strptime among string parsers, so str.to_integer(strict=True) is pushed: filter(s != 'x').filter(s.str.to_integer() == 1) becomes one filter whose & parses every row and raises on the row the first filter removes, on both engines. With pushdown off it returns the row. PL-7 records it upstream.

The docs are silent on a raising expression other than a cast moving onto rows an earlier filter removed: a docs gap.

No upstream test pins a fallible function outside the fallible list; it may change between versions.

The reference: The reference evaluates an expression only on the rows its operator sees: a CASE branch on the rows that reach it, a filter’s condition on the rows below it, a join condition on the pairs it forms. Its issue analysis (Rel.HasIssue, SubstraitLean/Safety.lean; docs/how-it-works.md) trusts the engine’s evaluation order: CASE is lazy, AND/OR are not, and filter and join conditions may run before the filters below them. A plan verify reports error-free must run without error, so an engine that evaluates a raising expression on rows outside that model breaks the guarantee.

Options: evaluation_order.

Constructs: filter, cast, with_columns.

Settings: {"engine": ["in-memory", "streaming"], "predicate_pushdown": false}.

See: upstream ledger PL-7, ST-12.

Provenance:

Lead: the contradiction test shows a strict cast that would raise is dropped by FilterConstraintRule, which runs under simplify_expr, not under predicate_pushdown. sverify turns only predicate_pushdown off and keeps simplify_expression on, so a translated plan whose filter is a contradiction over a raising expression returns no rows where the reference raises. POLARS_PUSHDOWN_OPT_MAINTAIN_ERRORS=1 (or simplify_expression=False) keeps the error. SimplifyBooleanRule takes the same flag. To probe next round.

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/lazyframe/test_predicates.py#L1091-L1217 test_predicate_pushdown_split_pushable: filter(a == 1, a.cast(Int8) == 1) on 1, 999 places the strict cast’s filter above the pushed a == 1, so it returns the row 1; with POLARS_PUSHDOWN_OPT_MAINTAIN_ERRORS=1 it raises conversion from i64 to i8 failed. Not pushed past a join - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/lazyframe/test_predicates.py#L1007-L1076 test_predicate_pushdown_fallible_exprs_22284: a fallible filter stays above the filter before it; a filter after with_columns(str.to_datetime(strict=True)) moves below the projection, so the cast skips the bad row; with MAINTAIN_ERRORS it raises - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/lazyframe/test_predicates.py#L1722-L1737 test_filter_contradiction_fallible_error_handling: filter((cast_a > 0) & (cast_a <= 0)) with cast_a = a.cast(Int8, strict=True) on a = 1000 returns an empty frame (the contradiction is dropped and the cast never runs); with MAINTAIN_ERRORS it raises InvalidOperationError - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/optimizer/mod.rs#L298-L311 FilterConstraintRule and SimplifyBooleanRule run under simplify_expr, with maintain_errors from POLARS_PUSHDOWN_OPT_MAINTAIN_ERRORS, not under predicate_pushdown - 2026-11-04, manual-repro, by characterization pipeline round 4 (execution): Confirms the round-3 lead on both engines: FilterConstraintRule turns a contradiction such as (c > 0) & (c <= 0) into false and drops the filter, so a strict cast inside it that would raise never runs. It runs under simplify_expression, so sverify’s predicate_pushdown=False does not stop it; simplify_expression=False or POLARS_PUSHDOWN_OPT_MAINTAIN_ERRORS=1 keeps the error. A literal false conjunct does not drop the cast (the second control raises).

Informational: here the engine raises less than the reference, returning no rows where the reference raises. A verified plan already shows its expressions cannot raise on the rows they see, so dropping a raising branch cannot change its result; only a plan the checker leaves unverified could notice the difference.

Polars 2.1.0 (Python), macOS arm64. - c = pl.col('a').cast(pl.Int8, strict=True); pl.LazyFrame({'a': [1000]}).filter((c > 0) & (c <= 0)).collect(engine=e, optimizations=o) for e in ('in-memory', 'streaming') gives default flags: [] on both engines; predicate_pushdown=False: [] on both; predicate_pushdown=False, simplify_expression=False: InvalidOperationError (conversion from i64 to i8 failed) on both; with POLARS_PUSHDOWN_OPT_MAINTAIN_ERRORS=1 and predicate_pushdown=False: InvalidOperationError - controls with predicate_pushdown=False, streaming: filter(c > 0); filter(pl.lit(False) & (c > 0)) gives both raise InvalidOperationError - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/optimizer/mod.rs#L298-L311 FilterConstraintRule runs under simplify_expr - 2026-11-04, manual-repro, by characterization pipeline reverse pass: Added to the description: str.to_integer(strict=True) is missing from the fallible list (PL-7). - pl.LazyFrame({'s': ['1', 'x']}).filter(pl.col('s') != 'x').filter(pl.col('s').str.to_integer() == 1).collect(engine=e, optimizations=pl.QueryOptFlags(predicate_pushdown=pd)) for e in ('in-memory', 'streaming'), pd in (True, False); POLARS_IDEAL_MORSEL_SIZE=64 gives pushdown on: ComputeError 'strict integer parsing failed for 1 value(s): ["x"]' on both engines; off: [('1',)] on both; explain(): FILTER (col("s") != "x") & (col("s").str.to_integer([dyn int: 10]) == 1) (Polars 2.1.0, macOS arm64) - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-plan/src/plans/optimizer/predicate_pushdown/mod.rs#L266-L300 a Filter’s predicate joins the accumulated predicates and is pushed into the filter below unless pushdown_eligibility keeps it local; str.to_integer is not fallible, so it is pushed and merged with & - 2026-11-04, docs, by characterization pipeline reverse pass - https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.str.to_integer.html strict raises ParseError as ComputeError; no caveat that predicate pushdown can evaluate it on rows an earlier filter removed

polars-exec-row-order: Row order: groups, joins and unique keep no order unless asked

Engine: polars py-2.1.0; area: execution; status: open; evidence: probe.

Verdict: order only with maintain_order; a join’s default order may change between runs

group_by keeps the order of the groups consistent with the input only with maintain_order=True (which the streaming engine runs: its group_by adds a row index, takes its first value per group and sorts the groups by it); rows within a group always keep their order. A join’s maintain_order defaults to "none": the ordering might differ across Polars versions or between runs, and the docs say not to rely on an observed order.

Float sums depend on the summation tree: pairwise, Kahan when grouped, plain across chunks (polars-sum-float), so chunking and the thread count can change their last bits. The docs do not say so; sum only documents that small integers are cast to Int64.

Round 2 (source), streaming engine: a grouped sum is a NumSumReducer whose combine adds the partial sums; within a morsel a float column is summed pairwise (ChunkAgg::sum), per group row by row. The reducer is marked ORDER_INDEPENDENT, which the trait documents as “changing the result by at most floating-point rounding” when the values of a group are split into several states. The group-by sink builds pre-aggregates per pipeline and combines them partition by partition in the order they were produced, so the morsel distribution over threads decides the order of the float additions. Probes at POLARS_MAX_THREADS 1 and N go to the next round.

No upstream test pins the float sums’ variation with the thread count; it may change between versions.

The reference: The reference compares results as bags, so row order outside a sort is free; sum_order is open and allowed within the plan’s bound (polars-sum-float).

Options: parallel_nondeterminism.

Constructs: group_by, join, unique, sum.

Settings: {"POLARS_IDEAL_MORSEL_SIZE": ["100000 (default)", "64"], "POLARS_MAX_THREADS": "the host's core count", "POLARS_SORT_PARTITION_THRESHOLD_BYTES": ["33554432 (default)", "1024"], "engine": ["in-memory", "streaming"]}.

Provenance:

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-group-order in characterization/polars/probes-streaming.json - probe s-group-fsum-runs in characterization/polars/probes-streaming.json - probe s-fsum-runs in characterization/polars/probes-streaming.json - probe s-var-runs in characterization/polars/probes-streaming.json - 2026-11-04, probe, by characterization pipeline round 5 (probe_polars –diff, small morsels): Rerun on small inputs (3,000 or 5,000 rows) in a subprocess with POLARS_IDEAL_MORSEL_SIZE=64 and POLARS_SORT_PARTITION_THRESHOLD_BYTES=1024, which Polars reads at start-up: the same behaviour as the large inputs at the defaults (the unstable streaming sort, sums and var varying by run), so the probes stay cheap. The variables only move the thresholds; the large-input results at the defaults stand. - probe s-fsum-runs in characterization/polars/probes-streaming.json - probe s-group-fsum-runs in characterization/polars/probes-streaming.json - probe s-var-runs in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass: Corrected the description: it said maintain_order=True blocks the streaming engine. At py-2.1.0 the streaming group_by handles it (lower_group_by.rs#L930-L941, #L1190-L1205); the docs’ “blocks the possibility to run partitioned” is about the distributed engine. - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_group_by.rs#L930-L941 streaming group_by with maintain_order: row index added, first(row index) aggregated - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_group_by.rs#L1190-L1205 then the groups are sorted by it: maintain_order runs on the streaming engine - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-stream/src/physical_plan/lower_group_by.rs#L1279-L1289 only the sorted-keys fast path declines unsorted keys with maintain_order - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/aggregation/test_aggregations.py#L1036-L1057 test_agg_empty_sum_after_filter_14734: in-memory group order varies between runs without maintain_order - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming_sort.py#L638-L646 streaming group_by(maintain_order=True) equals in memory exactly - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/streaming/test_streaming.py#L199-L214 test_streaming_9776: maintain_order=True keeps first-occurrence group order

polars-exec-size-limits: Size limits: decimal precision 38; strings 4 GiB - 1 (panic past it); rows 2^32 - 1 by default

Engine: polars py-2.1.0; area: execution; status: open; evidence: engine_test.

Verdict: Decimal: at most 38 digits; a string or binary over u32::MAX bytes panics; the default runtime caps a frame at 2^32 - 1 rows

Decimal is a 128-bit type with at most 38 digits of precision (also the default when the precision is not given) and a non-negative scale. The page does not say what an overflow does; Polars’s tests pin that it raises ComputeError: overflow in decimal ….

Source at py-2.1.0: strings and binaries are binary views whose length is a u32. The builder asserts bytes.len() <= u32::MAX (push_value_into_buffer), so a longer value panics (a PanicException in Python) where it would be an error elsewhere. Lists are ListArray<i64>, with 64-bit offsets checked on push (Offsets::try_push returns an overflow error). Row indices are IdxSize, a u32 in the default polars-runtime-32 that pip install polars brings; polars[rt64] switches to u64 (bigidx). Past 2^32 - 1 rows the default runtime reports “Polars’ maximum length reached. Consider installing ‘polars[rt64]’.”, as an error at most sites and a panic at some (expect(LENGTH_LIMIT_MSG)).

Leads: py-polars/tests/unit/datatypes/test_decimal.py for decimal overflow (polars-decimal-arith covers arithmetic).

The docs are silent on the u32 string length and the u32 row limit of the default runtime: a docs gap.

No upstream test pins a string over u32::MAX bytes or a list offset overflow; it may change between versions.

The reference: The reference caps decimals at 38 digits and raises past them; it has no limit on string, blob or list lengths.

Options: size_limits.

Constructs: Decimal, String, Binary, List.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

polars-exec-streaming-fallback: Streaming engine: operations handed to the in-memory engine

Engine: polars py-2.1.0; area: execution; status: open; evidence: engine_test.

Verdict: per-node fallback for opaque UDFs, some joins, some group-bys, keep-last unique, median and implode, cumulative eval and non-elementwise functions

The streaming engine lowers the optimized plan node by node (lower_ir, lower_expr, lower_group_by in crates/polars-stream). What it cannot stream it wraps in a fallback node that sinks its whole input into a DataFrame and runs the in-memory engine on it: InMemoryMap, InMemoryJoin and ColumnarFunction, which the plan visualizer colours as “in-memory engine fallback”. The rest of the query keeps streaming; there is no whole-query fallback (polars-exec-engine).

Relations that fall back: a MapFunction that is not streamable (rechunk, a fast count, an opaque Python or Rust UDF not marked streamable, such as LazyFrame.map_batches); a join that is not an equi, semi, anti, cross or range join, an as-of join whose by keys are not known sorted the same way, a join with a non-equality condition other than a fused predicate, and any join with validate; unique(keep="last", maintain_order=True); a group-by with an apply, with rolling or dynamic options (when not handled by the sorted rolling node), over object columns, or whose aggregations the streaming group-by cannot split (more than one non-elementwise input to one aggregation, non-elementwise post-processing, all expressions input-independent); an anonymous scan.

Expressions that fall back, each computed over the full input in its own node: median and implode as plain aggregations, a cumulative eval, a window over that cannot be lowered to a streaming group-by, and every non-elementwise function or UDF without its own streaming rule (ColumnarFunction), such as map_batches without is_elementwise=True.

What changes in a fallback node: the in-memory engine’s algorithms (float sums, group_by row order) apply to that node’s materialized input, so a fact probed in memory holds there, and one probed streaming may not. Evaluation order is the same in both: both evaluate expressions with polars-expr’s physical expressions, so the masking of elementwise when/then branches (polars-exec-when-eager) applies in a fallback node too; a non-elementwise branch is computed over every row in either engine.

sverify’s Polars translation reaches several of these: map_batches without is_elementwise (_on_days, the measure results), LazyFrame.map_batches (window computation), implode in an ungrouped aggregation, and over. Probes of explain(engine="streaming") for each translated construct go to the next round.

The docs are silent on which constructs fall back and a fallback node’s results differing: a docs gap.

No upstream test pins the fallback of map_batches, LazyFrame.map_batches or implode; it may change between versions.

The reference: The reference has no engines. A profile’s facts hold for the engine they were probed on; sverify pins the streaming engine (#605), so a fact whose construct falls back must hold for the in-memory engine as well, and a probe must check which engine ran it.

Options: streaming_fallback.

Constructs: LazyFrame.collect, map_batches, join, join_asof, group_by, unique, median, over, cum_sum.

Settings: {"engine": "streaming"}.

Provenance:

polars-exec-strict-false: Non-raising forms: strict=False and wrap_numerical

Engine: polars py-2.1.0; area: execution; status: open; evidence: engine_test.

Verdict: strict=False gives NULL for an invalid cast; wrap_numerical wraps

cast(strict=False) gives NULL where the cast is invalid; wrap_numerical=True wraps overflowing numeric casts instead of marking them invalid (cast page). The probes confirm both for integer and double casts (polars-cast-int, polars-cast-double-to-int) and strict parsing for strings (polars-cast-string).

Polars has no general TRY(expr). String parsers (str.to_integer, str.strptime, str.to_date) take their own strict flag; which errors they turn into NULL, and whether a non-strict cast of nested types nulls the field or the row, goes to source and tests (py-polars/tests/unit/operations/test_cast.py: test_nested_strict_casts_failing, test_strict_cast_nested).

Round 2 (source): a strict cast is the non-strict cast followed by a check (handle_casting_failures) that raises when the output has more NULLs than the input; for nested targets the strictness is passed down to the children. So strict=False returns whatever the non-strict kernel produced: NULL for every value it cannot represent, with no classification of error kinds. Errors raised before the kernel (an unsupported pair of types, raised when the plan is typed) are not affected. CastOptions has three modes: Strict (the default), NonStrict and Overflowing (wrap_numerical=True).

The docs are silent on nested targets, where the field becomes NULL: a docs gap.

The reference: Substrait’s cast has failure_behavior: RETURN_NULL; the reference honours it on DuckDB only, and other engines refuse it until #271 (docs/reference-semantics.md).

Options: try_variants.

Constructs: cast, str.to_integer, str.strptime, str.to_date.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

polars-exec-when-eager: when/then/otherwise masks elementwise branches; &, | and coalesce evaluate every operand

Engine: polars py-2.1.0; area: execution; status: open; evidence: probe.

Verdict: an elementwise branch sees NULL on the rows its condition excludes; other branches, &, | and coalesce run on every row

The when docstring warns that Polars computes all expressions passed to when-then-otherwise in parallel and filters afterwards, so each expression must be valid on its own. The code at py-2.1.0 does better for most branches.

Masking (in memory and streaming): when a then or otherwise operand is elementwise and is not a bare column or literal, the planner records its input columns, and TernaryExpr evaluates it on a frame where those columns are NULL on the rows the condition does not send to it; a branch no row takes is skipped unless needed for broadcasting. The streaming engine keeps an elementwise expression whole in a select node, so the same code runs there. So pl.when(c).then(x.cast(t)) does not raise on rows c excludes; Polars’s tests check this on both engines, also for multi-chunk structs (#29799).

Still eager: a non-elementwise branch (aggregations, windows, map_batches without is_elementwise=True), which both engines compute on every row; every branch under group_by().agg or over(), where evaluate_on_groups runs all three operands with no masking; a branch whose inputs are only literals; and an elementwise branch that turns NULL into a raising value, such as x.fill_null('bad').cast(t).

coalesce is an ordinary function whose arguments are all evaluated on every row before it folds them, and & and | evaluate both sides on every row (in parallel). This is the operand order ST-12 records, and the soundness bug #604.

The raising expressions in Polars are strict casts (polars-cast-int, polars-cast-double-to-int, polars-cast-string), string parsing and temporal errors (polars-temporal-errors); integer division by zero gives NULL and integer overflow wraps (polars-int-divide, polars-int-arith).

To probe: common subexpression elimination descends into ternary branches outside group_by and may hoist a subexpression that occurs twice into a column computed on every row, which would undo the masking; and a strict cast in a then branch, a later coalesce argument and under group_by().agg, on the streaming engine.

The docs are stricter than the code, and Polars’s own tests pin the masking (test_when_then.py#L916-L960): PL-13 asks Polars to document it.

No upstream test pins an unmasked branch under group_by().agg or over; it may change between versions.

The reference: The reference evaluates an expression only on the rows its operator sees: a CASE branch on the rows that reach it, a filter’s condition on the rows below it, a join condition on the pairs it forms. Its issue analysis (Rel.HasIssue, SubstraitLean/Safety.lean; docs/how-it-works.md) trusts the engine’s evaluation order: CASE is lazy, AND/OR are not, and filter and join conditions may run before the filters below them. A plan verify reports error-free must run without error, so an engine that evaluates a raising expression on rows outside that model breaks the guarantee. sverify’s Polars translation would need its CASE written so that a raising branch only sees its rows, or the issue analysis to treat Polars’s CASE as eager.

Options: evaluation_order.

Constructs: when, then, otherwise, &, |, coalesce, cast.

Settings: {"engine": ["in-memory", "streaming"]}.

See: upstream ledger PL-6, PL-13, ST-12; issues #604.

Provenance:

Polars’s CI runs the unit tests by default, where a lazy collect() without engine runs the streaming engine (also once with POLARS_IDEAL_MORSEL_SIZE=4), and again with POLARS_ENGINE_AFFINITY=in-memory (.github/workflows/test-python.yml#L55-L78, #L168-L178); eager Series and DataFrame calls run in memory. Read at tag py-2.1.0 (commit 7f902bd3), not run. - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/functions/test_when_then.py#L916-L921 test_when_then_scalar_condition_masks_unselected_branch, engine in-memory and streaming: when(lit(False)).then(col('x').cast(Int64)).otherwise(0) over ‘bad’, ‘worse’ is 0, 0 without raising - 2026-11-04, probe, by characterization pipeline round 4 (streaming probes): Streaming agrees with memory: when(x == '1').then(x.cast(Int64)) does not raise on the masked rows; & and coalesce evaluate the strict cast on every row and raise on both engines (only the error’s counts differ, per morsel); a cast after a filter sees the filtered rows only. The CSE cases are in polars-exec-cse-hoist and PL-6.

python -m sverify_dev.probe_polars --diff --write runs each observation (STREAMING_OBSERVATIONS, python/tools/sverify_dev/probe_polars.py) with collect(engine='in-memory') and collect(engine='streaming') and records both results. Polars 2.1.0 (Python), macOS arm64. - probe s-when-masks-cast in characterization/polars/probes-streaming.json - probe s-and-eager-cast in characterization/polars/probes-streaming.json - probe s-coalesce-eager-cast in characterization/polars/probes-streaming.json - probe s-filter-then-cast in characterization/polars/probes-streaming.json - 2026-11-04, docs, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/src/polars/functions/whenthen.py#L48-L51 docs say every branch is computed on every row; the code masks elementwise branches (ternary.rs#L110-L203), and test_when_then.py#L916-L960 pins the masking on both engines: the docs are stricter than the code

frontend

polars-implicit-casts: Type promotion: the wider type; literals adopt the column’s

Engine: polars py-2.1.0; area: frontend; status: open; evidence: probe.

Verdict: supertypes; a literal that fits keeps the column’s type

Arithmetic on two columns promotes to the wider type (Int32 + Int64 is Int64, UInt64 + Int64 is Int128). A Python literal takes the column’s type where it fits (Int32 + 1 is Int32, and wraps there, polars-int-arith) and widens it where it does not (Int32 + 2⁴⁰ is Int64, Int32 + 1.5 is Float64). Numbers and strings do not mix (InvalidOperationError).

The docs are silent on the supertype rules and on how a literal takes a column’s type: a docs gap.

The reference: A Substrait call takes the types its plan declares; casts are explicit.

Options: implicit_casts.

Constructs: +, ==, lit.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The result types are the plan’s schema, the same on both engines.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe implicit-casts in characterization/polars/probes-streaming.json - 2026-11-04, source-reading, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/utils/supertype.rs#L246 (Int64, UInt64) => Int128 - https://github.com/pola-rs/polars/blob/py-2.1.0/crates/polars-core/src/utils/supertype.rs#L511-L571 dynamic literals: a float column keeps its type; an integer literal takes the smallest fitting type, then the supertype - 2026-11-04, engine-test, by characterization pipeline reverse pass - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/operations/arithmetic/test_arithmetic.py#L39-L112 test_arithmetic_supertype: Int32+Int64 → Int64, Int64+UInt64 → Int128 (lazy, both engines) - https://github.com/pola-rs/polars/blob/py-2.1.0/py-polars/tests/unit/datatypes/test_datatypes.py#L277-L325 dynamic literals: lit(2**40) → Int64; UInt32 * -2 → Int64

polars-literal-types: Literal types: Int32 up to 2³¹, then Int64, UInt64, Int128

Engine: polars py-2.1.0; area: frontend; status: no-finding; evidence: probe.

Verdict: the narrowest of Int32, Int64, UInt64, Int128

pl.lit(1) is Int32, 2⁴⁰ is Int64, 2⁶³ is UInt64, 2⁶⁴ is Int128; 1.5 is Float64, None is Null, a Decimal is Decimal(38, s).

The reference: Substrait literals carry their type; the frontends (the SQL lift, the Ibis producer) choose it.

Options: literal_types.

Constructs: lit.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

literal-types runs pl.select eagerly, in memory; s-literal-types collects the same literals from a lazy select on each engine and gives the same types.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe literal-types in characterization/polars/probes-streaming.json - probe s-literal-types in characterization/polars/probes-streaming.json

polars-type-names: Polars’s names of the types

Engine: polars py-2.1.0; area: frontend; status: no-finding; evidence: probe.

Verdict: Int8 … Int128, UInt*, Float32, Float64, String, Decimal(precision, scale), …

Polars names its types Int8, Int16, Int32, Int64, Int128, UInt8 … UInt64, Float32, Float64, Boolean, String, Binary, Date, Time, Datetime(time_unit, time_zone), Duration(time_unit), Decimal(precision, scale), List(T), Array(T, shape), Struct({...}), Null, Categorical.

The reference: typeof folds to DuckDB’s names whichever engine runs the plan (duckdb-typeof-names).

Options: type_names.

Constructs: dtype.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observation prints data types (str(pl.Int8), …), which no engine runs.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe type-names in characterization/polars/probes-streaming.json

io

polars-io: I/O: scans and sinks, object stores by storage_options, a Unity catalog client

Engine: polars py-2.1.0; area: io; status: no-finding; evidence: probe.

Verdict: Parquet, CSV, IPC, NDJSON, Delta, Iceberg; cloud via storage_options; pl.Catalog

Polars scans Parquet, CSV, IPC, NDJSON, lines, Delta, Iceberg, Lance and PyArrow datasets lazily (scan_*), sinks Parquet, CSV, IPC, NDJSON, Delta and Iceberg from a lazy query (sink_*), and writes those and Avro, Excel, JSON and databases from a frame (write_*).

Object stores (S3, GCS, Azure, HTTP) are reached through storage_options or a credential_provider on the scans. pl.Catalog is a client for Unity Catalog.

The reference: I/O is outside what a plan computes; sverify’s io modules read and write the formats.

Options: io_formats, object_stores, catalogs.

Constructs: scan_*, sink_*, write_*, Catalog.

Settings: {"engine": ["in-memory", "streaming"]}.

Provenance:

  • 2026-11-04, probe, by sverify_dev.probe_polars (session brave-borg-41f7cc): sverify_dev.probe_polars runs each expression on edge cases (0, ±1, ±2, 3, ±7, the bounds of the integer width; 0.5, ±2.5, 7, 1e308, ±inf, NaN; NULL), one row at a time, and compares each row with Lean’s outcome of the Substrait call under each candidate option (substrait-check --outcomes). Observations are Polars programs whose result is recorded as it is. Polars 2.1.0 (Python), Rust crates at tag py-2.1.0 (commit 7f902bd3), on macOS arm64.
  • 2026-11-04, probe, by characterization pipeline round 5 (streaming engine): Streaming gives what in memory gives on every probe and observation cited here.

The observation reads the Python API (dir, hasattr, pl.scan_parquet’s signature), which both engines share; sink_* runs on the streaming engine.

python -m sverify_dev.probe_polars --diff --write runs each probe one row at a time and on a 6000-row column with collect(engine='in-memory') and collect(engine='streaming'), and each observation twice: its df(...) frame’s select, with_columns, filter, sort, unique, join and group_by go through the streaming engine the second time (StreamingFrame); the s- observations collect a lazy query on each engine themselves. Polars 2.1.0 (Python), macOS arm64. - probe io-functions in characterization/polars/probes-streaming.json