fix(polars): coerce to Decimal without a Float64 round trip - #2574

Fixes #2109.

Summary

Decimal.coerce in the polars engine casts the column to Float64 before casting it to the target decimal, so any value that does not survive a float round trip loses digits:

import polars as pl
import pandera.polars as pa

schema = pa.DataFrameSchema(
    {
        "a": pa.Column(pl.Decimal(11, 2)),
        "b": pa.Column(pl.Decimal(38, 2)),
        "c": pa.Column(pl.Decimal(38, 0)),
    },
    coerce=True,
)
df = pl.DataFrame(
    {"a": ["16809.76"], "b": ["12345678901234567.89"], "c": [123456789012345678]},
    schema={"a": pl.String, "b": pl.String, "c": pl.Int64},
)
schema.validate(df)
# a: 16809.75 on polars 1.20.0 and 1.33.1 (16809.76 on 1.42.1 and 2.0.0)
# b: 12345678901234567.68 on every polars version
# c: 123456789012345680 on every polars version

Polars can cast integers, decimals and strings to Decimal directly, so the float step is not needed for them.

Changes

• pandera/engines/polars_engine.py: Decimal.coerce now picks the cast per column dtype:
  • integers and decimals are cast directly to the target decimal;
  • strings are parsed directly, and only values the direct parse rejects fall back to the previous Float64 path. Scientific notation ("1e3") is the case that needs this: on polars 1.20.0 and 1.33.1 the direct string-to-decimal cast rejects it, while the float path accepts it;
  • floats, booleans and anything else keep the previous Float64 path (polars does not cast Boolean to Decimal directly).
• New tests in tests/polars/test_polars_dtypes.py:
  • test_coerce_decimal_keeps_precision: strings, large integers, a wider decimal, scientific notation, floats and booleans.
  • test_coerce_decimal_keeps_precision_in_schema: the schema-level example from the issue. It is marked xfail under the narwhals backend, which does not apply coerce=True, the same way as the existing coercion tests in tests/polars/test_polars_container.py.
  • test_try_coerce_decimal_invalid_string: a non-numeric string still raises ParserError with that value as the failure case.

Tests

• New tests on unmodified main: 5 failed on polars 1.20.0, 4 failed on polars 2.0.0 (where "16809.76" happens to round correctly through the float).
• pytest tests/polars with this change:
  • polars 1.20.0: 478 passed, 5 skipped
  • polars 1.33.1: 478 passed, 5 skipped
  • polars 1.42.1: 482 passed, 4 skipped
  • polars 1.42.1 with PANDERA_USE_NARWHALS_BACKEND=True: 410 passed, 1 skipped, 75 xfailed
  • polars 2.0.0: 480 passed, 4 skipped, 2 failed. The failures (test_dataframe_model_with_annotated_nested_types and test_polars_pydantic.py::test_conversion_types) also fail on unmodified main with polars 2.0.0 and are not related to this change.
• pytest tests/base: 19 passed.
• prek run --files pandera/engines/polars_engine.py tests/polars/test_polars_dtypes.py: all hooks pass (ruff, ruff format, mypy, pyupgrade, flynt, codespell).

Backward compatibility

No existing tests were changed. The user-visible change is that coercing integer, decimal and string columns to pl.Decimal now keeps the exact input value instead of the nearest float.

• String columns: the previous Float64 path is still evaluated as the fallback, so every string that coerced before still coerces, and strings that failed before still fail with the same polars error.
• Integer and decimal columns: values that coerced before still coerce. A few values that failed only because the float rounding pushed them past the target precision now coerce, for example 999999999999999999 into pl.Decimal(18, 0).
• Float and boolean columns go through the same casts as before.