Repository navigation
Require pandera>=0.27 and un-xfail the dask series test - #1770
Thanatos9404 wants to merge 3 commits into
Conversation
Validating a dask series against a pandera SeriesSchema raised KeyError: 'dask' with pandera<0.23 and dask>=2025.1. Since dask-expr moved into dask, dask series live in `dask.dataframe.dask_expr`, and pandera before 0.23 only maps the old top-level `dask_expr` module name to its dask backend. SeriesSchema.validate looks the type up first, so the series path failed; DataFrameSchema maps over the partitions before any lookup, which is why dask dataframes were unaffected. The error also happens without Hamilton. pandera fixed the lookup in 0.23.0 (unionai-oss/pandera#1898). Set pandera>=0.23 for the pandera extra, the test and docs dependency groups, and the pandera integration test requirements. Remove the xfail from test_pandera_decorator_dask_series and add a test where only the last partition of a dask series breaks the schema. Fixes apache#1279
Dev-iL
left a comment
Dev-iL
left a comment
There was a problem hiding this comment.
LGTM. Pandera 0.23 is over 1.5 years old, so users are unlikely to be negatively impacted.
skrawcz
left a comment
skrawcz
left a comment
There was a problem hiding this comment.
AI-assisted preliminary review — findings must be fixed or technically refuted before human review.
The diagnosis of the Dask Series failure is correct, and the xfail removal plus last-partition regression test are focused. The proposed pandera>=0.23 floor is incomplete against current main, though. Current resolution permits Polars 2.1, and Pandera 0.23 through 0.25 crash while importing hamilton.plugins.h_pandera because their Polars engine calls pl.Categorical(ordering=...), which Polars 2 removed.
This was reproduced at the PR head: Pandera 0.23 + Dask 2025.1 + Polars 1.44 passes the integration file, while the same Pandera/Dask with Polars 2.1 fails during collection. In a sequential import check, 0.23–0.25 failed and 0.26 was the first tested Pandera release that imported with Polars 2.
Please raise or condition the floor to cover every allowed dependency combination and add a lower-bound compatibility check rather than validating only latest Pandera, or provide evidence that Polars 2 cannot occur in the supported environment.
pandera 0.23 to 0.25 fail to import pandera.typing.polars with polars 2 (pl.Categorical no longer takes `ordering`), and 0.26 fails pandas checks on Python 3.14. 0.27.0 is the first release that works with both, so it is the new floor for the pandera extra, the test and docs groups and the integration requirements. The pandas extra keeps pandera's own pandas and numpy floors, which moved out of its base requirements in 0.24. A new CI step reruns tests/integrations with pandera 0.27.0 after the run with the newest pandera, so the floor stays tested.
Thanatos9404
changed the title
|
Thanks, you're right. I merged main and raised every pin to 0.27: 0.23 to 0.25 fail to import |
Fixes #1279
The
KeyError: 'dask'comes from pandera rather than import order in Hamilton. Since dask 2025.1, dask-expr ships inside dask, so a dask series is adask.dataframe.dask_expr._collection.Series, and pandera before 0.23 maps only the old top-leveldask_exprmodule to its dask backend (fixed in unionai-oss/pandera#1898).DataFrameSchema.validatemaps over the partitions before that lookup, which is why only the series test broke.Changes
pandera[pandas]>=0.27for thepanderaextra and thedocsgroup, andpandera[dask,pandas]>=0.27in thetestgroup andtests/integrations/pandera/requirements.txt. 0.27 is the first release that works with everything main allows: 0.23 to 0.25 fail to importpandera.typing.polarswith polars 2, and 0.26.x fails pandas checks on Python 3.14 (0.27.0 added 3.14 support). Thepandasextra keeps pandera's own pandas and numpy floors.tests/integrationswith pandera 0.27.0 after the run with the newest pandera, so the floor stays tested.test_pandera_decorator_dask_seriesand added a test where only the last of two partitions of a dask series breaks the schema.How I tested this
tests/integrationswith pandera 0.27.0, dask 2026.8.0, pandas 3.1.1 and polars 2.1.0: 15 passed on Python 3.11, 3.12, 3.13 and 3.14. With 0.26.1 on 3.14, 3 tests fail. I couldn't run 3.15 locally, so that one rests on CI.prek run --filespasses on the changed files.Checklist