Skip to content

Upgrade to MyPy 1.2.0#30687

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:upgrade-mypy-to-1.2.0
Apr 17, 2023
Merged

Upgrade to MyPy 1.2.0#30687
potiuk merged 1 commit into
apache:mainfrom
potiuk:upgrade-mypy-to-1.2.0

Conversation

@potiuk

@potiuk potiuk commented Apr 17, 2023

Copy link
Copy Markdown
Member

Upgrading to latest (released a week ago) MyPy in the hopes it will fix some more problem with attrs after upgrading new packages, but it seems that even the latest MyPy does not know about the new typing changes introduced in attrs (traditionally mypy has attrs plugin that injects appropriate typing but apparently it needs to catch up with those changes.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk
Upgrade to MyPy 1.2.0
a126572 
Upgrading to latest (released a week ago) MyPy in the hopes it
will fix some more problem with attrs after upgrading new packages,
but it seems that even the latest MyPy does not know about the
new typing changes introduced in attrs (traditionally mypy has
attrs plugin that injects appropriate typing but apparently it
needs to catch up with those changes.
@potiuk
potiuk requested review from mobuchowski and uranusjr April 17, 2023 19:37
@potiuk

potiuk commented Apr 17, 2023 •
edited
Loading

Copy link
Copy Markdown
Member Author

Seems that mypy in our main after some more (unrelated) upgraded packages started failing. They seem unrelated, but hey, it's MyPy and it's heuristing on when to start complaining about things that seemed to be ok earlier after some unrelated changes is something that we all got used to I think,

¯\_(ツ)_/¯

Here is the failure:

https://github.com/apache/airflow/actions/runs/4724060012/jobs/8381284205

Those were the change deps:

< # This constraints file was automatically generated on 2023-04-17T00:25:15Z
---
> # This constraints file was automatically generated on 2023-04-17T17:55:16Z
124c124
< cfn-lint==0.77.0
---
> cfn-lint==0.77.1
292c292
< limits==3.3.1
---
> limits==3.4.0
314c314
< msal==1.21.0
---
> msal==1.22.0
483c483
< soupsieve==2.4
---
> soupsieve==2.4.1
578c578
< zstandard==0.20.0
---
> zstandard==0.21.0

And after upgrading those, mypy provider checks resulted in:

airflow/providers/openlineage/utils/utils.py:345: error: Argument 1 to "asdict"
has incompatible type "str"; expected "AttrsInstance"  [arg-type]
    ...            for dict_key, subval in attrs.asdict(item, recurse=False)....
                                                        ^
airflow/providers/openlineage/utils/utils.py:345: error: Argument 1 to "asdict"
has incompatible type "Dict[Any, Any]"; expected "AttrsInstance"  [arg-type]
    ...            for dict_key, subval in attrs.asdict(item, recurse=False)....
                                                        ^
airflow/providers/openlineage/utils/utils.py:345: error: Argument 1 to "asdict"
has incompatible type "Tuple[Any, ...]"; expected "AttrsInstance"  [arg-type]
    ...            for dict_key, subval in attrs.asdict(item, recurse=False)....
                                                        ^
airflow/providers/openlineage/utils/utils.py:345: error: Argument 1 to "asdict"
has incompatible type "List[Any]"; expected "AttrsInstance"  [arg-type]
    ...            for dict_key, subval in attrs.asdict(item, recurse=False)....

I upgraded mypy toi 1.2.0 in the hopes it will understand attrs a bit better, but this resulted in few more errors related to attrs (and some other small things I fixed in Pydantic AIP-44 classes:

airflow/serialization/serde.py:160: note: Possible overload variants:
airflow/serialization/serde.py:160: note:     def asdict(obj: DataclassInstance) -> Dict[str, Any]
airflow/serialization/serde.py:160: note:     def [_T] asdict(obj: DataclassInstance, *, dict_factory: Callable[[List[Tuple[str, Any]]], _T]) -> _T
airflow/decorators/task_group.py:117: error: Argument 1 to "evolve" of
"_TaskGroupFactory[FParams, None]" has incompatible type
"_TaskGroupFactory[FParams, None]"; expected
"_TaskGroupFactory[FParams, FReturn]"  [arg-type]
            return attr.evolve(self, tg_kwargs={**self.tg_kwargs, **kwargs...
                               ^
airflow/decorators/task_group.py:117: error: Argument 1 to "evolve" of
"_TaskGroupFactory[FParams, DAGNode]" has incompatible type
"_TaskGroupFactory[FParams, DAGNode]"; expected
"_TaskGroupFactory[FParams, FReturn]"  [arg-type]
            return attr.evolve(self, tg_kwargs={**self.tg_kwargs, **kwargs...
                               ^
airflow/decorators/task_group.py:123: error: Argument 1 to "evolve" of
"_TaskGroupFactory[FParams, None]" has incompatible type
"_TaskGroupFactory[FParams, None]"; expected
"_TaskGroupFactory[FParams, FReturn]"  [arg-type]
            return attr.evolve(self, partial_kwargs=kwargs)
                               ^
airflow/decorators/task_group.py:123: error: Argument 1 to "evolve" of
"_TaskGroupFactory[FParams, DAGNode]" has incompatible type
"_TaskGroupFactory[FParams, DAGNode]"; expected
"_TaskGroupFactory[FParams, FReturn]"  [arg-type]
            return attr.evolve(self, partial_kwargs=kwargs)

Unless we have a better idea (@uranusjr ?) temporary ignoring those, seem to be the best way (And as a bonus we wil have latest-and-greatest MyPy.

@potiuk
potiuk merged commit cbde23e into apache:main Apr 17, 2023
@potiuk
potiuk deleted the upgrade-mypy-to-1.2.0 branch April 17, 2023 20:38
wookiist pushed a commit to wookiist/airflow that referenced this pull request Apr 19, 2023
@potiuk @jaewookoh-toss
Upgrade to MyPy 1.2.0 (apache#30687)
d714154 
Upgrading to latest (released a week ago) MyPy in the hopes it
will fix some more problem with attrs after upgrading new packages,
but it seems that even the latest MyPy does not know about the
new typing changes introduced in attrs (traditionally mypy has
attrs plugin that injects appropriate typing but apparently it
needs to catch up with those changes.
@ephraimbuddy ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label May 8, 2023
@ephraimbuddy ephraimbuddy added this to the Airflow 2.7.0 milestone May 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eladkal eladkal eladkal approved these changes
@kaxil kaxil Awaiting requested review from kaxil
@ashb ashb Awaiting requested review from ashb
@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin
@uranusjr uranusjr Awaiting requested review from uranusjr
@mobuchowski mobuchowski Awaiting requested review from mobuchowski

Assignees

No one assigned

Labels

area:dev-tools area:providers area:serialization type:misc/internal Changelog: Misc changes that should appear in change log

Projects

None yet

Milestone

Airflow 2.7.0

Development

Successfully merging this pull request may close these issues.

3 participants

@potiuk @eladkal @ephraimbuddy