Skip to content

windows: use sys.platform == "win32" instead of is_windows#35640

Merged
tgamblin merged 1 commit intodevelopfrom
use-mypy-compatible-platform-check
Mar 5, 2023
Merged

windows: use sys.platform == "win32" instead of is_windows#35640
tgamblin merged 1 commit intodevelopfrom
use-mypy-compatible-platform-check

Conversation

@tgamblin
Copy link
Copy Markdown
Member

mypy only understands sys.platform == "win32", not indirect assignments of that value to things like is_windows. See python/mypy#9242 (comment).

If we don't use the accepted platform checks, mypy registers many Windows-only symbols as not present on Linux, when it should skip the checks for platform-specific code.

I'd like to get us to where we can enable check_untyped_defs in mypy (see #35639), so I've pulled this off of #35639 and made it a first PR. I realize this may look uglier to some people, but I don't see a good way around it as long as mypy is going to be dumb.

@spackbot-app spackbot-app bot added commands compilers core PR affects Spack core functionality fetching locking sbang tests General test capability(ies) utilities labels Feb 23, 2023
@tgamblin tgamblin requested review from alalazo and haampie February 23, 2023 10:14
@tgamblin tgamblin force-pushed the use-mypy-compatible-platform-check branch from 8fa6d6f to 03b6fc6 Compare February 23, 2023 10:16
@tgamblin tgamblin marked this pull request as ready for review February 23, 2023 10:18
@tgamblin tgamblin force-pushed the use-mypy-compatible-platform-check branch from 03b6fc6 to 20b3b49 Compare February 23, 2023 10:42
@alalazo
Copy link
Copy Markdown
Member

alalazo commented Feb 23, 2023

For reference #34779 is loosely related to the sys.platform =="win32" issue1, and orthogonal to the change here.

Footnotes

  1. It gathers platform specific changes together into their own "private" folders.

@scheibelp
Copy link
Copy Markdown
Member

I wanted the check to be abstracted into a method in case multiple things would be "equivalently-Windows" from a Spack point of view. No such case exists yet (although the thread you linked is an example of treating "cygwin" and "win32" the same), but that could be handled later. However, when it comes to that, we run into the problem again.

I don't think it's a major problem here but I'm generally concerned about mypy considerations negatively affecting the API. Is there some way to give mypy extra information about this?

Side note: my concern in #34779 (comment) does not apply to this PR.

@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Feb 23, 2023

There is apparently no way to get mypy to understand anything but the checks documented in PEP 484, which are intentionally "simple". The mypy developers don't seem particularly enthused about covering more cases, and someone would have to decide which ones to support, etc. See python/mypy#8547 (comment).

One issue in the current code is that is_windows is defined in lots of places, and it's not even used in some files, but that's not caught because it's assigned to a module-scope variable. So, when making this, I actually removed several unneeded instances of is_windows = sys.platform == "win32". They probably became unused with refactoring.

IMO this means we need to standardize on the sys.platform == "win32" idiom for now -- it's honestly pretty readable and sys.platform != "win32" might even be more readable than not is_windows (at least to me).

I'd say let's switch to using this, then think about whether more stuff can be consolidated in a checker-friendly way in #34779.

@johnwparent
Copy link
Copy Markdown
Contributor

@tgamblin would mypy better handle this case if there was a singular definition for is_windows somewhere and each module that actually needed it just imported it? IK that would at least throw an unused import if refactoring removed the need for the is_windows statement.
If that would solve the problem, I have a branch implementing something along those lines that's been languishing as I focus on other efforts, but I could spin up on that and PR pretty quickly. Personally I find a lot of sys.platform == "win32" clutters up a stanza of code pretty easily/quickly and I've gotten feedback from other community members to that effect as well.

@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Mar 3, 2023

@johnwparent: unfortunately no, there's no way to make mypy smarter about this.

I also find that sys.platform == "win32" is ugly, but OTOH I think that does a good job of making you want to find a way to not have to write it :). I don't think is_windows or an is_windows() method is necessarily the best way to hide this (and does it really hide it?). I think it would be better if we instead came up with some abstractions to use that worked well in both places, and buried the if inside those.

So, really, I think we have to do it this way unless mypy changes -- I don't see where making the relatively small number of lines here slightly prettier is worth giving up the ability to mypy check the whole codebase.

@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Mar 3, 2023

@scheibelp @johnwparent any additional thoughts?

@johnwparent
Copy link
Copy Markdown
Contributor

From what you're saying about mypy it doesn't really seem like there's another option, so not much of a surface to have thoughts on LOL.
Seems like our hands are tied, so LGTM.
I'll start mocking out ways to abstract logic like that in a way mypy can understand, do you have a good way to test what mypy can handle/docs? FWIW I think implementing the pathlib refactor in full will resolve a lot of these anyway.

scheibelp
scheibelp previously approved these changes Mar 3, 2023
johnwparent
johnwparent previously approved these changes Mar 3, 2023
@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Mar 4, 2023

I think it's really just the stuff in PEP 484 above, with no smart data flow analysis to track the value of the expression. I suspect it's bc mypy is mostly tracking types -- not values -- and they don't want to implement what you'd need to statically analyze the assignment to is_windows.

@tgamblin tgamblin dismissed stale reviews from johnwparent and scheibelp via 967ed95 March 4, 2023 00:08
@tgamblin tgamblin force-pushed the use-mypy-compatible-platform-check branch 2 times, most recently from 967ed95 to ae0b409 Compare March 4, 2023 00:41
`mypy` only understands `sys.platform == "win32"`, not indirect assignments of that
value to things like `is_windows`. If we don't use the accepted platform checks, `mypy`
registers many Windows-only symbols as not present on Linux, when it should skip the
checks for platform-specific code.
@tgamblin tgamblin force-pushed the use-mypy-compatible-platform-check branch from ae0b409 to 64e251a Compare March 4, 2023 00:44
@tgamblin tgamblin enabled auto-merge (squash) March 4, 2023 01:01
Copy link
Copy Markdown
Member

@scheibelp scheibelp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In all cases except two I see a direct replacement

In one case an extra is-windows check is added which is unnecessary (but does not hurt either)

)
except FileNotFoundError as e:
if e.winerror == 2:
if sys.platform == "win32" and e.winerror == 2:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this extra check is unnecessary because this object cannot be instantiated on non-windows systems

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s another unfortunate mypy thing. The winerror attribute doesn’t exist on non-windows systems but mypy sees it statically, so without this you get an error at typecheck time.

@tgamblin tgamblin disabled auto-merge March 5, 2023 15:57
@tgamblin tgamblin merged commit 42a0241 into develop Mar 5, 2023
@tgamblin tgamblin deleted the use-mypy-compatible-platform-check branch March 5, 2023 15:58
jmcarcell pushed a commit to key4hep/spack that referenced this pull request Apr 13, 2023
…#35640)

`mypy` only understands `sys.platform == "win32"`, not indirect assignments of that
value to things like `is_windows`. If we don't use the accepted platform checks, `mypy`
registers many Windows-only symbols as not present on Linux, when it should skip the
checks for platform-specific code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants