🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166197

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 48 Pending

As of commit e01d4a7 with merge base a2da693 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Converting to draft to find the CI jobs I need to disable

Wait, does pyrefly have complete coverage vs MYPY? I can understanding defaulting to pyrefly for quick on commit CI checks and using mypy for merging, but this seems premature to switch to pyrefly as the only supported type checker.

@Skylion007 we now cover the same files that were specified in both the mypy and mypy strict files (With the execption of ~5 files that I will be enabling in the coming days). Having two type checkers is hard to maintain, and users are finding it noisy.
Are there any files you've found that are not covered?

@Skylion007 what criteria would you like to see before switching it over?

@Skylion007 we now cover the same files that were specified in both the mypy and mypy strict files (With the execption of ~5 files that I will be enabling in the coming days). Having two type checkers is hard to maintain, and users are finding it noisy. Are there any files you've found that are not covered?

I meant more in type rule coverage rather than lint rule coverage. Do we have any metrics on the type of lints supported for pyrefly vs mypy? I would hate to accidentally introduce errors that would get flagged by downstream users of torch who use mypy for a donwstream library, especially given it's marketshare.

The biggest issue to me is that pyrefly does not have rule specific ignore types that mypy supports. Like you may have multiple typing errors on one line and because of one known issue, you can only silence the entire line even if another typing error occurs on the same line. This is especially common with function calls for instance.

@Skylion007 We have support for error specific suppressions, same as Mypy. I put together a demo for you here

The suppressions i've been adding to the codebase all have a specific error attached. When it comes to conformance of the type specification, you can find a comparision here https://htmlpreview.github.io/?https://github.com/python/typing/blob/main/conformance/results/results.html

While there are a few bugs we are actively working on fixing, we do have a higher conformance rate than mypy does currently. As an added bonus, we are fast enough to run over all files both locally and in CI. This means users can get a more consistent type check.

Once we've moved to Pyrefly, I am planning on helping improve the number/quality of type annotations in the codebase, which should help downstream users regardless of the type checker they are using.

Are there any other concerns I can help address?

For additional clarity, we are actively tracking and working on improving Pyrefly for Pytorch: https://github.com/facebook/pyrefly/issues?q=is%3Aissue%20state%3Aopen%20label%3Apytorch

If there's any other specific issues I can address, I'll happily bring them up with the team and prioritize them to make sure they're included in our next release :)

@Skylion007 I just realized the tool to auto add suppressions is adding the wrong syntax, which is why the specific suppression confusion is happening. I'll put up some PR's to remedy that before I take this out of draft mode.

thanks for reporting that concern!

@Skylion007 circling back to my previous question here - I've updated all of the pyrefly suppression comments to be much more specific and am adding a warning for users whose setups are missing stub files.

Are there any other metrics or features you would like to see before we switch to pyrefly as a typechecker?

@pytorchbot merge -f "avoid more problems creeping in"

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

wrote my concerns about dropping mypy support fully here: #166883 (comment)