Fix virtual environment details in `knot_benchmark` by AlexWaygood · Pull Request #13228 · astral-sh/ruff

AlexWaygood · 2024-09-03T11:39:44Z

Following #13227, pyright is looking for a virtual environment in a directory called '--threads', and no such directory exists. This PR fixes that, which slows pyright down again when checking black from 251.9ms to 1.751s on my machine

AlexWaygood · 2024-09-03T11:50:30Z

I think this is a fairer comparison between mypy/pyright/redknot... but I'm still seeing lots of errors regarding missing imports from both mypy and pyright if I run uv run benchmark --project black -vv locally.

Mypy:

src/blackd/middlewares.py:3:1: error: Cannot find implementation or library stub for module named "aiohttp.web_request"  [import-not-found]
src/blackd/middlewares.py:4:1: error: Cannot find implementation or library stub for module named "aiohttp.web_response"  [import-not-found]
src/black/output.py:11:1: error: Cannot find implementation or library stub for module named "click"  [import-not-found]
src/black/report.py:9:1: error: Cannot find implementation or library stub for module named "click"  [import-not-found]
src/black/cache.py:12:1: error: Cannot find implementation or library stub for module named "platformdirs"  [import-not-found]
src/black/files.py:21:1: error: Cannot find implementation or library stub for module named "packaging.specifiers"  [import-not-found]
src/black/files.py:22:1: error: Cannot find implementation or library stub for module named "packaging.version"  [import-not-found]
src/black/files.py:34:1: error: Cannot find implementation or library stub for module named "tomli"  [import-not-found]

Pyright:

  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:20:6 - warning: Import "mypy_extensions" could not be resolved from source (reportMissingModuleSource)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:21:6 - error: Import "packaging.specifiers" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:22:6 - error: Import "packaging.version" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:23:6 - error: Import "pathspec" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:24:6 - error: Import "pathspec.patterns.gitwildmatch" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:42:12 - warning: Import "colorama" could not be resolved from source (reportMissingModuleSource)

So something else may be going wrong somewhere...

MichaReiser · 2024-09-03T11:59:17Z

Thanks. I was just about to revert the commit because I don't see any performance improvements by just using the max number of threads (but e.g. 4 threads is faster on my machine for black).

So something else may be going wrong somewhere...

This is what I pointed out in the original PR summary and asked for help:

I'm a bit surprised that I e.g. see an error for Black. I would appreciate it if someone could verify that the diagnostics are reasonable.

#13026

AlexWaygood · 2024-09-03T12:01:32Z

This is what I pointed out in the original PR summary and asked for help:

Yeah, it's odd! I checked the output of the tools at the time and I don't remember seeing anything like this. I remember everything looking similar to what I'd expect at the time. I'm looking into it now.

Adding the --verbose flag to pyright's invocation in the script indicates that it is successfully finding the virtual environment:

Could not resolve source for 'mypy_extensions' in file '/private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/src/black/nodes.py'
  Looking in root directory of execution environment 'file:///private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92'
  Attempting to resolve using root path 'file:///private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92'
  Looking in extraPath 'file:///private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/src'
  Attempting to resolve using root path 'file:///private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/src'
  Finding python search paths
  Found path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib'; looking for site-packages
  Did not find 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/site-packages', so looking for python subdirectory
  Found path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/python3.12/site-packages'
  Did not find 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib64'
  Found path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/Lib'; looking for site-packages
  Did not find 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/Lib/site-packages', so looking for python subdirectory
  Found path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/Lib/python3.12/site-packages'
  Found the following 'site-packages' dirs
    file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/python3.12/site-packages
  Looking in python search path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/python3.12/site-packages'
  Attempting to resolve using root path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/python3.12/site-packages'

MichaReiser · 2024-09-03T12:04:12Z

Yeah it's not that all packages are missing, just some. The packages that are missing are also missing in the dependencies arrays (which I took from mypy-primer)

AlexWaygood · 2024-09-03T12:07:49Z

The packages that are missing are also missing in the dependencies arrays (which I took from mypy-primer)

In the logs I posted above, I see mypy errors about not being able to resolve click, and pyright errors about not being able to resolve packaging. Both of those should be py.typed packages we have installed into the virtual environment, no?

ruff/scripts/knot_benchmark/src/benchmark/projects.py

Lines 96 to 109 in c2aac5f

    
           Project( 
        
               name="black", 
        
               repository="https://github.com/psf/black", 
        
               revision="c20423249e9d8dfb8581eebbfc67a13984ee45e9", 
        
               include=["src"], 
        
               dependencies=[ 
        
                   "aiohttp", 
        
                   "click", 
        
                   "pathspec", 
        
                   "tomli", 
        
                   "platformdirs", 
        
                   "packaging", 
        
               ], 
        
           ),

MichaReiser · 2024-09-03T12:12:51Z

This is the only error that I get when running mypy over black

Benchmark 1: mypy
src/blackd/__init__.py:91:22: error: List item 0 has incompatible type "Callable[[Request, Callable[[Request], Awaitable[StreamResponse]]], Awaitable[StreamResponse]]"; expected "Middleware"  [list-item]
Warning: unused section(s) in pyproject.toml: module = ['tests.data.*']
Found 1 error in 1 file (checked 40 source files)

AlexWaygood · 2024-09-03T13:35:39Z

(We debugged this offline -- it seems like something changed in uv between v0.3 and v0.4? We're not sure what, but specifying --python for the uv pip install fixes things, and is more explicit anyway 😄)

codspeed-hq · 2024-09-03T13:36:33Z

CodSpeed Performance Report

Merging #13228 will degrade performances by 4.45%

_{Comparing alex/redknot-bench (61ee689) with main (c2aac5f)}

Summary

❌ 1 regressions
✅ 31 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`main`	`alex/redknot-bench`	Change
❌	`linter/default-rules[pydantic/types.py]`	1.8 ms	1.9 ms	-4.45%

Fix --venv-path for pyright in knot_benchmark

71656cc

AlexWaygood changed the title ~~Fix --venv-path for pyright in knot_benchmark~~ Fix --venvpath for pyright in knot_benchmark Sep 3, 2024

AlexWaygood mentioned this pull request Sep 3, 2024

Enable multithreading for pyright #13227

Merged

MichaReiser approved these changes Sep 3, 2024

View reviewed changes

MichaReiser added the ty Multi-file analysis & type inference label Sep 3, 2024

fix installation of dependencies into the venv

61ee689

MichaReiser approved these changes Sep 3, 2024

View reviewed changes

AlexWaygood changed the title ~~Fix --venvpath for pyright in knot_benchmark~~ Fix virtual environment details in knot_benchmark Sep 3, 2024

AlexWaygood merged commit 50c8ee5 into main Sep 3, 2024

AlexWaygood deleted the alex/redknot-bench branch September 3, 2024 13:35

This was referenced Sep 3, 2024

add a test for ty_benchmark that asserts we have roughly the number of expected errors astral-sh/ty#241

Closed

Make mypy pass on black in knot_benchmark #13235

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Fix virtual environment details in `knot_benchmark`#13228

Fix virtual environment details in `knot_benchmark`#13228
AlexWaygood merged 2 commits intomainfrom
alex/redknot-bench

AlexWaygood commented Sep 3, 2024

Uh oh!

AlexWaygood commented Sep 3, 2024

Uh oh!

MichaReiser commented Sep 3, 2024

Uh oh!

AlexWaygood commented Sep 3, 2024

Uh oh!

MichaReiser commented Sep 3, 2024

Uh oh!

AlexWaygood commented Sep 3, 2024

Uh oh!

MichaReiser commented Sep 3, 2024

Uh oh!

AlexWaygood commented Sep 3, 2024

Uh oh!

codspeed-hq bot commented Sep 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

AlexWaygood commented Sep 3, 2024

Uh oh!

AlexWaygood commented Sep 3, 2024

Uh oh!

MichaReiser commented Sep 3, 2024

Uh oh!

AlexWaygood commented Sep 3, 2024

Uh oh!

MichaReiser commented Sep 3, 2024

Uh oh!

AlexWaygood commented Sep 3, 2024

Uh oh!

MichaReiser commented Sep 3, 2024

Uh oh!

AlexWaygood commented Sep 3, 2024

Uh oh!

codspeed-hq bot commented Sep 3, 2024

CodSpeed Performance Report

Merging #13228 will degrade performances by 4.45%

Summary

Benchmarks breakdown

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants