Skip to content

Comments

Enable multithreading for pyright#13227

Merged
MichaReiser merged 1 commit intomainfrom
micha/benchmark-pyright-threads
Sep 3, 2024
Merged

Enable multithreading for pyright#13227
MichaReiser merged 1 commit intomainfrom
micha/benchmark-pyright-threads

Conversation

@MichaReiser
Copy link
Member

@MichaReiser MichaReiser commented Sep 3, 2024

Summary

Enable multithreading for pyright for a fairer comparison.

uv run benchmark  --min-runs=3 --project=black
black (cold)
Benchmark 1: knot
  Time (mean ± σ):      34.4 ms ±   1.7 ms    [User: 126.6 ms, System: 74.6 ms]
  Range (min … max):    31.8 ms …  39.2 ms    75 runs
 
  Warning: Ignoring non-zero exit code.
 
Benchmark 2: Pyright
  Time (mean ± σ):     245.0 ms ±   4.3 ms    [User: 156.2 ms, System: 32.6 ms]
  Range (min … max):   238.0 ms … 252.4 ms    12 runs
 
  Warning: Ignoring non-zero exit code.
 
Benchmark 3: mypy
  Time (mean ± σ):      3.270 s ±  0.042 s    [User: 3.128 s, System: 0.135 s]
  Range (min … max):    3.222 s …  3.302 s    3 runs
 
  Warning: Ignoring non-zero exit code.
 
Summary
  knot ran
    7.11 ± 0.37 times faster than Pyright
   94.97 ± 4.76 times faster than mypy
black (warm)
Benchmark 1: mypy
  Time (mean ± σ):     478.5 ms ±  18.8 ms    [User: 416.1 ms, System: 61.1 ms]
  Range (min … max):   459.0 ms … 507.8 ms    6 runs
 
  Warning: Ignoring non-zero exit code.

It improves pyright's performance from ~3s to 245ms

@MichaReiser MichaReiser added the ty Multi-file analysis & type inference label Sep 3, 2024
@MichaReiser MichaReiser enabled auto-merge (squash) September 3, 2024 11:23
AlexWaygood
AlexWaygood reviewed Sep 3, 2024
View reviewed changes
scripts/knot_benchmark/src/benchmark/cases.py
Comment on lines 136 to +137
"--venvpath",
"--threads",
Copy link
Member

@AlexWaygood AlexWaygood Sep 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I run uv run benchmark with -vv locally on this PR branch, it's clear that pyright is no longer resolving any third-party types installed into the virtual environment, because the --venvpath flag requires a value:

Suggested change
"--venvpath",
"--threads",
"--threads",
"--venvpath",

Copy link
Member

@AlexWaygood AlexWaygood Sep 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/cache.py:12:6 - error: Import "platformdirs" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/cache.py:14:6 - error: Import "_black_version" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/cache.py:144:59 - error: Expression of type "dict[str, tuple[float | int | str, ...]]" is incompatible with declared type "Dict[str, Tuple[float, int, str]]" (reportAssignmentType)
/private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/concurrency.py
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/concurrency.py:18:6 - warning: Import "mypy_extensions" could not be resolved from source (reportMissingModuleSource)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/concurrency.py:34:16 - error: Import "uvloop" could not be resolved (reportMissingImports)
/private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/files.py
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/files.py:20:6 - warning: Import "mypy_extensions" could not be resolved from source (reportMissingModuleSource)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/files.py:21:6 - error: Import "packaging.specifiers" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/files.py:22:6 - error: Import "packaging.version" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/files.py:23:6 - error: Import "pathspec" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/files.py:24:6 - error: Import "pathspec.patterns.gitwildmatch" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpm3ut7zl3/src/black/files.py:42:12 - warning: Import "colorama" could not be resolved from source (reportMissingModuleSource)

Copy link
Member

@AlexWaygood AlexWaygood Sep 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed #13228

@MichaReiser MichaReiser merged commit c2aac5f into main Sep 3, 2024
@MichaReiser MichaReiser deleted the micha/benchmark-pyright-threads branch September 3, 2024 11:24
@AlexWaygood AlexWaygood mentioned this pull request Sep 3, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AlexWaygood AlexWaygood AlexWaygood left review comments

Assignees

No one assigned

Labels

ty Multi-file analysis & type inference

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MichaReiser @AlexWaygood