Conversation
| INCREMENTAL = "incremental" | ||
| """Incremental check between two commits.""" |
There was a problem hiding this comment.
Not sure if we should remove this one again. I don't have an incremental benchmark yet but the idea would be to switch between two revisions.
MichaReiser
force-pushed
the
knot-mypy-pyright-benchmarks
branch
from
5694376 to
8da745b
Compare
|
AlexWaygood
left a comment
AlexWaygood
left a comment
There was a problem hiding this comment.
Some comments from reading through the code
AlexWaygood
left a comment
AlexWaygood
left a comment
There was a problem hiding this comment.
Basically this all looks good to me in terms of structure, nothing sticks out
| "--venvpath", | ||
| str( | ||
| venv.path.parent | ||
| ), # This is not the path to the venv folder, but the folder that contains the venv... |
There was a problem hiding this comment.
wow, bizarre. Do we also need to specify --venv=venv along with this argument, so that it knows which subdirectory inside the --venv-path directory the venv is in? https://github.com/microsoft/pyright/blob/main/docs/import-resolution.md#configuring-your-python-environment
There was a problem hiding this comment.
Yes we do. But there's no venv CLI option. So we have to create a config
for project in projects:
with tempfile.TemporaryDirectory() as cwd:
cwd = Path(cwd)
project.clone(cwd)
venv = Venv.create(cwd)
venv.install(project.dependencies)
# Set the `venv` config for pyright. Pyright only respects the `--venvpath`
# CLI option when `venv` is set in the configuration... 🤷♂️
with open(cwd / "pyrightconfig.json", "w") as f:
f.write(json.dumps(dict(venv=venv.name)))There was a problem hiding this comment.
pyright's config setup is so odd...
MichaReiser
force-pushed
the
knot-mypy-pyright-benchmarks
branch
from
8da745b to
9ccaa8a
Compare
MichaReiser
force-pushed
the
knot-mypy-pyright-benchmarks
branch
from
9ccaa8a to
ee2e088
Compare
Co-authored-by: Alex Waygood <[email protected]>
| "pip", | ||
| "install", | ||
| "--quiet", | ||
| *dependencies, |
There was a problem hiding this comment.
Worth adding --exclude-newer? While you pin the commit, you don't pin the deps
3 participants
Summary
This PR adds a basic benchmark runner for red knot. These benchmarks are different from codspeed in that they benchmark
entire projects. They are not micro-benchmarks.
The benchmarks aren't representive! Don't draw any conclusion from them today.
Red Knot only implements a tiny tiny subset of Mypy's and Pyright's functionality.
This work is inspired by mypy primer but it serves
a different purpose. We may want to add Red Knot to mypy primer when we have a more complete feature set (and a published binary)
Test Plan
Ran the benchmarks for each project and verified the diagnostics I saw:
I'm a bit surprised that I e.g. see an error for Black. I would appreciate it if someone could verify that the diagnostics are reasonable.