Skip to content

Comments

[ty] Add micro-benchmark for #711#18979

Merged
sharkdp merged 1 commit intomainfrom
david/benchmark-711
Jun 27, 2025
Merged

[ty] Add micro-benchmark for #711#18979
sharkdp merged 1 commit intomainfrom
david/benchmark-711

Conversation

@sharkdp
Copy link
Contributor

@sharkdp sharkdp commented Jun 27, 2025 •
edited
Loading

Summary

Add a benchmark for the problematic case in astral-sh/ty#711, which will potentially be solved in #18955

@sharkdp sharkdp added internal An internal refactor or improvement performance Potential performance improvement ty Multi-file analysis & type inference labels Jun 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2025

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

@codspeed-hq
Copy link

codspeed-hq bot commented Jun 27, 2025 •
edited
Loading

CodSpeed Instrumentation Performance Report

Merging #18979 will not alter performance

Comparing david/benchmark-711 (390c45b) with main (e5e3d99)

Summary

✅ 37 untouched benchmarks
🆕 2 new benchmarks
⁉️ 1 (👁 1) dropped benchmarks

Benchmarks breakdown

Benchmark BASE HEAD Change
🆕 ty_micro[complex_constrained_attributes_1] N/A 56.8 ms N/A
🆕 ty_micro[complex_constrained_attributes_2] N/A 6.3 s N/A
👁 ty_micro[many_attribute_assignments] 56.7 ms N/A N/A

@sharkdp sharkdp merged commit 3c18d85 into main Jun 27, 2025
37 checks passed
@sharkdp sharkdp deleted the david/benchmark-711 branch June 27, 2025 09:34
@AlexWaygood
Copy link
Member

AlexWaygood commented Jun 27, 2025 •
edited
Loading

Oof... I just ran the benchmarks locally to see if a change I was experimenting with had a performance impact, and this new benchmark takes over a minute to run (much longer than any other ty benchmark):

Benchmarking ty_micro[complex_constrained_attributes_2]: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 85.2s, or reduce sample count to 10.
ty_micro[complex_constrained_attributes_2]
                        time:   [808.20 ms 808.88 ms 809.60 ms]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

The command I ran locally was cargo bench -p ruff_benchmark --bench=ty. For comparison, the next-slowest benchmark was project/anyio, for which the report was:

Benchmarking project/anyio: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 14.5s, or reduce sample count to 30.
project/anyio           time:   [131.96 ms 132.08 ms 132.23 ms]
                        change: [−0.9888% −0.7683% −0.5652%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  2 (2.00%) high severe

Can we make this slightly less diabolical for now, so that it's possible to run benchmarks locally in a reasonable time?

@sharkdp
Copy link
Contributor Author

sharkdp commented Jun 27, 2025

Oof... I just ran the benchmarks locally to see if a change I was experimenting with had a performance impact, and this new benchmark takes over a minute to run (much longer than any other ty benchmark):

Oh, sorry! Will fix that later. In the meantime, you can use a filter argument or reduce the amount of samples (--sample-size 10), I think. The true fix will be #18955

dcreager added a commit that referenced this pull request Jun 27, 2025
@dcreager
Merge branch 'main' into dcreager/unpack-tuple
4b722d2 
* main:
  [ty] Add builtins to completions derived from scope (#18982)
  [ty] Don't add incorrect subdiagnostic for unresolved reference (#18487)
  [ty] Simplify `KnownClass::check_call()` and `KnownFunction::check_call()` (#18981)
  [ty] Add micro-benchmark for #711 (#18979)
  [`flake8-annotations`] Make `ANN401` example error out-of-the-box (#18974)
  [`flake8-async`] Make `ASYNC110` example error out-of-the-box (#18975)
  [pandas]: Fix issue on `non pandas` dataframe `in-place` usage (PD002) (#18963)
  [`pylint`] Fix `PLC0415` example (#18970)
  [ty] Add environment variable to dump Salsa memory usage stats (#18928)
  [`pylint`] Fix `PLW0108` autofix introducing a syntax error when the lambda's body contains an assignment expression (#18678)
  Bump 0.12.1 (#18969)
  [`FastAPI`] Add fix safety section to `FAST002` (#18940)
  [ty] Add regression test for leading tab mis-alignment in diagnostic rendering (#18965)
  [ty] Resolve python environment in `Options::to_program_settings` (#18960)
  [`ruff`] Fix false positives and negatives in `RUF010` (#18690)
  [ty] Fix rendering of long lines that are indented with tabs
  [ty] Add regression test for diagnostic rendering panic
  [ty] Move venv and conda env discovery to `SearchPath::from_settings` (#18938)
@sharkdp sharkdp mentioned this pull request Jun 27, 2025
Merged
@sharkdp
Copy link
Contributor Author

sharkdp commented Jun 27, 2025

Will fix that later

#19001

sharkdp added a commit that referenced this pull request Jun 27, 2025
@sharkdp
[ty] Reduce 'complex_constrained_attributes_2' runtime (#19001)
90cb0d3 
Re: #18979 (comment)

Each check increases the runtime by a factor of 3, so this should be an
order of magnitude faster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

internal An internal refactor or improvement performance Potential performance improvement ty Multi-file analysis & type inference

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sharkdp @AlexWaygood