You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cpython 3.13 installed with UV slow and not compiled with --enable-experimental-jit=yes-off` #535
The benchmark is very simple (the goal was to be able to observe an effect of the new JIT in CPython 3.13):
def short_calcul(n):
result = 0
foriin range(1, n+1):
result += i
return result
def long_calcul(num):
result = 0
foriin range(num):
result += short_calcul(i) - short_calcul(i)
return result
The results:
$ /usr/bin/python3 bench_loops_sum.py
3.11.2 (main, Sep 14 2024, 03:00:30) [GCC 12.2.0]
Number of long_calcul per second: 56.10
$ pypy bench_loops_sum.py
3.11.11 (b38de282cead, Feb 05 2025, 16:26:37)
[PyPy 7.3.18 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
Number of long_calcul per second: 1992.83
$ python bench_loops_sum.py
3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:10:22) [GCC 13.3.0]
Number of long_calcul per second: 51.12
$ PYTHON_JIT=1 python bench_loops_sum.py
3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:10:22) [GCC 13.3.0]
Number of long_calcul per second: 60.39
$ python bench_loops_sum.py
3.13.2 (main, Feb 12 2025, 14:51:17) [Clang 19.1.6 ]
Number of long_calcul per second: 40.91
$ python bench_loops_sum.py
3.14.0a5 (main, Feb 12 2025, 14:51:40) [Clang 19.1.6 ]
Number of long_calcul per second: 51.41
This is bad:
CPython 3.13 installed with UV is slow
CPython installed with UV are not compiled with --enable-experimental-jit=yes-off
FWIW, I don't see any slowdown on conda-forge 3.13 vs uv 3.14.0a5 on my machine (AMD64 Linux) on this benchmark:
(py313-conda-forge) ken@ken-Legion-5-Pro-16IAH7H:~/Documents/GitHub/cpython$ time python ./bm_calc.py
real 0m1.492s
user 0m1.488s
sys 0m0.003s
time uv run --python 3.14.0a5 python ./bm_calc.py
real 0m1.267s
user 0m1.255s
sys 0m0.011s
Oh wow, I do see a significant slowdown on 3.13 (compare the previous comment)
(cpython) ken@ken-Legion-5-Pro-16IAH7H:~/Documents/GitHub/cpython$ time uv run --python 3.13 python ./bm_calc.py
real 0m1.939s
user 0m1.897s
sys 0m0.025s
❯ uv run -p 3.13.2 bench.py
3.13.2 (main, Feb 12 2025, 14:59:08) [Clang 19.1.6 ]
Number of long_calcul per second: 54.34
❯ uv run -p 3.13.2 bench.py
3.13.2 (main, Mar 11 2025, 17:30:09) [Clang 20.1.0 ]
Number of long_calcul per second: 77.16
❯ /Users/zb/workspace/conda/my-env/bin/python ../uv/example/bench.py
3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:02:48) [Clang 18.1.8 ]
Number of long_calcul per second: 69.25
Benchmark 1: uv 3.13.2
Time (mean ± σ): 279.1 ms ± 2.8 ms [User: 275.3 ms, System: 2.8 ms]
Range (min … max): 274.9 ms … 283.4 ms 10 runs
Benchmark 2: conda-forge 3.13.2
Time (mean ± σ): 331.7 ms ± 6.3 ms [User: 328.3 ms, System: 2.5 ms]
Range (min … max): 324.8 ms … 347.7 ms 10 runs
Summary
uv 3.13.2 ran
1.19 ± 0.03 times faster than conda-forge 3.13.2
Yeah, it was the LLVM 20 upgrade with the backport for the LLVM 19 regression (conda-forge is on LLVM 18 on macOS and gcc on Linux) llvm/llvm-project#114990. This was merged here in #553
Note that on Linux, I still get a notable difference : Python 3.13 conda-forge compiled with GGC is approximately 35% faster than Python 3.13 UV - PBS compiled with Clang 20.
Activity
zanieb commentedon Feb 24, 2025
Thanks for the report. Do you know if Conda builds with any particular performance flags?
zanieb commentedon Feb 25, 2025
I did an actual performance run (Linux x86-64) and it looks like there's a significant difference here
I consider this fairly high priority, but I don't know what the source of the difference is.
zanieb commentedon Feb 25, 2025
Looking at https://github.com/conda-forge/python-feedstock/blob/main/recipe/build_base.sh and not seeing anything obvious.
zanieb commentedon Feb 25, 2025
@paugier What platform and architecture did you run your benchmarks on?
zanieb commentedon Feb 25, 2025
It looks like they might use -O3 instead of -O2
https://github.com/conda-forge/python-feedstock/blob/50a4e2d4203f05082fcbb93e14541180de3aa8ac/recipe/build_base.sh#L98C1-L104C3
edit: This is already the default in CPython
https://github.com/python/cpython/blob/94a7279fbdcfc2ccb6e7bb3c1cec863cce30a591/configure#L9600
zanieb commentedon Feb 25, 2025
Our v3 builds are a bit better, but that's not the bulk of it (geometric mean: 1.20x slower)
Fidget-Spinner commentedon Feb 25, 2025
FWIW, I don't see any slowdown on conda-forge 3.13 vs uv 3.14.0a5 on my machine (AMD64 Linux) on this benchmark:
In fact, the conda forge is significantly slower
zanieb commentedon Feb 25, 2025
3.14 <-> 3.13 doesn't feel like a fair comparison since we're using the tail calling interpreter.
Fidget-Spinner commentedon Feb 25, 2025
Oh wow, I do see a significant slowdown on 3.13 (compare the previous comment)
Set `enable-experimental-jit=yes-off` on Python 3.13 / Linux (#538)
paugier commentedon Mar 3, 2025
linux-x86_64-gnu. Sorry for this late answer. I was offline skying with the family.
zanieb commentedon Mar 11, 2025
Should be fixed in the next release
zanieb commentedon Mar 12, 2025
And using hyperfine
paugier commentedon Mar 12, 2025
That's awesome but could you also tell us what has been done to get this result?
zanieb commentedon Mar 16, 2025
Yeah, it was the LLVM 20 upgrade with the backport for the LLVM 19 regression (conda-forge is on LLVM 18 on macOS and gcc on Linux) llvm/llvm-project#114990. This was merged here in #553
This is briefly discussed in https://github.com/astral-sh/python-build-standalone/releases/tag/20250311
paugier commentedon Mar 20, 2025
Note that on Linux, I still get a notable difference : Python 3.13 conda-forge compiled with GGC is approximately 35% faster than Python 3.13 UV - PBS compiled with Clang 20.
Details of the results here and code here.
Good news: one can now enable the JIT with PBS Python 3.13 and 3.14! But the effect is tiny, much less than with conda-forge Python 3.13.
zanieb commentedon Mar 20, 2025
I did a comprehensive benchmark and I'm seeing about a 6% difference
This seems far more inline with what I'd expect. The statically linked libpython probably accounts for the additional difference.
zanieb commentedon Mar 20, 2025
I can reproduce some of the difference you're reporting for that microbenchmark
(but I also expect this to be related to libpython)