You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
mypy-protobuf 3.7.0 causes enormous mypy processing time #707
After upgrading mypy-protobuf to version 3.7.0, we observed enormous memory consumption and processing time by mypy. Upon further analysis, we discovered that this issue is primarily caused by a _pb2_grpc.pyi file. It appears that each RPC method within a single service contributes to the increased processing time, and this becomes noticeable when there are 11 RPC methods.
Here's a simple example to illustrate this: mypy takes more than 5 minutes to process stubs of this proto file.
Apparently, this issue arises because of the introduction of generics in 3.7.0. Some combination of generics and typevars with constraints significantly impacts the processing of __init__ methods. Therefore, removing __init__ methods restores normal processing.
If this is dues to python/mypy#19622, then there is not much can be done from mypy side: I am quite sure any correct algorithm has exponential complexity for this.
If this is something else, you can open an issue on mypy tracker (with the generated stubs), and we can check if anything can be done performance-wise from mypy side.
Fwiw this isn't a problem on ty (which may or may not even be fully resolving these overloads, I haven't checked) or pyright (which is where I did most of the development on this feature).
I've identified another problem and created a pull request to mypy (python/mypy#20378). This change should enhance the processing of overloads. However, due to the issue mentioned (python/mypy#19622), we may still encounter performance issues if a service has numerous RPC methods.
Now, mypy takes 2 seconds to process the file with 13 RPCs, but for 15 RPCs, it takes about 30 seconds. Even if we optimize something else, it will never be enough to process dozens of constrained parameters.
There is 1 downside. Which is where I could use some input.
The __init__ method is no longer typed, and stubtest complains about it as well. I tried adding __init__ to the main class with a Union[grpc.aio.Channel, grpc.Channel] but that started down a rabbit hole of mypy not being able to type narrow on __new__. I'm still spending some time on that, but i'm not finding any clear answers. Pyright seemed to work fine and narrow things.
I'm going to keep exploring the mypy errors, but I think this might work better.
Hm, btw if the problem is specific to overloads, there is a chance python/mypy#18159 may fix this. (That PR is however quite old, and not yet ready to be merged.)
The time is spent almost exclusively in here. Because of the multiple generics and the cross product check the number of calls is astronomical.
I'm not sure a reduction from O(N^2V^2K) to O(N^2V^K) is actually enough, as when I started testing with 40-50 rpc's it probably needs to somehow be constant time instead. I would need to look at how pyright does it, unless someone else has looked at that already.
There are only 2 overloads per class, and each type var only has 2 bounds. But each typevar is actually another generic, so that may play into it as well, I have not fully groked the algorithm at this point.
I actually spent some time looking for a way to annotate the generated code to skip the check. Because it's generated code, we know it's correct, and it doesn't actually need to be validated.
In this case there are only 2 overloads, and neither accepts a type var, the argument just determines the value that the type vars will take.
OK, good, then it means it is not a pressing issue, but I will still look at this, I have couple ideas how to make this faster without sacrificing too much safety.
To be clear, the current state is unusable after ~13 RPC methods. Even with just the 2 overloads. Moving to the new overload should fix it, but even just the 2 with many type vars is too much.
Is there a way to annotate something for mypy to not do exhaustive checks on? Or is it all or nothing.
Hm, I don't think there is a way to skip this check with existing mypy versions. We can add some logic to skip this check altogether if overload-overlap error code is disabled, but then the mypy config is controlled by the user, so this is tricky.
I don't see any difference between generics and unions from a typing perspective. All RPC methods must be either synchronous or asynchronous. Neither unions nor generics can represent this requirement at the moment. Both represent that any of them can be synchronous or asynchronous, with eventual narrowing. Until we can write something like this:
Just confirming that we are hitting the same issue described here. I'm reverting to 3.6 for the time being. Our CICD pipelines cannot even run before getting killed with OOM issues. A colleague of mine had around 37 Gb of memory before his computer froze. No issues with 3.6.0.
Activity
aidandj commentedon Dec 6, 2025
What version of mypy
subnix commentedon Dec 7, 2025
@aidandj The latest one -
1.19.0. I also tried1.18.2and1.17.1, but it didn't help.ilevkivskyi commentedon Dec 8, 2025
If this is dues to python/mypy#19622, then there is not much can be done from mypy side: I am quite sure any correct algorithm has exponential complexity for this.
If this is something else, you can open an issue on mypy tracker (with the generated stubs), and we can check if anything can be done performance-wise from mypy side.
aidandj commentedon Dec 8, 2025
I'll work on reproducing and see what I can find.
Fwiw this isn't a problem on ty (which may or may not even be fully resolving these overloads, I haven't checked) or pyright (which is where I did most of the development on this feature).
fix: optimize overloaded signatures check
fix: optimize overloaded signatures check
subnix commentedon Dec 8, 2025
I've identified another problem and created a pull request to mypy (python/mypy#20378). This change should enhance the processing of overloads. However, due to the issue mentioned (python/mypy#19622), we may still encounter performance issues if a service has numerous RPC methods.
fix: optimize overloaded signatures check (#20378)
aidandj commentedon Dec 8, 2025
@WouldYouKindly does this fix your issue?
subnix commentedon Dec 8, 2025
Compiled mypy builds with the fix are available here: https://github.com/mypyc/mypy_mypyc-wheels/releases/tag/v1.20.0%2Bdev.88cf8e08f3e04f797d216869df0f8d5e030aa07c
Now, mypy takes 2 seconds to process the file with 13 RPCs, but for 15 RPCs, it takes about 30 seconds. Even if we optimize something else, it will never be enough to process dozens of constrained parameters.
aidandj commentedon Dec 8, 2025
Ok, I might have another solution. But I may need a 2nd set of eyes on it.
Instead of overriding init and using generics, I can override
__new__and return different stub classes.This passes all our existing tests, and I can run 100+ methods per service with no real change in performance.
There is 1 downside. Which is where I could use some input.
The
__init__method is no longer typed, and stubtest complains about it as well. I tried adding__init__to the main class with aUnion[grpc.aio.Channel, grpc.Channel]but that started down a rabbit hole of mypy not being able to type narrow on__new__. I'm still spending some time on that, but i'm not finding any clear answers. Pyright seemed to work fine and narrow things.I'm going to keep exploring the mypy errors, but I think this might work better.
aidandj commentedon Dec 8, 2025
#710
@subnix can you test this?
ilevkivskyi commentedon Dec 9, 2025
Hm, btw if the problem is specific to overloads, there is a chance python/mypy#18159 may fix this. (That PR is however quite old, and not yet ready to be merged.)
aidandj commentedon Dec 9, 2025
The time is spent almost exclusively in here. Because of the multiple generics and the cross product check the number of calls is astronomical.
I'm not sure a reduction from O(N^2V^2K) to O(N^2V^K) is actually enough, as when I started testing with 40-50 rpc's it probably needs to somehow be constant time instead. I would need to look at how pyright does it, unless someone else has looked at that already.
There are only 2 overloads per class, and each type var only has 2 bounds. But each typevar is actually another generic, so that may play into it as well, I have not fully groked the algorithm at this point.
ilevkivskyi commentedon Dec 9, 2025
There is a chance pyright simply does this the usual pyright way, like e.g. https://github.com/microsoft/pyright/blob/5f7ce347e8fbf80c826870b4a95395916cce3454/packages/pyright-internal/src/analyzer/checker.ts#L723-L734.
That said, I will take a closer look closer at this,
itertools.product()is kind of a dangerous thing to use.aidandj commentedon Dec 9, 2025
I actually spent some time looking for a way to annotate the generated code to skip the check. Because it's generated code, we know it's correct, and it doesn't actually need to be validated.
In this case there are only 2 overloads, and neither accepts a type var, the argument just determines the value that the type vars will take.
ilevkivskyi commentedon Dec 9, 2025
OK, good, then it means it is not a pressing issue, but I will still look at this, I have couple ideas how to make this faster without sacrificing too much safety.
aidandj commentedon Dec 9, 2025
I think the new overload will work, I want to play with it a bit longer, because I had used the init overload a lot before releasing.
Is there a way to annotate something for mypy to not do exhaustive checks on? Or is it all or nothing.
aidandj commentedon Dec 9, 2025
To be clear, the current state is unusable after ~13 RPC methods. Even with just the 2 overloads. Moving to the new overload should fix it, but even just the 2 with many type vars is too much.
ilevkivskyi commentedon Dec 9, 2025
Hm, I don't think there is a way to skip this check with existing mypy versions. We can add some logic to skip this check altogether if
overload-overlaperror code is disabled, but then the mypy config is controlled by the user, so this is tricky.aidandj commentedon Dec 9, 2025
Yeah, I was hoping something had been done already, but it wouldn't work in a user config.
subnix commentedon Dec 10, 2025
I don't see any difference between generics and unions from a typing perspective. All RPC methods must be either synchronous or asynchronous. Neither unions nor generics can represent this requirement at the moment. Both represent that any of them can be synchronous or asynchronous, with eventual narrowing. Until we can write something like this:
Given the challenges with analyzing generics, I would prefer unions.
wardVD commentedon Dec 16, 2025
Just confirming that we are hitting the same issue described here. I'm reverting to 3.6 for the time being. Our CICD pipelines cannot even run before getting killed with OOM issues. A colleague of mine had around 37 Gb of memory before his computer froze. No issues with
3.6.0.aidandj commentedon Dec 21, 2025
Closed in 4.0.0