Speed up make_simplified_union for Literal[string]s (issue #9169)#9192
Merged
JukkaL merged 4 commits intopython:masterfrom Jul 31, 2020
Merged
Speed up make_simplified_union for Literal[string]s (issue #9169)#9192JukkaL merged 4 commits intopython:masterfrom
JukkaL merged 4 commits intopython:masterfrom
Conversation
Contributor
Author
|
travis has actually passed, it just doesn't show up like that on github for whatever reason |
JukkaL
approved these changes
Jul 31, 2020
Collaborator
JukkaL
left a comment
JukkaL
left a comment
There was a problem hiding this comment.
Thanks for the performance fix!
This looks like a reasonable workaround for the immediate issue. A more general fix could split the items into literal and non-literal types, and manipulate the literal items using the more efficient algorithm, but also efficiently support unions containing non-literal types.
Changing mypy so that the simplification happens less often would definitely be nice, but it will be non-trivial to implement.
Contributor
Author
mypy used to be like that, but 25b1c4d broke it |
huguesb
pushed a commit
to huguesb/mypy
that referenced
this pull request
Sep 1, 2020
In PR python#9192 a fast path was created to address the slowness reported in issue python#9169 wherein large Union or literal types would dramatically slow down typechecking. It is desirable to extend this fast path to cover Enum types, as these can also leverage the O(n) set-based fast path instead of the O(n**2) fallback. This is seen to bring down the typechecking of a single fairly simple chain of `if` statements operating on a large enum (~3k members) from ~40min to 12s in real-world code! Note that the timing is taken from a pure-python run of mypy, as opposed to a compiled version.
huguesb
pushed a commit
to huguesb/mypy
that referenced
this pull request
Sep 1, 2020
In PR python#9192 a fast path was created to address the slowness reported in issue python#9169 wherein large Union or literal types would dramatically slow down typechecking. It is desirable to extend this fast path to cover Enum types, as these can also leverage the O(n) set-based fast path instead of the O(n**2) fallback. This is seen to bring down the typechecking of a single fairly simple chain of `if` statements operating on a large enum (~3k members) from ~40min to 12s in real-world code! Note that the timing is taken from a pure-python run of mypy, as opposed to a compiled version.
huguesb
pushed a commit
to huguesb/mypy
that referenced
this pull request
Sep 2, 2020
In PR python#9192 a fast path was created to address the slowness reported in issue python#9169 wherein large Union or literal types would dramatically slow down typechecking. It is desirable to extend this fast path to cover Enum types, as these can also leverage the O(n) set-based fast path instead of the O(n**2) fallback. This is seen to bring down the typechecking of a single fairly simple chain of `if` statements operating on a large enum (~3k members) from ~40min to 12s in real-world code! Note that the timing is taken from a pure-python run of mypy, as opposed to a compiled version.
huguesb
pushed a commit
to huguesb/mypy
that referenced
this pull request
Sep 2, 2020
In PR python#9192 a fast path was created to address the slowness reported in issue python#9169 wherein large Union or literal types would dramatically slow down typechecking. It is desirable to extend this fast path to cover Enum types, as these can also leverage the O(n) set-based fast path instead of the O(n**2) fallback. This is seen to bring down the typechecking of a single fairly simple chain of `if` statements operating on a large enum (~3k members) from ~40min to 12s in real-world code! Note that the timing is taken from a pure-python run of mypy, as opposed to a compiled version.
huguesb
pushed a commit
to huguesb/mypy
that referenced
this pull request
Sep 2, 2020
In PR python#9192 a fast path was created to address the slowness reported in issue python#9169 wherein large Union or literal types would dramatically slow down typechecking. It is desirable to extend this fast path to cover Enum types, as these can also leverage the O(n) set-based fast path instead of the O(n**2) fallback. This is seen to bring down the typechecking of a single fairly simple chain of `if` statements operating on a large enum (~3k members) from ~40min to 12s in real-world code! Note that the timing is taken from a pure-python run of mypy, as opposed to a compiled version.
huguesb
pushed a commit
to huguesb/mypy
that referenced
this pull request
Sep 3, 2020
In PR python#9192 a fast path was created to address the slowness reported in issue python#9169 wherein large Union or literal types would dramatically slow down typechecking. It is desirable to extend this fast path to cover Enum types, as these can also leverage the O(n) set-based fast path instead of the O(n**2) fallback. This is seen to bring down the typechecking of a single fairly simple chain of `if` statements operating on a large enum (~3k members) from ~40min to 12s in real-world code! Note that the timing is taken from a pure-python run of mypy, as opposed to a compiled version.
huguesb
pushed a commit
to huguesb/mypy
that referenced
this pull request
Sep 3, 2020
In PR python#9192 a fast path was created to address the slowness reported in issue python#9169 wherein large Union or literal types would dramatically slow down typechecking. It is desirable to extend this fast path to cover Enum types, as these can also leverage the O(n) set-based fast path instead of the O(n**2) fallback. This is seen to bring down the typechecking of a single fairly simple chain of `if` statements operating on a large enum (~3k members) from ~40min to 12s in real-world code! Note that the timing is taken from a pure-python run of mypy, as opposed to a compiled version.
msullivan
pushed a commit
that referenced
this pull request
Jul 7, 2021
In PR #9192 a fast path was created to address the slowness reported in issue #9169 wherein large Union or literal types would dramatically slow down typechecking. It is desirable to extend this fast path to cover Enum types, as these can also leverage the O(n) set-based fast path instead of the O(n**2) fallback. This is seen to bring down the typechecking of a single fairly simple chain of `if` statements operating on a large enum (~3k members) from ~40min to 12s in real-world code! Note that the timing is taken from a pure-python run of mypy, as opposed to a compiled version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
make_simplified_unionno longer runs for 0.5 seconds every time it's called with a Union containing 270 literals of strings.like I explained in #9169, this only fixes half of the problem and I'm not capable of fixing the other half. This function is still called 1098 times for the "reduced" example code, and IMO it should be called only once.