[dynamo] prevent recompilation limit exceeded on external_utils.wrap_inline #163503
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/163503
Note: Links to docs will display an error until the docs builds have been completed. ❌ 29 New Failures, 4 Cancelled JobsAs of commit b2d00f6 with merge base 93bc987 ( NEW FAILURES - The following jobs have failed:
CANCELLED JOBS - The following jobs were cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
williamwen42
added
the
keep-going
…ine" This will prevent recompilations due to wrap_inline's guard on `fn` from hitting the recompilation limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
StrongerXi
reviewed
Sep 23, 2025
| if fn not in _wrap_inline_cache: | ||
| _wrap_inline_cache[fn] = deepcopy_code(inner.__code__) # type: ignore[attr-defined] | ||
|
|
||
| inner.__code__ = _wrap_inline_cache[fn] # type: ignore[attr-defined] |
Contributor
There was a problem hiding this comment.
Checking my understanding:
Given different functions f1 and f2, your PR makes wrap_inline(f1) and wrap_inline(f2) have different code objects. Wouldn't that trigger recompilation (as opposed to "don't recompile" as you documented)?
On the other hand, given the same f, existing code (without your PR) already have inner.__code__ being the same for multiple wrap_inline(f) calls, so there'll be no recompilation anyways, e.g.,
import torch
def f(x):
return x + 1
f1 = torch._dynamo.external_utils.wrap_inline(f)
f2 = torch._dynamo.external_utils.wrap_inline(f)
f1 = torch.compile(f1, backend="eager")
f2 = torch.compile(f2, backend="eager")
f1(torch.rand(10))
f2(torch.rand(10))Could you create a minimal example to show what exactly is causing unwanted recompilation over wrap_inline(...)?
Contributor
There was a problem hiding this comment.
Given different functions f1 and f2, your PR makes wrap_inline(f1) and wrap_inline(f2) have different code objects. Wouldn't that trigger recompilation (as opposed to "don't recompile" as you documented)?
I think I got it, by returning a new code object, you are changing recompilations into compilation over different frames (i.e., code objects).
So this is mainly for compiling a wrap_inline-ed function at top level (or after graph breaks, however that might get triggered). But you still want to treat different instances of wrap_inline(f)s (with same input function object) as the same frame for compilation (basically my example), thus the cache.
Curious, why are we hitting this case now?
Member
Author
There was a problem hiding this comment.
Minimal example is in the added unit test.
I'm hitting this case now because my wrapped nested graph break tests add an additional frame to each top-level compile call through wrap_inline - but then this causes recompilation limits to be exceeded. I then figured that this problem could happen more generally with wrap_inline - if we compile enough top level functions that are wrap_inline'd, then we would suddenly start hitting the recompile limit.
The current problem with this PR is that we create a new wrapper with a new code object for each instance of the function, which is problematic for two instances of the function with the same code object. This PR would then compile the inner function twice.
Contributor
There was a problem hiding this comment.
I'm hitting this case now because my wrapped nested graph break tests add an additional frame to each top-level compile call through wrap_inline
Makes sense, I assume this is only for the tests, i.e., not fundamental to nested graph break use cases in the wild? If so then I think it's totally okay to lazy evaluate the "different func objects but same code objects" problem.
Member
Author
There was a problem hiding this comment.
Yeah, this is not fundamental to nested graph breaks.
williamwen42
changed the title
StrongerXi
approved these changes
Sep 23, 2025
Contributor
StrongerXi
left a comment
StrongerXi
left a comment
There was a problem hiding this comment.
I'd like to understand why we are hitting this problem for wrap_inline, etc. now. Because in theory this is a dynamo problem for all non-inplace function wrappers.
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
mlazos
approved these changes
Sep 30, 2025
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
…utils.wrap_inline" This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on `fn`. If we call wrap_inline on enough different functions, we would hit the recompile limit. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
This will prevent recompilations (i.e. logs to TORCH_LOGS="recompiles") due to wrap_inline's guard on
fn. If we call wrap_inline on enough different functions, we would hit the recompile limit.cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @Lucaskabela