-
Notifications
You must be signed in to change notification settings - Fork 506
CPAOT support for large version bubble, take #2 #7428
Conversation
As I don't see any way to revive the original PR I accidentally merged in and then reverted, I have created a new one. In the initial commit I have pushed the previous change. As the second commit I have added the CompilationModuleGroup change suggested by Simon - using separate sets for input assemblies vs. for the version bubble. This change modifies the existing --inputbubble CPAOT option to work the same as Crossgen /largeversionbubble i.e. the main module and all reference assemblies are considered to belong to the same version bubble. The gist of the change deals with encoding module overrides in signatures. I have implemented a new R2R header table ManifestMetadataTableNode and added logic for composing and emitting the extra AssemblyRef list. One challenging bit was that we need to emit all the signatures before emitting the ManifestMetadataTableNode because only the emission of the signatures shakes out all the required AssemblyRef's. I have added a simple loop to ImportSectionsTableNode / ImportSectionNode that traverses and emits all registered import cell signatures. For instance entrypoints, I didn't find any manner of encoding a module override for their signatures in the native hashtable so for now I added filtering that we're only storing those instance entrypoints that have the current input module as the reference module. One can theoretically imagine that a module may contain arbitrary generic instantiations spanning other modules when larger bubbles are on. While working on the change I also realized that the current way of basing signature contexts on input modules in CorInfoImpl.ReadyToRun was incorrect - when a method gets inlined, we still need the original signature context because that's what defines the ambient module used in the CoreCLR signature parser. I have deleted the field and instead I added a new method GetSignatureContext which currently returns ReadyToRunCoregenNodeFactory.InputModuleContext. Once we start implementing "single-file" (i.e. compiling multiple MSIL modules into a single PE), we'll change this to something more fine-grained, most likely based on some lookup on MethodBeingCompiled. I found out that part of the framework libraries were crashing in release build with large version bubble on due to the presence of duplicate symbols. I found out that CPAOT was still inconsistent w.r.t. means for formatting typesystem entities in AppendMangledName methods. I went over all these methods and unified them to use the name mangled for formatting types, methods and fields. As part of this cleanup I could subsequently remove the WriteTo methods on RuntimeDeterminedTypeHelper that are no longer needed. Rerunning the Pri#1 tests with my new support for issues.targets filtering out some annoying timing out GC tests I found out that several tests failed during CPAOT compilation in the manifest metadata table node. It turns out that the AssemblyRef metadata table is not always unique, in particular I hit duplication of the "mscorlib" reference in the table. This change makes CPAOT resilient towards this type of issues. Thanks Tomas
Based on PR discussion I have modified the R2R single assembly compilation module group to keep input assemblies separate from the version bubble module set. With this change, CPAOT dependency analysis no longer attempts to compose the code graph across all assemblies, not only those being actually compiled, while keeping the more relaxed inlining conditions in case of larger version bubble. Based on this additional delta I could finally remove the artificial filtering of methods in MethodEntryPointTableNode. To facilitate this clean separation, I have added two new methods to CompilationModuleGroup for the version bubble checks - VersionsWithType and VersionsWithMethodBody. I have subsequently audited all CompilationModuleGroup queries in the CPAOT compiler and I changed the checks semantically referring to the version bubble to use the new methods. Thanks Tomas
src/ILCompiler.ReadyToRun/src/Compiler/ReadyToRunSingleAssemblyCompilationModuleGroup.cs
Show resolved
Hide resolved
nattress
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM - I like the new module group API for version bubbling
src/ILCompiler/src/Program.cs
Outdated
| versionBubbleModules.Add(module); | ||
|
|
||
| } | ||
| catch (Exception ex) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the reasoning behind making a failure to resolve version bubble assemblies non-fatal? Intuitively, I would think if a path is mis-specified we'd want to die early.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As discussed offline, the problem is in the pre-existing script SimpleTest.targets that has no simple way of filtering out non-managed assemblies from the reference assembly set. In 2nd commit I have added a descriptive comment explaining this issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we catch the exception specific to that? I think we would still want to not silently move on for access denied, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, thanks, fixed in 4th commit.
When determining reference module for encoding type signatures, we're special-casing primitive types as they don't need any reference module and use special constants in the signature encoding scheme. I previously didn't realize that object and string need the same treatment. I have also added an explanatory comment to Program.cs as to why we're using a try block when opening reference modules. Thanks Tomas
As I haven't found out how to reopen my original PR after reverting it, I have prepared a new one. In the first commit I have pushed the original change rebased to the preparatory change moving CanInline
to CompilationModuleGroup. In the second commit I added logic for clean separation of input assembly set from the version bubble assembly set.
Earlier in CPAOT development, we used to use CompilationModuleGroup.ContainsType / ContainsMethodBody for checking whether a given typesystem element belongs to either of these sets. To facilitate the separation I have added two new methods on CompilationModuleGroup, VersionsWithType and VersionsWithMethodBody. I audited all calls to ContainsMethodBody & ContainsType in the CPAOT codebase and I modified those querying the version bubble to call VersionsWithType/MethodBody instead.
Thanks
Tomas