Skip to content

Commit 0055ffd

Browse files
EgorBoCopilot
andauthored
Don't require GDV targets to be inlineable (#132375)
GDV used to bail out when the target wasn't inlineable. A direct call still beats a virtual one, so keep the candidate and just don't inline it. `DOTNET_JitGuardedDevirtualizationRequireInlining=1` restores the old behavior. Class probes only. I know it doesn't match Andy's numbers, but many things have changed since then (e.g. CET), also, I assume we never tested it on arm64. So far it's a pure improvement on most kinds of scenarious (interface, virtual calls, etc.), see benchmarks below. We still might want to avoid doing it for virtual calls (esp for large target methods where the cost of the type check + direct call is a noise anyway) only on CoreCLR, but I propose we do it for everything and watch dotnet/performance reports. Should help NativeAOT the most, where `getExactClasses` gives exact devirtualization but targets often aren't inlineable (if at least one target of `getExactClasses` wasn't inlineable we used to bail on the whole thing). CoreCLR, default env, all targets `[NoInlining]` (so baseline does no GDV at all). Columns = how often the guessed type hits, ratio = new/base, lower is better. | Benchmark | Mono | Hit90 | Hit50 | Hit33 | Hit25 | |---|---|---|---|---|---| | `DirectControl` (noise floor) | 1.01 | 0.99 | 1.01 | 1.01 | 1.00 | | `Interface` | $\color{green}{0.72}$ | $\color{green}{0.48}$ | $\color{green}{0.45}$ | $\color{green}{0.64}$ | $\color{green}{0.56}$ | | `InterfaceShared` (`__Canon`) | $\color{green}{0.88}$ | $\color{green}{0.50}$ | $\color{green}{0.51}$ | $\color{green}{0.62}$ | $\color{green}{0.59}$ | | `InterfaceCostlyTarget` | $\color{green}{0.86}$ | $\color{green}{0.58}$ | $\color{green}{0.55}$ | $\color{green}{0.65}$ | $\color{green}{0.60}$ | | `Virtual` | $\color{green}{0.86}$ | $\color{green}{0.78}$ | $\color{green}{0.63}$ | $\color{green}{0.64}$ | $\color{green}{0.62}$ | | `VirtualShared` (`__Canon`) | $\color{red}{1.19}$ | $\color{green}{0.55}$ | $\color{green}{0.55}$ | $\color{green}{0.52}$ | $\color{green}{0.63}$ | | `InterfaceHugeTarget` (GDV in both) | 1.10 | 1.10 | 1.03 | 1.01 | 1.00 | Only regression is monomorphic shared-generic virtual - a well-predicted vtable call is already as cheap as compare+direct call. rt-sz report: MichalStrehovsky/rt-sz#246 smoke-test benchmarks: EgorBot/Benchmarks#489 (comment) --------- Co-authored-by: Copilot <[email protected]> Copilot-Session: 38498b8d-c116-4637-b069-d1d275634193
1 parent c2af89b commit 0055ffd

6 files changed

Lines changed: 163 additions & 85 deletions

File tree

‎src/coreclr/jit/compiler.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8239,6 +8239,8 @@ class Compiler
82398239
CORINFO_RESOLVED_TOKEN* pResolvedToken,
82408240
CORINFO_RESOLVED_TOKEN* pUnboxedResolvedToken);
82418241

8242+
bool canKeepNonInlineableGdvCandidate(GenTreeCall* call);
8243+
82428244
int getGDVMaxTypeChecks()
82438245
{
82448246
int typeChecks = JitConfig.JitGuardedDevirtualizationMaxTypeChecks();

‎src/coreclr/jit/importercalls.cpp‎

Lines changed: 101 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -2150,14 +2150,15 @@ GenTree* Compiler::impFixupCallStructReturn(GenTreeCall* call, CORINFO_CLASS_HAN
21502150

21512151
assert(retRegCount >= 2);
21522152

2153-
if (!call->CanTailCall() && !call->IsInlineCandidate())
2153+
if (!call->CanTailCall() && !call->IsInlineCandidate() && !call->IsGuardedDevirtualizationCandidate())
21542154
{
21552155
// Force a call returning multi-reg struct to be always of the IR form
21562156
// tmp = call
21572157
//
21582158
// No need to assign a multi-reg struct to a local var if:
21592159
// - It is a tail call or
2160-
// - The call is marked for in-lining later
2160+
// - The call is marked for in-lining later or
2161+
// - The call is a guarded devirtualization candidate (fate not yet known)
21612162
return impStoreMultiRegValueToVar(call, retClsHnd DEBUGARG(call->GetUnmanagedCallConv()));
21622163
}
21632164
return call;
@@ -9029,16 +9030,23 @@ void Compiler::addGuardedDevirtualizationCandidate(GenTreeCall* call,
90299030
// Gather some information for later. Note we actually allocate InlineCandidateInfo
90309031
// here, as the devirtualized half of this call will likely become an inline candidate.
90319032
//
9032-
InlineCandidateInfo* pInfo = new (this, CMK_Inlining) InlineCandidateInfo;
9033-
9034-
pInfo->guardedMethodHandle = methodHandle;
9035-
pInfo->guardedMethodInstParamLookup = {};
9036-
pInfo->guardedMethodResolvedToken = {};
9037-
pInfo->guardedMethodUnboxedResolvedToken = {};
9038-
pInfo->guardedClassHandle = classHandle;
9039-
pInfo->likelihood = likelihood;
9040-
pInfo->exactContextHandle = contextHandle;
9041-
pInfo->originalMethodHandle = originalMethodHandle;
9033+
// Value-initialize: the candidate may be expanded without ever going through
9034+
// impCheckCanInline, so every field has to be in a well-defined state.
9035+
//
9036+
InlineCandidateInfo* pInfo = new (this, CMK_Inlining) InlineCandidateInfo{};
9037+
9038+
pInfo->guardedMethodHandle = methodHandle;
9039+
pInfo->guardedClassHandle = classHandle;
9040+
pInfo->likelihood = likelihood;
9041+
pInfo->exactContextHandle = contextHandle;
9042+
pInfo->originalMethodHandle = originalMethodHandle;
9043+
pInfo->clsAttr = classAttr;
9044+
pInfo->methAttr = methodAttr;
9045+
pInfo->preexistingSpillTemp = BAD_VAR_NUM;
9046+
9047+
// The call only carries its IL offset in debug builds, and not this early.
9048+
//
9049+
pInfo->ilOffset = BAD_IL_OFFSET;
90429050

90439051
if (instParamLookup != nullptr)
90449052
{
@@ -9102,6 +9110,59 @@ void Compiler::impConvertToUserCallAndMarkForInlining(GenTreeCall* call)
91029110
}
91039111
}
91049112

9113+
//------------------------------------------------------------------------
9114+
// canKeepNonInlineableGdvCandidate: check if we can keep a guarded devirtualization
9115+
// candidate whose target we're not going to inline.
9116+
//
9117+
// Arguments:
9118+
// call -- guarded devirtualization candidate
9119+
//
9120+
// Return Value:
9121+
// true if the candidate can be kept just for the sake of devirtualization.
9122+
//
9123+
// Notes:
9124+
// A direct call is cheaper than a virtual/interface call, so we keep such candidates
9125+
// by default and only bail out where the expansion would be illegal or would cost
9126+
// us a better optimization.
9127+
//
9128+
bool Compiler::canKeepNonInlineableGdvCandidate(GenTreeCall* call)
9129+
{
9130+
assert(call->IsGuardedDevirtualizationCandidate());
9131+
9132+
if (JitConfig.JitGuardedDevirtualizationRequireInlining() != 0)
9133+
{
9134+
return false;
9135+
}
9136+
9137+
// Class-based GDV only for now; method-based (e.g. delegate) GDV still requires inlining.
9138+
//
9139+
if (call->GetGDVCandidateInfo(0)->guardedClassHandle == NO_CLASS_HANDLE)
9140+
{
9141+
return false;
9142+
}
9143+
9144+
// An explicit tail call has to stay a tail call, so don't perturb its shape.
9145+
//
9146+
// Implicit ones are fine: fgMorphPotentialTailCall can tail call out of the
9147+
// BBJ_ALWAYS blocks the expansion produces. That includes recursive ones -
9148+
// the tail-recursion-to-loop transform requires a non-virtual callee, so
9149+
// devirtualizing here is what makes it possible in the first place.
9150+
//
9151+
if (call->IsTailPrefixedCall())
9152+
{
9153+
return false;
9154+
}
9155+
9156+
// NextCallReturnAddress needs the call to stay exactly where it is.
9157+
//
9158+
if (info.compHasNextCallRetAddr)
9159+
{
9160+
return false;
9161+
}
9162+
9163+
return true;
9164+
}
9165+
91059166
//------------------------------------------------------------------------
91069167
// impMarkInlineCandidate: determine if this call can be subsequently inlined
91079168
//
@@ -9113,8 +9174,8 @@ void Compiler::impConvertToUserCallAndMarkForInlining(GenTreeCall* call)
91139174
//
91149175
// Notes:
91159176
// Mostly a wrapper for impMarkInlineCandidateHelper that also undoes
9116-
// guarded devirtualization for virtual calls where the method we'd
9117-
// devirtualize to cannot be inlined.
9177+
// guarded devirtualization when it's not worth doing (or not legal) once
9178+
// we know we can't inline the target.
91189179

91199180
void Compiler::impMarkInlineCandidate(GenTree* callNode,
91209181
CORINFO_CONTEXT_HANDLE exactContextHnd,
@@ -9136,19 +9197,37 @@ void Compiler::impMarkInlineCandidate(GenTree* callNode,
91369197
if (call->IsGuardedDevirtualizationCandidate())
91379198
{
91389199
assert(call->GetInlineCandidatesCount() > 0);
9200+
9201+
// We usually still want to devirtualize a target we can't inline,
9202+
// see canKeepNonInlineableGdvCandidate for the exceptions.
9203+
//
9204+
const bool keepNonInlineable = canKeepNonInlineableGdvCandidate(call);
9205+
91399206
for (uint8_t candidateId = 0; candidateId < call->GetInlineCandidatesCount(); candidateId++)
91409207
{
91419208
InlineResult inlineResult(this, call, nullptr, "impMarkInlineCandidate for GDV");
91429209

91439210
// Do the actual evaluation
91449211
impMarkInlineCandidateHelper(call, candidateId, exactContextHnd, callInfo, inlinersContext, &inlineResult);
9145-
// Ignore non-inlineable candidates
9146-
// TODO: Consider keeping them to just devirtualize without inlining, at least for interface
9147-
// calls on NativeAOT, but that requires more changes elsewhere too.
9212+
91489213
if (!inlineResult.IsCandidate())
91499214
{
9150-
call->RemoveGDVCandidateInfo(this, candidateId);
9151-
candidateId--;
9215+
if (!keepNonInlineable)
9216+
{
9217+
JITDUMP("Revoking guarded devirtualization candidate %u of call [%06u]: target method can't be "
9218+
"inlined\n",
9219+
candidateId, dspTreeID(call));
9220+
9221+
call->RemoveGDVCandidateInfo(this, candidateId);
9222+
candidateId--;
9223+
continue;
9224+
}
9225+
9226+
JITDUMP("Keeping GDV candidate %u of call [%06u] for devirtualization only: target can't be inlined\n",
9227+
candidateId, dspTreeID(call));
9228+
9229+
assert(!call->GetGDVCandidateInfo(candidateId)->isInlineable);
9230+
assert(call->GetGDVCandidateInfo(candidateId)->guardedClassHandle != NO_CLASS_HANDLE);
91529231
}
91539232
}
91549233

@@ -9166,25 +9245,6 @@ void Compiler::impMarkInlineCandidate(GenTree* callNode,
91669245
InlineResult inlineResult(this, call, nullptr, "impMarkInlineCandidate");
91679246
impMarkInlineCandidateHelper(call, 0, exactContextHnd, callInfo, inlinersContext, &inlineResult);
91689247
}
9169-
9170-
// If this call is an inline candidate or is not a guarded devirtualization
9171-
// candidate, we're done.
9172-
if (call->IsInlineCandidate() || !call->IsGuardedDevirtualizationCandidate())
9173-
{
9174-
return;
9175-
}
9176-
9177-
// If we can't inline the call we'd guardedly devirtualize to,
9178-
// we undo the guarded devirtualization, as the benefit from
9179-
// just guarded devirtualization alone is likely not worth the
9180-
// extra jit time and code size.
9181-
//
9182-
// TODO: it is possibly interesting to allow this, but requires
9183-
// fixes elsewhere too...
9184-
JITDUMP("Revoking guarded devirtualization candidacy for call [%06u]: target method can't be inlined\n",
9185-
dspTreeID(call));
9186-
9187-
call->ClearInlineInfo();
91889248
}
91899249

91909250
//------------------------------------------------------------------------
@@ -9473,6 +9533,8 @@ void Compiler::impMarkInlineCandidateHelper(GenTreeCall* call,
94739533
call->SetSingleInlineCandidateInfo(inlineCandidateInfo);
94749534
}
94759535

9536+
inlineCandidateInfo->isInlineable = true;
9537+
94769538
// Let the strategy know there's another candidate.
94779539
impInlineRoot()->m_inlineStrategy->NoteCandidate();
94789540

@@ -11042,17 +11104,7 @@ void Compiler::impCheckCanInline(GenTreeCall* call,
1104211104
}
1104311105
else
1104411106
{
11045-
pInfo = new (pParam->pThis, CMK_Inlining) InlineCandidateInfo;
11046-
11047-
// Null out bits we don't use when we're just inlining
11048-
//
11049-
pInfo->guardedClassHandle = nullptr;
11050-
pInfo->guardedMethodHandle = nullptr;
11051-
pInfo->guardedMethodInstParamLookup = {};
11052-
pInfo->guardedMethodResolvedToken = {};
11053-
pInfo->guardedMethodUnboxedResolvedToken = {};
11054-
pInfo->originalMethodHandle = nullptr;
11055-
pInfo->likelihood = 0;
11107+
pInfo = new (pParam->pThis, CMK_Inlining) InlineCandidateInfo{};
1105611108
}
1105711109

1105811110
pInfo->methInfo = methInfo;

‎src/coreclr/jit/indirectcalltransformer.cpp‎

Lines changed: 49 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -572,16 +572,6 @@ class IndirectCallTransformer
572572
JITDUMP("\n----------------\n\n*** %s contemplating [%06u] in " FMT_BB " \n", Name(),
573573
m_compiler->dspTreeID(m_origCall), m_currBlock->bbNum);
574574

575-
// We currently need inline candidate info to guarded devirt.
576-
//
577-
if (!m_origCall->IsInlineCandidate())
578-
{
579-
JITDUMP("*** %s Bailing on [%06u] -- not an inline candidate\n", Name(),
580-
m_compiler->dspTreeID(m_origCall));
581-
ClearFlag();
582-
return;
583-
}
584-
585575
m_likelihood = m_origCall->GetGDVCandidateInfo(0)->likelihood;
586576
assert((m_likelihood >= 0) && (m_likelihood <= 100));
587577
JITDUMP("Likelihood of correct guess is %u\n", m_likelihood);
@@ -848,10 +838,20 @@ class IndirectCallTransformer
848838
}
849839
else
850840
{
851-
// If there's a spill temp already associated with this inline candidate,
852-
// use that instead of allocating a new temp.
841+
// Only candidates that made it through impMarkInlineCandidateHelper get a
842+
// spill temp, so candidate 0 may not be the one carrying it.
853843
//
854-
m_returnTemp = inlineInfo->preexistingSpillTemp;
844+
m_returnTemp = BAD_VAR_NUM;
845+
for (uint8_t i = 0; i < m_origCall->GetInlineCandidatesCount(); i++)
846+
{
847+
const unsigned spillTemp = m_origCall->GetGDVCandidateInfo(i)->preexistingSpillTemp;
848+
if (spillTemp != BAD_VAR_NUM)
849+
{
850+
// Same call site, so all candidates must agree.
851+
assert((m_returnTemp == BAD_VAR_NUM) || (m_returnTemp == spillTemp));
852+
m_returnTemp = spillTemp;
853+
}
854+
}
855855

856856
if (m_returnTemp != BAD_VAR_NUM)
857857
{
@@ -958,22 +958,6 @@ class IndirectCallTransformer
958958
GenTreeCall* call = m_compiler->gtCloneCandidateCall(m_origCall);
959959
call->gtArgs.GetThisArg()->SetEarlyNode(m_compiler->gtNewLclvNode(thisTemp, TYP_REF));
960960

961-
// If the original call was flagged as one that might inspire enumerator de-abstraction
962-
// cloning, move the flag to the devirtualized call.
963-
//
964-
if (m_compiler->hasImpEnumeratorGdvLocalMap())
965-
{
966-
Compiler::NodeToUnsignedMap* const map = m_compiler->getImpEnumeratorGdvLocalMap();
967-
unsigned enumeratorLcl = BAD_VAR_NUM;
968-
if (map->Lookup(m_origCall, &enumeratorLcl))
969-
{
970-
JITDUMP("Flagging [%06u] for enumerator cloning via V%02u\n", m_compiler->dspTreeID(call),
971-
enumeratorLcl);
972-
map->Remove(m_origCall);
973-
map->Set(call, enumeratorLcl);
974-
}
975-
}
976-
977961
INDEBUG(call->SetIsGuarded());
978962

979963
JITDUMP("Direct call [%06u] in block " FMT_BB "\n", m_compiler->dspTreeID(call), block->bbNum);
@@ -1022,16 +1006,26 @@ class IndirectCallTransformer
10221006
//
10231007
assert(!call->IsVirtual() && !call->IsDelegateInvoke());
10241008

1025-
// If the devirtualizer was unable to transform the call to invoke the unboxed entry, the inline info
1026-
// we set up may be invalid. We won't be able to inline anyways. So demote the call as an inline candidate.
1009+
// Don't inline if the candidate was kept for devirtualization only, or if the
1010+
// devirtualizer couldn't use the unboxed entry (which invalidates the inline info).
1011+
// Either way we keep the direct call, we just don't re-mark it as a candidate.
10271012
//
10281013
CORINFO_METHOD_HANDLE unboxedMethodHnd = inlineInfo->guardedMethodUnboxedResolvedToken.hMethod;
1029-
if ((unboxedMethodHnd != nullptr) && (methodHnd != unboxedMethodHnd))
1014+
const bool unboxedEntryMismatch = (unboxedMethodHnd != nullptr) && (methodHnd != unboxedMethodHnd);
1015+
1016+
if (!inlineInfo->isInlineable || unboxedEntryMismatch)
10301017
{
1031-
// Demote this call to a non-inline candidate
1032-
//
1033-
JITDUMP("Devirtualization was unable to use the unboxed entry; so marking call (to boxed entry) as not "
1034-
"inlineable\n");
1018+
if (unboxedEntryMismatch)
1019+
{
1020+
JITDUMP("Devirtualization was unable to use the unboxed entry; so marking call (to boxed entry) as "
1021+
"not inlineable\n");
1022+
}
1023+
else
1024+
{
1025+
JITDUMP("Target of this GDV candidate is not inlineable; leaving the devirtualized call as a plain "
1026+
"direct call\n");
1027+
m_compiler->Metrics.NoInlineGDV++;
1028+
}
10351029

10361030
call->gtFlags &= ~GTF_CALL_INLINE_CANDIDATE;
10371031
call->ClearInlineInfo();
@@ -1048,6 +1042,25 @@ class IndirectCallTransformer
10481042
}
10491043
else
10501044
{
1045+
// If the original call was flagged as one that might inspire enumerator
1046+
// de-abstraction cloning, move the flag to the devirtualized call.
1047+
//
1048+
// Done here rather than right after the clone so a candidate we won't inline
1049+
// doesn't consume the mapping and hide it from one we will.
1050+
//
1051+
if (m_compiler->hasImpEnumeratorGdvLocalMap())
1052+
{
1053+
Compiler::NodeToUnsignedMap* const map = m_compiler->getImpEnumeratorGdvLocalMap();
1054+
unsigned enumeratorLcl = BAD_VAR_NUM;
1055+
if (map->Lookup(m_origCall, &enumeratorLcl))
1056+
{
1057+
JITDUMP("Flagging [%06u] for enumerator cloning via V%02u\n", m_compiler->dspTreeID(call),
1058+
enumeratorLcl);
1059+
map->Remove(m_origCall);
1060+
map->Set(call, enumeratorLcl);
1061+
}
1062+
}
1063+
10511064
// Add the call.
10521065
//
10531066
m_compiler->fgNewStmtAtEnd(block, call, m_stmt->GetDebugInfo());

‎src/coreclr/jit/inline.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -621,6 +621,11 @@ struct InlineCandidateInfo : public HandleHistogramProfileCandidateInfo
621621
unsigned clsAttr;
622622
unsigned methAttr;
623623

624+
// True if the target of this candidate can be inlined. GDV candidates are kept
625+
// around even when it can't be, so this is what tells the two apart.
626+
//
627+
bool isInlineable;
628+
624629
CorInfoInitClassResult initClassResult;
625630
InlineContext* inlinersContext;
626631

‎src/coreclr/jit/jitconfigvalues.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -742,6 +742,11 @@ RELEASE_CONFIG_INTEGER(JitEnableGuardedDevirtualization, "JitEnableGuardedDevirt
742742
// Max number is MAX_GDV_TYPE_CHECKS defined above ^. -1 means it's up to JIT to decide
743743
RELEASE_CONFIG_INTEGER(JitGuardedDevirtualizationMaxTypeChecks, "JitGuardedDevirtualizationMaxTypeChecks", -1)
744744

745+
// Whether a guarded devirtualization candidate has to be inlineable.
746+
// 0 - keep it even if we won't inline it, a direct call is still cheaper.
747+
// 1 - drop it if the target can't be inlined (legacy behavior).
748+
RELEASE_CONFIG_INTEGER(JitGuardedDevirtualizationRequireInlining, "JitGuardedDevirtualizationRequireInlining", 0)
749+
745750
// Various policies for GuardedDevirtualization (0x4B == 75)
746751
RELEASE_CONFIG_INTEGER(JitGuardedDevirtualizationChainLikelihood, "JitGuardedDevirtualizationChainLikelihood", 0x4B)
747752
RELEASE_CONFIG_INTEGER(JitGuardedDevirtualizationChainStatements, "JitGuardedDevirtualizationChainStatements", 1)

‎src/coreclr/jit/jitmetadatalist.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ JITMETADATAMETRIC(GDV, int, 0)
6464
JITMETADATAMETRIC(ClassGDV, int, 0)
6565
JITMETADATAMETRIC(MethodGDV, int, 0)
6666
JITMETADATAMETRIC(MultiGuessGDV, int, 0)
67+
JITMETADATAMETRIC(NoInlineGDV, int, 0)
6768
JITMETADATAMETRIC(ChainedGDV, int, 0)
6869
JITMETADATAMETRIC(EnumeratorGDV, int, 0)
6970
JITMETADATAMETRIC(InlinerBranchFold, int, 0)

0 commit comments

Comments
 (0)