Skip to content

JIT: save pgo data in inline context, use it for call optimization #116241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 3, 2025
Merged

JIT: save pgo data in inline context, use it for call optimization #116241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0f66fc8
JIT: save pgo data in inline context, use it for call optimization
AndyAyersMS Jun 2, 2025
e3c40a8
Dump likely class in gtDispTree
EgorBo Jun 2, 2025
b828888
fix scope of ifdef'd code so we always capture the pgo data
AndyAyersMS Jun 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion src/coreclr/jit/compiler.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2531,6 +2531,7 @@ void Compiler::compInitOptions(JitFlags* jitFlags)
fgPgoDisabled = true;
}
}
#endif // DEBUG

// A successful result implies a non-NULL fgPgoSchema
//
Expand All @@ -2550,6 +2551,11 @@ void Compiler::compInitOptions(JitFlags* jitFlags)
break;
}
}

// Stash pointers to PGO info on the context so
// we can access contextually it later.
//
compInlineContext->SetPgoInfo(PgoInfo(this));
}

// A failed result implies a NULL fgPgoSchema
Expand All @@ -2559,7 +2565,6 @@ void Compiler::compInitOptions(JitFlags* jitFlags)
{
assert(fgPgoSchema == nullptr);
}
#endif // DEBUG
}

bool enableInliningMethodsWithEH = JitConfig.JitInlineMethodsWithEH() > 0;
Expand Down
3 changes: 2 additions & 1 deletion src/coreclr/jit/compiler.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7529,7 +7529,8 @@ class Compiler
CORINFO_CLASS_HANDLE* classGuesses,
CORINFO_METHOD_HANDLE* methodGuesses,
int* candidatesCount,
unsigned* likelihoods);
unsigned* likelihoods,
bool verboseLogging = true);
Copy link
Preview

Copilot AI Jun 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The pickGDV parameter list is growing; consider grouping related flags (like verboseLogging) and PGO inputs into a struct to simplify calls and reduce maintenance risk.

Copilot uses AI. Check for mistakes.


void considerGuardedDevirtualization(GenTreeCall* call,
IL_OFFSET ilOffset,
Expand Down
21 changes: 19 additions & 2 deletions src/coreclr/jit/gentree.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8347,9 +8347,11 @@ GenTreeCall* Compiler::gtNewCallNode(gtCallTypes callType,
// These get updated after call node is built.
node->gtInlineObservation = InlineObservation::CALLEE_UNUSED_INITIAL;
node->gtRawILOffset = BAD_IL_OFFSET;
node->gtInlineContext = compInlineContext;

#endif

node->gtInlineContext = compInlineContext;

// Spec: Managed Retval sequence points needs to be generated while generating debug info for debuggable code.
//
// Implementation note: if not generating MRV info genCallSite2ILOffsetMap will be NULL and
Expand Down Expand Up @@ -9952,9 +9954,10 @@ GenTreeCall* Compiler::gtCloneExprCallHelper(GenTreeCall* tree)
#if defined(DEBUG)
copy->gtInlineObservation = tree->gtInlineObservation;
copy->gtRawILOffset = tree->gtRawILOffset;
copy->gtInlineContext = tree->gtInlineContext;
#endif

copy->gtInlineContext = tree->gtInlineContext;

copy->CopyOtherRegFlags(tree);

// We keep track of the number of no return calls, so if we've cloned
Expand Down Expand Up @@ -13013,6 +13016,20 @@ void Compiler::gtDispTree(GenTree* tree,
}
}

// Dump profile if any
if (call->IsHelperCall() && impIsCastHelperMayHaveProfileData(eeGetHelperNum(call->gtCallMethHnd)))
{
CORINFO_CLASS_HANDLE likelyClasses[MAX_GDV_TYPE_CHECKS] = {};
unsigned likelyLikelihoods[MAX_GDV_TYPE_CHECKS] = {};
int likelyClassCount = 0;
pickGDV(call, call->gtCastHelperILOffset, false, likelyClasses, nullptr, &likelyClassCount,
likelyLikelihoods, false);
if (likelyClassCount > 0)
{
printf(" (%d%% likely '%s')", likelyLikelihoods[0], eeGetClassName(likelyClasses[0]));
}
}

gtDispCommonEndLine(tree);

if (!topOnly)
Expand Down
5 changes: 1 addition & 4 deletions src/coreclr/jit/gentree.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5660,12 +5660,9 @@ struct GenTreeCall final : public GenTree

// IL offset of the call wrt its parent method.
IL_OFFSET gtRawILOffset;
#endif // defined(DEBUG)

// In DEBUG we report even non inline candidates in the inline tree in
// fgNoteNonInlineCandidate. We need to keep around the inline context for
// this as normally it's part of the candidate info.
class InlineContext* gtInlineContext;
#endif // defined(DEBUG)

bool IsHelperCall() const
{
Expand Down
24 changes: 10 additions & 14 deletions src/coreclr/jit/helperexpansion.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1942,10 +1942,6 @@ enum class TypeCheckPassedAction
CallHelper_AlwaysThrows,
};

// Some arbitrary limit on the number of guesses we can make
// The actual number of guesses is usually much smaller
#define MAX_CAST_GUESSES 8

//------------------------------------------------------------------------------
// PickCandidatesForTypeCheck: picks classes to use as fast type checks against
// the object being casted. The function also defines the strategy to follow
Expand All @@ -1954,7 +1950,7 @@ enum class TypeCheckPassedAction
// Arguments:
// comp - Compiler instance
// castHelper - Cast helper call to expand
// candidates - [out] Classes (guesses) to use in the fast path (up to MAX_CAST_GUESSES)
// candidates - [out] Classes (guesses) to use in the fast path (up to MAX_GDV_TYPE_CHECKS)
// commonCls - [out] Common denominator class for the fast and the fallback paths.
// likelihoods - [out] Likelihoods of successful type checks [0..100]
// typeCheckFailed - [out] Action to perform if the type check fails
Expand Down Expand Up @@ -2160,9 +2156,9 @@ static int PickCandidatesForTypeCheck(Compiler* comp,
/////////////////////////////////////////////////////////////////////////////////////////////////////

// Let's re-use GDV's threshold on how many guesses we can make (can be 3 by default).
const int maxTypeChecks = min(comp->getGDVMaxTypeChecks(), MAX_CAST_GUESSES);
const int maxTypeChecks = min(comp->getGDVMaxTypeChecks(), MAX_GDV_TYPE_CHECKS);

CORINFO_CLASS_HANDLE exactClasses[MAX_CAST_GUESSES] = {};
CORINFO_CLASS_HANDLE exactClasses[MAX_GDV_TYPE_CHECKS] = {};
const int numExactClasses = comp->info.compCompHnd->getExactClasses(castToCls, maxTypeChecks, exactClasses);
bool allTrulyExact = true;
for (int i = 0; i < numExactClasses; i++)
Expand Down Expand Up @@ -2234,9 +2230,9 @@ static int PickCandidatesForTypeCheck(Compiler* comp,
// 3) Consult with PGO data
/////////////////////////////////////////////////////////////////////////////////////////////////////

CORINFO_CLASS_HANDLE likelyClasses[MAX_CAST_GUESSES] = {};
unsigned likelyLikelihoods[MAX_CAST_GUESSES] = {};
int likelyClassCount = 0;
CORINFO_CLASS_HANDLE likelyClasses[MAX_GDV_TYPE_CHECKS] = {};
unsigned likelyLikelihoods[MAX_GDV_TYPE_CHECKS] = {};
int likelyClassCount = 0;
comp->pickGDV(castHelper, castHelper->gtCastHelperILOffset, false, likelyClasses, nullptr, &likelyClassCount,
likelyLikelihoods);

Expand Down Expand Up @@ -2364,8 +2360,8 @@ bool Compiler::fgLateCastExpansionForCall(BasicBlock** pBlock, Statement* stmt,
TypeCheckFailedAction typeCheckFailedAction;
TypeCheckPassedAction typeCheckPassedAction;
CORINFO_CLASS_HANDLE commonCls;
CORINFO_CLASS_HANDLE expectedExactClasses[MAX_CAST_GUESSES] = {};
unsigned likelihoods[MAX_CAST_GUESSES] = {};
CORINFO_CLASS_HANDLE expectedExactClasses[MAX_GDV_TYPE_CHECKS] = {};
unsigned likelihoods[MAX_GDV_TYPE_CHECKS] = {};

const int numOfCandidates = PickCandidatesForTypeCheck(this, call, expectedExactClasses, &commonCls, likelihoods,
&typeCheckFailedAction, &typeCheckPassedAction);
Expand Down Expand Up @@ -2450,8 +2446,8 @@ bool Compiler::fgLateCastExpansionForCall(BasicBlock** pBlock, Statement* stmt,
// Block 2: typeCheckBb(s)
// TODO-InlineCast: if likelyCls == expectedCls we can consider saving to a local to re-use.

BasicBlock* typeChecksBbs[MAX_CAST_GUESSES] = {};
BasicBlock* lastTypeCheckBb = nullcheckBb;
BasicBlock* typeChecksBbs[MAX_GDV_TYPE_CHECKS] = {};
BasicBlock* lastTypeCheckBb = nullcheckBb;
for (int candidateId = 0; candidateId < numOfCandidates; candidateId++)
{
const CORINFO_CLASS_HANDLE expectedCls = expectedExactClasses[candidateId];
Expand Down
38 changes: 27 additions & 11 deletions src/coreclr/jit/importercalls.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6750,24 +6750,30 @@ void Compiler::addFatPointerCandidate(GenTreeCall* call)
// methodGuesses - [out] the methods to guess for (mutually exclusive with classGuess)
// candidatesCount - [out] number of guesses
// likelihoods - [out] estimates of the likelihoods that the guesses will succeed
// verboseLogging - whether or not to do verbose logging
//
void Compiler::pickGDV(GenTreeCall* call,
IL_OFFSET ilOffset,
bool isInterface,
CORINFO_CLASS_HANDLE* classGuesses,
CORINFO_METHOD_HANDLE* methodGuesses,
int* candidatesCount,
unsigned* likelihoods)
unsigned* likelihoods,
bool verboseLogging)
{
*candidatesCount = 0;

// Get the relevant pgo info for this call
//
PgoInfo pgoInfo(call->gtInlineContext);

const int maxLikelyClasses = MAX_GDV_TYPE_CHECKS;
LikelyClassMethodRecord likelyClasses[maxLikelyClasses];
unsigned numberOfClasses = 0;
if (call->IsVirtualStub() || call->IsVirtualVtable() || call->IsHelperCall())
{
numberOfClasses =
getLikelyClasses(likelyClasses, maxLikelyClasses, fgPgoSchema, fgPgoSchemaCount, fgPgoData, ilOffset);
numberOfClasses = getLikelyClasses(likelyClasses, maxLikelyClasses, pgoInfo.PgoSchema, pgoInfo.PgoSchemaCount,
pgoInfo.PgoData, ilOffset);
}

const int maxLikelyMethods = MAX_GDV_TYPE_CHECKS;
Expand All @@ -6783,18 +6789,21 @@ void Compiler::pickGDV(GenTreeCall* call,
if (!IsAot() && (call->IsVirtualVtable() || call->IsDelegateInvoke()))
{
assert(!call->IsHelperCall());
numberOfMethods =
getLikelyMethods(likelyMethods, maxLikelyMethods, fgPgoSchema, fgPgoSchemaCount, fgPgoData, ilOffset);
numberOfMethods = getLikelyMethods(likelyMethods, maxLikelyMethods, pgoInfo.PgoSchema, pgoInfo.PgoSchemaCount,
pgoInfo.PgoData, ilOffset);
}

if ((numberOfClasses < 1) && (numberOfMethods < 1))
{
JITDUMP("No likely class or method, sorry\n");
if (verboseLogging)
{
JITDUMP("No likely class or method, sorry\n");
}
return;
}

#ifdef DEBUG
if ((verbose || JitConfig.EnableExtraSuperPmiQueries()) && (numberOfClasses > 0))
if ((verbose || JitConfig.EnableExtraSuperPmiQueries()) && (numberOfClasses > 0) && verboseLogging)
{
JITDUMP("Likely classes for call [%06u]", dspTreeID(call));
if (!call->IsHelperCall())
Expand Down Expand Up @@ -6964,8 +6973,12 @@ void Compiler::pickGDV(GenTreeCall* call,
classGuesses[guessIdx] = (CORINFO_CLASS_HANDLE)likelyClasses[guessIdx].handle;
likelihoods[guessIdx] = likelyClasses[guessIdx].likelihood;
*candidatesCount = *candidatesCount + 1;
JITDUMP("Accepting type %s with likelihood %u as a candidate\n", eeGetClassName(classGuesses[guessIdx]),
likelihoods[guessIdx])

if (verboseLogging)
{
JITDUMP("Accepting type %s with likelihood %u as a candidate\n",
eeGetClassName(classGuesses[guessIdx]), likelihoods[guessIdx])
}
}
else
{
Expand All @@ -6988,8 +7001,11 @@ void Compiler::pickGDV(GenTreeCall* call,
return;
}

JITDUMP("Not guessing for method; likelihood is below %s call threshold %u\n",
call->IsDelegateInvoke() ? "delegate" : "virtual", likelihoodThreshold);
if (verboseLogging)
{
JITDUMP("Not guessing for method; likelihood is below %s call threshold %u\n",
call->IsDelegateInvoke() ? "delegate" : "virtual", likelihoodThreshold);
}
}
}

Expand Down
20 changes: 20 additions & 0 deletions src/coreclr/jit/inline.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,7 @@ InlineContext::InlineContext(InlineStrategy* strategy)
, m_Code(nullptr)
, m_Callee(nullptr)
, m_RuntimeContext(nullptr)
, m_PgoInfo()
, m_ILSize(0)
, m_ImportedILSize(0)
, m_ActualCallOffset(BAD_IL_OFFSET)
Expand Down Expand Up @@ -1823,3 +1824,22 @@ bool InlineStrategy::IsInliningDisabled()

#endif // defined(DEBUG)
}

PgoInfo::PgoInfo()
{
PgoSchema = nullptr;
PgoSchemaCount = 0;
PgoData = nullptr;
}

PgoInfo::PgoInfo(Compiler* compiler)
{
PgoSchema = compiler->fgPgoSchema;
PgoSchemaCount = compiler->fgPgoSchemaCount;
PgoData = compiler->fgPgoData;
}

PgoInfo::PgoInfo(InlineContext* context)
{
*this = context->GetPgoInfo();
}
31 changes: 31 additions & 0 deletions src/coreclr/jit/inline.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -715,6 +715,21 @@ struct InlineInfo
BasicBlock* iciBlock; // The basic block iciStmt is in.
};

//------------------------------------------------------------------------
// PgoInfo
// Schema and data for a method's PGO data.
//
struct PgoInfo
{
PgoInfo();
PgoInfo(Compiler* compiler);
PgoInfo(InlineContext* inlineContext);

ICorJitInfo::PgoInstrumentationSchema* PgoSchema; // pgo schema for method
BYTE* PgoData; // pgo data for the method
unsigned PgoSchemaCount; // count of schema elements
};

// InlineContext tracks the inline history in a method.
//
// Notes:
Expand Down Expand Up @@ -870,6 +885,21 @@ class InlineContext
}
#endif

const PgoInfo& GetPgoInfo()
Copy link
Preview

Copilot AI Jun 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Consider adding a brief doc comment explaining what GetPgoInfo, SetPgoInfo, and HasPgoInfo do, and how PgoInfo is intended to be used within inlining.

Copilot uses AI. Check for mistakes.

{
return m_PgoInfo;
}

void SetPgoInfo(const PgoInfo& info)
{
m_PgoInfo = info;
}

bool HasPgoInfo() const
{
return (m_PgoInfo.PgoSchema != nullptr) && (m_PgoInfo.PgoSchemaCount > 0) && (m_PgoInfo.PgoData != nullptr);
}

private:
InlineContext(InlineStrategy* strategy);

Expand All @@ -880,6 +910,7 @@ class InlineContext
const BYTE* m_Code; // address of IL buffer for the method
CORINFO_METHOD_HANDLE m_Callee; // handle to the method
CORINFO_CONTEXT_HANDLE m_RuntimeContext; // handle to the exact context
PgoInfo m_PgoInfo; // profile data
unsigned m_ILSize; // size of IL buffer for the method
unsigned m_ImportedILSize; // estimated size of imported IL
ILLocation m_Location; // inlining statement location within parent
Expand Down
Loading