Skip to content

Conversation

@fhahn
Copy link
Contributor

@fhahn fhahn commented Nov 2, 2025

Create phi recipes for scalar resume value up front in addInitialSkeleton during initial construction. This will allow moving the remaining code dealing with resume values to VPlan transforms/construction.

@llvmbot
Copy link
Member

llvmbot commented Nov 2, 2025

@llvm/pr-subscribers-vectorizers

@llvm/pr-subscribers-llvm-transforms

Author: Florian Hahn (fhahn)

Changes

Create phi recipes for scalar resume value up front in addInitialSkeleton during initial construction. This will allow moving the remaining code dealing with resume values to VPlan transforms/construction.


Full diff: https://github.com/llvm/llvm-project/pull/166099.diff

3 Files Affected:

  • (modified) llvm/lib/Transforms/Vectorize/VPlan.h (+3-1)
  • (modified) llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp (+9)
  • (modified) llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp (+15-21)
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index aba6d351a8e5d..1aa1d0f894495 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -1079,7 +1079,7 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
   OpcodeTy Opcode;
 
   /// An optional name that can be used for the generated IR instruction.
-  const std::string Name;
+  std::string Name;
 
   /// Returns true if we can generate a scalar for the first lane only if
   /// needed.
@@ -1179,6 +1179,8 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
 
   /// Returns the symbolic name assigned to the VPInstruction.
   StringRef getName() const { return Name; }
+
+  void setName(StringRef NewName) { Name = NewName.str(); }
 };
 
 /// A specialization of VPInstruction augmenting it with a dedicated result
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 1a66d2049a8db..8c65f5e2c8b18 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -527,6 +527,15 @@ static void addInitialSkeleton(VPlan &Plan, Type *InductionTy, DebugLoc IVDL,
   Plan.getEntry()->swapSuccessors();
 
   createExtractsForLiveOuts(Plan, MiddleVPBB);
+
+  VPBuilder ScalarPHBuilder(ScalarPH);
+  for (const auto &[PhiR, ScalarPhiR] : zip_equal(
+           drop_begin(HeaderVPBB->phis()), Plan.getScalarHeader()->phis())) {
+    auto *VectorPhiR = cast<VPPhi>(&PhiR);
+    auto *ResumePhiR = ScalarPHBuilder.createScalarPhi(
+        {VectorPhiR, VectorPhiR->getOperand(0)}, VectorPhiR->getDebugLoc());
+    cast<VPIRPhi>(&ScalarPhiR)->addOperand(ResumePhiR);
+  }
 }
 
 std::unique_ptr<VPlan>
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index f50bf29970597..6987dab35a4f4 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -4376,9 +4376,10 @@ void VPlanTransforms::addBranchWeightToMiddleTerminator(
 /// Create and return a ResumePhi for \p WideIV, unless it is truncated. If the
 /// induction recipe is not canonical, creates a VPDerivedIVRecipe to compute
 /// the end value of the induction.
-static VPInstruction *addResumePhiRecipeForInduction(
-    VPWidenInductionRecipe *WideIV, VPBuilder &VectorPHBuilder,
-    VPBuilder &ScalarPHBuilder, VPTypeAnalysis &TypeInfo, VPValue *VectorTC) {
+static VPValue *addResumePhiRecipeForInduction(VPWidenInductionRecipe *WideIV,
+                                               VPBuilder &VectorPHBuilder,
+                                               VPTypeAnalysis &TypeInfo,
+                                               VPValue *VectorTC) {
   auto *WideIntOrFp = dyn_cast<VPWidenIntOrFpInductionRecipe>(WideIV);
   // Truncated wide inductions resume from the last lane of their vector value
   // in the last vector iteration which is handled elsewhere.
@@ -4404,9 +4405,7 @@ static VPInstruction *addResumePhiRecipeForInduction(
                                                 WideIV->getDebugLoc());
   }
 
-  auto *ResumePhiRecipe = ScalarPHBuilder.createScalarPhi(
-      {EndValue, Start}, WideIV->getDebugLoc(), "bc.resume.val");
-  return ResumePhiRecipe;
+  return EndValue;
 }
 
 void VPlanTransforms::addScalarResumePhis(
@@ -4419,21 +4418,18 @@ void VPlanTransforms::addScalarResumePhis(
   VPBuilder VectorPHBuilder(
       cast<VPBasicBlock>(VectorRegion->getSinglePredecessor()));
   VPBuilder MiddleBuilder(MiddleVPBB, MiddleVPBB->getFirstNonPhi());
-  VPBuilder ScalarPHBuilder(ScalarPH);
-  for (VPRecipeBase &ScalarPhiR : Plan.getScalarHeader()->phis()) {
-    auto *ScalarPhiIRI = cast<VPIRPhi>(&ScalarPhiR);
+  for (VPRecipeBase &PhiR : Plan.getScalarPreheader()->phis()) {
+    auto *ResumePhiR = cast<VPPhi>(&PhiR);
 
     // TODO: Extract final value from induction recipe initially, optimize to
     // pre-computed end value together in optimizeInductionExitUsers.
-    auto *VectorPhiR =
-        cast<VPHeaderPHIRecipe>(Builder.getRecipe(&ScalarPhiIRI->getIRPhi()));
+    auto *VectorPhiR = cast<VPHeaderPHIRecipe>(ResumePhiR->getOperand(0));
     if (auto *WideIVR = dyn_cast<VPWidenInductionRecipe>(VectorPhiR)) {
-      if (VPInstruction *ResumePhi = addResumePhiRecipeForInduction(
-              WideIVR, VectorPHBuilder, ScalarPHBuilder, TypeInfo,
-              &Plan.getVectorTripCount())) {
-        assert(isa<VPPhi>(ResumePhi) && "Expected a phi");
-        IVEndValues[WideIVR] = ResumePhi->getOperand(0);
-        ScalarPhiIRI->addOperand(ResumePhi);
+      if (VPValue *ResumeV = addResumePhiRecipeForInduction(
+              WideIVR, VectorPHBuilder, TypeInfo, &Plan.getVectorTripCount())) {
+        IVEndValues[WideIVR] = ResumeV;
+        ResumePhiR->setOperand(0, ResumeV);
+        ResumePhiR->setName("bc.resume.val");
         continue;
       }
       // TODO: Also handle truncated inductions here. Computing end-values
@@ -4455,10 +4451,8 @@ void VPlanTransforms::addScalarResumePhis(
       ResumeFromVectorLoop = MiddleBuilder.createNaryOp(
           VPInstruction::ExtractLastElement, {ResumeFromVectorLoop}, {},
           "vector.recur.extract");
-    StringRef Name = IsFOR ? "scalar.recur.init" : "bc.merge.rdx";
-    auto *ResumePhiR = ScalarPHBuilder.createScalarPhi(
-        {ResumeFromVectorLoop, VectorPhiR->getStartValue()}, {}, Name);
-    ScalarPhiIRI->addOperand(ResumePhiR);
+    ResumePhiR->setName(IsFOR ? "scalar.recur.init" : "bc.merge.rdx");
+    ResumePhiR->setOperand(0, ResumeFromVectorLoop);
   }
 }
 

@fhahn fhahn force-pushed the vplan-create-scalar-phis-earlier branch from 2175eae to b7cec87 Compare November 4, 2025 19:39
Comment on lines -390 to -391
VPValue *Zero = Plan->getConstantInt(32, 0);
Plan->getScalarHeader()->front().addOperand(Zero);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initial plan now already has the operands of the scalar header phis set up correctly, so this change is not needed any more to fix them up.

Comment on lines +349 to +350
VPValue *Zero = Plan->getConstantInt(32, 0);
Plan->getScalarHeader()->front().addOperand(Zero);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initial plan now already has the operands of the scalar header phis set up correctly, so here we now add anothe operand to trigger a mismatch between #incoming values and #predecessors

@fhahn fhahn force-pushed the vplan-create-scalar-phis-earlier branch from b7cec87 to 38e1813 Compare November 11, 2025 22:59
Copy link
Contributor Author

@fhahn fhahn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping :)

Comment on lines -4440 to -4582
auto *VectorPhiR =
cast<VPHeaderPHIRecipe>(Builder.getRecipe(&ScalarPhiIRI->getIRPhi()));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removes one of the few remaining instances that retrieve recipes via IR referneces.

fhahn added a commit to fhahn/llvm-project that referenced this pull request Nov 19, 2025
…NFC)

This patch splits off VPReductionRecipe creation for in-loop reductions
to a separate transform from adjustInLoopReductions, which has been
renamed.

The new transform has been updated to work directly on VPInstructions,
and gets applied after header phis have been processed, once on VPlan0.

Builds on top of llvm#168291 and
llvm#166099 which should be
reviewed first.
@fhahn fhahn force-pushed the vplan-create-scalar-phis-earlier branch from 38e1813 to 7e109a7 Compare November 19, 2025 22:55
@fhahn
Copy link
Contributor Author

fhahn commented Nov 19, 2025

ping

@github-actions
Copy link

github-actions bot commented Nov 19, 2025

🐧 Linux x64 Test Results

  • 186453 tests passed
  • 4871 tests skipped

Create phi recipes for scalar resume value up front in
addInitialSkeleton during initial construction. This will allow moving
the remaining code dealing with resume values to VPlan
transforms/construction.
@fhahn fhahn requested review from Mel-Chen and lukel97 November 20, 2025 21:22
@fhahn fhahn force-pushed the vplan-create-scalar-phis-earlier branch from 7e109a7 to 67c16bd Compare November 20, 2025 21:22
@fhahn fhahn force-pushed the vplan-create-scalar-phis-earlier branch from 28596f9 to 47d4d48 Compare November 21, 2025 14:48
static void addScalarResumePhis(VPlan &Plan, VPRecipeBuilder &Builder,
DenseMap<VPValue *, VPValue *> &IVEndValues);
/// Update the resume phis in the scalar preheader after creating wide recipes
/// for first-order recurrences, reductions and inductions, End values for
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// for first-order recurrences, reductions and inductions, End values for
/// for first-order recurrences, reductions and inductions. End values for

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks!

@fhahn fhahn enabled auto-merge (squash) November 22, 2025 20:16
@fhahn fhahn merged commit 080ca90 into llvm:main Nov 22, 2025
9 of 10 checks passed
llvm-sync bot pushed a commit to arm/arm-toolchain that referenced this pull request Nov 22, 2025
…) (#166099)

Create phi recipes for scalar resume value up front in addInitialSkeleton during initial construction. This will allow moving the remaining code dealing with resume values to VPlan transforms/construction.

PR: llvm/llvm-project#166099
nekoshirro pushed a commit to nekoshirro/Alchemist-LLVM that referenced this pull request Nov 24, 2025
Create phi recipes for scalar resume value up front in addInitialSkeleton during initial construction. This will allow moving the remaining code dealing with resume values to VPlan transforms/construction.

PR: llvm/llvm-project#166099
Signed-off-by: Hafidz Muzakky <[email protected]>
aadeshps-mcw pushed a commit to aadeshps-mcw/llvm-project that referenced this pull request Nov 26, 2025
)

Create phi recipes for scalar resume value up front in addInitialSkeleton during initial construction. This will allow moving the remaining code dealing with resume values to VPlan transforms/construction.

PR: llvm#166099
Priyanshu3820 pushed a commit to Priyanshu3820/llvm-project that referenced this pull request Nov 26, 2025
)

Create phi recipes for scalar resume value up front in addInitialSkeleton during initial construction. This will allow moving the remaining code dealing with resume values to VPlan transforms/construction.

PR: llvm#166099
augusto2112 pushed a commit to augusto2112/llvm-project that referenced this pull request Dec 3, 2025
)

Create phi recipes for scalar resume value up front in addInitialSkeleton during initial construction. This will allow moving the remaining code dealing with resume values to VPlan transforms/construction.

PR: llvm#166099
fhahn added a commit that referenced this pull request Dec 15, 2025
…68291)

Together with #168289 &
#166099 we can construct header
phis once up front, after creating VPlan0, as the
induction/reduction/first-order-recurrence classification applies across
all VFs.

Depends on #168289 &
#166099 

PR: #168291
llvm-sync bot pushed a commit to arm/arm-toolchain that referenced this pull request Dec 15, 2025
… (NFC). (#168291)

Together with llvm/llvm-project#168289 &
llvm/llvm-project#166099 we can construct header
phis once up front, after creating VPlan0, as the
induction/reduction/first-order-recurrence classification applies across
all VFs.

Depends on llvm/llvm-project#168289 &
llvm/llvm-project#166099

PR: llvm/llvm-project#168291
fhahn added a commit to fhahn/llvm-project that referenced this pull request Dec 16, 2025
…NFC)

This patch splits off VPReductionRecipe creation for in-loop reductions
to a separate transform from adjustInLoopReductions, which has been
renamed.

The new transform has been updated to work directly on VPInstructions,
and gets applied after header phis have been processed, once on VPlan0.

Builds on top of llvm#168291 and
llvm#166099 which should be
reviewed first.
fhahn added a commit to fhahn/llvm-project that referenced this pull request Dec 16, 2025
…NFC)

This patch splits off VPReductionRecipe creation for in-loop reductions
to a separate transform from adjustInLoopReductions, which has been
renamed.

The new transform has been updated to work directly on VPInstructions,
and gets applied after header phis have been processed, once on VPlan0.

Builds on top of llvm#168291 and
llvm#166099 which should be
reviewed first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants