[SelectionDAG] Scalarize <1 x T> vector types for atomic store#197165
Merged
Conversation
llvmorg-github-actions
Bot
added
backend:X86
llvm:SelectionDAG
|
@llvm/pr-subscribers-llvm-selectiondag @llvm/pr-subscribers-backend-x86 Author: jofrn Changes`store atomic <1 x T>` is not valid. This change legalizes This is the store-side counterpart to #148894. 3 Files Affected:
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h b/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
index a7d9974faee61..9c37eb8065ba5 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
@@ -877,6 +877,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue ScalarizeVecOp_VSETCC(SDNode *N);
SDValue ScalarizeVecOp_VSTRICT_FSETCC(SDNode *N, unsigned OpNo);
SDValue ScalarizeVecOp_STORE(StoreSDNode *N, unsigned OpNo);
+ SDValue ScalarizeVecOp_ATOMIC_STORE(AtomicSDNode *N);
SDValue ScalarizeVecOp_FP_ROUND(SDNode *N, unsigned OpNo);
SDValue ScalarizeVecOp_STRICT_FP_ROUND(SDNode *N, unsigned OpNo);
SDValue ScalarizeVecOp_FP_EXTEND(SDNode *N);
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
index 81e9e301f2572..c6fc5e2152528 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
@@ -900,6 +900,9 @@ bool DAGTypeLegalizer::ScalarizeVectorOperand(SDNode *N, unsigned OpNo) {
case ISD::STORE:
Res = ScalarizeVecOp_STORE(cast<StoreSDNode>(N), OpNo);
break;
+ case ISD::ATOMIC_STORE:
+ Res = ScalarizeVecOp_ATOMIC_STORE(cast<AtomicSDNode>(N));
+ break;
case ISD::STRICT_FP_ROUND:
Res = ScalarizeVecOp_STRICT_FP_ROUND(N, OpNo);
break;
@@ -1162,6 +1165,15 @@ SDValue DAGTypeLegalizer::ScalarizeVecOp_STORE(StoreSDNode *N, unsigned OpNo){
N->getMemOperand()->getFlags(), N->getAAInfo());
}
+/// If the value to store is a vector that needs to be scalarized, it must be
+/// <1 x ty>. Just store the element.
+SDValue DAGTypeLegalizer::ScalarizeVecOp_ATOMIC_STORE(AtomicSDNode *N) {
+ SDValue ScalarVal = GetScalarizedVector(N->getVal());
+ return DAG.getAtomic(ISD::ATOMIC_STORE, SDLoc(N),
+ N->getMemoryVT().getVectorElementType(), N->getChain(),
+ ScalarVal, N->getBasePtr(), N->getMemOperand());
+}
+
/// If the value to round is a vector that needs to be scalarized, it must be
/// <1 x ty>. Convert the element instead.
SDValue DAGTypeLegalizer::ScalarizeVecOp_FP_ROUND(SDNode *N, unsigned OpNo) {
diff --git a/llvm/test/CodeGen/X86/atomic-load-store.ll b/llvm/test/CodeGen/X86/atomic-load-store.ll
index 867a4acb791bc..8e74276019e17 100644
--- a/llvm/test/CodeGen/X86/atomic-load-store.ll
+++ b/llvm/test/CodeGen/X86/atomic-load-store.ll
@@ -165,6 +165,63 @@ define <1 x i64> @atomic_vec1_i64_align(ptr %x) nounwind {
ret <1 x i64> %ret
}
+define void @store_atomic_vec1_i32(ptr %x, <1 x i32> %v) {
+; CHECK-LABEL: store_atomic_vec1_i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movl %esi, (%rdi)
+; CHECK-NEXT: retq
+ store atomic <1 x i32> %v, ptr %x release, align 4
+ ret void
+}
+
+define void @store_atomic_vec1_i8(ptr %x, <1 x i8> %v) {
+; CHECK-O3-LABEL: store_atomic_vec1_i8:
+; CHECK-O3: # %bb.0:
+; CHECK-O3-NEXT: movb %sil, (%rdi)
+; CHECK-O3-NEXT: retq
+;
+; CHECK-O0-LABEL: store_atomic_vec1_i8:
+; CHECK-O0: # %bb.0:
+; CHECK-O0-NEXT: movb %sil, %al
+; CHECK-O0-NEXT: movb %al, (%rdi)
+; CHECK-O0-NEXT: retq
+ store atomic <1 x i8> %v, ptr %x release, align 1
+ ret void
+}
+
+define void @store_atomic_vec1_i16(ptr %x, <1 x i16> %v) {
+; CHECK-O3-LABEL: store_atomic_vec1_i16:
+; CHECK-O3: # %bb.0:
+; CHECK-O3-NEXT: movw %si, (%rdi)
+; CHECK-O3-NEXT: retq
+;
+; CHECK-O0-LABEL: store_atomic_vec1_i16:
+; CHECK-O0: # %bb.0:
+; CHECK-O0-NEXT: movw %si, %ax
+; CHECK-O0-NEXT: movw %ax, (%rdi)
+; CHECK-O0-NEXT: retq
+ store atomic <1 x i16> %v, ptr %x release, align 2
+ ret void
+}
+
+define void @store_atomic_vec1_i64(ptr %x, <1 x i64> %v) nounwind {
+; CHECK-LABEL: store_atomic_vec1_i64:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq %rsi, (%rdi)
+; CHECK-NEXT: retq
+ store atomic <1 x i64> %v, ptr %x release, align 8
+ ret void
+}
+
+define void @store_atomic_vec1_ptr(ptr %x, <1 x ptr> %v) nounwind {
+; CHECK-LABEL: store_atomic_vec1_ptr:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq %rsi, (%rdi)
+; CHECK-NEXT: retq
+ store atomic <1 x ptr> %v, ptr %x release, align 8
+ ret void
+}
+
define <2 x i8> @atomic_vec2_i8(ptr %x) {
; CHECK-SSE-O3-LABEL: atomic_vec2_i8:
; CHECK-SSE-O3: # %bb.0:
|
|
|
arsenm
approved these changes
May 12, 2026
| store atomic <1 x ptr> %v, ptr %x release, align 8 | ||
| ret void | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Maybe test half/bfloat/float/double
Contributor
Author
There was a problem hiding this comment.
they are already in the float commit (above the current commit in the stack).
Unaligned atomic vector stores with size >1 are lowered to calls. Adding their tests separately here.
jofrn
force-pushed
the
users/jofrn/scalarize-vec-atomic-store
branch
from
3e365f3 to
3e49a56
Compare
jofrn
changed the base branch from
main
to
users/jofrn/atomic-store-unaligned-tests
Contributor
Should also fix this |
`store atomic <1 x T>` is not valid. This change legalizes vector types of atomic store via scalarization in SelectionDAG so that it can, for example, translate from `v1i32` to `i32`.
jofrn
force-pushed
the
users/jofrn/scalarize-vec-atomic-store
branch
from
3e49a56 to
bbd3398
Compare
pedroMVicente
pushed a commit
to pedroMVicente/llvm-project
that referenced
this pull request
May 19, 2026
) Unaligned atomic vector stores with size >1 are lowered to calls. Adding their tests separately here. Store-side counterpart to llvm#148896. Stacked below llvm#197165.
jofrn
added a commit
that referenced
this pull request
May 19, 2026
When lowering `atomic store <1 x T>` vector types with floats (i.e. during scalarization in the selection DAG), selection can fail since this pattern is unsupported. To support this, floats can be casted to an integer type of the same size. Store-side counterpart to #148895. Stacked on top of #197165; and below of #197618.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
store atomic <1 x T>is not valid. This change legalizesvector types of atomic store via scalarization in SelectionDAG
so that it can, for example, translate from
v1i32toi32.This is the store-side counterpart to #148894. Stacked on top of #197372; and below of #197166.