I was looking back on #61956 (comment) and since the current implementation hasn't changed since, I decided to try to and see if I can create an example.
And yes, here is a playground example showing three invalid transmutes, all stemming from different points in the code (two alignment-based and one regarding treating "non-null-optimized" enums as newtypes).
What we special-case is a transmute from Outer1(Pointer1(Inner1(T))) to Outer2(Pointer2(Inner2(T))), where:
T is a generic param (or an associated type of one)
- presumably
T: ?Sized, because T: Sized would result in Pointer(Inner(T)) having a known layout, even when the exact choice of T isn't known
Inner{1,2}(T) wrap T in any number of structs, with any choice of prefix fields
- only the effect on
Pointer(Inner(T)) containing T::Metadata matters
Outer{1,2}(X) are newtypes
- that is, they only have an
X field and optionally some ZST fields
- we also allow
Option-like enums here, if the Pointer type they wrap is non-null, but that shouldn't matter - it's effectively "desugaring" the enum optimization to a newtype
- however, it does matter whether the
enum opted out of the optimization, which turns out we're not checking
Now, I confused myself with alignment originally, but what's important to note is that even if a newtype of X contains higher alignment ZSTs around X, the only difference that will make is on the alignment (and size, through additional padding) of the whole newtype, not on the position of X in the newtype.
Because of that, Outer1 and Outer2 could have different alignments and that wouldn't change the fact that reading the pointer from one of them and writing it into the other would work, it's still at offset 0 and has the same size.
But that's not what we do, since transmute is not field-based, and if we copy the larger size, we're reading or writing more bytes than would be legal.
Looking at the LLVM IR of my example, test_align does look like it's copying 64 bytes, which should definitely be UB (but it might take some effort to cause a LLVM optimization to trigger).
OTOH, test_option_like is even worse, since the pointer goes in the #[repr(C)] enum tag, and the value inside the Some is garbage, so running it in release mode crashes trying to print the resulting value.
What's a bit sad is I feel like I remember seeing an assert! for equal transmute sizes in codegen, which would catch such a situation and turn it into an ICE, but I'm guessing it got removed?
cc @nagisa
I was looking back on #61956 (comment) and since the current implementation hasn't changed since, I decided to try to and see if I can create an example.
And yes, here is a playground example showing three invalid
transmutes, all stemming from different points in the code (two alignment-based and one regarding treating "non-null-optimized"enums as newtypes).What we special-case is a
transmutefromOuter1(Pointer1(Inner1(T)))toOuter2(Pointer2(Inner2(T))), where:Tis a generic param (or an associated type of one)T: ?Sized, becauseT: Sizedwould result inPointer(Inner(T))having a known layout, even when the exact choice ofTisn't knownInner{1,2}(T)wrapTin any number ofstructs, with any choice of prefix fieldsPointer(Inner(T))containingT::MetadatamattersOuter{1,2}(X)are newtypesXfield and optionally some ZST fieldsOption-likeenums here, if thePointertype they wrap is non-null, but that shouldn't matter - it's effectively "desugaring" theenumoptimization to a newtypeenumopted out of the optimization, which turns out we're not checkingNow, I confused myself with alignment originally, but what's important to note is that even if a newtype of
Xcontains higher alignment ZSTs aroundX, the only difference that will make is on the alignment (and size, through additional padding) of the whole newtype, not on the position ofXin the newtype.Because of that,
Outer1andOuter2could have different alignments and that wouldn't change the fact that reading the pointer from one of them and writing it into the other would work, it's still at offset0and has the same size.But that's not what we do, since
transmuteis not field-based, and if we copy the larger size, we're reading or writing more bytes than would be legal.Looking at the LLVM IR of my example,
test_aligndoes look like it's copying64bytes, which should definitely be UB (but it might take some effort to cause a LLVM optimization to trigger).OTOH,
test_option_likeis even worse, since the pointer goes in the#[repr(C)] enumtag, and the value inside theSomeis garbage, so running it in release mode crashes trying to print the resulting value.What's a bit sad is I feel like I remember seeing an
assert!for equaltransmutesizes in codegen, which would catch such a situation and turn it into an ICE, but I'm guessing it got removed?cc @nagisa