[NativeAOT] BestFitMapping tests fail for StringBuilder and layout classes, and crash for string arrays #133395

Description

The five assembly variants under src\tests\Interop\PInvoke\BestFitMapping are disabled on NativeAOT using the old dotnet/runtimelab#155 link (now #69919).

Temporarily enabling them revealed StringBuilder failures. Temporarily bypassing only the StringBuilder cases then exposed additional layout-class failures and a process crash while testing string arrays. This issue tracks the full test so all of these cases can be investigated before enabling it.

Reproduction

Observed on Windows x64 with a Checked NativeAOT product build and the en-US test path, based on dotnet/runtime commit fb98b57, with subsequent local interop-test enablement changes. No regression range has been established.

1. Remove the NativeAOT ActiveIssue from src\tests\Interop\PInvoke\BestFitMapping\Program.cs.

2. Build and run:

   src\tests\build.cmd nativeaot checked tree Interop priority 1
   src\tests\run.cmd runnativeaottests checked

3. To expose the failures after StringBuilder, temporarily bypass Test.ValidateStringBuilder in src\tests\Interop\PInvoke\BestFitMapping\Test.cs on NativeAOT, and repeat the build and run. This diagnostic bypass should not be required for the eventual fix.

Observed failures

1. Byref StringBuilder behavior

All five assembly variants fail in Test.ValidateStringBuilder. For example:

Assert.NotEqual() Failure: Strings are equal
Expected: Not "\\//::"
Actual:       "\\//::"
   at Xunit.Assert.NotEqual[T](T, T, IEqualityComparer`1)
   at Test.Validate[T,U](Boolean, Boolean, Test.Functions`1, Test.DataContext`2)
   at Test.ValidateStringBuilder(Boolean, Boolean, Test.Functions`1)
   at Char.PInvoke_Default.RunTest(Boolean, Boolean)
   at Program.TestEntryPoint()

The assertion compares the original and returned contents after an [In, Out] ref StringBuilder call. NativeAOT updates the original builder rather than replacing it, so both references observe the converted contents. The false/false variant similarly reports equal "?????" strings.

2. Byref layout-class behavior

With only StringBuilder validation bypassed, Assembly_Default and Assembly_False_False pass their char/direct-string checks and reach LPStr.PInvoke_Default.RunTest. They fail while validating LPStrTestClass:

 ---- Marshal String
 ---- Marshal String[]
 ---- Marshal LPStrTestStruct
 ---- Marshal LPStrTestClass
Xunit.Sdk.NotEqualException: Assert.NotEqual() Failure: Strings are equal
Expected: Not "\\//::"
Actual:       "\\//::"
   at Xunit.Assert.NotEqual[T](T, T, IEqualityComparer`1)
   at Test.Validate[T,U](Boolean, Boolean, Test.Functions`1, Test.DataContext`2)
   at LPStr.PInvoke_Default.RunTest(Boolean, Boolean)
   at Program.TestEntryPoint()

The existing test expects the original reference object to remain unchanged while the byref result contains the marshalled value.

3. String-array crash

In the same run, Assembly_False_True gets through the char tests and the direct LPStr string checks, then terminates the merged runner while validating string[]:

Validating LPStr marshalling...
 -- Validate P/Invokes: BestFitMapping not set, ThrowOnUnmappableChar not set
 ---- Marshal String
 ---- Marshal String[]
Expected: 100
Actual: -1073741819
END EXECUTION - FAILED

-1073741819 is 0xC0000005 (access violation). This configuration has assembly-level BestFitMapping(false, ThrowOnUnmappableChar = true). Invalid/unmappable input should result in the expected managed exception, not terminate the process.

The exact crashing instruction has not been identified. One source-level investigation lead is array exception cleanup: NativeAOT's ArrayMarshaller leaves input buffers uninitialized and emits cleanup over the full array length, including when an element conversion throws before initialization completes.

Related issues / duplicate check

• Support BestFitMapping in marshalling runtimelab#172 records the earlier StringBuilder AreNotEqual failure in this test family.
• StringBuilder marshalling #123529 tracks StringBuilder marshalling more generally.
• PInvoke/Array/MarshalArrayAsParam/AsLPArray/AsLPArrayTest failing #81674 is closed and concerns other NativeAOT array cases, including empty arrays and out-only buffers; the crash above occurs in the best-fit/unmappable-character test path.
• JITed struct marshaler IL stub is missing a GC reference during codegen #33950 is a closed CoreCLR GC-stress/JIT struct-marshaler issue, rather than this NativeAOT run.

Searches for BestFitMapping, ThrowOnUnmappableChar, LPStrTestClass, NativeAOT array marshalling, and layout-class marshalling did not find an existing report matching the newly observed layout-class failure and string-array crash. The StringBuilder overlap is noted above.