To see where the wasm library tests stand with AOT compilation with respect to how many pass/fail, what the failures are, how many hang, the 268 test suites were ran locally.

Dropbox paper with findings

All CI Results

Among all the test suites failing on CI, the following are the main unique test failures observed.

Unique Test Failures based on CI logs

• a) Assert.True() Failure [Wasm][AOT] Assert.True() failure #49246
• b) System.ExecutionEngineException : AOT Compilation failed for method wasm AOT library test failure - AOT Compilation failed for method #49219
• c) System.ExecutionEngineException : Attempting to JIT compile method wasm AOT library test failure - Attempting to JIT compile method #46708
• d) console.error: RuntimeError: memory access out of bounds wasm AOT library test failure - memory access out of bounds #46700
• e) System.NotImplementedException : The method or operation is not implemented. [Wasm][AOT] System.NotImplementedException : The method or operation is not implemented. #49229
• f) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Threading.CancellationTokenSource:ExecuteCallbackHandlers (bool)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• g) System.ExecutionEngineException : AOT Compilation failed for method 'bool Microsoft.Extensions.Internal.ParameterDefaultValue:TryGetDefaultValue (System.Reflection.ParameterInfo,object&)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• h) System.ExecutionEngineException : Attempting to JIT compile method '(wrapper dynamic-method) object Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder:ResolveService (Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder/ILEmitResolverBuilderRuntimeContext,Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope)' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information. wasm AOT library test failure - Attempting to JIT compile method #46708
• i) System.ExecutionEngineException : Attempting to JIT compile method '(wrapper dynamic-method) object object:lambda_method wasm AOT library test failure - Attempting to JIT compile method #46708
• j) AOT Compilation failed for method 'object Microsoft.Extensions.DependencyInjection.ActivatorUtilities/ConstructorMatcher:CreateInstance (System.IServiceProvider)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• k) System.ExecutionEngineException : AOT Compilation failed for method 'System.Threading.Tasks.Task System.Net.Http.HttpContent:LoadIntoBufferAsync (long,System.Threading.CancellationToken)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• l) EventName and ExceptionJson is supposed to be null but it isn't. [Wasm][AOT] Microsoft.Extensions.Logging.EventSource.Tests failures #49240
• m) LoggerName is supposed to be null but it isn't. [Wasm][AOT] Microsoft.Extensions.Logging.EventSource.Tests failures #49240
• n) console.error: RuntimeError: invalid index into function table wasm AOT library test failure - invalid index into function table #46731
• p) System.Composition.Hosting.CompositionFailedException : No export was found for the contract [Wasm][AOT] System.Composition.Hosting.CompositionFailedException #49230
• q) System.Composition.Hosting.CompositionFailedException : No importing constructor was found on type [Wasm][AOT] System.Composition.Hosting.CompositionFailedException #49230
• u) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Data.DataColumnCollection:BaseAdd (System.Data.DataColumn)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• v) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Data.DataRow:set_Item (System.Data.DataColumn,object)' wasm AOT library test failure - AOT Compilation failed for method #49219
• w) System.IO.DirectoryNotFoundException : Could not find a part of the path or System.IO.FileNotFoundException : Could not find file [Wasm][AOT] System.IO.DirectoryNotFoundException And System.IO.FileNotFoundException #49232 (@fanyang-mono working on this)
• x) Assert.NotEmpty() Failure [Wasm][AOT] System.Diagnostics.TraceSource.Tests failures #49241
• y) Assert.Contains() Failure [Wasm][AOT] System.Diagnostics.TraceSource.Tests failures #49241
• z) System.ExecutionEngineException : AOT Compilation failed for method 'void System.IO.MemoryMappedFiles.Tests.MemoryMappedFileTests_CreateFromFile wasm AOT library test failure - AOT Compilation failed for method #49219
• ab) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Net.Http.HttpContent:CopyTo (System.IO.Stream,System.Net.TransportContext,System.Threading.CancellationToken)'. wasm AOT library test failure - AOT Compilation failed for method #49219
  Special case (Invariant.Tests)
• ac) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Net.Mime.ContentType:ParseValue ()'. wasm AOT library test failure - AOT Compilation failed for method #49219
• ad) [byte difference] [Wasm][AOT] System.Reflection.Metadata.Tests failures #49245
• ae) System.InvalidOperationException : Failed to compare two elements in the array. wasm AOT library test failure - System.InvalidOperationException #46759
• ag) Assert.Throws() Failure [Wasm][AOT] System.Runtime.CompilerServices.Unsafe.Tests failures #49242
• aj) console.error: RuntimeError: function signature mismatch wasm AOT library test failure - function signature mismatch #46761 log
• ak) System.NullReferenceException : Object reference not set to an instance of an object. [Wasm][AOT] System.NullReferenceException : Object reference not set to an instance of an object. #49234
• al) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Numerics.Tests.divremTest:VerifyDivRemString (string)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• am) Assertion at /__w/1/s/src/mono/mono/metadata/class-init.c:2746, condition '<disabled>' not met wasm AOT library test failure - condition <disabled> not met #46698 log
• an) Assertion: should not be reached at /__w/1/s/src/mono/mono/metadata/class-accessors.c:86 [Wasm][AOT] Assertion: should not be reached at src/mono/mono/metadata/class-accessors.c:83 #45304
• ao) Assertion at /__w/1/s/src/mono/mono/mini/mini-generic-sharing.c:2168, condition '<disabled>' not met wasm AOT library test failure - condition <disabled> not met #46698 log
• ap) Process 24507 didn't exit within 00:15:00 and will be killed [Wasm][AOT] System.Threading.Tasks.Dataflow.Tests failures #49243
• aq) Assertion at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3965, condition '<disabled>' not met wasm AOT library test failure - condition <disabled> not met #46698 log
• as) console.error: Unhandled exception in _finalize_startup Error: FS error [Wasm][AOT] System.IO.FileSystem.Tests failures #49244
• au) Assertion: should not be reached at /__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:91 [Wasm][AOT] Assertion: should not be reached at /__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:91 #49225

EnableAggressiveTrimming related failures
r) System.ArgumentException : Type Moq.Internals.InterfaceProxy is not valid base type for interface proxy, because it does not have accessible parameterless constructor. Only a non-sealed class with non-private default constructor can be used as base type for interface proxy. Please use some other valid type. #46760 log
s) System.InvalidOperationException : Sequence contains no matching element #46759
t) System.ArgumentNullException : Value cannot be null. (Parameter 'method') #46758 (@naricc working on this)
af) System.NotSupportedException : Use of ResourceManager for custom types is disabled. Set the MSBuild Property CustomResourceTypesSupport to true in order to enable it.
ah) console.error: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) ---> System.NullReferenceException: Object reference not set to an instance of an object.
ai) Could not load file or assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies log

The Unique failures may appear across several test suites. Each test suite below has failed on CI, and are tagged with the corresponding failure by the alphabetical tag.

Suites that did not produce test results

Error: Failed to load method 0x6000001

• System.ComponentModel.Composition.Tests -- ai
• System.Diagnostics.Tracing.Tests -- ai
• System.Runtime.Loader.DefaultContext.Tests -- ai
• System.Runtime.Loader.RefEmitLoadContext.Tests -- ai
• System.Runtime.Tests -- Precompiling failed for log

Suites that failed with 1 or more test failures

• System.Data.Common.Tests -- u
• System.Security.Cryptography.Algorithms.Tests -- aj
• System.Numerics.Vectors.Tests -- c
• System.Linq.Queryable.Tests -- i, d
• System.IO.FileSystem.Tests -- as
• System.Linq.Expressions.Tests -- an
• System.Runtime.Serialization.Xml.Tests -- an
• Microsoft.Extensions.DependencyInjection.Tests -- g, h, i, j, k
• System.Runtime.Serialization.Json.Tests -- c
• System.Composition.Tests -- i, p, q
• System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests -- w
• System.ComponentModel.TypeConverter.Tests -- i, d, t
• System.Threading.Channels.Tests -- c, f
• System.IO.Pipelines.Tests -- f, y
• System.Data.DataSetExtensions.Tests -- u, v, w, missing 33 tests
• System.Runtime.Numerics.Tests -- ak, al
• System.ServiceModel.Syndication.Tests -- c, w
• System.Collections.Immutable.Tests -- d
• System.ComponentModel.Composition.Registration.Tests -- i
• System.Reflection.DispatchProxy.Tests -- c
• System.Net.Http.Json.Functional.Tests -- c, f
• Microsoft.Extensions.Configuration.Json.Tests -- a, f
• Microsoft.Extensions.Logging.EventSource.Tests -- g, l, m
• Microsoft.Extensions.Configuration.Functional.Tests -- f
• System.IO.Compression.Tests -- w
• Common.Tests -- a, b
• System.Collections.Concurrent.Tests -- e, f
• System.IO.MemoryMappedFiles.Tests -- z
• System.Composition.Convention.Tests -- i
• System.Threading.Tests -- f
• System.Xml.XmlSchemaSet.Tests -- a, w
• Microsoft.Extensions.Primitives.Tests -- f, r
• System.Composition.TypedParts.Tests -- i
• Microsoft.Extensions.Http.Tests -- i, k, t
• System.Reflection.Emit.Lightweight.Tests -- c
• Microsoft.Extensions.Configuration.Ini.Tests -- a, f
• System.IO.Tests -- Random text Kæ￿ $@#�'ã†�ìŽ A5 not AOT related
• Microsoft.Extensions.Configuration.Xml.Tests -- a, f
• System.Collections.Tests -- e, killed
• System.Text.RegularExpressions.Tests -- au
• System.CodeDom.Tests -- d
• System.Drawing.Primitives.Tests -- c
• System.Memory.Tests -- c
• System.ObjectModel.Tests -- d
• Microsoft.Extensions.Options.Tests -- f
• Microsoft.Extensions.Configuration.Tests -- f
• Microsoft.Extensions.Configuration.UserSecrets.Tests -- f
• Microsoft.Extensions.Logging.Testing.Tests -- g
• System.IO.Packaging.Tests -- w
• System.Memory.Data.Tests -- f, c
• System.Net.Primitives.Functional.Tests -- e
• Microsoft.Extensions.FileProviders.Composite.Tests -- a, r
• System.Diagnostics.TraceSource.Tests -- x, y
• Microsoft.Extensions.Configuration.Binder.Tests -- e
• Microsoft.Extensions.Configuration.FileExtensions.Tests -- r
• System.Composition.Hosting.Tests -- i
• System.Globalization.Tests -- e
• System.Net.WebSockets.Tests -- c
• System.Reflection.CoreCLR.Tests -- EnableAggressiveTrimming failure
• System.Transactions.Local.Tests -- ak
• System.Diagnostics.DiagnosticSource.Tests -- t
• System.Net.Http.Json.Unit.Tests -- c
• System.Net.Http.Unit.Tests -- k, ab
• System.Resources.Reader.Tests -- af
• System.Runtime.CompilerServices.Unsafe.Tests -- ag
• System.Threading.Tasks.Extensions.Tests -- s
• System.Reflection.MetadataLoadContext.Tests -- 360 failures -- missing 78 tests -- w
• System.Diagnostics.StackTrace.Tests -- 22 failures -- a
• System.Reflection.Tests -- missing ~1100 tests -- d
• Microsoft.CSharp.Tests -- c, d
• Microsoft.Extensions.Logging.Tests -- n
• Microsoft.VisualBasic.Core.Tests -- am
• System.Dynamic.Runtime.Tests -- an
• System.Linq.Tests -- i
• System.Net.Mail.Functional.Tests -- ac
• System.Private.Runtime.InteropServices.JavaScript.Tests -- ao
• System.Reflection.Context.Tests -- am
• System.Reflection.Emit.ILGeneration.Tests -- c
• System.Reflection.Emit.Tests -- c
• System.Reflection.Metadata.Tests -- c, w, ad, ae
• System.Runtime.Extensions.Tests -- d
• System.Runtime.InteropServices.Tests -- c, ah
• System.Runtime.Loader.Tests -- am
• System.Runtime.Serialization.Json.ReflectionOnly.Tests -- ao
• System.Runtime.Serialization.Xml.ReflectionOnly.Tests -- ao
• System.Text.Json.Tests -- b, c, d
• System.Threading.Tasks.Dataflow.Tests -- ap
• System.Threading.Tasks.Parallel.Tests -- f
• System.Threading.Tasks.Tests -- d
• System.ValueTuple.Tests -- aj
• System.Xml.Linq.xNodeBuilder.Tests -- aq
• System.Xml.XmlSerializer.ReflectionOnly.Tests -- i
• System.Xml.XmlSerializer.Tests -- ao
• System.Xml.Xsl.XslCompiledTransformApi.Tests -- w
• System.Xml.Xsl.XslTransformApi.Tests -- w

Tests failing in CI

• System.Collections.Specialized.Tests -- d
• System.IO.Compression.ZipFile.Tests -- w
• System.Private.Uri.Functional.Tests -- n
• System.Xml.RW.XmlWriterApi.Tests -- aj
• System.Xml.XPath.XmlDocument.Tests -- aj
• System.Xml.XmlDocument.Tests -- w
• System.Xml.XmlResolver.Tests -- w
• System.Xml.RW.XmlSystemPathResolver.Tests -- w