Skip to content

Improve WASM test output via xharness #42860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
akoeplinger merged 1 commit into from
Oct 5, 2020
Merged

Improve WASM test output via xharness #42860

akoeplinger merged 1 commit into from
Oct 5, 2020

Conversation

@akoeplinger
Copy link
Member

@akoeplinger akoeplinger commented Sep 29, 2020 •
edited
Loading

With dotnet/xharness#315 we can now make the test output similar to the regular xunit desktop runner.

It no longer prints the passed tests by default, only failing tests. To get the old behavior, add the -v flag back to xharness in tests.mobile.targets or check the new wasm-console.log file in the xharness-output folder.

  XHarness command issued: wasm test --app=. --engine=V8 --engine-arg=--stack-trace-limit=1000 --js-file=runtime.js --output-directory=/Users/alexander/dev/runtime/artifacts/bin/System.Runtime.Handles.Tests/net5.0-Release/browser-wasm/AppBundle/xharness-output -- --run WasmTestRunner.dll System.Runtime.Handles.Tests.dll -notrait category=OuterLoop -notrait category=failing
  info: 21:57:45.3244690 v8 --expose_wasm --stack-trace-limit=1000 runtime.js -- --run WasmTestRunner.dll System.Runtime.Handles.Tests.dll -notrait category=OuterLoop -notrait category=failing
  info: Arguments: --run,WasmTestRunner.dll,System.Runtime.Handles.Tests.dll,-notrait,category=OuterLoop,-notrait,category=failing
  info: console.debug: MONO_WASM: Initializing mono runtime
  info: console.debug: MONO_WASM: ICU data archive(s) loaded, disabling invariant mode
  info: console.debug: mono_wasm_runtime_ready fe00e07a-5519-4dfe-b35a-f867dbaf2e28
  info: Initializing.....
  info: Discovering: System.Runtime.Handles.Tests.dll (method display = ClassAndMethod, method display options = None)
  info: Discovered:  System.Runtime.Handles.Tests.dll (found 14 of 15 test cases)
  info: Starting:    System.Runtime.Handles.Tests.dll
  fail: [FAIL] SafeWaitHandleTests.SafeWaitHandle_False
  info: Assert.False() Failure
  info: Expected: False
  info: Actual:   True
  info:    at SafeWaitHandleTests.SafeWaitHandle_False()
  info:    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
  info: Finished:    System.Runtime.Handles.Tests.dll
  info:
  info: === TEST EXECUTION SUMMARY ===
  info: Total: 14, Errors: 0, Failed: 1, Skipped: 0, Time: 0.097211s
  info:
  info: 21:57:46.3323620 Process exited with 1

Fixes #42746

/cc @CoffeeFlux @safern

@ghost
Copy link

ghost commented Sep 29, 2020

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

CoffeeFlux
CoffeeFlux approved these changes Sep 29, 2020
View reviewed changes
Copy link
Contributor

@CoffeeFlux CoffeeFlux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be held back for the monthly infra rollout? If not, LGTM, thanks so much!

@akoeplinger
Copy link
Member Author

akoeplinger commented Sep 29, 2020

Does this need to be held back for the monthly infra rollout? If not, LGTM, thanks so much!

No, the monthly infra rollout is only when we require a new dotnet SDK. xharness is just a package that gets downloaded during the build :)

safern
safern approved these changes Sep 29, 2020
View reviewed changes
Copy link
Member

@safern safern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, the browser legs, timed out, might be related to the change?

@akoeplinger
Copy link
Member Author

akoeplinger commented Sep 30, 2020

@safern no, that was because of an infra issue: https://github.com/dotnet/core-eng/issues/11001

@runfoapp runfoapp bot mentioned this pull request Sep 30, 2020
Closed
@safern
Copy link
Member

safern commented Sep 30, 2020

Hmm, interesting:

All tests are failing with:

�[41m�[30mfail�[39m�[22m�[49m: Microsoft.AspNetCore.Server.Kestrel[0]
      Heartbeat.OnHeartbeat
      System.TypeLoadException: Could not load type 'Microsoft.Extensions.Primitives.InplaceStringBuilder' from assembly 'Microsoft.Extensions.Primitives, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
         at Microsoft.Net.Http.Headers.DateTimeFormatter.ToRfc1123String(DateTimeOffset dateTime, Boolean quoted)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.DateHeaderValueManager.SetDateValues(DateTimeOffset value)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.OnHeartbeat()
�[41m�[30mfail�[39m�[22m�[49m: Microsoft.AspNetCore.Server.Kestrel[0]

@akoeplinger
Copy link
Member Author

akoeplinger commented Sep 30, 2020 •
edited
Loading

Only the browser wasm tests which rely on Kestrel. I think it's a mismatch with the aspnet version that is getting used (I updated Microsoft.Extensions packages to 5.0 in xharness), looking.

@premun
Copy link
Member

premun commented Oct 1, 2020

Btw the broken characters can be fixed by setting the XHARNESS_DISABLE_COLORED_OUTPUT env var: https://github.com/dotnet/xharness#other-settings

@akoeplinger akoeplinger force-pushed the wasm-logging branch 2 times, most recently from 46c081f to 09f8a3d Compare October 2, 2020 21:47
@akoeplinger
Improve WASM test output via xharness
225f013 
With dotnet/xharness#315 we can now make the test output similar to the regular xunit desktop runner.
It no longer prints the passed tests by default, only failing tests. To get the old behavior, add the `-v` flag back to xharness in tests.mobile.targets or check the new wasm-console.log file in the xharness-output folder.

```
  XHarness command issued: wasm test --app=. --engine=V8 --engine-arg=--stack-trace-limit=1000 --js-file=runtime.js --output-directory=/Users/alexander/dev/runtime/artifacts/bin/System.Runtime.Handles.Tests/net5.0-Release/browser-wasm/AppBundle/xharness-output -- --run WasmTestRunner.dll System.Runtime.Handles.Tests.dll -notrait category=OuterLoop -notrait category=failing
  info: 21:57:45.3244690 v8 --expose_wasm --stack-trace-limit=1000 runtime.js -- --run WasmTestRunner.dll System.Runtime.Handles.Tests.dll -notrait category=OuterLoop -notrait category=failing
  info: Arguments: --run,WasmTestRunner.dll,System.Runtime.Handles.Tests.dll,-notrait,category=OuterLoop,-notrait,category=failing
  info: console.debug: MONO_WASM: Initializing mono runtime
  info: console.debug: MONO_WASM: ICU data archive(s) loaded, disabling invariant mode
  info: console.debug: mono_wasm_runtime_ready fe00e07a-5519-4dfe-b35a-f867dbaf2e28
  info: Initializing.....
  info: Discovering: System.Runtime.Handles.Tests.dll (method display = ClassAndMethod, method display options = None)
  info: Discovered:  System.Runtime.Handles.Tests.dll (found 14 of 15 test cases)
  info: Starting:    System.Runtime.Handles.Tests.dll
  fail: [FAIL] SafeWaitHandleTests.SafeWaitHandle_False
  info: Assert.False() Failure
  info: Expected: False
  info: Actual:   True
  info:    at SafeWaitHandleTests.SafeWaitHandle_False()
  info:    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
  info: Finished:    System.Runtime.Handles.Tests.dll
  info:
  info: === TEST EXECUTION SUMMARY ===
  info: Total: 14, Errors: 0, Failed: 1, Skipped: 0, Time: 0.097211s
  info:
  info: 21:57:46.3323620 Process exited with 1
```
@akoeplinger akoeplinger merged commit bd33fc4 into dotnet:master Oct 5, 2020
@akoeplinger akoeplinger deleted the wasm-logging branch October 5, 2020 10:37
@akoeplinger akoeplinger mentioned this pull request Oct 5, 2020
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@safern safern safern approved these changes

+1 more reviewer

@CoffeeFlux CoffeeFlux CoffeeFlux approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area-Infrastructure-mono

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[wasm] Simplify logging output for readability

5 participants

@akoeplinger @safern @premun @CoffeeFlux @Dotnet-GitSync-Bot