Skip to content

[Xamarin.Android.Build.Tests] Add support for testing the UI #2855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jonpryor merged 6 commits into from
Apr 16, 2019
Merged

[Xamarin.Android.Build.Tests] Add support for testing the UI #2855

jonpryor merged 6 commits into from
Apr 16, 2019

Conversation

@dellis1972
Copy link
Contributor

@dellis1972 dellis1972 commented Mar 20, 2019 •
edited
Loading

Context #2608

This commit adds support for testing that the
app launches on device AND can be interacted
with. It does this by making use of a few features
of adb.

First is adb input. This allow us to send taps and
clicks to the device. It simulates user input. So
we can use this to say click a button.

Next up is the uiautomator this allows use to get
the complete ui layout for the active activity.
Using

exec-out uiautomator dump /dev/tty

we can send this data to the console. As it is
xml we can then parse it looking for our ui element
we want. We can then get the bounds of the control
and pass that information to adb input tap.

This commit also adds support for testing that
the timezone set on a device is correctly picked
up by mono.

And finally we add support for testing that the
debugger can actaully connect to the application.
These tests are only run if the commerial parts
of the product are available.

Seeing as all of these new tests REQUIRE the
emulator or device they have been split out into
a new MSBuildDeviceIntegration project. This
is built as part of the main solution. However
it is run as part of the make run-apk-tests
call. This is so we can be sure the emulator is
running.

In future any MSBuild related tests that REQUIRE
a device should be placed in this new project.

@dellis1972 dellis1972 added the do-not-merge PR should not be merged. label Mar 20, 2019
@dellis1972 dellis1972 force-pushed the addondevicetesting branch 3 times, most recently from 83d7e15 to 3da30ec Compare April 1, 2019 09:01
@dellis1972 dellis1972 removed the do-not-merge PR should not be merged. label Apr 1, 2019
@dellis1972 dellis1972 force-pushed the addondevicetesting branch from f13fd74 to efdfb57 Compare April 2, 2019 10:01
@dellis1972 dellis1972 marked this pull request as ready for review April 2, 2019 10:51
jonathanpeppers
jonathanpeppers reviewed Apr 2, 2019
View reviewed changes
@dellis1972 dellis1972 force-pushed the addondevicetesting branch from d8857ba to 2044732 Compare April 3, 2019 10:25
@jonpryor
Copy link
Contributor

jonpryor commented Apr 3, 2019

Conceptual question: for this to work, there needs to be an Android device available on which to run the test and manipulate the UI.

Currently, unless I've forgotten something, Xamarin.Android.Build.Tests.dll doesn't consistently execute in an environment in which the emulator is running; the emulator is currently created as part of the RunApkTests target, which is run in parallel with the RunNUnitTests target. There is no guaranteed reliability that the emulator will -- or will not! -- be running when the local NUnit tests are run.

Perhaps we're just really lucky so far?

@dellis1972 dellis1972 force-pushed the addondevicetesting branch 3 times, most recently from b7b1dd7 to bc6ba03 Compare April 9, 2019 12:26
@dellis1972 dellis1972 force-pushed the addondevicetesting branch 4 times, most recently from 0bca494 to 5e6d395 Compare April 10, 2019 13:21
jonpryor
jonpryor reviewed Apr 10, 2019
View reviewed changes
@dellis1972 dellis1972 force-pushed the addondevicetesting branch 2 times, most recently from 5a05ee3 to 953ebc5 Compare April 10, 2019 19:42
jonpryor
jonpryor reviewed Apr 11, 2019
View reviewed changes
jonpryor
jonpryor reviewed Apr 11, 2019
View reviewed changes
@jonpryor
Copy link
Contributor

jonpryor commented Apr 11, 2019

Ouch: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/517/testReport/Xamarin.Android.Build.Tests.DeploymentTest/CheckTimeZoneInfoIsCorrect/__Australia_NSW_____Release/

System.InvalidOperationException : Cannot process request because the process (87383) has exited.

  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6d83221bea054562936f2f38bd41deeb>:0

@dellis1972
Copy link
Contributor Author

dellis1972 commented Apr 11, 2019

Test failures are as follows

Test Result (5 failures / -1)
SslTest.Mono.Android_Tests, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
SslTest.Mono.Android_TestsMultiDex, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
SslTest.Mono.Android_TestsAppBundle, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
AndroidClientHandlerTests.Mono.Android_Tests, Xamarin.Android.NetTests.AndroidClientHandlerTests.Sanity_Tls_1_2_Url_WithMonoClientHandlerFails / Release
AndroidClientHandlerTests.Mono.Android_TestsAppBundle, Xamarin.Android.NetTests.AndroidClientHandlerTests.Sanity_Tls_1_2_Url_WithMonoClientHandlerFails / Release

@dellis1972
Copy link
Contributor Author

dellis1972 commented Apr 11, 2019

Ouch: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/517/testReport/Xamarin.Android.Build.Tests.DeploymentTest/CheckTimeZoneInfoIsCorrect/__Australia_NSW_____Release/

System.InvalidOperationException : Cannot process request because the process (87383) has exited.

  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6d83221bea054562936f2f38bd41deeb>:0

That is a very very weird crash. Maybe its from the nunit runner... Its an actual test crash.

@dellis1972
[Xamarin.Android.Build.Tests] Add support for testing the UI
fb81f6a 
This commit adds support for testing that the
app launches on device AND can be interacted
with. It does this by making use of a few features
of `adb`.

First is `adb input`. This allow us to send taps and
clicks to the device. It simulates user input. So
we can use this to say click a button.

Next up is the `uiautomator` this allows use to get
the complete ui layout for the active activity.
Using

	exec-out uiautomator dump /dev/tty

we can send this data to the console. As it is
xml we can then parse it looking for our ui element
we want. We can then get the bounds of the control
and pass that information to `adb input tap`.

This commit also adds support for testing that
the timezone set on a device is correctly picked
up by mono.

And finally we add support for testing that the
debugger can actaully connect to the application.
These tests are only run if the commerial parts
of the product are available.

Seeing as all of these new tests REQUIRE the
emulator or device they have been split out into
a new `MSBuildDeviceIntegration` project. This
is built as part of the main solution. However
it is run as part of the `make run-apk-tests`
call. This is so we can be sure the emulator is
running.

In future any MSBuild related tests that REQUIRE
a device should be placed in this new project.
@dellis1972
Copy link
Contributor Author

dellis1972 commented Apr 15, 2019

Not sure why this test failed https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/569/testReport/junit/Mono/Android_TestsMultiDex/Possible_Crash___Release/

It does not seem to be related to this PR.

@dellis1972
Copy link
Contributor Author

dellis1972 commented Apr 15, 2019

Commercial build failure is

 EXEC : error : option --boot-class-path not allowed with target 11

looks like that java version problem

@jonpryor
Copy link
Contributor

jonpryor commented Apr 16, 2019

@dellis1972 asked:

Not sure why this test failed: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/569/testReport/junit/Mono/Android_TestsMultiDex/Possible_Crash___Release/

The problem is described in commit a972624:

We have been seeing adb crash during RunApkTests:

If you download xa-test-results-v9.2.99.163_Darwin-x86_64_pr_07dd6aee-Release.zip, then bin/TestRelease/msbuild-20190415T070436-Target-RunApkTests.binlog contains:

Task "RunInstrumentationTests"
  Task Parameter:AdbTarget=-s emulator-5570
  Task Parameter:LogLevel=Verbose
  Task Parameter:PackageName=Mono.Android_TestsMultiDex
  Task Parameter:Component=Mono.Android_TestsMultiDex/xamarin.android.runtimetests.TestInstrumentation
  Task Parameter:NUnit2TestResultsFile=…bin/TestRelease/TestResult-Mono.Android_TestsMultiDex.xml
  Task Parameter:LogcatFilename=…/bin/TestRelease/logcat-Release-Aot-Mono.Android_TestsMultiDex.txt
  Task Parameter:ToolExe=adb
  Task Parameter:ToolPath=/Users/builder/android-toolchain/sdk/platform-tools
  Task Parameter:Timeout=3600000
  Executing: /Users/builder/android-toolchain/sdk/platform-tools/adb -s emulator-5570  shell am instrument  -e "loglevel Verbose" -w "Mono.Android_TestsMultiDex/xamarin.android.runtimetests.TestInstrumentation"
  Executing: /Users/builder/android-toolchain/sdk/platform-tools/adb -s emulator-5570  logcat -v threadtime -d
    appending stdout to file: …/bin/TestRelease/logcat-Release-Aot-Mono.Android_TestsMultiDex.txt
/Users/builder/jenkins/workspace/xamarin-android-pr-pipeline-release/xamarin-android/build-tools/scripts/TestApks.targets(226,5): warning : * daemon not running; starting now at tcp:5037

i.e. adb is "not running" when it most certainly should have already been running, i.e. it crashed.

Looks like bumping to platform-tools 28.0.2 didn't entirely fix things. :-(

@jonpryor jonpryor merged commit bf58a88 into dotnet:master Apr 16, 2019
@jonathanpeppers
Copy link
Member

jonathanpeppers commented Apr 16, 2019

Ah! we merged this and it breaks the Windows build...

external\debugger-libs\Mono.Debugging.Soft\SoftDebuggerSession.cs(2606,30): Error CS0234: The type or namespace name 'Unix' does not exist in the namespace 'Mono' (are you missing an assembly reference?)
Process 'msbuild.exe' exited with code '1'.

jonpryor pushed a commit that referenced this pull request Apr 16, 2019
@dellis1972 @jonpryor
[MSBuildDeviceIntegration] Fix build error when Cecil libs (#2985)
8f2c598 
Commit 66d0165 [broke the macOS build][0]:

	…/xamarin-android/tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.targets(18,5): error MSB3030: Could not copy the file "../../bin/TestRelease/Xamarin.Android.Cecil.Mdb.dll" because it was not found.

This was probably "overlooked" because the builds for PR #2980 didn't
include bf58a88/PR #2855.

Update the paths used for `Xamarin.Android.Cecil*.dll` so that
`MSBuildDeviceIntegration.targets` can run without error.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/463/
@github-actions github-actions bot locked and limited conversation to collaborators Jan 31, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@jonathanpeppers jonathanpeppers jonathanpeppers left review comments

+1 more reviewer

@jonpryor jonpryor jonpryor left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dellis1972 @jonpryor @jonathanpeppers