Skip to content

Add support for running RDK tests using starboard/tools/testing/test_runner.py - #5689

Merged
niranjanyardi merged 3 commits into
youtube:25.lts.1+from
andrunko:rdk-test-runner-support
May 23, 2025
Merged

niranjanyardi merged 3 commits into
youtube:25.lts.1+from
andrunko:rdk-test-runner-support

Conversation

@andrunko

@andrunko andrunko commented May 7, 2025

Copy link
Copy Markdown
Collaborator

This updates the current RDK test launcher to use the elf_loader_sandbox to run the tests, similar to other linux platforms, opposed to using a jsonrpc call.

The test launcher script will use SSH/rsync to connect/copy resources to the device, so for it to work the SSH device configuration should be based on https://wiki.rdkcentral.com/display/RDK/Test+on+board, including the default ports listed there.

This PR also includes a small bugfix for the Raspi test launcher when using test_runner.py.

b/415848657

@andrunko
andrunko force-pushed the rdk-test-runner-support branch from 149fc84 to d56da2f Compare May 8, 2025 04:28
@niranjanyardi niranjanyardi added the on_device Trigger on-device tests label May 9, 2025

@niranjanyardi niranjanyardi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we move all files under starboard/rdk to somewhere inside starboard/contrib/rdk ?

@niranjanyardi

Copy link
Copy Markdown
Member

@hlwarriner / @yuying-y can you ptal at the evergreen launcher changes ?

@jonastsai

Copy link
Copy Markdown
Contributor

Two issues I encountered so far with your change:

  1. cannot find ./out/evergreen-arm-hardfp_devel/app_launcher/starboard/tools/testing/test_runner.py
  2. when running nplb, the UI remains on the screen and the nplb video is not displayed (but I can hear the audio).

@niranjanyardi

Copy link
Copy Markdown
Member

Two issues I encountered so far with your change:

  1. cannot find ./out/evergreen-arm-hardfp_devel/app_launcher/starboard/tools/testing/test_runner.py
  2. when running nplb, the UI remains on the screen and the nplb video is not displayed (but I can hear the audio).

@andrunko ptal

Comment thread starboard/rdk/launcher.py Resolved
Comment thread starboard/evergreen/shared/launcher.py Resolved
@andrunko
andrunko force-pushed the rdk-test-runner-support branch from d56da2f to 6c4e8bc Compare May 19, 2025 19:02
@andrunko

Copy link
Copy Markdown
Collaborator Author

can we move all files under starboard/rdk to somewhere inside starboard/contrib/rdk ?

We could but as explained offline, we would need to duplicate basically all files for each platform RDK supports (atm we only support arm, but upstream also supports arm64 and rpi), due to the fact that the RDK sources don't have a fixed location (can change depending on a build var).

Also this script (starboard/rdk/launcher.py) depends on the directory structure defined in starboard/evergreen/shared/launcher.py, which builds the resources used by test_runner (the ones copied to the device which are then run by the launcher).

That said, I would keep them where they are, but please let me know what you think and if needed I can updated.

@andrunko

Copy link
Copy Markdown
Collaborator Author

Two issues I encountered so far with your change:

  1. cannot find ./out/evergreen-arm-hardfp_devel/app_launcher/starboard/tools/testing/test_runner.py

As mentioned elsewhere, please use starboard/tools/testing/test_runner.py directly.

  1. when running nplb, the UI remains on the screen and the nplb video is not displayed (but I can hear the audio).

This should be fixed with latest changes, please let me know of any issue.

@niranjanyardi

Copy link
Copy Markdown
Member

can we move all files under starboard/rdk to somewhere inside starboard/contrib/rdk ?

andre to create a follow up bug here

@hlwarriner hlwarriner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly lgtm but I left a few comments for the evergreen launcher changes.

Comment thread starboard/evergreen/shared/launcher.py Resolved
Comment thread starboard/evergreen/shared/launcher.py Resolved
Comment thread starboard/evergreen/shared/launcher.py Resolved
@andrunko
andrunko force-pushed the rdk-test-runner-support branch 2 times, most recently from 48293bd to a4d1b7d Compare May 20, 2025 18:36

@jonastsai jonastsai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirm the new PR works with my RDK CB25 image. It can run all the unittest which includes NPLB.

@andrunko
andrunko force-pushed the rdk-test-runner-support branch 2 times, most recently from 02aa61b to 63a50ab Compare May 22, 2025 13:07

@hlwarriner hlwarriner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as long as the open comment is addressed before merging - thanks.

Comment thread starboard/evergreen/shared/launcher.py Resolved
Comment thread starboard/evergreen/shared/launcher.py Resolved

@niranjanyardi niranjanyardi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a TODO in starboard/rdk/launcher.py

somewhere at the begnining of the file. maybe around line 33

Something like
// TODO(b/xxxxxxxx): Move all files under starboard/rdk to starboard/contrib/rdk

@andrunko

Copy link
Copy Markdown
Collaborator Author

Please add a TODO in starboard/rdk/launcher.py

somewhere at the begnining of the file. maybe around line 33

Something like // TODO(b/xxxxxxxx): Move all files under starboard/rdk to starboard/contrib/rdk

Ack, will add the TODO - should we say "Consider moving all files..." instead? Given there are the other constraints we discussed previously (e.g. duplicating code when supporting multiple rdk platforms, etc) to weight in before actuall moving it?

andrunko added 3 commits May 22, 2025 22:02
…_runner.py`

This updates the current RDK test launcher to use the `elf_loader_sandbox` to
run the tests, similar to other linux platforms, opposed to using a `jsonrpc` call.

Also update ssh/rsync options to properly connect to devices using a default RDK setup.

b/415848657
…t_runner.py` for raspi tests

The current implementation relies in the `first_run` entry being available
on `globals()`, but this was always true given `globals()` also includes
method names which was the case (i.e. `first_run()`), so let's rename it.

b/415848657
This should fix tests that require video playback.

b/415848657
@andrunko
andrunko force-pushed the rdk-test-runner-support branch from 63a50ab to 9644561 Compare May 23, 2025 01:02
@andrunko

Copy link
Copy Markdown
Collaborator Author

I believe I've addressed all feedback, please let me know if I missed something.

@niranjanyardi
niranjanyardi merged commit 89a8e77 into youtube:25.lts.1+ May 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

on_device Trigger on-device tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants