E2E CI V2 by EQuimper · Pull Request #6713 · rainbow-me/rainbow

EQuimper · 2025-06-26T19:04:44Z

Fixes APP-2922

What changed (plus any additional context for devs)

Speed up build by reusing artifacts with RNEF github action. Native build are now reused if the dependencies did not change. Then the JS is re-bundled and replaced in the previously compiled binary.
Rewrite the e2e test runner to support sharding across jobs. The idea here is to have one job build the app, then multiple jobs each pull the app and run a subset of tests. The maestro test runner (running maestro command on a folder) doesn't support this so I wrote a script that will discover the tests and run maestro individually on each test file. This runner supports sharding by passing a shard number and the total number of shards. It will then only run tests at <test_index> % <total_shards> == <shard_index>. This allows also better control of logs structure and retries. Retries are now handled by the script and no longer need retry blocks in all tests.
Move iOS tests to github hosted runners to leverage concurrency. We only have 3 hosted runners so this allows having proper sharding on iOS too. Need to discuss costs first before moving forward with that one.

Screen recordings / screenshots

N/A

What to test

CI passes FAST

socket-security · 2025-06-27T14:28:57Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/@rnef/platform-ios@0.8.11
	npm/@rnef/config@0.8.11
	npm/@rnef/provider-github@0.8.11
	npm/@rnef/plugin-metro@0.8.11
	npm/@rnef/tools@0.8.11
	npm/@rnef/platform-android@0.8.11
	npm/@rnef/platform-apple-helpers@0.8.11
	npm/@types/adm-zip@0.5.7
	npm/@clack/core@0.5.0
	npm/@rnef/cli@0.8.11
	npm/adm-zip@0.5.16
	npm/ts-regex-builder@1.8.2
	npm/nano-spawn@0.2.1
	npm/@clack/prompts@0.11.0

View full report

jinchung · 2025-07-07T16:14:36Z

Launch in simulator or device for cba0b4f

jinchung · 2025-07-08T03:13:25Z

Launch in simulator or device for 5b369eb

jinchung · 2025-07-08T15:27:04Z

Launch in simulator or device for b070c7f

jinchung · 2025-07-29T19:51:28Z

Launch in simulator or device for cec8af1

jinchung · 2025-08-01T21:58:25Z

Launch in simulator or device for 24229be

jinchung · 2025-08-05T06:30:37Z

Launch in simulator or device for ed992c5

jinchung · 2025-08-08T04:00:54Z

Launch in simulator or device for 6d73f17

WoLewicki

Left some comments

WoLewicki · 2025-08-12T15:51:54Z

-    timeout-minutes: 90
+  android-e2e-build:
+    if: github.event.pull_request.draft == false
+    runs-on: blacksmith-8vcpu-ubuntu-2404


Is this machine considered the best option for this type of jobs?

Yes, tested different sizes and this one is most cost effective. However for running the tests we found that the blacksmith runners would crash / disconnect more often when running android emulators so we're using github hosted ones for now. Ideally we'd use these runners for everything, but can't currently.

WoLewicki · 2025-08-12T15:53:48Z

      - name: Checkout repo
        uses: actions/checkout@v4

-      - name: Setup Java environment


Guess the whole environment is set up by the chosen machine?

The RNEF github actions does it internally if I remember correctly.

WoLewicki · 2025-08-12T15:57:51Z

+      DEBUG: false
+      SHARD_TOTAL: 4
+      SHARD_INDEX: ${{ matrix.shard-index }}
+      ANDROID_EMULATOR_API_LEVEL: 31


Don't we want the newest ones? e.g. API 35 pixel_9? Or is it not available?

From our testing newer versions are less stable and randomly crash or disconnect a lot more.

WoLewicki · 2025-08-12T16:25:41Z


-      - name: Version debug
-        run: |
-          npx react-native info


Is it connected to this PR?

Earlier versions removed the community CLI for RNEF CLI, but decided to keep both. Could revert this change.

Haven't seen any problems in another project using RNEF but its up to you

Co-authored-by: Wojciech Lewicki <[email protected]>

janicduplessis · 2025-08-13T02:10:48Z

@WoLewicki I think I addressed all your comments and fixed minor conflict in podfile.lock

jinchung · 2025-08-13T04:42:48Z

Launch in simulator or device for c139783

WoLewicki · 2025-08-13T15:14:07Z

I'd approve it but CI is failing.

janicduplessis · 2025-08-13T15:25:34Z

Looks like something broke when RNEF needs to do a full rebuild, I tried bumping the version to see if it fixes it. Looks like this Run OUTPUT=$(npx rnef remote-cache list --name rnef-***-release-9a6f61ff7d935c23bbf89e7e73358ec4b9f39604 --json) returns non-zero exit code now which makes the action fail.

janicduplessis · 2025-08-13T15:49:41Z

Not sure what happened, but it worked now. Should be good to go once tests pass!

WoLewicki · 2025-08-13T15:52:49Z

Hmm so I think we should trigger some clean build again, should e.g. bumping some native dependency be enough?

WoLewicki · 2025-08-13T15:53:00Z

Oh, ios timed out?

janicduplessis · 2025-08-13T15:59:26Z

Hmm so I think we should trigger some clean build again, should e.g. bumping some native dependency be enough?

I checked the build on android and it did a full build again, I saw the exact same code path that errored in the previous code execute with no problem :/

janicduplessis · 2025-08-13T16:00:21Z

Bumped timeout on iOS, I think combinaison of using github hosted runners + the arm change might make builds a bit slower. It failed literally at the last second when it was uploading the ipa :( Added 15 min to be safe.

janicduplessis · 2025-08-13T16:45:01Z

@WoLewicki All passed! iOS build is still pending, but this doesn't affect that one.

WoLewicki · 2025-08-13T16:52:48Z

I checked the build on android and it did a full build again, I saw the exact same code path that errored in the previous code execute with no problem :/

So do you think we should go forward and merge it then? If so, I'm up for it too and we can handle some problems in follow-ups.

janicduplessis · 2025-08-13T16:55:18Z

Yea I think it is fine, I’ll fix any issues we find in follow ups

jinchung · 2025-08-13T18:38:50Z

Launch in simulator or device for 91900f5

* Record e2e test failures * Cleanup

jinchung · 2025-08-14T02:10:42Z

Launch in simulator or device for 42e6c00

jinchung · 2025-08-14T15:35:42Z

Launch in simulator or device for e2eb60f

janicduplessis · 2025-08-18T21:02:25Z

@jinchung Are we good to merge this?

* Use RNEF CLI * Move tests to flow folder * Remove retry blocks * Fix * Update e2e/README.md Co-authored-by: Wojciech Lewicki <[email protected]> * fixes * Try bumping rnef * Increase timeout * Record e2e test failures (#6796) * Record e2e test failures * Cleanup * Set xcode version --------- Co-authored-by: Janic Duplessis <[email protected]> Co-authored-by: Wojciech Lewicki <[email protected]>

@janicduplessis

The `Ensure Simulator is fully booted` step in `ios-e2e.yml` polls `xcrun simctl list | grep "Booted"` with a 120-second timeout. The `simulator-action` that creates the simulator already supports a `wait_for_boot: true` input that uses Apple's `simctl bootstatus` internally, which is the supported mechanism for detecting simulator readiness. The custom loop was added in #6713 (Aug 2025) during the GH Actions migration with no documented rationale; presumably the action's native wait was simply overlooked at the time. We noticed the grep-based detection is fragile while working on the iOS 26 SDK migration. An iOS 26.4 simulator booted in ~36 seconds, yet our custom loop timed out at 120s because the `grep 'Booted'` pattern didn't match reliably during the boot transition. Swapping to the action's native wait fixed it cleanly in that context. This change backports the combined swap to develop's current iPhone 16 / iOS 18.5 setup. Behavior should be equivalent on green runs and more robust on slow boots. @janicduplessis pointed out on this PR that the built-in wait has historically been flaky: `simctl bootstatus` sometimes returns while the simulator isn't actually ready, or hangs outright. We actually observed that hang pattern on one shard here. To combat this the change also bumps the simulator-action from v4 to v5, which addresses this exact class of flake. v5 ships `boot_timeout_seconds: 360` and `boot_retries: 2` as defaults (see [#563](futureware-tech/simulator-action#563)), so a hung `bootstatus` call now bails after 6 minutes and retries up to twice before failing the step. No explicit config needed on our side. Ref FEPLAT-81.

janicduplessis mentioned this pull request Jul 2, 2025

Run e2e android tests on github runner #6712

Merged

janicduplessis force-pushed the eq/rnef-2 branch 5 times, most recently from 65f5c32 to a935ff2 Compare July 4, 2025 16:43

janicduplessis force-pushed the @janic/upgrade-rn branch 4 times, most recently from 47b83a0 to 31a3a8f Compare July 18, 2025 14:22

EQuimper force-pushed the eq/rnef-2 branch from 67ea73f to b5949f8 Compare July 28, 2025 16:18

janicduplessis force-pushed the eq/rnef-2 branch from cbde820 to 39ba534 Compare August 1, 2025 20:41

Base automatically changed from @janic/upgrade-rn to develop August 5, 2025 01:25

janicduplessis force-pushed the eq/rnef-2 branch 4 times, most recently from c8f6be9 to 9386443 Compare August 5, 2025 03:00

EQuimper and others added 3 commits August 7, 2025 21:28

Use RNEF CLI

f35593d

Move tests to flow folder

8294b37

Remove retry blocks

2f8c636

janicduplessis force-pushed the eq/rnef-2 branch from e6f04d9 to c427f43 Compare August 8, 2025 01:34

rainbow-me deleted a comment from github-actions Bot Aug 8, 2025

janicduplessis requested a review from WoLewicki August 11, 2025 16:11

WoLewicki reviewed Aug 12, 2025

View reviewed changes

janicduplessis and others added 3 commits August 12, 2025 12:54

Update e2e/README.md

0b14dd0

Co-authored-by: Wojciech Lewicki <[email protected]>

fixes

2dfaa3d

Merge remote-tracking branch 'origin/develop' into eq/rnef-2

b894095

Try bumping rnef

2d4f5ed

Increase timeout

bb1e0fb

Record e2e test failures (#6796)

bfc2022

* Record e2e test failures * Cleanup

Set xcode version

8c83cc7

jinchung merged commit 65e1902 into develop Aug 18, 2025

jinchung deleted the eq/rnef-2 branch August 18, 2025 22:28

olerass mentioned this pull request Apr 21, 2026

ci: drop custom simulator boot-wait loop for iOS e2e #7375

Merged

Conversation

EQuimper commented Jun 26, 2025 • edited by janicduplessis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed (plus any additional context for devs)

Screen recordings / screenshots

What to test

Uh oh!

socket-security Bot commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jinchung commented Jul 7, 2025

Uh oh!

jinchung commented Jul 8, 2025

Uh oh!

jinchung commented Jul 8, 2025

Uh oh!

jinchung commented Jul 29, 2025

Uh oh!

jinchung commented Aug 1, 2025

Uh oh!

jinchung commented Aug 5, 2025

Uh oh!

jinchung commented Aug 8, 2025

Uh oh!

WoLewicki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janicduplessis commented Aug 13, 2025

Uh oh!

jinchung commented Aug 13, 2025

Uh oh!

WoLewicki commented Aug 13, 2025

Uh oh!

janicduplessis commented Aug 13, 2025

Uh oh!

janicduplessis commented Aug 13, 2025

Uh oh!

WoLewicki commented Aug 13, 2025

Uh oh!

WoLewicki commented Aug 13, 2025

Uh oh!

janicduplessis commented Aug 13, 2025

Uh oh!

janicduplessis commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janicduplessis commented Aug 13, 2025

Uh oh!

WoLewicki commented Aug 13, 2025

Uh oh!

janicduplessis commented Aug 13, 2025

Uh oh!

jinchung commented Aug 13, 2025

Uh oh!

jinchung commented Aug 14, 2025

Uh oh!

EQuimper commented Jun 26, 2025 •

edited by janicduplessis

Loading

socket-security Bot commented Jun 27, 2025 •

edited

Loading

janicduplessis commented Aug 13, 2025 •

edited

Loading