Skip to content

Commit 497a54d

Browse files
thymikeeclaude
andauthored
test: expand replay coverage and CI reporting with JUnit (#317)
* test: add benchmark .ad scripts for iOS and Android (02-06) Add 10 new replay scripts covering deep navigation, scroll discovery, text input & keyboard, app lifecycle, and swipe gestures for both iOS simulator and Android emulator. Each script combines multiple command types (snapshot, click, wait, is, find, get, scroll, scrollintoview, swipe, back, home, keyboard, screenshot, appstate) to provide comprehensive baseline coverage for version-over-version comparison. https://claude.ai/code/session_01VkHLzVc7NULQFaWHnhuBCm * test: move replay benchmarks out of CI suite * test: trim replay benchmark script comments * test: relaunch replay app opens * fix: stop focusing iOS simulator for screenshots * fix: open Simulator app on cold boot * test: trim iOS cold boot coverage * test: move replay benchmarks into integration suites * ci: align Android emulator with local setup * ci: raise iOS replay daemon timeout * test: add replay junit reports * ci: use supported Android emulator profile * fix: stabilize replay cleanup and Android CI flows * test: trim flaky replay coverage * fix: exclude replay tests from daemon request timeout * test: relax android scroll discovery replay * refactor: rename ios simulator app opener * ci: remove obsolete android replay timeout override * test: remove duplicate android replay opens * test: relax ios swipe replay assertion * ci: move full replay suites to nightly * ci: extract shared replay workflow setup * ci: fail nightly replay jobs on test errors --------- Co-authored-by: Claude <[email protected]>
1 parent 40425fc commit 497a54d

35 files changed

Lines changed: 1005 additions & 196 deletions
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: "Boot iOS Test Simulator"
2+
description: "Resolve and boot the preferred iOS simulator for replay tests"
3+
4+
inputs:
5+
runtime-version:
6+
description: "Preferred iOS runtime version"
7+
required: true
8+
preferred-device-name:
9+
description: "Preferred simulator device name"
10+
required: false
11+
default: "iPhone 17 Pro"
12+
13+
runs:
14+
using: "composite"
15+
steps:
16+
- name: Resolve and boot iOS test simulator
17+
run: |
18+
set -euo pipefail
19+
RUNTIME_TOKEN="SimRuntime.iOS-${{ inputs.runtime-version }}"
20+
export RUNTIME_TOKEN
21+
export PREFERRED_DEVICE_NAME="${{ inputs.preferred-device-name }}"
22+
UDID="$(
23+
xcrun simctl list devices -j | node -e '
24+
const fs = require("node:fs");
25+
const runtimeToken = process.env.RUNTIME_TOKEN;
26+
const preferredDeviceName = process.env.PREFERRED_DEVICE_NAME;
27+
const payload = JSON.parse(fs.readFileSync(0, "utf8"));
28+
const entries = Object.entries(payload.devices ?? {});
29+
const iosRuntimes = entries.filter(([runtime]) => runtime.includes("SimRuntime.iOS-"));
30+
const runtimeMatches = iosRuntimes.filter(([runtime]) => runtime.includes(runtimeToken));
31+
const pool = runtimeMatches.length > 0 ? runtimeMatches : iosRuntimes;
32+
const available = pool.flatMap(([runtime, devices]) =>
33+
(devices ?? [])
34+
.filter((device) => device.isAvailable)
35+
.map((device) => ({ ...device, runtime })),
36+
);
37+
const preferred =
38+
available.find((device) => device.state === "Booted" && device.name === preferredDeviceName) ??
39+
available.find((device) => device.name === preferredDeviceName) ??
40+
available.find((device) => device.state === "Booted") ??
41+
available[0];
42+
if (!preferred?.udid) process.exit(1);
43+
process.stdout.write(preferred.udid);
44+
'
45+
)"
46+
xcrun simctl shutdown all || true
47+
xcrun simctl boot "$UDID" || true
48+
xcrun simctl bootstatus "$UDID" -b
49+
shell: bash
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: "Setup Android Replay Host"
2+
description: "Prepare the Android CI host for replay tests and expose the agent-device home path"
3+
4+
outputs:
5+
agent-home-dir:
6+
description: "Resolved agent-device home directory"
7+
value: ${{ steps.agent-home.outputs.dir }}
8+
9+
runs:
10+
using: "composite"
11+
steps:
12+
- name: Resolve agent-device home
13+
id: agent-home
14+
run: echo "dir=$HOME/.agent-device" >> "$GITHUB_OUTPUT"
15+
shell: bash
16+
17+
- name: Enable KVM
18+
run: |
19+
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
20+
sudo udevadm control --reload-rules
21+
sudo udevadm trigger --name-match=kvm
22+
shell: bash
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
name: "Setup Apple Replay"
2+
description: "Resolve shared Apple replay cache/build state and expose the agent-device home path"
3+
4+
inputs:
5+
derived-path:
6+
description: "Derived data path used by the replay runner"
7+
required: true
8+
cache-key-prefix:
9+
description: "Prefix for the cache key"
10+
required: true
11+
cache-key-suffix:
12+
description: "Optional suffix for the cache key"
13+
required: false
14+
default: ""
15+
build-command:
16+
description: "Command used to build replay artifacts when the cache is cold"
17+
required: true
18+
xcuitest-platform:
19+
description: "Optional AGENT_DEVICE_XCUITEST_PLATFORM value"
20+
required: false
21+
default: ""
22+
xcuitest-destination:
23+
description: "Optional AGENT_DEVICE_XCUITEST_DESTINATION value"
24+
required: false
25+
default: ""
26+
clean-derived:
27+
description: "Optional AGENT_DEVICE_IOS_CLEAN_DERIVED value"
28+
required: false
29+
default: ""
30+
31+
outputs:
32+
agent-home-dir:
33+
description: "Resolved agent-device home directory"
34+
value: ${{ steps.agent-home.outputs.dir }}
35+
cache-hit:
36+
description: "Whether the replay cache was restored"
37+
value: ${{ steps.restore-prebuilt.outputs.cache-hit }}
38+
39+
runs:
40+
using: "composite"
41+
steps:
42+
- name: Resolve Xcode cache key
43+
id: xcode
44+
run: |
45+
set -euo pipefail
46+
XCODE_VERSION="$(xcodebuild -version | tr '\n' ' ' | sed -E 's/[[:space:]]+/ /g; s/[[:space:]]$//')"
47+
XCODE_KEY="$(echo "$XCODE_VERSION" | tr ' ' '-' | tr -cd '[:alnum:]._-')"
48+
echo "key=$XCODE_KEY" >> "$GITHUB_OUTPUT"
49+
shell: bash
50+
51+
- name: Resolve prebuild source hash
52+
id: source-hash
53+
run: |
54+
set -euo pipefail
55+
echo "value=${{ hashFiles('ios-runner/**', 'package.json', 'pnpm-lock.yaml') }}" >> "$GITHUB_OUTPUT"
56+
shell: bash
57+
58+
- name: Cache replay prebuilt
59+
id: restore-prebuilt
60+
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.2.3
61+
with:
62+
path: ${{ inputs.derived-path }}
63+
key: ${{ inputs.cache-key-prefix }}-${{ steps.xcode.outputs.key }}${{ inputs.cache-key-suffix }}-${{ steps.source-hash.outputs.value }}
64+
65+
- name: Resolve agent-device home
66+
id: agent-home
67+
run: echo "dir=$HOME/.agent-device" >> "$GITHUB_OUTPUT"
68+
shell: bash
69+
70+
- name: Build replay artifacts
71+
if: steps.restore-prebuilt.outputs.cache-hit != 'true'
72+
run: ${{ inputs.build-command }}
73+
shell: bash
74+
env:
75+
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ inputs.derived-path }}
76+
AGENT_DEVICE_XCUITEST_PLATFORM: ${{ inputs.xcuitest-platform }}
77+
AGENT_DEVICE_XCUITEST_DESTINATION: ${{ inputs.xcuitest-destination }}
78+
AGENT_DEVICE_IOS_CLEAN_DERIVED: ${{ inputs.clean-derived }}
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: "Upload Agent Device Artifacts"
2+
description: "Upload shared agent-device diagnostics, session logs, and replay artifacts"
3+
4+
inputs:
5+
artifact-name:
6+
description: "Artifact name"
7+
required: true
8+
agent-home-dir:
9+
description: "Resolved agent-device home directory"
10+
required: true
11+
12+
runs:
13+
using: "composite"
14+
steps:
15+
- name: Upload artifacts
16+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
17+
with:
18+
name: ${{ inputs.artifact-name }}
19+
if-no-files-found: ignore
20+
path: |
21+
${{ inputs.agent-home-dir }}/daemon.log
22+
${{ inputs.agent-home-dir }}/logs/**
23+
${{ inputs.agent-home-dir }}/sessions/**
24+
test/artifacts/**
25+
test/screenshots/**

‎.github/workflows/android.yml‎

Lines changed: 12 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ concurrency:
1414
cancel-in-progress: true
1515

1616
jobs:
17-
integration-android:
18-
name: Integration Tests
17+
smoke-android:
18+
name: Smoke Tests
1919
runs-on: ubuntu-latest
2020
timeout-minutes: 80
2121
continue-on-error: true
@@ -26,33 +26,23 @@ jobs:
2626
- name: Setup toolchain
2727
uses: ./.github/actions/setup-node-pnpm
2828

29-
- name: Resolve agent-device home
30-
id: android-agent-home
31-
run: echo "dir=$HOME/.agent-device" >> "$GITHUB_OUTPUT"
29+
- name: Setup Android replay host
30+
id: android-replay-host
31+
uses: ./.github/actions/setup-android-replay-host
3232

33-
- name: Enable KVM
34-
run: |
35-
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
36-
sudo udevadm control --reload-rules
37-
sudo udevadm trigger --name-match=kvm
38-
39-
- name: Run Android integration test
33+
- name: Run Android smoke replay
4034
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b # v2.35.0
4135
with:
42-
api-level: 35
36+
api-level: 36
4337
arch: x86_64
4438
profile: pixel_7
39+
target: google_apis_playstore
4540
emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim -no-metrics
46-
script: node --experimental-strip-types src/bin.ts test test/integration/replays/android --retries 3
41+
script: node --experimental-strip-types src/bin.ts test test/integration/replays/android/01-settings.ad --retries 3 --report-junit test/artifacts/replays-android-smoke.junit.xml
4742

4843
- name: Upload Android artifacts
4944
if: always()
50-
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
45+
uses: ./.github/actions/upload-agent-device-artifacts
5146
with:
52-
name: android-artifacts
53-
if-no-files-found: ignore
54-
path: |
55-
${{ steps.android-agent-home.outputs.dir }}/daemon.log
56-
${{ steps.android-agent-home.outputs.dir }}/sessions/**
57-
test/artifacts/**
58-
test/screenshots/**
47+
artifact-name: android-artifacts
48+
agent-home-dir: ${{ steps.android-replay-host.outputs.agent-home-dir }}

‎.github/workflows/ios.yml‎

Lines changed: 25 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -14,16 +14,15 @@ concurrency:
1414
cancel-in-progress: true
1515

1616
jobs:
17-
integration-ios:
18-
name: Integration Tests
17+
smoke-ios:
18+
name: Smoke Tests
1919
runs-on: macos-26
2020
timeout-minutes: 80
2121
continue-on-error: true
2222
env:
23-
IOS_RUNTIME_VERSION: '26.2'
23+
IOS_RUNTIME_VERSION: "26.2"
2424
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived
2525
AGENT_DEVICE_IOS_SIMCTL_LIST_TIMEOUT_MS: "60000"
26-
AGENT_DEVICE_DAEMON_TIMEOUT_MS: "300000"
2726
AGENT_DEVICE_IOS_BOOT_TIMEOUT_MS: "180000"
2827
AGENT_DEVICE_IOS_APP_LAUNCH_TIMEOUT_MS: "60000"
2928
steps:
@@ -33,89 +32,36 @@ jobs:
3332
- name: Setup toolchain
3433
uses: ./.github/actions/setup-node-pnpm
3534

36-
- name: Resolve Xcode cache key
37-
id: xcode
38-
run: |
39-
set -euo pipefail
40-
XCODE_VERSION="$(xcodebuild -version | tr '\n' ' ' | sed -E 's/[[:space:]]+/ /g; s/[[:space:]]$//')"
41-
XCODE_KEY="$(echo "$XCODE_VERSION" | tr ' ' '-' | tr -cd '[:alnum:]._-')"
42-
echo "key=$XCODE_KEY" >> "$GITHUB_OUTPUT"
43-
44-
- name: Resolve prebuild source hash
45-
id: source-hash
46-
run: |
47-
set -euo pipefail
48-
echo "value=${{ hashFiles('ios-runner/**', 'package.json', 'pnpm-lock.yaml') }}" >> "$GITHUB_OUTPUT"
49-
50-
- name: Cache iOS runner prebuilt
51-
id: restore-prebuilt
52-
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.2.3
35+
- name: Setup Apple replay
36+
id: apple-replay
37+
uses: ./.github/actions/setup-apple-replay
5338
with:
54-
path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
55-
key: ios-runner-prebuilt-${{ steps.xcode.outputs.key }}-ios-${{ env.IOS_RUNTIME_VERSION }}-${{ steps.source-hash.outputs.value }}
56-
57-
- name: Resolve agent-device home
58-
id: ios-agent-home
59-
run: echo "dir=$HOME/.agent-device" >> "$GITHUB_OUTPUT"
39+
derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
40+
cache-key-prefix: ios-runner-prebuilt
41+
cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }}
42+
build-command: sh ./scripts/build-xcuitest-apple.sh
43+
xcuitest-platform: ios
44+
xcuitest-destination: platform=iOS Simulator,name=iPhone 17 Pro,OS=${{ env.IOS_RUNTIME_VERSION }}
45+
clean-derived: "1"
6046

61-
- name: Build iOS integration artifacts
62-
if: steps.restore-prebuilt.outputs.cache-hit != 'true'
63-
env:
64-
AGENT_DEVICE_XCUITEST_PLATFORM: ios
65-
AGENT_DEVICE_XCUITEST_DESTINATION: platform=iOS Simulator,name=iPhone 17 Pro,OS=${{ env.IOS_RUNTIME_VERSION }}
66-
AGENT_DEVICE_IOS_CLEAN_DERIVED: '1'
67-
run: |
68-
sh ./scripts/build-xcuitest-apple.sh
69-
70-
- name: Resolve and boot iOS test simulator
71-
run: |
72-
set -euo pipefail
73-
RUNTIME_TOKEN="SimRuntime.iOS-${IOS_RUNTIME_VERSION//./-}"
74-
export RUNTIME_TOKEN
75-
UDID="$(
76-
xcrun simctl list devices -j | node -e '
77-
const fs = require("node:fs");
78-
const runtimeToken = process.env.RUNTIME_TOKEN;
79-
const payload = JSON.parse(fs.readFileSync(0, "utf8"));
80-
const entries = Object.entries(payload.devices ?? {});
81-
const iosRuntimes = entries.filter(([runtime]) => runtime.includes("SimRuntime.iOS-"));
82-
const runtimeMatches = iosRuntimes.filter(([runtime]) => runtime.includes(runtimeToken));
83-
const pool = runtimeMatches.length > 0 ? runtimeMatches : iosRuntimes;
84-
const available = pool.flatMap(([runtime, devices]) =>
85-
(devices ?? [])
86-
.filter((device) => device.isAvailable)
87-
.map((device) => ({ ...device, runtime })),
88-
);
89-
const preferred =
90-
available.find((device) => device.state === "Booted" && device.name === "iPhone 17 Pro") ??
91-
available.find((device) => device.name === "iPhone 17 Pro") ??
92-
available.find((device) => device.state === "Booted") ??
93-
available[0];
94-
if (!preferred?.udid) process.exit(1);
95-
process.stdout.write(preferred.udid);
96-
'
97-
)"
98-
xcrun simctl shutdown all || true
99-
xcrun simctl boot "$UDID" || true
100-
xcrun simctl bootstatus "$UDID" -b
47+
- name: Boot iOS test simulator
48+
uses: ./.github/actions/boot-ios-test-simulator
49+
with:
50+
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
51+
preferred-device-name: iPhone 17 Pro
10152

102-
- name: Run iOS simulator replay suite
103-
run: node --experimental-strip-types src/bin.ts test test/integration/replays/ios/simulator --retries 3
53+
- name: Run iOS simulator smoke replay
54+
run: node --experimental-strip-types src/bin.ts test test/integration/replays/ios/simulator/01-settings.ad --retries 3 --report-junit test/artifacts/replays-ios-simulator-smoke.junit.xml
10455

105-
- name: Run iOS physical device replay suite
56+
- name: Run iOS physical device smoke replay
10657
if: env.IOS_UDID != ''
10758
env:
10859
IOS_UDID: ${{ vars.IOS_UDID }}
109-
run: node --experimental-strip-types src/bin.ts test test/integration/replays/ios/device --udid "$IOS_UDID" --retries 3
60+
run: node --experimental-strip-types src/bin.ts test test/integration/replays/ios/device/01-physical-lifecycle.ad --udid "$IOS_UDID" --retries 3 --report-junit test/artifacts/replays-ios-device-smoke.junit.xml
11061

11162
- name: Upload iOS artifacts
11263
if: always()
113-
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
64+
uses: ./.github/actions/upload-agent-device-artifacts
11465
with:
115-
name: ios-artifacts
116-
if-no-files-found: ignore
117-
path: |
118-
${{ steps.ios-agent-home.outputs.dir }}/daemon.log
119-
${{ steps.ios-agent-home.outputs.dir }}/sessions/**
120-
test/artifacts/**
121-
test/screenshots/**
66+
artifact-name: ios-artifacts
67+
agent-home-dir: ${{ steps.apple-replay.outputs.agent-home-dir }}

0 commit comments

Comments
 (0)