Skip to content

Commit 98ae4fb

Browse files
committed
fix(shorebird_ci): retry a test batch that died loading a suite
## Summary One suite losing its VM service connection under `--coverage` reds a whole Run Tests step, and a static batch is up to fifty suites. The flake is open upstream at dart-lang/native#3738, so a run that dies with that signature now happens a second time. Only that signature is retried. A failing test or a compile error still comes back on the first run. - The wrapper covers both static reusable workflows and both dynamic coverage steps. The dynamic step w/o coverage stays bare because nothing there opens a connection to drop - The checked-in `_shorebird_ci_dart.yaml` carries the generator's change verbatim. That is what puts the retry in this repo's CI w/o a shorebird_ci release, so the activate pin stays at 0.3.0 ## Testing - New tests run each of the four generated steps over a stub SDK on `PATH`. A dropped connection passes on the second run, a failing test is not retried, a connection that drops twice fails - Take the retry back out of the generator and 9 of the 12 fail. The three that pass assert a single run, which is free with no retry in the step - Drop only the two `grep` guards and the four "a failing test is not retried" tests fail instead. That is the direction those three guard
1 parent f1d3937 commit 98ae4fb

5 files changed

Lines changed: 264 additions & 20 deletions

File tree

‎.github/workflows/_shorebird_ci_dart.yaml‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,11 +103,31 @@ jobs:
103103
if [ -z "$files" ]; then
104104
exit 0
105105
fi
106+
# A suite whose VM service connection drops as it loads fails every
107+
# suite running alongside it:
108+
# https://github.com/dart-lang/native/issues/3738
109+
retry_test() {
110+
set -o pipefail
111+
local log attempt
112+
log=$(mktemp)
113+
for attempt in 1 2; do
114+
if [ "$attempt" -gt 1 ]; then
115+
echo "::warning::a suite died loading, so the tests run again"
116+
fi
117+
if dart test "$@" 2>&1 | tee "$log"; then
118+
return 0
119+
fi
120+
grep -q 'Failed to load' "$log" || return 1
121+
grep -q 'Connection closed before full header was received' "$log" || return 1
122+
done
123+
return 1
124+
}
125+
export -f retry_test
106126
if [ -n "$MIN_COVERAGE" ] && [ "$MIN_COVERAGE" != "0" ] && [ "$RUNNER_OS_NAME" = "ubuntu-latest" ]; then
107127
# Pinned until https://github.com/dart-lang/tools/issues/2667 is fixed.
108128
dart pub global activate coverage 1.15.1
109129
COVERAGE_DIR="$(pwd -W 2>/dev/null || pwd)/coverage"
110-
printf '%s\n' "$files" | xargs -n 50 dart test --coverage="$COVERAGE_DIR"
130+
printf '%s\n' "$files" | xargs -n 50 bash -c 'retry_test "$@"' _ --coverage="$COVERAGE_DIR"
111131
if [ -z "$(find coverage -name '*.json' -print -quit)" ]; then
112132
echo "::error::dart test wrote no coverage files to coverage/"
113133
exit 1
@@ -128,7 +148,7 @@ jobs:
128148
jq --arg name "$PACKAGE_NAME" '.packages |= map(select(.name == $name))' "$PACKAGE_CONFIG" > "$SCOPED_CONFIG"
129149
dart pub global run coverage:format_coverage --lcov --in=coverage --out=coverage/lcov.info --packages="$SCOPED_CONFIG" --report-on="$REPORT_ON" --check-ignore
130150
else
131-
printf '%s\n' "$files" | xargs -n 50 dart test
151+
printf '%s\n' "$files" | xargs -n 50 bash -c 'retry_test "$@"' _
132152
fi
133153
- name: Check Coverage
134154
if: inputs.has_unit_tests && inputs.min_coverage != '' && inputs.min_coverage != '0' && matrix.os == 'ubuntu-latest'

‎cspell.config.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ words:
4646
- dylib
4747
- endtemplate
4848
- Entra
49+
- esac # bash case terminator
4950
- eseidel
5051
- exitcode
5152
- exportoptions
@@ -98,6 +99,8 @@ words:
9899
- noaudio # From .github dir, doesn't show up in "**" check?
99100
- NOAUTH
100101
- nonobvious # From analysis_options.yaml
102+
- noprofile # bash --noprofile
103+
- norc # bash --norc
101104
- nserror
102105
- OCSF # Open Cybersecurity Schema Framework, the audit log format
103106
- Oltman
@@ -107,6 +110,7 @@ words:
107110
- pana
108111
- pbxproj
109112
- pinact # Third-party action-pinning tool referenced in shorebird_ci
113+
- pipefail
110114
- Perfetto # Chrome trace viewer at ui.perfetto.dev
111115
- PKCE # OAuth Proof Key for Code Exchange (RFC 7636)
112116
- pkcs

‎packages/shorebird_ci/lib/src/commands/generate_command.dart‎

Lines changed: 50 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -692,11 +692,11 @@ ${hasBlocLint ? _staticBlocLintStep : ''} - name: Run Tests
692692
if [ -z "\$files" ]; then
693693
exit 0
694694
fi
695-
if $collectCondition; then
695+
${_retryTestFunction(executable: 'dart', exported: true)} if $collectCondition; then
696696
# Pinned until https://github.com/dart-lang/tools/issues/2667 is fixed.
697697
dart pub global activate coverage 1.15.1
698698
COVERAGE_DIR="\$(pwd -W 2>/dev/null || pwd)/coverage"
699-
printf '%s\\n' "\$files" | xargs -n 50 dart test --coverage="\$COVERAGE_DIR"
699+
printf '%s\\n' "\$files" | xargs -n 50 bash -c 'retry_test "\$@"' _ --coverage="\$COVERAGE_DIR"
700700
if [ -z "\$(find coverage -name '*.json' -print -quit)" ]; then
701701
echo "::error::dart test wrote no coverage files to coverage/"
702702
exit 1
@@ -717,7 +717,7 @@ ${hasBlocLint ? _staticBlocLintStep : ''} - name: Run Tests
717717
jq --arg name "\$PACKAGE_NAME" '.packages |= map(select(.name == \$name))' "\$PACKAGE_CONFIG" > "\$SCOPED_CONFIG"
718718
dart pub global run coverage:format_coverage --lcov --in=coverage --out=coverage/lcov.info --packages="\$SCOPED_CONFIG" --report-on="\$REPORT_ON" --check-ignore
719719
else
720-
printf '%s\\n' "\$files" | xargs -n 50 dart test
720+
printf '%s\\n' "\$files" | xargs -n 50 bash -c 'retry_test "\$@"' _
721721
fi
722722
- name: Check Coverage
723723
if: inputs.has_unit_tests && inputs.min_coverage != '' && inputs.min_coverage != '0' && matrix.os == 'ubuntu-latest'
@@ -854,11 +854,11 @@ ${hasBlocLint ? _staticBlocLintStep : ''} - name: Run Tests
854854
if [ -z "\$files" ]; then
855855
exit 0
856856
fi
857-
if $collectCondition; then
857+
${_retryTestFunction(executable: 'flutter', exported: true)} if $collectCondition; then
858858
COVERAGE_DIR="\$(pwd -W 2>/dev/null || pwd)/coverage"
859-
printf '%s\\n' "\$files" | xargs -n 50 flutter test --coverage --merge-coverage --coverage-path="\$COVERAGE_DIR/lcov.info"
859+
printf '%s\\n' "\$files" | xargs -n 50 bash -c 'retry_test "\$@"' _ --coverage --merge-coverage --coverage-path="\$COVERAGE_DIR/lcov.info"
860860
else
861-
printf '%s\\n' "\$files" | xargs -n 50 flutter test
861+
printf '%s\\n' "\$files" | xargs -n 50 bash -c 'retry_test "\$@"' _
862862
fi
863863
- name: Check Coverage
864864
if: inputs.has_unit_tests && inputs.min_coverage != '' && inputs.min_coverage != '0' && matrix.os == 'ubuntu-latest'
@@ -939,6 +939,38 @@ $codecovStep - name: Integration Tests
939939
run: bloc lint .
940940
''';
941941

942+
/// Shell function a test step calls in place of `[executable] test`, which
943+
/// runs the same arguments a second time when the first run died loading a
944+
/// suite rather than failing a test. Only that one signature is retried: a
945+
/// failing test, a compile error and every other load failure come back as
946+
/// they are. [exported] hands the function to the `bash -c` that `xargs`
947+
/// spawns per batch of suites.
948+
static String _retryTestFunction({
949+
required String executable,
950+
required bool exported,
951+
}) =>
952+
'''
953+
# A suite whose VM service connection drops as it loads fails every
954+
# suite running alongside it:
955+
# https://github.com/dart-lang/native/issues/3738
956+
retry_test() {
957+
set -o pipefail
958+
local log attempt
959+
log=\$(mktemp)
960+
for attempt in 1 2; do
961+
if [ "\$attempt" -gt 1 ]; then
962+
echo "::warning::a suite died loading, so the tests run again"
963+
fi
964+
if $executable test "\$@" 2>&1 | tee "\$log"; then
965+
return 0
966+
fi
967+
grep -q 'Failed to load' "\$log" || return 1
968+
grep -q 'Connection closed before full header was received' "\$log" || return 1
969+
done
970+
return 1
971+
}
972+
${exported ? ' export -f retry_test\n' : ''}''';
973+
942974
// ── Dynamic (affected_packages + matrix) ─────────────────────────
943975

944976
String _buildDynamicYaml(
@@ -1249,14 +1281,22 @@ ${_checkoutStep(hasSubmodules: hasSubmodules)}''';
12491281
required bool hasCodecov,
12501282
required String executable,
12511283
}) {
1284+
// Left bare: without `--coverage` there is no VM service connection
1285+
// for a loading suite to drop, which is all the retry covers.
12521286
if (!hasCodecov) return ' - run: $executable test\n';
1253-
if (isFlutter) return ' - run: flutter test --coverage\n';
1254-
return r'''
1287+
if (isFlutter) {
1288+
return '''
1289+
- name: Run Tests
1290+
run: |
1291+
${_retryTestFunction(executable: 'flutter', exported: false)} retry_test --coverage
1292+
''';
1293+
}
1294+
return '''
12551295
- name: Run Tests
12561296
# Pinned until https://github.com/dart-lang/tools/issues/2667 is fixed.
12571297
run: |
1258-
dart pub global activate coverage 1.15.1 && \
1259-
dart test --coverage=coverage && \
1298+
${_retryTestFunction(executable: 'dart', exported: false)} dart pub global activate coverage 1.15.1 && \\
1299+
retry_test --coverage=coverage && \\
12601300
dart pub global run coverage:format_coverage --lcov --in=coverage --out=coverage/lcov.info --report-on=lib --check-ignore
12611301
''';
12621302
}

‎packages/shorebird_ci/test/generate_command_test.dart‎

Lines changed: 183 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,16 @@ Future<int?> runGenerate(
3232
]);
3333
}
3434

35+
/// A generated Run Tests step, with the workflow and job holding it.
36+
typedef TestSite = ({
37+
String name,
38+
String executable,
39+
String style,
40+
bool hasCodecov,
41+
String workflow,
42+
String job,
43+
});
44+
3545
void main() {
3646
setUpTempDir('shorebird_ci_generate_');
3747

@@ -798,10 +808,6 @@ void main() {
798808
);
799809
}
800810

801-
String readWorkflow(String name) => File(
802-
p.join(tempDir.path, '.github', 'workflows', name),
803-
).readAsStringSync();
804-
805811
test('dynamic: no bloc steps when no package uses bloc_lint', () async {
806812
createPackage(tempDir, 'packages/foo', 'foo');
807813
createPackage(tempDir, 'packages/bar', 'bar', flutterLine: 'any');
@@ -1329,10 +1335,6 @@ void main() {
13291335
});
13301336

13311337
group('v0.3.0 features', () {
1332-
String readWorkflow(String name) => File(
1333-
p.join(tempDir.path, '.github', 'workflows', name),
1334-
).readAsStringSync();
1335-
13361338
test('--filters-only updates dorny/paths-filter in place', () async {
13371339
createPackage(tempDir, 'packages/foo', 'foo');
13381340
createPackage(tempDir, 'packages/bar', 'bar');
@@ -1617,6 +1619,179 @@ shorebird_ci:
16171619
},
16181620
);
16191621
});
1622+
1623+
group('Run Tests step', () {
1624+
/// Every generated step the retry wraps.
1625+
const sites = <TestSite>[
1626+
(
1627+
name: 'static dart',
1628+
executable: 'dart',
1629+
style: 'static',
1630+
hasCodecov: false,
1631+
workflow: '_shorebird_ci_dart.yaml',
1632+
job: 'ci',
1633+
),
1634+
(
1635+
name: 'static flutter',
1636+
executable: 'flutter',
1637+
style: 'static',
1638+
hasCodecov: false,
1639+
workflow: '_shorebird_ci_flutter.yaml',
1640+
job: 'ci',
1641+
),
1642+
(
1643+
name: 'dynamic dart',
1644+
executable: 'dart',
1645+
style: 'dynamic',
1646+
hasCodecov: true,
1647+
workflow: 'shorebird_ci.yaml',
1648+
job: 'dart_ci',
1649+
),
1650+
(
1651+
name: 'dynamic flutter',
1652+
executable: 'flutter',
1653+
style: 'dynamic',
1654+
hasCodecov: true,
1655+
workflow: 'shorebird_ci.yaml',
1656+
job: 'flutter_ci',
1657+
),
1658+
];
1659+
1660+
/// The `run:` script of the Run Tests step in [job] of [workflow].
1661+
String runTestsStep(String workflow, String job) {
1662+
final steps =
1663+
((loadYaml(workflow) as YamlMap)['jobs'] as YamlMap)[job] as YamlMap;
1664+
final step = (steps['steps'] as YamlList).cast<YamlMap>().firstWhere(
1665+
(step) => step['name'] == 'Run Tests',
1666+
);
1667+
return step['run'] as String;
1668+
}
1669+
1670+
/// Writes an [executable] into [dir] that answers each `test` invocation
1671+
/// with the next entry of [runs] and every other invocation with success,
1672+
/// recording the `test` ones in `attempts` beside itself. The last entry
1673+
/// answers every `test` invocation after it.
1674+
void writeStub(
1675+
Directory dir,
1676+
String executable,
1677+
List<({String output, int exitCode})> runs,
1678+
) {
1679+
final cases = [
1680+
for (final (index, run) in runs.indexed)
1681+
'''
1682+
${index == runs.length - 1 ? '*' : index + 1})
1683+
printf '%s\\n' '${run.output}'
1684+
exit ${run.exitCode}
1685+
;;''',
1686+
].join('\n');
1687+
final stub = File(p.join(dir.path, executable))
1688+
..writeAsStringSync('''
1689+
#!/usr/bin/env bash
1690+
if [ "\$1" != test ]; then
1691+
exit 0
1692+
fi
1693+
attempts="\$(dirname "\$0")/attempts"
1694+
echo "$executable \$*" >> "\$attempts"
1695+
case "\$(grep -c . "\$attempts")" in
1696+
$cases
1697+
esac
1698+
''');
1699+
Process.runSync('chmod', ['+x', stub.path]);
1700+
}
1701+
1702+
/// Runs [site]'s generated step over one test file, against an SDK
1703+
/// answering with [runs]. Returns the step's exit code and how many
1704+
/// times it ran the SDK's `test`.
1705+
Future<({int exitCode, int testRuns})> runStep(
1706+
TestSite site,
1707+
List<({String output, int exitCode})> runs,
1708+
) async {
1709+
final isFlutter = site.executable == 'flutter';
1710+
createPackage(
1711+
tempDir,
1712+
'pkg',
1713+
'pkg',
1714+
addTestDir: true,
1715+
flutterLine: isFlutter ? 'any' : null,
1716+
);
1717+
if (site.hasCodecov) {
1718+
File(p.join(tempDir.path, 'codecov.yml')).writeAsStringSync('');
1719+
}
1720+
initGitRepo(tempDir);
1721+
await runGenerate(tempDir, extra: ['--style', site.style]);
1722+
1723+
final stubDir = Directory(p.join(tempDir.path, 'stub'))..createSync();
1724+
writeStub(stubDir, site.executable, runs);
1725+
final script = File(p.join(tempDir.path, 'step.sh'))
1726+
..writeAsStringSync(
1727+
runTestsStep(readWorkflow(site.workflow), site.job),
1728+
);
1729+
1730+
// The static workflows declare `shell: bash`, which GitHub runs with
1731+
// `-eo pipefail`. The dynamic ones take the default `bash -e`, so the
1732+
// step cannot lean on its caller for pipefail.
1733+
final result = Process.runSync(
1734+
'bash',
1735+
[
1736+
'--noprofile',
1737+
'--norc',
1738+
if (site.style == 'static') ...['-eo', 'pipefail'] else '-e',
1739+
script.path,
1740+
],
1741+
workingDirectory: p.join(tempDir.path, 'pkg'),
1742+
environment: {
1743+
'MIN_COVERAGE': '',
1744+
'PATH': '${stubDir.path}:${Platform.environment['PATH']}',
1745+
'RUNNER_OS_NAME': 'ubuntu-latest',
1746+
'SHARD_INDEX': '0',
1747+
'TOTAL_SHARDS': '1',
1748+
},
1749+
);
1750+
printOnFailure('${result.stdout}${result.stderr}');
1751+
final attempts = File(p.join(stubDir.path, 'attempts'));
1752+
return (
1753+
exitCode: result.exitCode,
1754+
testRuns: attempts.existsSync()
1755+
? attempts.readAsLinesSync().where((l) => l.isNotEmpty).length
1756+
: 0,
1757+
);
1758+
}
1759+
1760+
const droppedConnection =
1761+
'Failed to load "test/example_test.dart": '
1762+
'Connection closed before full header was received';
1763+
1764+
for (final site in sites) {
1765+
group(site.name, () {
1766+
test('a dropped suite connection passes on the second run', () async {
1767+
final step = await runStep(site, [
1768+
(output: droppedConnection, exitCode: 1),
1769+
(output: 'All tests passed!', exitCode: 0),
1770+
]);
1771+
1772+
expect(step, (exitCode: 0, testRuns: 2));
1773+
}, testOn: '!windows');
1774+
1775+
test('a failing test is not retried and fails the step', () async {
1776+
final step = await runStep(site, [
1777+
(output: 'Some tests failed.', exitCode: 1),
1778+
]);
1779+
1780+
expect(step.exitCode, isNot(0));
1781+
expect(step.testRuns, 1);
1782+
}, testOn: '!windows');
1783+
1784+
test('a connection that drops twice fails the step', () async {
1785+
final step = await runStep(site, [
1786+
(output: droppedConnection, exitCode: 1),
1787+
]);
1788+
1789+
expect(step.exitCode, isNot(0));
1790+
expect(step.testRuns, 2);
1791+
}, testOn: '!windows');
1792+
});
1793+
}
1794+
});
16201795
}
16211796

16221797
String _readMain(Directory repoRoot) {

0 commit comments

Comments
 (0)