Skip to content

Commit 3485af1

Browse files
luisleo526claude
andcommitted
tests: the ownership shards pass their ids to unittest by name and fail closed
The independent read of f539b76 found that unittest.main(module=None, defaultTest=ids, argv=[prog]) ignores defaultTest and falls back to discovery, so each CTest shard ran zero tests (exit 0 before Python 3.12, exit 5 from 3.12). The ids now go into argv, where unittest loads them by name, and the sharded entry point exits 1 unless the number of tests it ran equals the number it selected (and all passed); an empty shard is refused. The CMake headroom comment now uses the heaviest shard (15 of the 39 checker runs, an estimate, about 2.4x on sanitizers) instead of the average, and the README states what a shard checks. Cases, assertions, fixtures, the checker and the 180 s timeout are untouched. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent f539b76 commit 3485af1

3 files changed

Lines changed: 27 additions & 10 deletions

File tree

‎tests/CMakeLists.txt‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3774,10 +3774,13 @@ add_test(NAME test_selected_window_ownership_manifest
37743774
COMMAND ${Python3_EXECUTABLE} -I -B ${PROJECT_SOURCE_DIR}/scripts/check_selected_window_ownership.py)
37753775
# CR-196: lab release/debug/sanitizers/kernel took 57.0/65.9/85.5/56.9 s.
37763776
# Loaded pool release/kernel/native-live took 125.7/126.8/132.3 s; debug and
3777-
# sanitizers timed out at 180.02 s. Four shards keep the same 180 s budget:
3778-
# 126/4 = 31.5 s, or 132.3/4 = 33.1 s using the slowest passing measurement.
3779-
# 180/33.1 = 5.4x estimated headroom before overhead (60 s/shard allows 3x).
3780-
# Cases have unequal costs; the supervisor verifies each shard's loaded time.
3777+
# sanitizers timed out at 180.02 s. Four shards keep the same 180 s budget.
3778+
# The cases are unequal: the heaviest shard (3) holds 15 of the 39 checker
3779+
# runs. ESTIMATE, not measured: 132.3 s x 15/39 = 50.9 s on the loaded pool in
3780+
# Release; scaling by the lab sanitizers/release ratio (85.5/57.0 = 1.5) gives
3781+
# about 76 s for sanitizers, i.e. 2.4x headroom against 180 s. The lab run of
3782+
# the four shard rows is the measurement; each shard also fails closed when it
3783+
# runs a number of tests different from the number it selected.
37813784
set(_pf_ownership_control_shards)
37823785
foreach(_pf_ownership_shard 1 2 3 4)
37833786
add_test(NAME test_selected_window_ownership_manifest_controls_shard_${_pf_ownership_shard}of4

‎tests/ownership/README.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -289,10 +289,13 @@ python3 -I -B tests/ownership/test_manifest_checker.py --shard 1/4
289289
```
290290

291291
`--shard I/N` requires integers with `1 <= I <= N` and selects
292-
`sorted(loader case ids)[I-1::N]`. The separate cheap
293-
`test_shards_partition_every_case_exactly_once` guard runs in every shard:
294-
for each shard count from 1 through 8 it independently discovers at least 30
295-
cases and checks the exact assignments, disjointness and full coverage.
292+
`sorted(loader case ids)[I-1::N]`, passes those ids to unittest by name and
293+
exits non-zero unless it ran exactly the number of tests it selected (9, 9, 9
294+
and 8 for shards 1 to 4 of 4: 8, 8, 8 and 7 cases plus the guard). The separate cheap `test_shards_partition_every_case_exactly_once`
295+
guard runs in every shard: for each shard count from 1 through 8 it rebuilds
296+
the assignment from the loader's ids with index residues (the same loader
297+
expression, a different assignment rule) and checks the exact assignments,
298+
disjointness and full coverage, with at least 30 cases discovered.
296299
Preflight continues to run the complete unsharded suite. The positive
297300
`test_selected_window_ownership_manifest` and independent
298301
`test_selected_window_ownership_d4` CTest rows keep their existing commands.

‎tests/ownership/test_manifest_checker.py‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -522,7 +522,18 @@ def parse_shard(value: str) -> tuple[int, int]:
522522
index, count = args.shard[0]
523523
guards = [test.id() for test in unittest.TestLoader().loadTestsFromTestCase(
524524
ManifestCheckerShardingTests)]
525-
unittest.main(module=None, defaultTest=shard_case_ids(index, count) + guards,
526-
argv=[sys.argv[0]])
525+
selected = shard_case_ids(index, count) + guards
526+
if len(selected) == len(guards):
527+
parser.error("shard %d/%d selects no case" % (index, count))
528+
# The ids go in argv: with module=None unittest ignores defaultTest and
529+
# would fall back to discovery (zero tests, exit 0 before Python 3.12).
530+
program = unittest.main(module=None, argv=[sys.argv[0], *selected], exit=False)
531+
# A shard that ran a different number of tests than it selected fails,
532+
# whatever unittest's own exit status says.
533+
if program.result.testsRun != len(selected) or not program.result.wasSuccessful():
534+
print("shard %d/%d: ran %d tests, selected %d" % (
535+
index, count, program.result.testsRun, len(selected)), file=sys.stderr)
536+
sys.exit(1)
537+
print("shard %d/%d: ran %d tests as selected" % (index, count, len(selected)))
527538
else:
528539
unittest.main()

0 commit comments

Comments
 (0)