Repository navigation
benchmarks: a host without ps is not read as quiet; run_all.sh records the busy count as unknown and labels the figures quiet-unverified - #387
Merged
Conversation
…s the busy count as unknown and labels the figures quiet-unverified
The benchmark test file runs the real run_all.sh against stand-ins, and run_all.sh's quiet_gate called `ps -axo comm=,args=` unconditionally. On a host without ps (the lab's CLOUD task image) the script exited 127 at that line and test_benchmark_harness failed there: test_absent_pynecore_harness_completes_without_pynecore_outputs and test_absent_harness_without_committed_indicator_output_skips_three_way_report ("AssertionError: 127 != 0 ... ps: command not found").
quiet_gate now checks `command -v ps`. With ps, nothing changes: the same count, the same log line, the same tsv columns, the same report. Without ps the busy count is UNKNOWN, never 0: the timing-batch log line says "busy count unknown (ps absent)", the busy column of _workdir/speed_loads.tsv holds "unknown (ps absent)", and every figure of that run is labelled in the generated report (the line "quiet-unverified: no ps on this host, the busy count was not measured; do not use these figures as performance evidence" is appended to _workdir/speed.md and printed again as the last line of the run). With QUIET_LOAD_MAX set the wait loop decides on load alone and still ends by QUIET_WAIT_S; the label still applies. When PyneCore timings exist and the notice is set, aggregate.py writes to _workdir/speed.md instead of the committed benchmarks/results/speed.md, so unverified figures never replace the committed table. The header comment of run_all.sh says what happens without ps.
Tests (benchmarks/tests/test_run_all.py): the report cases now stub sysctl and ps so they no longer depend on the host's ps; three new cases: test_quiet_gate_without_ps_labels_figures and test_quiet_gate_without_ps_uses_load_only run run_all.sh with a PATH of symlinks that has no ps (exit 0, the log line, the tsv column, the label in speed.md and as the last output line, the committed speed.md byte-unchanged, with and without the PyneCore harness); test_quiet_gate_with_ps_preserves_verified_output runs with a ps stand-in and asserts the old output exactly and that none of the new text appears. Red first: the three new cases against the base run_all.sh fail with exit 127 and "ps: command not found" (4 subtest failures in the two missing-ps cases; the with-ps case passes on the base, as it should); on this tree the whole file passes (11 tests).
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526
added a commit
that referenced
this pull request
Oct 14, 2026
* docs: the 1.6.0-rc.1 changelog entry The changelog section for the first candidate of 1.6.0, tag v1.6.0-rc.1 on commit 04607c1: the selected-window contract revision 1.3 and the typed failures behind it (#388), finite trade and summary doubles in every report writer (#384, #388), the favicon of the API documentation (#355), the removed CI time limits and command-line options (#378), and the tests, CI and benchmarks changes since 1.5.0 (#375, #376, #377, #382, #383, #385, #386, #387). The entry is inserted above the 1.5.0 section; no existing line changes. The date is the UTC date of the tag. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * README: link the apex pineforge.dev, not the www host The README header link pointed at https://www.pineforge.dev. The product site lives at the apex, https://pineforge.dev; the www host is being redirected to it, and search indexes both forms today. One link changed, nothing else. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * CHANGELOG: add the selected-window mode and typed failures to the 1.5.0 notes Two things that are in the v1.5.0 tag were missing from its notes: the opt-in selected-window mode with retained pre-roll bars (#363, already in v1.5.0-rc.1) and the typed failures that docker/run_json.py raises for exceptions from the selected-window helper modules (_typed_failure). The new subsection "Additions to the 1.5.0 notes" only states what the code and the #363 text say, with the same "development-only parity candidate" limits (contract revision 1.3-parity, no claim of complete value-ownership coverage, cannot be promoted to stable). It adds lines and changes none. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> --------- Co-authored-by: Claude Sonnet 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
benchmarks/run_all.shno longer fails on a host withoutps, and it no longer lets such a host pass as quiet.quiet_gatecounted running build/test processes withps -axo comm=,args=before every timing batch. On a host that has nops(the lab's CLOUD task image) the script exited 127 at that line, sotest_benchmark_harnessfailed there:test_absent_pynecore_harness_completes_without_pynecore_outputsandtest_absent_harness_without_committed_indicator_output_skips_three_way_report(AssertionError: 127 != 0 ... ps: command not found).quiet_gatenow checkscommand -v ps:ps: nothing changes (same count, same log line, same columns in_workdir/speed_loads.tsv, same report).ps: the busy count is unknown, never 0. The timing-batch log line saysbusy count unknown (ps absent), the busy column holdsunknown (ps absent), and every figure of that run is labelled: the linequiet-unverified: no ps on this host, the busy count was not measured; do not use these figures as performance evidenceis appended to_workdir/speed.mdand printed as the last line of the run. WithQUIET_LOAD_MAXset the wait loop decides on the load average alone and still ends byQUIET_WAIT_S. When PyneCore timings exist, the aggregate writes to_workdir/speed.md, so unverified figures never replace the committedbenchmarks/results/speed.md.Tests
benchmarks/tests/test_run_all.py: the report cases stubsysctlandps, so they no longer depend on the host'sps. Three new cases: two runrun_all.shwith aPATHthat has nops(exit 0; the log line; the tsv column; the label inspeed.mdand as the last output line; the committedspeed.mdbyte-unchanged; with and without the PyneCore harness; with and withoutQUIET_LOAD_MAX), and one runs with apsstand-in and asserts the old output exactly and that none of the new text appears.Red first: against the base
run_all.shthe two without-pscases fail with exit 127 (ps: command not found); on this tree the whole file passes (11 tests).Verification
lab remoteCLOUD job (release) on this tree, which is the host class where the failure occurred: passed.check_doc_reverts.py(base 80e26e8),check_doc_lint.py,check_twin_parity.py,bash -n benchmarks/run_all.sh.Only
benchmarks/run_all.shandbenchmarks/tests/test_run_all.pychange. Row counts and floors do not move.🤖 Generated with Claude Code