Skip to content

benchmarks: a host without ps is not read as quiet; run_all.sh records the busy count as unknown and labels the figures quiet-unverified - #387

Merged
luisleo526 merged 1 commit into
mainfrom
lvb/psfix
Oct 13, 2026
Merged

luisleo526 merged 1 commit into
mainfrom
lvb/psfix

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Summary

benchmarks/run_all.sh no longer fails on a host without ps, and it no longer lets such a host pass as quiet.

quiet_gate counted running build/test processes with ps -axo comm=,args= before every timing batch. On a host that has no ps (the lab's CLOUD task image) the script exited 127 at that line, so test_benchmark_harness failed there: test_absent_pynecore_harness_completes_without_pynecore_outputs and test_absent_harness_without_committed_indicator_output_skips_three_way_report (AssertionError: 127 != 0 ... ps: command not found).

quiet_gate now checks command -v ps:

  • With ps: nothing changes (same count, same log line, same columns in _workdir/speed_loads.tsv, same report).
  • Without ps: the busy count is unknown, never 0. The timing-batch log line says busy count unknown (ps absent), the busy column holds unknown (ps absent), and every figure of that run is labelled: the line quiet-unverified: no ps on this host, the busy count was not measured; do not use these figures as performance evidence is appended to _workdir/speed.md and printed as the last line of the run. With QUIET_LOAD_MAX set the wait loop decides on the load average alone and still ends by QUIET_WAIT_S. When PyneCore timings exist, the aggregate writes to _workdir/speed.md, so unverified figures never replace the committed benchmarks/results/speed.md.

Tests

benchmarks/tests/test_run_all.py: the report cases stub sysctl and ps, so they no longer depend on the host's ps. Three new cases: two run run_all.sh with a PATH that has no ps (exit 0; the log line; the tsv column; the label in speed.md and as the last output line; the committed speed.md byte-unchanged; with and without the PyneCore harness; with and without QUIET_LOAD_MAX), and one runs with a ps stand-in and asserts the old output exactly and that none of the new text appears.

Red first: against the base run_all.sh the two without-ps cases fail with exit 127 (ps: command not found); on this tree the whole file passes (11 tests).

Verification

  • lab remote CLOUD job (release) on this tree, which is the host class where the failure occurred: passed.
  • Local: check_doc_reverts.py (base 80e26e8), check_doc_lint.py, check_twin_parity.py, bash -n benchmarks/run_all.sh.

Only benchmarks/run_all.sh and benchmarks/tests/test_run_all.py change. Row counts and floors do not move.

🤖 Generated with Claude Code

…s the busy count as unknown and labels the figures quiet-unverified

The benchmark test file runs the real run_all.sh against stand-ins, and run_all.sh's quiet_gate called `ps -axo comm=,args=` unconditionally. On a host without ps (the lab's CLOUD task image) the script exited 127 at that line and test_benchmark_harness failed there: test_absent_pynecore_harness_completes_without_pynecore_outputs and test_absent_harness_without_committed_indicator_output_skips_three_way_report ("AssertionError: 127 != 0 ... ps: command not found").

quiet_gate now checks `command -v ps`. With ps, nothing changes: the same count, the same log line, the same tsv columns, the same report. Without ps the busy count is UNKNOWN, never 0: the timing-batch log line says "busy count unknown (ps absent)", the busy column of _workdir/speed_loads.tsv holds "unknown (ps absent)", and every figure of that run is labelled in the generated report (the line "quiet-unverified: no ps on this host, the busy count was not measured; do not use these figures as performance evidence" is appended to _workdir/speed.md and printed again as the last line of the run). With QUIET_LOAD_MAX set the wait loop decides on load alone and still ends by QUIET_WAIT_S; the label still applies. When PyneCore timings exist and the notice is set, aggregate.py writes to _workdir/speed.md instead of the committed benchmarks/results/speed.md, so unverified figures never replace the committed table. The header comment of run_all.sh says what happens without ps.

Tests (benchmarks/tests/test_run_all.py): the report cases now stub sysctl and ps so they no longer depend on the host's ps; three new cases: test_quiet_gate_without_ps_labels_figures and test_quiet_gate_without_ps_uses_load_only run run_all.sh with a PATH of symlinks that has no ps (exit 0, the log line, the tsv column, the label in speed.md and as the last output line, the committed speed.md byte-unchanged, with and without the PyneCore harness); test_quiet_gate_with_ps_preserves_verified_output runs with a ps stand-in and asserts the old output exactly and that none of the new text appears. Red first: the three new cases against the base run_all.sh fail with exit 127 and "ps: command not found" (4 subtest failures in the two missing-ps cases; the with-ps case passes on the base, as it should); on this tree the whole file passes (11 tests).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@luisleo526
luisleo526 merged commit 9372b11 into main Oct 13, 2026
20 checks passed
@luisleo526
luisleo526 deleted the lvb/psfix branch October 13, 2026 23:12
luisleo526 added a commit that referenced this pull request Oct 14, 2026
* docs: the 1.6.0-rc.1 changelog entry

The changelog section for the first candidate of 1.6.0, tag v1.6.0-rc.1 on
commit 04607c1: the selected-window contract
revision 1.3 and the typed failures behind it (#388), finite trade and summary
doubles in every report writer (#384, #388), the favicon of the API
documentation (#355), the removed CI time limits and command-line options (#378),
and the tests, CI and benchmarks changes since 1.5.0 (#375, #376, #377, #382,
#383, #385, #386, #387).

The entry is inserted above the 1.5.0 section; no existing line changes. The
date is the UTC date of the tag.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

* README: link the apex pineforge.dev, not the www host

The README header link pointed at https://www.pineforge.dev. The product site
lives at the apex, https://pineforge.dev; the www host is being redirected to
it, and search indexes both forms today. One link changed, nothing else.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

* CHANGELOG: add the selected-window mode and typed failures to the 1.5.0 notes

Two things that are in the v1.5.0 tag were missing from its notes: the opt-in
selected-window mode with retained pre-roll bars (#363, already in
v1.5.0-rc.1) and the typed failures that docker/run_json.py raises for
exceptions from the selected-window helper modules (_typed_failure).

The new subsection "Additions to the 1.5.0 notes" only states what the code
and the #363 text say, with the same "development-only parity candidate"
limits (contract revision 1.3-parity, no claim of complete value-ownership
coverage, cannot be promoted to stable). It adds lines and changes none.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

---------

Co-authored-by: Claude Sonnet 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant