Skip to content

Commit 9372b11

Browse files
luisleo526claude
andauthored
benchmarks: a host without ps is not read as quiet; run_all.sh records the busy count as unknown and labels the figures quiet-unverified (#387)
benchmarks: a host without ps is not read as quiet; run_all.sh records the busy count as unknown and labels the figures quiet-unverified The benchmark test file runs the real run_all.sh against stand-ins, and run_all.sh's quiet_gate called `ps -axo comm=,args=` unconditionally. On a host without ps (the lab's CLOUD task image) the script exited 127 at that line and test_benchmark_harness failed there: test_absent_pynecore_harness_completes_without_pynecore_outputs and test_absent_harness_without_committed_indicator_output_skips_three_way_report ("AssertionError: 127 != 0 ... ps: command not found"). quiet_gate now checks `command -v ps`. With ps, nothing changes: the same count, the same log line, the same tsv columns, the same report. Without ps the busy count is UNKNOWN, never 0: the timing-batch log line says "busy count unknown (ps absent)", the busy column of _workdir/speed_loads.tsv holds "unknown (ps absent)", and every figure of that run is labelled in the generated report (the line "quiet-unverified: no ps on this host, the busy count was not measured; do not use these figures as performance evidence" is appended to _workdir/speed.md and printed again as the last line of the run). With QUIET_LOAD_MAX set the wait loop decides on load alone and still ends by QUIET_WAIT_S; the label still applies. When PyneCore timings exist and the notice is set, aggregate.py writes to _workdir/speed.md instead of the committed benchmarks/results/speed.md, so unverified figures never replace the committed table. The header comment of run_all.sh says what happens without ps. Tests (benchmarks/tests/test_run_all.py): the report cases now stub sysctl and ps so they no longer depend on the host's ps; three new cases: test_quiet_gate_without_ps_labels_figures and test_quiet_gate_without_ps_uses_load_only run run_all.sh with a PATH of symlinks that has no ps (exit 0, the log line, the tsv column, the label in speed.md and as the last output line, the committed speed.md byte-unchanged, with and without the PyneCore harness); test_quiet_gate_with_ps_preserves_verified_output runs with a ps stand-in and asserts the old output exactly and that none of the new text appears. Red first: the three new cases against the base run_all.sh fail with exit 127 and "ps: command not found" (4 subtest failures in the two missing-ps cases; the with-ps case passes on the base, as it should); on this tree the whole file passes (11 tests). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent e11d9fc commit 9372b11

2 files changed

Lines changed: 123 additions & 6 deletions

File tree

‎benchmarks/run_all.sh‎

Lines changed: 31 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
# this and no cmake --build / ctest / ci_verify runs (poll 60 s, up to
3232
# QUIET_WAIT_S, default 14400); unset = record the load and go on.
3333
# Loads land in _workdir/speed_loads.tsv and speed.md's header.
34+
# Without ps, wait on load alone; quiet is unverified and figures are labelled quiet-unverified.
3435
#
3536
# Exit status: every engine run first removes the slot's previous trade list,
3637
# so a failed run never leaves one behind for compare.py to grade. PineForge is
@@ -297,17 +298,27 @@ fi
297298

298299
# Record the host load before a timing batch (and, with QUIET_LOAD_MAX, wait
299300
# for a quiet host first). One line per batch in _workdir/speed_loads.tsv.
301+
# Without ps, the busy count is unknown and figures are labelled quiet-unverified.
302+
QUIET_UNVERIFIED_NOTICE=""
300303
quiet_gate() {
301-
local label="$1" waited=0 load busy
304+
local label="$1" waited=0 load busy ps_available
302305
while :; do
303306
load=$( (sysctl -n vm.loadavg 2>/dev/null || cat /proc/loadavg) | tr -d '{}' | awk '{print $1}')
304307
# Build/test processes by executable name (a full-command grep also
305308
# matches any process whose arguments merely quote these commands).
306-
busy=$(ps -axo comm=,args= | awk '{ c = $1; sub(".*/", "", c)
307-
if (c == "ctest" || (c == "cmake" && / --build /) || (c ~ /^[Pp]ython/ && /ci_verify\.py/)) n++ }
308-
END { print n + 0 }')
309+
if command -v ps >/dev/null 2>&1; then
310+
ps_available=1
311+
busy=$(ps -axo comm=,args= | awk '{ c = $1; sub(".*/", "", c)
312+
if (c == "ctest" || (c == "cmake" && / --build /) || (c ~ /^[Pp]ython/ && /ci_verify\.py/)) n++ }
313+
END { print n + 0 }')
314+
else
315+
ps_available=0
316+
busy="unknown (ps absent)"
317+
QUIET_UNVERIFIED_NOTICE="quiet-unverified: no ps on this host, the busy count was not measured; do not use these figures as performance evidence"
318+
fi
309319
if [[ -z "${QUIET_LOAD_MAX:-}" ]] \
310-
|| { awk -v l="${load}" -v m="${QUIET_LOAD_MAX}" 'BEGIN { exit !(l < m) }' && [[ "${busy}" == 0 ]]; }; then
320+
|| { awk -v l="${load}" -v m="${QUIET_LOAD_MAX}" 'BEGIN { exit !(l < m) }' \
321+
&& [[ "${ps_available}" == 0 || "${busy}" == 0 ]]; }; then
311322
break
312323
fi
313324
(( waited >= ${QUIET_WAIT_S:-14400} )) && fail "host never quiet before ${label}: load ${load}, busy ${busy}"
@@ -316,7 +327,11 @@ quiet_gate() {
316327
done
317328
printf '%s\t%s\t%s\t%s\n' "${label}" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${load}" "${busy}" \
318329
>> "${WORKDIR}/speed_loads.tsv"
319-
log "timing batch ${label}: 1-min load ${load}, build/test processes ${busy}"
330+
if [[ "${ps_available}" == 0 ]]; then
331+
log "timing batch ${label}: 1-min load ${load}, busy count ${busy}"
332+
else
333+
log "timing batch ${label}: 1-min load ${load}, build/test processes ${busy}"
334+
fi
320335
}
321336

322337
if [[ "${SKIP_SPEED:-0}" != "1" ]]; then
@@ -368,12 +383,19 @@ if [[ "${SKIP_SPEED:-0}" != "1" ]]; then
368383
>/dev/null 2>"${WORKDIR}/vbt_speed.err"
369384
VBT_ARGS=(--vectorbt "${WORKDIR}/vbt_speed.json")
370385
fi
386+
# Unverified figures, even with PyneCore timings, must not replace the committed table.
387+
if [[ -n "${QUIET_UNVERIFIED_NOTICE}" ]] && pynecore_harness_has speed/time_pynecore.py; then
388+
PC_ARGS+=(--out "${WORKDIR}/speed.md")
389+
fi
371390
(cd "${BENCH_DIR}" && uv run python speed/aggregate.py \
372391
--pineforge "${WORKDIR}/pf_speed.json" \
373392
--pineforge-format "${PF_FMT}" \
374393
${PC_ARGS[@]+"${PC_ARGS[@]}"} \
375394
--pinets "${WORKDIR}/pt_speed.json" \
376395
--loads "${WORKDIR}/speed_loads.tsv" ${VBT_ARGS[@]+"${VBT_ARGS[@]}"})
396+
if [[ -n "${QUIET_UNVERIFIED_NOTICE}" ]]; then
397+
printf '\n%s\n' "${QUIET_UNVERIFIED_NOTICE}" >> "${WORKDIR}/speed.md"
398+
fi
377399
fi
378400

379401
# --- 7) reports -------------------------------------------------------
@@ -392,3 +414,6 @@ fi
392414

393415
log "done."
394416
log "results: ${BENCH_DIR}/results/{summary,trade_comparison,indicator_comparison}.md"
417+
if [[ -n "${QUIET_UNVERIFIED_NOTICE}" ]]; then
418+
printf '%s\n' "${QUIET_UNVERIFIED_NOTICE}"
419+
fi

‎benchmarks/tests/test_run_all.py‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,12 @@ def prepare_report_stubs(self) -> None:
209209
node = self.root / "bin" / "node"
210210
node.write_text('#!/bin/sh\nprintf "{}\\n"\n')
211211
node.chmod(0o755)
212+
# Report tests do not measure the host: keep gate readings deterministic,
213+
# including where sysctl or ps is unavailable or disallowed.
214+
for tool, output in (("sysctl", "{ 0.25 0.25 0.25 }"), ("ps", "ctest ctest")):
215+
stub = self.root / "bin" / tool
216+
stub.write_text(f'#!/bin/sh\nprintf "{output}\\n"\n')
217+
stub.chmod(0o755)
212218
(self.strategies / "_indicators").mkdir()
213219
canonical = bench / "runners" / "run_pineforge_canonical"
214220
canonical.write_text('#!/bin/sh\nprintf "fresh\\n" > "$2"\n')
@@ -308,6 +314,92 @@ def test_absent_pynecore_harness_completes_without_pynecore_outputs(self) -> Non
308314
# The committed canonical PyneCore indicator output is still there, so the three-way report runs.
309315
self.assertTrue((bench / "_workdir" / "indicator_report.called").exists())
310316

317+
def quiet_gate_path(self, *, ps_present: bool = False) -> Path:
318+
# Only these tools and the existing stand-ins are visible; ps cannot
319+
# leak in through the host PATH, including in xargs' child shells.
320+
no_ps_bin = self.root / "quiet-bin"
321+
no_ps_bin.mkdir()
322+
for tool in ("bash", "python3", "dirname", "mkdir", "cmp", "cp", "basename",
323+
"rm", "wc", "tr", "xargs", "awk", "date", "sleep", "cat"):
324+
executable = shutil.which(tool)
325+
self.assertIsNotNone(executable, f"required test tool missing: {tool}")
326+
(no_ps_bin / tool).symlink_to(executable)
327+
for tool in ("uv", "node", "pyne", "sysctl"):
328+
(no_ps_bin / tool).symlink_to(self.root / "bin" / tool)
329+
if ps_present:
330+
(no_ps_bin / "ps").symlink_to(self.root / "bin" / "ps")
331+
self.assertIsNotNone(shutil.which("ps", path=str(no_ps_bin)))
332+
else:
333+
self.assertIsNone(shutil.which("ps", path=str(no_ps_bin)))
334+
return no_ps_bin
335+
336+
def assert_unverified_quiet_gate(self, load_max: str | None) -> None:
337+
ok = self.slot("001-ok")
338+
self.prepare_report_stubs()
339+
no_ps_bin = self.quiet_gate_path()
340+
bench = self.root / "benchmarks"
341+
for harness_dir in (None, str(self.harness)):
342+
with self.subTest(harness_dir=harness_dir):
343+
self.reset_report_case(ok)
344+
res = self.run_all(
345+
PATH=str(no_ps_bin), PYNECORE_HARNESS_DIR=harness_dir, SKIP_PYNE="0",
346+
SKIP_SPEED="0", SKIP_VECTORBT="0", SKIP_REPORTS="0",
347+
SKIP_INDICATORS="0", SKIP_PINETS="0", SKIP_PINEFORGE="0",
348+
SLOTS="", RUNNER="docker", QUIET_LOAD_MAX=load_max, QUIET_WAIT_S="0")
349+
self.assertEqual(res.returncode, 0, res.stderr)
350+
rows = (bench / "_workdir" / "speed_loads.tsv").read_text().splitlines()
351+
batches = (["pynecore"] if harness_dir else []) + ["pinets", "vectorbt"]
352+
self.assertEqual([row.split("\t")[0] for row in rows], batches)
353+
for row in rows:
354+
columns = row.split("\t")
355+
self.assertEqual(len(columns), 4, row)
356+
self.assertEqual(columns[2:], ["0.25", "unknown (ps absent)"], row)
357+
timing_lines = [line for line in res.stdout.splitlines() if "timing batch" in line]
358+
self.assertEqual(len(timing_lines), len(batches))
359+
for line in timing_lines:
360+
self.assertIn("busy count unknown (ps absent)", line)
361+
notice = ("quiet-unverified: no ps on this host, the busy count was not measured; "
362+
"do not use these figures as performance evidence")
363+
self.assertIn(notice, (bench / "_workdir" / "speed.md").read_text())
364+
self.assertEqual(res.stdout.splitlines()[-1], notice)
365+
self.assertEqual((bench / "results" / "speed.md").read_bytes(),
366+
b"COMMITTED SPEED TABLE\n")
367+
368+
def test_quiet_gate_without_ps_labels_figures(self) -> None:
369+
self.assert_unverified_quiet_gate(None)
370+
371+
def test_quiet_gate_without_ps_uses_load_only(self) -> None:
372+
self.assert_unverified_quiet_gate("1")
373+
374+
def test_quiet_gate_with_ps_preserves_verified_output(self) -> None:
375+
ok = self.slot("001-ok")
376+
self.prepare_report_stubs()
377+
ps_bin = self.quiet_gate_path(ps_present=True)
378+
bench = self.root / "benchmarks"
379+
for harness_dir in (None, str(self.harness)):
380+
with self.subTest(harness_dir=harness_dir):
381+
self.reset_report_case(ok)
382+
res = self.run_report_case(PATH=str(ps_bin), PYNECORE_HARNESS_DIR=harness_dir)
383+
self.assertEqual(res.returncode, 0, res.stderr)
384+
rows = (bench / "_workdir" / "speed_loads.tsv").read_text().splitlines()
385+
batches = (["pynecore"] if harness_dir else []) + ["pinets", "vectorbt"]
386+
self.assertEqual([row.split("\t")[0] for row in rows], batches)
387+
for row in rows:
388+
columns = row.split("\t")
389+
self.assertEqual(len(columns), 4, row)
390+
self.assertEqual(columns[2:], ["0.25", "1"], row)
391+
timing_lines = [line for line in res.stdout.splitlines() if "timing batch" in line]
392+
self.assertEqual(len(timing_lines), len(batches))
393+
for batch, line in zip(batches, timing_lines):
394+
self.assertTrue(line.endswith(
395+
f"timing batch {batch}: 1-min load 0.25, build/test processes 1"), line)
396+
report = (bench / ("results" if harness_dir else "_workdir") / "speed.md").read_text()
397+
self.assertEqual(report, "SPY TABLE\n")
398+
for output in (res.stdout, res.stderr, "\n".join(rows), report):
399+
self.assertNotIn("quiet-unverified", output)
400+
self.assertNotIn("unknown", output)
401+
self.assertNotIn("ps absent", output)
402+
311403
def test_absent_harness_without_committed_indicator_output_skips_three_way_report(self) -> None:
312404
ok = self.slot("001-ok")
313405
self.prepare_report_stubs()

0 commit comments

Comments
 (0)