Skip to content

Commit 8f0ff31

Browse files
committed
fix(test): escape benchmark paths, keep stderr, use monotonic clock
Review follow-up for the benchmark harness isolation. - Both benchmark scripts build the index request with a JSON-escaped repository path (python3 json.dumps, the spelling soak-test.sh already uses): a path containing a quote or a backslash produced a payload the server could not parse. - benchmark-index.sh reads its three timestamps from one helper backed by time.monotonic_ns(); an NTP step mid-run no longer skews or negates a figure whose purpose is comparison across runs. - benchmark-search-graph.sh keeps the stderr of the index call and of the response parse in a run-private temp file (removed by the EXIT trap) and prints it, plus the first 500 bytes of the response, under the existing "did not report a project" line. - The contract test now refuses the index from its fixture and requires the cause to reach the output, and (POSIX only) drives a repository path with a quote and a backslash and requires the recorded request to parse as JSON with the resolved path. Both assertions fail against the previous head of this branch. Signed-off-by: Anton Standrik <[email protected]>
1 parent 406a43f commit 8f0ff31

3 files changed

Lines changed: 86 additions & 7 deletions

File tree

‎scripts/benchmark-index.sh‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,17 @@ trap 'cbm_test_runtime_cleanup "$BINARY"' EXIT
1919

2020
# Resolve symlinks
2121
REPO=$(cd "$REPO" && pwd -P)
22+
# One pre-escaped spelling of the path for every request below, the way the
23+
# soak harness builds its own: a repository path may legitimately contain a
24+
# quote or a backslash, and hand-built JSON turns that into a parse error.
25+
REPO_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$REPO")
26+
27+
# Elapsed time is read from a monotonic clock, never the wall clock: an NTP
28+
# step mid-run would otherwise skew — or negate — a figure whose whole purpose
29+
# is comparison across runs. Its reference point is fixed per boot on every
30+
# platform CPython supports here, so the three readings below are comparable
31+
# even though each comes from its own process.
32+
bench_now_ms() { python3 -c "import time; print(time.monotonic_ns() // 1000000)"; }
2233

2334
OUT="$RESULTS_DIR/$LANG"
2435
mkdir -p "$OUT"
@@ -45,18 +56,18 @@ echo "$LOC" > "$OUT/loc.txt"
4556
# alone. setup-time.txt keeps the activation cost attributable and
4657
# total-time.txt is their sum — the figure comparable with earlier runs, which
4758
# paid activation inside the index timing whenever no daemon was already warm.
48-
SETUP_START_MS=$(python3 -c "import time; print(int(time.time()*1000))")
59+
SETUP_START_MS=$(bench_now_ms)
4960
if ! "$BINARY" daemon start >/dev/null 2>&1; then
5061
echo " $LANG: private daemon did not start" >&2
5162
exit 1
5263
fi
5364

5465
# Index via CLI and capture timing
55-
START_MS=$(python3 -c "import time; print(int(time.time()*1000))")
66+
START_MS=$(bench_now_ms)
5667

57-
INDEX_JSON=$("$BINARY" cli index_repository "{\"repo_path\":\"$REPO\",\"mode\":\"full\"}" 2>/dev/null || echo '{"error":"index failed"}')
68+
INDEX_JSON=$("$BINARY" cli index_repository "{\"repo_path\":$REPO_JSON,\"mode\":\"full\"}" 2>/dev/null || echo '{"error":"index failed"}')
5869

59-
END_MS=$(python3 -c "import time; print(int(time.time()*1000))")
70+
END_MS=$(bench_now_ms)
6071
ELAPSED=$((END_MS - START_MS))
6172

6273
echo "$INDEX_JSON" > "$OUT/00-index.json"

‎scripts/benchmark-search-graph.sh‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,22 +22,41 @@ REPO=$(cd "$REPO" && pwd -P)
2222
# shellcheck source=test-runtime.sh
2323
source "$(dirname "${BASH_SOURCE[0]}")/test-runtime.sh"
2424
cbm_test_runtime_init
25-
trap 'cbm_test_runtime_cleanup "$BINARY"' EXIT
25+
BENCH_TMP=""
26+
trap 'cbm_test_runtime_cleanup "$BINARY"; [ -z "$BENCH_TMP" ] || rm -rf -- "$BENCH_TMP"' EXIT
27+
BENCH_TMP=$(mktemp -d)
28+
INDEX_ERR="$BENCH_TMP/index-stderr.log"
2629

2730
if ! "$BINARY" daemon start >/dev/null 2>&1; then
2831
echo "private daemon did not start" >&2
2932
exit 1
3033
fi
31-
INDEX_JSON=$("$BINARY" cli index_repository "{\"repo_path\":\"$REPO\",\"mode\":\"full\"}" 2>/dev/null || echo '{}')
34+
# One pre-escaped spelling of the path, the way the soak harness builds its
35+
# own: a repository path may legitimately contain a quote or a backslash, and
36+
# hand-built JSON turns that into a parse error.
37+
REPO_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$REPO")
38+
# Index and parse keep their stderr instead of discarding it: without it the
39+
# failure below names only its symptom, and the cause — an unreadable
40+
# repository, a refused daemon, a malformed envelope — is unrecoverable.
41+
INDEX_JSON=$("$BINARY" cli index_repository "{\"repo_path\":$REPO_JSON,\"mode\":\"full\"}" \
42+
2>"$INDEX_ERR" || echo '{}')
3243
PROJECT=$(echo "$INDEX_JSON" | python3 -c "
3344
import json, sys
3445
d = json.load(sys.stdin)
3546
if 'content' in d:
3647
d = json.loads(d['content'][0]['text'])
3748
print(d.get('project', ''))
38-
" 2>/dev/null || echo "")
49+
" 2>>"$INDEX_ERR" || echo "")
3950
if [ -z "$PROJECT" ]; then
4051
echo "index of $REPO did not report a project" >&2
52+
if [ -s "$INDEX_ERR" ]; then
53+
echo "--- index/parse stderr ---" >&2
54+
cat "$INDEX_ERR" >&2
55+
fi
56+
if [ -n "$INDEX_JSON" ]; then
57+
echo "--- index response (first 500 bytes) ---" >&2
58+
printf '%.500s\n' "$INDEX_JSON" >&2
59+
fi
4160
exit 1
4261
fi
4362

‎tests/test_benchmark_runtime_isolation_contract.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,11 @@ ENV_PROBE="$WORKDIR/environment-probe"
3333
cat > "$ENV_PROBE" <<'EOF'
3434
#!/usr/bin/env bash
3535
printf '%s\t%s\n' "${CBM_CACHE_DIR-}" "${CBM_RUNTIME_DIR-}" >> "$CBM_BENCH_ENV_PROBE"
36+
if [[ "${1-} ${2-}" == "cli index_repository" ]]; then
37+
[[ -z "${CBM_BENCH_REQUEST_LOG-}" ]] || printf '%s\n' "${3-}" >> "$CBM_BENCH_REQUEST_LOG"
38+
echo "probe: index refused" >&2
39+
exit 1
40+
fi
3641
[[ "${1-} ${2-}" == "daemon status" ]] && exit 1
3742
exit 0
3843
EOF
@@ -93,4 +98,48 @@ CBM_BENCH_ENV_PROBE="$SEARCH_LOG" \
9398
> "$WORKDIR/search.out" 2>&1 || true
9499
assert_isolated "benchmark-search-graph" "$SEARCH_LOG"
95100

101+
# A refused index must name its cause. The fixture writes one line to stderr and
102+
# exits non-zero; discarding it leaves "did not report a project" as the only
103+
# thing an operator sees.
104+
grep -q -- '--- index/parse stderr ---' "$WORKDIR/search.out" ||
105+
fail "benchmark-search-graph hid the index stderr behind its own message"
106+
grep -q 'probe: index refused' "$WORKDIR/search.out" ||
107+
fail "benchmark-search-graph did not surface the cause of the index failure"
108+
109+
# The request carrying the repository path has to be built as JSON. A path may
110+
# legitimately contain a quote or a backslash — hand-built JSON turns that into
111+
# a payload the server cannot parse, or one that means something else. NTFS
112+
# rejects both characters in a path component, so this case is POSIX-only.
113+
case "$(uname -s)" in
114+
MINGW* | MSYS* | CYGWIN*) ;;
115+
*)
116+
QUIRKY_REPO="$WORKDIR/re\"po\\dir"
117+
mkdir -p "$QUIRKY_REPO"
118+
echo 'def bench(): return 1' > "$QUIRKY_REPO/bench.py"
119+
QUIRKY_RESOLVED=$(cd "$QUIRKY_REPO" && pwd -P)
120+
REQUEST_LOG="$WORKDIR/requests.log"
121+
CBM_CACHE_DIR="$CALLER_CACHE" \
122+
CBM_RUNTIME_DIR="$CALLER_RUNTIME" \
123+
CBM_BENCH_ENV_PROBE="$WORKDIR/quirky-environment.log" \
124+
CBM_BENCH_REQUEST_LOG="$REQUEST_LOG" \
125+
"$ROOT/scripts/benchmark-search-graph.sh" "$ENV_PROBE" "$QUIRKY_REPO" \
126+
> "$WORKDIR/quirky.out" 2>&1 || true
127+
python3 - "$REQUEST_LOG" "$QUIRKY_RESOLVED" <<'PY' || fail "benchmark-search-graph built an index request that is not valid JSON for a quoted path"
128+
import json
129+
import sys
130+
131+
request_log, expected = sys.argv[1], sys.argv[2]
132+
try:
133+
lines = [line for line in open(request_log).read().splitlines() if line.strip()]
134+
except OSError:
135+
sys.exit("the search benchmark sent no index request")
136+
if not lines:
137+
sys.exit("the search benchmark sent no index request")
138+
payload = json.loads(lines[0])
139+
if payload.get("repo_path") != expected:
140+
sys.exit(f"repo_path is {payload.get('repo_path')!r}, expected {expected!r}")
141+
PY
142+
;;
143+
esac
144+
96145
echo "PASS: benchmark harnesses isolate their daemon runtime and cache from the caller"

0 commit comments

Comments
 (0)