Skip to content

Commit 704f633

Browse files
committed
fix(perf): reject bench-history runs below the minimum
`parse-history-cli` validated a copy with `:runs` raised to 3 but returned the original options, so `--runs 2` passed validation and then measured two rounds, too few for the four-in-five agreement rule. Validate the options that are returned, so fewer than three rounds is rejected as in the PR comparison, and say so in both `--runs` helps. Reported by Pullfrog on #2061.
1 parent df22c5d commit 704f633

3 files changed

Lines changed: 8 additions & 4 deletions

File tree

‎apps/ccusage/scripts/compare-pr-performance_test.clj‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,11 @@
2323
(is (= ["v20.0.20" "main"] (:refs (parse-history-cli ["--real" "v20.0.20" "main"]))))
2424
(is (true? (:real (parse-history-cli ["--real" "main"]))))
2525
(is (thrown-with-msg? clojure.lang.ExceptionInfo #"at least one git ref"
26-
(parse-history-cli [])))))
26+
(parse-history-cli []))))
27+
(testing "history rejects too few rounds instead of measuring them"
28+
(is (thrown-with-msg? clojure.lang.ExceptionInfo #"--runs must be at least 3"
29+
(parse-history-cli ["--runs" "2" "main"])))
30+
(is (= 3 (:runs (parse-history-cli ["--runs" "3" "main"]))))))
2731

2832
(deftest formatting
2933
(is (= "999ms" (format-duration 999)))

‎apps/ccusage/scripts/compare_pr_performance/data.clj‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
:fixture-dir {:ref "DIR" :desc "Claude fixture (CLAUDE_CONFIG_DIR)"}
1515
:codex-fixture-dir {:ref "DIR" :desc "Codex fixture (CODEX_HOME)"}
1616
:output {:ref "FILE"}
17-
:runs {:coerce :long :default 11 :ref "N" :desc "Interleaved measured rounds"}
17+
:runs {:coerce :long :default 11 :ref "N" :desc "Interleaved measured rounds; at least 3"}
1818
:warmup {:coerce :long :default 2 :ref "N" :desc "Interleaved warmup rounds"}
1919
:threshold {:coerce :double :default 0.1 :ref "RATIO" :desc "CPU-time and RSS change that counts as a regression"}
2020
:help {:alias :h :coerce :boolean}})

‎apps/ccusage/scripts/compare_pr_performance/history.clj‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
:codex-fixture-dir {:ref "DIR" :desc "Codex fixture; generated under the cache when omitted"}
44
:size-mib {:coerce :long :default 1024 :ref "N" :desc "Size of each generated fixture"}
55
:real {:coerce :boolean :desc "Also measure the logs this environment points at (default dirs, CLAUDE_CONFIG_DIR, CODEX_HOME)"}
6-
:runs {:coerce :long :default 5 :ref "N"}
6+
:runs {:coerce :long :default 5 :ref "N" :desc "Interleaved measured rounds; at least 3"}
77
:warmup {:coerce :long :default 1 :ref "N"}
88
:threshold {:coerce :double :default 0.1 :ref "RATIO"}
99
:commands {:ref "LIST" :desc "Comma-separated report commands"}
@@ -37,7 +37,7 @@
3737
(when-not (:help opts)
3838
(when (empty? args)
3939
(fail! "Pass at least one git ref or ccusage binary path" {:kind :invalid-cli}))
40-
(assert-sample-options (assoc opts :runs (max 3 (:runs opts)))))
40+
(assert-sample-options opts))
4141
(assoc opts :refs (vec args))))
4242

4343
(defn history-help-text []

0 commit comments

Comments
 (0)