Skip to content

About

Math-Verify failure audit: unreviewed manuscript, code and evidence

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

11 Commits

Folders and files

Repository files navigation

When the Verifier Cannot Decide

Tracing Math-Verify Failures Through an RLVR Reward Pipeline

Yufeiyang Chen · manuscript revised 12 October 2026

Read the paper · Download companion materials

This is a publicly available research manuscript. It has not been peer reviewed or posted to arXiv. The companion archive contains code and evidence for inspection; it does not contain model weights, upstream datasets or the complete original raw-log archive.

Study

The study traces failure information from Math-Verify through a pinned verl reward wrapper, manager and batch aggregation. Controlled worker-failure tests reproduce a known sparse-metadata problem: the position of a failed row can determine whether its flag is lost or the batch raises an exception. The manuscript cites the related upstream proposals #6545 and #7151. Its contribution is the specific wrapper-to-consumer failure trace and the paired corpus and timeout/recovery measurements. The local aggregation prototype supplies 21 regression fixtures, not a new repair algorithm.

A pre-declared replay covers 93,733 stored mathematical responses. Longer checking limits recover one zero-to-one verdict and leave 18 responses undecided. A separate extension covers all 193,767 stored answers. Matched shared-pool tests and a fixed-source comparison distinguish the behavior of wrappers and execution settings.

These results are conditional on the fixed corpora and declared protocols. Longer checking is not mathematical ground truth. Controlled failure injections and shared-pool diagnostics do not measure failure rates in training, and no training benefit has been measured. The paper states the sampling assumptions, comparison units and limits of inference.

Inspect the evidence

Extract the companion archive into a fresh directory. From its root, run:

python3 -B experiments/check_artifacts.py --index package_manifest.json
python3 -B experiments/compare_runs.py

These commands inspect the distributed snapshot without installing packages or downloading data. The archive includes compact reports, derived score records, selected event traces, synthetic checks, exact upstream source revisions and a file manifest. Its README explains which records are included and which are omitted.

Replaying the measurements requires the documented dependencies, upstream sources and separately obtained datasets. The measured environment was macOS arm64 with CPython 3.12.8. A fresh installation and Linux have not been experimentally validated. Follow the archive's instructions in a fresh environment; do not overwrite retained original records or an existing measurement environment.

Public evidence clarification (3 November 2026)

The manuscript PDF and its reported measurements are unchanged. The companion now includes EVIDENCE_STATUS.md and a read-only, standard-library audit:

python3 -B experiments/audit_public_evidence.py

This independently counts the distributed default-pass derived records and checks recorded regression outcomes. It is not a new scoring experiment or an independent raw-event replication. The corrected main reference records and complete historical reanalysis inputs remain unavailable in the public package, so k=1 and u=18 are reported values, not independently reconstructed by this audit. The note also distinguishes reference-protocol unresolved rows from all library execution failures and lists the evidence still needed. No training benefit or fresh-machine replication has been added.

Preprint package (3 November 2026, awaiting author review)

Evidence audit and editable revision (3 November 2026, awaiting author review)

This branch adds, without changing paper.pdf or the companion archive:

  • audit/EVIDENCE_MAP.md: claim-to-evidence table and every check run.
  • audit/GAPS.md: what still cannot be verified and the smallest input that would close each gap.
  • audit/REPRODUCE.md: commands, environment and stored results.
  • audit/upstream_chain/: real-function failure-chain tests and two patches for verl 0c4c3d4 (a port of the open upstream PR #7151, and a validation-accumulation fix).
  • audit/experiments/e3_linux/: a new, pre-specified Linux replay of the labelled answer set.
  • paper/: an editable LaTeX reconstruction from the public PDF (not the original source), its revision log, build notes and the compiled revised PDF.
  • audit/THESIS_HANDOFF.md: items a dependent thesis must re-check.

New executions ran on Linux x86-64 (CPython 3.12.8); they are additional verification, not a replay of the historical macOS environment. Recounting the distributed derived records is not an independent reproduction from raw data, and k = 1 / u = 18 remain report-only values.

Contribution and assistance

The author selected the research topic, research route and methods. AI agents provided substantial assistance with implementation, experiment execution, code analysis and manuscript drafting. The manuscript contains the full contribution and AI-use statement. Human answer labels in the supplementary analysis were supplied by the cited dataset authors.

Licences and fixed files

  • Manuscript: CC BY 4.0.
  • Original companion code and synthetic fixtures: Apache-2.0; see LICENSE and NOTICE inside the archive. Third-party sources and datasets retain their own licences and notices.

SHA-256 of the distributed files:

8efbb3471d23a8b5396bc1605fb5cf29d4769ee6e0214ede8e63e76f69c46811  paper.pdf
2aaf2113ebc11a5396dc17408a40534419bb14b70589de30a832c146a1381ae4  rewardfail-companion.zip

About

Math-Verify failure audit: unreviewed manuscript, code and evidence

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages