Repository navigation
fix(leaderboard): keep the sympy check working where SIGALRM is unavailable - #4325
Open
barlowa124 wants to merge 1 commit into
Open
barlowa124 wants to merge 1 commit into
barlowa124 wants to merge 1 commit into
Conversation
…ilable The leaderboard math is_equiv guards sympy with the same SIGALRM-based timeout as minerva_math and putnam_axiom. Where the handler cannot be installed (Windows, worker threads) the AttributeError or ValueError reached the catch-all in is_equiv and scored every comparison that needs sympy as wrong. Skip arming when the handler cannot be installed instead of raising; the comparison then runs without a time limit. Refs EleutherAI#4286.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #4286.
The SIGALRM timeout census on that issue lists six sites. Two open PRs cover five of them: #4287 guards
minerva_mathandputnam_axiom, and #4318 guards the twotime_limitcopies where a missed disarm leavesITIMER_REALarmed. This PR covers the sixth,leaderboard/math/utils.py, which has the same unguardedsignal.signal/signal.alarmpair and the same wrong-score failure mode on Windows and off the main thread.The change mirrors #4287: if
signal.signalraisesAttributeError(no SIGALRM) orValueError(not the main thread), the timeout stays unarmed and the comparison runs without a time limit rather than reaching theis_equivcatch-all and scoring 0.New tests in
tests/test_minerva_math.pycover both failure modes for the leaderboard copy:monkeypatch.delattr(signal, "SIGALRM")and a worker-thread call. Both fail on the unguarded code (is_equiv("\frac{1}{2}", "0.5")returnsFalse) and pass with the fix. The three other failures in that file are the NORMS issue tracked in #4073 and occur identically onupstream/main.