robustness-eval

Set up:

git clone ...
cd ...
uv venv
source .venv/bin/activate
uv sync

Generate variants

python robustness-analyses/main.py augment \
    data/aimo-2025-reference.jsonl \
    prompts/paraphrase.txt \
    data/ \
    --api-model "gpt-5.2-2025-12-11" \
    --provider openai \
    --n-variants 10

Generate predictions

Predictions for base problems:

python robustness-analyses/main.py predict \
    data/aimo-2025-reference.jsonl \
    predictions/ \
    --api-model "gpt-5.2-2025-12-11" \
    --provider openai \
    --n-repeats 10

Predictions for augmented problems:

python robustness-analyses/main.py predict \
    data/aimo-2025-reference___paraphrase=gpt-5.2-2025-12-11.jsonl \
    predictions/ \
    --api-model "gpt-5.2-2025-12-11" \
    --provider openai \
    --n-repeats 1

Evaluate

Evaluate standalone base prediction file:

python robustness-analyses/main.py eval \
    predictions/aimo-2025-reference___eval=gpt-5.2-2025-12-11.jsonl

Evaluate augmented prediction file:

python robustness-analyses/main.py eval \
    predictions/aimo-2025-reference___paraphrase=gpt-5.2-2025-12-11___eval=gpt-5.2-2025-12-11.jsonl

Or compare the augmented to the base:

python robustness-analyses/main.py eval \
    predictions/aimo-2025-reference___paraphrase=gpt-5.2-2025-12-11___eval=gpt-5.2-2025-12-11.jsonl \
    --base-pred-file predictions/aimo-2025-reference___eval=gpt-5.2-2025-12-11.jsonl