autprompt-lab is a ~500-line, CPU-only study of automatic prompt optimisation against a frozen model with an exact verifier. Three search procedures - random restarts, greedy hill climbing, and Promptbreeder-style population evolution - are handed the same budget of model queries on the same frozen base model, and the only difference is how each spends that budget searching text. The honest, non-obvious finding: on this landscape the simplest budgeted searcher wins, so we report the measurement instead of cherry-picking a win for the fancy method.
- Frozen base model: a ~110k-parameter decoder-only transformer is fine-tuned to read a prompt (a framing prefix + an operation paraphrase) followed by a short list of digits and emit the requested statistic as two digits. After fine-tuning its weights are frozen; the study never touches them again.
- Verifiable fitness: the answer is a plain integer graded by an exact verifier, so a prompt's score is real held-out accuracy - never a model's opinion or a hand-assigned number.
- Prompt space: 3 framings x 8 operations x 6 paraphrases each. Only one framing and one paraphrase per operation were ever shown in training, so which unseen wording the frozen model still obeys is an empirical property it must be measured for, not declared.
- Black-box budget: an optimiser may only query the model with a candidate prompt. Cost is charged per query, which is the currency that matters when the model is a paid API.
Every accuracy number comes from sampling the trained model's own generations and
grading them with the exact verifier. The three methods are compared on identical
frozen weights at an identical query budget, and a full grid-scan gives the ceiling
they are all measured against. Crucially the prose in the results block is rendered
mechanically from the committed JSON by experiments/make_report.py, and a CI test
asserts the README equals it byte-for-byte - so numbers and superlatives cannot drift
or be overclaimed.
pip install -e . # torch is the only runtime dependency
python -m autprompt.cli demo # one seeded run, ~a minute
python experiments/run_study.py # full 3-seed study -> results/optimisation.json
python experiments/make_report.py # re-render the README block from the JSONEvery figure below is produced by experiments/run_study.py on CPU and committed as results/optimisation.json; the tables are rendered by experiments/make_report.py. Same frozen base model per seed, same held-out verifier, same budget of model queries - the only thing that differs is the prompt-search procedure. Mean over 3 seeds, 120 queries each.
- prompt space: 120-query budget over 3 framings x 8 operations x 6 paraphrases each
- fitness = exact verifier accuracy over 32 held-out digit lists; nothing is modelled or assumed
- full grid-scan ceiling (see every cell) = 0.927
| search procedure | final accuracy | std | % of ceiling |
|---|---|---|---|
| Greedy hill climb | 0.872 | 0.116 | 94% |
| Population evolution (Promptbreeder-style) | 0.815 | 0.026 | 88% |
| Random restarts | 0.676 | 0.024 | 73% |
On this landscape the simplest budgeted searcher wins: Greedy hill climb reaches 0.872 (94% of the grid-scan ceiling) within 120 queries, ahead of Population evolution (Promptbreeder-style) (0.815) and Random restarts (0.676). That is the honest, non-obvious result this lab exists to report: prompt accuracy decomposes almost independently per operation, so greedy single-edit coordinate ascent climbs it efficiently, and the extra machinery of population evolution pays a diversity tax it cannot recoup inside a tight query budget. We ship the measurement, not a cherry-picked win for the fancy method.
| model queries | 12 | 24 | 48 | 72 | 96 | 120 |
|---|---|---|---|---|---|---|
| Random restarts | 0.652 | 0.663 | 0.676 | 0.676 | 0.676 | 0.676 |
| Greedy hill climb | 0.578 | 0.615 | 0.772 | 0.797 | 0.872 | 0.872 |
| Population evolution (Promptbreeder-style) | 0.652 | 0.652 | 0.706 | 0.754 | 0.811 | 0.815 |
All three start near the random prompt and separate as the budget grows; Greedy hill climb keeps gaining while Random restarts has already plateaued, i.e. random restarts exhaust the reachable set long before the budget ends.
- The framing 'role' prefix barely matters. A grid scan gives each of the three framings a best-achievable accuracy of 0.927/0.922/0.927, a spread of only 0.005. Accuracy is carried by the paraphrase, not the persona line, which is why random and evolution end on the trained framing only ~33% of the time - chance across three framings. (Hill climb always 'finds' the trained framing only because it never edits that slot; it inherits its random start, not a discovery.) A system prompt that contributes nothing measurable is exactly what a verifier-guarded search should expose.
- Random already scores 0.68, not ~0. If unseen paraphrases simply broke the model, a random prompt would sit near the all-wrong floor. It does not, because the model transfers across synonyms that share the key word ('third number' works for the trained 'third value'). The optimiser is exploiting genuine, measured generalisation, not memorising one lucky string.
- Deliberately toy: 8 positional/statistic operations on 6 digits, a ~110k-parameter transformer. Real prompt optimisation targets free-form instruction following; the procedure (black-box budgeted search scored by a verifier) is what transfers, not this task.
- The landscape is near-separable, which is precisely why hill climbing wins here. Population/LLM-proposer methods earn their cost when good phrasings are hard to reach by single edits; on this task they are not, and we say so rather than rigging a win.
- We freeze the model and only search prompts. This isolates prompt optimisation from fine-tuning; the accuracy ceiling is a property of the frozen weights, and no method beats the grid-scan ceiling.