fix: wire eval_path and iterations_left into MetaAgent.forward#36
fix: wire eval_path and iterations_left into MetaAgent.forward#36xbrxr03 wants to merge 1 commit into
Conversation
MetaAgent.forward accepts eval_path and iterations_left but silently drops them — the agent never sees prior evaluation results or knows its iteration budget, making recursive self-improvement amnesic. Changes: - Add _read_eval_summaries() helper that scans eval_path for JSON score files and markdown reports, extracting score-like fields (pass_rate, accuracy, reward, success, etc.) into a prompt-friendly summary - Append eval summaries to the agent instruction when available - Include iteration budget in the instruction when iterations_left is provided, guiding the agent to prioritize high-impact changes - Add 9 unit tests for _read_eval_summaries covering edge cases (None paths, empty dirs, corrupt JSON, mixed file types) Fixes facebookresearch#30
|
Hi @xbrxr03! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
|
Closing this PR due to lack of maintainer engagement. I'm happy to reopen if there's interest. Thanks for the great project. |
2 participants
Summary
MetaAgent.forwardacceptseval_pathanditerations_leftparameters but silently drops them — the agent never sees prior evaluation results or knows its iteration budget, making recursive self-improvement amnesic.Fixes #30.
What changed
meta_agent.py_read_eval_summaries(eval_path)helper that scans the eval directory for:pass_rate,accuracy,reward,success,score,metric,result)eval_pathhas results, they are appended to the agent instruction so the meta-agent can learn from prior generationsiterations_leftis provided, the instruction includes the remaining budget and guidance to prioritize high-impact improvementstest_meta_agent.py(new)_read_eval_summariescovering: None/empty paths, nonexistent directories, JSON extraction, markdown preview, irrelevant key filtering, mixed files, corrupt JSONTesting
All edge cases handled: missing directories, corrupt JSON, empty eval folders, mixed file types. The helper degrades gracefully — if eval_path is missing or contains no relevant data, the agent falls back to the original bare instruction.
Notes
run_meta_agent.py— callers already pass these args correctly, they were just being ignored_read_eval_summariesfunction is deliberately conservative: it only extracts fields with score-like names, not arbitrary JSON keys, to keep the prompt focused