Skip to content

Fix AgentNetBench evaluator scoring edge cases - #56

Merged
XinyuanWangCS merged 1 commit into
xlang-ai:mainfrom
FU-max-boop:docs/agentnetbench-evaluator-validity
May 25, 2026
Merged

XinyuanWangCS merged 1 commit into
xlang-ai:mainfrom
FU-max-boop:docs/agentnetbench-evaluator-validity

Conversation

@FU-max-boop

Copy link
Copy Markdown
Contributor

Summary

This PR tightens several AgentNetBench action-scoring edge cases and adds offline regression tests for them.

Changes:

  • avoid installing editdistance at import time; use a built-in Levenshtein fallback if the optional package is missing
  • score scroll actions by direction and amount instead of action type only
  • compare press / hotkey key sequences while preserving order and repeated keys
  • penalize extra predicted actions after an otherwise correct action

Why

These edge cases can make an evaluator report overly optimistic scores:

  • opposite-direction scrolls could receive full credit
  • ["ctrl", "v"] and ["v", "ctrl"] were treated as equivalent because key comparison used sets
  • repeated keys such as ["down", "down"] could collapse to one key
  • a correct first action followed by unrelated extra actions was not penalized
  • importing the evaluator could mutate the runtime by installing a package

Validation

From the repo root:

python3 evaluation/agentnetbench/test_eval.py

Result:

Ran 6 tests in 0.001s
OK

I also checked unittest discovery:

python3 -m unittest discover evaluation/agentnetbench -p 'test_eval.py'

Result:

Ran 6 tests in 0.003s
OK

The tests cover:

  • missing optional editdistance
  • wrong-direction scroll
  • same-direction scroll amount similarity
  • hotkey order mismatch
  • repeated-key mismatch
  • extra-action penalty

Scope / limits

This PR does not change the model runner or sample data. It only updates local action scoring and adds offline regression coverage for the changed behavior.

@XinyuanWangCS
XinyuanWangCS merged commit dfc91ba into xlang-ai:main May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants