Skip to content

fix: reset per-attempt cost in SetStatusAgentHook.on_setup_attempt - #1568

Open
BlueX888 wants to merge 1 commit into
SWE-agent:mainfrom
BlueX888:fix/prep-status-hook-cost-double-count
Open

BlueX888 wants to merge 1 commit into
SWE-agent:mainfrom
BlueX888:fix/prep-status-hook-cost-double-count

Conversation

@BlueX888

Copy link
Copy Markdown

Reference Issues/PRs

Fixes #1567

What does this implement/fix? Explain your changes.

Summary

SetStatusAgentHook renders the per-instance status line in the batch progress UI. On a retry attempt it double-counted the previous attempt's spend until the new attempt's first step completed, so the status line showed roughly 2x the actual spend (e.g. Attempt 2 Step 1 ($2.00) when only $1.00 had been spent).

Why (root cause)

sweagent/run/run_batch.py:347 attaches a SetStatusAgentHook to every batch agent. With a retry loop, RetryAgent._next_attempt (sweagent/agent/agents.py:321) re-attaches the same hook objects to a fresh DefaultAgent, whose setup fires on_setup_attempt (sweagent/agent/agents.py:594) at the start of every attempt. In on_setup_attempt (sweagent/agent/hooks/status.py:16-20) the finished attempt's cost was folded into _previous_cost:

# Costs will be reset for the next attempt
self._previous_cost += self._cost

but self._cost was never reset, even though the comment says it would be. Because the status line shows ${self._previous_cost + self._cost:.2f} (sweagent/agent/hooks/status.py:29), every status line of the new attempt displayed the previous attempt's spend twice until on_step_done (sweagent/agent/hooks/status.py:31) overwrote _cost with the new attempt's instance_cost. The previous+current split only yields the true total spend if _cost is zeroed at the attempt boundary.

Changes

  • sweagent/agent/hooks/status.py: reset self._cost = 0.0 in on_setup_attempt right after folding the finished attempt's cost into _previous_cost, so _previous_cost + _cost is the true total spend from the first status line of a retry attempt onward. For non-retry agents this is a no-op (_cost is still 0.0 at the single setup call).
  • tests/test_agent_hooks.py: new regression test that drives the hook through two attempts and asserts the status line shows $1.00 (not $2.00) at the start of attempt 2, and $1.25 after attempt 2's first step adds $0.25.

Testing

New regression test, failing on main (3ea751c):

$ python -m pytest tests/test_agent_hooks.py -v
>       assert statuses[-1] == "Attempt 2 Step   1 ($1.00)"
E       AssertionError: assert 'Attempt 2 Step   1 ($2.00)' == 'Attempt 2 Step   1 ($1.00)'
1 failed

Passing with this fix:

$ python -m pytest tests/test_agent_hooks.py -v
tests/test_agent_hooks.py::test_status_hook_cost_not_double_counted_on_retry PASSED [100%]
1 passed

Existing tests around the changed module:

$ python -m pytest tests/test_agent.py tests/test_agent_hooks.py -v
12 passed, 2 xfailed            # both xfails pre-existing (test_run_autosubmit, test_successful_submission)

$ python -m pytest -m "not slow" tests/test_run_batch.py tests/test_batch_instance.py -v
8 passed, 3 deselected           # deselected = slow tests that spin up real batch runs

Lint/format (pre-commit config, ruff):

$ ruff check sweagent/agent/hooks/status.py tests/test_agent_hooks.py
All checks passed!
$ ruff format --check sweagent/agent/hooks/status.py tests/test_agent_hooks.py
2 files already formatted

Note: in the sandbox used for this work, the GitHub-API/network-dependent tests (tests/test_run_hooks.py should_open_pr tests, tests/test_env_utils.py::test_get_associated_commit_urls, tests/test_run_replay.py::test_replay) fail or stall identically on unpatched main, so they could not be exercised here; they are unaffected by this change (no code path they use was modified).

on_setup_attempt folds the finished attempt's cost into _previous_cost but
never resets _cost, even though every retry attempt starts from a fresh
agent whose instance_cost starts at 0 again. The status line shows
_previous_cost + _cost, so until the new attempt's first step completed,
each status line displayed the previous attempt's cost twice (roughly 2x
the actual spend in the batch progress UI).

@chessing234 chessing234 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the retry lifecycle and cost folding/reset. The focused status-hook regression test passes.

No blocking issue found in the reviewed diff. Validation was limited to the focused checks described above; the full integration suite was not run.

@chessing234 chessing234 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes as a reviewer-directed hold pending specific revision requirements. No additional technical defect is asserted in this follow-up; the findings and validation limits in my earlier review still apply.

@chessing234 chessing234 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed retry-attempt cost accumulation and reset against the lifecycle call sites. The status-hook regression test passed.

No new blocking defect found in this review. These were focused local checks; the full integration suite was not run.

@BlueX888

Copy link
Copy Markdown
Author

Thanks for taking a look, and sorry for the slow reply on our side.

Status right now: head is e97d7c0d, pre-commit.ci passes, and there have been no new commits since 2026-10-05.

We're treating the 2026-10-05 hold as still in force. Your note mentions "specific revision requirements", but we can't find them written down anywhere on this PR, and we'd rather ask than guess at what you meant.

Could you tell us whether there is anything specific we should do to lift the hold? If so, please spell it out and we'll do it. We're also happy to run the full integration suite if that would help.

sun04222 added a commit to sun04222/SWE-agent that referenced this pull request Oct 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SetStatusAgentHook never resets _cost on retry attempts, double-counting the previous attempt's spend in the batch status line

2 participants