Skip to content

fix(core): dispatch detailed bugbot review comments to agents - #1334

Merged
harsh-batheja merged 3 commits into
mainfrom
feat/issue-895
Apr 22, 2026
Merged

harsh-batheja merged 3 commits into
mainfrom
feat/issue-895

Conversation

@harsh-batheja

Copy link
Copy Markdown
Contributor

Summary

Fixes #895 — the bugbot-comments reaction was sending a generic message, leaving the agent to rediscover comments on its own. The naive GET /pulls/{pr}/comments first-page scan misses new comments on later pages and trusts a stale gh pr checks status, causing the agent to falsely report "all reviews addressed."

Changes

  • packages/core/src/lifecycle-manager.ts
    • Add formatAutomatedCommentsMessage(AutomatedComment[]), mirroring formatCIFailureMessage. Lists each comment (severity, bot name, path:line, first-line excerpt, URL) and embeds correct-API guidance.
    • In maybeDispatchReviewBacklog, override the reaction config's message with the formatted one for send-to-agent dispatches of bugbot-comments. Fingerprint/dedup path unchanged.
  • packages/core/src/config.ts — update the default bugbot-comments reaction text to embed the /reviews → /reviews/{id}/comments → paginated /pulls/{pr}/comments (checking in_reply_to_id) sequence as a fallback for custom configs that don't rely on the dispatcher.
  • packages/core/src/__tests__/lifecycle-manager.test.ts — update the "dispatches automated review comments only once" test to assert the new message contains comment details AND the correct-API guidance.

Why this design

The core already paginates getAutomatedComments correctly and knows every bot comment. The old code threw that data away when dispatching and let the agent re-query with a broken pattern. Passing the already-fetched data to the agent removes the failure mode entirely; embedding the correct verification steps guards against agents that still want to re-check.

Test plan

  • pnpm --filter @aoagents/ao-core typecheck — passes
  • pnpm --filter @aoagents/ao-core test — 792 tests pass
  • pnpm typecheck (all packages) — passes
  • pnpm lint — no errors (32 pre-existing warnings unrelated to this change)
  • Manual verification on a PR with paginated bugbot comments once merged

The bugbot-comments reaction previously sent a generic "fix the issues"
message, leaving the agent to discover comments on its own. The typical
agent flow — fetch `GET /pulls/{pr}/comments` (first page only) and
filter for a bugbot marker — misses new comments when the response is
paginated, and trusts a stale `gh pr checks` status.

Now we format the already-fetched `AutomatedComment[]` into a detailed
message that lists each comment (severity, path:line, bot name, excerpt,
URL) and embeds explicit API-level guidance so the agent can verify via
`/reviews` + `/reviews/{id}/comments` + paginated `/pulls/{pr}/comments`
(checking `in_reply_to_id` for replies) instead of the naive first-page
scan.

Closes #895
@github-actions

github-actions Bot commented Apr 19, 2026 •

Copy link
Copy Markdown
Contributor

Test Coverage Report

Metric Value
Lines covered 774/867
Lines not covered 93/867
Overall coverage 89.3%
Per-file breakdown
File Coverage
packages/core/src/config.ts 161/195 (82.6%)
packages/core/src/format-automated-comments.ts 16/16 (100.0%)
packages/core/src/lifecycle-manager.ts 597/656 (91.0%)

Uncovered lines

  • packages/core/src/config.ts: L33, L43, L214-L215, L222-L223, L298, L310-L311, L342, L353, L457-L458, L466, L468, L471-L472, L474, L720-L724, L738, L751, L779, L781-L782, L785-L787, L790, L792, L821
  • packages/core/src/lifecycle-manager.ts: L85, L89, L91, L101, L164, L195, L197, L396-L397, L407, L413-L414, L430, L451-L453, L481, L548-L551, L568-L572, L575, L626, L644, L648-L649, L692, L733, L815, L842-L843, L955, L959-L961, L966, L971, L977-L978, L1004, L1012, L1034, L1040-L1041, L1050, L1919, L2008, L2014, L2019, L2040, L2050, L2063-L2064, L2074

Follow-up to #895 review:

- Extract formatAutomatedCommentsMessage to a module-level helper
  (packages/core/src/format-automated-comments.ts) so it is directly
  unit-testable rather than closure-scoped inside createLifecycleManager.
- Accept optional PRInfo and interpolate owner/repo/number into the
  verification guidance when dispatched from the lifecycle; fall back
  to OWNER/REPO/PR placeholders otherwise.
- Drop the dead `if (c.url)` guard (url is required on AutomatedComment).
- Append `…` when the first-line excerpt is truncated at 160 chars.
- Simplify the config.ts default message to a short fallback and note
  inline that the lifecycle dispatcher injects the rich listing.
- Add 11 focused unit tests for the formatter (interpolation, ellipsis,
  first-line extraction, missing path/line, severity rendering, ordering).

@illegalcall illegalcall left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the bugbot detail-dispatch change. The extraction into format-automated-comments.ts with its own test suite is clean, PR-slug interpolation with placeholder fallback is well thought out, and typecheck + core tests pass on a fresh checkout. A few issues to address before merge — inline comments below.

Repo-wide note (not line-specific): this PR changes runtime behavior in @aoagents/ao-core but has no .changeset/*.md entry. Please add a patch-level changeset so the next release notes reflect the dispatched-message change.

Summary of inline findings:

  • High: silent override of user-customized reactions.bugbot-comments.message
  • High: step-2 guidance omits --paginate, reintroducing the pagination failure mode from #895
  • High: bot comment bodies inserted verbatim into the agent prompt (prompt-injection surface)
  • Med: line === 0 is treated as falsy and drops the line number
  • Med: blank/markdown-heading first lines produce empty or misleading excerpts
  • Low: "reply to resolve" wording doesn't match GitHub's review-thread resolution flow

Comment thread packages/core/src/lifecycle-manager.ts Resolved
Resolves the 6 comments from @illegalcall on #1334:

- H: Sentinel-gate the default-message override — export
  DEFAULT_BUGBOT_COMMENTS_MESSAGE from config.ts and only replace the
  reaction message when it matches the sentinel. User-customized
  messages in project YAML are left untouched.
- H: Add --paginate to step 2 of the verification guidance. Step 2 was
  previously missing --paginate, reintroducing the exact pagination
  failure mode #895 is meant to fix.
- H: Prompt-injection mitigation — strip backticks from comment body
  excerpts, wrap each excerpt in a code span so the content cannot
  break out, and add an "untrusted third-party data" preamble telling
  the agent not to treat excerpts as instructions.
- M: Preserve line number when c.line === 0 (file-level or 0-indexed
  tools). Was previously treated as falsy and dropped.
- M: Skip leading blank lines in excerpt extraction; strip leading
  markdown heading markers (### Title) and whole-line bold/italic
  wrappers before truncating.
- L: Correct the "reply resolves the thread" wording — replying alone
  does not resolve a review thread on GitHub; resolution is a separate
  "Resolve conversation" action.

Also adds a patch-level changeset for @aoagents/ao-core and expands
the test suite to 811 tests (was 803) covering sentinel override,
custom-message passthrough, line===0, backtick escaping, heading
stripping, and step-2 pagination regression.
@harsh-batheja
harsh-batheja merged commit 331f1ce into main Apr 22, 2026
10 checks passed
harshitsinghbhandari pushed a commit to harshitsinghbhandari/agent-orchestrator that referenced this pull request May 1, 2026
- Rewrite the @aoagents/ao-core changeset to describe only what feat/1226
  contributes: event enrichment with schemaVersion: 2, issueTitle
  persistence, executeReaction refactor, maybeDispatchCIFailureDetails,
  and bugbot-comments enrichment. Drop the false claims about adding
  spawn-target and format-automated-comments (those modules came from
  upstream PRs OrchestratorInc#1330 and OrchestratorInc#1334).
- Delete stop-kill-all-resilience.md — its claims (kill-all loop,
  fd-safety in tryAcquire, allSessionPrefixes unify) are no longer
  in the branch after the rebase took upstream's canonical stop logic.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
harshitsinghbhandari pushed a commit that referenced this pull request May 1, 2026
* fix: serialize ao start and stop numbered orchestrators (#1306)

* fix: restore dead orchestrators on start (#1306)

* fix: harden startup lock handling (#1306)

* feat(core): enrich events with PR title, description, and URL

Adds PR and issue context to all event payloads sent to notifiers.
External consumers (Telegram, Discord, n8n) can now display meaningful
information without making additional API calls.

Changes:
- Add buildEventContext() helper to extract PR/issue context from session
- Enrich all createEvent() calls with context data (pr, issueId, issueTitle, branch)
- Store issueTitle in session metadata during spawn
- Add issueTitle field to SessionMetadata interface
- Update executeReaction() to accept session for context access
- Add tests for event enrichment

The context includes:
- pr: { url, title, number, branch } when PR exists
- issueId: issue identifier
- issueTitle: issue title (from tracker during spawn)
- branch: session branch name

Events before PR creation gracefully omit PR fields (pr: null).
Existing webhook consumers that ignore unknown fields are unaffected.

Closes #1226

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* fix(core): address review comments for event enrichment

- Add issueTitle to readMetadata/writeMetadata for proper persistence
- Create ReactionSessionContext type for type-safe system events
- Replace unsafe `as unknown as Session` cast with proper union type
- Add end-to-end test verifying issueTitle persistence during spawn

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* fix(cli): include lock file path in startup lock error message

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address review feedback — fd safety, kill-all resilience, issueTitle restore

- Restore try/catch/finally in tryAcquire for fd leak prevention
- Wrap stop command's sm.list()+kill in try/catch so dashboard shutdown
  always runs even on session listing failure
- Add per-iteration error handling in kill-all loop with partial failure
  reporting (spinner.warn for mixed results)
- Unify allSessionPrefixes derivation between start and stop commands
- Propagate issueTitle through archive restore path
- Add clarifying comments on agentInfo.summary fallback and intentional
  prNumber/prUrl duplication in event data
- Add ora warn mock for stop tests
- Update changeset to minor (event enrichment is a feature) and add CLI
  changeset for stop resilience
- Add tests for kill-all error mid-loop and issueTitle archive restore

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address harsh-batheja review — title/summary split, lock grace, context namespace

- Separate PR title from agent summary in EventContext: title is null
  until enrichment cache populates; summary is a distinct field so
  webhook consumers never confuse the task summary for a PR title.
- Restore UNPARSEABLE_LOCK_GRACE_MS (5s mtime grace) and
  isStaleUnparseableLock lost during merge conflict resolution —
  prevents lockfile-steal race when process A just created the file
  but hasn't written metadata yet.
- Fix orchestrator sort: extract numeric suffix instead of
  localeCompare so -10 sorts after -2, not before.
- Namespace context under data.context instead of spreading into data
  to prevent field collisions with reaction-specific keys.
- Add schemaVersion: 2 to all enriched events so consumers can
  migrate away from top-level prNumber/prUrl (kept for compat,
  marked for removal in v3).
- Update event enrichment tests for nested context structure.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: resolve merge conflicts — use formatReviewCommentsMessage, fix batch enrichment mock

- Replace formatAutomatedCommentsMessage with upstream's formatReviewCommentsMessage
  for automated review comment dispatch (fixes type mismatch with ReviewComment[])
- Make createMockSCM's enrichSessionsPRBatch dynamically resolve from individual
  method mocks so test overrides (e.g. getPRState("closed")) propagate correctly
- Add explicit enrichSessionsPRBatch to merge-conflict-tracking test to avoid
  unexpected getMergeability calls from the dynamic mock
- Remove all debug console.log statements added during troubleshooting

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: wire up maybeDispatchCIFailureDetails and record dispatch hash on transition

The function was defined but never called after merge conflict resolution
dropped the call site. Added it back to the Promise.allSettled alongside
maybeDispatchReviewBacklog and maybeDispatchMergeConflicts.

Also updated the transition-reaction early-return to record the dispatch
hash, since the transition path now enriches the CI message with detailed
check info from the batch cache — preventing duplicate sends on subsequent
polls.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: remove stale duplicate test from rebase

Removes the orphaned numbered-orchestrator restoration test left over
from feat/1226 history. Upstream's canonical model test (same name,
expects "app-orchestrator" via ensureOrchestrator) supersedes it.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* docs: remove misleading CLI CHANGELOG entry

The "Restore the most recently active dead orchestrator on ao start"
entry described upstream's ensureOrchestrator behavior (#1487), not
work contributed by this PR.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* docs(changeset): correct scope and drop stale CLI changeset

- Rewrite the @aoagents/ao-core changeset to describe only what feat/1226
  contributes: event enrichment with schemaVersion: 2, issueTitle
  persistence, executeReaction refactor, maybeDispatchCIFailureDetails,
  and bugbot-comments enrichment. Drop the false claims about adding
  spawn-target and format-automated-comments (those modules came from
  upstream PRs #1330 and #1334).
- Delete stop-kill-all-resilience.md — its claims (kill-all loop,
  fd-safety in tryAcquire, allSessionPrefixes unify) are no longer
  in the branch after the rebase took upstream's canonical stop logic.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

---------

Co-authored-by: Claude Opus 4.5 <[email protected]>
alerios pushed a commit to AIMagicFlow/agent-orchestrator that referenced this pull request Oct 14, 2026
…tratorInc#1334)

* fix(core): dispatch detailed bugbot review comments to agents

The bugbot-comments reaction previously sent a generic "fix the issues"
message, leaving the agent to discover comments on its own. The typical
agent flow — fetch `GET /pulls/{pr}/comments` (first page only) and
filter for a bugbot marker — misses new comments when the response is
paginated, and trusts a stale `gh pr checks` status.

Now we format the already-fetched `AutomatedComment[]` into a detailed
message that lists each comment (severity, path:line, bot name, excerpt,
URL) and embeds explicit API-level guidance so the agent can verify via
`/reviews` + `/reviews/{id}/comments` + paginated `/pulls/{pr}/comments`
(checking `in_reply_to_id` for replies) instead of the naive first-page
scan.

Closes OrchestratorInc#895

* refactor(core): extract bugbot comment formatter, address review

Follow-up to OrchestratorInc#895 review:

- Extract formatAutomatedCommentsMessage to a module-level helper
  (packages/core/src/format-automated-comments.ts) so it is directly
  unit-testable rather than closure-scoped inside createLifecycleManager.
- Accept optional PRInfo and interpolate owner/repo/number into the
  verification guidance when dispatched from the lifecycle; fall back
  to OWNER/REPO/PR placeholders otherwise.
- Drop the dead `if (c.url)` guard (url is required on AutomatedComment).
- Append `…` when the first-line excerpt is truncated at 160 chars.
- Simplify the config.ts default message to a short fallback and note
  inline that the lifecycle dispatcher injects the rich listing.
- Add 11 focused unit tests for the formatter (interpolation, ellipsis,
  first-line extraction, missing path/line, severity rendering, ordering).

* fix(core): address review feedback on bugbot detail dispatch

Resolves the 6 comments from @illegalcall on OrchestratorInc#1334:

- H: Sentinel-gate the default-message override — export
  DEFAULT_BUGBOT_COMMENTS_MESSAGE from config.ts and only replace the
  reaction message when it matches the sentinel. User-customized
  messages in project YAML are left untouched.
- H: Add --paginate to step 2 of the verification guidance. Step 2 was
  previously missing --paginate, reintroducing the exact pagination
  failure mode OrchestratorInc#895 is meant to fix.
- H: Prompt-injection mitigation — strip backticks from comment body
  excerpts, wrap each excerpt in a code span so the content cannot
  break out, and add an "untrusted third-party data" preamble telling
  the agent not to treat excerpts as instructions.
- M: Preserve line number when c.line === 0 (file-level or 0-indexed
  tools). Was previously treated as falsy and dropped.
- M: Skip leading blank lines in excerpt extraction; strip leading
  markdown heading markers (### Title) and whole-line bold/italic
  wrappers before truncating.
- L: Correct the "reply resolves the thread" wording — replying alone
  does not resolve a review thread on GitHub; resolution is a separate
  "Resolve conversation" action.

Also adds a patch-level changeset for @aoagents/ao-core and expands
the test suite to 811 tests (was 803) covering sentinel override,
custom-message passthrough, line===0, backtick escaping, heading
stripping, and step-2 pagination regression.
alerios pushed a commit to AIMagicFlow/agent-orchestrator that referenced this pull request Oct 14, 2026
…tratorInc#1326)

* fix: serialize ao start and stop numbered orchestrators (OrchestratorInc#1306)

* fix: restore dead orchestrators on start (OrchestratorInc#1306)

* fix: harden startup lock handling (OrchestratorInc#1306)

* feat(core): enrich events with PR title, description, and URL

Adds PR and issue context to all event payloads sent to notifiers.
External consumers (Telegram, Discord, n8n) can now display meaningful
information without making additional API calls.

Changes:
- Add buildEventContext() helper to extract PR/issue context from session
- Enrich all createEvent() calls with context data (pr, issueId, issueTitle, branch)
- Store issueTitle in session metadata during spawn
- Add issueTitle field to SessionMetadata interface
- Update executeReaction() to accept session for context access
- Add tests for event enrichment

The context includes:
- pr: { url, title, number, branch } when PR exists
- issueId: issue identifier
- issueTitle: issue title (from tracker during spawn)
- branch: session branch name

Events before PR creation gracefully omit PR fields (pr: null).
Existing webhook consumers that ignore unknown fields are unaffected.

Closes OrchestratorInc#1226

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* fix(core): address review comments for event enrichment

- Add issueTitle to readMetadata/writeMetadata for proper persistence
- Create ReactionSessionContext type for type-safe system events
- Replace unsafe `as unknown as Session` cast with proper union type
- Add end-to-end test verifying issueTitle persistence during spawn

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* fix(cli): include lock file path in startup lock error message

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address review feedback — fd safety, kill-all resilience, issueTitle restore

- Restore try/catch/finally in tryAcquire for fd leak prevention
- Wrap stop command's sm.list()+kill in try/catch so dashboard shutdown
  always runs even on session listing failure
- Add per-iteration error handling in kill-all loop with partial failure
  reporting (spinner.warn for mixed results)
- Unify allSessionPrefixes derivation between start and stop commands
- Propagate issueTitle through archive restore path
- Add clarifying comments on agentInfo.summary fallback and intentional
  prNumber/prUrl duplication in event data
- Add ora warn mock for stop tests
- Update changeset to minor (event enrichment is a feature) and add CLI
  changeset for stop resilience
- Add tests for kill-all error mid-loop and issueTitle archive restore

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address harsh-batheja review — title/summary split, lock grace, context namespace

- Separate PR title from agent summary in EventContext: title is null
  until enrichment cache populates; summary is a distinct field so
  webhook consumers never confuse the task summary for a PR title.
- Restore UNPARSEABLE_LOCK_GRACE_MS (5s mtime grace) and
  isStaleUnparseableLock lost during merge conflict resolution —
  prevents lockfile-steal race when process A just created the file
  but hasn't written metadata yet.
- Fix orchestrator sort: extract numeric suffix instead of
  localeCompare so -10 sorts after -2, not before.
- Namespace context under data.context instead of spreading into data
  to prevent field collisions with reaction-specific keys.
- Add schemaVersion: 2 to all enriched events so consumers can
  migrate away from top-level prNumber/prUrl (kept for compat,
  marked for removal in v3).
- Update event enrichment tests for nested context structure.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: resolve merge conflicts — use formatReviewCommentsMessage, fix batch enrichment mock

- Replace formatAutomatedCommentsMessage with upstream's formatReviewCommentsMessage
  for automated review comment dispatch (fixes type mismatch with ReviewComment[])
- Make createMockSCM's enrichSessionsPRBatch dynamically resolve from individual
  method mocks so test overrides (e.g. getPRState("closed")) propagate correctly
- Add explicit enrichSessionsPRBatch to merge-conflict-tracking test to avoid
  unexpected getMergeability calls from the dynamic mock
- Remove all debug console.log statements added during troubleshooting

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: wire up maybeDispatchCIFailureDetails and record dispatch hash on transition

The function was defined but never called after merge conflict resolution
dropped the call site. Added it back to the Promise.allSettled alongside
maybeDispatchReviewBacklog and maybeDispatchMergeConflicts.

Also updated the transition-reaction early-return to record the dispatch
hash, since the transition path now enriches the CI message with detailed
check info from the batch cache — preventing duplicate sends on subsequent
polls.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: remove stale duplicate test from rebase

Removes the orphaned numbered-orchestrator restoration test left over
from feat/1226 history. Upstream's canonical model test (same name,
expects "app-orchestrator" via ensureOrchestrator) supersedes it.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* docs: remove misleading CLI CHANGELOG entry

The "Restore the most recently active dead orchestrator on ao start"
entry described upstream's ensureOrchestrator behavior (OrchestratorInc#1487), not
work contributed by this PR.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* docs(changeset): correct scope and drop stale CLI changeset

- Rewrite the @aoagents/ao-core changeset to describe only what feat/1226
  contributes: event enrichment with schemaVersion: 2, issueTitle
  persistence, executeReaction refactor, maybeDispatchCIFailureDetails,
  and bugbot-comments enrichment. Drop the false claims about adding
  spawn-target and format-automated-comments (those modules came from
  upstream PRs OrchestratorInc#1330 and OrchestratorInc#1334).
- Delete stop-kill-all-resilience.md — its claims (kill-all loop,
  fd-safety in tryAcquire, allSessionPrefixes unify) are no longer
  in the branch after the rebase took upstream's canonical stop logic.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

---------

Co-authored-by: Claude Opus 4.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(agent): review-check logic misses new bugbot comments from latest push

2 participants