Repository navigation
fix(core): dispatch detailed bugbot review comments to agents - #1334
Merged
Merged
Conversation
The bugbot-comments reaction previously sent a generic "fix the issues"
message, leaving the agent to discover comments on its own. The typical
agent flow — fetch `GET /pulls/{pr}/comments` (first page only) and
filter for a bugbot marker — misses new comments when the response is
paginated, and trusts a stale `gh pr checks` status.
Now we format the already-fetched `AutomatedComment[]` into a detailed
message that lists each comment (severity, path:line, bot name, excerpt,
URL) and embeds explicit API-level guidance so the agent can verify via
`/reviews` + `/reviews/{id}/comments` + paginated `/pulls/{pr}/comments`
(checking `in_reply_to_id` for replies) instead of the naive first-page
scan.
Closes #895
Contributor
Test Coverage Report
Per-file breakdown
Uncovered lines
|
Follow-up to #895 review: - Extract formatAutomatedCommentsMessage to a module-level helper (packages/core/src/format-automated-comments.ts) so it is directly unit-testable rather than closure-scoped inside createLifecycleManager. - Accept optional PRInfo and interpolate owner/repo/number into the verification guidance when dispatched from the lifecycle; fall back to OWNER/REPO/PR placeholders otherwise. - Drop the dead `if (c.url)` guard (url is required on AutomatedComment). - Append `…` when the first-line excerpt is truncated at 160 chars. - Simplify the config.ts default message to a short fallback and note inline that the lifecycle dispatcher injects the rich listing. - Add 11 focused unit tests for the formatter (interpolation, ellipsis, first-line extraction, missing path/line, severity rendering, ordering).
illegalcall
requested changes
Apr 21, 2026
illegalcall
left a comment
Collaborator
There was a problem hiding this comment.
Review of the bugbot detail-dispatch change. The extraction into format-automated-comments.ts with its own test suite is clean, PR-slug interpolation with placeholder fallback is well thought out, and typecheck + core tests pass on a fresh checkout. A few issues to address before merge — inline comments below.
Repo-wide note (not line-specific): this PR changes runtime behavior in @aoagents/ao-core but has no .changeset/*.md entry. Please add a patch-level changeset so the next release notes reflect the dispatched-message change.
Summary of inline findings:
- High: silent override of user-customized
reactions.bugbot-comments.message - High: step-2 guidance omits
--paginate, reintroducing the pagination failure mode from #895 - High: bot comment bodies inserted verbatim into the agent prompt (prompt-injection surface)
- Med:
line === 0is treated as falsy and drops the line number - Med: blank/markdown-heading first lines produce empty or misleading excerpts
- Low: "reply to resolve" wording doesn't match GitHub's review-thread resolution flow
illegalcall
approved these changes
Apr 21, 2026
Resolves the 6 comments from @illegalcall on #1334: - H: Sentinel-gate the default-message override — export DEFAULT_BUGBOT_COMMENTS_MESSAGE from config.ts and only replace the reaction message when it matches the sentinel. User-customized messages in project YAML are left untouched. - H: Add --paginate to step 2 of the verification guidance. Step 2 was previously missing --paginate, reintroducing the exact pagination failure mode #895 is meant to fix. - H: Prompt-injection mitigation — strip backticks from comment body excerpts, wrap each excerpt in a code span so the content cannot break out, and add an "untrusted third-party data" preamble telling the agent not to treat excerpts as instructions. - M: Preserve line number when c.line === 0 (file-level or 0-indexed tools). Was previously treated as falsy and dropped. - M: Skip leading blank lines in excerpt extraction; strip leading markdown heading markers (### Title) and whole-line bold/italic wrappers before truncating. - L: Correct the "reply resolves the thread" wording — replying alone does not resolve a review thread on GitHub; resolution is a separate "Resolve conversation" action. Also adds a patch-level changeset for @aoagents/ao-core and expands the test suite to 811 tests (was 803) covering sentinel override, custom-message passthrough, line===0, backtick escaping, heading stripping, and step-2 pagination regression.
5 tasks
harshitsinghbhandari
pushed a commit
to harshitsinghbhandari/agent-orchestrator
that referenced
this pull request
May 1, 2026
- Rewrite the @aoagents/ao-core changeset to describe only what feat/1226 contributes: event enrichment with schemaVersion: 2, issueTitle persistence, executeReaction refactor, maybeDispatchCIFailureDetails, and bugbot-comments enrichment. Drop the false claims about adding spawn-target and format-automated-comments (those modules came from upstream PRs OrchestratorInc#1330 and OrchestratorInc#1334). - Delete stop-kill-all-resilience.md — its claims (kill-all loop, fd-safety in tryAcquire, allSessionPrefixes unify) are no longer in the branch after the rebase took upstream's canonical stop logic. Co-Authored-By: Claude Opus 4.7 <[email protected]>
harshitsinghbhandari
pushed a commit
that referenced
this pull request
May 1, 2026
* fix: serialize ao start and stop numbered orchestrators (#1306) * fix: restore dead orchestrators on start (#1306) * fix: harden startup lock handling (#1306) * feat(core): enrich events with PR title, description, and URL Adds PR and issue context to all event payloads sent to notifiers. External consumers (Telegram, Discord, n8n) can now display meaningful information without making additional API calls. Changes: - Add buildEventContext() helper to extract PR/issue context from session - Enrich all createEvent() calls with context data (pr, issueId, issueTitle, branch) - Store issueTitle in session metadata during spawn - Add issueTitle field to SessionMetadata interface - Update executeReaction() to accept session for context access - Add tests for event enrichment The context includes: - pr: { url, title, number, branch } when PR exists - issueId: issue identifier - issueTitle: issue title (from tracker during spawn) - branch: session branch name Events before PR creation gracefully omit PR fields (pr: null). Existing webhook consumers that ignore unknown fields are unaffected. Closes #1226 Co-Authored-By: Claude Opus 4.5 <[email protected]> * fix(core): address review comments for event enrichment - Add issueTitle to readMetadata/writeMetadata for proper persistence - Create ReactionSessionContext type for type-safe system events - Replace unsafe `as unknown as Session` cast with proper union type - Add end-to-end test verifying issueTitle persistence during spawn Co-Authored-By: Claude Opus 4.5 <[email protected]> * fix(cli): include lock file path in startup lock error message Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address review feedback — fd safety, kill-all resilience, issueTitle restore - Restore try/catch/finally in tryAcquire for fd leak prevention - Wrap stop command's sm.list()+kill in try/catch so dashboard shutdown always runs even on session listing failure - Add per-iteration error handling in kill-all loop with partial failure reporting (spinner.warn for mixed results) - Unify allSessionPrefixes derivation between start and stop commands - Propagate issueTitle through archive restore path - Add clarifying comments on agentInfo.summary fallback and intentional prNumber/prUrl duplication in event data - Add ora warn mock for stop tests - Update changeset to minor (event enrichment is a feature) and add CLI changeset for stop resilience - Add tests for kill-all error mid-loop and issueTitle archive restore Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address harsh-batheja review — title/summary split, lock grace, context namespace - Separate PR title from agent summary in EventContext: title is null until enrichment cache populates; summary is a distinct field so webhook consumers never confuse the task summary for a PR title. - Restore UNPARSEABLE_LOCK_GRACE_MS (5s mtime grace) and isStaleUnparseableLock lost during merge conflict resolution — prevents lockfile-steal race when process A just created the file but hasn't written metadata yet. - Fix orchestrator sort: extract numeric suffix instead of localeCompare so -10 sorts after -2, not before. - Namespace context under data.context instead of spreading into data to prevent field collisions with reaction-specific keys. - Add schemaVersion: 2 to all enriched events so consumers can migrate away from top-level prNumber/prUrl (kept for compat, marked for removal in v3). - Update event enrichment tests for nested context structure. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: resolve merge conflicts — use formatReviewCommentsMessage, fix batch enrichment mock - Replace formatAutomatedCommentsMessage with upstream's formatReviewCommentsMessage for automated review comment dispatch (fixes type mismatch with ReviewComment[]) - Make createMockSCM's enrichSessionsPRBatch dynamically resolve from individual method mocks so test overrides (e.g. getPRState("closed")) propagate correctly - Add explicit enrichSessionsPRBatch to merge-conflict-tracking test to avoid unexpected getMergeability calls from the dynamic mock - Remove all debug console.log statements added during troubleshooting Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: wire up maybeDispatchCIFailureDetails and record dispatch hash on transition The function was defined but never called after merge conflict resolution dropped the call site. Added it back to the Promise.allSettled alongside maybeDispatchReviewBacklog and maybeDispatchMergeConflicts. Also updated the transition-reaction early-return to record the dispatch hash, since the transition path now enriches the CI message with detailed check info from the batch cache — preventing duplicate sends on subsequent polls. Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: remove stale duplicate test from rebase Removes the orphaned numbered-orchestrator restoration test left over from feat/1226 history. Upstream's canonical model test (same name, expects "app-orchestrator" via ensureOrchestrator) supersedes it. Co-Authored-By: Claude Opus 4.7 <[email protected]> * docs: remove misleading CLI CHANGELOG entry The "Restore the most recently active dead orchestrator on ao start" entry described upstream's ensureOrchestrator behavior (#1487), not work contributed by this PR. Co-Authored-By: Claude Opus 4.7 <[email protected]> * docs(changeset): correct scope and drop stale CLI changeset - Rewrite the @aoagents/ao-core changeset to describe only what feat/1226 contributes: event enrichment with schemaVersion: 2, issueTitle persistence, executeReaction refactor, maybeDispatchCIFailureDetails, and bugbot-comments enrichment. Drop the false claims about adding spawn-target and format-automated-comments (those modules came from upstream PRs #1330 and #1334). - Delete stop-kill-all-resilience.md — its claims (kill-all loop, fd-safety in tryAcquire, allSessionPrefixes unify) are no longer in the branch after the rebase took upstream's canonical stop logic. Co-Authored-By: Claude Opus 4.7 <[email protected]> --------- Co-authored-by: Claude Opus 4.5 <[email protected]>
alerios
pushed a commit
to AIMagicFlow/agent-orchestrator
that referenced
this pull request
Oct 14, 2026
…tratorInc#1334) * fix(core): dispatch detailed bugbot review comments to agents The bugbot-comments reaction previously sent a generic "fix the issues" message, leaving the agent to discover comments on its own. The typical agent flow — fetch `GET /pulls/{pr}/comments` (first page only) and filter for a bugbot marker — misses new comments when the response is paginated, and trusts a stale `gh pr checks` status. Now we format the already-fetched `AutomatedComment[]` into a detailed message that lists each comment (severity, path:line, bot name, excerpt, URL) and embeds explicit API-level guidance so the agent can verify via `/reviews` + `/reviews/{id}/comments` + paginated `/pulls/{pr}/comments` (checking `in_reply_to_id` for replies) instead of the naive first-page scan. Closes OrchestratorInc#895 * refactor(core): extract bugbot comment formatter, address review Follow-up to OrchestratorInc#895 review: - Extract formatAutomatedCommentsMessage to a module-level helper (packages/core/src/format-automated-comments.ts) so it is directly unit-testable rather than closure-scoped inside createLifecycleManager. - Accept optional PRInfo and interpolate owner/repo/number into the verification guidance when dispatched from the lifecycle; fall back to OWNER/REPO/PR placeholders otherwise. - Drop the dead `if (c.url)` guard (url is required on AutomatedComment). - Append `…` when the first-line excerpt is truncated at 160 chars. - Simplify the config.ts default message to a short fallback and note inline that the lifecycle dispatcher injects the rich listing. - Add 11 focused unit tests for the formatter (interpolation, ellipsis, first-line extraction, missing path/line, severity rendering, ordering). * fix(core): address review feedback on bugbot detail dispatch Resolves the 6 comments from @illegalcall on OrchestratorInc#1334: - H: Sentinel-gate the default-message override — export DEFAULT_BUGBOT_COMMENTS_MESSAGE from config.ts and only replace the reaction message when it matches the sentinel. User-customized messages in project YAML are left untouched. - H: Add --paginate to step 2 of the verification guidance. Step 2 was previously missing --paginate, reintroducing the exact pagination failure mode OrchestratorInc#895 is meant to fix. - H: Prompt-injection mitigation — strip backticks from comment body excerpts, wrap each excerpt in a code span so the content cannot break out, and add an "untrusted third-party data" preamble telling the agent not to treat excerpts as instructions. - M: Preserve line number when c.line === 0 (file-level or 0-indexed tools). Was previously treated as falsy and dropped. - M: Skip leading blank lines in excerpt extraction; strip leading markdown heading markers (### Title) and whole-line bold/italic wrappers before truncating. - L: Correct the "reply resolves the thread" wording — replying alone does not resolve a review thread on GitHub; resolution is a separate "Resolve conversation" action. Also adds a patch-level changeset for @aoagents/ao-core and expands the test suite to 811 tests (was 803) covering sentinel override, custom-message passthrough, line===0, backtick escaping, heading stripping, and step-2 pagination regression.
alerios
pushed a commit
to AIMagicFlow/agent-orchestrator
that referenced
this pull request
Oct 14, 2026
…tratorInc#1326) * fix: serialize ao start and stop numbered orchestrators (OrchestratorInc#1306) * fix: restore dead orchestrators on start (OrchestratorInc#1306) * fix: harden startup lock handling (OrchestratorInc#1306) * feat(core): enrich events with PR title, description, and URL Adds PR and issue context to all event payloads sent to notifiers. External consumers (Telegram, Discord, n8n) can now display meaningful information without making additional API calls. Changes: - Add buildEventContext() helper to extract PR/issue context from session - Enrich all createEvent() calls with context data (pr, issueId, issueTitle, branch) - Store issueTitle in session metadata during spawn - Add issueTitle field to SessionMetadata interface - Update executeReaction() to accept session for context access - Add tests for event enrichment The context includes: - pr: { url, title, number, branch } when PR exists - issueId: issue identifier - issueTitle: issue title (from tracker during spawn) - branch: session branch name Events before PR creation gracefully omit PR fields (pr: null). Existing webhook consumers that ignore unknown fields are unaffected. Closes OrchestratorInc#1226 Co-Authored-By: Claude Opus 4.5 <[email protected]> * fix(core): address review comments for event enrichment - Add issueTitle to readMetadata/writeMetadata for proper persistence - Create ReactionSessionContext type for type-safe system events - Replace unsafe `as unknown as Session` cast with proper union type - Add end-to-end test verifying issueTitle persistence during spawn Co-Authored-By: Claude Opus 4.5 <[email protected]> * fix(cli): include lock file path in startup lock error message Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address review feedback — fd safety, kill-all resilience, issueTitle restore - Restore try/catch/finally in tryAcquire for fd leak prevention - Wrap stop command's sm.list()+kill in try/catch so dashboard shutdown always runs even on session listing failure - Add per-iteration error handling in kill-all loop with partial failure reporting (spinner.warn for mixed results) - Unify allSessionPrefixes derivation between start and stop commands - Propagate issueTitle through archive restore path - Add clarifying comments on agentInfo.summary fallback and intentional prNumber/prUrl duplication in event data - Add ora warn mock for stop tests - Update changeset to minor (event enrichment is a feature) and add CLI changeset for stop resilience - Add tests for kill-all error mid-loop and issueTitle archive restore Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address harsh-batheja review — title/summary split, lock grace, context namespace - Separate PR title from agent summary in EventContext: title is null until enrichment cache populates; summary is a distinct field so webhook consumers never confuse the task summary for a PR title. - Restore UNPARSEABLE_LOCK_GRACE_MS (5s mtime grace) and isStaleUnparseableLock lost during merge conflict resolution — prevents lockfile-steal race when process A just created the file but hasn't written metadata yet. - Fix orchestrator sort: extract numeric suffix instead of localeCompare so -10 sorts after -2, not before. - Namespace context under data.context instead of spreading into data to prevent field collisions with reaction-specific keys. - Add schemaVersion: 2 to all enriched events so consumers can migrate away from top-level prNumber/prUrl (kept for compat, marked for removal in v3). - Update event enrichment tests for nested context structure. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: resolve merge conflicts — use formatReviewCommentsMessage, fix batch enrichment mock - Replace formatAutomatedCommentsMessage with upstream's formatReviewCommentsMessage for automated review comment dispatch (fixes type mismatch with ReviewComment[]) - Make createMockSCM's enrichSessionsPRBatch dynamically resolve from individual method mocks so test overrides (e.g. getPRState("closed")) propagate correctly - Add explicit enrichSessionsPRBatch to merge-conflict-tracking test to avoid unexpected getMergeability calls from the dynamic mock - Remove all debug console.log statements added during troubleshooting Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: wire up maybeDispatchCIFailureDetails and record dispatch hash on transition The function was defined but never called after merge conflict resolution dropped the call site. Added it back to the Promise.allSettled alongside maybeDispatchReviewBacklog and maybeDispatchMergeConflicts. Also updated the transition-reaction early-return to record the dispatch hash, since the transition path now enriches the CI message with detailed check info from the batch cache — preventing duplicate sends on subsequent polls. Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: remove stale duplicate test from rebase Removes the orphaned numbered-orchestrator restoration test left over from feat/1226 history. Upstream's canonical model test (same name, expects "app-orchestrator" via ensureOrchestrator) supersedes it. Co-Authored-By: Claude Opus 4.7 <[email protected]> * docs: remove misleading CLI CHANGELOG entry The "Restore the most recently active dead orchestrator on ao start" entry described upstream's ensureOrchestrator behavior (OrchestratorInc#1487), not work contributed by this PR. Co-Authored-By: Claude Opus 4.7 <[email protected]> * docs(changeset): correct scope and drop stale CLI changeset - Rewrite the @aoagents/ao-core changeset to describe only what feat/1226 contributes: event enrichment with schemaVersion: 2, issueTitle persistence, executeReaction refactor, maybeDispatchCIFailureDetails, and bugbot-comments enrichment. Drop the false claims about adding spawn-target and format-automated-comments (those modules came from upstream PRs OrchestratorInc#1330 and OrchestratorInc#1334). - Delete stop-kill-all-resilience.md — its claims (kill-all loop, fd-safety in tryAcquire, allSessionPrefixes unify) are no longer in the branch after the rebase took upstream's canonical stop logic. Co-Authored-By: Claude Opus 4.7 <[email protected]> --------- Co-authored-by: Claude Opus 4.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #895 — the bugbot-comments reaction was sending a generic message, leaving the agent to rediscover comments on its own. The naive
GET /pulls/{pr}/commentsfirst-page scan misses new comments on later pages and trusts a stalegh pr checksstatus, causing the agent to falsely report "all reviews addressed."Changes
packages/core/src/lifecycle-manager.tsformatAutomatedCommentsMessage(AutomatedComment[]), mirroringformatCIFailureMessage. Lists each comment (severity, bot name,path:line, first-line excerpt, URL) and embeds correct-API guidance.maybeDispatchReviewBacklog, override the reaction config'smessagewith the formatted one forsend-to-agentdispatches ofbugbot-comments. Fingerprint/dedup path unchanged.packages/core/src/config.ts— update the defaultbugbot-commentsreaction text to embed the/reviews→/reviews/{id}/comments→ paginated/pulls/{pr}/comments(checkingin_reply_to_id) sequence as a fallback for custom configs that don't rely on the dispatcher.packages/core/src/__tests__/lifecycle-manager.test.ts— update the "dispatches automated review comments only once" test to assert the new message contains comment details AND the correct-API guidance.Why this design
The core already paginates
getAutomatedCommentscorrectly and knows every bot comment. The old code threw that data away when dispatching and let the agent re-query with a broken pattern. Passing the already-fetched data to the agent removes the failure mode entirely; embedding the correct verification steps guards against agents that still want to re-check.Test plan
pnpm --filter @aoagents/ao-core typecheck— passespnpm --filter @aoagents/ao-core test— 792 tests passpnpm typecheck(all packages) — passespnpm lint— no errors (32 pre-existing warnings unrelated to this change)