Skip to content

Improve address-review command: fix bot filtering, add thread resolution#2555

Merged
justin808 merged 1 commit intomasterfrom
jg/review-cycle-analysis
Mar 8, 2026
Merged

Improve address-review command: fix bot filtering, add thread resolution#2555
justin808 merged 1 commit intomasterfrom
jg/review-cycle-analysis

Conversation

@justin808
Copy link
Copy Markdown
Member

@justin808 justin808 commented Mar 8, 2026

Summary

Fixed five critical issues in the address-review command that were preventing proper feedback resolution:

  1. Bot comment filtering: Now deduplicates bot feedback instead of discarding it, since many actionable comments come from bots (claude[bot], coderabbitai, greptile)
  2. Thread resolution: Added GraphQL mutation to resolve review threads after addressing concerns
  3. ID preservation: Step 4 now preserves review comment and thread IDs through triage
  4. Internal consistency: Removed conflicting guidance about bot comments between Step 3 and Bot Reviewer Awareness section
  5. Closure paths: Added explicit reply-and-resolve behavior for DISCUSS and SKIPPED items when user explicitly requests closure

Pull Request checklist

  • Changes are documentation/tooling only (no code changes requiring tests)

Note

Low Risk
Documentation-only changes that adjust internal review workflow guidance; no runtime code paths or data handling are modified.

Overview
Updates the .claude/commands/address-review.md workflow to triage review comments into MUST-FIX, DISCUSS, and SKIPPED, deduplicate overlaps (including bot feedback), and create todos only for MUST-FIX items.

Adds guidance to preserve comment/thread IDs, reply only after completing user-selected items, and resolve addressed review threads via a GitHub GraphQL resolveReviewThread mutation; example output and instructions are updated accordingly.

Written by Cursor Bugbot for commit bdb4381. Configure here.

Summary by CodeRabbit

  • Documentation
    • Enhanced review comment management with improved triage and classification workflow. Added intelligent deduplication, categorization of review items, and automated thread resolution capabilities. The updated process provides clearer organization of feedback and refined output formatting with item counts to guide user prioritization and response.

…ion, preserve comment IDs

- Remove blanket bot comment skip; deduplicate bot feedback instead
- Add GraphQL thread resolution with proper conditions
- Preserve review comment/thread IDs through triage for reply/resolve
- Resolve internal inconsistency between Step 3 and Bot Reviewer Awareness section
- Add explicit closure path for DISCUSS and SKIPPED items via user request
- Fixes reply-and-resolve feedback loop for addressed review threads
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 8, 2026

Walkthrough

The changes restructure the AI-assisted review command workflow to introduce a triage phase that classifies comments into MUST-FIX, DISCUSS, and SKIPPED categories, followed by selective todo creation from MUST-FIX items only, user presentation of triage results, and post-reply thread resolution via GraphQL. Comment filtering is enhanced with bot comment deduplication and granular classification rules.

Changes

Cohort / File(s) Summary
AI Review Command Documentation
.claude/commands/address-review.md
Restructured workflow to include triage classification, bot comment deduplication, selective MUST-FIX todo creation, user presentation of categorized items, and post-reply thread resolution logic with GraphQL mutations. Steps 4–7 renamed and refocused. Output examples updated to show triage-driven categorization with counts.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant AIAssistant as AI Assistant
    participant GraphQLAPI as GraphQL API

    User->>AIAssistant: Submit review for triage
    AIAssistant->>AIAssistant: Classify comments into<br/>MUST-FIX, DISCUSS, SKIPPED
    AIAssistant->>AIAssistant: Deduplicate bot comments
    AIAssistant->>AIAssistant: Create todos from MUST-FIX only
    AIAssistant->>User: Present triage results<br/>(categorized items & counts)
    User->>AIAssistant: Approve triage & provide direction
    AIAssistant->>AIAssistant: Address selected items,<br/>generate replies
    AIAssistant->>GraphQLAPI: Resolve review threads<br/>(when addressed & thread ID available)
    GraphQLAPI-->>AIAssistant: Thread resolution confirmed
    AIAssistant->>User: Summary of addressed items
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 A triage tale of threads so grand,
We sort and sift with careful hand,
MUST-FIX first, the rest deferred,
Then GraphQL whispers each resolved word!
New workflows bloom, efficient and bright! 🌿✨

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main changes: fixing bot filtering and adding thread resolution capabilities to the address-review command.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch jg/review-cycle-analysis

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Mar 8, 2026

Review: Improve address-review command

Overall this is a solid improvement. The triage system (MUST-FIX / DISCUSS / SKIPPED) is well-designed and will meaningfully reduce noise, and the thread-resolution flow closes an important feedback loop that was previously missing. A few issues worth addressing:

Critical gap: thread IDs are not available from the REST API

The PR instructs Step 4 to "preserve the original review comment ID and thread ID when available," and Step 7 then uses that thread ID in the resolveReviewThread GraphQL mutation. However, the REST endpoint used in Step 3 (GET /pulls/{PR_NUMBER}/comments) does not return thread IDs — it only returns comment database IDs. Thread IDs are GraphQL node IDs and must be retrieved separately via a GraphQL query.

Without a mechanism to fetch thread IDs, the "preserve thread ID" instruction in Step 4 is a no-op, and the resolveReviewThread mutation will never have a valid threadId to work with. A GraphQL query like the one below should be added (probably alongside the REST fetch in Step 3):

gh api graphql -f query='query($owner:String!,$name:String!,$number:Int!){
  repository(owner:$owner,name:$name){
    pullRequest(number:$number){
      reviewThreads(first:100){
        nodes{
          id isResolved
          comments(first:1){ nodes{ databaseId } }
        }
      }
    }
  }
}' -f owner="$OWNER" -f name="$NAME" -F number="$PR_NUMBER"

This maps each comment's databaseId (which matches the REST API id) to its parent thread's GraphQL node ID.

Pagination limitation silently removed

The previous "Known Limitations" section included:

Large PRs: PRs with many comments may require pagination (not currently handled)

This bullet was deleted in this PR without adding pagination support. Since both the REST API and the GraphQL reviewThreads query above default to limited page sizes, the limitation still exists. The note should be retained (or pagination actually implemented).

Minor: triage criteria for MUST-FIX vs DISCUSS overlaps

"clear inconsistencies with adjacent code" appears in the MUST-FIX definition, but identical wording also appears in the Step 3 filtering rules as "actionable by default." This duplication is harmless but slightly confusing; a brief cross-reference ("see Step 4 triage rules") in Step 3 would keep the definitions as the single source of truth.

Positive highlights

  • Deduplicating bot feedback rather than discarding it wholesale is the right call for this repo.
  • The conditions guarding thread resolution (pending fix, user approval, already-resolved check) are appropriately conservative.
  • The example output is clear and immediately actionable.

- Deduplicate overlapping comments before classifying them. Keep one representative item for the underlying issue.
- Verify factual claims locally before classifying a comment as `MUST-FIX`.
- If a claim appears wrong, classify it as `SKIPPED` and note briefly why.
- Preserve the original review comment ID and thread ID when available so the command can reply to the correct place and resolve the correct thread later.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The REST API (GET /pulls/{PR}/comments) returns each comment's numeric id but not its GraphQL thread ID. The resolveReviewThread mutation in Step 7 requires a GraphQL node ID (e.g. PRT_kwDO...), not the REST integer.

Without an additional GraphQL query to map comment IDs → thread IDs, this instruction is silently a no-op. Suggest adding a companion GraphQL fetch here (or in Step 3) like:

gh api graphql -f query='query($owner:String!,$name:String!,$number:Int!){
  repository(owner:$owner,name:$name){
    pullRequest(number:$number){
      reviewThreads(first:100){
        nodes{
          id isResolved
          comments(first:1){ nodes{ databaseId } }
        }
      }
    }
  }
}' -f owner="$OWNER" -f name="$NAME" -F number="$PR_NUMBER"

Then cross-reference databaseId (= REST id) to the thread id node and store both when building the triage list.

# Known Limitations

- Rate limiting: GitHub API has rate limits; if you hit them, wait a few minutes
- Private repos: Requires appropriate `gh` authentication scope
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pagination limitation bullet that existed in the previous version was removed here without adding pagination support. Both the REST GET /pulls/{PR}/comments endpoint and the GraphQL reviewThreads(first:100) query are subject to page-size limits on larger PRs. Suggest restoring this note:

Suggested change
- Private repos: Requires appropriate `gh` authentication scope
- Large PRs: PRs with many comments may require pagination (not currently handled). The GraphQL `reviewThreads` query above uses `first:100`; increase or add cursor-based pagination for PRs with more threads.

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented Mar 8, 2026

Greptile Summary

This PR improves the address-review slash command by replacing the blanket "skip all bot comments" rule with a smarter triage system (MUST-FIX / DISCUSS / SKIPPED), and adds a GraphQL resolveReviewThread mutation to mark threads as resolved after they are addressed. The direction is clearly right, but two gaps remain that will prevent the new thread-resolution feature from working in practice.

Key changes:

  • Bot comments are now deduplicated and triaged rather than unconditionally discarded
  • A structured three-category triage workflow (Steps 4–6) replaces the old single "Create Todo" step
  • The GraphQL resolveReviewThread mutation is documented in Step 7 for post-reply thread closure

Issues found:

  • Thread ID unavailable at resolution time: The Step 3 REST API fetch (jq filter) never extracts node_id from comments, and there is no instruction to perform a GraphQL query to map comments to their parent PullRequestReviewThread.id. Without the thread node ID the resolveReviewThread mutation in Step 7 cannot be populated, silently failing the core new feature.
  • Pagination limitation silently removed: The old Known Limitations note about REST API pagination being unhandled was deleted, but the gh api commands in Step 3 still have no --paginate flag or ?per_page=100 parameter. On large PRs the agent will triage only the first ~30 comments with no warning to the user.

Confidence Score: 2/5

  • The PR introduces a new thread-resolution feature that cannot work as written because the required thread node IDs are never fetched.
  • The triage restructuring is well-conceived and the bot-comment fix is sound, but the headline new capability (resolving review threads via GraphQL) is broken by a data-collection gap: no thread ID is captured in the REST API step, and no GraphQL fallback is provided. Additionally, removing the pagination limitation note without fixing pagination can cause silent data loss on large PRs. Both are correctness issues in a command that will be run autonomously.
  • .claude/commands/address-review.md — specifically Step 3 (jq filter missing node_id / no GraphQL thread query) and the Known Limitations section (pagination note removed without fix)

Important Files Changed

Filename Overview
.claude/commands/address-review.md Improves bot-comment handling, adds triage categories, and introduces thread resolution via GraphQL — but the REST API fetch never captures the thread node IDs required by the new resolveReviewThread mutation, and the pagination limitation was silently removed without being fixed.

Sequence Diagram

sequenceDiagram
    participant U as User
    participant A as Agent
    participant GH as GitHub REST API
    participant GQL as GitHub GraphQL API

    U->>A: /address-review <PR_NUMBER>
    A->>GH: GET /repos/{repo}/pulls/{PR}/comments
    GH-->>A: comments (id, path, body, line, user, in_reply_to_id)
    Note over A: ⚠️ node_id / thread_id NOT fetched

    A->>A: Filter replies (in_reply_to_id set)
    A->>A: Deduplicate bot comments
    A->>A: Triage → MUST-FIX / DISCUSS / SKIPPED
    A->>A: TodoWrite (MUST-FIX items only)
    A-->>U: Present triage summary

    U->>A: "address item 1"
    A->>A: Apply code / doc fix
    A->>GH: POST /pulls/{PR}/comments/{id}/replies
    GH-->>A: reply created

    A->>GQL: resolveReviewThread(threadId: ???)
    Note over A,GQL: ⚠️ threadId unavailable — mutation will fail
    GQL-->>A: thread resolved (if threadId known)

    A-->>U: Item addressed & thread resolved
Loading

Comments Outside Diff (2)

  1. .claude/commands/address-review.md, line 50-52 (link)

    Thread ID not captured in REST API fetch

    The jq filters in Step 3 extract {id, path, body, line, start_line, user, in_reply_to_id} but omit node_id. Step 4 then instructs the agent to "preserve the original review comment ID and thread ID", and Step 7 uses a <THREAD_ID> in the GraphQL resolveReviewThread mutation.

    The problem is that the REST API comment id (numeric) is not the same as the PullRequestReviewThread.id (GraphQL node ID) required by the mutation. The thread node ID can only be obtained via GraphQL — either by querying pullRequest.reviewThreads or by fetching a comment's node_id and traversing to its parent thread.

    As written, the agent will have no way to populate <THREAD_ID> correctly without an additional undescribed step. Either:

    1. Add node_id to the jq filter and document a follow-up GraphQL query to map comment node IDs to their parent thread IDs, or
    2. Replace the REST API call with a single GraphQL query that fetches thread IDs alongside comment data:
    gh api graphql -f query='
      query($owner:String!, $name:String!, $number:Int!) {
        repository(owner:$owner, name:$name) {
          pullRequest(number:$number) {
            reviewThreads(first:100) {
              nodes {
                id isResolved
                comments(first:1) {
                  nodes { databaseId body path line author { login } }
                }
              }
            }
          }
        }
      }
    ' -f owner=<OWNER> -f name=<REPO> -F number=<PR_NUMBER>

    Without this, the thread-resolution feature added in Step 7 cannot function reliably.

  2. .claude/commands/address-review.md, line 196-200 (link)

    Pagination limitation silently dropped

    The original Known Limitations section contained the note:

    Large PRs: PRs with many comments may require pagination (not currently handled)

    This note was removed in the PR, but the gh api REST call in Step 3 still has no pagination logic. GitHub's REST API returns at most 30 items per page by default (max 100 with ?per_page=100). On a PR with many review comments the agent will silently triage only the first page, causing missed comments with no warning.

    The limitation note should be restored (or pagination should be implemented, e.g. with --paginate):

Last reviewed commit: bdb4381

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdb43819c2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +140 to +143
Use GitHub GraphQL to resolve the thread:

```bash
gh api graphql -f query='mutation($threadId:ID!) { resolveReviewThread(input:{threadId:$threadId}) { thread { id isResolved } } }' -f threadId="<THREAD_ID>"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Retrieve thread IDs before resolving review threads

Step 7 requires passing <THREAD_ID> to resolveReviewThread, but the fetch commands in Step 3 still only project comment fields like id, path, body, and line and never collect a thread identifier (or a value that is then mapped to one). In the common /address-review <PR_NUMBER> flow this leaves no thread ID to supply, so addressed comments cannot actually be resolved even though the command now instructs doing so. Add an explicit thread-ID retrieval/mapping step before this mutation.

Useful? React with 👍 / 👎.

@justin808 justin808 merged commit f2a0ab9 into master Mar 8, 2026
14 of 15 checks passed
@justin808 justin808 deleted the jg/review-cycle-analysis branch March 8, 2026 02:52
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is ON, but it could not run because the branch was deleted or merged before autofix could start.

Use GitHub GraphQL to resolve the thread:

```bash
gh api graphql -f query='mutation($threadId:ID!) { resolveReviewThread(input:{threadId:$threadId}) { thread { id isResolved } } }' -f threadId="<THREAD_ID>"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thread IDs never fetched but required for resolution

High Severity

The new thread resolution feature in Step 7 requires a <THREAD_ID> for the resolveReviewThread GraphQL mutation, but Step 3's REST API calls and jq filters never fetch thread IDs. The GitHub REST API for PR review comments doesn't expose review thread IDs at all — they're only available via GraphQL (e.g., querying pullRequest.reviewThreads). Step 4's instruction to "preserve the thread ID when available" is a no-op since the data is never obtained. A GraphQL query to fetch thread IDs (and map them to comment IDs) needs to be added to Step 3.

Additional Locations (2)

Fix in Cursor Fix in Web

justin808 added a commit that referenced this pull request Mar 9, 2026
## Summary

- Add changelog entries for 6 user-visible PRs merged since v16.4.0.rc.6
that were missing from `[Unreleased]`
- Update existing #2561 entry to include #2568 contributor credit

### New entries added

| Section | PR | Description |
|---|---|---|
| Added | #2539 | Environment-variable-driven ports in Procfile
templates |
| Fixed | #2417 | Rspack generator config path fix |
| Fixed | #2419 | Precompile hook load-based execution fix |
| Fixed | #2577 | `create-react-on-rails-app` validation improvements |
| Pro Fixed | #2416 | StreamResponse status fallback fix |
| Pro Fixed | #2566 | Empty-string license plan mismatch fix |

### Skipped PRs (not user-visible)

Docs (#2406, #2414, #2479, #2494, #2518, #2537, #2544), CI/internal
(#2533, #2547, #2555, #2557, #2558, #2564), dependabot (#2387, #2541),
dev dependencies (#2559, #2569, #2573).

## Test plan

- [ ] Verify changelog formatting matches existing entries
- [ ] Verify all user-visible PRs since v16.4.0.rc.6 are covered

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Documentation-only changelog updates with no runtime or build behavior
changes.
> 
> **Overview**
> Updates `CHANGELOG.md`’s **[Unreleased]** section to include
previously missing user-facing entries: Procfile templates now support
env-driven ports, several generator/`bin/dev` precompile-hook and
rspack-path fixes are documented, and `create-react-on-rails-app`
validation improvements are noted.
> 
> Also adds two Pro fix entries (StreamResponse status fallback and
license plan empty-string validation) and updates the existing `bin/dev`
precompile-hook entry to credit an additional PR/contributor.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
e75d2b5. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: Claude Opus 4.6 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant