Add async iterator to fix/enable streaming responses by estill01 · Pull Request #240 · BerriAI/litellm

estill01 · 2023-08-29T22:46:32Z

Add __aiter__ and __anext__ to CustomStreamWrapper class so that it returns an async iterator

vercel · 2023-08-29T22:46:36Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 29, 2023 10:47pm

estill01 · 2023-08-29T22:47:32Z

I'm building a streaming OpenAI chat response interface using completion and it seems this is necessary

estill01 · 2023-08-29T22:51:06Z

Here's my server log output:

index.py", line 39, in stream_response
     async for chunk in response:
TypeError: 'async for' requires an object with __aiter__ method, got CustomStreamWrapper

ishaan-jaff · 2023-08-29T23:01:25Z

looks awesome ! thanks for the contribution @estill01 ! Is the key difference you're trying to do

async for chunk in response and we only support for chunk in response

ishaan-jaff · 2023-08-29T23:02:22Z

if so we can add this to docs too, under Streaming - happy to add that in a separate PR / track it as an issue

ghost · 2023-08-29T23:11:28Z

This is amazing!

Can you help me understand the async case better @estill01?

I used LiteLLM for streaming on our playground - https://litellm.ai/playground, and just used streaming as is. How do you use async here?

For context, here's my API call code from my Next.js frontend:

api call

        const response = await fetch(apiUrl, {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
          },
          body: JSON.stringify(body)
        })

stream output parsing

if (contentType && contentType.includes("text/event-stream")) {
            const response_body = response.body;
            const reader = response_body.getReader();
            const decoder = new TextDecoder();
            let done = false;

            while (!done) {
              const { value, done: readingDone } = await reader.read();
              done = readingDone;

              let chunkValue = decoder.decode(value);
              // Check if chunkValue starts with "data: " and remove it
              if (chunkValue.startsWith('data: ')) {
                chunkValue = chunkValue.slice(6);
              }
              // Split by new lines
              const lines = chunkValue.split('\n');

              // Check & Parse each line 
              lines.forEach(line => {
                if (line.startsWith('data: ')) {
                  line = line.slice(6); // remove "data: " 
                }
    
                if(line.trim() === "") return; // skip empty lines
    
                try {
                  const data = JSON.parse(line);
                  const content = data.choices[0].delta.content;
                  console.log("model:", body.model);
                  console.log("content:", content);
                  if (content) {
                    switch (body.model) {
                      case model1:
                        setValue1((prev) => prev + content);
                        break;
                      case model2:
                        setValue2((prev) => prev + content);
                        break;
                      case model3:
                        setValue3((prev) => prev + content);
                        break;
                      default:
                        console.error('Unexpected model:', body.model);
                    }
                  }
                } 
                catch (error) {
                  console.error("Error parsing JSON:", error);
                }
              });
            }

estill01 · 2023-08-29T23:46:50Z

looks awesome ! thanks for the contribution @estill01 ! Is the key difference you're trying to do

async for chunk in response and we only support for chunk in response

Yeah - exactly

estill01 · 2023-08-29T23:53:51Z

Can you help me understand the async case better @estill01?

Sure - Without an async iterator it blocks execution of other tasks that could be running concurrently while it's waiting for the next chunk of the response e.g. llm output sanitizer logic.

It also provides a DX boost so that it's harder to write code that doesn't work with completion

…comment Adds "What's this and why am I getting it?" links to docs.litellm.ai/blog/ agent-shin-triage from the four comments contributors actually read when something went wrong: PR close, PR grace warning, issue close, issue grace warning. PR comments also link the rubric section directly from the QA-proof bullet so contributors can self-serve "what counts as proof" without pinging a maintainer. Pins the new guarantees in tests: blog link must appear in all four comments, and the PR close comment must continue to flag mocked-dependency unit tests as insufficient proof. The linked blog post is in BerriAI/litellm-docs PR #240; the URL will 404 until that lands. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

@agent-shin

…and review-gate label lifecycle (#30433) * feat(triage): auto-close stale PRs with Greptile score <4/5 Adds .github/scripts/close_low_quality_prs.py and a daily workflow that closes PRs which: - are open for at least 7 days, and - carry a most-recent greptile-apps review with Confidence Score <4/5, - and are not drafts or opt-out-labeled ('do not close', 'wip', etc.). Each closure posts an explanatory comment telling the contributor how to bring the PR back (rebase, re-request greptile, reopen at 4+/5). The 4/5 bar is already documented in the PR template (.github/pull_request_template.md), so this just enforces it. Tested with a dry run against the live BerriAI/litellm backlog of 1000 open PRs: 100 candidates identified, 598 PRs pass the bar (4+/5), 186 are too young, 97 are drafts, 19 lack any Greptile review and are left alone. Workflow defaults to closing 25 PRs/run as a safety net and supports workflow_dispatch with overrides (close=false for a dry run, custom min_age_days/min_score/limit). 18 unit tests cover score extraction (HTML/markdown/plain text, login variants, multi-review picks latest) and per-PR evaluation (drafts, opt-out labels, age, missing/passing/failing scores). Co-authored-by: Mateo Wang <[email protected]> * docs(templates): require expected/actual + QA proof for external contributions PR template: - Make the rubric explicit at the top: link an issue, OR provide a clear problem description + expected vs. actual + visual QA proof. - Add dedicated sections for each piece so the bot has a deterministic shape to read. - Keep the existing 'Linear ticket' section for internal contributors (they're exempt from the auto-triage rubric). Bug report template: - Split 'What happened?' into 'Actual behavior' + 'Expected behavior'. - Make logs/screenshot a required textarea. - Warning banner at the top tells external contributors that incomplete reports will be auto-closed (with re-evaluation on reopen). Feature request template: - Require a concrete use case + example in the motivation field, not just a one-liner pitch. - Same auto-triage warning banner. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): Agent Shin LLM-as-judge for external PRs and issues Adds a new triage flow that evaluates external pull requests and issues against the project's contribution rubric and, when configured to do so, auto-closes non-conforming ones with an explanatory comment. Contributors can update + reopen to be re-evaluated. Scope: - Internal BerriAI contributors (author_association OWNER/MEMBER/COLLABORATOR) and bot accounts are skipped entirely. - 'Fixes #1234' / 'Resolves https://github.com/.../issues/N' in the PR body short-circuits to PASS without burning LLM tokens. - LLM judge returns structured JSON (verdict, missing[], explanation); parser tolerates markdown fences and embedded JSON. - LLM errors NEVER close PRs/issues — failure surfaces as 'skip-llm-error'. Safety: - pull_request_target / issues triggers are FORCED dry-run in the workflow; only manual workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true) takes destructive action. - Default mode writes verdicts to GITHUB_STEP_SUMMARY only — no public comments until the team flips the AGENT_SHIN_ENABLED repo variable. - LLM uses an OpenAI-compatible endpoint (model and base URL configurable via repo variables; key via OPENAI_API_KEY secret). Files: - .github/scripts/triage_with_llm.py - judge orchestrator + CLI - .github/workflows/triage_pr_with_llm.yml - .github/workflows/triage_issue_with_llm.yml - tests/test_litellm/test_github_triage_with_llm.py - 33 unit tests End-to-end validated against four real PRs (#28117 internal collaborator, #28108 bot, #28129 'Fixes #28128', #28116 no linked issue) and issue #28132 with a stubbed LLM judge: each path produces the expected action. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): scope Greptile auto-closer to external contributors + dry-run by default - close_low_quality_prs.py now filters by GitHub author_association via the REST API: PRs from OWNER / MEMBER / COLLABORATOR (and bot accounts) are skipped with a new 'skip-internal' summary bucket. - close_low_quality_prs.yml now defaults workflow_dispatch close=false, and ignores 'close=true' unless the new repo variable AGENT_SHIN_ENABLED is set to 'true'. Scheduled runs are dry-run only until the team flips that switch. - Updated unit tests: one new test asserting internal authors are skipped, and an autouse fixture treats unspecified test PRs as external so the rest of the suite still exercises the close path. Co-authored-by: Mateo Wang <[email protected]> * fix(workflows): scheduled cron closes PRs; safe --close strip in triage Co-authored-by: Yassin Kortam <[email protected]> * fix(triage): scheduled cron stays dry-run; dedent prompts before interpolation - close_low_quality_prs.yml: only workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true) actually closes PRs. Scheduled runs are always dry-run, matching the safety invariant documented for triage_pr/issue. - triage_with_llm.py: textwrap.dedent on an f-string with multi-line interpolated bodies fails because the body's 2nd+ lines start at column 0, making the common-indent zero. Dedent the static template first, then .format() the title/body in. Co-authored-by: Yassin Kortam <[email protected]> * Fix bugs in auto-close PR triage scripts - close_low_quality_prs.py: Treat author_association API lookup failures as internal (fail-safe) so transient errors don't cause internal contributors' PRs to be auto-closed. - triage_with_llm.py: Update summary heading from 'Would post comment:' to 'Posted comment:' since this branch only runs after the comment has already been posted. Co-authored-by: Yassin Kortam <[email protected]> * feat(triage): default Agent Shin to gpt-5.4-mini with reasoning_effort=none - Bump DEFAULT_MODEL from gpt-4o-mini to gpt-5.4-mini (more modern; 4M total context window per OpenAI catalog, JSON-schema response format, function calling all supported). - For gpt-5.x family models, pass reasoning_effort="none" via extra_body. gpt-5.x rejects temperature != 1 unless reasoning_effort is explicitly "none"; setting it lets us keep temperature=0 for deterministic JSON rubric judgments. extra_body works across openai SDK versions regardless of whether they natively type the kwarg. - For non-gpt5 overrides (TRIAGE_MODEL=gpt-4o-mini etc.), reasoning_effort is not sent. - 4 new unit tests cover: gpt-5.4-mini -> reasoning_effort=none, capitalized/dated gpt-5 variants -> reasoning_effort=none, gpt-4o-mini -> no extra_body, base_url passthrough. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — drop dead gh_json and fix --optout-label append-with-default - Removed the unused gh_json helper (bugbot low-severity dead code). - Replaced argparse `action="append", default=[...]` with default=None + DEFAULT_OPTOUT_LABELS fallback. The mutable-default + append combo silently APPENDS to the canonical defaults instead of replacing them, so --optout-label could not actually scope the opt-out list. - Added tests covering both the canonical default and the flag-replaces-defaults behavior. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — tighten linked-issue regex, fail-safe author_association, fix empty TRIAGE_MODEL Three independent bugbot findings against triage_with_llm.py: 1. LINKED_ISSUE_PATTERN included weak keywords (`see`, `ref`, `addresses`) so casual mentions like "See #1234 for context" were short-circuited to pass-linked-issue without ever calling the LLM — contradicting the prompt's own "a bare issue number without a closing keyword counts only if it's clearly the related issue (not a passing mention)" rubric. Limit the regex to GitHub's documented PR-closing keywords (fixes/fix/fixed/closes/close/closed/resolves/resolve/resolved). 2. is_internal_contributor() treated an empty/missing author_association as external (eligible for the destructive close path), while the sibling is_external_pr_author() in close_low_quality_prs.py fail-safes the same case as internal. Align the two so a partial/unknown GitHub response can never make a PR eligible for auto-close. 3. argparse `default=os.environ.get("TRIAGE_MODEL", DEFAULT_MODEL)` returns the empty string when GitHub Actions exposes an unset repo variable as an empty-string env var (the optional vars.TRIAGE_MODEL case in the workflow). Use `os.environ.get(...) or DEFAULT_MODEL` so empty -> default, matching the existing OPENAI_BASE_URL pattern. Tests: - Casual mentions now must fall through to the LLM (parametrized); added an orchestration test ensuring "See #1234" reaches the judge. - Empty/missing author_association now fails safe (parametrized). - Empty TRIAGE_MODEL env var falls back to DEFAULT_MODEL; explicit TRIAGE_MODEL is still honored. Co-authored-by: Mateo Wang <[email protected]> * fix(workflows): bugbot — gate Agent Shin --close on '= true' not '!= false' The PR and issue Agent Shin workflows gated the destructive --close flag with [ "${DISPATCH_CLOSE:-false}" != "false" ]. That pattern treats anything other than the literal string "false" as enabling closure — "True", "yes", "1", typos, accidental whitespace, etc. The workflow_dispatch input UI is a 'true'/'false' choice dropdown so the form is constrained, but the API (`gh workflow run -f close=...`) accepts any string, and a CI cron / external invoker passing a non-canonical truthy value would have silently enabled real contributor PR closures. Mirror the sibling Greptile closer's [ "${CLOSE_FLAG}" = "true" ] pattern: only the EXACT string "true" enables --close; every other value (including the unset/empty default) resolves to dry-run. This is the fail-safe philosophy applied everywhere else in this PR. Added tests/test_litellm/test_github_triage_workflows.py with two parametrized invariants: 1. The destructive gate uses '= "true"' for its env-var comparison (either bare '${ENV}' or '${ENV:-false}' form accepted), and never the fail-open '!= "false"' pattern. 2. Every destructive gate is also gated on AGENT_SHIN_ENABLED being "true" — either by entering the close branch on '=' or by bailing out early on '!=' — so flipping the repo variable off is a true kill switch regardless of per-run inputs. Manually verified the test fails on the buggy '!= "false"' pattern and passes on the fix, so it would have caught the regression at PR time. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): close any PR (incl. drafts, any age); add @agent-shin reconsider flow Follow-up to PR #28117. Three behavior changes + one new workflow, addressing the team's concerns on the original review: 1) Apply auto-close to ALL open PRs, not just those over a week old. - close_low_quality_prs.py: --min-age-days default flipped from 7 to 0. The flag is preserved as an opt-in safety net for one-off backfill runs that want to spare very-young PRs, but the daily scheduled sweep now closes external-author PRs as soon as Greptile scores them <4/5. - close_low_quality_prs.yml: workflow_dispatch input default also flipped to 0; doc comments updated. 2) Apply auto-close to draft PRs too. - close_low_quality_prs.py: removed the skip-draft branch in evaluate_pr. Drafts are NOT a free pass — the team's intent is 'open PR count == PRs internal collaborators need to action on', so a draft Greptile scored 2/5 still belongs in the closed bucket. Authors who genuinely need a long-lived draft can attach the 'wip' opt-out label, which is unchanged. - The 'skip-draft' action is gone; the 'wip' label still skips. 3) Address the 'OSS contributors cannot reopen a bot-closed PR' wrinkle. GitHub does NOT let an external (non-write-access) contributor reopen a PR that was closed by a bot or maintainer (long-standing limitation). The original PR's close-comments told contributors to 'Reopen the PR — I'll re-evaluate automatically', which is broken for the very audience this triage targets. Two changes: a) Reword every close-comment (Greptile sweep + Agent Shin PR close + Agent Shin issue close + PR template) to recommend: - Open a new PR with the updated branch (primary path). - Or comment '@agent-shin reconsider' on the closed PR for a re-evaluation that, on pass, reopens the PR via the bot's GH_TOKEN write access. b) Add the @agent-shin reconsider workflow: - .github/workflows/triage_reconsider.yml: new 'issue_comment'-triggered workflow. Authorizes only the PR/issue author or an internal collaborator (OWNER/MEMBER/COLLABORATOR), gated via a step output so unauthorized commenters never reach the destructive steps. Globally gated on AGENT_SHIN_ENABLED='true' (positive form, matching the test_github_triage_workflows guardrail patterns). - triage_with_llm.py: --reconsider mode. On a closed PR/issue, re-runs the LLM judge (or linked-issue regex short-circuit) and: - on pass: reopens via reopen_pr/reopen_issue + posts a 'Re-evaluated and reopened' comment. - on fail: leaves closed and posts a 'still missing X' comment so the contributor can iterate again. Reconsider-on-open is a no-op ('skip-not-closed'). Internal-author + bot-account skips still take priority over reconsider. 4) Greptile-on-closed-PRs question: the team asked whether Greptile can re-review a closed PR. Greptile's docs don't address this and we shouldn't promise behavior we can't verify, so the new close-comment wording does NOT instruct contributors to 're-request greptile on the closed PR'. Instead it points them at the new-PR path (which Greptile definitely reviews) or the @agent-shin reconsider trigger (which re-runs the LiteLLM-side rubric judge, not Greptile). Tests: 93 passing (was 59). - test_github_close_low_quality_prs.py: replaced 'skip drafts' test with 'closes drafts when score is low' + 'closes brand-new PR when min_age=0' + 'no skip when min_age=0'. The 'skip too young' assertion is preserved as opt-in. - test_github_triage_with_llm.py: 6 new TestTriageOrchestration cases for reconsider mode (skip-not-closed on open, reopen on pass, still-failing comment on fail, linked-issue short-circuit reopen, skip internal author in reconsider, reopen-issue on pass) + a new TestCloseCommentText class that pins the user-facing 'open a new PR' + '@agent-shin reconsider' wording. - test_github_triage_workflows.py: added triage_reconsider.yml to the destructive-gate guardrail table; AGENT_SHIN_ENABLED is its own destructive gate (no separate per-run flag needed). Co-authored-by: Mateo Wang <[email protected]> * test(triage): pin safe behavior for curly braces in PR/issue title+body Adds regression tests covering the bugbot high-severity finding that str.format() would crash on user-supplied content containing { or }. Empirically str.format() does NOT re-parse interpolated values — only the template literal is scanned for replacement fields — so the bug does not exist in the current code, but pinning the safe behavior prevents a future templating change from silently reintroducing it. Also pins the dedented prompt shape (no leading 8-space indentation on template lines) so a future change to the build_*_prompt functions can't silently regress the LLM judge prompt format on multi-line bodies. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — reconsider dry-run + bot-closed guard + rate limit Address three Greptile/veria-ai concerns on the @agent-shin reconsider flow: 1. **Reconsider had no dry-run path.** The previous reconsider mode ignored `--close` and always posted comments + reopened on a pass. A local operator running `python triage_with_llm.py --reconsider --pr N` would silently take destructive GitHub actions with no way to preview. Reconsider now honors `close=False` the same way regular triage does and returns `would-reopen` / `would-reconsider-still-failing` for step-summary rendering. 2. **Reconsider could reopen maintainer-closed PRs/issues** (Medium security finding from veria-ai). The workflow only checked that the commenter was authorized — it did NOT check that the most recent close was performed by Agent Shin. A contributor could comment `@agent-shin reconsider` on a PR a maintainer closed for non-rubric reasons (duplicate, security report, design rejection) and have the bot reopen it. Add `was_closed_by_agent_shin()` which inspects the issue events API for the most recent `closed` actor and only permits reopen when that actor matches the configured bot login (default `github-actions[bot]`, overridable via env). Fail-closed on missing events. 3. **No rate-limiting on the reconsider trigger.** Every `@agent-shin reconsider` comment burns CI minutes + an OpenAI API call. Add a 10-minute cooldown via `seconds_since_last_reconsider_verdict()` which greps the issue's comment list for the bot's own verdict marker (``). Inside the window the triage returns `skip-rate-limited` and the LLM never runs. Workflow update: - `triage_reconsider.yml` now passes `--close` only when `AGENT_SHIN_ENABLED=true`, matching the pattern of `triage_pr_with_llm.yml`. The script runs in both states so the verdict still appears in the step summary for QA. Tests: - Add 5 reconsider safety tests: dry-run for pass / fail / linked-issue short-circuit, bot-closed-guard refusal on maintainer close, rate-limit refusal inside the cooldown window, and cooldown-elapsed acceptance. - Add unit tests for `was_closed_by_agent_shin` (bot / maintainer / missing actor / env-override) and `seconds_since_last_reconsider_verdict` (no marker / multiple markers / non-bot comment with marker / bot comment without marker). - Pin the `` marker in both reopen and still-failing comments — dropping it would silently break the cooldown. Existing reconsider tests updated to pass `close=True` (the production path now) + stub the new guards via `_stub_reconsider_guards`. 112 tests pass (was 93). Co-authored-by: Mateo Wang <[email protected]> * feat(triage): 1-day grace period before close + SwiftWinds immediate-close bypass - Add a 24-hour grace window between the first low-quality detection and the actual auto-close. The first detection posts a warning comment that explicitly says "You have 1 day to address this before this PR is auto-closed" and points the contributor at: * `@agent-shin reconsider` to request another look (and re-open) * `@greptileai` to request a fresh Greptile review — works even after the PR is closed - Both `triage_with_llm.py` (LLM judge) and `close_low_quality_prs.py` (Greptile-score closer) share the same `` HTML marker so a warning posted by either path is recognized by both. - Add IMMEDIATE_CLOSE_LOGINS = {swiftwinds} to bypass BOTH the grace period AND the dry-run / AGENT_SHIN_ENABLED gating. SwiftWinds is the user's personal account (no push permissions to litellm) used to dogfood the bot; user explicitly asked: "For SwiftWinds, just close immediately. Faster iteration that way." - Update the standard close comments to mention that `@greptileai` works even after the PR is closed. - Add 23 new tests covering: warn-grace on first detection, skip during grace window, close after grace expires, SwiftWinds bypass (case insensitive, with close=False, no random-login false positives), the grace-warning text invariants, and the SwiftWinds entry in the IMMEDIATE_CLOSE_LOGINS constant. Co-authored-by: Mateo Wang <[email protected]> * fix: skip grace-period text in close comment for IMMEDIATE_CLOSE_LOGINS For PRs from IMMEDIATE_CLOSE_LOGINS (e.g. swiftwinds), evaluate_pr returns 'close' immediately without ever posting a grace warning, so the close comment should not reference a 1-day grace period. Make close_pr take a grace_period_elapsed flag, default True, and pass False from the main loop when the close path was the immediate-close branch. Co-authored-by: Yassin Kortam <[email protected]> * fix(close-low-quality-prs): report actual closes in dry-run summary IMMEDIATE_CLOSE_LOGINS PRs are closed even when the global --close flag is not set, but the summary used the global dry-run flag to choose between 'would close' and 'closed'. Split the count so operators can see both actual closures and dry-run would-be closures. Co-authored-by: Yassin Kortam <[email protected]> * chore(triage): vendor Agent Shin (#28117) onto demo branch Brings the Agent Shin OSS-triage scripts, workflows, issue/PR templates, and tests from PR #28117 onto this branch so the new review-gate feature and its end-to-end demo are self-contained and runnable in CI. https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ * feat(triage): add "ready for review" label lifecycle to Agent Shin Adds review_gate(), a state machine that keeps a `ready for review` label in sync with whether an external PR clears BOTH gates — the LLM rubric and Greptile's most recent confidence score: - pass (untagged) -> add label + "ready for review" / "all clear" comment - pass (already tagged) -> no-op (idempotent across re-runs) - regress (Greptile < 4/5 or QA proof removed) -> remove label + "what's missing" comment, PR stays open - recover after a regression -> "all clear again" comment + re-add the label - fail & untagged, < 24h old -> one-time "what's missing" notice (grace window) - fail & untagged, > 24h old -> close + comment (reopen via @agent-shin reconsider) The label itself is the persisted state, so comments fire only on transitions (never on every scheduled run). All side effects are gated behind --close, so the dry-run contract matches the existing triage flow. Lifecycle comments use hidden HTML markers and deliberately avoid the auto-close marker so they never trip the reconsider provenance check. Relocates the shared Greptile helpers (extract_greptile_score, SCORE_PATTERN, GREPTILE_BOT_LOGINS, parse_iso8601) into triage_with_llm.py so the daily sweep and the review gate read the score through one implementation, and adds the review_gate.yml workflow (dry-run unless AGENT_SHIN_ENABLED=true) plus 18 unit tests covering every branch and a full pass->regress->recover cycle. https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ * Port review-gate feature from #28758 onto #28147 triage scripts Adds the "ready for review" label lifecycle (originally PR #28758) on top of #28147's refactored triage_with_llm.py. The original commit was authored against an older snapshot of #28117 and could not be applied cleanly, so the additions were re-applied surgically: - New constants: READY_FOR_REVIEW_LABEL, DEFAULT_GRACE_DAYS, DEFAULT_MIN_GREPTILE_SCORE, READY/REGRESSED/WITHIN_GRACE markers, GREPTILE_BOT_LOGINS, SCORE_PATTERN, AGENT_SHIN_AUTO_CLOSE_MARKER. - New helpers: add_label, remove_label, extract_greptile_score, parse_iso8601 (the latter two mirrored from close_low_quality_prs.py so the daily sweep and the review gate read the score through the same logic). - New comment formatters: format_ready_for_review_comment, format_all_clear_comment, format_regression_comment, format_within_grace_comment. - New entry point: review_gate() implementing the pass/regress/recover state machine, with the label itself acting as persisted state so transition comments fire only on actual transitions. - main() learns --review-gate, --grace-days, --min-greptile-score and dispatches to review_gate() when the flag is set. Verified via tests/test_litellm/test_github_review_gate.py (18 tests) and the existing triage suites (144 more) — all 162 pass. Co-Authored-By: Claude Opus 4.7 <[email protected]> * agent_shin: extract shared constants/helpers; cover review_gate.yml in guardrail tests Bug 1: `triage_with_llm.py` and `close_low_quality_prs.py` each defined their own copies of `extract_greptile_score`, `parse_iso8601`, `GREPTILE_BOT_LOGINS`, `SCORE_PATTERN`, `GRACE_COMMENT_MARKER`, `GRACE_PERIOD_SECONDS`, `IMMEDIATE_CLOSE_LOGINS`, and `AGENT_SHIN_DEFAULT_BOT_LOGIN`. The comments explicitly said the two copies had to stay in sync, but nothing enforced it. A future change to one (e.g. extending `SCORE_PATTERN` for a new Greptile output format) would silently diverge from the other and the daily sweep and the LLM judge would disagree on which PRs have low scores. Extract these to `.github/scripts/agent_shin_shared.py` and re-export them from each script so the existing test attribute access (`triage_module.GRACE_COMMENT_MARKER`, etc.) keeps working without any test changes. Bug 2: `review_gate.yml` is a destructive workflow (close PRs, add/remove labels, post comments) with the same gating philosophy as the others (`AGENT_SHIN_ENABLED = "true"` + a per-run `CLOSE_FLAG = "true"`), but it was missing from `DESTRUCTIVE_GATE_ENV` in the guardrail tests. Add it so a future regression (e.g. flipping to `!= "false"`) is caught by the same parameterized invariants as every other workflow. Co-authored-by: Yassin Kortam <[email protected]> * agent_shin: fix bug bundle (gated LLM key, author-filtered marker dedup, dedup gh/grace helpers) Co-authored-by: Yassin Kortam <[email protected]> * agent_shin: fix review_gate close-after-regression and case-insensitive label match Co-authored-by: Yassin Kortam <[email protected]> * feat(triage): add one-shot 7-day heads-up sweep for Agent Shin rollout Adds a rollout-day workflow that comments on every open external PR/issue that the new triage bot WOULD auto-close, giving contributors 7 days to fix their description before any destructive action runs. Why now: merging this PR enables Agent Shin in dry-run. The follow-up "enact" PR (next Monday) flips the destructive paths on. Without this heads-up, contributors would get a close-comment on day 8 with no prior warning. The heads-up names the cutoff date, lists the rubric, calls out each PR/issue's specific missing pieces, and explains the recovery paths (@agent-shin reconsider for PRs, edit + reopen for issues). Files - .github/scripts/_agent_shin_actions.py — thin maybe_post_comment / maybe_close_* / maybe_add_label / etc. wrappers. Each is a single `if dry_run: log; return; else: call_through()` so a dry-run preview differs from the real run in exactly one call site per mutation. The call-through goes via `triage_with_llm.<name>` (module-qualified) so monkeypatching the underlying function in tests is reflected here. - .github/scripts/triage_rollout_heads_up.py — the sweep. Iterates every open PR + issue via `gh pr list` / `gh issue list`, runs the future rubric (review_gate for PRs, triage(kind="issue") for issues), and posts the heads-up on any item that would be auto-closed. Idempotent via a `` marker. Defaults to dry- run; --close opts in to real posts. --close-on overrides the cutoff date (defaults to today + 7 days). - .github/workflows/triage_rollout_heads_up.yml — one-shot workflow. Triggers on push to litellm_internal_staging filtered to the script path (fires on rollout merge) plus workflow_dispatch with a dry_run input that defaults to "true" for safe manual re-runs. - tests/test_litellm/test_triage_rollout_heads_up.py — 28 unit tests covering: the dry-run wrappers (each maybe_* gates correctly), the _would_be_closed predicate for PR vs. issue results, the comment formatter (cutoff/rubric/marker/recovery wording), per-item dispatch (skip-not-open, skip-internal-author, skip-already-notified, skip-passing, would-post/posted), and the sweep loop end-to-end. Local preview (no GitHub mutations): python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm Real run (what the workflow does): python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm --close TODO: replace the placeholder ROLLOUT_BLOG_URL with the canonical docs URL once the litellm-docs PR ships. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: gate reconsider workflow OPENAI_API_KEY + remove dead actions wrappers - Mirror sibling Agent Shin workflows by only exposing OPENAI_API_KEY in triage_reconsider.yml when vars.AGENT_SHIN_ENABLED == 'true'. Previously the secret was unconditionally exposed, so any PR/issue author could trigger paid LLM calls by commenting '@agent-shin reconsider' even while the bot was supposed to be in dry-run. - Remove the six unused dry-run wrappers (maybe_close_pr, maybe_close_issue, maybe_reopen_pr, maybe_reopen_issue, maybe_add_label, maybe_remove_label) from _agent_shin_actions.py — only maybe_post_comment is used by rollout scripts. Drop the associated tests that exercised the now-removed functions. Co-authored-by: Yassin Kortam <[email protected]> * fix: address triage script edge cases - triage_rollout_heads_up.py: replace %-d strftime specifier (GNU-only) with portable day formatting so the script doesn't crash on Windows. - close_low_quality_prs.py: skip malformed JSON lines in fetch_pr_comments instead of letting one bad line abort the daily sweep, matching the pattern in triage_with_llm._iter_paginated_json. - triage_with_llm.py: move has_linked_issue short-circuit before build_pr_prompt to avoid unnecessary prompt construction on PRs that link an issue. Co-authored-by: Yassin Kortam <[email protected]> * fix(scripts): per-PR error isolation and limit grace warnings in close_low_quality_prs - Wrap per-PR processing in try/except so a transient GitHub API failure on one PR no longer aborts the entire daily sweep (mirrors the pattern already used in triage_rollout_heads_up.py). - Have --limit bound *all* destructive write actions (closures and grace warnings combined), not just closures. Prevents a backlog of newly failing PRs from flooding contributors with comments in a single run. Co-authored-by: Yassin Kortam <[email protected]> * fix(agent-shin): remove 1000-PR cap on bulk sweeps; sweep entire backlog Both bulk-sweep scripts hardcoded `gh {pr,issue} list --limit 1000`, and gh lists newest-first — so the OLDEST ~900 PRs and ~380 issues were silently dropped. That's exactly the stale backlog the daily closer and one-shot rollout heads-up exist to catch. Extract a single `list_open_items(kind, *, repo, fields)` helper into `agent_shin_shared.py` with `GH_LIST_ALL_LIMIT = 100_000` — a ceiling far above any realistic open backlog so gh paginates until the queue is exhausted. `fetch_open_prs` and `_list_open_numbers` both delegate to it, so the limit lives in exactly one place going forward. Verified live against BerriAI/litellm: - `fetch_open_prs` -> 1981 PRs (was 1000) - `_list_open_numbers(issue)` -> 1382 issues (was 1000) - `_list_open_numbers(pr)` -> 1981 PRs (was 1000) Adds 7 regression tests asserting the new limit is passed, the dedicated `gh {pr,issue} list` command + fields are used per kind, bad kind raises ValueError, and both callers delegate to the shared helper. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(agent-shin): require non-mocked end-to-end QA proof for PR pass The PR rubric previously passed any PR with a linked issue, regardless of whether it showed the fix actually working. Sample spot-check found 21/25 recent external PRs passing, including ones that linked an issue but provided zero QA evidence. Tighten the rubric so a pass now requires BOTH: (1) CONTEXT — a linked issue OR a clear problem description with expected-vs-actual behavior. (2) END-TO-END QA PROOF — at least one of: (a) screenshot(s) of the fix working, (b) screen recording / video, (c) specific commands actually run, paired with their real output, against the real system. Mocked unit tests, generic 'I tested it' claims, 'all tests pass' without output, and the linked issue itself are explicitly excluded from QA proof. Also add 'qa_proof_type' to the JSON schema so the per-PR report surfaces which kind of proof (or 'none') the judge saw. Re-sample on the same 25 recent external PRs shifts the verdict distribution from 21 pass / 4 fail to 4 pass / 21 fail, with zero prior-fails now passing — the stricter rule catches PRs that ship only with unit-test claims and no real integration evidence. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(agent-shin): link blog explainer from every action-required bot comment Adds "What's this and why am I getting it?" links to docs.litellm.ai/blog/ agent-shin-triage from the four comments contributors actually read when something went wrong: PR close, PR grace warning, issue close, issue grace warning. PR comments also link the rubric section directly from the QA-proof bullet so contributors can self-serve "what counts as proof" without pinging a maintainer. Pins the new guarantees in tests: blog link must appear in all four comments, and the PR close comment must continue to flag mocked-dependency unit tests as insufficient proof. The linked blog post is in BerriAI/litellm-docs PR #240; the URL will 404 until that lands. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review_gate): raise sweep limit from 1000 to 100000 to match GH_LIST_ALL_LIMIT gh lists newest-first, so capping at 1000 silently drops the oldest open PRs — exactly the stale ones the daily sweep is meant to reconcile. Use the same ceiling as agent_shin_shared.GH_LIST_ALL_LIMIT so the workflow sees the entire backlog. Co-authored-by: Yassin Kortam <[email protected]> * Fix three Agent Shin triage edge cases - review_gate: expire the regression-marker short-circuit after grace_days so PRs that were regressed and then abandoned can eventually be closed. - review_gate: when the rubric short-circuits to pass via the linked-issue regex but Greptile drags the PR below the bar, replace the synthetic 'LLM was not called' explanation with the real Greptile shortfall so regression / close comments are not misleading. - triage_rollout_heads_up._comments_have_marker: drop the unused 'kind' parameter and filter by bot author so a contributor quoting the heads-up via 'Quote reply' cannot trick the idempotency check, matching the pattern in triage_with_llm._has_marker. Co-authored-by: Yassin Kortam <[email protected]> * fix: pass min_greptile_score through to ready-for-review comment text Co-authored-by: Yassin Kortam <[email protected]> * feat(agent-shin): warmer triage comments — bullet-train emoji, 'what you got right' section, softer 'park this for later' framing User feedback on the auto-triage comments contributors will see: 1. Tone — the previous 'You have 1 day to address this before this PR is auto-closed' framing reads as an ultimatum. Replace with: 'If the description isn't updated in the next 1 day, I'll auto-close this PR. That's not us saying we don't care about the change — we want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later," not a rejection. Take your time.' 2. Positive feedback — the previous comments only listed what was missing. Now every close + grace-warning comment opens with a 'What you got right:' section rendered from the judge's per-field flags. Contributors see a checkmark for everything they got right (linked issue, problem description, expected/actual, QA proof for PRs; runnable repro, screenshot/log, expected/actual, motivation+example for issues) before the gaps. The block is omitted entirely when nothing is present so we never render 'What you got right: (nothing).' 3. Reconsider trigger — the previous grace warning told contributors to comment '@agent-shin reconsider' during the grace window. They don't need to — the bot re-checks on every sweep. The new copy says 'just update the description, no need to ping me' for the grace path, and reserves '@agent-shin reconsider' for the post-close recovery path. 4. Bullet-train emoji — replace 👋 with 🚄 (Shinkansen, the symbol of Agent Shin) across every action-required comment: PR close, PR grace warning, issue close, issue grace warning, within-grace, Greptile- closer grace warning, rollout heads-up. Pinned in tests so a future refactor can't silently revert. 5. Greptile-post-close — the @greptileai bullet now explicitly says 'a low Greptile score isn't a blocker either,' since the previous copy buried the fact that @greptileai works after auto-close. Comment templates updated: format_pr_close_comment, format_issue_close_comment, format_grace_warning_pr_comment, format_grace_warning_issue_comment, format_within_grace_comment (triage_with_llm.py); format_grace_warning_comment (close_low_quality_prs.py); format_heads_up_comment header (triage_rollout_heads_up.py). New helpers: _format_present_for_pr / _format_present_for_issue / _format_present_block, driven off the existing per-field flags the LLM judge already emits — no prompt change needed. New tests pin: bullet-train emoji in every action-required comment; 'What you got right' appears with ✅ bullets when fields are present; the block is omitted when no fields are present; 'park this for later' / 'not a rejection' softer framing; grace warnings tell the contributor 'no need to ping' during the grace window (reconsider is the post-close path only). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(agent-shin): gate triage on a dogfood allowlist Add ALLOWLIST_LOGINS to agent_shin_shared so Agent Shin only acts on the named accounts while the set is non-empty. mateo-berri and SwiftWinds are allowlisted for the dogfood rollout; everyone else is skipped with skip-not-allowlisted across all four entrypoints (triage, review gate, the daily low-quality sweep, and the rollout heads-up). For an allowlisted author the usual internal/external classification is bypassed, so a maintainer's own org account still gets triaged during testing. Emptying the set lifts the restriction and restores full triage for the public rollout. The gate is dependency-injected via an `allowlist` parameter defaulting to the constant, so the internal/external-skip paths stay testable. * feat(agent-shin): tighten QA-proof and issue rubrics, ack reconsider with reactions Reorder the end-to-end QA proof options to video, then screenshots, then exact commands with their real output across the PR template, the LLM judge prompts, and every contributor-facing comment, and spell out that mocked or stubbed runs (including pytest on the repo's own unit tests, which mock the provider, DB, and network) never count as proof. QA proof is now required of all contributors, not just external ones. Tighten the issue bug-report rubric to require end-to-end evidence of the bug (the "before" half: a video, screenshot, or command paired with real output) plus expected vs. actual behavior, drop the bias toward PASS, and collapse the separate has_repro/has_proof flags into a single has_repro signal. Standardize the bullet-train emoji and strip em dashes from the bot's public-facing messages, and route issue recovery through @agent-shin reconsider since GitHub doesn't let OSS authors reopen an issue a bot closed. Acknowledge an @agent-shin reconsider the moment it's accepted with an eyes reaction and a thumbs-up once the run finishes, both gated on AGENT_SHIN_ENABLED so dry-run leaves no trace. * fix(agent-shin): shorten auto-close grace to 2 hours and drop the instant-close bypass Two dogfooding changes to the Agent Shin grace window. First, the warn-then-close grace (GRACE_PERIOD_SECONDS) drops from a day to 2 hours so the "fix it before it closes" loop can be exercised in one sitting; the constant carries a note to bump it back up for the public rollout. Second, remove IMMEDIATE_CLOSE_LOGINS entirely. SwiftWinds (the external dogfood account) used to skip the grace window and close on first detection, which also meant closing real PRs even during a scheduled dry run because the per-PR override flipped dry_run off. It now follows the same warn-then-close path as every other author, so a low-quality PR is warned first and only closed once the 2-hour window elapses. This also closes the Greptile finding that the sweep could mutate real PRs while AGENT_SHIN_ENABLED was still off. The review gate's separate age-based grace (DEFAULT_GRACE_DAYS) is left unchanged. Regression tests pin that SwiftWinds now warns-grace instead of closing instantly, and that a dry-run sweep over a closeable PR reports "would close" without making any GitHub mutation. * fix(agent-shin): gate reconsider reopen on an Agent Shin close marker was_closed_by_agent_shin only checked that the most recent close actor was the bot identity. That identity defaults to github-actions[bot], which is shared by every workflow in the repo (stale/duplicate sweeps included), so a contributor could @agent-shin reconsider an item another workflow closed and, if the description passed the rubric, get it reopened even though Agent Shin was never the closer. Require a second, Agent-Shin-specific signal alongside the actor check: an auto-close comment stamped with a hidden AGENT_SHIN_CLOSE_MARKER. Both close paths (the grace-period close and the review-gate close) flow through format_pr_close_comment / format_issue_close_comment, so stamping the marker there covers every real close while leaving the grace warnings unmarked. The guard stays fail-closed: no marker, no reopen. This also replaces the unused AGENT_SHIN_AUTO_CLOSE_MARKER constant (a visible phrase the guard never consulted) with the hidden marker the guard now relies on. * fix(agent-shin): stamp close marker on sweep closes and disclose regression deadline The daily Greptile sweep's close comment advertised `@agent-shin reconsider` but never stamped AGENT_SHIN_CLOSE_MARKER, so the reconsider reopen guard (was_closed_by_agent_shin), which now also requires that marker, silently rejected every sweep-closed PR with `skip-not-bot-closed`. Move the marker into agent_shin_shared so both close paths share one source of truth, extract format_close_comment so the sweep close comment is unit-testable, and stamp the marker there. Also disclose the grace_days deadline in the review-gate regression comment; it promised "the PR stays open" without mentioning that a still-failing PR is auto-closed grace_days after the notice, which would surprise contributors with a close they were never warned about. * fix(triage): tighten Agent Shin reconsider reopen guards The bot-closed guard accepted any historical Agent Shin marker comment on the thread as proof that Agent Shin owned the latest close, so a post-reopen close by another workflow under the shared `github-actions[bot]` identity could still satisfy the gate and let `@agent-shin reconsider` reopen a PR that Agent Shin did not close this cycle. `fetch_last_close_event` now also returns the latest `closed` event timestamp, and `was_closed_by_agent_shin` requires the most recent Agent Shin marker comment to sit at (or just before) that timestamp, with a small skew window for clock drift between the events and comments APIs. In the same path the LLM verdict check used `decision != "fail"` to choose the reopen branch, which treated a missing, empty, or typo verdict as a pass. Reopen is destructive, so the check now requires an explicit `decision == "pass"` and ambiguous verdicts fall through to the "still failing" branch instead. * style(agent-shin): black-format reconsider guard hardening * docs(agent-shin): scope dry-run wrapper docstring to the single existing helper The module docstring claimed it wrapped every Agent Shin mutation and referenced post_comment/close_pr/etc., but only maybe_post_comment exists. Describe the single helper accurately while keeping the dry-run pattern guidance for any future wrapper. * chore(agent-shin): defer issue/PR template changes to the rollout PR The triage and review-gate automation is gated to the allowlisted authors (mateo-berri, SwiftWinds) and AGENT_SHIN_ENABLED, so during this rollout it only acts on internal PRs/issues. The issue and PR templates have no such gate; they change for every contributor on merge and advertise that an LLM bot auto-closes external submissions, which won't happen while the allowlist is the sole author gate. Revert bug_report.yml, feature_request.yml, and pull_request_template.md to base so the public-facing messaging lands with the rollout flip instead of ahead of it. The scripts embed their own rubric and never read these files, so triage behavior is unchanged. * ci(agent-shin): hash-pin the openai install in privileged triage workflows The triage workflows install the OpenAI client with `pip install "openai>=1.40.0"`, a floating lower bound that resolves openai and its whole transitive tree to whatever PyPI serves at run time. These jobs run under pull_request_target with a write-scoped GITHUB_TOKEN, and the install plus the triage run happen on every PR open regardless of the AGENT_SHIN_ENABLED dry-run gate (that gate only withholds the LLM key and the destructive --close path), so a compromised release would execute during install or import while the token is in scope. Install instead from a new .github/scripts/triage-requirements.txt that pins openai==2.33.0 and every transitive dependency to an exact version with sha256 hashes, via pip --require-hashes. The workflows already sparse-checkout .github/scripts from the base repo (never fork code), so the pinned file is trusted. Add static guardrails to test_github_triage_workflows.py that fail if any installer workflow reverts to a floating openai install or if the requirements file loses its exact pins or hashes. * ci(agent-shin): gate rollout heads-up real run behind manual dispatch The rollout heads-up workflow fired its real `--close` sweep on every push to litellm_internal_staging that touched the script, and exposed OPENAI_API_KEY unconditionally, unlike every sibling triage workflow which only exposes the key on an enabled or dispatched run. That made merging the script post real heads-up comments (bounded only by the dogfood allowlist), which contradicts the inert-by-default safety invariant; once the allowlist is cleared for the public rollout, any later edit to the file would sweep the whole open backlog with real writes. The heads-up cannot be gated on AGENT_SHIN_ENABLED: its whole job is to warn contributors before that flag flips on, so it has to run while the flag is still off. Instead the automatic push trigger now stays dry-run, and the real one-shot sweep is a deliberate manual workflow_dispatch with dry_run=false, the sole path that adds `--close`. OPENAI_API_KEY is exposed only on that dispatch, matching the sibling workflows. Add static guardrails that fail if the push path regains a `--close`, if the dispatch gate stops fail-closing on the exact string "false", or if the key is exposed unconditionally again. --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Mateo Wang <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Mateo <[email protected]>

* fix(anthropic): price and surface response service_tier in cost tracking (#30558) * feat: add dev and wildcard proxy configs for local testing (#30556) * fix(proxy): list public team model name in /v1/models (#30588) * fix(proxy): optionally surface public team model name in /v1/models Behind general_settings.use_team_public_model_name (default False). When enabled, /v1/models and /models surface the public team_public_model_name for team-scoped (BYOK) models instead of the internal routing key model_name_{team_id}_{uuid} -- consistent with /v1/model/info and OpenAI-compatible. Off by default so the listing's model ids stay backward-compatible for callers that scripted against the internal name; routing by the internal name is unchanged regardless of the flag. Presentation-layer only: access-group, auth, and routing semantics are unchanged; non-team models are pass-through. * fix(proxy): default team model listings to public names * test(proxy): cover team model listing metadata * test(proxy): cover empty team listing deployments * refactor(proxy): simplify team model listing translation * fix(proxy): resolve public team model name on GET /v1/models/{id} The listing endpoints advertise team_public_model_name, but the retrieve endpoint validated and looked up by the raw id, so a public name 404'd. Resolve the public name back to the internal routing key (scoped to the caller's accessible models so colliding names never cross teams), look up by it, and echo the public name back as the response id. * test(proxy): cover public-name resolution on model retrieve * refactor(proxy): extract team model-name translation into TeamModelNameTranslator Move the team-scoped (BYOK) listing/retrieve name translation out of proxy_server.py into a dedicated common_utils module. Static methods with general_settings injected so the logic is unit-testable without globals and proxy_server.py stays thin. * refactor(proxy): use TeamModelNameTranslator in model_list and model_info * test(proxy): target TeamModelNameTranslator for model-name translation * fix(proxy): type create_model_info_response return as dict[str, object] * fix(proxy): keep internal routing key for team model listing metadata lookup Add listing_entries returning (public response id, internal lookup id) so include_metadata=true resolves fallbacks against the routing key the router indexes by, instead of the translated public name (which never matches). * fix(proxy): build /v1/models metadata from internal key, show public id * test(proxy): cover team listing fallback metadata via internal key * fix(proxy): use builtin dict generics in create_model_info_response (UP006) --------- Co-authored-by: Tushar More <[email protected]> Co-authored-by: Ishaan Jaffer <[email protected]> * ci: drop mypy entirely, standardize type checking on basedpyright (#30648) * ci: drop redundant mypy type-check gate, standardize on basedpyright Type checking ran both mypy (via the pydantic.mypy plugin) and basedpyright. pydantic v2 emits dataclass_transform, so basedpyright understands models natively with no plugin, and its gated rules already cover what the mypy pass caught (no-untyped-def, no-any-return, valid-type, import-not-found all map to basedpyright equivalents). Running both meant two checkers, two budgets, and a plugin only mypy could load. This removes the mypy type-check gate: the lint-mypy/lint-mypy-budget-update Makefile targets, the CI MyPy step, mypy-code-budget.json, the budget-ratchet entry, and the vestigial [tool.mypy] pydantic plugin block (the gating pass used litellm/mypy.ini, which never loaded the plugin). type_check_gate.py is specialized to basedpyright since the mypy parsing path is now unused. mypy stays a dev dependency because the Any-discipline gate (scripts/check_any_discipline.py) imports it as a library to detect Any-typed values; it is no longer run as a type checker. * ci: remove the Any-discipline gate, rely on basedpyright's reportAny The Any-discipline gate (scripts/check_any_discipline.py) was the last consumer of mypy: it imported mypy as a library to detect values whose inferred type contains Any, gated per-file against any-discipline-budget.json. basedpyright already reports the same class of finding through reportAny/reportExplicitAny, which are gated tree-wide in basedpyright-code-budget.json, so the separate gate (and the mypy dependency behind it) is redundant. Removes the gate end to end: check_any_discipline.py and its test, the any-discipline CI job, the lint-any/lint-any-budget-update Makefile targets, any-discipline-budget.json, litellm/mypy.ini, the .mypy_cache_any references, and mypy from the dev dependencies. budget_ratchet_check.py drops the any-discipline entry and the now-unused zero-floor mechanism (rewritten as a comprehension). check_type_discipline.py drops the any-ok suppression token, since # any-ok suppressed only the deleted gate; the 134 now-orphaned # any-ok comments across 14 files are stripped (they never affected basedpyright, which uses # pyright: ignore). uv.lock is intentionally left untouched: uv still considers it consistent with the mypy-removed pyproject (uv lock --check and uv sync --frozen both pass), and a relock bumps 30+ unrelated packages because of the moving exclude-newer window. A future intentional relock will prune the now-unreferenced mypy entry. * build: relock to drop mypy from uv.lock CI's uv 0.10.9 honors the repo's exclude-newer window and correctly flags the lockfile as out of sync once mypy leaves pyproject; my earlier local uv 0.8.17 could not parse exclude-newer and silently passed --check. Relocking with the pinned CI version removes only mypy and its transitive librt, with no other version changes. * feat(guardrails): surface OpenAI moderation violation_categories on guardrail traces (#30659) The OpenAI moderation guardrail (and the ai-platform-moderation guardrail built on it) stamped the whole moderation model response into the guardrail trace as guardrail_response. That blob carries the full category_scores map plus categories and category_applied_input_types, which on OTEL backends that index span attributes (for example ELK, which caps indexed attribute values at 1024 chars) overflows the limit and gets truncated, so the violated categories cannot be reliably searched. Extract the flagged category names from the moderation response and pass them through tracing_detail to add_standard_logging_guardrail_information_to_request_data, mirroring the Bedrock hook. Both the legacy and v2 OTEL integrations already read violation_categories off the standard logging guardrail information and emit it as a short, queryable guardrail_violation_categories attribute, so dashboards can group and filter by violation category without parsing the large guardrail_response blob. Resolves LIT-3801 * fix(proxy): resolve list files credentials from team BYOK deployments (#30495) * fix(proxy): resolve list files credentials from team BYOK deployments GET /v1/files without target_model_names now prefers the team's own deployment (model_info.team_id) over shared global provider keys, so JWT team auth lists files against the correct upstream account. Co-authored-by: Cursor <[email protected]> * fix(proxy): scope list files credential lookup to team allowlist Remove the unrestricted deployment scan that could leak global provider keys to teams without access, normalize all-proxy-models to the team-scoped model list, and fix TID251 violations by using dict instead of Dict/Any. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> * feat(proxy): add --max_requests_before_restart_jitter to stagger worker restarts (#30601) Setting --max_requests_before_restart alone recycles every worker at almost the same time once they have served a similar number of requests, which under sustained load can drop a whole pod's capacity at once roughly every 7-10 days. This exposes a jitter knob that adds a random amount in [0, jitter] to the restart threshold per worker so restarts are staggered. It maps to uvicorn's limit_max_requests_jitter and gunicorn's max_requests_jitter. uvicorn only gained limit_max_requests_jitter in 0.41.0 while litellm still allows uvicorn>=0.33.0, so the uvicorn path feature-detects the parameter via the Config signature and warns instead of crashing on older versions. The flag has no effect without --max_requests_before_restart, so the kwarg is not forwarded in that case and a warning is printed on both the uvicorn and gunicorn paths. Resolves LIT-3774 * fix(health): correct bedrock embedding health checks (#30583) * fix(health): correct bedrock embedding health checks Health checks for Bedrock embedding deployments failed in two ways. A deployment configured without an explicit model_info.mode was probed as chat, so max_tokens was injected and Bedrock embeddings rejected it with 400 "extraneous key [max_tokens]". Separately, stripping the bedrock/ routing prefix dropped the provider, so a cross-region inference-profile id like us.cohere.embed-v4:0 failed downstream with "LLM Provider NOT provided". Resolve the deployment mode from the model cost map (which understands the bedrock/ and us./eu./apac. prefixes) before deciding whether to inject max_tokens, and pin custom_llm_provider to bedrock when stripping the prefix so the bare model id still resolves. ahealth_check now accepts any string mode so the resolved embedding mode routes the probe to the embedding handler. * fix(health): preserve explicit custom_llm_provider on bedrock probe The bedrock prefix-strip pinned custom_llm_provider to bedrock unconditionally, so a deployment that set custom_llm_provider: bedrock_converse had it overwritten at health-check time and the probe hit the Invoke endpoint instead of Converse, a different request format that can report a spurious failure. Only fill in bedrock when the deployment left the provider blank, which still resolves bare cross-region ids like us.cohere.embed-v4:0 while leaving an explicit provider untouched. * test(health): assert resolved mode reaches the ahealth_check probe The existing tests check _resolve_health_check_mode and the params builder in isolation, but nothing verified that _run_model_health_check actually threads the resolved mode into litellm.ahealth_check. Without that, a refactor that probed with model_info.get("mode") again would reintroduce the chat fallback for embedding deployments while every test stayed green. This drives _run_model_health_check with a bedrock embedding deployment and asserts the probe is called with mode=embedding and the embedding params. * fix(health): resolve probe mode once for reasoning_effort and audio_speech The reasoning_effort and audio_speech branches read model_info.mode directly, so an embedding deployment declared without an explicit mode (the case this PR targets) was still treated as chat-like: a configured health_check_reasoning_effort got injected into the embedding probe, which embeddings reject as an unknown field, and an auto-detected audio_speech deployment never had its voice set. Resolve the effective mode once from the cost map and reuse it for the max_tokens, reasoning_effort, and audio_speech decisions so they all agree with the mode threaded into ahealth_check. * test: harden remaining pass-through CI flakes (image-gen spend poll, ruby assistants timeout) (#30685) * test(proxy): poll for image-gen spend instead of a fixed 5s sleep test_key_info_spend_values_image_generation failed once on litellm_internal_staging (pipeline 82282) with "spend did not increase on an identical repeat image call" (assert 0.24966 > 0.24966). The test made the second image call, slept 5s, then read the key's spend once. Response caching is commented out in proxy_server_config.yaml and no sibling test enables it, so the likely cause is async/batched spend logging not having flushed the repeat call's cost within 5s, which the build_and_test job aggravates by running every tests/test_*.py against one shared proxy under pytest -n 4. Poll the key's spend for up to 60s and break as soon as it grows. This removes the timing flake while preserving the canary: if the repeat were genuinely unbilled (for example the proxy response cache being on), spend never grows, the poll times out, and the assertion still fails. * test(pass_through): raise ruby assistants client request_timeout to 600s The streaming assistants example in openai_assistants_passthrough_spec.rb hit Net::ReadTimeout on litellm_internal_staging (pipeline 82280), failing at roughly 125s which is ruby-openai's default request_timeout of 120s. An assistants run with the code_interpreter tool can occasionally take longer than that to stream its first content back through the pass-through. Raise the client's request_timeout to 600s, matching the 600s timeout the Python pass-through e2e tests already use, so a slow-but-healthy streaming run no longer trips the default read timeout. * test(pass_through): harden vertex spendlog poll against transient empty reads (#30683) test_basic_vertex_ai_pass_through_with_spendlog failed intermittently on litellm_internal_staging (pipelines 82155, 82196, 82209, 82230) with "Spend should be greater than before after 120s". Spend logging is async and batched, so the pass-through call's cost sometimes had not landed within the 120s poll window; one run ended on spend_after 0.0 because the final /global/spend/logs read returned nothing and "or 0.0" recorded that as zero spend. Widen the poll window to 240s and skip a transient empty read instead of treating it as 0.0, so a momentary endpoint hiccup on the last poll no longer fails an otherwise-billed call. The spend_after > spend_before assertion is unchanged, so a genuinely unbilled call still fails the test * fix(cost): stop non-string service_tier from silently dropping cost tracking (#30690) completion_cost read service_tier straight from the request optional_params and called service_tier.lower() on it, so a non-string value (dict/int/list, reachable via allowed_openai_params/drop_params) raised AttributeError. _response_cost_calculator swallowed that and returned response_cost=None, so the request's cost was silently lost. The isinstance guard alone is not enough: a surviving dict would crash again downstream in _get_service_tier_cost_key, which also calls .lower(). A request-level service_tier is only meaningful for pricing when it is a concrete billable tier string, so coerce any non-string value to None and defer to the tier the provider reports on the response usage, the same way "auto" already does. Adds a regression test driving a dict service_tier through completion_cost; it raises AttributeError before the fix and prices at the served tier after. * feat(proxy): warn at startup when custom_auth skips common_checks enforcement (#30665) When general_settings.custom_auth is configured but custom_auth_run_common_checks is not set, project/team/org enforcement (budgets, model-level rate limits, and model-access lists) silently does nothing for custom-auth requests, since the centralized common_checks gate returns early for custom auth. Emit a startup warning pointing operators at the flag so the misconfiguration is visible instead of failing silently. * fix(pod_lock): release cron lock by matching async_set_cache JSON encoding (#30600) acquire_lock stores the pod_id through async_set_cache, which JSON-encodes the value, so Redis holds the quoted string "<pod_id>". release_lock's Lua compare-and-delete compared the raw pod_id, so the equality check never matched and the lock was never deleted; it only cleared on TTL expiry. That stalled the spend-update drain whenever the leader pod restarted, letting the litellm_daily_*_spend_update_buffer lists grow unbounded in Redis. Compare against json.dumps(self.pod_id) so the release matches the stored value. The GET+DEL fallback already round-trips through async_get_cache and is unaffected. Co-authored-by: Claude <[email protected]> * ci: run a local fake OpenAI endpoint instead of the shared Railway mock (#30695) Several CI jobs run the proxy against a model whose api_base is a shared "fake OpenAI endpoint" hosted on Railway (exampleopenaiendpoint-production.up.railway.app) so the E2E runs return canned responses without paying for or depending on a live provider. When that single deployment is down, every one of those jobs fails with "404 Application not found" even though nothing in the PR is broken; the whole repo is coupled to the uptime of one free external service. This adds tests/_fake_openai_endpoint_server.py, a small canned-response OpenAI-shaped server (chat, text, embeddings, streaming with usage, and the "429" rate-limit special case), and a reusable start_fake_openai_endpoint CircleCI command that runs it on host port 8190 and waits until healthy. The affected jobs now inject FAKE_OPENAI_API_BASE pointing at the local server, and the example configs they mount resolve api_base from that env var. The intentionally bad fallback URL in proxy_server_config.yaml is left untouched so the fallback test still exercises a failing upstream. Wired into build_and_test, litellm_router_testing, db_migration_disable_update_check, proxy_logging_guardrails_model_info_tests, proxy_spend_accuracy_tests, proxy_multi_instance_tests, proxy_store_model_in_db_tests, and proxy_build_from_pip_tests. * ci(windows): pin uv to Python 3.11 so it ignores the preinstalled 3.14 (#30704) * feat(ui): migrate models page to App Router path route (#30677) * feat(ui): migrate models page to App Router path route Cut the Models + Endpoints page over from the legacy ?page=models switch in (dashboard)/page.tsx to a path route at (dashboard)/models-and-endpoints. Adding the MIGRATED_PAGES entry repoints the sidebar link and redirects old ?page=models bookmarks to /ui/models-and-endpoints. ModelsAndEndpointsView already sourced identity from useAuthorized() and its own data via useModelsInfo(), so the token/keys/modelData/setModelData props were dead; drop them from ModelDashboardProps (and the parent's now-unused setModelData state) to sever the last of the shared-state coupling. * test(ui): scope migration smoke's shell probe to the exact sidebar link The migration smoke used a loose `locator("a", { hasText: "Virtual Keys" })` to assert the dashboard shell rendered. The Models + Endpoints page content itself links to the "Virtual Keys page", so on that route the substring filter matched two anchors and tripped Playwright strict mode. Match the sidebar link by its exact accessible name instead, which resolves to just the nav item. * refactor(ui): remove orphaned pass-through-settings route (#30692) The `page == "pass-through-settings"` arm in (dashboard)/page.tsx is unreachable: it isn't a sidebar item and nothing in the app sets ?page=pass-through-settings. The Pass-Through Endpoints UI lives as a tab inside the Models + Endpoints view (ModelsAndEndpointsView renders PassThroughSettings), so the standalone switch arm is dead code. Remove it, its now-unused import, and the matching enum member in the e2e pages fixture. * fix(cost): stop non-string response service_tier from dropping cost tracking (#30706) completion_cost extracted service_tier from the response object and the usage object without an isinstance guard, so a non-string value (e.g. a dict) flowed straight into _get_service_tier_cost_key and raised AttributeError on service_tier.lower(). completion_cost re-raises, so the request's cost was lost. PR #30690 fixed only the request-level optional_params path. This extends the same guard to the response and usage paths by normalizing each extracted value: a non-string tier (and the routing-only "auto" sentinel) is not billable, so it coerces to None and pricing defers to the next concrete tier the provider served, falling back to standard pricing when none is present. Adds two regression tests driving a dict service_tier through completion_cost, one on the response object (defers to the served usage tier) and one on the usage object (prices at standard); both raise AttributeError before the fix. * feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) * feat(triage): auto-close stale PRs with Greptile score <4/5 Adds .github/scripts/close_low_quality_prs.py and a daily workflow that closes PRs which: - are open for at least 7 days, and - carry a most-recent greptile-apps review with Confidence Score <4/5, - and are not drafts or opt-out-labeled ('do not close', 'wip', etc.). Each closure posts an explanatory comment telling the contributor how to bring the PR back (rebase, re-request greptile, reopen at 4+/5). The 4/5 bar is already documented in the PR template (.github/pull_request_template.md), so this just enforces it. Tested with a dry run against the live BerriAI/litellm backlog of 1000 open PRs: 100 candidates identified, 598 PRs pass the bar (4+/5), 186 are too young, 97 are drafts, 19 lack any Greptile review and are left alone. Workflow defaults to closing 25 PRs/run as a safety net and supports workflow_dispatch with overrides (close=false for a dry run, custom min_age_days/min_score/limit). 18 unit tests cover score extraction (HTML/markdown/plain text, login variants, multi-review picks latest) and per-PR evaluation (drafts, opt-out labels, age, missing/passing/failing scores). Co-authored-by: Mateo Wang <[email protected]> * docs(templates): require expected/actual + QA proof for external contributions PR template: - Make the rubric explicit at the top: link an issue, OR provide a clear problem description + expected vs. actual + visual QA proof. - Add dedicated sections for each piece so the bot has a deterministic shape to read. - Keep the existing 'Linear ticket' section for internal contributors (they're exempt from the auto-triage rubric). Bug report template: - Split 'What happened?' into 'Actual behavior' + 'Expected behavior'. - Make logs/screenshot a required textarea. - Warning banner at the top tells external contributors that incomplete reports will be auto-closed (with re-evaluation on reopen). Feature request template: - Require a concrete use case + example in the motivation field, not just a one-liner pitch. - Same auto-triage warning banner. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): Agent Shin LLM-as-judge for external PRs and issues Adds a new triage flow that evaluates external pull requests and issues against the project's contribution rubric and, when configured to do so, auto-closes non-conforming ones with an explanatory comment. Contributors can update + reopen to be re-evaluated. Scope: - Internal BerriAI contributors (author_association OWNER/MEMBER/COLLABORATOR) and bot accounts are skipped entirely. - 'Fixes #1234' / 'Resolves https://github.com/.../issues/N' in the PR body short-circuits to PASS without burning LLM tokens. - LLM judge returns structured JSON (verdict, missing[], explanation); parser tolerates markdown fences and embedded JSON. - LLM errors NEVER close PRs/issues — failure surfaces as 'skip-llm-error'. Safety: - pull_request_target / issues triggers are FORCED dry-run in the workflow; only manual workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true) takes destructive action. - Default mode writes verdicts to GITHUB_STEP_SUMMARY only — no public comments until the team flips the AGENT_SHIN_ENABLED repo variable. - LLM uses an OpenAI-compatible endpoint (model and base URL configurable via repo variables; key via OPENAI_API_KEY secret). Files: - .github/scripts/triage_with_llm.py - judge orchestrator + CLI - .github/workflows/triage_pr_with_llm.yml - .github/workflows/triage_issue_with_llm.yml - tests/test_litellm/test_github_triage_with_llm.py - 33 unit tests End-to-end validated against four real PRs (#28117 internal collaborator, #28108 bot, #28129 'Fixes #28128', #28116 no linked issue) and issue #28132 with a stubbed LLM judge: each path produces the expected action. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): scope Greptile auto-closer to external contributors + dry-run by default - close_low_quality_prs.py now filters by GitHub author_association via the REST API: PRs from OWNER / MEMBER / COLLABORATOR (and bot accounts) are skipped with a new 'skip-internal' summary bucket. - close_low_quality_prs.yml now defaults workflow_dispatch close=false, and ignores 'close=true' unless the new repo variable AGENT_SHIN_ENABLED is set to 'true'. Scheduled runs are dry-run only until the team flips that switch. - Updated unit tests: one new test asserting internal authors are skipped, and an autouse fixture treats unspecified test PRs as external so the rest of the suite still exercises the close path. Co-authored-by: Mateo Wang <[email protected]> * fix(workflows): scheduled cron closes PRs; safe --close strip in triage Co-authored-by: Yassin Kortam <[email protected]> * fix(triage): scheduled cron stays dry-run; dedent prompts before interpolation - close_low_quality_prs.yml: only workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true) actually closes PRs. Scheduled runs are always dry-run, matching the safety invariant documented for triage_pr/issue. - triage_with_llm.py: textwrap.dedent on an f-string with multi-line interpolated bodies fails because the body's 2nd+ lines start at column 0, making the common-indent zero. Dedent the static template first, then .format() the title/body in. Co-authored-by: Yassin Kortam <[email protected]> * Fix bugs in auto-close PR triage scripts - close_low_quality_prs.py: Treat author_association API lookup failures as internal (fail-safe) so transient errors don't cause internal contributors' PRs to be auto-closed. - triage_with_llm.py: Update summary heading from 'Would post comment:' to 'Posted comment:' since this branch only runs after the comment has already been posted. Co-authored-by: Yassin Kortam <[email protected]> * feat(triage): default Agent Shin to gpt-5.4-mini with reasoning_effort=none - Bump DEFAULT_MODEL from gpt-4o-mini to gpt-5.4-mini (more modern; 4M total context window per OpenAI catalog, JSON-schema response format, function calling all supported). - For gpt-5.x family models, pass reasoning_effort="none" via extra_body. gpt-5.x rejects temperature != 1 unless reasoning_effort is explicitly "none"; setting it lets us keep temperature=0 for deterministic JSON rubric judgments. extra_body works across openai SDK versions regardless of whether they natively type the kwarg. - For non-gpt5 overrides (TRIAGE_MODEL=gpt-4o-mini etc.), reasoning_effort is not sent. - 4 new unit tests cover: gpt-5.4-mini -> reasoning_effort=none, capitalized/dated gpt-5 variants -> reasoning_effort=none, gpt-4o-mini -> no extra_body, base_url passthrough. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — drop dead gh_json and fix --optout-label append-with-default - Removed the unused gh_json helper (bugbot low-severity dead code). - Replaced argparse `action="append", default=[...]` with default=None + DEFAULT_OPTOUT_LABELS fallback. The mutable-default + append combo silently APPENDS to the canonical defaults instead of replacing them, so --optout-label could not actually scope the opt-out list. - Added tests covering both the canonical default and the flag-replaces-defaults behavior. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — tighten linked-issue regex, fail-safe author_association, fix empty TRIAGE_MODEL Three independent bugbot findings against triage_with_llm.py: 1. LINKED_ISSUE_PATTERN included weak keywords (`see`, `ref`, `addresses`) so casual mentions like "See #1234 for context" were short-circuited to pass-linked-issue without ever calling the LLM — contradicting the prompt's own "a bare issue number without a closing keyword counts only if it's clearly the related issue (not a passing mention)" rubric. Limit the regex to GitHub's documented PR-closing keywords (fixes/fix/fixed/closes/close/closed/resolves/resolve/resolved). 2. is_internal_contributor() treated an empty/missing author_association as external (eligible for the destructive close path), while the sibling is_external_pr_author() in close_low_quality_prs.py fail-safes the same case as internal. Align the two so a partial/unknown GitHub response can never make a PR eligible for auto-close. 3. argparse `default=os.environ.get("TRIAGE_MODEL", DEFAULT_MODEL)` returns the empty string when GitHub Actions exposes an unset repo variable as an empty-string env var (the optional vars.TRIAGE_MODEL case in the workflow). Use `os.environ.get(...) or DEFAULT_MODEL` so empty -> default, matching the existing OPENAI_BASE_URL pattern. Tests: - Casual mentions now must fall through to the LLM (parametrized); added an orchestration test ensuring "See #1234" reaches the judge. - Empty/missing author_association now fails safe (parametrized). - Empty TRIAGE_MODEL env var falls back to DEFAULT_MODEL; explicit TRIAGE_MODEL is still honored. Co-authored-by: Mateo Wang <[email protected]> * fix(workflows): bugbot — gate Agent Shin --close on '= true' not '!= false' The PR and issue Agent Shin workflows gated the destructive --close flag with [ "${DISPATCH_CLOSE:-false}" != "false" ]. That pattern treats anything other than the literal string "false" as enabling closure — "True", "yes", "1", typos, accidental whitespace, etc. The workflow_dispatch input UI is a 'true'/'false' choice dropdown so the form is constrained, but the API (`gh workflow run -f close=...`) accepts any string, and a CI cron / external invoker passing a non-canonical truthy value would have silently enabled real contributor PR closures. Mirror the sibling Greptile closer's [ "${CLOSE_FLAG}" = "true" ] pattern: only the EXACT string "true" enables --close; every other value (including the unset/empty default) resolves to dry-run. This is the fail-safe philosophy applied everywhere else in this PR. Added tests/test_litellm/test_github_triage_workflows.py with two parametrized invariants: 1. The destructive gate uses '= "true"' for its env-var comparison (either bare '${ENV}' or '${ENV:-false}' form accepted), and never the fail-open '!= "false"' pattern. 2. Every destructive gate is also gated on AGENT_SHIN_ENABLED being "true" — either by entering the close branch on '=' or by bailing out early on '!=' — so flipping the repo variable off is a true kill switch regardless of per-run inputs. Manually verified the test fails on the buggy '!= "false"' pattern and passes on the fix, so it would have caught the regression at PR time. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): close any PR (incl. drafts, any age); add @agent-shin reconsider flow Follow-up to PR #28117. Three behavior changes + one new workflow, addressing the team's concerns on the original review: 1) Apply auto-close to ALL open PRs, not just those over a week old. - close_low_quality_prs.py: --min-age-days default flipped from 7 to 0. The flag is preserved as an opt-in safety net for one-off backfill runs that want to spare very-young PRs, but the daily scheduled sweep now closes external-author PRs as soon as Greptile scores them <4/5. - close_low_quality_prs.yml: workflow_dispatch input default also flipped to 0; doc comments updated. 2) Apply auto-close to draft PRs too. - close_low_quality_prs.py: removed the skip-draft branch in evaluate_pr. Drafts are NOT a free pass — the team's intent is 'open PR count == PRs internal collaborators need to action on', so a draft Greptile scored 2/5 still belongs in the closed bucket. Authors who genuinely need a long-lived draft can attach the 'wip' opt-out label, which is unchanged. - The 'skip-draft' action is gone; the 'wip' label still skips. 3) Address the 'OSS contributors cannot reopen a bot-closed PR' wrinkle. GitHub does NOT let an external (non-write-access) contributor reopen a PR that was closed by a bot or maintainer (long-standing limitation). The original PR's close-comments told contributors to 'Reopen the PR — I'll re-evaluate automatically', which is broken for the very audience this triage targets. Two changes: a) Reword every close-comment (Greptile sweep + Agent Shin PR close + Agent Shin issue close + PR template) to recommend: - Open a new PR with the updated branch (primary path). - Or comment '@agent-shin reconsider' on the closed PR for a re-evaluation that, on pass, reopens the PR via the bot's GH_TOKEN write access. b) Add the @agent-shin reconsider workflow: - .github/workflows/triage_reconsider.yml: new 'issue_comment'-triggered workflow. Authorizes only the PR/issue author or an internal collaborator (OWNER/MEMBER/COLLABORATOR), gated via a step output so unauthorized commenters never reach the destructive steps. Globally gated on AGENT_SHIN_ENABLED='true' (positive form, matching the test_github_triage_workflows guardrail patterns). - triage_with_llm.py: --reconsider mode. On a closed PR/issue, re-runs the LLM judge (or linked-issue regex short-circuit) and: - on pass: reopens via reopen_pr/reopen_issue + posts a 'Re-evaluated and reopened' comment. - on fail: leaves closed and posts a 'still missing X' comment so the contributor can iterate again. Reconsider-on-open is a no-op ('skip-not-closed'). Internal-author + bot-account skips still take priority over reconsider. 4) Greptile-on-closed-PRs question: the team asked whether Greptile can re-review a closed PR. Greptile's docs don't address this and we shouldn't promise behavior we can't verify, so the new close-comment wording does NOT instruct contributors to 're-request greptile on the closed PR'. Instead it points them at the new-PR path (which Greptile definitely reviews) or the @agent-shin reconsider trigger (which re-runs the LiteLLM-side rubric judge, not Greptile). Tests: 93 passing (was 59). - test_github_close_low_quality_prs.py: replaced 'skip drafts' test with 'closes drafts when score is low' + 'closes brand-new PR when min_age=0' + 'no skip when min_age=0'. The 'skip too young' assertion is preserved as opt-in. - test_github_triage_with_llm.py: 6 new TestTriageOrchestration cases for reconsider mode (skip-not-closed on open, reopen on pass, still-failing comment on fail, linked-issue short-circuit reopen, skip internal author in reconsider, reopen-issue on pass) + a new TestCloseCommentText class that pins the user-facing 'open a new PR' + '@agent-shin reconsider' wording. - test_github_triage_workflows.py: added triage_reconsider.yml to the destructive-gate guardrail table; AGENT_SHIN_ENABLED is its own destructive gate (no separate per-run flag needed). Co-authored-by: Mateo Wang <[email protected]> * test(triage): pin safe behavior for curly braces in PR/issue title+body Adds regression tests covering the bugbot high-severity finding that str.format() would crash on user-supplied content containing { or }. Empirically str.format() does NOT re-parse interpolated values — only the template literal is scanned for replacement fields — so the bug does not exist in the current code, but pinning the safe behavior prevents a future templating change from silently reintroducing it. Also pins the dedented prompt shape (no leading 8-space indentation on template lines) so a future change to the build_*_prompt functions can't silently regress the LLM judge prompt format on multi-line bodies. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — reconsider dry-run + bot-closed guard + rate limit Address three Greptile/veria-ai concerns on the @agent-shin reconsider flow: 1. **Reconsider had no dry-run path.** The previous reconsider mode ignored `--close` and always posted comments + reopened on a pass. A local operator running `python triage_with_llm.py --reconsider --pr N` would silently take destructive GitHub actions with no way to preview. Reconsider now honors `close=False` the same way regular triage does and returns `would-reopen` / `would-reconsider-still-failing` for step-summary rendering. 2. **Reconsider could reopen maintainer-closed PRs/issues** (Medium security finding from veria-ai). The workflow only checked that the commenter was authorized — it did NOT check that the most recent close was performed by Agent Shin. A contributor could comment `@agent-shin reconsider` on a PR a maintainer closed for non-rubric reasons (duplicate, security report, design rejection) and have the bot reopen it. Add `was_closed_by_agent_shin()` which inspects the issue events API for the most recent `closed` actor and only permits reopen when that actor matches the configured bot login (default `github-actions[bot]`, overridable via env). Fail-closed on missing events. 3. **No rate-limiting on the reconsider trigger.** Every `@agent-shin reconsider` comment burns CI minutes + an OpenAI API call. Add a 10-minute cooldown via `seconds_since_last_reconsider_verdict()` which greps the issue's comment list for the bot's own verdict marker (``). Inside the window the triage returns `skip-rate-limited` and the LLM never runs. Workflow update: - `triage_reconsider.yml` now passes `--close` only when `AGENT_SHIN_ENABLED=true`, matching the pattern of `triage_pr_with_llm.yml`. The script runs in both states so the verdict still appears in the step summary for QA. Tests: - Add 5 reconsider safety tests: dry-run for pass / fail / linked-issue short-circuit, bot-closed-guard refusal on maintainer close, rate-limit refusal inside the cooldown window, and cooldown-elapsed acceptance. - Add unit tests for `was_closed_by_agent_shin` (bot / maintainer / missing actor / env-override) and `seconds_since_last_reconsider_verdict` (no marker / multiple markers / non-bot comment with marker / bot comment without marker). - Pin the `` marker in both reopen and still-failing comments — dropping it would silently break the cooldown. Existing reconsider tests updated to pass `close=True` (the production path now) + stub the new guards via `_stub_reconsider_guards`. 112 tests pass (was 93). Co-authored-by: Mateo Wang <[email protected]> * feat(triage): 1-day grace period before close + SwiftWinds immediate-close bypass - Add a 24-hour grace window between the first low-quality detection and the actual auto-close. The first detection posts a warning comment that explicitly says "You have 1 day to address this before this PR is auto-closed" and points the contributor at: * `@agent-shin reconsider` to request another look (and re-open) * `@greptileai` to request a fresh Greptile review — works even after the PR is closed - Both `triage_with_llm.py` (LLM judge) and `close_low_quality_prs.py` (Greptile-score closer) share the same `` HTML marker so a warning posted by either path is recognized by both. - Add IMMEDIATE_CLOSE_LOGINS = {swiftwinds} to bypass BOTH the grace period AND the dry-run / AGENT_SHIN_ENABLED gating. SwiftWinds is the user's personal account (no push permissions to litellm) used to dogfood the bot; user explicitly asked: "For SwiftWinds, just close immediately. Faster iteration that way." - Update the standard close comments to mention that `@greptileai` works even after the PR is closed. - Add 23 new tests covering: warn-grace on first detection, skip during grace window, close after grace expires, SwiftWinds bypass (case insensitive, with close=False, no random-login false positives), the grace-warning text invariants, and the SwiftWinds entry in the IMMEDIATE_CLOSE_LOGINS constant. Co-authored-by: Mateo Wang <[email protected]> * fix: skip grace-period text in close comment for IMMEDIATE_CLOSE_LOGINS For PRs from IMMEDIATE_CLOSE_LOGINS (e.g. swiftwinds), evaluate_pr returns 'close' immediately without ever posting a grace warning, so the close comment should not reference a 1-day grace period. Make close_pr take a grace_period_elapsed flag, default True, and pass False from the main loop when the close path was the immediate-close branch. Co-authored-by: Yassin Kortam <[email protected]> * fix(close-low-quality-prs): report actual closes in dry-run summary IMMEDIATE_CLOSE_LOGINS PRs are closed even when the global --close flag is not set, but the summary used the global dry-run flag to choose between 'would close' and 'closed'. Split the count so operators can see both actual closures and dry-run would-be closures. Co-authored-by: Yassin Kortam <[email protected]> * chore(triage): vendor Agent Shin (#28117) onto demo branch Brings the Agent Shin OSS-triage scripts, workflows, issue/PR templates, and tests from PR #28117 onto this branch so the new review-gate feature and its end-to-end demo are self-contained and runnable in CI. https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ * feat(triage): add "ready for review" label lifecycle to Agent Shin Adds review_gate(), a state machine that keeps a `ready for review` label in sync with whether an external PR clears BOTH gates — the LLM rubric and Greptile's most recent confidence score: - pass (untagged) -> add label + "ready for review" / "all clear" comment - pass (already tagged) -> no-op (idempotent across re-runs) - regress (Greptile < 4/5 or QA proof removed) -> remove label + "what's missing" comment, PR stays open - recover after a regression -> "all clear again" comment + re-add the label - fail & untagged, < 24h old -> one-time "what's missing" notice (grace window) - fail & untagged, > 24h old -> close + comment (reopen via @agent-shin reconsider) The label itself is the persisted state, so comments fire only on transitions (never on every scheduled run). All side effects are gated behind --close, so the dry-run contract matches the existing triage flow. Lifecycle comments use hidden HTML markers and deliberately avoid the auto-close marker so they never trip the reconsider provenance check. Relocates the shared Greptile helpers (extract_greptile_score, SCORE_PATTERN, GREPTILE_BOT_LOGINS, parse_iso8601) into triage_with_llm.py so the daily sweep and the review gate read the score through one implementation, and adds the review_gate.yml workflow (dry-run unless AGENT_SHIN_ENABLED=true) plus 18 unit tests covering every branch and a full pass->regress->recover cycle. https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ * Port review-gate feature from #28758 onto #28147 triage scripts Adds the "ready for review" label lifecycle (originally PR #28758) on top of #28147's refactored triage_with_llm.py. The original commit was authored against an older snapshot of #28117 and could not be applied cleanly, so the additions were re-applied surgically: - New constants: READY_FOR_REVIEW_LABEL, DEFAULT_GRACE_DAYS, DEFAULT_MIN_GREPTILE_SCORE, READY/REGRESSED/WITHIN_GRACE markers, GREPTILE_BOT_LOGINS, SCORE_PATTERN, AGENT_SHIN_AUTO_CLOSE_MARKER. - New helpers: add_label, remove_label, extract_greptile_score, parse_iso8601 (the latter two mirrored from close_low_quality_prs.py so the daily sweep and the review gate read the score through the same logic). - New comment formatters: format_ready_for_review_comment, format_all_clear_comment, format_regression_comment, format_within_grace_comment. - New entry point: review_gate() implementing the pass/regress/recover state machine, with the label itself acting as persisted state so transition comments fire only on actual transitions. - main() learns --review-gate, --grace-days, --min-greptile-score and dispatches to review_gate() when the flag is set. Verified via tests/test_litellm/test_github_review_gate.py (18 tests) and the existing triage suites (144 more) — all 162 pass. Co-Authored-By: Claude Opus 4.7 <[email protected]> * agent_shin: extract shared constants/helpers; cover review_gate.yml in guardrail tests Bug 1: `triage_with_llm.py` and `close_low_quality_prs.py` each defined their own copies of `extract_greptile_score`, `parse_iso8601`, `GREPTILE_BOT_LOGINS`, `SCORE_PATTERN`, `GRACE_COMMENT_MARKER`, `GRACE_PERIOD_SECONDS`, `IMMEDIATE_CLOSE_LOGINS`, and `AGENT_SHIN_DEFAULT_BOT_LOGIN`. The comments explicitly said the two copies had to stay in sync, but nothing enforced it. A future change to one (e.g. extending `SCORE_PATTERN` for a new Greptile output format) would silently diverge from the other and the daily sweep and the LLM judge would disagree on which PRs have low scores. Extract these to `.github/scripts/agent_shin_shared.py` and re-export them from each script so the existing test attribute access (`triage_module.GRACE_COMMENT_MARKER`, etc.) keeps working without any test changes. Bug 2: `review_gate.yml` is a destructive workflow (close PRs, add/remove labels, post comments) with the same gating philosophy as the others (`AGENT_SHIN_ENABLED = "true"` + a per-run `CLOSE_FLAG = "true"`), but it was missing from `DESTRUCTIVE_GATE_ENV` in the guardrail tests. Add it so a future regression (e.g. flipping to `!= "false"`) is caught by the same parameterized invariants as every other workflow. Co-authored-by: Yassin Kortam <[email protected]> * agent_shin: fix bug bundle (gated LLM key, author-filtered marker dedup, dedup gh/grace helpers) Co-authored-by: Yassin Kortam <[email protected]> * agent_shin: fix review_gate close-after-regression and case-insensitive label match Co-authored-by: Yassin Kortam <[email protected]> * feat(triage): add one-shot 7-day heads-up sweep for Agent Shin rollout Adds a rollout-day workflow that comments on every open external PR/issue that the new triage bot WOULD auto-close, giving contributors 7 days to fix their description before any destructive action runs. Why now: merging this PR enables Agent Shin in dry-run. The follow-up "enact" PR (next Monday) flips the destructive paths on. Without this heads-up, contributors would get a close-comment on day 8 with no prior warning. The heads-up names the cutoff date, lists the rubric, calls out each PR/issue's specific missing pieces, and explains the recovery paths (@agent-shin reconsider for PRs, edit + reopen for issues). Files - .github/scripts/_agent_shin_actions.py — thin maybe_post_comment / maybe_close_* / maybe_add_label / etc. wrappers. Each is a single `if dry_run: log; return; else: call_through()` so a dry-run preview differs from the real run in exactly one call site per mutation. The call-through goes via `triage_with_llm.<name>` (module-qualified) so monkeypatching the underlying function in tests is reflected here. - .github/scripts/triage_rollout_heads_up.py — the sweep. Iterates every open PR + issue via `gh pr list` / `gh issue list`, runs the future rubric (review_gate for PRs, triage(kind="issue") for issues), and posts the heads-up on any item that would be auto-closed. Idempotent via a `` marker. Defaults to dry- run; --close opts in to real posts. --close-on overrides the cutoff date (defaults to today + 7 days). - .github/workflows/triage_rollout_heads_up.yml — one-shot workflow. Triggers on push to litellm_internal_staging filtered to the script path (fires on rollout merge) plus workflow_dispatch with a dry_run input that defaults to "true" for safe manual re-runs. - tests/test_litellm/test_triage_rollout_heads_up.py — 28 unit tests covering: the dry-run wrappers (each maybe_* gates correctly), the _would_be_closed predicate for PR vs. issue results, the comment formatter (cutoff/rubric/marker/recovery wording), per-item dispatch (skip-not-open, skip-internal-author, skip-already-notified, skip-passing, would-post/posted), and the sweep loop end-to-end. Local preview (no GitHub mutations): python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm Real run (what the workflow does): python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm --close TODO: replace the placeholder ROLLOUT_BLOG_URL with the canonical docs URL once the litellm-docs PR ships. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: gate reconsider workflow OPENAI_API_KEY + remove dead actions wrappers - Mirror sibling Agent Shin workflows by only exposing OPENAI_API_KEY in triage_reconsider.yml when vars.AGENT_SHIN_ENABLED == 'true'. Previously the secret was unconditionally exposed, so any PR/issue author could trigger paid LLM calls by commenting '@agent-shin reconsider' even while the bot was supposed to be in dry-run. - Remove the six unused dry-run wrappers (maybe_close_pr, maybe_close_issue, maybe_reopen_pr, maybe_reopen_issue, maybe_add_label, maybe_remove_label) from _agent_shin_actions.py — only maybe_post_comment is used by rollout scripts. Drop the associated tests that exercised the now-removed functions. Co-authored-by: Yassin Kortam <[email protected]> * fix: address triage script edge cases - triage_rollout_heads_up.py: replace %-d strftime specifier (GNU-only) with portable day formatting so the script doesn't crash on Windows. - close_low_quality_prs.py: skip malformed JSON lines in fetch_pr_comments instead of letting one bad line abort the daily sweep, matching the pattern in triage_with_llm._iter_paginated_json. - triage_with_llm.py: move has_linked_issue short-circuit before build_pr_prompt to avoid unnecessary prompt construction on PRs that link an issue. Co-authored-by: Yassin Kortam <[email protected]> * fix(scripts): per-PR error isolation and limit grace warnings in close_low_quality_prs - Wrap per-PR processing in try/except so a transient GitHub API failure on one PR no longer aborts the entire daily sweep (mirrors the pattern already used in triage_rollout_heads_up.py). - Have --limit bound *all* destructive write actions (closures and grace warnings combined), not just closures. Prevents a backlog of newly failing PRs from flooding contributors with comments in a single run. Co-authored-by: Yassin Kortam <[email protected]> * fix(agent-shin): remove 1000-PR cap on bulk sweeps; sweep entire backlog Both bulk-sweep scripts hardcoded `gh {pr,issue} list --limit 1000`, and gh lists newest-first — so the OLDEST ~900 PRs and ~380 issues were silently dropped. That's exactly the stale backlog the daily closer and one-shot rollout heads-up exist to catch. Extract a single `list_open_items(kind, *, repo, fields)` helper into `agent_shin_shared.py` with `GH_LIST_ALL_LIMIT = 100_000` — a ceiling far above any realistic open backlog so gh paginates until the queue is exhausted. `fetch_open_prs` and `_list_open_numbers` both delegate to it, so the limit lives in exactly one place going forward. Verified live against BerriAI/litellm: - `fetch_open_prs` -> 1981 PRs (was 1000) - `_list_open_numbers(issue)` -> 1382 issues (was 1000) - `_list_open_numbers(pr)` -> 1981 PRs (was 1000) Adds 7 regression tests asserting the new limit is passed, the dedicated `gh {pr,issue} list` command + fields are used per kind, bad kind raises ValueError, and both callers delegate to the shared helper. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(agent-shin): require non-mocked end-to-end QA proof for PR pass The PR rubric previously passed any PR with a linked issue, regardless of whether it showed the fix actually working. Sample spot-check found 21/25 recent external PRs passing, including ones that linked an issue but provided zero QA evidence. Tighten the rubric so a pass now requires BOTH: (1) CONTEXT — a linked issue OR a clear problem description with expected-vs-actual behavior. (2) END-TO-END QA PROOF — at least one of: (a) screenshot(s) of the fix working, (b) screen recording / video, (c) specific commands actually run, paired with their real output, against the real system. Mocked unit tests, generic 'I tested it' claims, 'all tests pass' without output, and the linked issue itself are explicitly excluded from QA proof. Also add 'qa_proof_type' to the JSON schema so the per-PR report surfaces which kind of proof (or 'none') the judge saw. Re-sample on the same 25 recent external PRs shifts the verdict distribution from 21 pass / 4 fail to 4 pass / 21 fail, with zero prior-fails now passing — the stricter rule catches PRs that ship only with unit-test claims and no real integration evidence. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(agent-shin): link blog explainer from every action-required bot comment Adds "What's this and why am I getting it?" links to docs.litellm.ai/blog/ agent-shin-triage from the four comments contributors actually read when something went wrong: PR close, PR grace warning, issue close, issue grace warning. PR comments also link the rubric section directly from the QA-proof bullet so contributors can self-serve "what counts as proof" without pinging a maintainer. Pins the new guarantees in tests: blog link must appear in all four comments, and the PR close comment must continue to flag mocked-dependency unit tests as insufficient proof. The linked blog post is in BerriAI/litellm-docs PR #240; the URL will 404 until that lands. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review_gate): raise sweep limit from 1000 to 100000 to match GH_LIST_ALL_LIMIT gh lists newest-first, so capping at 1000 silently drops the oldest open PRs — exactly the stale ones the daily sweep is meant to reconcile. Use the same ceiling as agent_shin_shared.GH_LIST_ALL_LIMIT so the workflow sees the entire backlog. Co-authored-by: Yassin Kortam <[email protected]> * Fix three Agent Shin triage edge cases - review_gate: expire the regression-marker short-circuit after grace_days so PRs that were regressed and then abandoned can eventually be closed. - review_gate: when the rubric short-circuits to pass via the linked-issue regex but Greptile drags the PR below the bar, replace the synthetic 'LLM was not called' explanation with the real Greptile shortfall so regression / close comments are not misleading. - triage_rollout_heads_up._comments_have_marker: drop the unused 'kind' parameter and filter by bot author so a contributor quoting the heads-up via 'Quote reply' cannot trick the idempotency check, matching the pattern in triage_with_llm._has_marker. Co-authored-by: Yassin Kortam <[email protected]> * fix: pass min_greptile_score through to ready-for-review comment text Co-authored-by: Yassin Kortam <[email protected]> * feat(agent-shin): warmer triage comments — bullet-train emoji, 'what you got right' section, softer 'park this for later' framing User feedback on the auto-triage comments contributors will see: 1. Tone — the previous 'You have 1 day to address this before this PR is auto-closed' framing reads as an ultimatum. Replace with: 'If the description isn't updated in the next 1 day, I'll auto-close this PR. That's not us saying we don't care about the change — we want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later," not a rejection. Take your time.' 2. Positive feedback — the previous comments only listed what was missing. Now every close + grace-warning comment opens with a 'What you got right:' section rendered from the judge's per-field flags. Contributors see a checkmark for everything they got right (linked issue, problem description, expected/actual, QA proof for PRs; runnable repro, screenshot/log, expected/actual, motivation+example for issues) before the gaps. The block is omitted entirely when nothing is present so we never render 'What you got right: (nothing).' 3. Reconsider trigger — the previous grace warning told contributors to comment '@agent-shin reconsider' during the grace window. They don't need to — the bot re-checks on every sweep. The new copy says 'just update the description, no need to ping me' for the grace path, and reserves '@agent-shin reconsider' for the post-close recovery path. 4. Bullet-train emoji — replace 👋 with 🚄 (Shinkansen, the symbol of Agent Shin) across every action-required comment: PR close, PR grace warning, issue close, issue grace warning, within-grace, Greptile- closer grace warning, rollout heads-up. Pinned in tests so a future refactor can't silently revert. 5. Greptile-post-close — the @greptileai bullet now explicitly says 'a low Greptile score isn't a blocker either,' since the previous copy buried the fact that @greptileai works after auto-close. Comment templates updated: format_pr_close_comment, format_issue_close_comment, format_grace_warning_pr_comment, format_grace_warning_issue_comment, format_within_grace_comment (triage_with_llm.py); format_grace_warning_comment (close_low_quality_prs.py); format_heads_up_comment header (triage_rollout_heads_up.py). New helpers: _format_present_for_pr / _format_present_for_issue / _format_present_block, driven off the existing per-field flags the LLM judge already emits — no prompt change needed. New tests pin: bullet-train emoji in every action-required comment; 'What you got right' appears with ✅ bullets when fields are present; the block is omitted when no fields are present; 'park this for later' / 'not a rejection' softer framing; grace warnings tell the contributor 'no need to ping' during the grace window (reconsider is the post-close path only). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(agent-shin): gate triage on a dogfood allowlist Add ALLOWLIST_LOGINS to agent_shin_shared so Agent Shin only acts on the named accounts while the set is non-empty. mateo-berri and SwiftWinds are allowlisted for the dogfood rollout; everyone else is skipped with skip-not-allowlisted across all four entrypoints (triage, review gate, the daily low-quality sweep, and the rollout heads-up). For an allowlisted author the usual internal/external classification is bypassed, so a maintainer's own org account still gets triaged during testing. Emptying the set lifts the restriction and restores full triage for the public rollout. The gate is dependency-injected via an `allowlist` parameter defaulting to the constant, so the internal/external-skip paths stay testable. * feat(agent-shin): tighten QA-proof and issue rubrics, ack reconsider with reactions Reorder the end-to-end QA proof options to video, then screenshots, then exact commands with their real output across the PR template, the LLM judge prompts, and every contributor-facing comment, and spell out that mocked or stubbed runs (including pytest on the repo's own unit tests, which mock the provider, DB, and network) never count as proof. QA proof is now required of all contributors, not just external ones. Tighten the issue bug-report rubric to require end-to-end evidence of the bug (the "before" half: a video, screenshot, or command paired with real output) plus expected vs. actual behavior, drop the bias toward PASS, and collapse the separate has_repro/has_proof flags into a single has_repro signal. Standardize the bullet-train emoji and strip em dashes from the bot's public-facing messages, and route issue recovery through @agent-shin reconsider since GitHub doesn't let OSS authors reopen an issue a bot closed. Acknowledge an @agent-shin reconsider the moment it's accepted with an eyes reaction and a thumbs-up once the run finishes, both gated on AGENT_SHIN_ENABLED so dry-run leaves no trace. * fix(agent-shin): shorten auto-close grace to 2 hours and drop the instant-close bypass Two dogfooding changes to the Agent Shin grace window. First, the warn-then-close grace (GRACE_PERIOD_SECONDS) drops from a day to 2 hours so the "fix it before it closes" loop can be exercised in one sitting; the constant carries a note to bump it back up for the public rollout. Second, remove IMMEDIATE_CLOSE_LOGINS entirely. SwiftWinds (the external dogfood account) used to skip the grace window and close on first detection, which also meant closing real PRs even during a scheduled dry run because the per-PR override flipped dry_run off. It now follows the same warn-then-close path as every other author, so a low-quality PR is warned first and only closed once the 2-hour window elapses. This also closes the Greptile finding that the sweep could mutate real PRs while AGENT_SHIN_ENABLED was still off. The review gate's separate age-based grace (DEFAULT_GRACE_DAYS) is left unchanged. Regression tests pin that SwiftWinds now warns-grace instead of closing instantly, and that a dry-run sweep over a closeable PR reports "would close" without making any GitHub mutation. * fix(agent-shin): gate reconsider reopen on an Agent Shin close marker was_closed_by_agent_shin only checked that the most recent close actor was the bot identity. That identity defaults to github-actions[bot], which is shared by every workflow in the repo (stale/duplicate sweeps included), so a contributor could @agent-shin reconsider an item another workflow closed and, if the description passed the rubric, get it reopened even though Agent Shin was never the closer. Require a second, Agent-Shin-specific signal alongside the actor check: an auto-close comment stamped with a hidden AGENT_SHIN_CLOSE_MARKER. Both close paths (the grace-period close and the review-gate close) flow through format_pr_close_comment / format_issue_close_comment, so stamping the marker there covers every real close while leaving the grace warnings unmarked. The guard stays fail-closed: no marker, no reopen. This also replaces the unused AGENT_SHIN_AUTO_CLOSE_MARKER constant (a visible phrase the guard never consulted) with the hidden marker the guard now relies on. * fix(agent-shin): stamp close marker on sweep closes and disclose regression deadline The daily Greptile sweep's close comment advertised `@agent-shin reconsider` but never stamped AGENT_SHIN_CLOSE_MARKER, so the reconsider reopen guard (was_closed_by_agent_shin), which now also requires that marker, silently rejected every sweep-closed PR with `skip-not-bot-closed`. Move the marker into agent_shin_shared so both close paths share one source of truth, extract format_close_comment so the sweep close comment is unit-testable, and stamp the marker there. Also disclose the grace_days deadline in the review-gate regression comment; it promised "the PR stays open" without mentioning that a still-failing PR is auto-closed grace_days after the notice, which would surprise contributors with a close they were never warned about. * fix(triage): tighten Agent Shin reconsider reopen guards The bot-closed guard accepted any historical Agent Shin marker comment on the thread as proof that Agent Shin owned the latest close, so a post-reopen close by another workflow under the shared `github-actions[bot]` identity could still satisfy the gate and let `@agent-shin reconsider` reopen a PR that Agent Shin did not close this cycle. `fetch_last_close_event` now also returns the latest `closed` event timestamp, and `was_closed_by_agent_shin` requires the most recent Agent Shin marker comment to sit at (or just before) that timestamp, with a small skew window for clock drift between the events and comments APIs. In the same path the LLM verdict check used `decision != "fail"` to choose the reopen branch, which treated a missing, empty, or typo verdict as a pass. Reopen is destructive, so the check now requires an explicit `decision == "pass"` and ambiguous verdicts fall through to the "still failing" branch instead. * style(agent-shin): black-format reconsider guard hardening * docs(agent-shin): scope dry-run wrapper docstring to the single existing helper The module docstring claimed it wrapped every Agent Shin mutation and referenced post_comment/close_pr/etc., but only maybe_post_comment exists. Describe the single helper accurately while keeping the dry-run pattern guidance for any future wrapper. * chore(agent-shin): defer issue/PR template changes to the rollout PR The triage and review-gate automation is gated to the allowlisted authors (mateo-berri, SwiftWinds) and AGENT_SHIN_ENABLED, so during this rollout it only acts on internal PRs/issues. The issue and PR templates have no such gate; …

@agent-shin

…and review-gate label lifecycle (BerriAI#30433) * feat(triage): auto-close stale PRs with Greptile score <4/5 Adds .github/scripts/close_low_quality_prs.py and a daily workflow that closes PRs which: - are open for at least 7 days, and - carry a most-recent greptile-apps review with Confidence Score <4/5, - and are not drafts or opt-out-labeled ('do not close', 'wip', etc.). Each closure posts an explanatory comment telling the contributor how to bring the PR back (rebase, re-request greptile, reopen at 4+/5). The 4/5 bar is already documented in the PR template (.github/pull_request_template.md), so this just enforces it. Tested with a dry run against the live BerriAI/litellm backlog of 1000 open PRs: 100 candidates identified, 598 PRs pass the bar (4+/5), 186 are too young, 97 are drafts, 19 lack any Greptile review and are left alone. Workflow defaults to closing 25 PRs/run as a safety net and supports workflow_dispatch with overrides (close=false for a dry run, custom min_age_days/min_score/limit). 18 unit tests cover score extraction (HTML/markdown/plain text, login variants, multi-review picks latest) and per-PR evaluation (drafts, opt-out labels, age, missing/passing/failing scores). Co-authored-by: Mateo Wang <[email protected]> * docs(templates): require expected/actual + QA proof for external contributions PR template: - Make the rubric explicit at the top: link an issue, OR provide a clear problem description + expected vs. actual + visual QA proof. - Add dedicated sections for each piece so the bot has a deterministic shape to read. - Keep the existing 'Linear ticket' section for internal contributors (they're exempt from the auto-triage rubric). Bug report template: - Split 'What happened?' into 'Actual behavior' + 'Expected behavior'. - Make logs/screenshot a required textarea. - Warning banner at the top tells external contributors that incomplete reports will be auto-closed (with re-evaluation on reopen). Feature request template: - Require a concrete use case + example in the motivation field, not just a one-liner pitch. - Same auto-triage warning banner. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): Agent Shin LLM-as-judge for external PRs and issues Adds a new triage flow that evaluates external pull requests and issues against the project's contribution rubric and, when configured to do so, auto-closes non-conforming ones with an explanatory comment. Contributors can update + reopen to be re-evaluated. Scope: - Internal BerriAI contributors (author_association OWNER/MEMBER/COLLABORATOR) and bot accounts are skipped entirely. - 'Fixes BerriAI#1234' / 'Resolves https://github.com/.../issues/N' in the PR body short-circuits to PASS without burning LLM tokens. - LLM judge returns structured JSON (verdict, missing[], explanation); parser tolerates markdown fences and embedded JSON. - LLM errors NEVER close PRs/issues — failure surfaces as 'skip-llm-error'. Safety: - pull_request_target / issues triggers are FORCED dry-run in the workflow; only manual workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true) takes destructive action. - Default mode writes verdicts to GITHUB_STEP_SUMMARY only — no public comments until the team flips the AGENT_SHIN_ENABLED repo variable. - LLM uses an OpenAI-compatible endpoint (model and base URL configurable via repo variables; key via OPENAI_API_KEY secret). Files: - .github/scripts/triage_with_llm.py - judge orchestrator + CLI - .github/workflows/triage_pr_with_llm.yml - .github/workflows/triage_issue_with_llm.yml - tests/test_litellm/test_github_triage_with_llm.py - 33 unit tests End-to-end validated against four real PRs (BerriAI#28117 internal collaborator, BerriAI#28108 bot, BerriAI#28129 'Fixes BerriAI#28128', BerriAI#28116 no linked issue) and issue BerriAI#28132 with a stubbed LLM judge: each path produces the expected action. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): scope Greptile auto-closer to external contributors + dry-run by default - close_low_quality_prs.py now filters by GitHub author_association via the REST API: PRs from OWNER / MEMBER / COLLABORATOR (and bot accounts) are skipped with a new 'skip-internal' summary bucket. - close_low_quality_prs.yml now defaults workflow_dispatch close=false, and ignores 'close=true' unless the new repo variable AGENT_SHIN_ENABLED is set to 'true'. Scheduled runs are dry-run only until the team flips that switch. - Updated unit tests: one new test asserting internal authors are skipped, and an autouse fixture treats unspecified test PRs as external so the rest of the suite still exercises the close path. Co-authored-by: Mateo Wang <[email protected]> * fix(workflows): scheduled cron closes PRs; safe --close strip in triage Co-authored-by: Yassin Kortam <[email protected]> * fix(triage): scheduled cron stays dry-run; dedent prompts before interpolation - close_low_quality_prs.yml: only workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true) actually closes PRs. Scheduled runs are always dry-run, matching the safety invariant documented for triage_pr/issue. - triage_with_llm.py: textwrap.dedent on an f-string with multi-line interpolated bodies fails because the body's 2nd+ lines start at column 0, making the common-indent zero. Dedent the static template first, then .format() the title/body in. Co-authored-by: Yassin Kortam <[email protected]> * Fix bugs in auto-close PR triage scripts - close_low_quality_prs.py: Treat author_association API lookup failures as internal (fail-safe) so transient errors don't cause internal contributors' PRs to be auto-closed. - triage_with_llm.py: Update summary heading from 'Would post comment:' to 'Posted comment:' since this branch only runs after the comment has already been posted. Co-authored-by: Yassin Kortam <[email protected]> * feat(triage): default Agent Shin to gpt-5.4-mini with reasoning_effort=none - Bump DEFAULT_MODEL from gpt-4o-mini to gpt-5.4-mini (more modern; 4M total context window per OpenAI catalog, JSON-schema response format, function calling all supported). - For gpt-5.x family models, pass reasoning_effort="none" via extra_body. gpt-5.x rejects temperature != 1 unless reasoning_effort is explicitly "none"; setting it lets us keep temperature=0 for deterministic JSON rubric judgments. extra_body works across openai SDK versions regardless of whether they natively type the kwarg. - For non-gpt5 overrides (TRIAGE_MODEL=gpt-4o-mini etc.), reasoning_effort is not sent. - 4 new unit tests cover: gpt-5.4-mini -> reasoning_effort=none, capitalized/dated gpt-5 variants -> reasoning_effort=none, gpt-4o-mini -> no extra_body, base_url passthrough. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — drop dead gh_json and fix --optout-label append-with-default - Removed the unused gh_json helper (bugbot low-severity dead code). - Replaced argparse `action="append", default=[...]` with default=None + DEFAULT_OPTOUT_LABELS fallback. The mutable-default + append combo silently APPENDS to the canonical defaults instead of replacing them, so --optout-label could not actually scope the opt-out list. - Added tests covering both the canonical default and the flag-replaces-defaults behavior. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — tighten linked-issue regex, fail-safe author_association, fix empty TRIAGE_MODEL Three independent bugbot findings against triage_with_llm.py: 1. LINKED_ISSUE_PATTERN included weak keywords (`see`, `ref`, `addresses`) so casual mentions like "See BerriAI#1234 for context" were short-circuited to pass-linked-issue without ever calling the LLM — contradicting the prompt's own "a bare issue number without a closing keyword counts only if it's clearly the related issue (not a passing mention)" rubric. Limit the regex to GitHub's documented PR-closing keywords (fixes/fix/fixed/closes/close/closed/resolves/resolve/resolved). 2. is_internal_contributor() treated an empty/missing author_association as external (eligible for the destructive close path), while the sibling is_external_pr_author() in close_low_quality_prs.py fail-safes the same case as internal. Align the two so a partial/unknown GitHub response can never make a PR eligible for auto-close. 3. argparse `default=os.environ.get("TRIAGE_MODEL", DEFAULT_MODEL)` returns the empty string when GitHub Actions exposes an unset repo variable as an empty-string env var (the optional vars.TRIAGE_MODEL case in the workflow). Use `os.environ.get(...) or DEFAULT_MODEL` so empty -> default, matching the existing OPENAI_BASE_URL pattern. Tests: - Casual mentions now must fall through to the LLM (parametrized); added an orchestration test ensuring "See BerriAI#1234" reaches the judge. - Empty/missing author_association now fails safe (parametrized). - Empty TRIAGE_MODEL env var falls back to DEFAULT_MODEL; explicit TRIAGE_MODEL is still honored. Co-authored-by: Mateo Wang <[email protected]> * fix(workflows): bugbot — gate Agent Shin --close on '= true' not '!= false' The PR and issue Agent Shin workflows gated the destructive --close flag with [ "${DISPATCH_CLOSE:-false}" != "false" ]. That pattern treats anything other than the literal string "false" as enabling closure — "True", "yes", "1", typos, accidental whitespace, etc. The workflow_dispatch input UI is a 'true'/'false' choice dropdown so the form is constrained, but the API (`gh workflow run -f close=...`) accepts any string, and a CI cron / external invoker passing a non-canonical truthy value would have silently enabled real contributor PR closures. Mirror the sibling Greptile closer's [ "${CLOSE_FLAG}" = "true" ] pattern: only the EXACT string "true" enables --close; every other value (including the unset/empty default) resolves to dry-run. This is the fail-safe philosophy applied everywhere else in this PR. Added tests/test_litellm/test_github_triage_workflows.py with two parametrized invariants: 1. The destructive gate uses '= "true"' for its env-var comparison (either bare '${ENV}' or '${ENV:-false}' form accepted), and never the fail-open '!= "false"' pattern. 2. Every destructive gate is also gated on AGENT_SHIN_ENABLED being "true" — either by entering the close branch on '=' or by bailing out early on '!=' — so flipping the repo variable off is a true kill switch regardless of per-run inputs. Manually verified the test fails on the buggy '!= "false"' pattern and passes on the fix, so it would have caught the regression at PR time. Co-authored-by: Mateo Wang <[email protected]> * feat(triage): close any PR (incl. drafts, any age); add @agent-shin reconsider flow Follow-up to PR BerriAI#28117. Three behavior changes + one new workflow, addressing the team's concerns on the original review: 1) Apply auto-close to ALL open PRs, not just those over a week old. - close_low_quality_prs.py: --min-age-days default flipped from 7 to 0. The flag is preserved as an opt-in safety net for one-off backfill runs that want to spare very-young PRs, but the daily scheduled sweep now closes external-author PRs as soon as Greptile scores them <4/5. - close_low_quality_prs.yml: workflow_dispatch input default also flipped to 0; doc comments updated. 2) Apply auto-close to draft PRs too. - close_low_quality_prs.py: removed the skip-draft branch in evaluate_pr. Drafts are NOT a free pass — the team's intent is 'open PR count == PRs internal collaborators need to action on', so a draft Greptile scored 2/5 still belongs in the closed bucket. Authors who genuinely need a long-lived draft can attach the 'wip' opt-out label, which is unchanged. - The 'skip-draft' action is gone; the 'wip' label still skips. 3) Address the 'OSS contributors cannot reopen a bot-closed PR' wrinkle. GitHub does NOT let an external (non-write-access) contributor reopen a PR that was closed by a bot or maintainer (long-standing limitation). The original PR's close-comments told contributors to 'Reopen the PR — I'll re-evaluate automatically', which is broken for the very audience this triage targets. Two changes: a) Reword every close-comment (Greptile sweep + Agent Shin PR close + Agent Shin issue close + PR template) to recommend: - Open a new PR with the updated branch (primary path). - Or comment '@agent-shin reconsider' on the closed PR for a re-evaluation that, on pass, reopens the PR via the bot's GH_TOKEN write access. b) Add the @agent-shin reconsider workflow: - .github/workflows/triage_reconsider.yml: new 'issue_comment'-triggered workflow. Authorizes only the PR/issue author or an internal collaborator (OWNER/MEMBER/COLLABORATOR), gated via a step output so unauthorized commenters never reach the destructive steps. Globally gated on AGENT_SHIN_ENABLED='true' (positive form, matching the test_github_triage_workflows guardrail patterns). - triage_with_llm.py: --reconsider mode. On a closed PR/issue, re-runs the LLM judge (or linked-issue regex short-circuit) and: - on pass: reopens via reopen_pr/reopen_issue + posts a 'Re-evaluated and reopened' comment. - on fail: leaves closed and posts a 'still missing X' comment so the contributor can iterate again. Reconsider-on-open is a no-op ('skip-not-closed'). Internal-author + bot-account skips still take priority over reconsider. 4) Greptile-on-closed-PRs question: the team asked whether Greptile can re-review a closed PR. Greptile's docs don't address this and we shouldn't promise behavior we can't verify, so the new close-comment wording does NOT instruct contributors to 're-request greptile on the closed PR'. Instead it points them at the new-PR path (which Greptile definitely reviews) or the @agent-shin reconsider trigger (which re-runs the LiteLLM-side rubric judge, not Greptile). Tests: 93 passing (was 59). - test_github_close_low_quality_prs.py: replaced 'skip drafts' test with 'closes drafts when score is low' + 'closes brand-new PR when min_age=0' + 'no skip when min_age=0'. The 'skip too young' assertion is preserved as opt-in. - test_github_triage_with_llm.py: 6 new TestTriageOrchestration cases for reconsider mode (skip-not-closed on open, reopen on pass, still-failing comment on fail, linked-issue short-circuit reopen, skip internal author in reconsider, reopen-issue on pass) + a new TestCloseCommentText class that pins the user-facing 'open a new PR' + '@agent-shin reconsider' wording. - test_github_triage_workflows.py: added triage_reconsider.yml to the destructive-gate guardrail table; AGENT_SHIN_ENABLED is its own destructive gate (no separate per-run flag needed). Co-authored-by: Mateo Wang <[email protected]> * test(triage): pin safe behavior for curly braces in PR/issue title+body Adds regression tests covering the bugbot high-severity finding that str.format() would crash on user-supplied content containing { or }. Empirically str.format() does NOT re-parse interpolated values — only the template literal is scanned for replacement fields — so the bug does not exist in the current code, but pinning the safe behavior prevents a future templating change from silently reintroducing it. Also pins the dedented prompt shape (no leading 8-space indentation on template lines) so a future change to the build_*_prompt functions can't silently regress the LLM judge prompt format on multi-line bodies. Co-authored-by: Mateo Wang <[email protected]> * fix(triage): bugbot — reconsider dry-run + bot-closed guard + rate limit Address three Greptile/veria-ai concerns on the @agent-shin reconsider flow: 1. **Reconsider had no dry-run path.** The previous reconsider mode ignored `--close` and always posted comments + reopened on a pass. A local operator running `python triage_with_llm.py --reconsider --pr N` would silently take destructive GitHub actions with no way to preview. Reconsider now honors `close=False` the same way regular triage does and returns `would-reopen` / `would-reconsider-still-failing` for step-summary rendering. 2. **Reconsider could reopen maintainer-closed PRs/issues** (Medium security finding from veria-ai). The workflow only checked that the commenter was authorized — it did NOT check that the most recent close was performed by Agent Shin. A contributor could comment `@agent-shin reconsider` on a PR a maintainer closed for non-rubric reasons (duplicate, security report, design rejection) and have the bot reopen it. Add `was_closed_by_agent_shin()` which inspects the issue events API for the most recent `closed` actor and only permits reopen when that actor matches the configured bot login (default `github-actions[bot]`, overridable via env). Fail-closed on missing events. 3. **No rate-limiting on the reconsider trigger.** Every `@agent-shin reconsider` comment burns CI minutes + an OpenAI API call. Add a 10-minute cooldown via `seconds_since_last_reconsider_verdict()` which greps the issue's comment list for the bot's own verdict marker (``). Inside the window the triage returns `skip-rate-limited` and the LLM never runs. Workflow update: - `triage_reconsider.yml` now passes `--close` only when `AGENT_SHIN_ENABLED=true`, matching the pattern of `triage_pr_with_llm.yml`. The script runs in both states so the verdict still appears in the step summary for QA. Tests: - Add 5 reconsider safety tests: dry-run for pass / fail / linked-issue short-circuit, bot-closed-guard refusal on maintainer close, rate-limit refusal inside the cooldown window, and cooldown-elapsed acceptance. - Add unit tests for `was_closed_by_agent_shin` (bot / maintainer / missing actor / env-override) and `seconds_since_last_reconsider_verdict` (no marker / multiple markers / non-bot comment with marker / bot comment without marker). - Pin the `` marker in both reopen and still-failing comments — dropping it would silently break the cooldown. Existing reconsider tests updated to pass `close=True` (the production path now) + stub the new guards via `_stub_reconsider_guards`. 112 tests pass (was 93). Co-authored-by: Mateo Wang <[email protected]> * feat(triage): 1-day grace period before close + SwiftWinds immediate-close bypass - Add a 24-hour grace window between the first low-quality detection and the actual auto-close. The first detection posts a warning comment that explicitly says "You have 1 day to address this before this PR is auto-closed" and points the contributor at: * `@agent-shin reconsider` to request another look (and re-open) * `@greptileai` to request a fresh Greptile review — works even after the PR is closed - Both `triage_with_llm.py` (LLM judge) and `close_low_quality_prs.py` (Greptile-score closer) share the same `` HTML marker so a warning posted by either path is recognized by both. - Add IMMEDIATE_CLOSE_LOGINS = {swiftwinds} to bypass BOTH the grace period AND the dry-run / AGENT_SHIN_ENABLED gating. SwiftWinds is the user's personal account (no push permissions to litellm) used to dogfood the bot; user explicitly asked: "For SwiftWinds, just close immediately. Faster iteration that way." - Update the standard close comments to mention that `@greptileai` works even after the PR is closed. - Add 23 new tests covering: warn-grace on first detection, skip during grace window, close after grace expires, SwiftWinds bypass (case insensitive, with close=False, no random-login false positives), the grace-warning text invariants, and the SwiftWinds entry in the IMMEDIATE_CLOSE_LOGINS constant. Co-authored-by: Mateo Wang <[email protected]> * fix: skip grace-period text in close comment for IMMEDIATE_CLOSE_LOGINS For PRs from IMMEDIATE_CLOSE_LOGINS (e.g. swiftwinds), evaluate_pr returns 'close' immediately without ever posting a grace warning, so the close comment should not reference a 1-day grace period. Make close_pr take a grace_period_elapsed flag, default True, and pass False from the main loop when the close path was the immediate-close branch. Co-authored-by: Yassin Kortam <[email protected]> * fix(close-low-quality-prs): report actual closes in dry-run summary IMMEDIATE_CLOSE_LOGINS PRs are closed even when the global --close flag is not set, but the summary used the global dry-run flag to choose between 'would close' and 'closed'. Split the count so operators can see both actual closures and dry-run would-be closures. Co-authored-by: Yassin Kortam <[email protected]> * chore(triage): vendor Agent Shin (BerriAI#28117) onto demo branch Brings the Agent Shin OSS-triage scripts, workflows, issue/PR templates, and tests from PR BerriAI#28117 onto this branch so the new review-gate feature and its end-to-end demo are self-contained and runnable in CI. https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ * feat(triage): add "ready for review" label lifecycle to Agent Shin Adds review_gate(), a state machine that keeps a `ready for review` label in sync with whether an external PR clears BOTH gates — the LLM rubric and Greptile's most recent confidence score: - pass (untagged) -> add label + "ready for review" / "all clear" comment - pass (already tagged) -> no-op (idempotent across re-runs) - regress (Greptile < 4/5 or QA proof removed) -> remove label + "what's missing" comment, PR stays open - recover after a regression -> "all clear again" comment + re-add the label - fail & untagged, < 24h old -> one-time "what's missing" notice (grace window) - fail & untagged, > 24h old -> close + comment (reopen via @agent-shin reconsider) The label itself is the persisted state, so comments fire only on transitions (never on every scheduled run). All side effects are gated behind --close, so the dry-run contract matches the existing triage flow. Lifecycle comments use hidden HTML markers and deliberately avoid the auto-close marker so they never trip the reconsider provenance check. Relocates the shared Greptile helpers (extract_greptile_score, SCORE_PATTERN, GREPTILE_BOT_LOGINS, parse_iso8601) into triage_with_llm.py so the daily sweep and the review gate read the score through one implementation, and adds the review_gate.yml workflow (dry-run unless AGENT_SHIN_ENABLED=true) plus 18 unit tests covering every branch and a full pass->regress->recover cycle. https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ * Port review-gate feature from BerriAI#28758 onto BerriAI#28147 triage scripts Adds the "ready for review" label lifecycle (originally PR BerriAI#28758) on top of BerriAI#28147's refactored triage_with_llm.py. The original commit was authored against an older snapshot of BerriAI#28117 and could not be applied cleanly, so the additions were re-applied surgically: - New constants: READY_FOR_REVIEW_LABEL, DEFAULT_GRACE_DAYS, DEFAULT_MIN_GREPTILE_SCORE, READY/REGRESSED/WITHIN_GRACE markers, GREPTILE_BOT_LOGINS, SCORE_PATTERN, AGENT_SHIN_AUTO_CLOSE_MARKER. - New helpers: add_label, remove_label, extract_greptile_score, parse_iso8601 (the latter two mirrored from close_low_quality_prs.py so the daily sweep and the review gate read the score through the same logic). - New comment formatters: format_ready_for_review_comment, format_all_clear_comment, format_regression_comment, format_within_grace_comment. - New entry point: review_gate() implementing the pass/regress/recover state machine, with the label itself acting as persisted state so transition comments fire only on actual transitions. - main() learns --review-gate, --grace-days, --min-greptile-score and dispatches to review_gate() when the flag is set. Verified via tests/test_litellm/test_github_review_gate.py (18 tests) and the existing triage suites (144 more) — all 162 pass. * agent_shin: extract shared constants/helpers; cover review_gate.yml in guardrail tests Bug 1: `triage_with_llm.py` and `close_low_quality_prs.py` each defined their own copies of `extract_greptile_score`, `parse_iso8601`, `GREPTILE_BOT_LOGINS`, `SCORE_PATTERN`, `GRACE_COMMENT_MARKER`, `GRACE_PERIOD_SECONDS`, `IMMEDIATE_CLOSE_LOGINS`, and `AGENT_SHIN_DEFAULT_BOT_LOGIN`. The comments explicitly said the two copies had to stay in sync, but nothing enforced it. A future change to one (e.g. extending `SCORE_PATTERN` for a new Greptile output format) would silently diverge from the other and the daily sweep and the LLM judge would disagree on which PRs have low scores. Extract these to `.github/scripts/agent_shin_shared.py` and re-export them from each script so the existing test attribute access (`triage_module.GRACE_COMMENT_MARKER`, etc.) keeps working without any test changes. Bug 2: `review_gate.yml` is a destructive workflow (close PRs, add/remove labels, post comments) with the same gating philosophy as the others (`AGENT_SHIN_ENABLED = "true"` + a per-run `CLOSE_FLAG = "true"`), but it was missing from `DESTRUCTIVE_GATE_ENV` in the guardrail tests. Add it so a future regression (e.g. flipping to `!= "false"`) is caught by the same parameterized invariants as every other workflow. Co-authored-by: Yassin Kortam <[email protected]> * agent_shin: fix bug bundle (gated LLM key, author-filtered marker dedup, dedup gh/grace helpers) Co-authored-by: Yassin Kortam <[email protected]> * agent_shin: fix review_gate close-after-regression and case-insensitive label match Co-authored-by: Yassin Kortam <[email protected]> * feat(triage): add one-shot 7-day heads-up sweep for Agent Shin rollout Adds a rollout-day workflow that comments on every open external PR/issue that the new triage bot WOULD auto-close, giving contributors 7 days to fix their description before any destructive action runs. Why now: merging this PR enables Agent Shin in dry-run. The follow-up "enact" PR (next Monday) flips the destructive paths on. Without this heads-up, contributors would get a close-comment on day 8 with no prior warning. The heads-up names the cutoff date, lists the rubric, calls out each PR/issue's specific missing pieces, and explains the recovery paths (@agent-shin reconsider for PRs, edit + reopen for issues). Files - .github/scripts/_agent_shin_actions.py — thin maybe_post_comment / maybe_close_* / maybe_add_label / etc. wrappers. Each is a single `if dry_run: log; return; else: call_through()` so a dry-run preview differs from the real run in exactly one call site per mutation. The call-through goes via `triage_with_llm.<name>` (module-qualified) so monkeypatching the underlying function in tests is reflected here. - .github/scripts/triage_rollout_heads_up.py — the sweep. Iterates every open PR + issue via `gh pr list` / `gh issue list`, runs the future rubric (review_gate for PRs, triage(kind="issue") for issues), and posts the heads-up on any item that would be auto-closed. Idempotent via a `` marker. Defaults to dry- run; --close opts in to real posts. --close-on overrides the cutoff date (defaults to today + 7 days). - .github/workflows/triage_rollout_heads_up.yml — one-shot workflow. Triggers on push to litellm_internal_staging filtered to the script path (fires on rollout merge) plus workflow_dispatch with a dry_run input that defaults to "true" for safe manual re-runs. - tests/test_litellm/test_triage_rollout_heads_up.py — 28 unit tests covering: the dry-run wrappers (each maybe_* gates correctly), the _would_be_closed predicate for PR vs. issue results, the comment formatter (cutoff/rubric/marker/recovery wording), per-item dispatch (skip-not-open, skip-internal-author, skip-already-notified, skip-passing, would-post/posted), and the sweep loop end-to-end. Local preview (no GitHub mutations): python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm Real run (what the workflow does): python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm --close TODO: replace the placeholder ROLLOUT_BLOG_URL with the canonical docs URL once the litellm-docs PR ships. * fix: gate reconsider workflow OPENAI_API_KEY + remove dead actions wrappers - Mirror sibling Agent Shin workflows by only exposing OPENAI_API_KEY in triage_reconsider.yml when vars.AGENT_SHIN_ENABLED == 'true'. Previously the secret was unconditionally exposed, so any PR/issue author could trigger paid LLM calls by commenting '@agent-shin reconsider' even while the bot was supposed to be in dry-run. - Remove the six unused dry-run wrappers (maybe_close_pr, maybe_close_issue, maybe_reopen_pr, maybe_reopen_issue, maybe_add_label, maybe_remove_label) from _agent_shin_actions.py — only maybe_post_comment is used by rollout scripts. Drop the associated tests that exercised the now-removed functions. Co-authored-by: Yassin Kortam <[email protected]> * fix: address triage script edge cases - triage_rollout_heads_up.py: replace %-d strftime specifier (GNU-only) with portable day formatting so the script doesn't crash on Windows. - close_low_quality_prs.py: skip malformed JSON lines in fetch_pr_comments instead of letting one bad line abort the daily sweep, matching the pattern in triage_with_llm._iter_paginated_json. - triage_with_llm.py: move has_linked_issue short-circuit before build_pr_prompt to avoid unnecessary prompt construction on PRs that link an issue. Co-authored-by: Yassin Kortam <[email protected]> * fix(scripts): per-PR error isolation and limit grace warnings in close_low_quality_prs - Wrap per-PR processing in try/except so a transient GitHub API failure on one PR no longer aborts the entire daily sweep (mirrors the pattern already used in triage_rollout_heads_up.py). - Have --limit bound *all* destructive write actions (closures and grace warnings combined), not just closures. Prevents a backlog of newly failing PRs from flooding contributors with comments in a single run. Co-authored-by: Yassin Kortam <[email protected]> * fix(agent-shin): remove 1000-PR cap on bulk sweeps; sweep entire backlog Both bulk-sweep scripts hardcoded `gh {pr,issue} list --limit 1000`, and gh lists newest-first — so the OLDEST ~900 PRs and ~380 issues were silently dropped. That's exactly the stale backlog the daily closer and one-shot rollout heads-up exist to catch. Extract a single `list_open_items(kind, *, repo, fields)` helper into `agent_shin_shared.py` with `GH_LIST_ALL_LIMIT = 100_000` — a ceiling far above any realistic open backlog so gh paginates until the queue is exhausted. `fetch_open_prs` and `_list_open_numbers` both delegate to it, so the limit lives in exactly one place going forward. Verified live against BerriAI/litellm: - `fetch_open_prs` -> 1981 PRs (was 1000) - `_list_open_numbers(issue)` -> 1382 issues (was 1000) - `_list_open_numbers(pr)` -> 1981 PRs (was 1000) Adds 7 regression tests asserting the new limit is passed, the dedicated `gh {pr,issue} list` command + fields are used per kind, bad kind raises ValueError, and both callers delegate to the shared helper. * fix(agent-shin): require non-mocked end-to-end QA proof for PR pass The PR rubric previously passed any PR with a linked issue, regardless of whether it showed the fix actually working. Sample spot-check found 21/25 recent external PRs passing, including ones that linked an issue but provided zero QA evidence. Tighten the rubric so a pass now requires BOTH: (1) CONTEXT — a linked issue OR a clear problem description with expected-vs-actual behavior. (2) END-TO-END QA PROOF — at least one of: (a) screenshot(s) of the fix working, (b) screen recording / video, (c) specific commands actually run, paired with their real output, against the real system. Mocked unit tests, generic 'I tested it' claims, 'all tests pass' without output, and the linked issue itself are explicitly excluded from QA proof. Also add 'qa_proof_type' to the JSON schema so the per-PR report surfaces which kind of proof (or 'none') the judge saw. Re-sample on the same 25 recent external PRs shifts the verdict distribution from 21 pass / 4 fail to 4 pass / 21 fail, with zero prior-fails now passing — the stricter rule catches PRs that ship only with unit-test claims and no real integration evidence. * feat(agent-shin): link blog explainer from every action-required bot comment Adds "What's this and why am I getting it?" links to docs.litellm.ai/blog/ agent-shin-triage from the four comments contributors actually read when something went wrong: PR close, PR grace warning, issue close, issue grace warning. PR comments also link the rubric section directly from the QA-proof bullet so contributors can self-serve "what counts as proof" without pinging a maintainer. Pins the new guarantees in tests: blog link must appear in all four comments, and the PR close comment must continue to flag mocked-dependency unit tests as insufficient proof. The linked blog post is in BerriAI/litellm-docs PR BerriAI#240; the URL will 404 until that lands. * fix(review_gate): raise sweep limit from 1000 to 100000 to match GH_LIST_ALL_LIMIT gh lists newest-first, so capping at 1000 silently drops the oldest open PRs — exactly the stale ones the daily sweep is meant to reconcile. Use the same ceiling as agent_shin_shared.GH_LIST_ALL_LIMIT so the workflow sees the entire backlog. Co-authored-by: Yassin Kortam <[email protected]> * Fix three Agent Shin triage edge cases - review_gate: expire the regression-marker short-circuit after grace_days so PRs that were regressed and then abandoned can eventually be closed. - review_gate: when the rubric short-circuits to pass via the linked-issue regex but Greptile drags the PR below the bar, replace the synthetic 'LLM was not called' explanation with the real Greptile shortfall so regression / close comments are not misleading. - triage_rollout_heads_up._comments_have_marker: drop the unused 'kind' parameter and filter by bot author so a contributor quoting the heads-up via 'Quote reply' cannot trick the idempotency check, matching the pattern in triage_with_llm._has_marker. Co-authored-by: Yassin Kortam <[email protected]> * fix: pass min_greptile_score through to ready-for-review comment text Co-authored-by: Yassin Kortam <[email protected]> * feat(agent-shin): warmer triage comments — bullet-train emoji, 'what you got right' section, softer 'park this for later' framing User feedback on the auto-triage comments contributors will see: 1. Tone — the previous 'You have 1 day to address this before this PR is auto-closed' framing reads as an ultimatum. Replace with: 'If the description isn't updated in the next 1 day, I'll auto-close this PR. That's not us saying we don't care about the change — we want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later," not a rejection. Take your time.' 2. Positive feedback — the previous comments only listed what was missing. Now every close + grace-warning comment opens with a 'What you got right:' section rendered from the judge's per-field flags. Contributors see a checkmark for everything they got right (linked issue, problem description, expected/actual, QA proof for PRs; runnable repro, screenshot/log, expected/actual, motivation+example for issues) before the gaps. The block is omitted entirely when nothing is present so we never render 'What you got right: (nothing).' 3. Reconsider trigger — the previous grace warning told contributors to comment '@agent-shin reconsider' during the grace window. They don't need to — the bot re-checks on every sweep. The new copy says 'just update the description, no need to ping me' for the grace path, and reserves '@agent-shin reconsider' for the post-close recovery path. 4. Bullet-train emoji — replace 👋 with 🚄 (Shinkansen, the symbol of Agent Shin) across every action-required comment: PR close, PR grace warning, issue close, issue grace warning, within-grace, Greptile- closer grace warning, rollout heads-up. Pinned in tests so a future refactor can't silently revert. 5. Greptile-post-close — the @greptileai bullet now explicitly says 'a low Greptile score isn't a blocker either,' since the previous copy buried the fact that @greptileai works after auto-close. Comment templates updated: format_pr_close_comment, format_issue_close_comment, format_grace_warning_pr_comment, format_grace_warning_issue_comment, format_within_grace_comment (triage_with_llm.py); format_grace_warning_comment (close_low_quality_prs.py); format_heads_up_comment header (triage_rollout_heads_up.py). New helpers: _format_present_for_pr / _format_present_for_issue / _format_present_block, driven off the existing per-field flags the LLM judge already emits — no prompt change needed. New tests pin: bullet-train emoji in every action-required comment; 'What you got right' appears with ✅ bullets when fields are present; the block is omitted when no fields are present; 'park this for later' / 'not a rejection' softer framing; grace warnings tell the contributor 'no need to ping' during the grace window (reconsider is the post-close path only). * feat(agent-shin): gate triage on a dogfood allowlist Add ALLOWLIST_LOGINS to agent_shin_shared so Agent Shin only acts on the named accounts while the set is non-empty. mateo-berri and SwiftWinds are allowlisted for the dogfood rollout; everyone else is skipped with skip-not-allowlisted across all four entrypoints (triage, review gate, the daily low-quality sweep, and the rollout heads-up). For an allowlisted author the usual internal/external classification is bypassed, so a maintainer's own org account still gets triaged during testing. Emptying the set lifts the restriction and restores full triage for the public rollout. The gate is dependency-injected via an `allowlist` parameter defaulting to the constant, so the internal/external-skip paths stay testable. * feat(agent-shin): tighten QA-proof and issue rubrics, ack reconsider with reactions Reorder the end-to-end QA proof options to video, then screenshots, then exact commands with their real output across the PR template, the LLM judge prompts, and every contributor-facing comment, and spell out that mocked or stubbed runs (including pytest on the repo's own unit tests, which mock the provider, DB, and network) never count as proof. QA proof is now required of all contributors, not just external ones. Tighten the issue bug-report rubric to require end-to-end evidence of the bug (the "before" half: a video, screenshot, or command paired with real output) plus expected vs. actual behavior, drop the bias toward PASS, and collapse the separate has_repro/has_proof flags into a single has_repro signal. Standardize the bullet-train emoji and strip em dashes from the bot's public-facing messages, and route issue recovery through @agent-shin reconsider since GitHub doesn't let OSS authors reopen an issue a bot closed. Acknowledge an @agent-shin reconsider the moment it's accepted with an eyes reaction and a thumbs-up once the run finishes, both gated on AGENT_SHIN_ENABLED so dry-run leaves no trace. * fix(agent-shin): shorten auto-close grace to 2 hours and drop the instant-close bypass Two dogfooding changes to the Agent Shin grace window. First, the warn-then-close grace (GRACE_PERIOD_SECONDS) drops from a day to 2 hours so the "fix it before it closes" loop can be exercised in one sitting; the constant carries a note to bump it back up for the public rollout. Second, remove IMMEDIATE_CLOSE_LOGINS entirely. SwiftWinds (the external dogfood account) used to skip the grace window and close on first detection, which also meant closing real PRs even during a scheduled dry run because the per-PR override flipped dry_run off. It now follows the same warn-then-close path as every other author, so a low-quality PR is warned first and only closed once the 2-hour window elapses. This also closes the Greptile finding that the sweep could mutate real PRs while AGENT_SHIN_ENABLED was still off. The review gate's separate age-based grace (DEFAULT_GRACE_DAYS) is left unchanged. Regression tests pin that SwiftWinds now warns-grace instead of closing instantly, and that a dry-run sweep over a closeable PR reports "would close" without making any GitHub mutation. * fix(agent-shin): gate reconsider reopen on an Agent Shin close marker was_closed_by_agent_shin only checked that the most recent close actor was the bot identity. That identity defaults to github-actions[bot], which is shared by every workflow in the repo (stale/duplicate sweeps included), so a contributor could @agent-shin reconsider an item another workflow closed and, if the description passed the rubric, get it reopened even though Agent Shin was never the closer. Require a second, Agent-Shin-specific signal alongside the actor check: an auto-close comment stamped with a hidden AGENT_SHIN_CLOSE_MARKER. Both close paths (the grace-period close and the review-gate close) flow through format_pr_close_comment / format_issue_close_comment, so stamping the marker there covers every real close while leaving the grace warnings unmarked. The guard stays fail-closed: no marker, no reopen. This also replaces the unused AGENT_SHIN_AUTO_CLOSE_MARKER constant (a visible phrase the guard never consulted) with the hidden marker the guard now relies on. * fix(agent-shin): stamp close marker on sweep closes and disclose regression deadline The daily Greptile sweep's close comment advertised `@agent-shin reconsider` but never stamped AGENT_SHIN_CLOSE_MARKER, so the reconsider reopen guard (was_closed_by_agent_shin), which now also requires that marker, silently rejected every sweep-closed PR with `skip-not-bot-closed`. Move the marker into agent_shin_shared so both close paths share one source of truth, extract format_close_comment so the sweep close comment is unit-testable, and stamp the marker there. Also disclose the grace_days deadline in the review-gate regression comment; it promised "the PR stays open" without mentioning that a still-failing PR is auto-closed grace_days after the notice, which would surprise contributors with a close they were never warned about. * fix(triage): tighten Agent Shin reconsider reopen guards The bot-closed guard accepted any historical Agent Shin marker comment on the thread as proof that Agent Shin owned the latest close, so a post-reopen close by another workflow under the shared `github-actions[bot]` identity could still satisfy the gate and let `@agent-shin reconsider` reopen a PR that Agent Shin did not close this cycle. `fetch_last_close_event` now also returns the latest `closed` event timestamp, and `was_closed_by_agent_shin` requires the most recent Agent Shin marker comment to sit at (or just before) that timestamp, with a small skew window for clock drift between the events and comments APIs. In the same path the LLM verdict check used `decision != "fail"` to choose the reopen branch, which treated a missing, empty, or typo verdict as a pass. Reopen is destructive, so the check now requires an explicit `decision == "pass"` and ambiguous verdicts fall through to the "still failing" branch instead. * style(agent-shin): black-format reconsider guard hardening * docs(agent-shin): scope dry-run wrapper docstring to the single existing helper The module docstring claimed it wrapped every Agent Shin mutation and referenced post_comment/close_pr/etc., but only maybe_post_comment exists. Describe the single helper accurately while keeping the dry-run pattern guidance for any future wrapper. * chore(agent-shin): defer issue/PR template changes to the rollout PR The triage and review-gate automation is gated to the allowlisted authors (mateo-berri, SwiftWinds) and AGENT_SHIN_ENABLED, so during this rollout it only acts on internal PRs/issues. The issue and PR templates have no such gate; they change for every contributor on merge and advertise that an LLM bot auto-closes external submissions, which won't happen while the allowlist is the sole author gate. Revert bug_report.yml, feature_request.yml, and pull_request_template.md to base so the public-facing messaging lands with the rollout flip instead of ahead of it. The scripts embed their own rubric and never read these files, so triage behavior is unchanged. * ci(agent-shin): hash-pin the openai install in privileged triage workflows The triage workflows install the OpenAI client with `pip install "openai>=1.40.0"`, a floating lower bound that resolves openai and its whole transitive tree to whatever PyPI serves at run time. These jobs run under pull_request_target with a write-scoped GITHUB_TOKEN, and the install plus the triage run happen on every PR open regardless of the AGENT_SHIN_ENABLED dry-run gate (that gate only withholds the LLM key and the destructive --close path), so a compromised release would execute during install or import while the token is in scope. Install instead from a new .github/scripts/triage-requirements.txt that pins openai==2.33.0 and every transitive dependency to an exact version with sha256 hashes, via pip --require-hashes. The workflows already sparse-checkout .github/scripts from the base repo (never fork code), so the pinned file is trusted. Add static guardrails to test_github_triage_workflows.py that fail if any installer workflow reverts to a floating openai install or if the requirements file loses its exact pins or hashes. * ci(agent-shin): gate rollout heads-up real run behind manual dispatch The rollout heads-up workflow fired its real `--close` sweep on every push to litellm_internal_staging that touched the script, and exposed OPENAI_API_KEY unconditionally, unlike every sibling triage workflow which only exposes the key on an enabled or dispatched run. That made merging the script post real heads-up comments (bounded only by the dogfood allowlist), which contradicts the inert-by-default safety invariant; once the allowlist is cleared for the public rollout, any later edit to the file would sweep the whole open backlog with real writes. The heads-up cannot be gated on AGENT_SHIN_ENABLED: its whole job is to warn contributors before that flag flips on, so it has to run while the flag is still off. Instead the automatic push trigger now stays dry-run, and the real one-shot sweep is a deliberate manual workflow_dispatch with dry_run=false, the sole path that adds `--close`. OPENAI_API_KEY is exposed only on that dispatch, matching the sibling workflows. Add static guardrails that fail if the push path regains a `--close`, if the dispatch gate stops fail-closing on the exact string "false", or if the key is exposed unconditionally again. --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Mateo Wang <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: Mateo <[email protected]>

Add async iterator to fix/enable streaming responses

4d009a9

vercel Bot deployed to Preview August 29, 2023 22:47 View deployment

ishaan-jaff approved these changes Aug 29, 2023

View reviewed changes

ishaan-jaff merged commit 6651cab into BerriAI:main Aug 30, 2023

ghost mentioned this pull request Sep 4, 2023

Improve acomplete for high traffic scenarions #255

Closed

coderabbitai Bot mentioned this pull request Mar 29, 2026

Fix/issue 6400 provider fix aden-hive/hive#6551

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add async iterator to fix/enable streaming responses#240

Add async iterator to fix/enable streaming responses#240
ishaan-jaff merged 1 commit into
BerriAI:mainfrom
estill01:async_iterator_fix

estill01 commented Aug 29, 2023

Uh oh!

vercel Bot commented Aug 29, 2023 •

edited

Loading

Uh oh!

estill01 commented Aug 29, 2023

Uh oh!

estill01 commented Aug 29, 2023

Uh oh!

ishaan-jaff commented Aug 29, 2023

Uh oh!

ishaan-jaff commented Aug 29, 2023

Uh oh!

ghost commented Aug 29, 2023 •

edited by ghost

Loading

Uh oh!

estill01 commented Aug 29, 2023

Uh oh!

estill01 commented Aug 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

estill01 commented Aug 29, 2023

Uh oh!

vercel Bot commented Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

estill01 commented Aug 29, 2023

Uh oh!

estill01 commented Aug 29, 2023

Uh oh!

ishaan-jaff commented Aug 29, 2023

Uh oh!

ishaan-jaff commented Aug 29, 2023

Uh oh!

ghost commented Aug 29, 2023 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

api call

stream output parsing

Uh oh!

estill01 commented Aug 29, 2023

Uh oh!

estill01 commented Aug 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Aug 29, 2023 •

edited

Loading

ghost commented Aug 29, 2023 •

edited by ghost

Loading