test(github-copilot): cover live xhigh reasoning for non-Claude mini models (#59416) by saju01 · Pull Request #91728 · openclaw/openclaw

saju01 · 2026-06-09T15:29:32Z

What

Adds regression tests covering xhigh reasoning resolution for non-Claude "mini" Copilot models via the live /models extended-thinking compat path.

gpt-5.4-mini should gain xhigh when the live catalog advertises it (live extended-thinking compat applies).
gpt-5-mini must not over-grant xhigh (guards against the compat helper widening reasoning levels too broadly).

Refs #59416.

Why this PR shrank

This branch originally proposed a live /models catalog with static fallback union. That idea has since landed upstream independently (live discovery via 75405f64d0 / d2279591bf; the old models-defaults.ts helper was removed in 46c42d4a0d), with live-first precedence handled centrally — so the original implementation is now redundant and unmergeable as-was.

Rather than resurrect dead code, this PR is reset onto main and reduced to the one piece not yet covered upstream: the non-Claude mini xhigh reasoning boundary tests above.

Real behavior proof

Behavior or issue addressed: non-Claude Copilot mini model reasoning compatibility had no regression coverage for the xhigh boundary after the live catalog path landed upstream.
Real environment tested: OpenClaw pull-request validation on this exact PR head, exercising the repository's github-copilot extension test path.

Exact steps or command run after this patch:

vitest github-copilot extension tests
oxlint
oxfmt

Evidence after fix:

vitest (github-copilot extension): 21/21 passed
oxlint: 0 warnings / 0 errors
oxfmt: clean

Observed result after fix: the tests-only branch now covers both the gpt-5.4-mini xhigh allow case and the gpt-5-mini no-overgrant guard without changing runtime code.
What was not tested: live Copilot network calls; this PR intentionally adds regression coverage only.

Testing

vitest (github-copilot extension): 21/21 passed
oxlint: 0 warnings / 0 errors
oxfmt: clean

Tests-only change — no runtime behavior modified.

clawsweeper · 2026-06-09T15:33:04Z

Codex review: needs maintainer review before merge. Reviewed June 21, 2026, 8:25 PM ET / 00:25 UTC.

Summary
Adds two GitHub Copilot provider tests asserting live compat grants xhigh to gpt-5.4-mini but not to gpt-5-mini.

PR surface: Tests +31. Total +31 across 1 file.

Reproducibility: yes. for source-level review: current main consumes compat.supportedReasoningEfforts, and the linked live Copilot /models output gives the exact gpt-5.4-mini positive and gpt-5-mini negative cases. I did not run tests in this read-only review.

Review metrics: 1 noteworthy metric.

Provider thinking regression coverage: 2 tests added. The assertions cover both the live-catalog xhigh positive case and the no-overgrant negative case for Copilot mini models.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Rerun or clear the stale Real behavior proof workflow failure if branch protection still requires that check.

Risk before merge

[P1] A stale failed Real behavior proof check remains in the latest GitHub check history; maintainers should rerun or confirm branch protection if that workflow is still required.

Maintainer options:

Decide the mitigation before merge
Land the focused regression tests after ordinary maintainer review and required checks, while keeping broader Copilot live-catalog policy on its canonical issue path.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

[P2] No repair lane is needed; the remaining action is ordinary maintainer review and merge-gate handling for a narrow tests-only PR.

Security
Cleared: The diff only adds provider tests and does not change runtime code, dependencies, workflows, secrets, or supply-chain surfaces.

Review details

Best possible solution:

Land the focused regression tests after ordinary maintainer review and required checks, while keeping broader Copilot live-catalog policy on its canonical issue path.

Do we have a high-confidence way to reproduce the issue?

Yes for source-level review: current main consumes compat.supportedReasoningEfforts, and the linked live Copilot /models output gives the exact gpt-5.4-mini positive and gpt-5-mini negative cases. I did not run tests in this read-only review.

Is this the best way to solve the issue?

Yes. Since the live-catalog runtime behavior is already on main, focused provider-boundary regression coverage is the narrowest maintainable remaining change.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 1f6ae32cabb9.

Label changes

Label justifications:

P2: The PR protects a bounded GitHub Copilot provider thinking-capability path without broad runtime or security impact.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR and linked issue discussion include copied live Copilot /models output from an entitled account showing the relevant positive and negative mini-model capability metadata for this tests-only regression guard.
proof: sufficient: Contributor real behavior proof is sufficient. The PR and linked issue discussion include copied live Copilot /models output from an entitled account showing the relevant positive and negative mini-model capability metadata for this tests-only regression guard.

Evidence reviewed

PR surface:

Tests +31. Total +31 across 1 file.

View PR surface stats

Area	Files	Added	Net
Source	0	0	0
Tests	1	31	+31
Docs	0	0	0
Config	0	0	0
Generated	0	0	0
Other	0	0	0
Total	1	31	+31

What I checked:

PR diff scope: The live PR diff modifies only extensions/github-copilot/index.test.ts and adds the two mini-model thinking-profile assertions at the provider registration boundary. (extensions/github-copilot/index.test.ts:273, 0d0bb4527eab)
Provider entry point: Current main registers resolveThinkingProfile for GitHub Copilot and delegates extra levels to resolveCopilotExtendedThinkingLevels(modelId, compat), so the proposed tests exercise registered provider behavior. (extensions/github-copilot/index.ts:457, 1f6ae32cabb9)
Resolver contract: Current main adds xhigh when the model is statically allowlisted or catalog compat advertises xhigh; without that compat, a non-allowlisted non-Claude model does not get xhigh. (extensions/github-copilot/model-metadata.ts:101, 1f6ae32cabb9)
Live catalog mapping: The Copilot /models mapper normalizes live capabilities.supports.reasoning_effort into compat.supportedReasoningEfforts, which is the metadata path this PR locks down. (extensions/github-copilot/models.ts:155, 1f6ae32cabb9)
Current-main coverage gap: Current main has generic Copilot catalog-supported xhigh coverage but no provider thinking-profile assertions for the reported gpt-5.4-mini positive and gpt-5-mini negative cases. (extensions/github-copilot/index.test.ts:240, 1f6ae32cabb9)
PR head not on main: The proposed test commit is not an ancestor of current main, so the tests are not already implemented on main. (0d0bb4527eab)

Likely related people:

Eduardo Piva: Authored the merged GitHub Copilot live /models discovery and mapping commit that supplies compat.supportedReasoningEfforts. (role: introduced live catalog behavior; confidence: high; commits: 75405f64d0a7; files: extensions/github-copilot/index.ts, extensions/github-copilot/models.ts, extensions/github-copilot/models.test.ts)
Vincent Koc: Authored the earlier GitHub Copilot xhigh enablement for gpt-5.4 and later removed obsolete Copilot model defaults helpers in the same provider surface. (role: recent thinking-compat contributor; confidence: high; commits: a70fdc88e083, 46c42d4a0dee; files: extensions/github-copilot/index.ts, extensions/github-copilot/model-metadata.ts, extensions/github-copilot/models-defaults.ts)
liuhao1024: Authored the provider discovery runtime change that makes refreshable catalogs require runtime discovery, adjacent to how live Copilot rows become available. (role: adjacent runtime discovery contributor; confidence: medium; commits: d2279591bfa8; files: src/plugins/provider-discovery.runtime.ts, src/plugins/provider-discovery.runtime.test.ts)
saju01: Beyond this PR, authored a merged Copilot static Opus 4.8 catalog update in the same metadata area. (role: recent adjacent contributor; confidence: medium; commits: fbb776d92c29; files: extensions/github-copilot/openclaw.plugin.json, extensions/github-copilot/models.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

saju01 · 2026-06-12T12:10:51Z

@clawsweeper re-review

clawsweeper · 2026-06-12T12:10:55Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27415004192
Updated: 2026-06-12T12:20:05.301Z

saju01 · 2026-06-12T12:14:13Z

@clawsweeper re-review

saju01 · 2026-06-12T12:48:10Z

@clawsweeper re-review

clawsweeper · 2026-06-12T12:48:13Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27416672124
Updated: 2026-06-12T12:58:54.315Z

saju01 · 2026-06-19T18:54:56Z

Thanks for the review. You were right — the live /models catalog approach this PR originally proposed has since landed upstream (live discovery via 75405f64d0 / d2279591bf; models-defaults.ts removed in 46c42d4a0d), so that implementation is now redundant and the import it flagged is gone.

I've reset the branch onto main and scoped this down to the one gap not yet covered upstream: regression tests for xhigh reasoning on non-Claude mini models (gpt-5.4-mini should gain xhigh from live compat; gpt-5-mini should not over-grant). Tests-only, 21/21 vitest pass, oxlint/oxfmt clean.

Re-requesting review on the slimmed-down change. Happy to close instead if you'd rather fold these assertions in elsewhere.

openclaw#91728) Regression coverage for openclaw#59416: when live /models metadata wins over the static xhigh allowlist, non-Claude mini-family ids (e.g. gpt-5.4-mini) must gain xhigh from their resolved compat, while ids whose live effort list lacks xhigh (e.g. gpt-5-mini) must not over-grant it. The live-first model catalog union and models-defaults removal are already in origin/main (42a7ac4, ce83abf, ae5a2a3); this PR now contributes only the still-missing regression tests on top of that base. Co-authored-by: saju01 <[email protected]>

openclaw#91728) Regression coverage for openclaw#59416: when live /models metadata wins over the static xhigh allowlist, non-Claude mini-family ids (e.g. gpt-5.4-mini) must gain xhigh from their resolved compat, while ids whose live effort list lacks xhigh (e.g. gpt-5-mini) must not over-grant it. The live-first model catalog union and models-defaults removal are already in origin/main (75405f6, 46c42d4, d227959); this PR now contributes only the still-missing regression tests on top of that base. Co-authored-by: saju01 <[email protected]>

openclaw-barnacle Bot added extensions: github-copilot size: M triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 9, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 12, 2026

saju01 force-pushed the fix/copilot-live-model-catalog branch from 336c450 to a9ad1f1 Compare June 12, 2026 12:46

openclaw-barnacle Bot added size: S and removed size: M proof: sufficient ClawSweeper judged the real behavior proof convincing. labels Jun 12, 2026

saju01 force-pushed the fix/copilot-live-model-catalog branch from 4e1d319 to 0d0bb45 Compare June 19, 2026 18:49

openclaw-barnacle Bot added size: XS and removed size: S proof: sufficient ClawSweeper judged the real behavior proof convincing. labels Jun 19, 2026

saju01 changed the title ~~fix(github-copilot): prefer live model catalog~~ test(github-copilot): cover live xhigh reasoning for non-Claude mini models (#59416) Jun 19, 2026

openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 19, 2026

openclaw-barnacle Bot removed the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label Jun 19, 2026

vincentkoc added the proof: override Maintainer override for the external PR real behavior proof gate. label Jun 29, 2026

openclaw-barnacle Bot added triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. and removed triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jun 29, 2026

vincentkoc merged commit 78b03fb into openclaw:main Jun 29, 2026
235 of 261 checks passed

github-actions Bot mentioned this pull request Jun 29, 2026

📡 Upstream Digest — 2026-06-29 08:51 UTC curtismercier/openclaw-mods#1172

Open

difujia mentioned this pull request Jun 29, 2026

[Bug]: github-copilot Claude models capped at 128k on published 2026.6.10 (PR #91728 not released yet) #97844

Open

clawsweeper Bot mentioned this pull request Jul 4, 2026

fix(github-copilot): preserve thinking levels for Claude models #99798

Open

vincentkoc mentioned this pull request Jul 6, 2026

chore(jobs): prune resolved OpenClaw inbox jobs openclaw/clownfish#221

Merged

CarelvanHeerden mentioned this pull request Jul 15, 2026

[Bug]: direct anthropic-messages claude-opus-4-7 / opus-4-6 / sonnet-4-6 capped at 200k on 2026.6.10 despite ANTHROPIC_GA_1M_MODEL_PREFIXES listing them (stale plugin manifest, no user-side workaround) #108152

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

test(github-copilot): cover live xhigh reasoning for non-Claude mini models (#59416)#91728

test(github-copilot): cover live xhigh reasoning for non-Claude mini models (#59416)#91728
vincentkoc merged 1 commit into
openclaw:mainfrom
saju01:fix/copilot-live-model-catalog

saju01 commented Jun 9, 2026 •

edited by vincentkoc

Loading

Uh oh!

clawsweeper Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

saju01 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

saju01 commented Jun 12, 2026

Uh oh!

saju01 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

saju01 commented Jun 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

saju01 commented Jun 9, 2026 • edited by vincentkoc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why this PR shrank

Real behavior proof

Testing

Uh oh!

clawsweeper Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saju01 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saju01 commented Jun 12, 2026

Uh oh!

saju01 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saju01 commented Jun 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

saju01 commented Jun 9, 2026 •

edited by vincentkoc

Loading

clawsweeper Bot commented Jun 9, 2026 •

edited

Loading

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading