test(e2e): wait for resolved context window in visual chat scenario by p12tic · Pull Request #1941 · Zoo-Code-Org/Zoo-Code

p12tic · 2026-10-05T22:43:01Z

The chat visual scenario captured its screenshot before the selected model's metadata had resolved. Router models arrive asynchronously over the extension-host message channel, so the context-window readout briefly rendered the 1 fallback, producing non-deterministic screenshots. Wait for a resolved context window before capturing.

Failing runs:

The chat visual scenario captured its screenshot before the selected model's metadata had resolved. Router models arrive asynchronously over the extension-host message channel, so the context-window readout briefly rendered the `1` fallback, producing non-deterministic screenshots. Wait for a resolved context window before capturing. Failing runs: - https://github.com/Zoo-Code-Org/Zoo-Code/actions/runs/37288898485/job/111694365121 - https://github.com/Zoo-Code-Org/Zoo-Code/actions/runs/37287279414/job/111689058401 - https://github.com/Zoo-Code-Org/Zoo-Code/actions/runs/37347520689/job/111889755914

coderabbitai · 2026-10-05T22:43:10Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: Zoo-Code-Org/Zoo-Code/.coderabbit.yaml
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 3d7d7e91-1ba2-467e-85ed-c48319bdfed0

📥 Commits

Reviewing files that changed from the base of the PR and between 9af61f8 and 15438ac.

📒 Files selected for processing (1)

apps/vscode-e2e/src/visual/electron.visual.ts

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

📜 Recent review details

🧰 Additional context used

📚 Code guidelines (1)

apps/vscode-e2e/AGENTS.md — auto-discovered

📓 Path-based instructions (4)

Check strict typing and exhaustive behavior across normal, boundary, error, cancellation, retry, and compatibility paths.

⚙️ CodeRabbit configuration file

Files:

apps/vscode-e2e/src/visual/electron.visual.ts

Reserve end-to-end coverage for behavior that requires the real VS Code host, workspace APIs, extension activation, webview messaging, file watchers, or a full workflow.

⚙️ CodeRabbit configuration file

Files:

apps/vscode-e2e/src/visual/electron.visual.ts

Act as an adversarial second-opinion reviewer.

⚙️ CodeRabbit configuration file

Files:

apps/vscode-e2e/src/visual/electron.visual.ts

Source excerpt: Keep the Electron matrix limited to high-value shell integration scenes that require the real VS Code host.

📄 CodeRabbit inference engine (apps/vscode-e2e/AGENTS.md)

Files:

apps/vscode-e2e/src/visual/electron.visual.ts

🔇 Additional comments (1)

apps/vscode-e2e/src/visual/electron.visual.ts (1)

239-249: LGTM!

📝 Summary

Summary by CodeRabbit

Tests
- Visual checks now wait for the chat context-window readout to update before capturing screenshots in the relevant scenario. Other scenarios are unchanged.

Walkthrough

The chat visual test waits up to 60 seconds for context-window-size to change from its initial fallback value of "1" before taking screenshots. Other scenarios skip this wait.

Changes

Chat visual test

Layer / File(s)	Summary
Wait for chat context-window metadata `apps/vscode-e2e/src/visual/electron.visual.ts`	Chat scenarios wait for `context-window-size` to differ from `"1"`, with a 60-second timeout. Other scenarios do not perform this wait.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 15438

Chat screenshots wait for the selected model’s context-window metadata before capture. The configured model resolves to a value distinct from the fallback, and no actionable merge risk remains beyond normal checks.

Architecture Summary

Architecture risk: 🔵 Low · up to 15438

The change affects 1 system.

Changed systems: apps/vscode-e2e

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

observed — apps/vscode-e2e (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

observed — Modified behavior in apps/vscode-e2e/src/visual/electron.visual.ts: For chat scenarios, the test waits until context-window-size no longer displays "1", timing out after 60 seconds. The added comments describe asynchronous model metadata and the fallback value.

🚥 Pre-merge checks | ✅ 7 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description explains the screenshot failure and its cause, and includes failing run links. It does not provide the required approved issue link or a test procedure, so key template information is …	Add the approved GitHub issue number under “Related GitHub Issue” and document how the change was tested and how reviewers can reproduce or verify it under “Test Procedure”. Also complete the applicable pre-submission checklist items before…

✅ Passed checks (7 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Regression Evidence	✅ Passed	The PR only changes the Electron visual test: the chat scenario waits until `context-window-size` differs from the `1` fallback before capturing the existing screenshot. The wait is itself a focused a…
Security Boundaries	✅ Passed	The only change adds a Playwright wait for a fixed context-window selector in the chat visual test. It does not expose secrets or PII, execute or trust unvalidated input, or bypass approval or allowli…
Persistence Integrity	✅ Passed	The only changed file adds a chat-scene UI wait for the context-window readout to differ from `1`. The diff introduces no persistence path, write operation, rollback, or persisted default propagation …
Lifecycle Resource Cleanup	✅ Passed	The changed path adds a Playwright locator assertion for the chat scene, with a 60-second timeout. It does not create or retain a listener, watcher, provider, timer, or task. If the wait times out, th…
Title check	✅ Passed	The title clearly identifies the main change: waiting for the chat scenario’s context-window value to resolve before taking the visual screenshot.

Full details: Description check

Explanation

The description explains the screenshot failure and its cause, and includes failing run links. It does not provide the required approved issue link or a test procedure, so key template information is missing.

Resolution

Add the approved GitHub issue number under “Related GitHub Issue” and document how the change was tested and how reviewers can reproduce or verify it under “Test Procedure”. Also complete the applicable pre-submission checklist items before marking the PR ready for review.

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-05T22:43:14Z

Review status

Thanks for contributing. This comment tracks the review sequence and the next action.

Current step: Awaiting fresh human maintainer or CODEOWNER approval.

Automated review is complete for the latest commit but does not replace human approval.

Review-state labels are managed by this workflow; do not edit them manually. community-approved is managed the same way — do not add or remove it manually. It signals a fresh community code approval for the current head as an advisory priority only; maintainer review is still required.

codecov · 2026-10-05T22:52:46Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

p12tic requested review from JamesRobert20, edelauna, navedmerchant and taltas as code owners October 5, 2026 22:43

github-actions Bot added coderabbit-review-active Required CI passed; CodeRabbit review is active awaiting-coderabbit Waiting for CodeRabbit to approve the latest commit labels Oct 5, 2026

coderabbitai Bot approved these changes Oct 5, 2026

View reviewed changes

github-actions Bot added awaiting-maintainer CodeRabbit approved; waiting for a human maintainer and removed coderabbit-review-active Required CI passed; CodeRabbit review is active awaiting-coderabbit Waiting for CodeRabbit to approve the latest commit labels Oct 5, 2026

taltas approved these changes Oct 6, 2026

View reviewed changes

taltas added this pull request to the merge queue Oct 6, 2026

Merged via the queue into Zoo-Code-Org:main with commit 3bea5b1 Oct 6, 2026
21 checks passed

Navigation Menu

Repository navigation

test(e2e): wait for resolved context window in visual chat scenario - #1941

test(e2e): wait for resolved context window in visual chat scenario#1941
taltas merged 1 commit into
Zoo-Code-Org:mainfrom
p12tic:fix-test-screenshot-instability

p12tic commented Oct 5, 2026

Uh oh!

coderabbitai Bot commented Oct 5, 2026 •

edited

Loading

Summary by CodeRabbit

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Oct 5, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

p12tic commented Oct 5, 2026

Uh oh!

coderabbitai Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

Architecture Summary

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review status

Uh oh!

codecov Bot commented Oct 5, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading