Repository navigation
Commit 257c626
feat(videos): add Grok video bridge for non-OpenAI models (#582)
* feat(videos): add Grok video bridge for non-OpenAI models
Extends the image bridge to support asynchronous video generation via xAI
Grok Imagine Video. Video generation uses a submit→poll→download pattern
(POST /v1/videos/generations → GET /v1/videos/{id}) with heartbeat forwarding
to keep the SSE stream alive during the 30-180s generation window.
New files:
- src/images/xai-video-client.ts: submitVideoJob + pollVideoJob
- src/images/fulfill-video.ts: arg parsing, async polling generator, result builder
- tests/videos/: 32 tests (xai-video-client, fulfill-video, plan-video)
- docs-site/.../video-bridge.md: user guide
Modified files:
- src/images/loop.ts: unified image+video fulfillment, per-turn call caps
- src/images/types.ts: VideoBridgePlan/VideoCallResult shared types
- src/images/synthetic-tool.ts: buildVideoTool, VIDEO_GEN_TOOL_NAME
- src/images/plan.ts: planVideoBridge (opt-in via videoBridgeEnabled)
- src/images/artifacts.ts: downloadVideoToArtifact (200MB cap, SSRF-protected)
- src/server/responses/core.ts: video bridge wiring + tool dedup
- src/types.ts: videoGeneration flag, video config fields
Video bridge is opt-in (videoBridgeEnabled defaults to false). All upstream
image-bridge hardening preserved (SSRF, size caps, abort linking, pinned HTTPS).
* fix(videos): address Codex P1/P2 + CodeRabbit review findings
P1 fixes:
- Non-streaming requests no longer 400 when video bridge is enabled
(only image bridge requires stream=true; video-only skips bridge for
non-stream requests)
- Restore globalThis.fetch after video client tests (afterEach cleanup)
P2 fixes:
- Use clamped vidPlan.timeoutMs instead of raw config value
- Honor videoMaxRounds when both image+video bridges active (tighter wins)
- guessVideoExtFromMagic throws on unrecognized magic (was defaulting to mp4)
- Poll 4xx permanent failures (except 429) fail fast instead of retrying
- buildVideoResult uses pathToFileURL for cross-platform markdown links
- Initial poll interval restored to 5s (was 200ms)
- done-without-videoUrl returns error instead of spinning until timeout
- VideoBudget charges every streamed chunk (was only first chunk)
- downloadVideoToArtifact: reader cleanup on file-open failure
- readBoundedText: cancel reader body on size cap breach
- Namespaced video_gen tools preserved in dedup filter
- Fix JSON missing closing brace in docs
- Clarify API key auth requirement in docs prerequisites
* fix(videos): encode requestId, defer paidVideoCalls, guard plan! assertion
- Encode requestId in poll URL to prevent path injection (CodeRabbit)
- Move paidVideoCalls++ past arg validation so malformed calls don't burn budget
- Add early guard for undefined plan in image branch, removing non-null assertions
* fix(videos): batch-aware artifact pruning + clarify provider key docs
- Move pruneArtifacts from per-download to post-batch so multi-video turns
don't delete earlier videos before tool results are injected (Codex P2)
- Document that video bridge requires providers.xai with apiKey authMode,
not OAuth from 'ocx login xai' (Codex P2)
* fix(videos): web search coexistence, unified timeout budget, VideoBudget ceiling
Address Wibias CHANGES_REQUESTED review:
1. Web search coexistence (P1):
- When wsPlan is active, media bridge is skipped (existing behavior)
- Now emits console.warn so the user sees the skip instead of silent loss
- Documented priority rule in video-bridge.md
2. Timeout budget (P2):
- Start deadline BEFORE submitVideoJob, not after
- Poll receives remaining budget (deadline - now), min 5s floor
- Submit (60s) + poll now share one videoTimeoutMs deadline
3. VideoBudget aggregate ceiling (optional, addressed):
- VideoBudget now has a cap (600 MiB = 3 × single-download max)
- chargeVideoBudget() enforces the ceiling per-chunk during streaming
- Exceeding the budget throws mid-download (partial file is cleaned up)
4. New tests: done-without-videoUrl, permanent 4xx poll stop, requestId encoding
* fix(videos): gate tool_choice on imgPlan, shared deadline signal, buffer magic sniff
Address Wibias second CHANGES_REQUESTED review:
Required:
B. Gate image tool_choice rewriting on imgPlan — video-only turns no longer
rewrite image_generation/image_gen aliases to an undeclared tool (core.ts)
C. Shared timeout complete — deadline-bound AbortSignal passed into submitVideoJob;
if budget expired after submit, poll is skipped with timeout error (no 5s floor)
Preferred:
A. Buffer ≥12 bytes before magic-byte sniff — accumulate chunks until sniff minimum
so short first reads don't crash guessVideoExtFromMagic (artifacts.ts)
Optional:
D. Docs: scope web-search priority to runnable sidecar / non-runTurn path
* fix(videos): pass deadline-bound signal into poll generator
Pass linkedDeadline (not raw client signal) into pollVideoWithHeartbeats so
in-flight pollVideoJob fetches and sleep() calls abort when the wall-clock
budget expires. Abort error messages now distinguish deadline-expiry from
client-cancel based on elapsed time vs timeoutMs threshold.
* fix(videos): deadline abort classification, pruned-path guard, video aliases, DNS noun
Address Wibias fourth CHANGES_REQUESTED review:
1. Deadline abort classification (required #1):
- Replace 0.95 wall-clock heuristic with signal.reason.name === 'TimeoutError'
- deadlineSignal declared outside try so catch can check deadlineSignal.aborted
- Submit catch returns timeout error when deadlineSignal fires (not generic abort)
2. Don't deliver pruned artifact paths (required #2):
- After batch prune, scan fulfilled results and rewrite any whose path
no longer exists to ok:false with explanatory error
3. Wire video aliases (required #3):
- planVideoBridge seeds toolNames with VIDEO_GEN_TOOL_NAME + any request
function tools matching isVideoGenName (image-bridge parity)
- core.ts filter only strips unnamespaced video_gen aliases (namespaced
MCP tools are left alone)
4. Fix test fixtures (required #4):
- authMode 'api_key' → 'key' (valid OcxProviderConfig value)
5. DNS error noun (small):
- resolvePublicAddresses accepts optional noun param (default 'image')
- downloadVideoToArtifact passes 'video'
* fix(videos): complete pruned-path guard, skip namespaced aliases, mock sleep in tests
Address Wibias fifth CHANGES_REQUESTED review:
1. Finish pruned-path guard:
- Filter files[] through existsSync, not just path
- If all pruned: ok:false with no markdown/path
- If some survive: refresh path/files/count/markdown from survivors
2. Skip namespaced tools when seeding videoPlan.toolNames:
- if (t.namespace) continue — matches core.ts filter rule
3. Fix ubuntu CI timeout:
- Mock setTimeout in heartbeat test so 5s poll interval resolves instantly
- Suite now runs in ~156ms (was ~5s)
* fix(videos): stop hanging ubuntu CI with global setTimeout mock
cf76fb1 mocked globalThis.setTimeout to skip the 5s poll sleep. That races
parallel Bun workers and cancelled ubuntu-latest at the 12m job cap
(938f25e was green; cf76fb1 hung). Inject sleep/poll seams instead and drop
mock.module so video tests no longer poison each other.
---------
Co-authored-by: Your Name <[email protected]>
Co-authored-by: Wibias <[email protected]>1 parent a47aa3f commit 257c626
15 files changed
Lines changed: 1299 additions & 87 deletions
File tree
- docs-site/src/content/docs/guides
- src
- images
- lib
- server/responses
- tests/videos
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| |||
475 | 475 | | |
476 | 476 | | |
477 | 477 | | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
0 commit comments