Skip to content

Commit 257c626

Browse files
tizerluoYour NameWibias
authored
feat(videos): add Grok video bridge for non-OpenAI models (#582)
* feat(videos): add Grok video bridge for non-OpenAI models Extends the image bridge to support asynchronous video generation via xAI Grok Imagine Video. Video generation uses a submit→poll→download pattern (POST /v1/videos/generations → GET /v1/videos/{id}) with heartbeat forwarding to keep the SSE stream alive during the 30-180s generation window. New files: - src/images/xai-video-client.ts: submitVideoJob + pollVideoJob - src/images/fulfill-video.ts: arg parsing, async polling generator, result builder - tests/videos/: 32 tests (xai-video-client, fulfill-video, plan-video) - docs-site/.../video-bridge.md: user guide Modified files: - src/images/loop.ts: unified image+video fulfillment, per-turn call caps - src/images/types.ts: VideoBridgePlan/VideoCallResult shared types - src/images/synthetic-tool.ts: buildVideoTool, VIDEO_GEN_TOOL_NAME - src/images/plan.ts: planVideoBridge (opt-in via videoBridgeEnabled) - src/images/artifacts.ts: downloadVideoToArtifact (200MB cap, SSRF-protected) - src/server/responses/core.ts: video bridge wiring + tool dedup - src/types.ts: videoGeneration flag, video config fields Video bridge is opt-in (videoBridgeEnabled defaults to false). All upstream image-bridge hardening preserved (SSRF, size caps, abort linking, pinned HTTPS). * fix(videos): address Codex P1/P2 + CodeRabbit review findings P1 fixes: - Non-streaming requests no longer 400 when video bridge is enabled (only image bridge requires stream=true; video-only skips bridge for non-stream requests) - Restore globalThis.fetch after video client tests (afterEach cleanup) P2 fixes: - Use clamped vidPlan.timeoutMs instead of raw config value - Honor videoMaxRounds when both image+video bridges active (tighter wins) - guessVideoExtFromMagic throws on unrecognized magic (was defaulting to mp4) - Poll 4xx permanent failures (except 429) fail fast instead of retrying - buildVideoResult uses pathToFileURL for cross-platform markdown links - Initial poll interval restored to 5s (was 200ms) - done-without-videoUrl returns error instead of spinning until timeout - VideoBudget charges every streamed chunk (was only first chunk) - downloadVideoToArtifact: reader cleanup on file-open failure - readBoundedText: cancel reader body on size cap breach - Namespaced video_gen tools preserved in dedup filter - Fix JSON missing closing brace in docs - Clarify API key auth requirement in docs prerequisites * fix(videos): encode requestId, defer paidVideoCalls, guard plan! assertion - Encode requestId in poll URL to prevent path injection (CodeRabbit) - Move paidVideoCalls++ past arg validation so malformed calls don't burn budget - Add early guard for undefined plan in image branch, removing non-null assertions * fix(videos): batch-aware artifact pruning + clarify provider key docs - Move pruneArtifacts from per-download to post-batch so multi-video turns don't delete earlier videos before tool results are injected (Codex P2) - Document that video bridge requires providers.xai with apiKey authMode, not OAuth from 'ocx login xai' (Codex P2) * fix(videos): web search coexistence, unified timeout budget, VideoBudget ceiling Address Wibias CHANGES_REQUESTED review: 1. Web search coexistence (P1): - When wsPlan is active, media bridge is skipped (existing behavior) - Now emits console.warn so the user sees the skip instead of silent loss - Documented priority rule in video-bridge.md 2. Timeout budget (P2): - Start deadline BEFORE submitVideoJob, not after - Poll receives remaining budget (deadline - now), min 5s floor - Submit (60s) + poll now share one videoTimeoutMs deadline 3. VideoBudget aggregate ceiling (optional, addressed): - VideoBudget now has a cap (600 MiB = 3 × single-download max) - chargeVideoBudget() enforces the ceiling per-chunk during streaming - Exceeding the budget throws mid-download (partial file is cleaned up) 4. New tests: done-without-videoUrl, permanent 4xx poll stop, requestId encoding * fix(videos): gate tool_choice on imgPlan, shared deadline signal, buffer magic sniff Address Wibias second CHANGES_REQUESTED review: Required: B. Gate image tool_choice rewriting on imgPlan — video-only turns no longer rewrite image_generation/image_gen aliases to an undeclared tool (core.ts) C. Shared timeout complete — deadline-bound AbortSignal passed into submitVideoJob; if budget expired after submit, poll is skipped with timeout error (no 5s floor) Preferred: A. Buffer ≥12 bytes before magic-byte sniff — accumulate chunks until sniff minimum so short first reads don't crash guessVideoExtFromMagic (artifacts.ts) Optional: D. Docs: scope web-search priority to runnable sidecar / non-runTurn path * fix(videos): pass deadline-bound signal into poll generator Pass linkedDeadline (not raw client signal) into pollVideoWithHeartbeats so in-flight pollVideoJob fetches and sleep() calls abort when the wall-clock budget expires. Abort error messages now distinguish deadline-expiry from client-cancel based on elapsed time vs timeoutMs threshold. * fix(videos): deadline abort classification, pruned-path guard, video aliases, DNS noun Address Wibias fourth CHANGES_REQUESTED review: 1. Deadline abort classification (required #1): - Replace 0.95 wall-clock heuristic with signal.reason.name === 'TimeoutError' - deadlineSignal declared outside try so catch can check deadlineSignal.aborted - Submit catch returns timeout error when deadlineSignal fires (not generic abort) 2. Don't deliver pruned artifact paths (required #2): - After batch prune, scan fulfilled results and rewrite any whose path no longer exists to ok:false with explanatory error 3. Wire video aliases (required #3): - planVideoBridge seeds toolNames with VIDEO_GEN_TOOL_NAME + any request function tools matching isVideoGenName (image-bridge parity) - core.ts filter only strips unnamespaced video_gen aliases (namespaced MCP tools are left alone) 4. Fix test fixtures (required #4): - authMode 'api_key' → 'key' (valid OcxProviderConfig value) 5. DNS error noun (small): - resolvePublicAddresses accepts optional noun param (default 'image') - downloadVideoToArtifact passes 'video' * fix(videos): complete pruned-path guard, skip namespaced aliases, mock sleep in tests Address Wibias fifth CHANGES_REQUESTED review: 1. Finish pruned-path guard: - Filter files[] through existsSync, not just path - If all pruned: ok:false with no markdown/path - If some survive: refresh path/files/count/markdown from survivors 2. Skip namespaced tools when seeding videoPlan.toolNames: - if (t.namespace) continue — matches core.ts filter rule 3. Fix ubuntu CI timeout: - Mock setTimeout in heartbeat test so 5s poll interval resolves instantly - Suite now runs in ~156ms (was ~5s) * fix(videos): stop hanging ubuntu CI with global setTimeout mock cf76fb1 mocked globalThis.setTimeout to skip the 5s poll sleep. That races parallel Bun workers and cancelled ubuntu-latest at the 12m job cap (938f25e was green; cf76fb1 hung). Inject sleep/poll seams instead and drop mock.module so video tests no longer poison each other. --------- Co-authored-by: Your Name <[email protected]> Co-authored-by: Wibias <[email protected]>
1 parent a47aa3f commit 257c626

15 files changed

Lines changed: 1299 additions & 87 deletions

File tree

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
---
2+
title: Video Bridge
3+
description: Generate videos with Grok Imagine Video through a non-OpenAI model.
4+
---
5+
6+
## Overview
7+
8+
The Video Bridge lets you use xAI's Grok Imagine Video generation through any non-OpenAI model
9+
routed by opencodex. When enabled, a synthetic `video_gen` tool is injected into the conversation.
10+
The model calls it like any function tool; opencodex intercepts the call, submits a video generation
11+
job to xAI, polls until completion, and downloads the result.
12+
13+
## Prerequisites
14+
15+
- An `xai` provider entry with an **API key** (`ocx login xai` alone is not sufficient — the video bridge requires key auth, not OAuth)
16+
- A non-OpenAI model as your routed provider (e.g. Anthropic Claude, Google Gemini)
17+
- opencodex configured to route through the non-OpenAI provider
18+
19+
> **⚠ Provider key required:** The video bridge only activates when the `xai` provider uses
20+
> API key auth. Add this to your config:
21+
>
22+
> ```json
23+
> {
24+
> "providers": {
25+
> "xai": { "adapter": "openai-chat", "apiKey": "xai-…", "authMode": "key" }
26+
> }
27+
> }
28+
> ```
29+
>
30+
> If you onboarded via `ocx login xai` (OAuth), the provider stays in `authMode: "oauth"`
31+
> and the bridge silently won't activate. Set `XAI_API_KEY` in the environment **or**
32+
> hard-code the key as shown above.
33+
34+
## Configuration
35+
36+
Add `videoBridgeEnabled: true` to your `images` config:
37+
38+
```json
39+
{
40+
"images": {
41+
"bridgeEnabled": true,
42+
"videoBridgeEnabled": true,
43+
"videoBridgeModel": "grok-imagine-video",
44+
"videoMaxRounds": 2,
45+
"videoTimeoutMs": 300000
46+
}
47+
}
48+
```
49+
50+
| Option | Default | Description |
51+
|--------|---------|-------------|
52+
| `videoBridgeEnabled` | `false` | Master switch. Must be explicitly enabled. |
53+
| `videoBridgeModel` | `"grok-imagine-video"` | xAI video model id. |
54+
| `videoMaxRounds` | `2` | Max video-gen rounds before forced final answer. |
55+
| `videoTimeoutMs` | `300000` (5 min) | Per-video timeout including polling. |
56+
57+
## How It Works
58+
59+
1. opencodex detects a non-OpenAI routed model with `videoBridgeEnabled: true`
60+
2. A synthetic `video_gen` function tool is injected into the conversation
61+
3. When the model calls `video_gen`, opencodex submits a job to xAI's `/videos/generations`
62+
4. The bridge polls the job status every 5-15 seconds, sending heartbeat messages to keep the stream alive
63+
5. When the video is ready, it's downloaded to the artifacts directory
64+
6. The local file path is returned to the model as a tool result
65+
66+
## Supported Parameters
67+
68+
The `video_gen` tool accepts:
69+
70+
| Parameter | Type | Range | Description |
71+
|-----------|------|-------|-------------|
72+
| `prompt` | string | required | Detailed video generation prompt |
73+
| `duration` | integer | 1-15 | Video length in seconds |
74+
| `resolution` | string | `"480p"`, `"720p"` | Video resolution |
75+
| `aspect_ratio` | string | 7 ratios | `16:9`, `9:16`, `1:1`, `4:3`, `3:4`, `3:2`, `2:3` |
76+
77+
## Limitations
78+
79+
- **xAI only**: Video generation is only available through xAI's Grok Imagine Video API
80+
- **Asynchronous**: Video generation takes 30-120 seconds
81+
- **Cost**: Video generation is a paid xAI feature (~$0.05/sec @480p, ~$0.07/sec @720p)
82+
- **One video per call**: Each `video_gen` call produces one video
83+
- **Coexists with Image Bridge**: Both bridges can be enabled simultaneously
84+
- **Web search priority**: When a web search sidecar is active for a turn (non-`runTurn` adapter), the video bridge is skipped — the two cannot run concurrently. A `console.warn` is emitted so you can detect this in logs.
85+
- **Timeout covers submit + poll**: The `videoTimeoutMs` budget starts before job submission, so the submit call (60 s) and subsequent polling share the same deadline.

‎src/images/artifacts.ts‎

Lines changed: 145 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { readdirSync, readFileSync, statSync, unlinkSync, existsSync } from "node:fs";
2-
import { mkdir, writeFile } from "node:fs/promises";
2+
import { mkdir, writeFile, open, unlink } from "node:fs/promises";
33
import type { IncomingMessage } from "node:http";
44
import https from "node:https";
55
import type { RequestOptions } from "node:https";
@@ -475,3 +475,147 @@ export async function downloadImageToArtifact(
475475
// Retention is post-batch via pruneArtifacts (see fulfill.ts).
476476
return writeArtifactUnique(dir, "dl-", bytes, ext);
477477
}
478+
479+
const MAX_VIDEO_DOWNLOAD_BYTES = 200 * 1024 * 1024; // 200 MiB
480+
/** Aggregate per-turn video download cap (600 MiB = 3 × max single download). */
481+
const MAX_VIDEO_BYTES_PER_TURN = MAX_VIDEO_DOWNLOAD_BYTES * 3;
482+
483+
export interface VideoBudget {
484+
spent: number;
485+
/** Ceiling on total bytes across all downloads this turn. */
486+
cap: number;
487+
}
488+
489+
export function createVideoBudget(): VideoBudget {
490+
return { spent: 0, cap: MAX_VIDEO_BYTES_PER_TURN };
491+
}
492+
493+
/** Charge bytes to the budget; returns false if the ceiling would be exceeded. */
494+
export function chargeVideoBudget(budget: VideoBudget, bytes: number): boolean {
495+
if (budget.spent + bytes > budget.cap) return false;
496+
budget.spent += bytes;
497+
return true;
498+
}
499+
500+
export function guessVideoExtFromMagic(bytes: Uint8Array): string {
501+
if (bytes.byteLength < 12) throw new Error("video data too short for magic byte sniffing");
502+
const sig = Buffer.from(bytes.slice(0, 12)).toString("latin1");
503+
// MP4/QuickTime/MOV: bytes 4-7 == "ftyp" (ISO BMFF)
504+
if (sig.slice(4, 8) === "ftyp") return "mp4";
505+
// WebM/Matroska: \x1a\x45\xdf\xa3
506+
if (sig.startsWith("\x1a\x45\xdf\xa3")) return "webm";
507+
throw new Error("unrecognized video format — magic bytes do not match MP4 or WebM");
508+
}
509+
510+
/**
511+
* Download a video from a URL to an artifact file with a 200 MiB hard cap, streaming the body
512+
* to disk to avoid buffering. SSRF protection reuses the same destination policy + pinned HTTPS
513+
* as image downloads. Format is sniffed from magic bytes.
514+
*/
515+
export async function downloadVideoToArtifact(
516+
url: string,
517+
budget?: VideoBudget,
518+
signal?: AbortSignal,
519+
): Promise<string> {
520+
// For data: URLs, handle inline (unlikely for video but keep parity)
521+
if (url.startsWith("data:")) {
522+
const commaIdx = url.indexOf(",");
523+
const meta = url.slice(0, commaIdx);
524+
const data = url.slice(commaIdx + 1);
525+
const isBase64 = meta.includes(";base64");
526+
if (!isBase64) throw new Error("non-base64 data URI for video is not supported");
527+
const buf = Buffer.from(data, "base64");
528+
if (budget && !chargeVideoBudget(budget, buf.byteLength)) {
529+
throw new Error("video data URI exceeds per-turn download budget");
530+
}
531+
if (buf.byteLength > MAX_VIDEO_DOWNLOAD_BYTES) throw new Error("video data URI exceeds size cap");
532+
const ext = guessVideoExtFromMagic(buf);
533+
const dir = getArtifactsDir();
534+
await mkdir(dir, { recursive: true, mode: 0o700 });
535+
const name = `vid-${timestampPrefix()}-${crypto.randomUUID()}.${ext}`;
536+
const dest = join(dir, name);
537+
await writeFile(dest, buf, { mode: 0o600 });
538+
return dest;
539+
}
540+
541+
// SSRF protection: same validation as downloadImageToArtifact
542+
let parsedUrl: URL;
543+
try { parsedUrl = new URL(url); } catch { throw new Error("video URL is not valid"); }
544+
if (parsedUrl.protocol !== "https:") {
545+
throw new Error(`video URL must use HTTPS, got ${parsedUrl.protocol}`);
546+
}
547+
const assessment = assessUrlDestination(url);
548+
if (assessment && assessment.kind !== "public" && assessment.kind !== "hostname") {
549+
throw new Error(`video URL targets ${assessment.detail}`);
550+
}
551+
const resolved = await resolvePublicAddresses(url, "video");
552+
const pinned = pickPinnedAddress(resolved.addresses);
553+
const resp = await pinnedHttpsGet(url, pinned, signal, { maxBytes: MAX_VIDEO_DOWNLOAD_BYTES });
554+
if (!resp.ok) {
555+
try { await resp.body?.cancel(); } catch { /* ignore */ }
556+
throw new Error("video download failed: " + resp.status);
557+
}
558+
559+
const dir = getArtifactsDir();
560+
await mkdir(dir, { recursive: true, mode: 0o700 });
561+
562+
const reader = resp.body?.getReader();
563+
if (!reader) throw new Error("video download returned no body");
564+
565+
// Buffer at least 12 bytes for magic-byte sniffing before opening the file.
566+
// A single read() can return fewer bytes; accumulate until we have enough.
567+
let dest: string | undefined;
568+
let fh: { close(): Promise<void>; writeFile(data: Uint8Array): Promise<void> } | undefined;
569+
try {
570+
const sniffChunks: Uint8Array[] = [];
571+
let sniffLen = 0;
572+
while (sniffLen < 12) {
573+
const { value, done } = await reader.read();
574+
if (done) break;
575+
if (!value) continue;
576+
sniffChunks.push(value);
577+
sniffLen += value.byteLength;
578+
}
579+
if (sniffLen === 0) {
580+
throw new Error("video download returned empty body");
581+
}
582+
const sniffBuf = Buffer.concat(sniffChunks);
583+
const ext = guessVideoExtFromMagic(new Uint8Array(sniffBuf));
584+
const name = `vid-${timestampPrefix()}-${crypto.randomUUID()}.${ext}`;
585+
dest = join(dir, name);
586+
fh = await open(dest, "w", 0o600);
587+
// Write all buffered chunks and set up accounting
588+
let totalBytes = sniffBuf.byteLength;
589+
if (budget && !chargeVideoBudget(budget, totalBytes)) {
590+
throw new Error("video download exceeds per-turn budget");
591+
}
592+
if (totalBytes > MAX_VIDEO_DOWNLOAD_BYTES) {
593+
throw new Error("video download exceeds size cap");
594+
}
595+
await fh.writeFile(new Uint8Array(sniffBuf));
596+
for (;;) {
597+
const { value, done } = await reader.read();
598+
if (done) break;
599+
totalBytes += value.byteLength;
600+
if (budget && !chargeVideoBudget(budget, value.byteLength)) {
601+
throw new Error("video download exceeds per-turn budget");
602+
}
603+
if (totalBytes > MAX_VIDEO_DOWNLOAD_BYTES) {
604+
throw new Error("video download exceeds size cap");
605+
}
606+
await fh.writeFile(value);
607+
}
608+
await fh.close();
609+
try { await reader.cancel(); } catch { /* ignore */ }
610+
reader.releaseLock();
611+
return dest;
612+
} catch (err) {
613+
try { await reader.cancel(); } catch { /* ignore */ }
614+
reader.releaseLock();
615+
if (fh) {
616+
try { await fh.close(); } catch { /* ignore */ }
617+
}
618+
if (dest) await unlink(dest).catch(() => {});
619+
throw err;
620+
}
621+
}

0 commit comments

Comments
 (0)