Skip to content

Commit a62193c

Browse files
committed
feat(video): add xai and alibaba providers
1 parent 5e0b58f commit a62193c

19 files changed

Lines changed: 1193 additions & 11 deletions

CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,10 @@ Docs: https://docs.openclaw.ai
2929
- Agents/Claude CLI: expose OpenClaw tools to background Claude CLI runs through a loopback MCP bridge and switch bundled runs to stdin + `stream-json` partial-message streaming so prompts stop riding argv, long replies show live progress, and final session/usage metadata still land cleanly. (#35676) Thanks @mylukin.
3030
- ACPX/runtime: embed the ACP runtime directly in the bundled `acpx` plugin, remove the extra external ACP CLI hop, harden live ACP session binding and reuse, and add a generic `reply_dispatch` hook so bundled plugins like ACPX can own reply interception without hardcoded ACP paths in core auto-reply routing. (#61319)
3131
- Config/schema: enrich the exported `openclaw config schema` JSON Schema with field titles and descriptions so editors, agents, and other schema consumers receive the same config help metadata. (#60067) Thanks @solavrc.
32+
- Agents/cache: diagnostics: add prompt-cache break diagnostics, trace live cache scenarios through embedded runner paths, and show cache reuse explicitly in `openclaw status --verbose`. Thanks @vincentkoc.
33+
- Agents/cache: stabilize cache-relevant system prompt fingerprints by normalizing equivalent structured prompt whitespace, line endings, hook-added system context, and runtime capability ordering so semantically unchanged prompts reuse KV/cache more reliably. Thanks @vincentkoc.
34+
- Agents/tool prompts: remove the duplicate in-band tool inventory from agent system prompts so tool-calling models rely on the structured tool definitions as the single source of truth, improving prompt stability and reducing stale tool guidance.
35+
- Tools/video generation: add bundled xAI (`grok-imagine-video`) and Alibaba Model Studio Wan video providers, plus live-test/default model wiring for both.
3236
- Providers/CLI: remove bundled CLI text-provider backends and the `agents.defaults.cliBackends` surface, while keeping ACP harness sessions and Gemini media understanding on the native bundled providers.
3337
- Matrix/exec approvals: clarify unavailable-approval replies so Matrix no longer claims chat approvals are unsupported when native exec approvals are merely unconfigured. (#61424) Thanks @gumadeiras.
3438
- Docs/IRC: replace public IRC hostname examples with `irc.example.com` and recommend private servers for bot coordination while listing common public networks for intentional use.
@@ -101,6 +105,7 @@ Docs: https://docs.openclaw.ai
101105
- Agents/errors: surface an explicit disk-full message when local session or transcript writes fail with `ENOSPC`/`disk full`, so those runs stop degrading into opaque `NO_REPLY`-style failures. Thanks @vincentkoc.
102106
- Exec approvals: remove heuristic command-obfuscation gating from host exec so gateway and node runs rely on explicit policy, allowlist, and strict inline-eval rules only.
103107
- Agents/tool results: cap live tool-result persistence and overflow-recovery truncation at 40k characters so oversized tool output stays bounded without discarding recent context entirely.
108+
- Discord/video replies: split text-plus-video deliveries into a text reply followed by a media-only send, and let live provider auth checks honor manifest-declared API key env vars like `MODELSTUDIO_API_KEY`.
104109
- Config/All Settings: keep the raw config view intact when sensitive fields are blank instead of corrupting or dropping the rendered snapshot. (#28214) Thanks @solodmd.
105110
- Plugin SDK/facades: back-fill bundled plugin facade sentinels before plugin-id tracking re-enters config loading, so CLI/provider startup no longer crashes with `shouldNormalizeGoogleProviderConfig is not a function` or other empty-facade reads during bundled plugin re-entry. Thanks @adam91holt.
106111
- Plugins/facades: back-fill facade sentinels before tracked-plugin resolution re-enters config loading, so facade exports stay defined during circular provider normalization. (#61180) Thanks @adam91holt.

docs/tools/video-generation.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
summary: "Generate videos using configured providers such as OpenAI, Google, Qwen, and MiniMax"
2+
summary: "Generate videos using configured providers such as Alibaba, OpenAI, Google, Qwen, and MiniMax"
33
read_when:
44
- Generating videos via the agent
55
- Configuring video generation providers and models
@@ -17,7 +17,7 @@ The tool only appears when at least one video-generation provider is available.
1717

1818
## Quick start
1919

20-
1. Set an API key for at least one provider (for example `OPENAI_API_KEY`, `GEMINI_API_KEY`, or `QWEN_API_KEY`).
20+
1. Set an API key for at least one provider (for example `OPENAI_API_KEY`, `GEMINI_API_KEY`, `MODELSTUDIO_API_KEY`, or `QWEN_API_KEY`).
2121
2. Optionally set your preferred model:
2222

2323
```json5
@@ -38,13 +38,15 @@ The agent calls `video_generate` automatically. No tool allow-listing needed —
3838

3939
| Provider | Default model | Reference inputs | API key |
4040
| -------- | ------------------------------- | ------------------ | ---------------------------------------------------------- |
41+
| Alibaba | `wan2.6-t2v` | Yes, remote URLs | `MODELSTUDIO_API_KEY`, `DASHSCOPE_API_KEY`, `QWEN_API_KEY` |
4142
| BytePlus | `seedance-1-0-lite-t2v-250428` | 1 image | `BYTEPLUS_API_KEY` |
4243
| fal | `fal-ai/minimax/video-01-live` | 1 image | `FAL_KEY` |
4344
| Google | `veo-3.1-fast-generate-preview` | 1 image or 1 video | `GEMINI_API_KEY`, `GOOGLE_API_KEY` |
4445
| MiniMax | `MiniMax-Hailuo-2.3` | 1 image | `MINIMAX_API_KEY` |
4546
| OpenAI | `sora-2` | 1 image or 1 video | `OPENAI_API_KEY` |
4647
| Qwen | `wan2.6-t2v` | Yes, remote URLs | `QWEN_API_KEY`, `MODELSTUDIO_API_KEY`, `DASHSCOPE_API_KEY` |
4748
| Together | `Wan-AI/Wan2.2-T2V-A14B` | 1 image | `TOGETHER_API_KEY` |
49+
| xAI | `grok-imagine-video` | 1 image or 1 video | `XAI_API_KEY` |
4850

4951
Use `action: "list"` to inspect available providers and models at runtime:
5052

@@ -105,10 +107,12 @@ If a provider fails, the next candidate is tried automatically. If all fail, the
105107

106108
## Provider notes
107109

108-
- OpenAI uses the native video endpoint and currently defaults to `sora-2`.
110+
- Alibaba uses the DashScope / Model Studio async video endpoint and currently requires remote `http(s)` URLs for reference assets.
109111
- Google uses Gemini/Veo and supports a single image or video reference input.
110112
- MiniMax, Together, BytePlus, and fal currently support a single image reference input.
113+
- OpenAI uses the native video endpoint and currently defaults to `sora-2`.
111114
- Qwen supports image/video references, but the upstream DashScope video endpoint currently requires remote `http(s)` URLs for those references.
115+
- xAI uses the native xAI video API and supports text-to-video, image-to-video, and remote video edit/extend flows.
112116

113117
## Qwen reference inputs
114118

extensions/alibaba/index.ts

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
2+
import { buildAlibabaVideoGenerationProvider } from "./video-generation-provider.js";
3+
4+
export default definePluginEntry({
5+
id: "alibaba",
6+
name: "Alibaba Model Studio Plugin",
7+
description: "Bundled Alibaba Model Studio video provider plugin",
8+
register(api) {
9+
api.registerVideoGenerationProvider(buildAlibabaVideoGenerationProvider());
10+
},
11+
});
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{
2+
"id": "alibaba",
3+
"enabledByDefault": true,
4+
"providerAuthEnvVars": {
5+
"alibaba": ["MODELSTUDIO_API_KEY", "DASHSCOPE_API_KEY", "QWEN_API_KEY"]
6+
},
7+
"providerAuthChoices": [
8+
{
9+
"provider": "alibaba",
10+
"method": "api-key",
11+
"choiceId": "alibaba-model-studio-api-key",
12+
"choiceLabel": "Alibaba Model Studio API key",
13+
"groupId": "alibaba",
14+
"groupLabel": "Alibaba Model Studio",
15+
"groupHint": "DashScope / Model Studio API key",
16+
"optionKey": "alibabaModelStudioApiKey",
17+
"cliFlag": "--alibaba-model-studio-api-key",
18+
"cliOption": "--alibaba-model-studio-api-key <key>",
19+
"cliDescription": "Alibaba Model Studio API key"
20+
}
21+
],
22+
"contracts": {
23+
"videoGenerationProviders": ["alibaba"]
24+
},
25+
"configSchema": {
26+
"type": "object",
27+
"additionalProperties": false,
28+
"properties": {}
29+
}
30+
}

extensions/alibaba/package.json

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"name": "@openclaw/alibaba-provider",
3+
"version": "2026.4.5",
4+
"private": true,
5+
"description": "OpenClaw Alibaba Model Studio video provider plugin",
6+
"type": "module",
7+
"openclaw": {
8+
"extensions": [
9+
"./index.ts"
10+
]
11+
}
12+
}
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
import { describePluginRegistrationContract } from "../../test/helpers/plugins/plugin-registration-contract.js";
2+
3+
describePluginRegistrationContract({
4+
pluginId: "alibaba",
5+
videoGenerationProviderIds: ["alibaba"],
6+
requireGenerateVideo: true,
7+
});
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
import { afterEach, describe, expect, it, vi } from "vitest";
2+
import { buildAlibabaVideoGenerationProvider } from "./video-generation-provider.js";
3+
4+
const {
5+
resolveApiKeyForProviderMock,
6+
postJsonRequestMock,
7+
fetchWithTimeoutMock,
8+
assertOkOrThrowHttpErrorMock,
9+
resolveProviderHttpRequestConfigMock,
10+
} = vi.hoisted(() => ({
11+
resolveApiKeyForProviderMock: vi.fn(async () => ({ apiKey: "alibaba-key" })),
12+
postJsonRequestMock: vi.fn(),
13+
fetchWithTimeoutMock: vi.fn(),
14+
assertOkOrThrowHttpErrorMock: vi.fn(async () => {}),
15+
resolveProviderHttpRequestConfigMock: vi.fn((params) => ({
16+
baseUrl: params.baseUrl ?? params.defaultBaseUrl,
17+
allowPrivateNetwork: false,
18+
headers: new Headers(params.defaultHeaders),
19+
dispatcherPolicy: undefined,
20+
})),
21+
}));
22+
23+
vi.mock("openclaw/plugin-sdk/provider-auth-runtime", () => ({
24+
resolveApiKeyForProvider: resolveApiKeyForProviderMock,
25+
}));
26+
27+
vi.mock("openclaw/plugin-sdk/provider-http", () => ({
28+
assertOkOrThrowHttpError: assertOkOrThrowHttpErrorMock,
29+
fetchWithTimeout: fetchWithTimeoutMock,
30+
postJsonRequest: postJsonRequestMock,
31+
resolveProviderHttpRequestConfig: resolveProviderHttpRequestConfigMock,
32+
}));
33+
34+
describe("alibaba video generation provider", () => {
35+
afterEach(() => {
36+
resolveApiKeyForProviderMock.mockClear();
37+
postJsonRequestMock.mockReset();
38+
fetchWithTimeoutMock.mockReset();
39+
assertOkOrThrowHttpErrorMock.mockClear();
40+
resolveProviderHttpRequestConfigMock.mockClear();
41+
});
42+
43+
it("submits async Wan generation, polls task status, and downloads the resulting video", async () => {
44+
postJsonRequestMock.mockResolvedValue({
45+
response: {
46+
json: async () => ({
47+
request_id: "req-1",
48+
output: {
49+
task_id: "task-1",
50+
},
51+
}),
52+
},
53+
release: vi.fn(async () => {}),
54+
});
55+
fetchWithTimeoutMock
56+
.mockResolvedValueOnce({
57+
json: async () => ({
58+
output: {
59+
task_status: "SUCCEEDED",
60+
results: [{ video_url: "https://example.com/out.mp4" }],
61+
},
62+
}),
63+
headers: new Headers(),
64+
})
65+
.mockResolvedValueOnce({
66+
arrayBuffer: async () => Buffer.from("mp4-bytes"),
67+
headers: new Headers({ "content-type": "video/mp4" }),
68+
});
69+
70+
const provider = buildAlibabaVideoGenerationProvider();
71+
const result = await provider.generateVideo({
72+
provider: "alibaba",
73+
model: "wan2.6-r2v-flash",
74+
prompt: "animate this shot",
75+
cfg: {},
76+
inputImages: [{ url: "https://example.com/ref.png" }],
77+
durationSeconds: 6,
78+
audio: true,
79+
watermark: false,
80+
});
81+
82+
expect(postJsonRequestMock).toHaveBeenCalledWith(
83+
expect.objectContaining({
84+
url: "https://dashscope-intl.aliyuncs.com/api/v1/services/aigc/video-generation/video-synthesis",
85+
body: expect.objectContaining({
86+
model: "wan2.6-r2v-flash",
87+
input: expect.objectContaining({
88+
prompt: "animate this shot",
89+
img_url: "https://example.com/ref.png",
90+
}),
91+
parameters: expect.objectContaining({
92+
duration: 6,
93+
enable_audio: true,
94+
watermark: false,
95+
}),
96+
}),
97+
}),
98+
);
99+
expect(fetchWithTimeoutMock).toHaveBeenNthCalledWith(
100+
1,
101+
"https://dashscope-intl.aliyuncs.com/api/v1/tasks/task-1",
102+
expect.objectContaining({ method: "GET" }),
103+
120000,
104+
fetch,
105+
);
106+
expect(result.videos).toHaveLength(1);
107+
expect(result.videos[0]?.mimeType).toBe("video/mp4");
108+
expect(result.metadata).toEqual(
109+
expect.objectContaining({
110+
requestId: "req-1",
111+
taskId: "task-1",
112+
taskStatus: "SUCCEEDED",
113+
}),
114+
);
115+
});
116+
117+
it("fails fast when reference inputs are local buffers instead of remote URLs", async () => {
118+
const provider = buildAlibabaVideoGenerationProvider();
119+
120+
await expect(
121+
provider.generateVideo({
122+
provider: "alibaba",
123+
model: "wan2.6-i2v",
124+
prompt: "animate this local frame",
125+
cfg: {},
126+
inputImages: [{ buffer: Buffer.from("png-bytes"), mimeType: "image/png" }],
127+
}),
128+
).rejects.toThrow(
129+
"Alibaba Wan video generation currently requires remote http(s) URLs for reference images/videos.",
130+
);
131+
expect(postJsonRequestMock).not.toHaveBeenCalled();
132+
});
133+
});

0 commit comments

Comments
 (0)