Skip to content

Commit d0ac1b0

Browse files
authored
feat: add PDF analysis tool with native provider support (#31319)
* feat: add PDF analysis tool with native provider support New `pdf` tool for analyzing PDF documents with model-powered analysis. Architecture: - Native PDF path: sends raw PDF bytes directly to providers that support inline document input (Anthropic via DocumentBlockParam, Google Gemini via inlineData with application/pdf MIME type) - Extraction fallback: for providers without native PDF support, extracts text via pdfjs-dist and rasterizes pages to images via @napi-rs/canvas, then sends through the standard vision/text completion path Key features: - Single PDF (`pdf` param) or multiple PDFs (`pdfs` array, up to 10) - Page range selection (`pages` param, e.g. "1-5", "1,3,7-9") - Model override (`model` param) and file size limits (`maxBytesMb`) - Auto-detects provider capability and falls back gracefully - Same security patterns as image tool (SSRF guards, sandbox support, local path roots, workspace-only policy) Config (agents.defaults): - pdfModel: primary/fallbacks (defaults to imageModel, then session model) - pdfMaxBytesMb: max PDF file size (default: 10) - pdfMaxPages: max pages to process (default: 20) Model catalog: - Extended ModelInputType to include "document" alongside "text"/"image" - Added modelSupportsDocument() capability check Files: - src/agents/tools/pdf-tool.ts - main tool factory - src/agents/tools/pdf-tool.helpers.ts - helpers (page range, config, etc.) - src/agents/tools/pdf-native-providers.ts - direct API calls for Anthropic/Google - src/agents/tools/pdf-tool.test.ts - 43 tests covering all paths - Modified: model-catalog.ts, openclaw-tools.ts, config schema/types/labels/help * fix: prepare pdf tool for merge (#31319) (thanks @tyler6204)
1 parent 31b6e58 commit d0ac1b0

17 files changed

Lines changed: 2008 additions & 100 deletions

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Docs: https://docs.openclaw.ai
99
- CLI/Config validation: add `openclaw config validate` (with `--json`) to validate config files before gateway startup, and include detailed invalid-key paths in startup invalid-config errors. (#31220) thanks @Sid-Qin.
1010
- Sessions/Attachments: add inline file attachment support for `sessions_spawn` (subagent runtime only) with base64/utf8 encoding, transcript content redaction, lifecycle cleanup, and configurable limits via `tools.sessions_spawn.attachments`. (#16761) Thanks @napetrov.
1111
- Agents/Thinking defaults: set `adaptive` as the default thinking level for Anthropic Claude 4.6 models (including Bedrock Claude 4.6 refs) while keeping other reasoning-capable models at `low` unless explicitly configured.
12+
- Tools/PDF analysis: add a first-class `pdf` tool with native Anthropic and Google PDF provider support, extraction fallback for non-native models, configurable defaults (`agents.defaults.pdfModel`, `pdfMaxBytesMb`, `pdfMaxPages`), and docs/tests covering routing, validation, and registration. (#31319) Thanks @tyler6204.
1213
- Gateway/Container probes: add built-in HTTP liveness/readiness endpoints (`/health`, `/healthz`, `/ready`, `/readyz`) for Docker/Kubernetes health checks, with fallback routing so existing handlers on those paths are not shadowed. (#31272) Thanks @vincentkoc.
1314
- Android/Nodes: add `camera.list`, `device.permissions`, `device.health`, and `notifications.actions` (`open`/`dismiss`/`reply`) on Android nodes, plus first-class node-tool actions for the new device/notification commands. (#28260) Thanks @obviyus.
1415
- Discord/Thread bindings: replace fixed TTL lifecycle with inactivity (`idleHours`, default 24h) plus optional hard `maxAgeHours` lifecycle controls, and add `/session idle` + `/session max-age` commands for focused thread-bound sessions. (#27845) Thanks @osolmaz.

docs/gateway/configuration-reference.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -835,6 +835,12 @@ Time format in system prompt. Default: `auto` (OS preference).
835835
primary: "openrouter/qwen/qwen-2.5-vl-72b-instruct:free",
836836
fallbacks: ["openrouter/google/gemini-2.0-flash-vision:free"],
837837
},
838+
pdfModel: {
839+
primary: "anthropic/claude-opus-4-6",
840+
fallbacks: ["openai/gpt-5-mini"],
841+
},
842+
pdfMaxBytesMb: 10,
843+
pdfMaxPages: 20,
838844
thinkingDefault: "low",
839845
verboseDefault: "off",
840846
elevatedDefault: "on",
@@ -853,6 +859,11 @@ Time format in system prompt. Default: `auto` (OS preference).
853859
- `imageModel`: accepts either a string (`"provider/model"`) or an object (`{ primary, fallbacks }`).
854860
- Used by the `image` tool path as its vision-model config.
855861
- Also used as fallback routing when the selected/default model cannot accept image input.
862+
- `pdfModel`: accepts either a string (`"provider/model"`) or an object (`{ primary, fallbacks }`).
863+
- Used by the `pdf` tool for model routing.
864+
- If omitted, the PDF tool falls back to `imageModel`, then to best-effort provider defaults.
865+
- `pdfMaxBytesMb`: default PDF size limit for the `pdf` tool when `maxBytesMb` is not passed at call time.
866+
- `pdfMaxPages`: default maximum pages considered by extraction fallback mode in the `pdf` tool.
856867
- `model.primary`: format `provider/model` (e.g. `anthropic/claude-opus-4-6`). If you omit the provider, OpenClaw assumes `anthropic` (deprecated).
857868
- `models`: the configured model catalog and allowlist for `/model`. Each entry can include `alias` (shortcut) and `params` (provider-specific, for example `temperature`, `maxTokens`, `cacheRetention`, `context1m`).
858869
- `params` merge precedence (config): `agents.defaults.models["provider/model"].params` is the base, then `agents.list[].params` (matching agent id) overrides by key.

docs/tools/index.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,26 @@ Notes:
397397
- Only available when `agents.defaults.imageModel` is configured (primary or fallbacks), or when an implicit image model can be inferred from your default model + configured auth (best-effort pairing).
398398
- Uses the image model directly (independent of the main chat model).
399399

400+
### `pdf`
401+
402+
Analyze one or more PDF documents.
403+
404+
Core parameters:
405+
406+
- `pdf` (single path or URL)
407+
- `pdfs` (multiple paths or URLs, up to 10)
408+
- `prompt` (optional, defaults to "Analyze this PDF document.")
409+
- `pages` (optional page range like `1-5` or `1,3,7-9`)
410+
- `model` (optional model override)
411+
- `maxBytesMb` (optional size cap)
412+
413+
Notes:
414+
415+
- Native PDF provider mode is supported for Anthropic and Google models.
416+
- Non-native models use PDF extraction fallback, text first, then rasterized page images when needed.
417+
- `pages` filtering is only supported in extraction fallback mode. Native providers return a clear error when `pages` is set.
418+
- Defaults are configurable via `agents.defaults.pdfModel`, `agents.defaults.pdfMaxBytesMb`, and `agents.defaults.pdfMaxPages`.
419+
400420
### `message`
401421

402422
Send messages and channel actions across Discord/Google Chat/Slack/Telegram/WhatsApp/Signal/iMessage/MS Teams.

src/agents/model-catalog.ts

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,15 @@ import { ensureOpenClawModelsJson } from "./models-config.js";
55

66
const log = createSubsystemLogger("model-catalog");
77

8+
export type ModelInputType = "text" | "image" | "document";
9+
810
export type ModelCatalogEntry = {
911
id: string;
1012
name: string;
1113
provider: string;
1214
contextWindow?: number;
1315
reasoning?: boolean;
14-
input?: Array<"text" | "image">;
16+
input?: ModelInputType[];
1517
};
1618

1719
type DiscoveredModel = {
@@ -20,7 +22,7 @@ type DiscoveredModel = {
2022
provider: string;
2123
contextWindow?: number;
2224
reasoning?: boolean;
23-
input?: Array<"text" | "image">;
25+
input?: ModelInputType[];
2426
};
2527

2628
type PiSdkModule = typeof import("./pi-model-discovery.js");
@@ -60,12 +62,12 @@ function applyOpenAICodexSparkFallback(models: ModelCatalogEntry[]): void {
6062
});
6163
}
6264

63-
function normalizeConfiguredModelInput(input: unknown): Array<"text" | "image"> | undefined {
65+
function normalizeConfiguredModelInput(input: unknown): ModelInputType[] | undefined {
6466
if (!Array.isArray(input)) {
6567
return undefined;
6668
}
6769
const normalized = input.filter(
68-
(item): item is "text" | "image" => item === "text" || item === "image",
70+
(item): item is ModelInputType => item === "text" || item === "image" || item === "document",
6971
);
7072
return normalized.length > 0 ? normalized : undefined;
7173
}
@@ -248,6 +250,13 @@ export function modelSupportsVision(entry: ModelCatalogEntry | undefined): boole
248250
return entry?.input?.includes("image") ?? false;
249251
}
250252

253+
/**
254+
* Check if a model supports native document/PDF input based on its catalog entry.
255+
*/
256+
export function modelSupportsDocument(entry: ModelCatalogEntry | undefined): boolean {
257+
return entry?.input?.includes("document") ?? false;
258+
}
259+
251260
/**
252261
* Find a model in the catalog by provider and model ID.
253262
*/
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
import fs from "node:fs/promises";
2+
import os from "node:os";
3+
import path from "node:path";
4+
import { describe, expect, it } from "vitest";
5+
import type { OpenClawConfig } from "../config/config.js";
6+
import "./test-helpers/fast-core-tools.js";
7+
import { createOpenClawTools } from "./openclaw-tools.js";
8+
9+
async function withTempAgentDir<T>(run: (agentDir: string) => Promise<T>): Promise<T> {
10+
const agentDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-tools-pdf-"));
11+
try {
12+
return await run(agentDir);
13+
} finally {
14+
await fs.rm(agentDir, { recursive: true, force: true });
15+
}
16+
}
17+
18+
describe("createOpenClawTools PDF registration", () => {
19+
it("includes pdf tool when pdfModel is configured", async () => {
20+
await withTempAgentDir(async (agentDir) => {
21+
const cfg: OpenClawConfig = {
22+
agents: {
23+
defaults: {
24+
pdfModel: { primary: "openai/gpt-5-mini" },
25+
},
26+
},
27+
};
28+
29+
const tools = createOpenClawTools({ config: cfg, agentDir });
30+
expect(tools.some((tool) => tool.name === "pdf")).toBe(true);
31+
});
32+
});
33+
});

src/agents/openclaw-tools.ts

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import { createGatewayTool } from "./tools/gateway-tool.js";
1313
import { createImageTool } from "./tools/image-tool.js";
1414
import { createMessageTool } from "./tools/message-tool.js";
1515
import { createNodesTool } from "./tools/nodes-tool.js";
16+
import { createPdfTool } from "./tools/pdf-tool.js";
1617
import { createSessionStatusTool } from "./tools/session-status-tool.js";
1718
import { createSessionsHistoryTool } from "./tools/sessions-history-tool.js";
1819
import { createSessionsListTool } from "./tools/sessions-list-tool.js";
@@ -84,6 +85,18 @@ export function createOpenClawTools(options?: {
8485
modelHasVision: options?.modelHasVision,
8586
})
8687
: null;
88+
const pdfTool = options?.agentDir?.trim()
89+
? createPdfTool({
90+
config: options?.config,
91+
agentDir: options.agentDir,
92+
workspaceDir,
93+
sandbox:
94+
options?.sandboxRoot && options?.sandboxFsBridge
95+
? { root: options.sandboxRoot, bridge: options.sandboxFsBridge }
96+
: undefined,
97+
fsPolicy: options?.fsPolicy,
98+
})
99+
: null;
87100
const webSearchTool = createWebSearchTool({
88101
config: options?.config,
89102
sandboxed: options?.sandboxed,
@@ -173,6 +186,7 @@ export function createOpenClawTools(options?: {
173186
...(webSearchTool ? [webSearchTool] : []),
174187
...(webFetchTool ? [webFetchTool] : []),
175188
...(imageTool ? [imageTool] : []),
189+
...(pdfTool ? [pdfTool] : []),
176190
];
177191

178192
const pluginTools = resolvePluginTools({
Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
/**
2+
* Direct SDK/HTTP calls for providers that support native PDF document input.
3+
* This bypasses pi-ai's content type system which does not have a "document" type.
4+
*/
5+
6+
import { isRecord } from "../../utils.js";
7+
import { normalizeSecretInput } from "../../utils/normalize-secret-input.js";
8+
9+
type PdfInput = {
10+
base64: string;
11+
filename?: string;
12+
};
13+
14+
// ---------------------------------------------------------------------------
15+
// Anthropic – native PDF via Messages API
16+
// ---------------------------------------------------------------------------
17+
18+
type AnthropicDocBlock = {
19+
type: "document";
20+
source: {
21+
type: "base64";
22+
media_type: "application/pdf";
23+
data: string;
24+
};
25+
};
26+
27+
type AnthropicTextBlock = {
28+
type: "text";
29+
text: string;
30+
};
31+
32+
type AnthropicContentBlock = AnthropicDocBlock | AnthropicTextBlock;
33+
34+
type AnthropicResponseContent = Array<{ type: string; text?: string }>;
35+
36+
export async function anthropicAnalyzePdf(params: {
37+
apiKey: string;
38+
modelId: string;
39+
prompt: string;
40+
pdfs: PdfInput[];
41+
maxTokens?: number;
42+
baseUrl?: string;
43+
}): Promise<string> {
44+
const apiKey = normalizeSecretInput(params.apiKey);
45+
if (!apiKey) {
46+
throw new Error("Anthropic PDF: apiKey required");
47+
}
48+
49+
const content: AnthropicContentBlock[] = [];
50+
for (const pdf of params.pdfs) {
51+
content.push({
52+
type: "document",
53+
source: {
54+
type: "base64",
55+
media_type: "application/pdf",
56+
data: pdf.base64,
57+
},
58+
});
59+
}
60+
content.push({ type: "text", text: params.prompt });
61+
62+
const baseUrl = (params.baseUrl ?? "https://api.anthropic.com").replace(/\/+$/, "");
63+
const res = await fetch(`${baseUrl}/v1/messages`, {
64+
method: "POST",
65+
headers: {
66+
"Content-Type": "application/json",
67+
"x-api-key": apiKey,
68+
"anthropic-version": "2023-06-01",
69+
"anthropic-beta": "pdfs-2024-09-25",
70+
},
71+
body: JSON.stringify({
72+
model: params.modelId,
73+
max_tokens: params.maxTokens ?? 4096,
74+
messages: [{ role: "user", content }],
75+
}),
76+
});
77+
78+
if (!res.ok) {
79+
const body = await res.text().catch(() => "");
80+
throw new Error(
81+
`Anthropic PDF request failed (${res.status} ${res.statusText})${body ? `: ${body.slice(0, 400)}` : ""}`,
82+
);
83+
}
84+
85+
const json = (await res.json().catch(() => null)) as unknown;
86+
if (!isRecord(json)) {
87+
throw new Error("Anthropic PDF response was not JSON.");
88+
}
89+
90+
const responseContent = json.content as AnthropicResponseContent | undefined;
91+
if (!Array.isArray(responseContent)) {
92+
throw new Error("Anthropic PDF response missing content array.");
93+
}
94+
95+
const text = responseContent
96+
.filter((block) => block.type === "text" && typeof block.text === "string")
97+
.map((block) => block.text!)
98+
.join("");
99+
100+
if (!text.trim()) {
101+
throw new Error("Anthropic PDF returned no text.");
102+
}
103+
104+
return text.trim();
105+
}
106+
107+
// ---------------------------------------------------------------------------
108+
// Google Gemini – native PDF via generateContent API
109+
// ---------------------------------------------------------------------------
110+
111+
type GeminiPart = { inline_data: { mime_type: string; data: string } } | { text: string };
112+
113+
type GeminiCandidate = {
114+
content?: { parts?: Array<{ text?: string }> };
115+
};
116+
117+
export async function geminiAnalyzePdf(params: {
118+
apiKey: string;
119+
modelId: string;
120+
prompt: string;
121+
pdfs: PdfInput[];
122+
baseUrl?: string;
123+
}): Promise<string> {
124+
const apiKey = normalizeSecretInput(params.apiKey);
125+
if (!apiKey) {
126+
throw new Error("Gemini PDF: apiKey required");
127+
}
128+
129+
const parts: GeminiPart[] = [];
130+
for (const pdf of params.pdfs) {
131+
parts.push({
132+
inline_data: {
133+
mime_type: "application/pdf",
134+
data: pdf.base64,
135+
},
136+
});
137+
}
138+
parts.push({ text: params.prompt });
139+
140+
const baseUrl = (params.baseUrl ?? "https://generativelanguage.googleapis.com").replace(
141+
/\/+$/,
142+
"",
143+
);
144+
const url = `${baseUrl}/v1beta/models/${encodeURIComponent(params.modelId)}:generateContent?key=${encodeURIComponent(apiKey)}`;
145+
146+
const res = await fetch(url, {
147+
method: "POST",
148+
headers: { "Content-Type": "application/json" },
149+
body: JSON.stringify({
150+
contents: [{ role: "user", parts }],
151+
}),
152+
});
153+
154+
if (!res.ok) {
155+
const body = await res.text().catch(() => "");
156+
throw new Error(
157+
`Gemini PDF request failed (${res.status} ${res.statusText})${body ? `: ${body.slice(0, 400)}` : ""}`,
158+
);
159+
}
160+
161+
const json = (await res.json().catch(() => null)) as unknown;
162+
if (!isRecord(json)) {
163+
throw new Error("Gemini PDF response was not JSON.");
164+
}
165+
166+
const candidates = json.candidates as GeminiCandidate[] | undefined;
167+
if (!Array.isArray(candidates) || candidates.length === 0) {
168+
throw new Error("Gemini PDF returned no candidates.");
169+
}
170+
171+
const textParts = candidates[0].content?.parts?.filter((p) => typeof p.text === "string") ?? [];
172+
const text = textParts.map((p) => p.text!).join("");
173+
174+
if (!text.trim()) {
175+
throw new Error("Gemini PDF returned no text.");
176+
}
177+
178+
return text.trim();
179+
}

0 commit comments

Comments
 (0)