Summary
Think's built-in workspace read tool currently treats every file as text. That works for text files, but it means an agent using a vision-capable model cannot ask read to read an image from the workspace and have the underlying model actually receive the image bytes in a model-compatible multimodal format.
This came up while using @cloudflare/think with Workers AI Kimi K2.6 (@cf/moonshotai/kimi-k2.6), which supports vision inputs. The model can receive image/file parts when they are present in the prompt, but Think's workspace read tool only exposes readFile() text and line-numbered string output.
Related broader issue: #1392 tracks multimodal memory/context. This request is narrower: make Think's workspace read tool multimodal-aware for files already present in this.workspace.
Current Think implementation
Pinned to current main at 7976aede7ebc2969b706a21079a67558416d9473.
The workspace surface used by the tools does not include byte reads:
|
export type WorkspaceLike = Pick< |
|
Workspace, |
|
"readFile" | "writeFile" | "readDir" | "rm" | "glob" | "mkdir" | "stat" |
|
>; |
ReadOperations only exposes readFile() plus stat():
|
export interface ReadOperations { |
|
readFile(path: string): Promise<string | null>; |
|
stat(path: string): Promise<FileInfo | null> | FileInfo | null; |
|
} |
The workspace-backed read adapter only forwards text reads:
|
function workspaceReadOps(ws: WorkspaceLike): ReadOperations { |
|
return { |
|
readFile: (path) => ws.readFile(path), |
|
stat: (path) => ws.stat(path) |
|
}; |
The current read tool implementation always calls ops.readFile(path), splits on newlines, and returns line-numbered text:
|
export function createReadTool(options: ReadToolOptions) { |
|
const { ops } = options; |
|
|
|
return tool({ |
|
description: |
|
"Read the contents of a file. Returns the file content with line numbers. " + |
|
"Use offset and limit for large files. Returns null if the file does not exist.", |
|
inputSchema: z.object({ |
|
path: z.string().describe("Absolute path to the file"), |
|
offset: z |
|
.number() |
|
.int() |
|
.min(1) |
|
.optional() |
|
.describe("1-indexed line number to start reading from"), |
|
limit: z |
|
.number() |
|
.int() |
|
.min(1) |
|
.optional() |
|
.describe("Number of lines to read") |
|
}), |
|
execute: async ({ path, offset, limit }) => { |
|
const stat = await ops.stat(path); |
|
if (!stat) { |
|
return { error: `File not found: ${path}` }; |
|
} |
|
if (stat.type === "directory") { |
|
return { error: `${path} is a directory, not a file` }; |
|
} |
|
|
|
const content = await ops.readFile(path); |
|
if (content === null) { |
|
return { error: `Could not read file: ${path}` }; |
|
} |
|
|
|
const allLines = content.split("\n"); |
|
const totalLines = allLines.length; |
|
|
|
// Apply offset/limit |
|
const startLine = offset ? offset - 1 : 0; |
|
const endLine = limit ? startLine + limit : allLines.length; |
|
const lines = allLines.slice(startLine, endLine); |
|
|
|
// Format with line numbers, truncate long lines |
|
const numbered = lines.map((line, i) => { |
|
const lineNum = startLine + i + 1; |
|
const truncated = |
|
line.length > MAX_LINE_LENGTH |
|
? line.slice(0, MAX_LINE_LENGTH) + "... (truncated)" |
|
: line; |
|
return `${lineNum}\t${truncated}`; |
|
}); |
|
|
|
// Truncate if too many lines |
|
let output: string; |
|
if (numbered.length > MAX_LINES) { |
|
output = |
|
numbered.slice(0, MAX_LINES).join("\n") + |
|
`\n... (${numbered.length - MAX_LINES} more lines truncated)`; |
|
} else { |
|
output = numbered.join("\n"); |
|
} |
|
|
|
const result: Record<string, unknown> = { |
|
path, |
|
content: output, |
|
totalLines |
|
}; |
|
|
|
if (offset || limit) { |
|
result.fromLine = startLine + 1; |
|
result.toLine = Math.min(endLine, totalLines); |
|
} |
|
|
|
return result; |
|
} |
|
}); |
Prior art: Pi read tool
Pi's read tool was a major inspiration for Think's tool behavior, and it already treats read as a multimodal-capable operation.
Pinned to Pi main at 9dcde1e3fa3bea29e64a85cb1990eea41748f1b4.
Pi read imports and image helpers
import type { AgentTool } from "@mariozechner/pi-agent-core";
import type { Api, ImageContent, Model, TextContent } from "@mariozechner/pi-ai";
import { Text } from "@mariozechner/pi-tui";
import { constants } from "fs";
import { access as fsAccess, readFile as fsReadFile } from "fs/promises";
import { type Static, Type } from "typebox";
import { keyHint } from "../../modes/interactive/components/keybinding-hints.js";
import { getLanguageFromPath, highlightCode } from "../../modes/interactive/theme/theme.js";
import { formatDimensionNote, resizeImage } from "../../utils/image-resize.js";
import { detectSupportedImageMimeTypeFromFile } from "../../utils/mime.js";
Pi ReadOperations buffer reads and MIME hook
/**
* Pluggable operations for the read tool.
* Override these to delegate file reading to remote systems (for example SSH).
*/
export interface ReadOperations {
/** Read file contents as a Buffer */
readFile: (absolutePath: string) => Promise<Buffer>;
/** Check if file is readable (throw if not) */
access: (absolutePath: string) => Promise<void>;
/** Detect image MIME type, return null or undefined for non-images */
detectImageMimeType?: (absolutePath: string) => Promise<string | null | undefined>;
}
const defaultReadOperations: ReadOperations = {
readFile: (path) => fsReadFile(path),
access: (path) => fsAccess(path, constants.R_OK),
detectImageMimeType: detectSupportedImageMimeTypeFromFile,
};
Pi read tool declaration and description
export function createReadToolDefinition(
cwd: string,
options?: ReadToolOptions,
): ToolDefinition<typeof readSchema, ReadToolDetails | undefined> {
const autoResizeImages = options?.autoResizeImages ?? true;
const ops = options?.operations ?? defaultReadOperations;
return {
name: "read",
label: "read",
description: `Read the contents of a file. Supports text files and images (jpg, png, gif, webp). Images are sent as attachments. For text files, output is truncated to ${DEFAULT_MAX_LINES} lines or ${DEFAULT_MAX_BYTES / 1024}KB (whichever is hit first). Use offset/limit for large files. When you need the full file, continue with offset until complete.`,
promptSnippet: "Read file contents",
promptGuidelines: ["Use read to examine files instead of cat or sed."],
parameters: readSchema,
Pi non-vision image note
function getNonVisionImageNote(model: Model<Api> | undefined): string | undefined {
if (!model || model.input.includes("image")) {
return undefined;
}
return "[Current model does not support images. The image will be omitted from this request.]";
}
Pi image branch in execute
// Check if file exists and is readable.
await ops.access(absolutePath);
if (aborted) return;
const mimeType = ops.detectImageMimeType ? await ops.detectImageMimeType(absolutePath) : undefined;
let content: (TextContent | ImageContent)[];
let details: ReadToolDetails | undefined;
const nonVisionImageNote = getNonVisionImageNote(ctx?.model);
if (mimeType) {
// Read image as binary.
const buffer = await ops.readFile(absolutePath);
const base64 = buffer.toString("base64");
if (autoResizeImages) {
// Resize image if needed before sending it back to the model.
const resized = await resizeImage({ type: "image", data: base64, mimeType });
if (!resized) {
let textNote = `Read image file [${mimeType}]\n[Image omitted: could not be resized below the inline image size limit.]`;
if (nonVisionImageNote) textNote += `\n${nonVisionImageNote}`;
content = [{ type: "text", text: textNote }];
} else {
const dimensionNote = formatDimensionNote(resized);
let textNote = `Read image file [${resized.mimeType}]`;
if (dimensionNote) textNote += `\n${dimensionNote}`;
if (nonVisionImageNote) textNote += `\n${nonVisionImageNote}`;
content = [
{ type: "text", text: textNote },
{ type: "image", data: resized.data, mimeType: resized.mimeType },
];
}
} else {
let textNote = `Read image file [${mimeType}]`;
if (nonVisionImageNote) textNote += `\n${nonVisionImageNote}`;
content = [
{ type: "text", text: textNote },
{ type: "image", data: base64, mimeType },
];
}
}
Pi text branch in execute
} else {
// Read text content.
const buffer = await ops.readFile(absolutePath);
const textContent = buffer.toString("utf-8");
const allLines = textContent.split("\n");
const totalFileLines = allLines.length;
// Apply offset if specified. Convert from 1-indexed input to 0-indexed array access.
const startLine = offset ? Math.max(0, offset - 1) : 0;
const startLineDisplay = startLine + 1;
// Check if offset is out of bounds.
if (startLine >= allLines.length) {
throw new Error(`Offset ${offset} is beyond end of file (${allLines.length} lines total)`);
}
let selectedContent: string;
let userLimitedLines: number | undefined;
// If limit is specified by the user, honor it first. Otherwise truncateHead decides.
if (limit !== undefined) {
const endLine = Math.min(startLine + limit, allLines.length);
selectedContent = allLines.slice(startLine, endLine).join("\n");
userLimitedLines = endLine - startLine;
} else {
selectedContent = allLines.slice(startLine).join("\n");
}
// Apply truncation, respecting both line and byte limits.
const truncation = truncateHead(selectedContent);
let outputText: string;
if (truncation.firstLineExceedsLimit) {
// First line alone exceeds the byte limit. Point the model at a bash fallback.
const firstLineSize = formatSize(Buffer.byteLength(allLines[startLine], "utf-8"));
outputText = `[Line ${startLineDisplay} is ${firstLineSize}, exceeds ${formatSize(DEFAULT_MAX_BYTES)} limit. Use bash: sed -n '${startLineDisplay}p' ${path} | head -c ${DEFAULT_MAX_BYTES}]`;
details = { truncation };
} else if (truncation.truncated) {
// Truncation occurred. Build an actionable continuation notice.
const endLineDisplay = startLineDisplay + truncation.outputLines - 1;
const nextOffset = endLineDisplay + 1;
outputText = truncation.content;
if (truncation.truncatedBy === "lines") {
outputText += `\n\n[Showing lines ${startLineDisplay}-${endLineDisplay} of ${totalFileLines}. Use offset=${nextOffset} to continue.]`;
} else {
outputText += `\n\n[Showing lines ${startLineDisplay}-${endLineDisplay} of ${totalFileLines} (${formatSize(DEFAULT_MAX_BYTES)} limit). Use offset=${nextOffset} to continue.]`;
}
details = { truncation };
} else if (userLimitedLines !== undefined && startLine + userLimitedLines < allLines.length) {
// User-specified limit stopped early, but the file still has more content.
const remaining = allLines.length - (startLine + userLimitedLines);
const nextOffset = startLine + userLimitedLines + 1;
outputText = `${truncation.content}\n\n[${remaining} more lines in file. Use offset=${nextOffset} to continue.]`;
} else {
// No truncation and no remaining user-limited content.
outputText = truncation.content;
}
content = [{ type: "text", text: outputText }];
}
Pi MIME sniffing helper
import { open } from "node:fs/promises";
import { fileTypeFromBuffer } from "file-type";
const IMAGE_MIME_TYPES = new Set(["image/jpeg", "image/png", "image/gif", "image/webp"]);
const FILE_TYPE_SNIFF_BYTES = 4100;
export async function detectSupportedImageMimeTypeFromFile(filePath: string): Promise<string | null> {
const fileHandle = await open(filePath, "r");
try {
const buffer = Buffer.alloc(FILE_TYPE_SNIFF_BYTES);
const { bytesRead } = await fileHandle.read(buffer, 0, FILE_TYPE_SNIFF_BYTES, 0);
if (bytesRead === 0) {
return null;
}
const fileType = await fileTypeFromBuffer(buffer.subarray(0, bytesRead));
if (!fileType) {
return null;
}
if (!IMAGE_MIME_TYPES.has(fileType.mime)) {
return null;
}
return fileType.mime;
} finally {
await fileHandle.close();
}
}
Pi resize defaults and strategy
// 4.5MB of base64 payload. Provides headroom below Anthropic's 5MB limit.
const DEFAULT_MAX_BYTES = 4.5 * 1024 * 1024;
const DEFAULT_OPTIONS: Required<ImageResizeOptions> = {
maxWidth: 2000,
maxHeight: 2000,
maxBytes: DEFAULT_MAX_BYTES,
jpegQuality: 80,
};
interface EncodedCandidate {
data: string;
encodedSize: number;
mimeType: string;
}
function encodeCandidate(buffer: Uint8Array, mimeType: string): EncodedCandidate {
const data = Buffer.from(buffer).toString("base64");
return {
data,
encodedSize: Buffer.byteLength(data, "utf-8"),
mimeType,
};
}
/**
* Resize an image to fit within the specified max dimensions and encoded file size.
* Returns null if the image cannot be resized below maxBytes.
*
* Uses Photon (Rust/WASM) for image processing. If Photon is not available,
* returns null.
*
* Strategy for staying under maxBytes:
* 1. First resize to maxWidth/maxHeight
* 2. Try both PNG and JPEG formats, pick the smaller one
* 3. If still too large, try JPEG with decreasing quality
* 4. If still too large, progressively reduce dimensions until 1x1
*/
export async function resizeImage(img: ImageContent, options?: ImageResizeOptions): Promise<ResizedImage | null> {
Pi no-resize fast path
// Check if already within all limits (dimensions AND encoded size)
if (originalWidth <= opts.maxWidth && originalHeight <= opts.maxHeight && inputBase64Size < opts.maxBytes) {
return {
data: img.data,
mimeType: img.mimeType ?? `image/${format}`,
originalWidth,
originalHeight,
width: originalWidth,
height: originalHeight,
wasResized: false,
};
}
Pi resize loop and dimension note
while (true) {
const candidates = tryEncodings(currentWidth, currentHeight, qualitySteps);
for (const candidate of candidates) {
if (candidate.encodedSize < opts.maxBytes) {
return {
data: candidate.data,
mimeType: candidate.mimeType,
originalWidth,
originalHeight,
width: currentWidth,
height: currentHeight,
wasResized: true,
};
}
}
if (currentWidth === 1 && currentHeight === 1) {
break;
}
const nextWidth = currentWidth === 1 ? 1 : Math.max(1, Math.floor(currentWidth * 0.75));
const nextHeight = currentHeight === 1 ? 1 : Math.max(1, Math.floor(currentHeight * 0.75));
if (nextWidth === currentWidth && nextHeight === currentHeight) {
break;
}
currentWidth = nextWidth;
currentHeight = nextHeight;
}
return null;
} catch {
return null;
} finally {
if (image) {
image.free();
}
}
}
/**
* Format a dimension note for resized images.
* This helps the model understand the coordinate mapping.
*/
export function formatDimensionNote(result: ResizedImage): string | undefined {
if (!result.wasResized) {
return undefined;
}
const scale = result.originalWidth / result.width;
return `[Image: original ${result.originalWidth}x${result.originalHeight}, displayed at ${result.width}x${result.height}. Multiply coordinates by ${scale.toFixed(2)} to map to original image.]`;
}
Why this matters
A model can only reason over image content if the provider receives an actual image/file part. A line-numbered string representation of an image path, binary decode, or placeholder metadata is not enough.
This also makes file extension reliance fragile. Workspace files already have FileInfo.mimeType; uploaded or imported files may be extensionless, incorrectly named, or created by tools. The read path should prefer MIME metadata and byte sniffing over filename extensions.
Proposed behavior
Keep current text behavior unchanged for text files.
For images and other supported multimodal file types:
read should inspect stat.mimeType first.
- If MIME is missing or generic, it should sniff bytes for common formats such as PNG, JPEG, GIF, WebP, SVG, and PDF before falling back to filename extension.
- For
image/*, read should read bytes with readFileBytes() and return AI SDK-compatible multimodal tool output, e.g. type: "content" with image-data.
- The user-visible/tool-result object should avoid persisting large base64 payloads when possible. The
toModelOutput hook can re-read bytes and send image data to the model while the normal tool result remains small metadata.
- Unknown binary files should return metadata and a clear unsupported-binary result instead of attempting UTF-8 line numbering.
Strawman implementation sketch
This is intentionally a sketch, not a finished patch. The key point is: execute can return compact metadata, while toModelOutput can map image reads to model-facing multimodal content.
export type WorkspaceLike = Pick<
Workspace,
| "readFile"
| "readFileBytes"
| "writeFile"
| "readDir"
| "rm"
| "glob"
| "mkdir"
| "stat"
>;
export interface ReadOperations {
readFile(path: string): Promise<string | null>;
readFileBytes(path: string): Promise<Uint8Array | null>;
stat(path: string): Promise<FileInfo | null> | FileInfo | null;
}
type ReadToolOutput =
| {
kind: "text";
path: string;
content: string;
totalLines: number;
fromLine?: number;
toLine?: number;
}
| {
kind: "image";
path: string;
name: string;
mediaType: string;
sizeBytes: number;
}
| {
kind: "binary";
path: string;
name: string;
mediaType: string;
sizeBytes: number;
unsupported: true;
}
| { error: string };
export function createReadTool(options: ReadToolOptions) {
const { ops } = options;
return tool({
description:
"Read a workspace file. Text files return line-numbered text. " +
"Images are passed to vision-capable models as image content.",
inputSchema: z.object({
path: z.string().describe("Absolute path to the file"),
offset: z.number().int().min(1).optional(),
limit: z.number().int().min(1).optional()
}),
execute: async ({ path, offset, limit }): Promise<ReadToolOutput> => {
const stat = await ops.stat(path);
if (!stat) return { error: `File not found: ${path}` };
if (stat.type === "directory") {
return { error: `${path} is a directory, not a file` };
}
const mediaType = await detectWorkspaceMediaType({ ops, path, stat });
if (mediaType.startsWith("image/")) {
return {
kind: "image",
path,
name: stat.name,
mediaType,
sizeBytes: stat.size
};
}
if (isProbablyBinary(mediaType)) {
return {
kind: "binary",
path,
name: stat.name,
mediaType,
sizeBytes: stat.size,
unsupported: true
};
}
// Preserve today's line-numbered text behavior for text files.
return readTextWithLineNumbers({ ops, path, offset, limit });
},
toModelOutput: async ({ input, output }) => {
if (output.kind !== "image") {
return "content" in output
? { type: "text", value: output.content }
: { type: "json", value: output };
}
const bytes = await ops.readFileBytes(input.path);
if (!bytes) {
return { type: "error-text", value: `Could not read image bytes: ${input.path}` };
}
return {
type: "content",
value: [
{
type: "text",
text: `Read image ${output.path} (${output.mediaType}, ${output.sizeBytes} bytes).`
},
{
type: "image-data",
data: uint8ArrayToBase64(bytes),
mediaType: output.mediaType
}
]
};
}
});
}
Provider compatibility note: AI SDK v6 has ToolResultOutput content variants such as image-data and file-data. If a given provider cannot send multimodal content in a role: "tool" message, Think may need to normalize image tool results into the provider-supported prompt shape before the provider boundary. The acceptance test should verify the actual provider request contains an image/file part, not only that the tool returned image metadata.
Acceptance tests
- A text file read still returns the current line-numbered content shape, including
offset/limit behavior.
- An image file with
mimeType: "image/png" and no useful extension is read through read and reaches the next model call as image content.
- An image file with generic MIME but PNG/JPEG/GIF/WebP magic bytes is treated as an image without relying on the filename extension.
- Unknown binary files are not passed through
readFile() as mojibake text.
- The persisted tool result remains compact; large base64 image data should not be stored in normal chat/session history unless there is an explicit size-bound design for it.
Summary
Think's built-in workspace
readtool currently treats every file as text. That works for text files, but it means an agent using a vision-capable model cannot askreadto read an image from the workspace and have the underlying model actually receive the image bytes in a model-compatible multimodal format.This came up while using
@cloudflare/thinkwith Workers AI Kimi K2.6 (@cf/moonshotai/kimi-k2.6), which supports vision inputs. The model can receive image/file parts when they are present in the prompt, but Think's workspacereadtool only exposesreadFile()text and line-numbered string output.Related broader issue: #1392 tracks multimodal memory/context. This request is narrower: make Think's workspace
readtool multimodal-aware for files already present inthis.workspace.Current Think implementation
Pinned to current
mainat7976aede7ebc2969b706a21079a67558416d9473.The workspace surface used by the tools does not include byte reads:
agents/packages/think/src/tools/workspace.ts
Lines 20 to 23 in 7976aed
ReadOperationsonly exposesreadFile()plusstat():agents/packages/think/src/tools/workspace.ts
Lines 29 to 32 in 7976aed
The workspace-backed read adapter only forwards text reads:
agents/packages/think/src/tools/workspace.ts
Lines 69 to 73 in 7976aed
The current
readtool implementation always callsops.readFile(path), splits on newlines, and returns line-numbered text:agents/packages/think/src/tools/workspace.ts
Lines 158 to 235 in 7976aed
Prior art: Pi read tool
Pi's read tool was a major inspiration for Think's tool behavior, and it already treats
readas a multimodal-capable operation.Pinned to Pi
mainat9dcde1e3fa3bea29e64a85cb1990eea41748f1b4.Pi read imports and image helpers
Pi
ReadOperationsbuffer reads and MIME hookPi read tool declaration and description
Pi non-vision image note
Pi image branch in
executePi text branch in
executePi MIME sniffing helper
Pi resize defaults and strategy
Pi no-resize fast path
Pi resize loop and dimension note
Why this matters
A model can only reason over image content if the provider receives an actual image/file part. A line-numbered string representation of an image path, binary decode, or placeholder metadata is not enough.
This also makes file extension reliance fragile. Workspace files already have
FileInfo.mimeType; uploaded or imported files may be extensionless, incorrectly named, or created by tools. The read path should prefer MIME metadata and byte sniffing over filename extensions.Proposed behavior
Keep current text behavior unchanged for text files.
For images and other supported multimodal file types:
readshould inspectstat.mimeTypefirst.image/*,readshould read bytes withreadFileBytes()and return AI SDK-compatible multimodal tool output, e.g.type: "content"withimage-data.toModelOutputhook can re-read bytes and send image data to the model while the normal tool result remains small metadata.Strawman implementation sketch
This is intentionally a sketch, not a finished patch. The key point is:
executecan return compact metadata, whiletoModelOutputcan map image reads to model-facing multimodal content.Provider compatibility note: AI SDK v6 has
ToolResultOutputcontent variants such asimage-dataandfile-data. If a given provider cannot send multimodal content in arole: "tool"message, Think may need to normalize image tool results into the provider-supported prompt shape before the provider boundary. The acceptance test should verify the actual provider request contains an image/file part, not only that the tool returned image metadata.Acceptance tests
offset/limitbehavior.mimeType: "image/png"and no useful extension is read throughreadand reaches the next model call as image content.readFile()as mojibake text.