Skip to content

Commit a76c091

Browse files
yansigitcursoragent
andcommitted
fix(cursor): promote view_image past Desktop multi-agent suffix
Treat trailing non-image developer injections as transparent for SelectedImage resolution and tool-continuation gates so Codex Desktop collab guidance no longer drops vision after view_image. Also enforce the JPEG soft-cap with edge shrink and map Grok none/minimal to medium with the cursor- wire prefix. Co-authored-by: Cursor <[email protected]>
1 parent de8eb90 commit a76c091

10 files changed

Lines changed: 242 additions & 36 deletions

File tree

‎docs-site/src/content/docs/reference/adapters.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -181,8 +181,9 @@ advertised effort control on those models as proof of upstream-native reasoning
181181
- Exposes Cursor Router as `cursor/auto` plus explicit `cursor/auto-cost`,
182182
`cursor/auto-balance`, and `cursor/auto-intelligence` entries. Explicit levels are encoded in
183183
`requested_model.parameters` while the legacy `cursor/auto` entry retains the account/team default.
184-
- Keeps `cursor/grok-4.5-fast` as a selectable model while sending Cursor's canonical `grok-4.5`
185-
model with separate `effort` and `fast=true` parameters.
184+
- Sends regular `cursor/grok-4.5` tiers with Cursor's exact live-discovery wire ids
185+
(`cursor-grok-4.5-low`, `-medium`, or `-high`). Keeps `cursor/grok-4.5-fast` selectable while
186+
sending the canonical `grok-4.5` model with separate `effort` and `fast=true` parameters.
186187
- Cursor-native local filesystem/shell/network execution is denied by default. Explicit `mcpServers`
187188
and `desktopExecutor` integrations have separate opt-ins; `nativeLocalExec: "on"` enables the
188189
broader built-in executor and bypasses Codex approval/sandbox semantics, and legacy

‎docs-site/src/content/docs/reference/configuration/providers.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -272,11 +272,15 @@ selection on every request. They remain available when live discovery omits `def
272272

273273
Native Cursor vision uses `SelectedImage` (JPEG soft-cap + `blobIdWithData`) for models that can see
274274
images natively — Claude, Gemini, GPT, Kimi, and Grok among them. `auto` and `composer-*` stay on
275-
the curated `noVisionModels` list and use the vision describe sidecar instead.
275+
the curated `noVisionModels` list and use the vision describe sidecar instead. Trailing non-image
276+
`developer` injections (Codex Desktop multi-agent guidance after `view_image`) are transparent for
277+
SelectedImage promotion so the continuation still carries the image and promote nudge.
276278

277279
After pulling Cursor vision fixes, run `ocx ensure` so the proxy PID is the workspace `src/cli`
278280
binary rather than a stale install. Stale `providers.cursor.noVisionModels` stamps that list every
279-
Cursor model are healed back to the curated Auto/Composer/GLM set on OAuth reconcile.
281+
Cursor model are healed back to the curated Auto/Composer/GLM set on OAuth reconcile. For
282+
`cursor/grok-4.5`, Codex effort `none`/`minimal` maps to wire tier `medium` (some plans reject
283+
`-low` with Connect `not_found`); explicit `low` still passes through when the account exposes it.
280284

281285
Cursor server-driven local tools are disabled by default. Codex continues using its own tools such as
282286
`apply_patch` and `exec_command` with its own approval and sandbox policy:

‎src/adapters/cursor/effort-map.ts‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,11 +86,19 @@ function codexEffortRank(reasoning: string | undefined): "low" | "medium" | "hig
8686
/**
8787
* The Cursor effort suffix to use for `baseModelId` given a Codex reasoning effort, or `undefined` when
8888
* the model takes no suffix (bare). Literal model tiers pass through; unknown efforts clamp by rank.
89+
*
90+
* Grok 4.5 special case: Codex `none`/`minimal` map to `medium`, not `low`. Cursor Start fixes Grok
91+
* at medium and live Connect returns `not_found` for `grok-4.5-low` on some plans; explicit `low`
92+
* still passes through for Pro accounts that expose it.
8993
*/
9094
export function cursorEffortSuffix(baseModelId: string, reasoning: string | undefined): string | undefined {
9195
const tiers = CURSOR_MODEL_EFFORT_TIERS[baseModelId];
9296
if (!tiers || tiers.length === 0) return undefined;
9397
const requested = normalizeRequestedEffort(reasoning);
98+
const isGrok45 = baseModelId === "grok-4.5" || baseModelId === "grok-4.5-fast";
99+
if (isGrok45 && (requested === "none" || requested === "minimal") && tiers.includes("medium")) {
100+
return "medium";
101+
}
94102
if (requested && tiers.includes(requested)) return requested;
95103
switch (codexEffortRank(reasoning)) {
96104
case "low":
@@ -126,3 +134,14 @@ export function cursorWireModelIdWithEffort(baseModelId: string, effortSuffix: s
126134
}
127135
return `${baseModelId}-${effortSuffix}`;
128136
}
137+
138+
/**
139+
* Compose the exact flattened id sent by AgentService/Run. Discovery normalizes Cursor's optional
140+
* `cursor-` prefix only for catalog matching, but regular Grok 4.5 requests require that prefix on
141+
* the wire. Keep this separate from {@link cursorWireModelIdWithEffort} so discovery can continue
142+
* comparing canonical, prefix-free ids. Grok Fast uses requested_model parameters instead.
143+
*/
144+
export function cursorRequestWireModelIdWithEffort(baseModelId: string, effortSuffix: string): string {
145+
const flattened = cursorWireModelIdWithEffort(baseModelId, effortSuffix);
146+
return baseModelId === "grok-4.5" ? `cursor-${flattened}` : flattened;
147+
}

‎src/adapters/cursor/images.ts‎

Lines changed: 90 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ export const CURSOR_VISION_MAX_EDGE = 2000;
4040

4141
const CURSOR_VISION_JPEG_QUALITIES_DEFAULT = [85, 70, 55, 40] as const;
4242
const CURSOR_VISION_JPEG_QUALITIES_HIGH = [90, 80, 65, 50] as const;
43+
/** Stop shrinking below this longest edge when chasing the soft byte cap. */
44+
const CURSOR_VISION_SOFT_MIN_EDGE = 256;
45+
const CURSOR_VISION_SOFT_SHRINK = 0.85;
4346

4447
const CURSOR_VISION_PASSTHROUGH_MIME = new Set([
4548
"image/jpeg",
@@ -326,12 +329,17 @@ export function cursorImageAttachmentPath(uuid: string, mimeType: string): strin
326329
* Re-encode toward a JPEG under the soft vision cap when Bun can decode the payload.
327330
* Unsupported MIME or undecodable bytes are omitted (fail-closed) except for tiny
328331
* undecodable PNG/JPEG/GIF/WebP stubs used in unit tests (pass-through under soft cap).
332+
* After the quality ladder, edges shrink iteratively until the soft byte cap is met
333+
* (or the min edge floor is hit) so large clipboard PNGs do not leave >softMax JPEGs
334+
* that Cursor vision hallucinates on.
329335
*/
330336
export async function prepareCursorImageForWire(
331337
image: ResolvedCursorImage,
332338
): Promise<PrepareCursorImageOutcome> {
333339
const mime = image.mimeType.toLowerCase();
334340
const softMax = softMaxBytesForDetail(image.detail);
341+
const qualities = jpegQualitiesForDetail(image.detail);
342+
const lowestQuality = qualities[qualities.length - 1]!;
335343

336344
if (!CURSOR_VISION_PASSTHROUGH_MIME.has(mime)) {
337345
return { status: "omitted", reason: CURSOR_VISION_IMAGE_OMITTED };
@@ -357,28 +365,61 @@ export async function prepareCursorImageForWire(
357365
targetH = Math.max(1, Math.round(height * scale));
358366
}
359367

360-
let best: Uint8Array | undefined;
361-
for (const quality of jpegQualitiesForDetail(image.detail)) {
368+
const encodeAt = async (w: number, h: number, quality: number): Promise<Uint8Array> => {
362369
let pipeline = new Bun.Image(image.data);
363-
if (targetW > 0 && targetH > 0 && (targetW !== width || targetH !== height)) {
364-
pipeline = pipeline.resize(targetW, targetH);
370+
if (w > 0 && h > 0 && (w !== width || h !== height)) {
371+
pipeline = pipeline.resize(w, h);
372+
}
373+
return new Uint8Array(await pipeline.jpeg({ quality }).bytes());
374+
};
375+
376+
let best: Uint8Array | undefined;
377+
for (const quality of qualities) {
378+
const encoded = await encodeAt(targetW, targetH, quality);
379+
if (!best || encoded.byteLength < best.byteLength) best = encoded;
380+
if (encoded.byteLength <= softMax) {
381+
return {
382+
status: "ready",
383+
image: { ...image, data: encoded, mimeType: "image/jpeg" },
384+
};
385+
}
386+
}
387+
388+
// Quality ladder missed the soft cap — shrink edges until it fits or we hit the floor.
389+
while (
390+
best
391+
&& best.byteLength > softMax
392+
&& targetW > 0
393+
&& targetH > 0
394+
&& Math.max(targetW, targetH) > CURSOR_VISION_SOFT_MIN_EDGE
395+
) {
396+
const nextW = Math.max(1, Math.round(targetW * CURSOR_VISION_SOFT_SHRINK));
397+
const nextH = Math.max(1, Math.round(targetH * CURSOR_VISION_SOFT_SHRINK));
398+
if (Math.max(nextW, nextH) < CURSOR_VISION_SOFT_MIN_EDGE) {
399+
const scale = CURSOR_VISION_SOFT_MIN_EDGE / Math.max(targetW, targetH);
400+
targetW = Math.max(1, Math.round(targetW * scale));
401+
targetH = Math.max(1, Math.round(targetH * scale));
402+
} else {
403+
targetW = nextW;
404+
targetH = nextH;
365405
}
366-
const encoded = new Uint8Array(await pipeline.jpeg({ quality }).bytes());
406+
const encoded = await encodeAt(targetW, targetH, lowestQuality);
367407
if (!best || encoded.byteLength < best.byteLength) best = encoded;
368408
if (encoded.byteLength <= softMax) {
369409
return {
370410
status: "ready",
371411
image: { ...image, data: encoded, mimeType: "image/jpeg" },
372412
};
373413
}
414+
if (Math.max(targetW, targetH) <= CURSOR_VISION_SOFT_MIN_EDGE) break;
374415
}
375-
if (best && best.byteLength < image.data.byteLength) {
416+
417+
if (best) {
376418
return {
377419
status: "ready",
378420
image: { ...image, data: best, mimeType: "image/jpeg" },
379421
};
380422
}
381-
// Decoded but could not shrink under soft max — still under wire hard cap check later.
382423
return { status: "ready", image };
383424
} catch {
384425
// Tiny undecodable stubs (unit wire tests) may still be referenced as blobIdWithData.
@@ -487,11 +528,12 @@ export function buildSelectedContext(
487528
export function extractTrailingToolResultImagePromotion(
488529
messages: readonly OcxMessage[],
489530
): { parts: CursorImagePartRef[]; omittedOlder: number; promotedCallIds: Set<string> } {
490-
let start = messages.length;
491-
while (start > 0 && messages[start - 1]?.role === "toolResult") start -= 1;
531+
const effective = stripTrailingTransparentDeveloperMessages(messages);
532+
let start = effective.length;
533+
while (start > 0 && effective[start - 1]?.role === "toolResult") start -= 1;
492534

493535
const entries: Array<{ callId: string; part: CursorImagePartRef }> = [];
494-
for (const message of messages.slice(start)) {
536+
for (const message of effective.slice(start)) {
495537
if (message.role !== "toolResult") continue;
496538
for (const part of extractCursorImageParts(message.content)) {
497539
entries.push({ callId: message.toolCallId, part });
@@ -519,27 +561,61 @@ export function extractTrailingToolResultImageParts(
519561
return { parts, omittedOlder };
520562
}
521563

564+
/**
565+
* Trailing non-image developer messages (Codex Desktop multi-agent guidance) are
566+
* transparent for Cursor vision / tool-continuation detection so `view_image`
567+
* still promotes onto SelectedImage.
568+
*/
569+
export function isTransparentCursorVisionSuffix(message: OcxMessage): boolean {
570+
if (message.role !== "developer") return false;
571+
return extractCursorImageParts(message.content).length === 0;
572+
}
573+
574+
/** Drop trailing transparent developers; returns the same array reference when unchanged. */
575+
export function stripTrailingTransparentDeveloperMessages(
576+
messages: readonly OcxMessage[],
577+
): readonly OcxMessage[] {
578+
let end = messages.length;
579+
while (end > 0) {
580+
const message = messages[end - 1];
581+
if (!message || !isTransparentCursorVisionSuffix(message)) break;
582+
end -= 1;
583+
}
584+
return end === messages.length ? messages : messages.slice(0, end);
585+
}
586+
587+
/** True when, ignoring transparent developer suffix, the active turn is a toolResult. */
588+
export function cursorIsTrailingToolResultContinuation(
589+
messages: readonly OcxMessage[] | undefined,
590+
): boolean {
591+
if (!messages?.length) return false;
592+
return stripTrailingTransparentDeveloperMessages(messages).at(-1)?.role === "toolResult";
593+
}
594+
522595
/**
523596
* Resolve images for the active Cursor turn.
524597
* - Trailing user/developer: attach images (SelectedImage).
525598
* - Trailing toolResult run (e.g. Codex `view_image`): same channel — McpImageContent
526599
* alone is not consumed as vision for external Cursor models like grok-4.5.
600+
* - Trailing non-image developer injections (multi_agent_mode) are stripped first so
601+
* Desktop collab guidance does not hide a view_image promotion.
527602
*/
528603
export async function resolveActiveCursorImages(
529604
messages: readonly OcxMessage[] | undefined,
530605
signal?: AbortSignal,
531606
): Promise<ResolvedCursorImage[]> {
532607
if (!messages?.length) return [];
533608

534-
const last = messages.at(-1);
609+
const effective = stripTrailingTransparentDeveloperMessages(messages);
610+
const last = effective.at(-1);
535611
if (last?.role === "toolResult") {
536-
const { parts } = extractTrailingToolResultImageParts(messages);
612+
const { parts } = extractTrailingToolResultImageParts(effective);
537613
if (parts.length === 0) return [];
538614
return resolveCursorImageParts(parts, signal);
539615
}
540616

541-
for (let i = messages.length - 1; i >= 0; i--) {
542-
const message = messages[i];
617+
for (let i = effective.length - 1; i >= 0; i--) {
618+
const message = effective[i];
543619
if (!message) continue;
544620
if (message.role === "user" || message.role === "developer") {
545621
return resolveCursorImageParts(extractCursorImageParts(message.content), signal);

‎src/adapters/cursor/live-transport.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ import {
1313
type TranslatorBudget,
1414
} from "../../lib/translator-budget";
1515
import { activePromptText, prepareCursorRunRequest } from "./protobuf-request";
16-
import { prepareCursorRawMessages, resolveActiveCursorImages } from "./images";
16+
import { prepareCursorRawMessages, resolveActiveCursorImages, cursorIsTrailingToolResultContinuation } from "./images";
1717
import {
1818
createCursorContextUsageTracker,
1919
createCursorProtobufEventState,
@@ -391,7 +391,7 @@ export function finalizeAfterDrain(state: ReturnType<typeof createCursorProtobuf
391391
}
392392

393393
export function clientToolFinalizeGraceMsForRequest(request: CursorRunRequest, baseGraceMs = CLIENT_TOOL_FINALIZE_GRACE_MS): number {
394-
if (request.rawMessages?.at(-1)?.role === "toolResult") return baseGraceMs;
394+
if (cursorIsTrailingToolResultContinuation(request.rawMessages)) return baseGraceMs;
395395
const text = activePromptText(request);
396396
if (!cursorRequestHasShellAlias(request.tools) || !isGenericToolUseCountDemoPrompt(text)) return baseGraceMs;
397397
const requestedCount = requestedCursorToolUseCount(text);

‎src/adapters/cursor/protobuf-request.ts‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,9 @@ import {
1717
buildSelectedContext,
1818
CURSOR_VISION_MCP_IMAGE_OMITTED,
1919
CURSOR_VISION_PROMOTE_NUDGE,
20+
cursorIsTrailingToolResultContinuation,
2021
extractTrailingToolResultImagePromotion,
22+
stripTrailingTransparentDeveloperMessages,
2123
} from "./images";
2224
import { estimateTokens } from "../../lib/token-estimate";
2325
import {
@@ -202,7 +204,7 @@ function rootPromptMessages(request: CursorRunRequest, requestScope: CursorBlobR
202204
}
203205

204206
const externalModel = isCursorExternalWireModel(request.modelId);
205-
const lastRawIsToolResult = messages.at(-1)?.role === "toolResult";
207+
const lastRawIsToolResult = cursorIsTrailingToolResultContinuation(messages);
206208
const activeUserIndex = lastRawIsToolResult ? -1 : lastActionIndex(messages);
207209

208210
for (let i = 0; i < messages.length; i++) {
@@ -521,7 +523,10 @@ function conversationTurns(
521523
if (!messages?.length) return [];
522524
const end = lastActionIndex(messages);
523525
const externalModel = isCursorExternalWireModel(request.modelId);
524-
const historyEnd = messages.at(-1)?.role === "toolResult" ? messages.length : Math.max(0, end);
526+
// Transparent developer suffixes stay out of turns; they are not toolResult history.
527+
const historyEnd = cursorIsTrailingToolResultContinuation(messages)
528+
? stripTrailingTransparentDeveloperMessages(messages).length
529+
: Math.max(0, end);
525530
const start = externalModel ? Math.max(0, historyMessageStart) : 0;
526531
const turns: Uint8Array[] = [];
527532
let current: { userMessage: Uint8Array; steps: Uint8Array[] } | undefined;
@@ -683,7 +688,7 @@ function buildPreparedCursorRunRequest(
683688
// Tool-result-only turns resume the remembered Cursor conversation with results in history.
684689
// Exception: image-bearing tool results (Codex view_image) must ride SelectedImage on a
685690
// userMessageAction — McpImageContent inline bytes are not hydrated as vision for grok/etc.
686-
const lastRawIsToolResult = request.rawMessages?.at(-1)?.role === "toolResult";
691+
const lastRawIsToolResult = cursorIsTrailingToolResultContinuation(request.rawMessages);
687692
const selectedImages = request.selectedImages ?? [];
688693
const promoteToolResultImages = lastRawIsToolResult && selectedImages.length > 0;
689694
const omitToolResultImageCallIds = promoteToolResultImages

‎src/adapters/cursor/request-builder.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import type {
1010
import { isAllowedToolChoice, namespacedToolName, toolChoiceAliases, type OcxTool, type OcxToolChoice } from "../../types";
1111
import type { CursorRequestMessage, CursorRequestedModelParameter, CursorRunRequest } from "./types";
1212
import { cursorWireModelSelection, type CursorRoutingLevel } from "./discovery";
13-
import { cursorEffortSuffix, cursorWireModelIdWithEffort } from "./effort-map";
13+
import { cursorEffortSuffix, cursorRequestWireModelIdWithEffort } from "./effort-map";
1414
import {
1515
cursorMcpToolEncodedSize,
1616
cursorMcpToolsEncodedSize,
@@ -152,7 +152,7 @@ function normalizeCursorModelId(modelId: string, reasoning?: string): {
152152
],
153153
};
154154
}
155-
return { ...selection, modelId: suffix ? cursorWireModelIdWithEffort(id, suffix) : id };
155+
return { ...selection, modelId: suffix ? cursorRequestWireModelIdWithEffort(id, suffix) : id };
156156
}
157157

158158
function contentPartToText(part: OcxContentPart | OcxAssistantContentPart): string | undefined {

‎tests/cursor-blob.test.ts‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -998,6 +998,55 @@ describe("Cursor blob handshake", () => {
998998
)).toBe(true);
999999
});
10001000

1001+
test("promotes view_image SelectedImage when multi_agent developer trails the toolResult", () => {
1002+
// Mirrors Desktop: injectDeveloperMessage appends after view_image output.
1003+
resetCursorBlobStateForTests();
1004+
const imageBytes = new Uint8Array([137, 80, 78, 71, 13, 10, 26, 10]);
1005+
const collab = "<multi_agent_mode>Preferred sub-agent: model \"cursor/grok-4.5\", reasoning_effort \"high\"</multi_agent_mode>";
1006+
const bytes = encodeCursorRunRequest({
1007+
modelId: "grok-4.5",
1008+
conversationId: "c1",
1009+
system: ["You are helpful."],
1010+
messages: [{ role: "tool", content: "[tool_result]\ncall_id: call_view\nname: view_image\nis_error: false\noutput:" }],
1011+
selectedImages: [{ uuid: "from-view", mimeType: "image/png", data: imageBytes }],
1012+
rawMessages: [
1013+
{ role: "user", content: "What is in this image? Do not guess.", timestamp: 1 },
1014+
{
1015+
role: "assistant",
1016+
model: "cursor/grok-4.5",
1017+
timestamp: 2,
1018+
content: [{ type: "toolCall", id: "call_view", name: "view_image", arguments: { path: "/tmp/x.png" } }],
1019+
},
1020+
{
1021+
role: "toolResult",
1022+
toolCallId: "call_view",
1023+
toolName: "view_image",
1024+
content: [{ type: "image", imageUrl: `data:image/png;base64,${Buffer.from(imageBytes).toString("base64")}`, detail: "high" }],
1025+
isError: false,
1026+
timestamp: 3,
1027+
},
1028+
{ role: "developer", content: collab, timestamp: 4 },
1029+
],
1030+
});
1031+
1032+
expect(activeSelectedImages(bytes)?.length).toBe(1);
1033+
expect(actionText(bytes)).toBe(CURSOR_VISION_PROMOTE_NUDGE);
1034+
const toolCalls = conversationToolCallSteps(bytes);
1035+
expect(toolCalls.length).toBe(1);
1036+
const tool = toolCalls[0]?.message;
1037+
expect(tool?.case).toBe("toolCall");
1038+
if (tool?.case !== "toolCall") throw new Error("expected toolCall");
1039+
expect(tool.value.tool.case).toBe("mcpToolCall");
1040+
if (tool.value.tool.case !== "mcpToolCall") throw new Error("expected mcpToolCall");
1041+
const result = tool.value.tool.value.result;
1042+
expect(result?.result.case).toBe("success");
1043+
if (result?.result.case !== "success") throw new Error("expected success");
1044+
expect(result.result.value.content.every(item => item.content.case !== "image")).toBe(true);
1045+
expect(result.result.value.content.some(item =>
1046+
item.content.case === "text" && item.content.value.text === CURSOR_VISION_MCP_IMAGE_OMITTED
1047+
)).toBe(true);
1048+
});
1049+
10011050
test("image-only user attach keeps empty action text", () => {
10021051
resetCursorBlobStateForTests();
10031052
const imageBytes = new Uint8Array([137, 80, 78, 71, 13, 10, 26, 10]);

0 commit comments

Comments
 (0)