Skip to content

fix: send screenshot attachments to ChatGPT Codex#157

Merged
Sun-sunshine06 merged 5 commits intomainfrom
fix/codex-image-attachments
Apr 22, 2026
Merged

fix: send screenshot attachments to ChatGPT Codex#157
Sun-sunshine06 merged 5 commits intomainfrom
fix/codex-image-attachments

Conversation

@Sun-sunshine06
Copy link
Copy Markdown
Collaborator

Summary

Fixes a chatgpt-codex bug where uploaded screenshots were only reduced to filename hints, so models like gpt-5.4 could ignore the attached visual reference entirely. This patch keeps text attachments unchanged, encodes supported image attachments as data URLs in prompt preparation, and forwards them as Responses input_image parts on the ChatGPT Codex generate path.

Type of change

  • Bug fix
  • New feature
  • Refactor (no behavior change)
  • Documentation
  • Build / CI / tooling
  • Breaking change

Linked issue

  • Refs local repro: attached screenshot uploads were not being used by the model response.

Checklist

  • I read docs/VISION.md, docs/PRINCIPLES.md, and CLAUDE.md before starting
  • Commits are signed with DCO (git commit -s)
  • pnpm lint && pnpm typecheck && pnpm test passes locally
  • Added/updated tests for the change
  • Added a changeset (pnpm changeset) if user-visible
  • Updated docs if behavior changed

Validation run for this PR:

  • corepack pnpm --filter @open-codesign/desktop test -- --run src/main/prompt-context.test.ts src/main/codex-generate.test.ts
  • corepack pnpm --filter @open-codesign/desktop typecheck
  • pre-push checks passed: workspace typecheck + biome check .

Note on full test suite:

  • pnpm test still has existing Windows baseline failures unrelated to this patch (provider token-store permission assertion, core builtin-skill loader expectation, exporter PDF timeout, and opencode path separator assertions).

Screenshots / recordings (UI changes)

  • N/A

@github-actions github-actions Bot added docs Documentation area:desktop apps/desktop (Electron shell, renderer) area:core packages/core (generation orchestration) labels Apr 22, 2026
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] Unbounded image payload forwarding can exceed request limits and cause avoidable memory pressure — images are read and base64-encoded up to 10MB each, then every encoded image is appended to the same Codex request without a combined cap (apps/desktop/src/main/prompt-context.ts:43, apps/desktop/src/main/prompt-context.ts:109, apps/desktop/src/main/prompt-context.ts:155, apps/desktop/src/main/codex-generate.ts:157).
    Suggested fix:
    const MAX_TOTAL_IMAGE_BYTES = 4_000_000;
    let totalImageBytes = 0;
    
    for (const attachment of input.attachments) {
      if (!attachment.imageDataUrl) continue;
      const b64 = attachment.imageDataUrl.slice(attachment.imageDataUrl.indexOf(',') + 1);
      const bytes = Math.floor((b64.length * 3) / 4);
      if (totalImageBytes + bytes > MAX_TOTAL_IMAGE_BYTES) {
        throw new CodesignError(
          'Attached images are too large in total for ChatGPT Codex. Reduce image count/size.',
          ERROR_CODES.ATTACHMENT_TOO_LARGE,
        );
      }
      totalImageBytes += bytes;
      userContent.push({ type: 'input_image', image_url: attachment.imageDataUrl, detail: 'auto' });
    }

Summary
Review mode: initial

  • 1 issue found (Major), introduced by the current diff.
  • docs/VISION.md and docs/PRINCIPLES.md were not available in this checkout (Not found in repo/docs), so principle/vision checks were limited to in-repo code and CLAUDE.md.

Testing

  • Not run (automation). Add a Vitest case that enforces a combined image-byte limit and verifies a typed error is surfaced.

open-codesign Bot

Comment thread apps/desktop/src/main/codex-generate.ts Outdated
const userContent: ResponsesInputItem['content'] = [
{ type: 'input_text', text: buildUserPrompt(input.prompt, sections) },
];
for (const attachment of input.attachments) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forwarding every imageDataUrl here has no aggregate-size guard. With current limits, a user can attach multiple large images and produce a very large JSON request body (base64 expansion included), which risks request rejection and main-process memory spikes. Add a total image-byte cap before pushing input_image parts and throw a typed error when exceeded.

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • None.

Summary

  • Review mode: follow-up after new commits
  • No new issues found in the latest diff. The previously flagged aggregate image-size risk is now guarded by a total-byte cap with a typed error path (apps/desktop/src/main/codex-generate.ts).
  • docs/VISION.md and docs/PRINCIPLES.md: Not found in repo/docs.

Testing

  • Not run (automation). Residual gap: no explicit test for malformed/non-base64 imageDataUrl inputs.

open-codesign Bot

@github-actions github-actions Bot added the area:providers packages/providers (pi-ai adapter, model calls) label Apr 22, 2026
Comment thread packages/providers/src/index.ts Fixed
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • None.

Summary

  • Review mode: follow-up after new commits
  • No new high-confidence issues found in added/modified lines of the latest PR diff.
  • docs/VISION.md and docs/PRINCIPLES.md: Not found in repo/docs.
  • Residual risk: coverage is layer-local; not found in changed tests is an end-to-end assertion that a desktop image attachment is propagated through core.generate into provider userImages on the openai-codex-responses route.

Testing

  • Not run (automation).

open-codesign Bot

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] Silent drop of malformed image attachments on Codex wire — attachmentToImageInput returns null when imageDataUrl does not match the expected prefix, and imageInputsForWire filters those out without surfacing an error. This violates the project hard constraint that fallbacks must not be silent and can cause user-attached images to be omitted from model input with no UI signal. Evidence: packages/core/src/index.ts:176, packages/core/src/index.ts:190.
    Suggested fix:
    function attachmentToImageInput(attachment: AttachmentContext): { data: string; mimeType: string } | null {
      if (!attachment.imageDataUrl || !attachment.mediaType) return null;
      const prefix = `data:${attachment.mediaType};base64,`;
      if (!attachment.imageDataUrl.startsWith(prefix)) {
        throw new CodesignError(
          `Malformed image attachment payload for ${attachment.name}`,
          ERROR_CODES.ATTACHMENT_READ_FAILED,
        );
      }
      return {
        data: attachment.imageDataUrl.slice(prefix.length),
        mimeType: attachment.mediaType,
      };
    }

Summary

  • Review mode: follow-up after new commits
  • 1 issue found (silent fallback on malformed image payload handling).
  • docs/VISION.md and docs/PRINCIPLES.md were referenced in the PR template but are not present in this checkout (Not found in repo/docs), so constraint alignment was validated against CLAUDE.md plus diffed code paths only.

Testing

  • Not run (automation)

): { data: string; mimeType: string } | null {
if (!attachment.imageDataUrl || !attachment.mediaType) return null;
const prefix = `data:${attachment.mediaType};base64,`;
if (!attachment.imageDataUrl.startsWith(prefix)) return null;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Major] This branch silently drops malformed image payloads (return null then filtered out), which can omit user attachments without any surfaced error. Hard constraint says no silent fallbacks.

Suggested fix:

if (!attachment.imageDataUrl.startsWith(prefix)) {
  throw new CodesignError(
    `Malformed image attachment payload for ${attachment.name}`,
    ERROR_CODES.ATTACHMENT_READ_FAILED,
  );
}

@Sun-sunshine06 Sun-sunshine06 merged commit a965e58 into main Apr 22, 2026
7 checks passed
@Sun-sunshine06 Sun-sunshine06 deleted the fix/codex-image-attachments branch April 22, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core packages/core (generation orchestration) area:desktop apps/desktop (Electron shell, renderer) area:providers packages/providers (pi-ai adapter, model calls) docs Documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants