Skip to content

fix(pbl): don't cut an emoji in half when shortening evaluator prompt text - #2124

Merged
wyuc merged 1 commit into
THU-MAIC:mainfrom
kevin9327:fix/eval-prompts-surrogate-cut
Nov 3, 2026
Merged

wyuc merged 1 commit into
THU-MAIC:mainfrom
kevin9327:fix/eval-prompts-surrogate-cut

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

Summary

The evaluator prompt builders in lib/pbl/v2/operations/runtime/eval-prompts.ts shorten long learner text with text.slice(0, maxLength) + '…', which counts UTF-16 code units. When an emoji (or any character outside the BMP) sits across the limit, the slice keeps only its high surrogate, so the prompt sent to the evaluator model carries a lone surrogate. That is invalid Unicode: it shows up as \ud83d in the JSON request body, and some providers reject or replace it.

Two places do this:

  • promptText, used by formatProjectSynthesisChecks for the core concept (240), question, learner answer (700) and quality (40) in the final evaluation.
  • The closing-check answer in buildMilestoneEvalPrompt (140).

Related Issues

None filed. I found this while reading the evaluator prompt builders.

Changes

  • Add a small sliceCodeUnits(text, maxLength) helper that cuts one code unit earlier when the cut would split a surrogate pair, so the emoji is dropped whole. Limits stay in code units, so plain text is cut exactly where it was before.
  • Use it in promptText and for the closing-check answer.
  • Three tests in tests/pbl/v2/eval-prompts.test.ts.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or build changes

Verification

Steps to reproduce / test

  1. Record a stage_synthesis_check whose learner_answer is 699 ASCII characters followed by 😀.
  2. Call formatProjectSynthesisChecks(project).
  3. Before: the output ends the answer with \ud83d… (a lone high surrogate). After: it ends with the 699 characters and ….

What you personally verified

  • The two emoji tests fail on main (expected true to be false from the lone-surrogate check) and pass with the change. The plain-text test passes on both.
  • All 25 tests in tests/pbl/v2/eval-prompts.test.ts pass with vitest, run with a minimal config (the @/ alias and the workspace packages pointed at their sources) instead of a full pnpm install, so I did not run the repo's global setup files.
  • prettier --check (3.8.1) passes on both changed files.
  • Not verified: I did not run pnpm lint or npx tsc --noEmit locally. The change is a typed string helper and two call sites.

Evidence

Before the fix:

× evaluator prompt text cuts > does not cut an emoji in half when shortening a long learner answer
× evaluator prompt text cuts > does not cut an emoji in half when shortening a long closing answer
Tests  2 failed | 23 passed (25)

After:

Tests  25 passed (25)
  • CI passes (pnpm check && pnpm lint && npx tsc --noEmit)
  • Manually tested locally
  • Screenshots / recordings attached (if UI changes)

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have added/updated documentation as needed
  • My changes do not introduce new warnings

AI-assisted: I used an AI coding assistant for this change, reviewed the code and ran the tests.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T03:07:52.231700Z 28ded3d PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@wyuc wyuc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kevin9327. The cut now moves back only when it would split a surrogate pair, so plain text is unchanged. feedback.slice(0, 280) further down the same file still has the old pattern. It's model-written text, so it doesn't need to block this; a follow-up is welcome.

@wyuc
wyuc merged commit 70c83ef into THU-MAIC:main Nov 3, 2026
13 checks passed
wyuc pushed a commit that referenced this pull request Nov 3, 2026
…cript cuts (#2147)

Two more cuts in the PBL evaluator prompts counted UTF-16 code units, including one that keeps the tail of a role-play transcript, so an emoji at the limit left a lone surrogate. Both now drop the whole emoji, as #2124 did for the other cuts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants