prompt-score: strip pasted prose blocks so quoted reviews don't inflate#258
Merged
Conversation
Validating v2 against 300 real logged prompts showed the score was still a strong length proxy (Pearson r(score, log-words) 0.78) and the top of the distribution was dominated by prompts that paste a long review / feedback / another agent's prose and add a thin ask. The artifact segmenter strips machine output (logs, traces, diffs, fences) but not pasted natural language, so that quoted prose was counted as authored craft, crediting the quoted author, not the human. This is the prose analogue of the original log-paste bug. strip_quoted_prose removes long or multi-line quoted spans (straight, curly, and Japanese quotes) before feature extraction; short quotes (a term, a line of UI copy) are kept as concrete grounding. Removed prose no longer feeds grounding / diversity / length or language detection, and instead credits only the saturating evidence signal; a thin ask around a pasted block is flagged MostlyPaste. On the 300-prompt corpus this cut r(score, log-words) 0.78 -> 0.66, collapsed the 300+-authored-word bucket from 20 prompts to 4 (their pasted content is gone), and dropped the pasted-review prompts out of the top band. Notably, 8 of 10 'Japanese' prompts were English asks quoting Japanese reviews — they now correctly detect as English and score on the ask. English calibration byte-identical. 54 module + 5 integration tests green; clippy clean both configs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validating v2 against 300 real logged prompts showed the score was still a strong length proxy (Pearson r(score, log-words) 0.78) and the top of the distribution was dominated by prompts that paste a long review / feedback / another agent's prose and add a thin ask. The artifact segmenter strips machine output (logs, traces, diffs, fences) but not pasted natural language, so that quoted prose was counted as authored craft, crediting the quoted author, not the human. This is the prose analogue of the original log-paste bug.
strip_quoted_prose removes long or multi-line quoted spans (straight, curly, and Japanese quotes) before feature extraction; short quotes (a term, a line of UI copy) are kept as concrete grounding. Removed prose no longer feeds grounding / diversity / length or language detection, and instead credits only the saturating evidence signal; a thin ask around a pasted block is flagged MostlyPaste.
On the 300-prompt corpus this cut r(score, log-words) 0.78 -> 0.66, collapsed the 300+-authored-word bucket from 20 prompts to 4 (their pasted content is gone), and dropped the pasted-review prompts out of the top band. Notably, 8 of 10 'Japanese' prompts were English asks quoting Japanese reviews — they now correctly detect as English and score on the ask. English calibration byte-identical. 54 module + 5 integration tests green; clippy clean both configs.