prompt-score: add offline Japanese-language support#257
Merged
Conversation
The scorer is now script-aware. Japanese prompts were previously abstained (unscored) by the non-English guard; they are now scored on a dedicated path, dictionary-free and fully offline — no morphological-analyser/MeCab dependency, consistent with the module contract. Design: detect_lang() routes on kana presence. Japanese is tokenized by character-class run (tokenize_ja, no spaces to split on) for diversity/ repetition, sized by a char/2 word-equivalent, and matched against parallel JA_* slot lexicons by substring (a space-less script has no word boundaries). The slot rubric, multiplicative core, guards, and aggregation are unchanged — they consume feature counts, so they work for either script. Readability indices are English-specific, so Japanese uses a sentence-length clarity proxy (ja_clarity); the English Flesch/FK/Fog path is untouched. Sentence counting now also splits on 。!?. Languages other than English/Japanese still abstain. Japanese calibration parallels English closely (vague~12 nascent, tactical~53, rich~56, loaded~58 proficient). English calibration is byte-identical — the JA path is fully isolated. PromptScore gains a public field so callers can label the score. Known limitation (documented): substring matching lets a precise action stem (変更) also match inside a prohibition (変更しない), so a pure-prohibition JP prompt scores a little objective credit it wouldn't in English. 49 module + 5 integration tests green; clippy clean both configs.
The initial Japanese support dispatched each slot to a single lexicon set by detected language, so a code-switched prompt lost whatever was not the dominant script (an English 'Do not change ...' in an otherwise-Japanese prompt scored nothing). Make slot detection bilingual: detect_lang now only selects the tokenizer and readability path, while every slot counts hits from BOTH the English and Japanese lexicons and sums them. Safe for the pure cases: ja_hits is zero on kana-free text and the English word-gated counter is ~zero on kana text, so single-language calibration is unchanged. Fixes a double-count this exposed: JA_OUTPUT_SHAPE had ASCII json/yaml/markdown, which then counted twice against the always-matched English OUTPUT_SHAPE (drifted English tactical 51.1 to 51.7). JA_* lexicons now hold Japanese-script entries only; the invariant is documented. imperative_open and executable_acceptance are likewise unioned across languages. Adds the missing coverage the reviewer asked for: two mixed English+Japanese prose tests proving an English constraint is credited in a JP-dominant prompt and a JP constraint alongside an English ask. English calibration byte-identical; Japanese prompts containing ASCII (e.g. cargo test) now correctly gain the English-lexicon credit they were missing. 51 module + 5 integration tests green; clippy clean both configs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.