Repository navigation
Prefer fuzzy alignment over a shorter partial exact match - #591
inamdarmihir wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
dae7c03 to
c9e1941
Compare
When an extraction shared only its opening tokens with the source, the exact phase accepted that short block as MATCH_LESSER and fuzzy alignment never ran for it. With paraphrased extractions (common for CJK text) this produced char_interval spans covering a couple of tokens instead of the full passage. Try LCS fuzzy alignment before settling for MATCH_LESSER and use it only when it matches more tokens than the exact block. Fixes google#289
c9e1941 to
08407ea
Compare
|
Your branch is 1 commits behind git fetch origin main
git merge origin/main
git pushNote: Enable "Allow edits by maintainers" to allow automatic updates. |
|
Your branch is 8 commits behind git fetch origin main
git merge origin/main
git pushNote: Enable "Allow edits by maintainers" to allow automatic updates. |
|
Your branch is 13 commits behind git fetch origin main
git merge origin/main
git pushNote: Enable "Allow edits by maintainers" to allow automatic updates. |
Description
WordAligner.align_extractionsaccepts the first exact difflib block that starts at an extraction's first token and labels itMATCH_LESSERwhen it is shorter than the extraction. Fuzzy alignment only runs for extractions that are still unaligned, so it never runs for these. When the model paraphrases an extraction that shares only its opening tokens with the source (very common with CJK text), the resultingchar_intervalcovers just those few tokens. In #289 a ~50 token extraction got an interval covering two characters.This change tries LCS fuzzy alignment before settling for
MATCH_LESSER, and uses the fuzzy span only when it matches strictly more tokens than the exact block (min_matches). Otherwise the existingMATCH_LESSERresult is kept, so cases where the partial block is already the best answer are unchanged. Nothing changes whenenable_fuzzy_alignment=False,accept_match_lesser=False, or with the legacy fuzzy algorithm.Reproduction from the issue's data with
UnicodeTokenizer: before,match_lesserover每份; after,match_fuzzyover每份汉堡约200克，含548大卡热量，其中蛋白质21.56克、脂肪18.12克、碳水化合物75.7克，能提供充足能量，但作为快餐食品，营养均衡方面有所欠缺.Behavior note:
tests/prompt_validation_test.pyhad a case (type-2 diabetesvsType 2 diabetes.) pinningMATCH_LESSER; it now resolves toMATCH_FUZZYwith the full span. It is still a non-exact alignment, so the validation issue count is unchanged.Fixes #289
Bug fix
How Has This Been Tested?
Added
shared_prefix_upgraded_to_fuzzyandshared_prefix_stays_lesser_without_fuzzycases toAlignEntitiesTest.tests/create_provider_plugin_test.pyhas 3 failures onmainin my environment (offline editable install / build isolation) unrelated to this change.Checklist:
pylintover the affected code.