Repository navigation
Regex allow_list: a backreference in a later term points at an earlier term's group #2433
Description
Describe the bug
With allow_list_match="regex", _remove_allow_list joins all terms into one pattern with "|".join(allow_list) (analyzer_engine.py:467). The capture groups then get numbers across the whole list. A backreference in a later term now points at a group of an earlier term. That term no longer matches its own text, and the result stays in the output with no error.
To Reproduce
import regex as re
from presidio_analyzer import AnalyzerEngine, RecognizerResult
text = "Contact: John Smith, ticket AA-AA"
person = RecognizerResult("PERSON", 9, 19, 0.85) # "John Smith"
ticket = RecognizerResult("TICKET", 28, 33, 0.9) # "AA-AA"
def kept(allow_list):
res = AnalyzerEngine._remove_allow_list(
[person, ticket], allow_list, text, re.DOTALL | re.MULTILINE | re.IGNORECASE, "regex")
return [text[r.start:r.end] for r in res]
print("alone 1:", kept([r"^(John) Smith$"]))
print("alone 2:", kept([r"^(\w\w)-\1$"]))
print("together:", kept([r"^(John) Smith$", r"^(\w\w)-\1$"]))$ .venv/bin/python presidio_allowlist_repro.py
alone 1: ['AA-AA']
alone 2: ['John Smith']
together: ['AA-AA']
# presidio-analyzer from main at 17d1946, Python 3.14.7, regex 2026.9.29
# not run: analyze() end to end with an NLP engine; it calls _remove_allow_list at line 310
Expected behavior
together: []. The analyze docstring says that with regex, "results which match with any regex condition in the allow_list would be allowed and not be returned as potential PII." AA-AA matches the second condition.
Additional context
Wrapping each term in (?:...) does not help, because the group numbers still shift. Since the module already imports regex, a branch reset fixes it in the join itself: "(?|" + "|".join(allow_list) + ")" numbers the groups per alternative, and on the case above it matches AA-AA and John Smith and still rejects AA-BB. Compiling each term on its own also works, but it costs one timeout-guarded search per term per result. #2061 fixed another side effect of the same join.
If the branch reset works for you, I'd send it as a PR with this case as a test.
Activity
jamesconwaydev commentedon Oct 31, 2026
I ran your case through
analyze()with two pattern recognizers on main at 17d1946, and it behaves the same end to end. The order matters as well.[r"^(John) Smith$", r"^(\w\w)-\1$"]keepsAA-AA, and the reverse order keeps nothing.Before you send the branch reset, there are terms it can't compile. A term with its own top-level
|becomes separate branches, and a call like(?1)becomes ambiguous once another term has a group 1. Usingregex2026.9.29 directly, with the engine's flags and.search():(?|join^(\d\d)-\1$|^(\w\w)=\2$AA=AAinvalid group reference^(John)$,^(\d\d)-\1$|^(\w\w)=\2$AA=AAinvalid group reference^(John)$,^(\w\w)-(?1)$AA-BBambiguous group referenceThe first row is the one to watch, since a single-term list that works today would stop working. The compile is outside the
try, so from reading the code I'd expect that to surface as an exception fromanalyze(), though I didn't patch it in and run that.Compiling each term once and checking them in turn matched in every case I tried, at the per-term search cost you mentioned. I haven't measured that cost.
chrikrah commentedon Nov 2, 2026
@jamesconwaydev your three rows reproduce. Wrapping each term as
(?:{t})inside the reset fixes the first two, but it fails open: a term with a stray)breaks out of its wrapper, compiles, and allow-lists every result.So my pick is per-term search, about 3 ms per 50 results at 20 terms, with your rows and the breakout term as tests. If a maintainer is fine with that direction, I'll open the PR.