Repository navigation
Open
Description
Is your feature request related to a problem? Please describe.
Organizations have their own vocabulary that the analyzer keeps reporting as PII although it never is: company and product names, internal role titles, project names, short codes. In German, spaCy reports e.g. "Contoso" or "Digitaler Arbeitsplatz" as LOCATION or ORGANIZATION.
The only way to suppress them is the per-request allow_list of AnalyzerEngine.analyze() / /analyze:
- every client of the analyzer service has to maintain and send the same list with each request; a deployment cannot enforce one list for all clients, and the list cannot live next to
ANALYZER_CONF_FILE,NLP_CONF_FILEandRECOGNIZER_REGISTRY_CONF_FILE. allow_list_match="exact"is case-sensitive (word not in allow_list), so "contoso" or "CONTOSO" are still reported.allow_list_match="regex"usessearchinstead of a full match, so a term also drops every longer result that merely contains it ("Contoso Berlin" asLOCATION). Clients have to anchor and escape every term themselves.
Describe the solution you'd like
An allow_list section in the analyzer configuration read by AnalyzerEngineProvider, applied by AnalyzerEngine to every request in addition to the per-request allow_list:
supported_languages:
- de
allow_list:
terms: # inline terms ...
- Contoso
- Digitaler Arbeitsplatz
file: /app/conf/allow-list-de.txt # ... and/or one term per line, # for comments
match: exact # exact | regex, as for the request parameter
ignore_case: true # new: case-insensitive exact matching- optionally per language (
allow_list: {de: ..., en: ...}), since terms and their inflected forms differ by language. exactmatching with an option to ignore case and normalize whitespace, so multi-word terms also match across a line break.- an environment variable for the analyzer image (e.g.
ALLOW_LIST_FILE), like the existing*_CONF_FILEvariables.
Describe alternatives you've considered
- Per-request
allow_list: cannot be enforced centrally and has the matching limits above. - A custom recognizer: recognizers can only add results, not remove those of other recognizers (in particular NER results).
- Wrapping
app.py, subclassingAnalyzerEngineProviderand filtering the results ofanalyze()(what we do now): works, but relies on howapp.pybuilds its engine and has to be re-checked on every upgrade. - Post-processing in each client: duplicates the list and the matching logic everywhere.
Additional context
- presidio-analyzer 2.2.366 (analyzer service image).
AnalyzerEngineProvideralready reads engine options from the configuration file (supported_languages,default_score_threshold,context_aware_enhancer), so the allow list would fit the same mechanism.- Current filtering:
AnalyzerEngine._remove_allow_listinpresidio_analyzer/analyzer_engine.py.
Metadata
Metadata
Assignees
Labels
No labels
Activity