Skip to content

Support a configurable allow list in the analyzer configuration file #2455

Open

Description

@MaklTa

Is your feature request related to a problem? Please describe.
Organizations have their own vocabulary that the analyzer keeps reporting as PII although it never is: company and product names, internal role titles, project names, short codes. In German, spaCy reports e.g. "Contoso" or "Digitaler Arbeitsplatz" as LOCATION or ORGANIZATION.

The only way to suppress them is the per-request allow_list of AnalyzerEngine.analyze() / /analyze:

  • every client of the analyzer service has to maintain and send the same list with each request; a deployment cannot enforce one list for all clients, and the list cannot live next to ANALYZER_CONF_FILE, NLP_CONF_FILE and RECOGNIZER_REGISTRY_CONF_FILE.
  • allow_list_match="exact" is case-sensitive (word not in allow_list), so "contoso" or "CONTOSO" are still reported.
  • allow_list_match="regex" uses search instead of a full match, so a term also drops every longer result that merely contains it ("Contoso Berlin" as LOCATION). Clients have to anchor and escape every term themselves.

Describe the solution you'd like
An allow_list section in the analyzer configuration read by AnalyzerEngineProvider, applied by AnalyzerEngine to every request in addition to the per-request allow_list:

supported_languages:
  - de
allow_list:
  terms:                              # inline terms ...
    - Contoso
    - Digitaler Arbeitsplatz
  file: /app/conf/allow-list-de.txt   # ... and/or one term per line, # for comments
  match: exact                        # exact | regex, as for the request parameter
  ignore_case: true                   # new: case-insensitive exact matching
  • optionally per language (allow_list: {de: ..., en: ...}), since terms and their inflected forms differ by language.
  • exact matching with an option to ignore case and normalize whitespace, so multi-word terms also match across a line break.
  • an environment variable for the analyzer image (e.g. ALLOW_LIST_FILE), like the existing *_CONF_FILE variables.

Describe alternatives you've considered

  • Per-request allow_list: cannot be enforced centrally and has the matching limits above.
  • A custom recognizer: recognizers can only add results, not remove those of other recognizers (in particular NER results).
  • Wrapping app.py, subclassing AnalyzerEngineProvider and filtering the results of analyze() (what we do now): works, but relies on how app.py builds its engine and has to be re-checked on every upgrade.
  • Post-processing in each client: duplicates the list and the matching logic everywhere.

Additional context

  • presidio-analyzer 2.2.366 (analyzer service image).
  • AnalyzerEngineProvider already reads engine options from the configuration file (supported_languages, default_score_threshold, context_aware_enhancer), so the allow list would fit the same mechanism.
  • Current filtering: AnalyzerEngine._remove_allow_list in presidio_analyzer/analyzer_engine.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions