Skip to content

Add Malaysian recognizers (MY_NRIC first, then MY_SSM) #2458

Open

Description

@aimanmalib

Is your feature request related to a problem? Please describe.
Presidio has country-specific recognizers for Singapore, Thailand, the Philippines and others, but none for Malaysia. The Malaysian
identity card number (MyKad / NRIC) is the most common personal identifier in Malaysian documents: forms, payslips, insurance letters,
customer service chats. Stock Presidio detects none of them.

Describe the solution you'd like
A predefined MyNricRecognizer under country_specific/malaysia/, entity MY_NRIC, COUNTRY_CODE = "my", shipped enabled: false:

  • 12 digits YYMMDD-PB-####, written dashed, spaced or compact. There is no check digit, so validation is structural:
  • Scores stay low without context (0.5 dashed, 0.3 compact): a compact 12-digit run that passes the date and code check is still often
    an account, member or policy number. Context words in English and Malay (ic, nric, mykad, kad pengenalan, no. kp, ...)
    raise the score.

Then, in a separate PR after feedback on MY_NRIC:

  • MY_SSM: Companies Commission of Malaysia (SSM) registration numbers. Since 11 October 2019 the format is 12 digits: registration
    year, a two-digit entity type code (01 to 06) and a serial. The older company format (123456-X) would only be reported with context,
    because the same shape is common in product codes.

I am not proposing a phone recognizer. PhoneRecognizer does not list MY in its default regions, but in my data it still finds
0.956 of Malaysian numbers; its errors are false positives (timestamps, reference numbers), which a Malaysian recognizer would not fix
by itself.

Evidence so far
I wrote these as Presidio PatternRecognizer subclasses in MyPII (MIT) and built a synthetic
benchmark of Malay, English and Manglish (mixed) text, with hard negatives: account, membership, policy and invoice numbers that look
like MyKad or SSM numbers. All names and numbers are synthetic. The split below (1,000 documents, templates written by a model family
not used anywhere else) was sealed by sha256 before any system ran on it and scored once, following a pre-registration in the repo.
Exact-span F1:

system all entities MY_NRIC MY_SSM
stock Presidio 0.419 0.0 0.0
malaysia-pii-guard 0.782 (0.900 on the entities it supports) 0.826 not supported
zero-shot Llama 3.3 70B 0.671 0.661 0.603
zero-shot Claude Haiku 5.5 0.873 0.894 0.931
MyPII pattern recognizers 0.940 0.993 0.799

Caveats, so the numbers are not over-read:

  • The data are synthetic and I wrote both the generator and the recognizers.
  • MY_NRIC is strong (precision 0.993, recall 0.993), which is why I would start there. MY_SSM is the weak point: precision 0.677,
    and zero-shot LLMs do better on it.
  • On a later split with OCR noise and line-broken numbers, the pattern-only recognizers drop to F1 0.587. The current MyPII recognizers handle those with
    number normalisation and a small logistic-regression context classifier (0.980), but that classifier sits outside Presidio's pattern
    model, so the upstream PR would cover only the pattern and validation part.

Credit. MyPII is not the first MyKad detector. malaysia-pii-guard (MIT)
already validates the birth date and state code, and covers passports and bank accounts. It has its own Presidio-style engine and is
not packaged on PyPI. I used it as a baseline, and its author may well want to be involved in an upstream version.

Describe alternatives you've considered

  • Keep the recognizers in a separate package (MyPII; the PyPI release is pending). That works, but every Malaysian user has to find
    it, and Presidio's country filter (load_predefined_recognizers(countries=["my"])) cannot include it.
  • Zero-shot LLMs: reasonable F1, but a network call per document and the text leaves the machine.

Additional context

  • Malay text is analysed with supported_language="en": the recognizers are pattern-based, and the context words are listed in
    both languages. Happy to follow whatever convention you prefer for languages without a default spaCy pipeline.
  • MyPII also drops a match when a negative context word ("account", "akaun", "invoice", ...) is closer than any positive one. That is
    outside the standard context enhancer, so the upstream PR would leave it out unless you want it.

If this is welcome, I'll open a PR for MY_NRIC with tests following developing_recognizers.md, and wait for review before MY_SSM.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions