Repository navigation
Description
Is your feature request related to a problem? Please describe.
Presidio has country-specific recognizers for Singapore, Thailand, the Philippines and others, but none for Malaysia. The Malaysian
identity card number (MyKad / NRIC) is the most common personal identifier in Malaysian documents: forms, payslips, insurance letters,
customer service chats. Stock Presidio detects none of them.
Describe the solution you'd like
A predefined MyNricRecognizer under country_specific/malaysia/, entity MY_NRIC, COUNTRY_CODE = "my", shipped enabled: false:
- 12 digits
YYMMDD-PB-####, written dashed, spaced or compact. There is no check digit, so validation is structural:YYMMDDmust be a real calendar date not in the future (1900s or 2000s;000229is valid only as 2000).PBmust be one of the 86 place-of-birth codes published by the National Registration Department (JPN):
https://www.jpn.gov.my/my/kod-negeri
- Scores stay low without context (0.5 dashed, 0.3 compact): a compact 12-digit run that passes the date and code check is still often
an account, member or policy number. Context words in English and Malay (ic,nric,mykad,kad pengenalan,no. kp, ...)
raise the score.
Then, in a separate PR after feedback on MY_NRIC:
MY_SSM: Companies Commission of Malaysia (SSM) registration numbers. Since 11 October 2019 the format is 12 digits: registration
year, a two-digit entity type code (01 to 06) and a serial. The older company format (123456-X) would only be reported with context,
because the same shape is common in product codes.
I am not proposing a phone recognizer. PhoneRecognizer does not list MY in its default regions, but in my data it still finds
0.956 of Malaysian numbers; its errors are false positives (timestamps, reference numbers), which a Malaysian recognizer would not fix
by itself.
Evidence so far
I wrote these as Presidio PatternRecognizer subclasses in MyPII (MIT) and built a synthetic
benchmark of Malay, English and Manglish (mixed) text, with hard negatives: account, membership, policy and invoice numbers that look
like MyKad or SSM numbers. All names and numbers are synthetic. The split below (1,000 documents, templates written by a model family
not used anywhere else) was sealed by sha256 before any system ran on it and scored once, following a pre-registration in the repo.
Exact-span F1:
| system | all entities | MY_NRIC | MY_SSM |
|---|---|---|---|
| stock Presidio | 0.419 | 0.0 | 0.0 |
| malaysia-pii-guard | 0.782 (0.900 on the entities it supports) | 0.826 | not supported |
| zero-shot Llama 3.3 70B | 0.671 | 0.661 | 0.603 |
| zero-shot Claude Haiku 5.5 | 0.873 | 0.894 | 0.931 |
| MyPII pattern recognizers | 0.940 | 0.993 | 0.799 |
Caveats, so the numbers are not over-read:
- The data are synthetic and I wrote both the generator and the recognizers.
MY_NRICis strong (precision 0.993, recall 0.993), which is why I would start there.MY_SSMis the weak point: precision 0.677,
and zero-shot LLMs do better on it.- On a later split with OCR noise and line-broken numbers, the pattern-only recognizers drop to F1 0.587. The current MyPII recognizers handle those with
number normalisation and a small logistic-regression context classifier (0.980), but that classifier sits outside Presidio's pattern
model, so the upstream PR would cover only the pattern and validation part.
Credit. MyPII is not the first MyKad detector. malaysia-pii-guard (MIT)
already validates the birth date and state code, and covers passports and bank accounts. It has its own Presidio-style engine and is
not packaged on PyPI. I used it as a baseline, and its author may well want to be involved in an upstream version.
Describe alternatives you've considered
- Keep the recognizers in a separate package (MyPII; the PyPI release is pending). That works, but every Malaysian user has to find
it, and Presidio's country filter (load_predefined_recognizers(countries=["my"])) cannot include it. - Zero-shot LLMs: reasonable F1, but a network call per document and the text leaves the machine.
Additional context
- Malay text is analysed with
supported_language="en": the recognizers are pattern-based, and the context words are listed in
both languages. Happy to follow whatever convention you prefer for languages without a default spaCy pipeline. - MyPII also drops a match when a negative context word ("account", "akaun", "invoice", ...) is closer than any positive one. That is
outside the standard context enhancer, so the upstream PR would leave it out unless you want it.
If this is welcome, I'll open a PR for MY_NRIC with tests following developing_recognizers.md, and wait for review before MY_SSM.
Activity