Skip to content

Commit 2f85799

Browse files
rkishoreclaude
andcommitted
Link the invariant-globalization report, dotnet/machinelearning#7728
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 1140158 commit 2f85799

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ configuration, and every divergence is in tokenization.
3434
`Microsoft.ML.Tokenizers` 2.0.0 with `RemoveNonSpacingMarks = true`: under invariant mode
3535
the option does nothing on precomposed text, so `Café crème brûlée in São Paulo, naïve
3636
résumé` scores 0.333027, the same as with the option off. On text that arrives already
37-
decomposed, it still works.
37+
decomposed, it still works. Reported as
38+
[dotnet/machinelearning#7728](https://github.com/dotnet/machinelearning/issues/7728).
3839

3940
Invariant globalization is common where .NET is deployed:
4041

@@ -150,6 +151,9 @@ that `PREDICTIONS.md` was written for.
150151
`BertTokenizer` merges words separated only by `\n`, `\t` or `\r`.
151152
- [dotnet/machinelearning#7725](https://github.com/dotnet/machinelearning/issues/7725):
152153
`BertTokenizer` silently drops symbol characters.
154+
- [dotnet/machinelearning#7728](https://github.com/dotnet/machinelearning/issues/7728):
155+
`RemoveNonSpacingMarks = true` fails to strip accents from precomposed characters under
156+
invariant globalization.
153157
- [elbruno/elbruno.localembeddings#56](https://github.com/elbruno/elbruno.localembeddings/issues/56):
154158
accented words become `[UNK]` with the default model.
155159
- [iyulab/lm-supply#12](https://github.com/iyulab/lm-supply/issues/12): embeddings diverge

0 commit comments

Comments
 (0)