@@ -34,7 +34,8 @@ configuration, and every divergence is in tokenization.
3434 ` Microsoft.ML.Tokenizers ` 2.0.0 with ` RemoveNonSpacingMarks = true ` : under invariant mode
3535 the option does nothing on precomposed text, so `Café crème brûlée in São Paulo, naïve
3636 résumé` scores 0.333027, the same as with the option off. On text that arrives already
37- decomposed, it still works.
37+ decomposed, it still works. Reported as
38+ [ dotnet/machinelearning #7728 ] ( https://github.com/dotnet/machinelearning/issues/7728 ) .
3839
3940Invariant globalization is common where .NET is deployed:
4041
@@ -150,6 +151,9 @@ that `PREDICTIONS.md` was written for.
150151 ` BertTokenizer ` merges words separated only by ` \n ` , ` \t ` or ` \r ` .
151152- [ dotnet/machinelearning #7725 ] ( https://github.com/dotnet/machinelearning/issues/7725 ) :
152153 ` BertTokenizer ` silently drops symbol characters.
154+ - [ dotnet/machinelearning #7728 ] ( https://github.com/dotnet/machinelearning/issues/7728 ) :
155+ ` RemoveNonSpacingMarks = true ` fails to strip accents from precomposed characters under
156+ invariant globalization.
153157- [ elbruno/elbruno.localembeddings #56 ] ( https://github.com/elbruno/elbruno.localembeddings/issues/56 ) :
154158 accented words become ` [UNK] ` with the default model.
155159- [ iyulab/lm-supply #12 ] ( https://github.com/iyulab/lm-supply/issues/12 ) : embeddings diverge
0 commit comments