This is a documentation request for existing default behavior in the analysis-nori plugin.
The default nori analyzer uses the nori_part_of_speech token filter. Its default stoptags include XPN, so when Nori emits a meaning-carrying Korean prefix as XPN, that prefix is removed from the analyzed term stream.
Minimal examples verified against Lucene main (6b67118c4a0f):
bare KoreanTokenizer(MIXED, no user dict)
비급여 -> 비/XPN, 급여/NNG
부담보 -> 부/XPN, 담보/NNG
비갱신형 -> 비/XPN, 갱신/NNG, 형/XSN
주계약 -> 주/MM, 계약/NNG
KoreanAnalyzer(default chain)
비급여 -> 급여/NNG
부담보 -> 담보/NNG
비갱신형 -> 갱신/NNG
주계약 -> 계약/NNG
Elasticsearch-level _analyze corroboration from analysis-nori 8.15.3: analyzing 비급여 with the nori analyzer returns a single token 급여.
Small public-data Lucene demonstration (two documents, default KoreanAnalyzer):
import java.util.*; import org.apache.lucene.analysis.*; import org.apache.lucene.analysis.ko.*; import org.apache.lucene.analysis.tokenattributes.*; import org.apache.lucene.document.*; import org.apache.lucene.index.*; import org.apache.lucene.search.*; import org.apache.lucene.store.*;
public class MinimalNoriPrefixDemo {
static final String F = "body";
public static void main(String[] args) throws Exception {
Analyzer a = new KoreanAnalyzer();
ByteBuffersDirectory dir = new ByteBuffersDirectory();
IndexWriter w = new IndexWriter(dir, new IndexWriterConfig(a));
add(w, "1", "비급여 항목 안내"); add(w, "2", "급여 항목 안내"); w.commit();
List<String> qTerms = terms(a, "비급여");
System.out.println("query terms: " + qTerms);
IndexSearcher s = new IndexSearcher(DirectoryReader.open(w));
Query q = new TermQuery(new Term(F, qTerms.get(0)));
for (ScoreDoc hit : s.search(q, 10).scoreDocs) {
Document doc = s.storedFields().document(hit.doc);
System.out.printf("id=%s score=%.6f body=%s%n", doc.get("id"), hit.score, doc.get(F));
}
}
static void add(IndexWriter w, String id, String body) throws Exception {
Document d = new Document(); d.add(new TextField("id", id, Field.Store.YES));
d.add(new TextField(F, body, Field.Store.YES)); w.addDocument(d);
}
static List<String> terms(Analyzer a, String text) throws Exception {
List<String> out = new ArrayList<>(); TokenStream ts = a.tokenStream(F, text);
CharTermAttribute term = ts.addAttribute(CharTermAttribute.class);
ts.reset(); while (ts.incrementToken()) out.add(term.toString()); ts.end(); ts.close();
return out;
}
}Output:
query terms: [급여]
id=1 score=0.082873 body=비급여 항목 안내
id=2 score=0.082873 body=급여 항목 안내
Note that both documents score identically: after analysis the index cannot distinguish 비급여 (non-covered) from 급여 (covered) at all, although they mean the opposite in insurance/medical text.
Suggested documentation update: add a warning to docs/reference/elasticsearch-plugins/analysis-nori-speech.md after the default stoptags list, with the 비급여 example and the two verified remedies — registering such terms via user_dictionary_rules (term-specific, survives the POS filter as a single noun token), or defining a custom stoptags list that omits XPN (preserves all prefixes, can add prefix noise).
Happy to open a docs PR with the wording above if this is useful.
This is a documentation request for existing default behavior in the
analysis-noriplugin.The default
norianalyzer uses thenori_part_of_speechtoken filter. Its defaultstoptagsincludeXPN, so when Nori emits a meaning-carrying Korean prefix asXPN, that prefix is removed from the analyzed term stream.Minimal examples verified against Lucene main (
6b67118c4a0f):Elasticsearch-level
_analyzecorroboration fromanalysis-nori8.15.3: analyzing비급여with thenorianalyzer returns a single token급여.Small public-data Lucene demonstration (two documents, default KoreanAnalyzer):
Output:
Note that both documents score identically: after analysis the index cannot distinguish 비급여 (non-covered) from 급여 (covered) at all, although they mean the opposite in insurance/medical text.
Suggested documentation update: add a warning to
docs/reference/elasticsearch-plugins/analysis-nori-speech.mdafter the defaultstoptagslist, with the비급여example and the two verified remedies — registering such terms viauser_dictionary_rules(term-specific, survives the POS filter as a single noun token), or defining a customstoptagslist that omitsXPN(preserves all prefixes, can add prefix noise).Happy to open a docs PR with the wording above if this is useful.