Elasticsearch Version
7.17.2
Installed Plugins
No response
Java Version
bundled
OS Version
macOS: Darwin ChristophersMBP 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:45:05 PDT 2022; root:xnu-8020.101.4~15/RELEASE_X86_64 x86_64
Problem Description
Elasticsearch fails to retrieve expected documents when using match_phrase to query for text which expands into synonyms containing stop words. Specifically, it is the documents which only contain the forms of the synonyms with stop words which fail to match.
Take for example an index with one configured stop word and a couple of synonym mappings, one which contains a synonym with that stop word, the other without:
PUT localhost:9200/my-index
{
"mappings": {
"properties": {
"text": {
"type": "text",
"analyzer": "analyzer-index",
"search_analyzer": "analyzer-search"
}
}
},
"settings": {
"index": {
"analysis": {
"filter": {
"synonym": {
"type": "synonym_graph",
"synonyms": [
"FBI, Federal Bureau of Investigation",
"CIA, Central Intelligence Agency"
]
},
"stop": {
"type": "stop",
"stopwords": [
"of"
]
}
},
"analyzer": {
"analyzer-index": {
"filter": [
"stop"
],
"tokenizer": "standard"
},
"analyzer-search": {
"filter": [
"synonym",
"stop"
],
"tokenizer": "standard"
}
}
}
}
}
}
I index four documents covering every synonym:
POST localhost:9200/my-index/_doc
{ "text": "FBI" } <--(1)
POST localhost:9200/my-index/_doc
{ "text": "Federal Bureau of Investigation" } <--(2)
POST localhost:9200/my-index/_doc
{ "text": "CIA" } <--(3)
POST localhost:9200/my-index/_doc
{ "text": "Central Intelligence Agency" } <--(4)
I then submit match_phrase queries for FBI and Federal Bureau of Investigation. I expect docs (1) and (2) to match to both queries, but only doc (1) matches both queries. Neither queries match doc (2):
POST localhost:9200/my-index/_search
{
"query": {
"match_phrase": {
"text": "FBI"
}
}
}
<<<
{
"took": 43,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 6.054264,
"hits": [
{
"_index": "my-index",
"_type": "_doc",
"_id": "0EYvRIABRbBbD8CRbxxv",
"_score": 6.054264,
"_source": {
"text": "FBI"
}
}
]
}
}
POST localhost:9200/my-index/_search
{
"query": {
"match_phrase": {
"text": "Federal Bureau of Investigation"
}
}
}
<<<
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 6.054264,
"hits": [
{
"_index": "my-index",
"_type": "_doc",
"_id": "0EYvRIABRbBbD8CRbxxv",
"_score": 6.054264,
"_source": {
"text": "FBI"
}
}
]
}
}
match_phrase queries for CIA and Central Intelligence Agency both match expected docs (3) and (4), indicating the missing doc (2) match above may have something to do with stop words:
POST localhost:9200/my-index/_search
{
"query": {
"match_phrase": {
"text": "CIA"
}
}
}
<<<
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 6.054264,
"hits": [
{
"_index": "my-index",
"_type": "_doc",
"_id": "0kYvRIABRbBbD8CRfRzU",
"_score": 6.054264,
"_source": {
"text": "CIA"
}
},
{
"_index": "my-index",
"_type": "_doc",
"_id": "00YvRIABRbBbD8CRhRwS",
"_score": 3.9980984,
"_source": {
"text": "Central Intelligence Agency"
}
}
]
}
}
POST localhost:9200/my-index/_search
{
"query": {
"match_phrase": {
"text": "Central Intelligence Agency"
}
}
}
<<<
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 6.054264,
"hits": [
{
"_index": "my-index",
"_type": "_doc",
"_id": "0kYvRIABRbBbD8CRfRzU",
"_score": 6.054264,
"_source": {
"text": "CIA"
}
},
{
"_index": "my-index",
"_type": "_doc",
"_id": "00YvRIABRbBbD8CRhRwS",
"_score": 3.9980984,
"_source": {
"text": "Central Intelligence Agency"
}
}
]
}
}
Inspecting one of the problematic queries with the _validate API reveals details about how the query gets rewritten to accommodate the synonym and stop filters:
POST localhost:9200/my-index/_validate/query?rewrite=true
{
"query": {
"match_phrase": {
"text": "Federal Bureau of Investigation"
}
}
}
<<<
{
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"valid": true,
"explanations": [
{
"index": "my-index",
"valid": true,
"explanation": "spanOr([text:FBI, spanNear([text:Federal, SpanGap(:1), text:Bureau, text:Investigation], 0, true)])"
}
]
}
I observe a few things:
- The query
Federal Bureau of Investigation is recognized as a synonym and the query is rewritten to include the other synonym in the mapping.
- The stop word
of in Federal Bureau of Investigation is recognized as a stop word, omitted from the rewritten query, and replaced with a span gap of width 1.
- The span gap is sequenced incorrectly in relation to the other terms
Federal, Bureau, Investigation.
A query for FBI is rewritten functionally identically:
POST localhost:9200/my-index/_validate/query?rewrite=true
{
"query": {
"match_phrase": {
"text": "FBI"
}
}
}
<<<
{
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"valid": true,
"explanations": [
{
"index": "my-index",
"valid": true,
"explanation": "spanOr([spanNear([text:Federal, SpanGap(:1), text:Bureau, text:Investigation], 0, true), text:FBI])"
}
]
}
Reconstructing the rewritten span query manually, but with the gap in the correct position, yields the expected matches docs (1) and (2):
POST localhost:9200/my-index/_search
{
"query": {
"span_or": {
"clauses": [
{ "span_term": { "text": "FBI" }},
{ "span_near": {
"clauses": [
{ "span_term": { "text": "Federal" }},
{ "span_term": { "text": "Bureau" }},
{ "span_gap": { "text": 1 }},
{ "span_term": { "text": "Investigation" }}
]
}}
]
}
}
}
<<<
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 6.054264,
"hits": [
{
"_index": "my-index",
"_type": "_doc",
"_id": "0EYvRIABRbBbD8CRbxxv",
"_score": 6.054264,
"_source": {
"text": "FBI"
}
},
{
"_index": "my-index",
"_type": "_doc",
"_id": "0UYvRIABRbBbD8CRdxwL",
"_score": 3.9980984,
"_source": {
"text": "Federal Bureau of Investigation"
}
}
]
}
}
_validate agrees this query is functionally identical to the rewritten match_phrase query, albeit the span gap is sequenced where expected.
POST localhost:9200/my-index/_validate/query?rewrite=true
{
"query": {
"span_or": {
"clauses": [
{ "span_term": { "text": "FBI" }},
{ "span_near": {
"clauses": [
{ "span_term": { "text": "Federal" }},
{ "span_term": { "text": "Bureau" }},
{ "span_gap": { "text": 1 }},
{ "span_term": { "text": "Investigation" }}
]
}}
]
}
}
}
<<<
{
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"valid": true,
"explanations": [
{
"index": "my-index",
"valid": true,
"explanation": "spanOr([text:FBI, spanNear([text:Federal, text:Bureau, SpanGap(:1), text:Investigation], 0, true)])"
}
]
}
Steps to Reproduce
- Create an index with the following characteristics:
- A
stop token filter with one or more stop words
- A
synonym_graph token filter with a mapping containing a synonym with a stop word
- An analyzer defined with these
synonym_graph and stop filters, in that relative order
- A text field mapping with this analyzer as its
search_analyzer
PUT localhost:9200/my-index
{
"mappings": {
"properties": {
"text": {
"type": "text",
"analyzer": "analyzer-index",
"search_analyzer": "analyzer-search"
}
}
},
"settings": {
"index": {
"analysis": {
"filter": {
"synonym": {
"type": "synonym_graph",
"synonyms": [
"FBI, Federal Bureau of Investigation"
]
},
"stop": {
"type": "stop",
"stopwords": [
"of"
]
}
},
"analyzer": {
"analyzer-index": {
"filter": [],
"tokenizer": "standard"
},
"analyzer-search": {
"filter": [
"synonym",
"stop"
],
"tokenizer": "standard"
}
}
}
}
}
}
- Index a document containing, in its text field, the synonym with a stop word. This document is the expected result.
POST localhost:9200/my-index/_doc
{
"text": "Federal Bureau of Investigation"
}
- Use a
match_phrase query to search for this document. The query can be any of the synonyms in the mapping. The expected result will not match.
POST localhost:9200/my-index/_search
{
"query": {
"match_phrase": {
"text": "Federal Bureau of Investigation"
}
}
}
<<<
{
"took": 19,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
}
POST localhost:9200/my-index/_search
{
"query": {
"match_phrase": {
"text": "FBI"
}
}
}
<<<
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
}
Logs (if relevant)
No response
Elasticsearch Version
7.17.2
Installed Plugins
No response
Java Version
bundled
OS Version
macOS: Darwin ChristophersMBP 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:45:05 PDT 2022; root:xnu-8020.101.4~15/RELEASE_X86_64 x86_64
Problem Description
Elasticsearch fails to retrieve expected documents when using
match_phraseto query for text which expands into synonyms containing stop words. Specifically, it is the documents which only contain the forms of the synonyms with stop words which fail to match.Take for example an index with one configured stop word and a couple of synonym mappings, one which contains a synonym with that stop word, the other without:
I index four documents covering every synonym:
I then submit
match_phrasequeries forFBIandFederal Bureau of Investigation. I expect docs (1) and (2) to match to both queries, but only doc (1) matches both queries. Neither queries match doc (2):match_phrasequeries forCIAandCentral Intelligence Agencyboth match expected docs (3) and (4), indicating the missing doc (2) match above may have something to do with stop words:Inspecting one of the problematic queries with the
_validateAPI reveals details about how the query gets rewritten to accommodate the synonym and stop filters:I observe a few things:
Federal Bureau of Investigationis recognized as a synonym and the query is rewritten to include the other synonym in the mapping.ofinFederal Bureau of Investigationis recognized as a stop word, omitted from the rewritten query, and replaced with a span gap of width 1.Federal,Bureau,Investigation.A query for
FBIis rewritten functionally identically:Reconstructing the rewritten span query manually, but with the gap in the correct position, yields the expected matches docs (1) and (2):
_validateagrees this query is functionally identical to the rewrittenmatch_phrasequery, albeit the span gap is sequenced where expected.Steps to Reproduce
stoptoken filter with one or more stop wordssynonym_graphtoken filter with a mapping containing a synonym with a stop wordsynonym_graphandstopfilters, in that relative ordersearch_analyzermatch_phrasequery to search for this document. The query can be any of the synonyms in the mapping. The expected result will not match.Logs (if relevant)
No response