Skip to content

ElasticSearch Instrumentor support for msearch #2021

Description

@daldoyle

Describe your environment

python3.8
opentelemetry-instrumentation-elasticsearch==0.41b0

Steps to reproduce

The elastic library converts msearch queries into a series of stringified json objects, which the otlp instrumentation library then complains about since it tries to decode them and flatten the objects. Here's an example code snippet that causes the behavior -

# regular ES library, happens to be version 7.17.9
from elasticsearch import Elasticsearch

# OTLP libraries, instrumentation=0.41b0, sdk=1.20.0
from opentelemetry.instrumentation.elasticsearch import ElasticsearchInstrumentor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import set_tracer_provider

# setup tracer and span exporting
tracer_provider = TracerProvider()
set_tracer_provider(tracer_provider)

span_exporter = OTLPSpanExporter()
tracer_provider.add_span_processor(BatchSpanProcessor(span_exporter))

# generic ES constructor
es = Elasticsearch(['https://my_es_server:9200'])

# create multiple queries, doesn't really matter what's in here as long
# as they're syntactically valid. It has to be more than 1 query due to the way the ES
# library concatenates the stringified versions of the queries
queries = []
queries.append({"index": "index1"})
queries.append({"size": 0,
                "query": {
                    "bool": {
                        "filter": [
                            {
                                "range": {
                                    "start": {
                                        "gte": 123,
                                        "lte": 456,
                                        "format": "epoch_millis"
                                    }
                                }
                            },
                            {
                                "query_string": {
                                    "analyze_wildcard": True,
                                    "query": "(foo: bar)"
                                }
                            }
                        ]
                    }
                }
              })

# enable ES instrumentation
ElasticsearchInstrumentor().instrument()

# try to msearch multiple queries. Kaboom
res = es.msearch(body=queries)

What is the expected behavior?
I expect the elastic queries to execute and the responses returned, along with traces emitted.

What is the actual behavior?
The code ends up throwing an exception.

Traceback (most recent call last):
  File "example.py", line 50, in <module>
    res = es.msearch(body=queries)
  File "./lib/python3.8/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "./lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 1226, in msearch
    return self.transport.perform_request(
  File "./lib/python3.8/site-packages/opentelemetry/instrumentation/elasticsearch/__init__.py", line 242, in wrapper
    attributes[SpanAttributes.DB_STATEMENT] = sanitize_body(
  File "./lib/python3.8/site-packages/opentelemetry/instrumentation/elasticsearch/utils.py", line 56, in sanitize_body
    body = json.loads(body)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)

Additional context
Here's the relevant bit where the ES library is converting multiple queries into a bulk request. The latest version in main handles this quite differently since the transport library was split out, but it looks like it still serializes the queries so msearch is likely broken there as well.

https://github.com/elastic/elasticsearch-py/blob/7.17/elasticsearch/client/utils.py#L354

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions