Adding more context from GenAI SIG call:

• langchain, llamaindex, haystack offer retriever abstraction.
• the question is: should corresponding implementations be instrumented and if so, which conventions they should follow

My take:

Reading through lanchain, llamaindex, haystack docs, retriever is in most cases is an thin layer on top of a database or a search client which may be instrumented using database conventions and/or hypothetical search conventions.

Retrievers could be more complicated and combine multiple source of data or perform additional logic, in these cases, the spans they emit might be significantly different than underlying DB calls and then multiple layers may be instrumented at the same time. In the case of thin layer abstraction / wrapper, having two spans does not improve observability, but increases costs and noise.

When it comes to abstractions, there is a classic problem of instrumentation layers (ORM vs database spans, lanchain LLM vs underlying model-client spans, etc): both layers could be instrumented and there are pros and cons (DB layer has more low-level info, framework layer represents caller perspective better). The duplication is a common problem. Solutions may include:

• users decide to enable/disable corresponding instrumentation (by installing instr library or by enabling/disabling retrival spans on lanchain / llamaindex / etc instrumentation)
• suppressing internal DB spans in scope of higher level retrieval span (similar to http suppression or more broadly Instrumentation layers and suppressing duplicates oteps#172)

I think path forward for this PR:

• separate retrieval from GenAI domain. Most of the attributes defined here can be generic db or search attributes. Please follow the discussion in Should search engines follow database semantic conventions? #1869
• lanchain / llamaindex / etc instrumentation would cover retrieval following that db or search conventions and would provide an option to disable their retrieval instrumentation (e.g. when user prefers to use underlying existing DB client instrumentation)

Also related #1231

Hi @lmolkova,

Here are a few examples from trace loop and our instrumentation of proposed retrieval spans, also updated in the google doc linked to the issue
https://docs.google.com/document/d/1DVE3Ht686nuxww-Z1JNyy2VInn9LeCJQYw8NffbjGtg/edit?tab=t.0.

I believe, we also discussed that we are fine moving retrieval spans from the genai to db spaces, but would the optionality to have or enable/disable genai attributes be possible as well, similar to enabling/disabling retrievals as a whole? So that retrievals could parent a db + embedding operation.

This PR was marked stale due to lack of activity. It will be closed in 7 days.

This PR contains changes to area(s) that do not have an active SIG/project and will be auto-closed:

• database

Such changes may be rejected or put on hold until a new SIG/project is established.

Please refer to the Semantic Convention Areas
document to see the current active SIGs and also to learn how to kick start a new one.

Retrieval Semconv Proposal doc (sample telemetry, attributes), adding from the open issue to the PR for more exposure:
https://docs.google.com/document/d/1DVE3Ht686nuxww-Z1JNyy2VInn9LeCJQYw8NffbjGtg/edit?tab=t.0

This PR has been labeled as stale due to lack of activity. It will be automatically closed if there is no further activity over the next 7 days.

Commenting to keep PR open