Skip to content

Keep tool and agent names on GenAI semconv execute_tool and invoke_agent spans - #26394

Merged
aaronteo-db merged 2 commits into
mlflow:masterfrom
chrikrah:fix/genai-semconv-tool-agent-names
Oct 4, 2026
Merged

aaronteo-db merged 2 commits into
mlflow:masterfrom
chrikrah:fix/genai-semconv-tool-agent-names

Conversation

@chrikrah

@chrikrah chrikrah commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Related Issues/PRs

Closes #26316

What changes are proposed in this pull request?

_build_genai_span_name (translator.py:173-175) named TOOL and AGENT spans execute_tool and invoke_agent, or {operation} {model} with a model, dropping the tool or agent name in the span name.

  • Following the table in GenAI semconv export drops the tool and agent names from execute_tool and invoke_agent spans #26316, gen_ai.tool.name and gen_ai.agent.name come from the span name, which becomes execute_tool {name} or invoke_agent {name}, even with a model, per gen-ai-spans.md:1148 and gen-ai-agent-spans.md:555. An autologger that names spans {Class}.{method} exports that string, so a pydantic-ai 2.x agent.run() exports invoke_agent Agent.run.
  • A span with no name, or named after its operation, exports as before and gets no name attribute.
  • A name attribute the span already carries wins, because Strands and Semantic Kernel set one and name spans execute_tool {name} and invoke_agent {name}. The translator overwrote those names with the bare operation, or with invoke_agent {model} for a Strands agent with a model_id.

How is this PR tested?

  • Existing unit/integration tests
  • New unit/integration tests
  • Manual tests
# master at 59dac6f
$ python repro_26316.py    # the snippet from #26316
'get_weather' (TOOL) -> name='execute_tool' attrs=['gen_ai.operation.name', 'gen_ai.tool.call.arguments', 'gen_ai.tool.call.result']
'travel_planner' (AGENT) -> name='invoke_agent' attrs=['gen_ai.operation.name']
'my_pipeline' (CHAIN) -> name='CHAIN' attrs=['gen_ai.operation.name']
$ python repro_agent_run.py    # the same snippet with the one span pydantic_ai/autolog_v2.py:97 starts: Agent.run, typed AGENT
'Agent.run' (AGENT) -> name='invoke_agent' attrs=['gen_ai.operation.name']
$ python -m pytest tests/strands/test_semconv_probe.py -q -s | grep '^name='    # scratch test, not in this PR: a Strands agent with one tool, semconv export on
name='execute_tool' tool.name='sum' agent.name=None
name='invoke_agent' tool.name=None agent.name='agent'
$ python -m pytest tests/strands/test_semconv_probe_modelid.py -q -s | grep '^name=.*invoke'    # also scratch: the same agent with a model_id, as a default Agent() has
name='invoke_agent us.anthropic.claude-sonnet-4' tool.name=None agent.name='agent' model='us.anthropic.claude-sonnet-4'

# with this change
$ python repro_26316.py
'get_weather' (TOOL) -> name='execute_tool get_weather' attrs=['gen_ai.operation.name', 'gen_ai.tool.call.arguments', 'gen_ai.tool.call.result', 'gen_ai.tool.name']
'travel_planner' (AGENT) -> name='invoke_agent travel_planner' attrs=['gen_ai.agent.name', 'gen_ai.operation.name']
'my_pipeline' (CHAIN) -> name='CHAIN' attrs=['gen_ai.operation.name']
$ python repro_agent_run.py
'Agent.run' (AGENT) -> name='invoke_agent Agent.run' attrs=['gen_ai.agent.name', 'gen_ai.operation.name']
$ python -m pytest tests/strands/test_semconv_probe.py -q -s | grep '^name='
name='execute_tool sum' tool.name='sum' agent.name=None
name='invoke_agent agent' tool.name=None agent.name='agent'
$ python -m pytest tests/strands/test_semconv_probe_modelid.py -q -s | grep '^name=.*invoke'
name='invoke_agent agent' tool.name=None agent.name='agent' model='us.anthropic.claude-sonnet-4'
$ python -m pytest -q -n 4 -p no:cacheprovider tests/tracing/export/genai_semconv/test_translator.py tests/anthropic/test_anthropic_genai_semconv_converter.py \
    tests/bedrock/test_genai_semconv_converter.py tests/gemini/test_gemini_genai_semconv_converter.py \
    tests/openai/test_genai_semconv_converter.py tests/tracing/otel/test_span_translation.py
265 passed, 5 warnings in 23.56s
# baseline at 59dac6f: 257 passed, so the delta is the 8 new cases
# translator.py reverted, tests kept: test_translator.py gives 6 failed, 32 passed (test_full_tool_span,
# test_full_agent_span, both *_is_kept cases, the two updated test_span_name_and_kind rows)
$ pre-commit run --files mlflow/tracing/constant.py mlflow/tracing/export/genai_semconv/translator.py tests/tracing/export/genai_semconv/test_translator.py
ruff.....................................................................Passed
format...................................................................Passed
unresolved-import........................................................Passed
clint....................................................................Passed
typos....................................................................Passed
# every other hook Passed or Skipped; ruff 0.16.8 from the lint group, Python 3.12.3
# not run: the full test suite, and Semantic Kernel or pydantic-ai end to end (span names read from function_tracer.py:49 and autolog_v2.py:97)

Does this PR require documentation update?

  • No.

Does this PR require updating the MLflow Skills repository?

  • No.

Release Notes

Is this a user-facing change?

  • Yes. Give a description of this change to be included in the release notes for MLflow users.

GenAI semconv export now names tool and agent spans execute_tool {name} and invoke_agent {name} from the original span name, and sets gen_ai.tool.name and gen_ai.agent.name. Previously the names were the bare operation or {operation} {model}.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

  • area/tracing: MLflow Tracing features, tracing APIs, and LLM tracing functionality

How should the PR be classified in the release notes? Choose one:

  • rn/bug-fix - A user-facing bug fix

Is this PR a critical bugfix or security fix that should go into the next patch release?

@aaronteo-db @B-Step62 is invoke_agent Agent.run fine, or should autologgers set gen_ai.agent.name?

…_agent` spans

The GenAI semconv translator renamed TOOL and AGENT spans to the bare
operation and dropped the `mlflow.*` attributes, so the exported span no
longer said which tool or agent ran.

Set `gen_ai.tool.name` and `gen_ai.agent.name` from the span name, keeping
the value when an instrumentation such as Strands or Semantic Kernel already
set it, and name the spans `execute_tool {name}` and `invoke_agent {name}`.

Signed-off-by: Christopher Krah <[email protected]>
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor
Install mlflow from this PR

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/26394/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/26394/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/26394/merge

PR author's recent activity

In the last 14 days, @chrikrah opened 28 PRs across 14 repos (showing top 10):

Repository Open Closed Merged Total
i-am-bee/beeai-framework 0 0 5 5
punkpeye/fastmcp 1 0 3 4
docling-project/docling 2 0 1 3
github/advisory-database 2 0 0 2
traceloop/openllmetry 2 0 0 2
deepset-ai/haystack 1 0 1 2
simonw/llm 2 0 0 2
run-llama/llama_index 2 0 0 2
mlflow/mlflow 1 0 0 1
NVIDIA/garak 1 0 0 1

@github-actions github-actions Bot added area/tracing MLflow Tracing and its integrations rn/bug-fix Mention under Bug Fixes in Changelogs. team-review Trigger a team review request community Community/external contribution labels Oct 3, 2026
@github-actions github-actions Bot added the size/M label Oct 3, 2026
@aaronteo-db
aaronteo-db requested a balanced review from Copilot October 4, 2026 05:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Existing names set through MLflow’s attribute API retain JSON quotation marks in exported attributes and span names.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Updates GenAI semantic-convention exports to retain tool and agent names.

Changes:

  • Adds tool and agent semantic-convention keys.
  • Derives span names and attributes from existing names.
  • Adds translator coverage for tool and agent spans.
File Description
mlflow/​tracing/​constant.py Adds tool and agent name keys.
mlflow/​tracing/​export/​genai_semconv/​translator.py Preserves names during translation.
tests/​tracing/​export/​genai_semconv/​test_translator.py Tests the updated translation behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

MLflow span APIs JSON-serialize attribute values, so a `gen_ai.tool.name` or
`gen_ai.agent.name` set via `span.set_attribute` was exported with literal
quotes, e.g. `invoke_agent "travel_planner"`. Decode the value when it is a
JSON string and keep raw strings set directly on the OTel span as-is.

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Aaron Teo <[email protected]>
@aaronteo-db

Copy link
Copy Markdown
Collaborator

@chrikrah invoke_agent Agent.run is fine for this PR. An existing gen_ai.agent.name / gen_ai.tool.name already takes precedence, so autologgers can set the real name in as follow-up changes.

@aaronteo-db aaronteo-db left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! thanks for your contribution

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation matches the stated semantic-convention behavior and includes focused regression coverage.

Review effort: Balanced
Findings: None

Resolved since last review (1)

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Documentation preview for 66ec608 is available at:

More info
  • Ignore this comment if this PR does not change the documentation.
  • The preview is updated when a new commit is pushed to this PR.
  • This comment was created by this workflow run.
  • The documentation was built by this workflow run.

@aaronteo-db
aaronteo-db added this pull request to the merge queue Oct 4, 2026
Merged via the queue into mlflow:master with commit 9340ef7 Oct 4, 2026
67 of 69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/tracing MLflow Tracing and its integrations community Community/external contribution rn/bug-fix Mention under Bug Fixes in Changelogs. size/M team-review Trigger a team review request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GenAI semconv export drops the tool and agent names from execute_tool and invoke_agent spans

3 participants