Repository navigation
Python: ApplicationInsights agent span has incorrect token numbers #1320
Description
I am using:
- agent-framework v 1.0.0b251007 (But the same was true for v 1.0.0b251001)
- AzureOpenAIResponsesClient
setup_observability(enable_sensitive_data=True, applicationinsights_connection_string="omitted")
What is happening:
In the latest version, each independent call to the model is being tracked as expected, with their token usage correctly attached. The agent's span then also records token usage, but is set to the values of the final llm call. This is misleading
Expected behavior is either:
- The agent span does not report token usage because they are reported on the model call
- The agent span reports the sum of tokens used in it's children spans
Screenshot showing application insights with the agent's span. The token usage matches the second chat o4-mini span's token usage:
Metadata
Metadata
Labels
Type
Projects
- Status
Activity
jomalsan commentedon Oct 14, 2025
After thinking about this more, I think the agent span not reporting token usage is the correct call. This way when using external observability like Langfuse we don't double count the tokens
eavanvalkenburg commentedon Dec 10, 2025
Jon Malsan (@jomalsan) I had a good look at this, and I made a update to the way tokens are added, where for depending on the type of agent it will or won't add them to the invoke_agent span, and not for ChatAgents, so that should fix it. I spun up LangFuse locally to validate, and did notice that I had to go in and remove the tokenizer setup for the model that I'm using, because the agent span does have the model in it, so Langfuse was still doing some kind of calculation to add a token count and cost to that span, rather then just aggregating from the underlying chat spans, so be carefull about that. Once I have some other fixes in the observability space done, it will go in and we'll release. Thanks!