Python: [Bug]: Token count used in Compaction much differ from usage from LLM #7022

Description

TokenBudgetComposedStrategy estimates the token count of the LLM input, but its estimate differs significantly from the actual token usage reported by the LLM API.

Currently, TokenBudgetComposedStrategy counts tokens using:

tokenizer.count_tokens(json.dumps(message, ensure_ascii=True, ...))

However, this is not an accurate way to estimate the number of input tokens.

The reason is that tokenizer.count_tokens() is tokenizing the serialized JSON string rather than the actual message text that the API sends to the model. The serialized JSON contains escaped Unicode sequences such as "\u3053\u3093\u306b\u3061\u306f", whereas the model tokenizes the original text ("こんにちは").

As a result, when the input contains Japanese or other non-ASCII characters, ensure_ascii=True can significantly overestimate the token count. For example, Japanese text may be estimated as requiring roughly three times as many tokens as the actual API usage.

In _compaction.py:

def _serialize_message(message: Message) -> str:
    serialized_contents = [_serialize_content(content) for content in message.contents]
    payload = {
        "role": message.role,
        "message_id": message.message_id,
        "contents": serialized_contents,
    }
    return json.dumps(payload, ensure_ascii=True, sort_keys=True, default=str)

Code Sample

Error Messages / Stack Traces

Package Versions

agent framework 1.9.0

Python Version

python 3.14

Additional Context

No response