Skip to content

fix: report measured OpenAI token usage - #95

Merged
mldangelo merged 3 commits into
promptfoo:mainfrom
arunimshukla:fix/real-openai-token-usage
Oct 8, 2026
Merged

mldangelo merged 3 commits into
promptfoo:mainfrom
arunimshukla:fix/real-openai-token-usage

Conversation

@arunimshukla

Copy link
Copy Markdown
Contributor

Summary

Use the token usage reported by OpenAI across the ReAct loop instead of estimating usage from the number of messages.

The provider currently reports:

total: result.messages.length * 100
prompt: Math.floor(result.messages.length * 60)
completion: Math.floor(result.messages.length * 40)

and derives cost from a hard-coded per-token rate. Two runs with the same message count therefore report identical usage even when the API reports different token counts, and the cost cannot be correct for arbitrary configured models or OpenAI-compatible base URLs.

Changes

  • aggregate completion.usage across every successful ReAct completion
  • include the number of OpenAI requests in the aggregated usage
  • return measured usage through the Promptfoo provider response
  • stop emitting synthetic cost when the provider has no authoritative price source
  • add regressions for a two-request tool loop and provider response passthrough

If any successful completion omits usage, the aggregate is omitted rather than presenting a partial total as complete.

@arunimshukla
arunimshukla force-pushed the fix/real-openai-token-usage branch from 66ed106 to 3e1430c Compare September 27, 2026 16:59

@mldangelo mldangelo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed per-iteration token aggregation, omission of incomplete usage, and removal of fabricated cost. Added real-SDK loopback integration coverage for measured totals, zero usage, and missing usage. All 31 local tests and lint pass, as does current Node 20/22/24 CI.

@mldangelo
mldangelo merged commit 6478386 into promptfoo:main Oct 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants