Skip to content

fix: null is not an explicit reason — unavailable metrics must state why #238

Closed

Description

@terisuke

Problem

Issue #215 acceptance criterion: "Captures token/sec where local runtime exposes it, or records an explicit unavailable reason."

PR #237 outputs "tokens_per_sec_by_model": {} and "total_completion_tokens": null when the backend does not return usage. This is not an explicit reason — it is an implicit absence. A reader of the smoke output cannot distinguish between:

  • Backend returned usage but values were 0
  • Backend did not return usage at all
  • Instrumentation hook was broken and never fired
  • This metric was not implemented

Root Cause

  1. "null means unavailable" is not explicit: The criterion asks for an explicit reason, not implicit absence. An explicit reason would be: "token_throughput_status": "backend_omits_usage" or "token_throughput_unavailable_reason": "usage_field_absent".

  2. Did not read the criterion literally: "or records an explicit unavailable reason" means the output must contain a field whose value explains why the metric is unavailable. null does not explain.

Prevention

For the agent

  1. When a criterion says "explicit X", do not substitute implicit absence. An explicit field must exist in the output with a value that explains the situation.

  2. Pattern for unavailable metrics:

{
  "tokens_per_sec_by_model": {"qwen2.5:7b": null},
  "tokens_per_sec_unavailable_reason": "backend_response_omits_usage_field"
}
  1. Apply this pattern to every metric that can be unavailable: RAG latency ("not_invoked_no_embedding_client"), embedding latency ("not_invoked_no_embedding_client"), psutil RSS ("psutil_not_installed").

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions