Skip to content

fix: metrics reporting when LLM plugin is not enabled#12841

Closed
qiqizjl wants to merge 1 commit into
apache:masterfrom
qiqizjl:llm_prometheus
Closed

fix: metrics reporting when LLM plugin is not enabled#12841
qiqizjl wants to merge 1 commit into
apache:masterfrom
qiqizjl:llm_prometheus

Conversation

@qiqizjl

@qiqizjl qiqizjl commented Dec 25, 2025

Copy link
Copy Markdown

Summary

Fix incorrect metrics reporting when LLM plugin is not enabled.

Problem

When the LLM plugin is not enabled, the nginx variables for LLM metrics were defaulting to '0', which caused the metrics to be incorrectly reported as active even when no LLM functionality was being used.

Solution

Changed the default values from '0' to '' (empty string) for the following nginx variables:

  • llm_time_to_first_token
  • llm_prompt_tokens
  • llm_completion_tokens

Also updated the condition check in openai-base.lua to properly detect when LLM functionality is not active.

Changes

  • apisix/cli/ngx_tpl.lua: Changed default values from '0' to ''
  • apisix/plugins/ai-drivers/openai-base.lua: Updated condition check

🤖 Generated with Claude Code

@qiqizjl @claude
bugfix: fix metrics reporting when LLM plugin is not enabled
4980bc8 
Fix incorrect metrics reporting when LLM plugin is not enabled.
Change default values from '0' to '' for LLM-related nginx variables
(llm_time_to_first_token, llm_prompt_tokens, llm_completion_tokens)
to properly detect when LLM functionality is not active.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. bug Something isn't working labels Dec 25, 2025
@qiqizjl qiqizjl changed the title bugfix: fix metrics reporting when LLM plugin is not enabled fix: metrics reporting when LLM plugin is not enabled Dec 25, 2025
@Baoyuantop

Baoyuantop commented Dec 25, 2025

Copy link
Copy Markdown
Contributor

Hi @qiqizjl, we made this change in #12549. Could you explain the original requirements of this PR? Changing the default value from '0' to '' may affect users' existing JSON log format.

@qiqizjl

qiqizjl commented Dec 25, 2025 •
edited
Loading

Copy link
Copy Markdown
Author

Hi @qiqizjl, we made this change in #12549. Could you explain the original requirements of this PR? Changing the default value from '0' to '' may affect users' existing JSON log format.

We found in apisix/plugins/prometheus/exporter.lua#L376-L394 that metrics are reported when the value is not empty. When the LLM plugin is not enabled, the default value is 0, which results in invalid metrics.

@qiqizjl

qiqizjl commented Dec 25, 2025

Copy link
Copy Markdown
Author

Another possible solution is to check if request_type is ai_stream or ai_chat, and report accordingly. I tried changing the check to report if ctx.var.llm_time_to_first_token != 0, but this might affect abnormal requests. If possible, I can continue to refine this issue based on the discussion.

@flearc

flearc commented Dec 26, 2025

Copy link
Copy Markdown
Contributor

Another possible solution is to check if request_type is ai_stream or ai_chat, and report accordingly. I tried changing the check to report if ctx.var.llm_time_to_first_token != 0, but this might affect abnormal requests. If possible, I can continue to refine this issue based on the discussion.

    if llm_time_to_first_token ~= "0" then
        metrics.llm_latency:observe(tonumber(llm_time_to_first_token),
            gen_arr(route_id, service_id, consumer_name, balancer_ip,
            vars.request_type, vars.request_llm_model, vars.llm_model,
            unpack(extra_labels("llm_latency", ctx))))
    end
    if vars.llm_prompt_tokens ~= "0" then
        metrics.llm_prompt_tokens:inc(tonumber(vars.llm_prompt_tokens),
            gen_arr(route_id, service_id, consumer_name, balancer_ip,
            vars.request_type, vars.request_llm_model, vars.llm_model,
            unpack(extra_labels("llm_prompt_tokens", ctx))))
    end
    if vars.llm_completion_tokens ~= "0" then
        metrics.llm_completion_tokens:inc(tonumber(vars.llm_completion_tokens),
            gen_arr(route_id, service_id, consumer_name, balancer_ip,
            vars.request_type, vars.request_llm_model, vars.llm_model,
            unpack(extra_labels("llm_completion_tokens", ctx))))
    end

Adjust the if statement might help

LiteSun added a commit to LiteSun/incubator-apisix that referenced this pull request Dec 29, 2025
@LiteSun
fix: mcp-bridge flaky test (apache#12841)
8ef2c4c
@Baoyuantop

Baoyuantop commented Jan 26, 2026

Copy link
Copy Markdown
Contributor

Hi @qiqizjl, the proposed changes in this PR are incomplete and will cause a regression. We recommend creating an issue first to discuss a solution for this problem.

@Baoyuantop Baoyuantop added the wait for update wait for the author's response in this issue/PR label Jan 26, 2026
@qiqizjl qiqizjl mentioned this pull request Jan 29, 2026
Closed
@Baoyuantop

Baoyuantop commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

The current PR implementation is incorrect, so I'm closing it for now. You can resubmit the PR according to the discussion in the issue.

@Baoyuantop Baoyuantop closed this Mar 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

bug Something isn't working size:XS This PR changes 0-9 lines, ignoring generated files. wait for update wait for the author's response in this issue/PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qiqizjl @Baoyuantop @flearc