feat: decoupled prometheus exporter's calculation and output by SkyeYoung · Pull Request #12383 · apache/apisix

SkyeYoung · 2025-06-26T09:49:41Z

Description

This PR decouples the calculation and output processes of the Prometheus exporter. The "calculation" is performed in the privileged agent process at intervals defined by the refresh_interval(default is 15s) and written to a shared dict, while the "output" (i.e., the /apisix/prometheus/metrics API) is moved to the worker process, which only reads and returns the cached data in the shared dict.

The above are just the core changes. In fact, I encountered many other problems, which have been commented or annotated in the corresponding positions, and will not be repeated here.

For the testing part, since the Prometheus exporter currently refreshes data every 15 seconds, I used a smaller interval in the relevant tests to pass the original tests.

Which issue(s) this PR fixes:

Fixes #

Stress Testing

How to

install wrk2, git clone apisix
deploy etcd(./test.sh init-etcd), nginx(as upstream, ./test.sh start-nginx)
make run
create 10k routes(./test.sh create)
enable promethues(./test.sh enable-prometheus)
run benchmark(./test.sh benchmark)

Results

Key Performance Indicators

Scenario	CPU Usage	Memory Usage	P50 Latency	P90 Latency	P99 Latency
Prometheus Disabled	33.0%	144.3MB	3.58ms	5.99ms	8.99ms
Prometheus Enabled	38.3% (+5.2%)	100.2MB (-44.1MB)	4.22ms (+0.64ms)	7.07ms (+1.08ms)	10.66ms (+1.67ms)
1 Metrics Reqs	38.6% (+5.6%)	101.3MB (-42.9MB)	4.25ms (+0.68ms)	7.87ms (+1.87ms)	12.17ms (+3.18ms)
3 Metrics Reqs	39.1% (+6.1%)	100.1MB (-44.2MB)	3.88ms (+0.30ms)	7.17ms (+1.18ms)	11.92ms (+2.93ms)

Performance Impact Summary

Scenario	CPU Impact	Memory Impact	P50 Latency Impact	P90 Latency Impact	P99 Latency Impact
Prometheus Disabled (Baseline)	0.0%	0.0MB	0.00ms	0.00ms	0.00ms
Prometheus Enabled	+5.2%	-44.1MB	+0.64ms	+1.08ms	+1.67ms
1 Metrics Reqs	+5.6%	-42.9MB	+0.68ms	+1.87ms	+3.18ms
3 Metrics Reqs	+6.1%	-44.2MB	+0.30ms	+1.18ms	+2.93ms

Checklist

I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

…heus-exporter-concurrency

SkyeYoung · 2025-06-27T09:31:33Z

    -- config server status
-    local vars = ngx.var or {}
-    local hostname = vars.hostname or ""
+    local hostname = core.utils.gethostname() or ""


Because API disabled in the context of ngx.timer, context: ngx.timer.

…heus-exporter-concurrency

This reverts commit 521f4c9.

…heus-exporter-concurrency

membphis · 2025-07-22T11:45:09Z

+function _M.export_metrics()
    if not prometheus then
-        core.response.exit(200, "{}")
+       return core.response.exit(200, "{}")


the current way is good to me

membphis · 2025-07-22T11:47:07Z

  meta:
    lua_shared_dict:              # Nginx Lua shared memory zone. Size units are m or k.
      prometheus-metrics: 15m
+      prometheus-cache: 10m


pls add some comments, tell users when they need to modify it

membphis · 2025-07-22T11:50:17Z

+        core.log.error("Failed to collect metrics: ", res)
+        return
+    end
+    shdict_prometheus_cache:set(CACHED_METRICS_KEY, res)


need to capture the return value, it maybe fail

if err, tell user what is the reason, and tell the user to change the default size if the shdict is small

membphis · 2025-07-23T01:43:54Z

+    local _, err, forcible = shdict_prometheus_cache:set(CACHED_METRICS_KEY, res)
+
+    if err then
+        core.log.error("Failed to save metrics to shdict: ", err)


need more information, eg: the name of shdict and the size of value

if the value is to large, will fail when we call set method

membphis · 2025-07-23T01:45:15Z

+
    core.response.set_header("content_type", "text/plain")
-    return 200, core.table.concat(prometheus:metric_data())
+    local cached_metrics_text = shdict_prometheus_cache:get(CACHED_METRICS_KEY)


just confirm, no "err" is returned?

I checked the part code and found that the second return value, marked as "flags" in its documentation, actually returns an error when an error occurs.

Let me fix this part of the code.

membphis

LGTM

membphis · 2025-07-23T05:49:51Z

+
+    -- Clear the cached data after cache_exptime to prevent stale data in case of an error.
+    local _, err, forcible = shdict_prometheus_cache:set(CACHED_METRICS_KEY, res, cache_exptime)
+


remove this blank line

bzp2010 · 2025-07-23T05:59:03Z

-        require("apisix.plugins.prometheus.exporter").http_init(prometheus_enabled_in_stream)
-    elseif not is_http and core.table.array_find(stream_plugin_names, "prometheus") then
-        require("apisix.plugins.prometheus.exporter").stream_init()
+    if is_http and (enabled_in_http or enabled_in_stream) then


It might be better to add a link to this PR comment here.

https://github.com/apache/apisix/pull/12383/files#r2221993953

yurkovoznyak · 2025-10-23T08:27:47Z


+
+function _M.init_worker()
    -- someone's plugin needs to be initialized after prometheus


Hi @SkyeYoung, I'm using Prometheus metrics in my custom plugin, and with this change, there is no guarantee that the prometheus plugin is loaded before mine. What is the correct approach for defining custom metrics now?

Sorry for commenting on a closed PR

Hi @yurkovoznyak, could you please file an issue describing your problem in detail?

SkyeYoung added 11 commits June 26, 2025 09:43

feat: decoupled prometheus exporter's calculation and output

d32ec9f

fix: require process

35a38ad

fix

55619cf

fix: part of exportor logic

467363a

rewrite: use FFI to get nginx_status

e34a924

chore

d91509b

Merge remote-tracking branch 'upstream/master' into young/perf/promet…

d2b3bd4

…heus-exporter-concurrency

chore: simplify logic

b22e74f

chore: rm useless logic

27eec34

fix: lint

a42006f

fix: lint

740c39b

SkyeYoung commented Jun 27, 2025

View reviewed changes

Comment thread apisix/plugins/prometheus/exporter.lua

SkyeYoung commented Jun 27, 2025

View reviewed changes

SkyeYoung added 17 commits July 7, 2025 03:45

fix: try init cached metrics to pass tests

e4800f5

try fix logic

97e00e7

fix: init only one time

171406d

fix: lint

57959e6

chore: try fix tests

a22b9a7

test: fix prometheus related cases

27ac5c9

Merge remote-tracking branch 'upstream/master' into young/perf/promet…

1fb6217

…heus-exporter-concurrency

fix: logic

521f4c9

Revert "fix: logic"

226a14c

This reverts commit 521f4c9.

fix: init etcd logic

f35b711

Merge remote-tracking branch 'upstream/master' into young/perf/promet…

df84c76

…heus-exporter-concurrency

fix: logic

b998b64

fix

0d61202

fix: test case

3601800

fix: test cases

9523fed

test: improve stability

6ae0448

chore(ngx_tpl): rm prom privileged_agent conf

005cc83

membphis reviewed Jul 22, 2025

View reviewed changes

SkyeYoung added 5 commits July 22, 2025 13:12

fix: lint

72c6f22

fix: add usefull comments and logs

9d5e45c

chore: add comments

61f4812

fix: standardized error return

cf66013

fix: log forciable

6de5129

membphis self-requested a review July 23, 2025 01:34

SkyeYoung requested a review from bzp2010 July 23, 2025 01:44

membphis reviewed Jul 23, 2025

View reviewed changes

SkyeYoung added 2 commits July 23, 2025 02:24

fix: return and log shared.DICT.get err

0b6f1a9

fix: add more explicit error messages when ngx.shared.DICT.set fails

432af12

membphis requested changes Jul 23, 2025

View reviewed changes

Comment thread apisix/plugins/prometheus/exporter.lua Outdated

Comment thread apisix/plugins/prometheus/exporter.lua Outdated

SkyeYoung added 2 commits July 23, 2025 03:44

fix: use ngx.timer.every to reduce potential problems

4d77fc2

fix: add cache exptime

cd96ecd

dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jul 23, 2025

SkyeYoung requested a review from membphis July 23, 2025 04:43

membphis previously approved these changes Jul 23, 2025

View reviewed changes

SkyeYoung commented Jul 23, 2025

View reviewed changes

Comment thread apisix/plugins/prometheus/exporter.lua Outdated

chore: rm blank line

f28295d

SkyeYoung dismissed membphis’s stale review via f28295d July 23, 2025 06:06

SkyeYoung requested a review from membphis July 23, 2025 06:07

bzp2010 approved these changes Jul 23, 2025

View reviewed changes

SkyeYoung mentioned this pull request Jul 24, 2025

bug: the stream plugins should not be loaded in the http subsystem #12458

Closed

membphis approved these changes Jul 24, 2025

View reviewed changes

nic-6443 approved these changes Jul 25, 2025

View reviewed changes

SkyeYoung merged commit 30ae5df into apache:master Jul 25, 2025
39 of 40 checks passed

SkyeYoung deleted the young/perf/prometheus-exporter-concurrency branch July 25, 2025 01:30

yurkovoznyak reviewed Oct 23, 2025

View reviewed changes

Baoyuantop mentioned this pull request Mar 12, 2026

bug: lua_shared_dict prometheus-metrics overflow #11948

Closed


		-- Clear the cached data after cache_exptime to prevent stale data in case of an error.
		local _, err, forcible = shdict_prometheus_cache:set(CACHED_METRICS_KEY, res, cache_exptime)



		function _M.init_worker()
		-- someone's plugin needs to be initialized after prometheus

Uh oh!

Conversation

SkyeYoung commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Which issue(s) this PR fixes:

Stress Testing

How to

Results

Key Performance Indicators

Performance Impact Summary

Checklist

Uh oh!

Uh oh!

SkyeYoung Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SkyeYoung Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

membphis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

SkyeYoung commented Jun 26, 2025 •

edited

Loading

SkyeYoung Jun 27, 2025 •

edited

Loading

SkyeYoung Jul 23, 2025 •

edited

Loading