Skip to content

IcingaDB: Don't publish useless data to Redis#10359

Merged
yhabteab merged 1 commit intomasterfrom
do-not-publish-useless-stats
Mar 7, 2025
Merged

IcingaDB: Don't publish useless data to Redis#10359
yhabteab merged 1 commit intomasterfrom
do-not-publish-useless-stats

Conversation

@yhabteab
Copy link
Copy Markdown
Member

@yhabteab yhabteab commented Mar 4, 2025

The Icinga DB daemon processes the data from the IcingaApplication type only and Icinga DB Web also uses only those stats. However, before this commit, Icinga DB published all kinds of useless stats to Redis each second, like the number of (un)reachable hosts, services, and so on, which is waste of CPU and some other resources. This commit reduces the published data drastically to only those simple stats coming from the IcingaApplication type.

Before

docker exec -i redis redis-cli -p 6380 --raw XREAD streams icinga:stats 0 
icinga:stats
1741094716481-0
ApiListener
{"perfdata":[{"counter":false,"crit":null,"label":"api_num_conn_endpoints","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null},{"counter":false,"crit":null,"label":"api_num_endpoints","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null},{"counter":false,"crit":null,"label":"api_num_http_clients","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null},{"counter":false,"crit":null,"label":"api_num_json_rpc_anonymous_clients","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null},{"counter":false,"crit":null,"label":"api_num_json_rpc_relay_queue_item_rate","max":null,"min":null,"type":"PerfdataValue","unit":"","value":13.183333333333334,"warn":null},{"counter":false,"crit":null,"label":"api_num_json_rpc_relay_queue_items","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null},{"counter":false,"crit":null,"label":"api_num_json_rpc_sync_queue_item_rate","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null},{"counter":false,"crit":null,"label":"api_num_json_rpc_sync_queue_items","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null},{"counter":false,"crit":null,"label":"api_num_json_rpc_work_queue_item_rate","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null},{"counter":false,"crit":null,"label":"api_num_not_conn_endpoints","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null}],"status":{"api":{"conn_endpoints":[],"http":{"clients":0},"identity":"mbp-yhabteab","json_rpc":{"anonymous_clients":0,"relay_queue_item_rate":13.183333333333334,"relay_queue_items":0,"sync_queue_item_rate":0,"sync_queue_items":0,"work_queue_item_rate":0},"not_conn_endpoints":[],"num_conn_endpoints":0,"num_endpoints":0,"num_not_conn_endpoints":0,"zones":{"fooo":{"client_log_lag":0,"connected":true,"endpoints":["mbp-yhabteab"],"parent_zone":""}}}}}
CIB
{"perfdata":[],"status":{"active_host_checks":0.4166666666666667,"active_host_checks_15min":565,"active_host_checks_1min":25,"active_host_checks_5min":266,"active_service_checks":2.45,"active_service_checks_15min":13558,"active_service_checks_1min":147,"active_service_checks_5min":8352,"avg_execution_time":0.001536543679806127,"avg_latency":0.0033716023314160582,"current_concurrent_checks":0,"current_pending_callbacks":0,"max_execution_time":0.09525394439697266,"max_latency":0.10480809211730957,"min_execution_time":0.0005261898040771484,"min_latency":0.0004940032958984375,"num_hosts_acknowledged":0,"num_hosts_down":43,"num_hosts_flapping":0,"num_hosts_handled":0,"num_hosts_in_downtime":0,"num_hosts_pending":0,"num_hosts_problem":232,"num_hosts_unreachable":384,"num_hosts_up":103,"num_services_acknowledged":0,"num_services_critical":3017,"num_services_flapping":0,"num_services_handled":5485,"num_services_in_downtime":0,"num_services_ok":3428,"num_services_pending":0,"num_services_problem":8980,"num_services_unknown":2844,"num_services_unreachable":3401,"num_services_warning":3119,"passive_host_checks":0,"passive_host_checks_15min":0,"passive_host_checks_1min":0,"passive_host_checks_5min":0,"passive_service_checks":0,"passive_service_checks_15min":0,"passive_service_checks_1min":0,"passive_service_checks_5min":0,"remote_check_queue":0,"uptime":10598.02723813057}}
CheckerComponent
{"perfdata":[{"counter":false,"crit":null,"label":"checkercomponent_checker_idle","max":null,"min":null,"type":"PerfdataValue","unit":"","value":12938,"warn":null},{"counter":false,"crit":null,"label":"checkercomponent_checker_pending","max":null,"min":null,"type":"PerfdataValue","unit":"","value":0,"warn":null}],"status":{"checkercomponent":{"checker":{"idle":12938,"pending":0}}}}
ElasticsearchWriter
{"perfdata":[],"status":{"elasticsearchwriter":{}}}
FileLogger
{"perfdata":[],"status":{"filelogger":{}}}
GelfWriter
{"perfdata":[],"status":{"gelfwriter":{}}}
GraphiteWriter
{"perfdata":[],"status":{"graphitewriter":{}}}
IcingaApplication
{"perfdata":[],"status":{"icingaapplication":{"app":{"enable_event_handlers":true,"enable_flapping":true,"enable_host_checks":true,"enable_notifications":true,"enable_perfdata":true,"enable_service_checks":true,"endpoint_id":"b344bec6e673840b76f486318c25c75fde03c8d3","environment":"","node_name":"mbp-yhabteab","pid":94203,"program_start":1741084118381,"version":"v2.14.0-481-g5a7b3ac16"}}}}
Influxdb2Writer
{"perfdata":[],"status":{"influxdb2writer":{}}}
InfluxdbWriter
{"perfdata":[],"status":{"influxdbwriter":{}}}
NotificationComponent
{"perfdata":[],"status":{"notificationcomponent":{}}}
OpenTsdbWriter
{"perfdata":[],"status":{"opentsdbwriter":{}}}
PerfdataWriter
{"perfdata":[],"status":{"perfdatawriter":{}}}
SyslogLogger
{"perfdata":[],"status":{"sysloglogger":{}}}
config_dump_in_progress
false
icingadb_environment
"07d5e33e7da4205e9f9a9d7513fed06ddf5e91a8"
timestamp
1741094716414

After

docker exec -i redis redis-cli -p 6380 --raw XREAD streams icinga:stats 0 
icinga:stats
1741105038990-0
IcingaApplication
{"perfdata":[],"status":{"icingaapplication":{"app":{"enable_event_handlers":true,"enable_flapping":true,"enable_host_checks":true,"enable_notifications":true,"enable_perfdata":true,"enable_service_checks":true,"endpoint_id":"b344bec6e673840b76f486318c25c75fde03c8d3","environment":"","node_name":"mbp-yhabteab","pid":33394,"program_start":1741104995661,"version":"v2.14.0-481-g5a7b3ac16"}}}}
config_dump_in_progress
false
icingadb_environment
"07d5e33e7da4205e9f9a9d7513fed06ddf5e91a8"
timestamp
1741105038991

@yhabteab
IcingaDB: Don't publish useless data to Redis
6ca0611 
The Icinga DB daemon processes the data from the `IcingaApplication`
type only and Icinga DB Web also uses only those stats. However, before
this commit, Icinga DB published all kinds of useless stats to Redis
each second, like the number of (un)reachable hosts, services, and so
on, which is waste of CPU and some other resources. This commit reduces
the published data drastically to only those simple stats coming from
the `IcingaApplication` type.
@cla-bot cla-bot bot added the cla/signed label Mar 4, 2025
@yhabteab yhabteab added the area/icingadb New backend label Mar 4, 2025
@yhabteab yhabteab added this to the 2.15.0 milestone Mar 4, 2025
@yhabteab yhabteab requested a review from julianbrost March 4, 2025 17:11
@julianbrost julianbrost added the consider backporting Should be considered for inclusion in a bugfix release label Mar 5, 2025
julianbrost
julianbrost approved these changes Mar 7, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@julianbrost julianbrost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going through the users of the icinga:stats stream:

The Icinga DB Go daemon uses the following:

The tests for Icinga DB care about icingadb_environment (kept by this PR):
https://github.com/Icinga/icingadb/blob/e72762a29ea93078e21bcc2c109484fbfc24d2ac/tests/internal/utils/redis.go#L18-L27

Icinga DB Web seems to only care about the timestamp value (kept by this PR):
https://github.com/Icinga/icingadb-web/blob/4806939f3016b81cea53b57f645fa70f530daa78/library/Icingadb/Common/IcingaRedis.php#L200-L218

So removing the rest should be fine, contents of the icingadb_instance table and what's shown on the health page in Icinga DB Web are all still fine with this change.

@yhabteab yhabteab merged commit 206d7cd into master Mar 7, 2025
23 checks passed
@yhabteab yhabteab deleted the do-not-publish-useless-stats branch March 7, 2025 11:51
@yhabteab yhabteab removed the consider backporting Should be considered for inclusion in a bugfix release label Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@julianbrost julianbrost julianbrost approved these changes

Assignees

No one assigned

Labels

area/icingadb New backend cla/signed

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

3 participants

@yhabteab @julianbrost @Al2Klimov