Repository navigation
Description
Bug description
When a container is removed, the go.d docker collector also removes the per-container charts of any other container whose name begins with the removed container's name. For example, removing foo also removes foo-bar's charts. Those charts are never re-created while the Netdata plugin keeps running, so foo-bar silently drops out of docker.container_state, docker.container_health_status and docker.container_writable_layer_size. Its docker_container_down and docker_container_unhealthy alerts go to REMOVED.
Cause (v2.12.0, unchanged on master and in v2.12.1):
charts.go#L166-L174 matches chart IDs by prefix without a separator:
func (c *Collector) removeContainerCharts(name string) {
px := fmt.Sprintf("container_%s", name)
for _, chart := range *c.Charts() {
if strings.HasPrefix(chart.ID, px) {
chart.MarkRemove()
chart.MarkNotCreated()
}
}
}So container_foo matches container_foo-bar_state, container_foo-bar_health_status and container_foo-bar_writable_layer_size.
collect.go#L188-L193 deletes only foo from c.containers, and collect.go#L162-L165 adds charts only for names that are not already in c.containers. foo-bar stays in the map, so its charts are never added again.
Expected behavior
Removing container foo removes only foo's charts. foo-bar's charts and alerts stay in place while it keeps running.
Steps to reproduce
- With the go.d
dockercollector enabled, start two containers:
docker run -d --name foo-bar alpine sleep infinityanddocker run -d --name foo alpine sleep infinity - Wait for both containers' charts to appear (for example
docker.container_stateforfooand forfoo-bar). docker rm -f foo- After the next collection,
foo-bar's charts are gone (marked obsolete) even thoughfoo-baris still running. They do not come back. - Restart Netdata (or the go.d plugin):
foo-bar's charts come back.
Installation method
docker
System info
$ uname -srvmo
Linux 6.8.0-146-generic #146-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 3 16:12:30 UTC 2026 x86_64 GNU/Linux
$ grep -HvE "^#|URL" /etc/*release
/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=24.04
/etc/lsb-release:DISTRIB_CODENAME=noble
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 24.04.5 LTS"
/etc/os-release:PRETTY_NAME="Ubuntu 24.04.5 LTS"
/etc/os-release:VERSION_ID="24.04"
/etc/os-release:VERSION_CODENAME=noble
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
Docker Engine 29.8.2Netdata build info
Packaging:
Netdata Version ____________________________________________ : v2.12.0
Installation Type __________________________________________ : binpkg-deb
Package Architecture _______________________________________ : x86_64
Package Distro _____________________________________________ : ubuntu 24.04
Operating System:
Kernel _____________________________________________________ : Linux
Kernel Version _____________________________________________ : 6.8.0-146-generic
Operating System ___________________________________________ : Ubuntu
Operating System ID ________________________________________ : ubuntu
Operating System ID Like ___________________________________ : debian
Operating System Version ___________________________________ : 24.04.5 LTS (Noble Numbat)
Operating System Version ID ________________________________ : 24.04
Detection __________________________________________________ : /etc/os-release
(netdata -W buildinfo, Packaging and Operating System sections)Additional info
Suggested fix: add the separator:
px := fmt.Sprintf("container_%s_", name)This fixes the foo / foo-bar case. Container names may also contain _, though, so container_foo_ would still match a container named foo_state (container_foo_state_state). Matching the exact IDs built from the per-container chart templates avoids that, for example:
func (c *Collector) removeContainerCharts(name string) {
ids := make(map[string]bool)
for _, tmpl := range containerChartsTmpl {
ids[fmt.Sprintf(tmpl.ID, name)] = true
}
for _, chart := range *c.Charts() {
if ids[chart.ID] {
chart.MarkRemove()
chart.MarkNotCreated()
}
}
}(containerChartsTmpl is the template set that addContainerCharts uses, charts.go#L92-L96.)
Some other go.d collectors use the same unsuffixed HasPrefix(chart.ID, ...) removal on master, for example ntpd (peer_%s), nvme (device_%s), wireguard (device_%s, peer_%s), logstash ("pipeline_"+id), upsd, proxysql (mysql_user_, mysql_command_) and k8s_state (pod_%s, node_%s, deployment_%s, cronjob_%s). I have not checked whether each of those can actually collide.
Activity