Ever-increasing memory usage (memory leak?)

I've been observing increasing memory usages since upgrading to the out-of-tree barman-cloud plugin (on v1.26.0);
I've been suspicions before, but after more investigation I wanted to raise awareness more formally:

only the primary instance seems to be suffering from the memory leak (which is doing the WAL shipping)
from what I'm seeing, it seems to be the plugin-barman-cloud container that's increasing its memory slowly, but steadily; a good metric I've found is container_memory_working_set_bytes{container="plugin-barman-cloud"}
possible workarounds are given by failovering, restarting the cluster, albeit that might be overkill

Before and after migrating to the out-of-tree plugin:

3-node HA cluster, which was restarted and is starting to accumulate memory again; running out-of-tree for the whole duration of the graph:

Different 3-node HA cluster which was migrated to out-of-tree on 03.06.25 00:00, also starting to increase memory consumption:

`container_memory_working_set_bytes{container="plugin-barman-cloud"}` is the culprit

Am I really the only one facing this issue?

Update:
I've updated from 0.4.1 to 0.5.0 on 16.06 on a (different to the above) test system: the overall situation does not look any better:

No. You are not the only one. I see the same issue. I need to restart the cluster from time to time to not eat up all the memory.

Noticing this as well. One thing that also may be of benefit, if it's not in fact a memory leak, but just general memory usage, is the usage of the new GOMEMLIMIT Go feature. Not familiar with how the new plugin system works, or if it's easy to set some of these types of env vars, but I generally use this for Go services/daemons to keep memory pinned below a certain amount (it doesn't do as much GC if not close to GOMEMLIMIT, and as it gets closer to GOMEMLIMIT, GC becomes more and more aggressive).

I use CNPG on clusters with very low resources (PG running with <200MB memory), and the backup plugin consuming more than PG does is definitely an issue for me. Ideally, I'd like to see the average usage be <50MB for this plugin sidecar (higher during backups is fine though).

I have also same issue. Thanks for posting this one.

Yes. same for me.

Any update on this? Seems like a pretty critical issue, causing cluster pods to constantly OOM (multiple times a day for me). Is there anything users can do to help debug/troubleshoot/provide addl. info?

I have a similar issue.
It's not about ever-increasing memory usage, but a high memory usage in general, when a lot of WAL files are backed up.

CloudNativePG 1.27.0
Plugin-Barman-Cloud 0.6.0

    wal:
      maxParallel: 8

I'm seeing the same thing, new deployment of about three days old. I was surprised to see the high memory usage, especially when compared to the same setup on another cluster with the in-tree plugin.

I'm late, but tl;dr you can prevent runaway memory usage by capping the memory that plugin-barman-cloud uses by setting memory limits on the ObjectStore custom resource.

This doesn't automatically apply the memory limits to running pods associated with the CNPG cluster, so you'll need to restart the associated pods afterwards. E.g. ensure .spec.instanceSidecarConfiguration.resources has the associated memory limits you want. In my limited testing, 300Mi is near the lower bound for an empty CNPG database, likely more is needed in practice.

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  ...
spec:
  ...
  instanceSidecarConfiguration:
    resources:
      limits:
        memory: 300Mi

I did end up taking two go pprofs when the difference in memory reported at the pod level was >200Mi, however go tool pprof showed that the in use memory by the binary produced by this repo (plugin-barman-cloud's manager) wasn't consuming any more memory at all:

❯ go tool pprof -top -base /path/to/backup-cache2.002.pb.gz /path/to/backup-cache2.003.pb.gz
File: manager
Build ID: 201b2fe15b1c8d540d3b0b9397cd1d7f26c82ab3
Type: inuse_space
Time: 2025-08-20 20:58:16 PDT
Showing nodes accounting for -512.05kB, 100% of 512.05kB total
      flat  flat%   sum%        cum   cum%
 -512.05kB   100%   100%  -512.05kB   100%  runtime.acquireSudog
         0     0%   100%  -512.05kB   100%  runtime.chanrecv
         0     0%   100%  -512.05kB   100%  runtime.chanrecv1
         0     0%   100%  -512.05kB   100%  runtime.unique_runtime_registerUniqueMapCleanup.func2

pprof dumps attached here if anyone's curious to poke around
backup-cache2.002.pb.gz
backup-cache2.003.pb.gz

# The container plugin-barman-cloud seemingly has 243Mi allocated to it
❯ kubectl top po -n mjlshen --containers
POD        NAME                  CPU(cores)   MEMORY(bytes)
mjlshen-1   plugin-barman-cloud   1m           243Mi
mjlshen-1   postgres              3m           46Mi

❯ kubectl debug -n mjlshen mjlshen-1 -it --image=ubuntu:latest --target=plugin-barman-cloud

# From within the container, the main `manager` process is only taking ~52Mi though
$ ps aux --sort=-%mem
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
26             1  0.0  0.3 1281080 53600 ?       Ssl  Aug20   0:01 /manager instance
26          1133  0.0  0.0   4296  3620 pts/0    Ss   04:26   0:00 /bin/bash
26          1152  0.0  0.0   7628  3584 pts/0    R+   04:27   0:00 ps aux --sort=-%mem
26          1034  0.0  0.0   4372  2396 pts/0    Ss+  04:09   0:00 /bin/sh -l

I was stumped here for a while, but later learned about slab_reclaimable memory. It makes sense that plugin-barman-cloud would utilize slab memory during the backup process and this is counted againsted a container's memory usage (kubernetes/kubernetes#43916). Here are some manual steps just to demonstrate that this memory growth consists of slab_reclaimable memory, though I don't recommend running these command regularly to remediate the situation by "freeing up" ~230Mi:

# Get onto the "node" (for me, a node is a docker container since this was a kind cluster)
$ docker exec -it kind-control-plane bash

# Look at memory stats to see slab_reclaimable at ~228Mi (number in bytes, divide by 1024^2)
$ grep slab_reclaimable /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod1e887441_d89a_4de7_9825_85c9dd27fb07.slice/cri-containerd-6792222719d8560fd3e2730db9aa45df67fc7c08bc79298820adf9a004f5c635.scope/memory.stat
slab_reclaimable 239973216

# Manually drop the reclaimable slab cache
# Writing 2 tells the kernel to specifically drop reclaimable slab objects. Of note, writing 1
# drops the active file cache, writing 3 drops both the active file and reclaimable slab cache
$ sync; echo 2 > /proc/sys/vm/drop_caches

# Memory has been freed up!
$ grep slab_reclaimable /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod1e887441_d89a_4de7_9825_85c9dd27fb07.slice/cri-containerd-6792222719d8560fd3e2730db9aa45df67fc7c08bc79298820adf9a004f5c635.scope/memory.stat
slab_reclaimable 61216

# Check back in from Kubernetes' perspective
$ exit
exit
$ kubectl top po -n mjlshen --containers
POD        NAME                  CPU(cores)   MEMORY(bytes)
mjlshen-1   plugin-barman-cloud   1m           14Mi
mjlshen-1   postgres              3m           46Mi

As plugin-barman-cloud approaches a configured memory limit, the slab_reclaimable memory will typically be freed first without the entire container getting necessarily OOMKilled, but I'm doing some further testing in that area now. It's not necessarily a bad thing that plugin-barman-cloud keeps using more and more slab_reclaimable memory as long as it's available, but in terms of overall Kubernetes cluster health, I did not think it was a good idea to let it keep growing until the node itself came under memory pressure, so a memory limit on the plugin-barman-cloud sidecar seemed to be the right middle ground for me.

What I'm still not sure about and may need someone more familiar with the in-tree --> plugin split for backups is why the in-tree implementation doesn't run into this runaway slab_reclaimable memory growth at all

@mjlshen

@mjlshen I can confirm, that setting the memory limit does work

I set it to 500Mi because with 300Mi the pods did not start any more, even without doing a backup --> OOMKilled

Same issue here, but want to add that memory leak is not the only issue - CPU usage increased dramatically. It's test cluster without real load, but it eats 1 core consistently, while on old barman same test cluster eats just 0.175 of the core (with btw includes postgres itself**).

Designated primary barman plugin resource usage

update: barman plugin version 0.9.0
still a huge memory foodprint
what I noticed, the memory drop's with a checkpoint;

https://cloud-native.slack.com/archives/C08MW1HKF40/p1755605394502759

pr-659 seems to fix the memory handling dramatically
no sidecar resources set
maxParallel: 8

climbing over time, I will continue to run tests

after upgrading to 1.28.0-rc1 I did a 2,5h run (220GB loaded)

one thing I noticed in the 1.28.0-rc1 is, that for about 0,5h the backups where keeping up with the generated WAL files

on 1.27.1 the "ready" WAL files started piling up right from the beginning

fix: prevent memory leak by periodically cleaning up expired cache en…

We also experience this issue. Any ETA for the fix?

@mnencia

@mnencia or @armru is there any estimated date for a new plugin release?

Navigation Menu

Repository navigation

Ever-increasing memory usage (memory leak?) #385

Before and after migrating to the out-of-tree plugin:

3-node HA cluster, which was restarted and is starting to accumulate memory again; running out-of-tree for the whole duration of the graph:

Different 3-node HA cluster which was migrated to out-of-tree on 03.06.25 00:00, also starting to increase memory consumption:

`container_memory_working_set_bytes{container="plugin-barman-cloud"}` is the culprit

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Ever-increasing memory usage (memory leak?) #385

Description

Before and after migrating to the out-of-tree plugin:

3-node HA cluster, which was restarted and is starting to accumulate memory again; running out-of-tree for the whole duration of the graph:

Different 3-node HA cluster which was migrated to out-of-tree on 03.06.25 00:00, also starting to increase memory consumption:

container_memory_working_set_bytes{container="plugin-barman-cloud"} is the culprit

Activity

hwo-wd commented on Jun 16, 2025

mkrueger92 commented on Jun 22, 2025

lrstanley commented on Jun 23, 2025

anishbista60 commented on Jul 4, 2025

anishbista60 commented on Jul 4, 2025

priyashpatil commented on Jul 5, 2025

lrstanley commented on Jul 27, 2025

1 remaining item

smiyc commented on Aug 21, 2025

LordGaav commented on Aug 25, 2025

mjlshen commented on Aug 27, 2025

mjlshen commented on Aug 27, 2025

smiyc commented on Aug 28, 2025

dragoangel commented on Oct 18, 2025

smiyc commented on Nov 11, 2025

smiyc commented on Nov 13, 2025

smiyc commented on Nov 13, 2025

smiyc commented on Nov 13, 2025

HaveFun83 commented on Dec 1, 2025

HaveFun83 commented on Dec 8, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`container_memory_working_set_bytes{container="plugin-barman-cloud"}` is the culprit