Skip to content

[Bug]: kubectl cnpg logs pretty leaks a time.Timer per log line (defer inside an unbounded loop) #10893

Description

@Anand-240

Is there an existing issue already for this bug?

  • I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

  • I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

  • I have read the troubleshooting guide and I think this is a new bug.

Contact Details

[email protected]

Version

trunk (main)

What version of Kubernetes are you using?

1.35

What is your Kubernetes environment?

Self-managed: kind (evaluation)

How did you install the operator?

YAML manifest

What happened?

Summary

In prettyCmd.group(), a new time.Timer is created and a defer timer.Stop() is registered on every iteration of an unbounded for loop. Because defer runs only when the function returns — not at the end of each iteration — the timers and their deferred Stop() calls accumulate for the entire lifetime of the log stream, causing unbounded memory growth.

This was found by code inspection on the main branch; it is environment-independent.

Affected code

internal/cmd/plugin/logs/pretty/pretty.go, function group() (line 145); the defect is the for loop at lines 163-165:

logLoop:
	for {
		timer := time.NewTimer(1 * time.Second)
		defer timer.Stop()        // accumulates; only runs when group() returns

		select {
		case <-ctx.Done():
			break logLoop
		case <-timer.C:
			pushLogGroup()
		case logRecord, ok := <-logChannel:
			...
		}
	}

Why it happens

defer executes when the enclosing function returns, not at the end of each loop iteration. The loop runs once per incoming log record, so each record creates a fresh time.Timer and pushes another deferred Stop() onto the stack. None of them are released until group() returns, i.e. until the whole log stream ends. The deferred closures also retain a reference to each timer, preventing garbage collection.

Impact

For a long-running stream such as kubectl cnpg logs cluster <name> -f on a busy cluster (or piping a large historical log through pretty), memory and the deferred-call stack grow without bound, proportional to the number of log lines processed.

Secondary effect: because each iteration installs a brand-new 1-second timer, the inactivity-based flush is effectively reset on every record.

Steps to reproduce

  1. Stream a high volume of logs through the command, e.g. kubectl cnpg logs cluster <name> -f | kubectl cnpg logs pretty, on a busy cluster.
  2. Observe the process RSS grow steadily and unbounded with the number of log lines, instead of staying flat.

Isolated reproduction (mirroring the loop structure): feeding 1000 records produces 1001 time.NewTimer allocations whose Stop() calls do not run until the loop exits.

Expected behaviour

Memory usage should stay flat regardless of how many log lines are streamed; at most one timer should exist at a time.

Suggested fix

Create the timer once before the loop and Reset it inside, or use a single time.Ticker created before the loop with defer ticker.Stop() registered once outside the loop.

Additional context

The buggy line has existed unchanged since the logs pretty command was introduced (#5779). No open or closed issue/PR currently covers it.

Are you willing to contribute?

Yes — I'd like to be assigned to this. My plan: move the timer creation out of the loop (single time.Timer created before for, drained and Reset on each iteration, with defer timer.Stop() registered once), and confirm the inactivity-flush behaviour is preserved.

Cluster resource

N/A — found by source-code inspection, not from a running cluster.

Relevant log output

N/A — code-level defect; no runtime logs apply.

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

  1. dosubot commented on Jun 13, 2026

    @dosubot

    Good catch! I can confirm the issue exists in the current code. The group() function in internal/cmd/plugin/logs/pretty/pretty.go does indeed create a new time.Timer and register a defer timer.Stop() on every loop iteration [1]. Since defer only fires when the enclosing function returns, all those timers and deferred calls pile up for the lifetime of the stream.

    There are no existing PRs or issues addressing this — the most recent change to the file was PR #10008 (March 2026), which only added a linter suppression [2].

    Your proposed fix (single time.Timer before the loop, drained+Reset per iteration, one defer outside) is the idiomatic approach and should preserve the 1-second inactivity flush semantics while keeping memory flat.

    To reply, just mention @dosu.


    Docs are dead. Just use Dosu.

    Leave Feedback Ask Dosu about cloudnative-pg Share Dosu with your team

  2. Anand-240 commented on Jun 13, 2026

    @Anand-240
    ContributorAuthor

    I'd like to work on this.

  3. Anand-240 commented on Jun 20, 2026

    @Anand-240
    ContributorAuthor

    Hi @gbartolini, just following up on this — I'd love to work on the fix if it's available. My plan is to replace the per-iteration time.NewTimer with a single time.Ticker created before the loop, with defer ticker.Stop() registered once. Let me know if I can proceed!

  4. added
    bug 🐛Something isn't working
    and removed
    triagePending triage
    on Jun 21, 2026
  5. added theissue type on Jun 21, 2026
  6. added 2 commits that reference this issue on Jun 22, 2026
    2c6bbb3
    9946ec8
  7. added 3 commits that reference this issue on Jun 22, 2026
    0d505d5
    28f8a68
    fca7dd7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bug 🐛Something isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions