fix(postgres): garbage-collect init-job after completion (#3055) by scooby87 · Pull Request #3069 · cozystack/cozystack

scooby87 · 2026-06-25T10:55:40Z

Problem

After deleting a Postgres app CR, the chart-managed init Job <release>-init-job and its completed Pod remain in the tenant namespace indefinitely (reported in #3055).

The init Job is a Helm hook (post-install,post-upgrade) whose only delete policy is before-hook-creation. That policy deletes the previous hook only right before a new one is created (next upgrade) — it never fires on helm uninstall. With no hook-succeeded and no TTL, the completed Job and Pod leak.

Root cause

packages/apps/postgres/templates/init-job.yaml — the Job has no garbage-collection mechanism for its completed state.

Fix

Add ttlSecondsAfterFinished: 120 to the Job spec. The native Kubernetes TTL controller then GCs the Job and its Pod ~2 minutes after completion, regardless of the Helm hook lifecycle.

The init Job is a one-shot, idempotent reconciler of users/databases/roles (init-script.yaml) — once it has run it is not needed until the next upgrade (when Helm recreates it), so GC-ing the finished Job is safe.

This approach deliberately does not touch the hook-delete-policy / hook-timeout semantics, which the bootstrap.recovery path depends on (see the header comment in init-job.yaml). Under bootstrap.enabled the Job is skipped entirely, so the change is inert there.

Tests

Added a helm-unittest suite packages/apps/postgres/tests/init_job_cleanup_test.yaml:

asserts the init Job renders spec.ttlSecondsAfterFinished: 120
asserts the init Job is skipped under bootstrap.enabled: true

Verified locally with helm template (TTL present on default render; Job absent under bootstrap recovery).

Test plan

helm template renders ttlSecondsAfterFinished: 120 on the init Job
helm unittest packages/apps/postgres green in CI
e2e test-apps-postgres green

Fixes #3055

Summary by CodeRabbit

New Features
- Postgres bootstrap init Jobs now automatically clean up after completion using a post-finish TTL.
Tests
- Added a new integration test suite to verify the init Job renders with the expected cleanup TTL, and that the init Job is skipped when bootstrap is enabled.

The post-install/post-upgrade init Job is a one-shot reconciler of users/databases/roles. Its hook-delete-policy was `before-hook-creation` only, which never fires on uninstall, so the completed Job and its Pod lingered in the tenant namespace forever after the Postgres CR was deleted (#3055). Add `ttlSecondsAfterFinished: 120` so the Kubernetes TTL controller GCs the Job and its Pod ~2 minutes after completion, regardless of the Helm hook lifecycle. This deliberately avoids touching the hook-timeout semantics that the bootstrap.recovery path depends on. Add a helm-unittest suite locking in the TTL and the bootstrap-skip behaviour to prevent regression. Fixes #3055 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Алексей Артамонов <[email protected]>

gemini-code-assist · 2026-06-25T10:55:48Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a resource leak where completed init Jobs and Pods remained in the tenant namespace indefinitely after a Postgres app CR was deleted. By implementing a TTL on the Job, the system now automatically garbage-collects these resources after a short duration, improving cluster hygiene without disrupting existing Helm hook lifecycle requirements.

Highlights

Garbage Collection: Added ttlSecondsAfterFinished: 120 to the Postgres init Job to ensure it and its associated Pod are automatically cleaned up by Kubernetes after completion.
Testing: Introduced a new helm-unittest suite to verify the TTL configuration and ensure the Job is correctly skipped when bootstrap recovery is enabled.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	Gemini (@gemini-code-assist)	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

coderabbitai · 2026-06-25T10:55:59Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: dcec5b7a-df10-402a-a55b-2643185479e8

📥 Commits

Reviewing files that changed from the base of the PR and between cee4227 and af1cc79.

📒 Files selected for processing (1)

packages/apps/postgres/tests/init_job_cleanup_test.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

packages/apps/postgres/tests/init_job_cleanup_test.yaml

📝 Walkthrough

Walkthrough

The Postgres init Job now sets a 3600-second completion TTL. A new test suite renders the template, checks the TTL value, and verifies that enabling bootstrap skips the Job entirely.

Changes

Postgres init-job cleanup

Layer / File(s)	Summary
Init Job TTL `packages/apps/postgres/templates/init-job.yaml`	The init Job template sets `spec.ttlSecondsAfterFinished` to `3600`.
Cleanup tests `packages/apps/postgres/tests/init_job_cleanup_test.yaml`	The test suite renders `templates/init-job.yaml` for `tenant-test` and checks the TTL value and bootstrap-disabled output.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 I hopped through the hooks in a tidy little line,
A TTL of 3600 makes the old Job decline.
The bootstrap path whispers, “No Job today,”
And I nibble a carrot for cleanup that stays.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: cleaning up the Postgres init Job after completion.
Linked Issues check	✅ Passed	The TTL-based cleanup and matching tests address the orphaned init Job/Pod cleanup issue in `#3055`.
Out of Scope Changes check	✅ Passed	The changes stay focused on the init Job manifest and its cleanup test.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/init-job-and-its-pod-are-not-cleaned

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

gemini-code-assist

Code Review

This pull request adds a 'ttlSecondsAfterFinished' configuration set to 120 seconds for the Postgres 'init-job' template to ensure it is garbage-collected after completion, and introduces a unit test suite to verify this behavior. The reviewer recommends increasing this TTL to 3600 seconds (1 hour) to allow sufficient time for troubleshooting failed jobs before they are cleaned up, which requires corresponding updates to the test assertions and comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-25T10:57:06Z

+  # One-shot reconciler Job: once it has applied users/databases/roles it is no
+  # longer needed. Let the Kubernetes TTL controller garbage-collect the Job and
+  # its Pod ~2 minutes after completion so it does not linger after the release
+  # is deleted (cozystack/cozystack#3055). This does not touch the Helm hook
+  # lifecycle the bootstrap.recovery path relies on.
+  ttlSecondsAfterFinished: 120


While setting a TTL for the completed init Job is a great way to prevent resource leaks, a 120-second (2-minute) TTL might be too short for troubleshooting if the Job fails. Since ttlSecondsAfterFinished cleans up both successful and failed Jobs, a failure during database initialization would leave operators with a very narrow window to inspect the logs of the failed Pod. Increasing this to 3600 seconds (1 hour) provides a much safer buffer for production troubleshooting while still ensuring the resources are eventually garbage-collected.

# One-shot reconciler Job: once it has applied users/databases/roles it is no # longer needed. Let the Kubernetes TTL controller garbage-collect the Job and # its Pod ~1 hour after completion so it does not linger after the release # is deleted (cozystack/cozystack#3055). This does not touch the Helm hook # lifecycle the bootstrap.recovery path relies on. ttlSecondsAfterFinished: 3600

gemini-code-assist · 2026-06-25T10:57:06Z

+# Locks in cozystack/cozystack#3055: the post-install/post-upgrade init Job
+# is a one-shot reconciler of users/databases/roles. Without a TTL it lingers
+# in the namespace forever after the Postgres CR is deleted (its hook-delete-policy
+# is `before-hook-creation` only, which never fires on uninstall). A native
+# ttlSecondsAfterFinished lets the Kubernetes TTL controller GC the Job and its
+# Pod ~2 minutes after it completes, regardless of Helm hook lifecycle — without
+# touching the hook-timeout semantics the recovery flow depends on.


Update the comment in the test file to reflect the increased TTL of 1 hour instead of 2 minutes.

# Locks in cozystack/cozystack#3055: the post-install/post-upgrade init Job # is a one-shot reconciler of users/databases/roles. Without a TTL it lingers # in the namespace forever after the Postgres CR is deleted (its hook-delete-policy # is 'before-hook-creation' only, which never fires on uninstall). A native # ttlSecondsAfterFinished lets the Kubernetes TTL controller GC the Job and its # Pod ~1 hour after it completes, regardless of Helm hook lifecycle — without # touching the hook-timeout semantics the recovery flow depends on.

gemini-code-assist · 2026-06-25T10:57:06Z

+      - equal:
+          path: spec.ttlSecondsAfterFinished
+          value: 120


Update the expected ttlSecondsAfterFinished value in the unit test to match the increased TTL of 3600 seconds.

- equal: path: spec.ttlSecondsAfterFinished value: 3600

Drop the explanatory comment added alongside ttlSecondsAfterFinished; the rationale lives in the PR/commit history. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Алексей Артамонов <[email protected]>

Address review feedback on #3069: - ttlSecondsAfterFinished 120 -> 3600 so a failed init Job leaves a 1h debugging window before the TTL controller collects it (the TTL also reaps failed Jobs, not just successful ones). - Load templates/init-script.yaml in the unittest suite: the Job's checksum/config annotation includes that template, so rendering init-job.yaml in isolation failed in CI. Select the Job via documentSelector and assert the 3600 TTL. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Алексей Артамонов <[email protected]>

Signed-off-by: Алексей Артамонов <[email protected]>

Aleksei Sviridkin (lexfrei)

LGTM — adds ttlSecondsAfterFinished: 3600 so the Kubernetes TTL-after-finished controller garbage-collects the one-shot init Job (and its Pod) after completion, independent of the Helm hook lifecycle that never fired on uninstall.

Business context: the post-install/post-upgrade init Job lingered in the namespace forever after the Postgres instance was deleted — its only hook-delete-policy (before-hook-creation) never triggers on uninstall (#3055).

Verified: native ttlSecondsAfterFinished is honored by the TTL controller regardless of Helm, and it doesn't touch the hook-timeout semantics the bootstrap-recovery flow depends on. The test pins ttl=3600 and that the init Job is skipped (count: 0) under bootstrap.enabled. No blockers.

scooby87 requested review from IvanHunters, Andrey Kolkov (androndo), Andrei Kvapil (kvaps), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp), myasnikovdaniil and Kirill Ilin (sircthulhu) as code owners June 25, 2026 10:55

github-actions Bot added area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files labels Jun 25, 2026

gemini-code-assist Bot reviewed Jun 25, 2026

View reviewed changes

scooby87 and others added 3 commits June 25, 2026 13:58

fix unittest assetions

af1cc79

Signed-off-by: Алексей Артамонов <[email protected]>

Aleksei Sviridkin (lexfrei) approved these changes Jun 26, 2026

View reviewed changes

scooby87 merged commit 4db5652 into main Jun 29, 2026
12 checks passed

scooby87 deleted the fix/init-job-and-its-pod-are-not-cleaned branch June 29, 2026 11:34

Navigation Menu

Repository navigation

fix(postgres): garbage-collect init-job after completion (#3055) - #3069

fix(postgres): garbage-collect init-job after completion (#3055)#3069
scooby87 merged 4 commits into
mainfrom
fix/init-job-and-its-pod-are-not-cleaned

scooby87 commented Jun 25, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jun 25, 2026

Uh oh!

coderabbitai Bot commented Jun 25, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jun 25, 2026

Uh oh!

gemini-code-assist Bot Jun 25, 2026

Uh oh!

gemini-code-assist Bot Jun 25, 2026

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

scooby87 commented Jun 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root cause

Fix

Tests

Test plan

Summary by CodeRabbit

Uh oh!

gemini-code-assist Bot commented Jun 25, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

coderabbitai Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

scooby87 commented Jun 25, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 25, 2026 •

edited

Loading