Skip to content

fix(openbao): garbage-collect the data PVC on delete (#3058) - #3073

Merged
scooby87 merged 1 commit into
mainfrom
fix/openbao-data-pvc-orphaned
Jun 29, 2026
Merged

scooby87 merged 1 commit into
mainfrom
fix/openbao-data-pvc-orphaned

Conversation

@scooby87

@scooby87 scooby87 commented Jun 25, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

After deleting an OpenBAO app CR, the StatefulSet data PVC (data-<release>-0)
was left orphaned in the tenant namespace, consuming storage permanently. The
OpenBAO app renders a Flux HelmRelease over the vendored upstream OpenBAO chart,
and StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion
by default — the chart set no persistentVolumeClaimRetentionPolicy.

This passes a retention policy through the HelmRelease values. The upstream chart
renders it onto the StatefulSet from server.persistentVolumeClaimRetentionPolicy
(verified in the vendored server-statefulset.yaml) — not from
server.dataStorage.*:

server:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete
    whenScaled: Retain

A test target is added to the package Makefile so the new helm-unittest suite
(asserting the passthrough) actually runs in CI — hack/helm-unit-tests.sh only
runs packages that define a test target.

Warning

Deleting an OpenBAO app now permanently removes its data PVC and the data it holds.
This is intentional and fixes the storage leak. whenScaled: Retain is kept on
purpose so scaling an HA/raft cluster down does not destroy a member's data —
only full deletion of the release reclaims storage. Back up before deleting.

Fixes #3058.

Release note

fix(openbao): set persistentVolumeClaimRetentionPolicy so the data PVC is removed when an OpenBAO app is deleted, instead of being orphaned. WARNING: deleting an OpenBAO app now permanently removes its data PVC and the data it holds (scale-down still retains data).

Summary by CodeRabbit

  • New Features
    • Added configurable PVC retention behavior for OpenBao: storage is deleted when the release is removed and retained when scaling down.
  • Tests
    • Added a Helm unit testing workflow to run chart/unit tests.
    • Introduced assertions to confirm PVC retention settings are correctly passed through and rendered as expected.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a storage leak where OpenBAO data PVCs remained orphaned in the tenant namespace after an application was deleted. By explicitly setting the persistentVolumeClaimRetentionPolicy in the HelmRelease, the StatefulSet now correctly cleans up associated volumes upon deletion while maintaining data safety during scale-down operations.

Highlights

  • PVC Retention Policy: Configured persistentVolumeClaimRetentionPolicy to ensure data PVCs are deleted when an OpenBAO app is removed, preventing storage leaks.
  • CI Testing: Added a test target to the Makefile and implemented a new helm-unittest suite to verify the retention policy passthrough.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files labels Jun 25, 2026
@coderabbitai

coderabbitai Bot commented Jun 25, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4f2fb95b-71e0-4b86-8486-72fa81cc12f1

📥 Commits

Reviewing files that changed from the base of the PR and between bbdb766 and 9dfb078.

📒 Files selected for processing (3)
  • packages/apps/openbao/Makefile
  • packages/apps/openbao/templates/openbao.yaml
  • packages/apps/openbao/tests/pvc_retention_test.yaml
✅ Files skipped from review due to trivial changes (1)
  • packages/apps/openbao/tests/pvc_retention_test.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/apps/openbao/templates/openbao.yaml

📝 Walkthrough

Walkthrough

OpenBao now passes persistentVolumeClaimRetentionPolicy through the server values, setting delete-on-release and retain-on-scale-down behavior for the StatefulSet PVC. A Helm unittest was added, and the chart Makefile gained a test target.

Changes

OpenBao PVC retention passthrough

Layer / File(s) Summary
PVC retention passthrough and tests
packages/apps/openbao/templates/openbao.yaml, packages/apps/openbao/tests/pvc_retention_test.yaml, packages/apps/openbao/Makefile
persistentVolumeClaimRetentionPolicy is passed through for the OpenBao server PVC, the Helm unittest checks whenDeleted and whenScaled, and the Makefile adds a helm unittest . target.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related issues

  • Issue 3058: The PR adds the persistentVolumeClaimRetentionPolicy passthrough needed to delete the StatefulSet PVC when the OpenBao app is deleted.
  • Issue 3059: This also matches the same StatefulSet PVC retention handling objective by wiring persistentVolumeClaimRetentionPolicy through the Helm chart.

Possibly related PRs

Suggested reviewers

  • kvaps
  • lllamnyp
  • androndo

Poem

(_/)\
( •_•) The bunny hops through chartland bright,
/ >🍃 With Delete on farewell and Retain just right.
Helm tests squeak, “Yes, all passed today!”
No lonely PVCs can linger away.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: deleting OpenBAO now garbage-collects the data PVC.
Linked Issues check ✅ Passed The PR passes through PVC retention policy values and tests the behavior, satisfying issue #3058's delete-and-retain-on-scale-down goals.
Out of Scope Changes check ✅ Passed The Makefile test target and explanatory comments support the PVC retention fix and do not appear unrelated.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/openbao-data-pvc-orphaned

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dosubot dosubot Bot added the area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) label Jun 25, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request configures the persistent volume claim (PVC) retention policy for the OpenBAO StatefulSet, ensuring that the data PVC is deleted when the release is deleted, but retained when scaled down. It also adds a corresponding unit test suite and a test target in the Makefile. The review feedback suggests declaring the test target as .PHONY in the Makefile to prevent potential conflicts with a file or directory named test.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +7 to +8
test:
helm unittest .

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

Declaring the test target as .PHONY is a best practice to prevent make from skipping the recipe if a file or directory named test is ever created in this directory.

.PHONY: test
test:
	helm unittest .

@scooby87 scooby87 removed the area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review label Jun 25, 2026
After deleting an OpenBAO app CR, the StatefulSet data PVC (data-<release>-0)
was left orphaned in the tenant namespace, consuming storage permanently.
StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion
by default, and the chart set no persistentVolumeClaimRetentionPolicy.

Pass a retention policy through the HelmRelease values. The vendored upstream
chart renders it onto the StatefulSet from
server.persistentVolumeClaimRetentionPolicy (not server.dataStorage.*):
whenDeleted: Delete reclaims the data PVC on release deletion, while
whenScaled: Retain keeps a member's data PVC on HA scale-down.

Also add a 'test' Makefile target so the package's helm-unittest suite runs in
CI (hack/helm-unit-tests.sh only runs packages that define a 'test' target).

Signed-off-by: Алексей Артамонов <[email protected]>
@scooby87
scooby87 force-pushed the fix/openbao-data-pvc-orphaned branch from bbdb766 to 9dfb078 Compare June 25, 2026 15:24
@github-actions github-actions Bot added the area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review label Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — passes server.persistentVolumeClaimRetentionPolicy={whenDeleted: Delete, whenScaled: Retain} through to the upstream chart so the data PVC is garbage-collected on release deletion instead of orphaned.

Business context: the OpenBAO StatefulSet's volumeClaimTemplate PVC is never GC'd by default, leaving it orphaned after the instance is deleted (#3058).

Verified: the vendored upstream chart consumes this value — packages/system/openbao/charts/openbao/templates/server-statefulset.yaml renders persistentVolumeClaimRetentionPolicy from .Values.server.persistentVolumeClaimRetentionPolicy (k8s >= 1.23), and the wrapper's openbao.server.* path maps onto it. whenScaled: Retain correctly preserves raft member data on scale-down.

Non-blocking follow-ups

  1. whenDeleted: Delete is a destructive behavior change for a secrets store: deleting an instance now also destroys its data PVC, whereas before the data survived (orphaned). The sibling ClickHouse change used a breaking-change marker (fix(clickhouse)!); consider the same here, and/or a one-line release note so operators know instance deletion is now irreversible for the data volume.
  2. The helm-unittest only asserts the value is passed through the HelmRelease (it can't render the subchart) — the upstream-consumption check above is what actually proves the fix end-to-end. Fine as-is; noting the test's scope.

@scooby87
scooby87 merged commit 49555e5 into main Jun 29, 2026
12 checks passed
@scooby87
scooby87 deleted the fix/openbao-data-pvc-orphaned branch June 29, 2026 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openbao: StatefulSet data PVC is orphaned after deletion

2 participants