Skip to content

Commit 238b3ee

Browse files
authored
fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but only the `s3_lifecycle` job of its plugin worker applies it, and that job is scheduled by the admin server. Cozystack deployed neither, so a lifecycle rule returned 200 and no object ever expired, on the platform backup bucket and on tenant buckets alike. That also made the documented way to bound backup storage a no-op. This PR adds the admin server and a worker to Simple and MultiZone SeaweedFS instances, behind a new `lifecycle.enabled` value that is **off by default**. A Client instance is unchanged, because its rules are enforced where the instance it connects to runs. The worker registers for `s3_lifecycle` only, so vacuum, balancing and erasure coding are not started as a side effect. The admin keeps its job state on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default when that is empty; `volume.storageClass` is not reused, because in MultiZone it can name a class local to one zone or node. The admin UI is protected by a generated password, because a tenant namespace admits traffic from the whole cluster. The dashboard Role lists the two new WorkloadMonitors. Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart renders no `[grpc.s3]` section, so the s3 gateway served its internal gRPC in plaintext while the worker dials it with mTLS, and every lifecycle delete failed the TLS handshake. The section is backported from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart patch, applied with `--forward` so the next `make update` stops at the point where the patch must be dropped instead of silently reversing it. The s3 pods are restarted once so they load the section. This also repairs IAM cache propagation between s3 replicas, which failed in the same way, and it applies whether or not lifecycle is enabled. The data-plane release now retries a failed install or upgrade in place (`RetryOnFailure`) instead of rolling it back. With the previous default remediation and unlimited retries, an upgrade that waited out its timeout, for example on a tenant quota with no room for the admin volume, rolled back and retried forever, restarting the s3 pods twice per cycle. The worker's hostname anti-affinity is relaxed to a preference so a rollout finishes on a single-node cluster. #### Why lifecycle is off by default The daily walk that SeaweedFS uses to expire objects sends its deletes without an object identity, and the s3 server skips its compare-and-swap when no identity is given; it does not re-check the object's age either. A key overwritten between the walk's listing and the delete loses its new content. This path is unchanged from 4.45 through 4.52 and on upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`, `identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup buckets write unique keys and are barely exposed; a tenant bucket that rewrites a fixed key is. On top of that, turning enforcement on would start applying, on upgrade, every rule that was stored and ignored until now. Both are retention changes an operator should choose, so nothing changes for existing instances until `lifecycle.enabled` is set. Making it the default again is left to a separate PR that carries object identity on the walk path; upstream report: seaweedfs/seaweedfs#12451. #### Turning lifecycle on and off When `lifecycle.enabled` is set, rules that have been stored but ignored start applying within minutes, and objects they have already expired are deleted on that first pass, so review existing bucket lifecycle rules first. Set `lifecycle.storageClass` before enabling, and prefer a replicated class: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. Moving the volume means disabling lifecycle, deleting the `admin-data-seaweedfs-admin-0` PVC, changing the class and enabling lifecycle again. Each instance gains two small pods and a 1Gi PVC; when the tenant quota has no room for them, the data-plane release keeps retrying in place until it has. When `lifecycle.enabled` is turned off again, the parent release removes the admin credentials Secret while the data-plane release removes the admin StatefulSet only on its own reconcile, shortly after. An admin pod that restarts in that window fails to start on the missing Secret until the StatefulSet is gone; nothing is lost, and the order is left as is. The admin PVC stays until the instance is deleted, and a later re-enable starts the admin with a new password. Verified on a test cluster with the 4.45 chart: without the backport, the worker found the expired objects and every delete failed with `tls: first record does not look like a TLS handshake`; with this branch and lifecycle enabled, a rule with a past expiration date removed the matching objects and left the others, both on a manual run and on the first scheduled pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up needed, not opened yet: `operations/services/backup-classes.md` (v1.7 and next) advises an S3 lifecycle policy, which the bundled SeaweedFS applies only once `lifecycle.enabled` is set; the SeaweedFS reference page gains `lifecycle.enabled` and `lifecycle.storageClass` from the regenerated README on the next release - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The terraform provider does not model the SeaweedFS application, so the new values do not reach it. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added optional, daily S3 bucket lifecycle-rule processing for supported SeaweedFS configurations. It is disabled by default; Client instances rely on lifecycle rules configured on the connected SeaweedFS. * Added configurable storage for lifecycle job state and dashboard visibility for lifecycle components. * Enabled the S3 gRPC server to use its internal TLS identity when security is configured. * **Bug Fixes** * Improved storage-class handling so numeric-looking names remain strings and unset classes are omitted where appropriate. * Clarified backup cleanup guidance, including when lifecycle rules apply and how to reclaim storage. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 3badbde + f5114ae commit 238b3ee

30 files changed

Lines changed: 1229 additions & 29 deletions

‎examples/backups/kafka/92-scenario-user-restore.md‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,6 @@ The script:
7171
had already replayed. The Pod log names each topic before its replay and
7272
again once it landed; empty or delete every topic it shows as started, not
7373
only the one the refusal names, then re-run.
74-
- The tarball in object storage is not managed by the Cozystack `Backup`
75-
lifecycle: deleting the `Backup` CR removes the reference but leaves the S3
76-
object in place. Tenants who need to reclaim space should delete the object
77-
(or set a bucket lifecycle policy).
74+
- The tarball in object storage is not managed by the Cozystack `Backup` lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 object in place. Tenants who need to reclaim space should delete the object, or set a lifecycle rule on the bucket. A Simple or MultiZone Cozystack SeaweedFS applies such rules once a day only while its `lifecycle.enabled` is on, which is not the default; a Client instance applies none itself, so the SeaweedFS it connects to must have `lifecycle.enabled` on. Another object store applies them only if it implements bucket lifecycle.
7875
- Consumer-group offsets, topic configs, ACLs and users are not part of the
7976
backup; consumers start from the restored topic's fresh offsets.

‎examples/backups/nats/92-scenario-user-restore.md‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,6 @@ The script:
5555

5656
- Both flows rely on the strategy template branching on `.Mode`. See
5757
`01-create-strategy.sh`.
58-
- The tarball in object storage is not managed by the Cozystack `Backup`
59-
lifecycle: deleting the `Backup` CR removes the reference but leaves the S3
60-
object in place. Tenants who need to reclaim space should delete the object
61-
(or set a bucket lifecycle policy).
58+
- The tarball in object storage is not managed by the Cozystack `Backup` lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 object in place. Tenants who need to reclaim space should delete the object, or set a lifecycle rule on the bucket. A Simple or MultiZone Cozystack SeaweedFS applies such rules once a day only while its `lifecycle.enabled` is on, which is not the default; a Client instance applies none itself, so the SeaweedFS it connects to must have `lifecycle.enabled` on. Another object store applies them only if it implements bucket lifecycle.
6259
- `nats stream restore` recreates the stream's configuration from the snapshot;
6360
consumers are restored as captured at backup time.

‎internal/backupcontroller/psmdbstrategy_controller.go‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1376,9 +1376,10 @@ func (r *BackupJobReconciler) createMongoDBBackupArtifact(
13761376
// annotation (honoured before any apiserver read, so it still frees a Backup
13771377
// whose operator CR has become unreadable) and a going-away namespace (the psmdb
13781378
// delete-backup exec cannot run there) each make the driver strip its own
1379-
// finalizer and release, leaving the object for a bucket lifecycle policy. This
1380-
// bounds only what the driver contributes: a Get that fails with something other
1381-
// than NotFound/NoMatchError still requeues (below), and the psmdb operator's own
1379+
// finalizer and release, leaving the object to manual cleanup, or to a
1380+
// lifecycle rule on the bucket where the object store applies one. This bounds
1381+
// only what the driver contributes: a Get that fails with something other than
1382+
// NotFound/NoMatchError still requeues (below), and the psmdb operator's own
13821383
// release-lock finalizer keeps the CR Terminating while the operator is down,
13831384
// whatever the driver strips. A down operator in a live namespace is likewise
13841385
// not auto-released: the Backup stays Terminating as a visible signal until an

‎packages/extra/seaweedfs/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,4 +78,7 @@
7878
| `s3.resources.cpu` | Number of CPU cores allocated. | `quantity` | `""` |
7979
| `s3.resources.memory` | Amount of memory allocated. | `quantity` | `""` |
8080
| `s3.resourcesPreset` | Default sizing preset used when `resources` is omitted. | `string` | `t1.small` |
81+
| `lifecycle` | S3 bucket lifecycle configuration. | `object` | `{}` |
82+
| `lifecycle.enabled` | Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new. | `bool` | `false` |
83+
| `lifecycle.storageClass` | StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again. | `string` | `""` |
8184

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{{- if and .Values.lifecycle.enabled (ne .Values.topology "Client") }}
2+
{{- $name := printf "%s-admin-credentials" .Release.Name }}
3+
{{- /* The admin reads these into its environment when it starts and nothing
4+
restarts it on a Secret change, so a password regenerated on every render
5+
would leave the Secret holding one the running server does not accept.
6+
Keep the one already issued. */}}
7+
{{- $password := randAlphaNum 40 | b64enc }}
8+
{{- with lookup "v1" "Secret" .Release.Namespace $name }}
9+
{{- with (dig "data" "adminPassword" "" .) }}
10+
{{- $password = . }}
11+
{{- end }}
12+
{{- end }}
13+
apiVersion: v1
14+
kind: Secret
15+
metadata:
16+
name: {{ $name }}
17+
type: Opaque
18+
data:
19+
adminUser: {{ "admin" | b64enc }}
20+
adminPassword: {{ $password }}
21+
{{- end }}

‎packages/extra/seaweedfs/templates/dashboard-resourcemap.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@ rules:
2727
- {{ $.Release.Name }}-filer
2828
- {{ $.Release.Name }}-db
2929
- {{ $.Release.Name }}-s3
30+
{{- if .Values.lifecycle.enabled }}
31+
- {{ $.Release.Name }}-admin
32+
- {{ $.Release.Name }}-worker
33+
{{- end }}
3034
{{- if eq .Values.topology "Simple" }}
3135
- {{ $.Release.Name }}-volume
3236
{{- range $poolName, $pool := .Values.volume.pools }}

‎packages/extra/seaweedfs/templates/seaweedfs-db.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,6 @@ spec:
4242
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.db.resourcesPreset .Values.db.resources $) | nindent 8 }}
4343
size: {{ .Values.db.size }}
4444
{{- with .Values.db.storageClass }}
45-
storageClass: {{ . }}
45+
storageClass: {{ . | quote }}
4646
{{- end }}
4747
{{- end }}

‎packages/extra/seaweedfs/templates/seaweedfs.yaml‎

Lines changed: 43 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -214,10 +214,21 @@ spec:
214214
namespace: cozy-system
215215
interval: 30s
216216
timeout: 10m
217+
{{- /* Flux remediates a failed install by uninstalling and a failed upgrade
218+
by rolling back, and with retries: -1 it does so on every attempt. A
219+
release waiting on tenant quota for the admin volume would then roll
220+
the s3 pods back and forth with the [grpc.s3] restart annotation.
221+
RetryOnFailure keeps what was applied and retries it. */}}
217222
install:
223+
strategy:
224+
name: RetryOnFailure
225+
retryInterval: 30s
218226
remediation:
219227
retries: -1
220228
upgrade:
229+
strategy:
230+
name: RetryOnFailure
231+
retryInterval: 30s
221232
remediation:
222233
retries: -1
223234
valuesFrom:
@@ -249,7 +260,7 @@ spec:
249260
type: "persistentVolumeClaim"
250261
size: "{{ .Values.volume.size }}"
251262
{{- with .Values.volume.storageClass }}
252-
storageClass: {{ . }}
263+
storageClass: {{ . | quote }}
253264
{{- end }}
254265
maxVolumes: 0
255266
{{- if .Values.volume.diskType }}
@@ -268,7 +279,7 @@ spec:
268279
type: "persistentVolumeClaim"
269280
size: "{{ $pool.size | default $.Values.volume.size }}"
270281
{{- with ($pool.storageClass | default $.Values.volume.storageClass) }}
271-
storageClass: "{{ . }}"
282+
storageClass: {{ . | quote }}
272283
{{- end }}
273284
maxVolumes: 0
274285
extraArgs:
@@ -284,7 +295,7 @@ spec:
284295
type: "persistentVolumeClaim"
285296
size: "{{ $zone.size | default $.Values.volume.size }}"
286297
{{- with ($zone.storageClass | default $.Values.volume.storageClass) }}
287-
storageClass: "{{ . }}"
298+
storageClass: {{ . | quote }}
288299
{{- end }}
289300
maxVolumes: 0
290301
nodeSelector: |
@@ -309,7 +320,7 @@ spec:
309320
type: "persistentVolumeClaim"
310321
size: "{{ $pool.size | default $zone.size | default $.Values.volume.size }}"
311322
{{- with ($pool.storageClass | default $zone.storageClass | default $.Values.volume.storageClass) }}
312-
storageClass: "{{ . }}"
323+
storageClass: {{ . | quote }}
313324
{{- end }}
314325
maxVolumes: 0
315326
nodeSelector: |
@@ -371,6 +382,16 @@ spec:
371382
secretName: {{ .Release.Name }}-s3-ingress-tls
372383
{{- end }}
373384
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.s3.resourcesPreset .Values.s3.resources $) | nindent 10 }}
385+
admin:
386+
enabled: {{ .Values.lifecycle.enabled }}
387+
secret:
388+
existingSecret: {{ .Release.Name }}-admin-credentials
389+
{{- with .Values.lifecycle.storageClass }}
390+
data:
391+
storageClass: {{ . | quote }}
392+
{{- end }}
393+
worker:
394+
enabled: {{ .Values.lifecycle.enabled }}
374395
cosi:
375396
driverName: "{{ .Release.Namespace }}.seaweedfs.objectstorage.k8s.io"
376397
bucketClassName: "{{ .Release.Namespace }}"
@@ -503,4 +524,22 @@ spec:
503524
app.kubernetes.io/component: s3
504525
app.kubernetes.io/name: seaweedfs
505526
version: {{ $.Chart.Version }}
527+
{{- if .Values.lifecycle.enabled }}
528+
{{- range $component := list "admin" "worker" }}
529+
---
530+
apiVersion: cozystack.io/v1alpha1
531+
kind: WorkloadMonitor
532+
metadata:
533+
name: {{ $.Release.Name }}-{{ $component }}
534+
spec:
535+
replicas: 1
536+
minReplicas: 1
537+
kind: seaweedfs
538+
type: {{ $component }}
539+
selector:
540+
app.kubernetes.io/component: {{ $component }}
541+
app.kubernetes.io/name: seaweedfs
542+
version: {{ $.Chart.Version }}
543+
{{- end }}
544+
{{- end }}
506545
{{- end }}
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
suite: seaweedfs metadata database storage class reaches its release as a string
2+
3+
templates:
4+
- templates/seaweedfs-db.yaml
5+
6+
release:
7+
name: seaweedfs
8+
namespace: tenant-root
9+
10+
tests:
11+
# A StorageClass name may be all digits; handed on unquoted it reaches the
12+
# database release as a number, and a line break in it would open new keys
13+
# in that release's values.
14+
- it: hands a numeric storage class name on as a string
15+
set:
16+
db:
17+
storageClass: "123"
18+
asserts:
19+
- equal:
20+
path: spec.values.db.storageClass
21+
value: "123"
22+
23+
- it: keeps a storage class with a line break inside its own field
24+
set:
25+
db:
26+
storageClass: "local\n replicas: 9"
27+
asserts:
28+
- equal:
29+
path: spec.values.db.storageClass
30+
value: "local\n replicas: 9"

0 commit comments

Comments
 (0)