Skip to content

Commit 6918dd3

Browse files
TheSentinel454codexclaudeElrond
authored
Schedule the deletion drain safely (#7827)
## Why The deletion engine already exposes a one-shot `buzz-admin deletions drain` command, but operators have no safe way to schedule it. A generic workload wrapper could expose arbitrary commands or inherit relay credentials, which would add a second control plane and widen the blast radius. ## What This PR adds an opt-in Kubernetes CronJob for the typed deletion drain command. It is stacked on #7818, which adds owner-request admission. ## How The chart keeps the job disabled by default. The CronJob uses `Forbid`, zero Kubernetes retries, bounded runtime and history, and only the database, Redis, and object-store settings required by the executor. PostgreSQL leases, checkpoints, and retry timing remain the sole durable retry authority. The closed `_operator-jobs.tpl` helper centralizes safe CronJob mechanics without creating an arbitrary command or environment registry. This leaves the touched system simpler than a general job framework would. ## Risk The deployment blast radius is low because the job is disabled by default. When enabled, it can run destructive deletion work that the database has already approved, so the chart limits concurrency, retries, credentials, and execution time. ## Testing Rendered every `deploy/charts/buzz/ci/*.yaml` fixture with `helm template`; all passed at `e68cb753bb6673aa1baa60046969bdb2d4738b62`. Ran the full independent Desktop, Tauri, and web lanes; all passed at `e68cb753bb6673aa1baa60046969bdb2d4738b62`. Flutter package resolution made no progress within three minutes, so the mobile lane is not claimed green. ## Bigger picture A later stack layer must inventory and approve admitted owner requests. Until then, this scheduler cannot make a new owner request destructive. Generated with Codex --------- Signed-off-by: tornquist <[email protected]> Signed-off-by: Codex <[email protected]> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
1 parent 662014d commit 6918dd3

16 files changed

Lines changed: 713 additions & 7 deletions

‎ARCHITECTURE.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -750,10 +750,19 @@ Subcommands:
750750
| `remove-member` | Remove a pubkey from the relay membership list (`--pubkey`, optional `--role` guard); publishes kind:13534 roster |
751751
| `list-members` | List all relay members |
752752
| `generate-key` | Generate a new Nostr keypair (for bootstrapping) |
753+
| `deletions` | Submit, inspect, approve, abort, unblock, run, or drain durable whole-community deletion requests |
754+
| `storage-snapshot` | Run one isolated S3 accounting scan and publish its complete Postgres snapshot |
753755
| `reconcile-channels` | Emit kind:39000/39002 discovery events for channels missing them (idempotent) |
754756
755757
The `buzz-admin` binary is shipped in the relay Docker image (`/usr/local/bin/buzz-admin`) and is the recommended way to manage relay membership in production. Use `./run.sh add-member`, `./run.sh remove-member`, and `./run.sh list-members` in Docker Compose deployments.
756758
759+
Kubernetes deployments may schedule the typed one-shot
760+
`buzz-admin deletions drain` command directly. The pod owns its bounded
761+
Postgres/Redis clients and S3 client; it does not call relay HTTP. Durable
762+
requests, leases, retry timing, and checkpoints in Postgres are the handoff and
763+
execution authority, so Kubernetes uses `Forbid` concurrency and zero Job
764+
retries rather than introducing a second retry system.
765+
757766
---
758767
759768
### buzz-test-client — Integration Test Harness

‎deploy/charts/buzz/Chart.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ description: |
77
PostgreSQL and Redis. Configurable for single-node evaluation
88
(subcharts on) and HA production (external services, existingSecret).
99
type: application
10-
version: 0.1.9
10+
version: 0.1.10
1111
appVersion: "0.1.0"
1212
home: https://github.com/block/buzz
1313
sources:
@@ -24,7 +24,7 @@ maintainers:
2424
annotations:
2525
artifacthub.io/changes: |
2626
- kind: added
27-
description: Optional isolated storage-accounting CronJob with durable relay snapshots.
27+
description: Optional typed deletion-drain operator CronJob using durable database leases.
2828
artifacthub.io/license: Apache-2.0
2929

3030
# Optional eval-only subcharts. Production deploys disable both and point

‎deploy/charts/buzz/README.md‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ This chart has two operating profiles selected by values:
1212
## Quickstart (eval only)
1313

1414
```sh
15-
helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.8 \
15+
helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.10 \
1616
--create-namespace --namespace buzz \
1717
--set quickstart=true \
1818
--set postgresql.enabled=true \
@@ -116,6 +116,28 @@ is still parsed strictly, but reachability and addressing errors surface on the
116116
first storage operation. `/_readiness` tests no external dependency in either
117117
case — see the readiness contract below.
118118

119+
## Community deletion operator job
120+
121+
`operatorJobs.deletionDrain` is a disabled-by-default, typed CronJob for
122+
`/usr/local/bin/buzz-admin deletions drain`. It runs inside the relay image with
123+
bounded Job lifetime/history, `concurrencyPolicy: Forbid`, `backoffLimit: 0`,
124+
and no relay HTTP call. Postgres deletion requests, leases, retries, and
125+
checkpoints remain the execution authority.
126+
127+
The pod receives only `DATABASE_URL`, `REDIS_URL`, and required S3
128+
configuration/credential variables. It does not receive the relay private key,
129+
git-hook secret, relay URL, service links, or a generic environment registry.
130+
The chart disables the ordinary Kubernetes API service-account token mount;
131+
platform workload-identity admission may still inject its own projected token
132+
and provider environment variables. Schedule,
133+
deadline, history, termination grace, resources, service account, pod labels,
134+
and pod annotations are independently configurable under
135+
`operatorJobs.deletionDrain`.
136+
137+
See [`docs/operator-community-deletion.md`](../../../docs/operator-community-deletion.md)
138+
for enablement, permissions, the staffed first-run procedure, failure recovery,
139+
and the current explicit approval/alerting boundaries.
140+
119141
### Early-startup telemetry contract
120142

121143
`buzz_process_lifecycle` JSON records are the authoritative history for the

‎deploy/charts/buzz/templates/_helpers.tpl‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,13 @@
1717
{{- end -}}
1818
{{- end -}}
1919

20+
{{/* Kubernetes CronJob names are limited to 52 characters. */}}
21+
{{- define "buzz.cronJobName" -}}
22+
{{- $maxBaseLength := sub 51 (len .suffix) | int -}}
23+
{{- $base := include "buzz.fullname" .root | trunc $maxBaseLength | trimSuffix "-" -}}
24+
{{- printf "%s-%s" $base .suffix -}}
25+
{{- end -}}
26+
2027
{{- define "buzz.chart" -}}
2128
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
2229
{{- end -}}
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
{{/* Closed rendering foundation for typed Buzz operator jobs. */}}
2+
3+
{{- define "buzz.operatorCronJob" -}}
4+
{{- $root := .root -}}
5+
{{- if eq .type "deletionDrain" -}}
6+
{{- $job := $root.Values.operatorJobs.deletionDrain -}}
7+
{{- range $label := list "app.kubernetes.io/name" "app.kubernetes.io/instance" "app.kubernetes.io/component" -}}
8+
{{- if hasKey $job.podLabels $label -}}
9+
{{- fail (printf "operatorJobs.deletionDrain.podLabels may not set chart-owned label %q" $label) -}}
10+
{{- end -}}
11+
{{- end -}}
12+
apiVersion: batch/v1
13+
kind: CronJob
14+
metadata:
15+
name: {{ include "buzz.cronJobName" (dict "root" $root "suffix" "deletion-drain") }}
16+
labels:
17+
{{- include "buzz.labels" $root | nindent 4 }}
18+
app.kubernetes.io/component: deletion-drain
19+
spec:
20+
schedule: {{ $job.schedule | quote }}
21+
concurrencyPolicy: Forbid
22+
successfulJobsHistoryLimit: {{ $job.successfulJobsHistoryLimit }}
23+
failedJobsHistoryLimit: {{ $job.failedJobsHistoryLimit }}
24+
jobTemplate:
25+
spec:
26+
activeDeadlineSeconds: {{ $job.activeDeadlineSeconds }}
27+
backoffLimit: 0
28+
template:
29+
metadata:
30+
labels:
31+
{{- include "buzz.selectorLabels" $root | nindent 12 }}
32+
app.kubernetes.io/component: deletion-drain
33+
{{- with $job.podLabels }}
34+
{{- toYaml . | nindent 12 }}
35+
{{- end }}
36+
annotations:
37+
{{- toYaml $job.podAnnotations | nindent 12 }}
38+
spec:
39+
restartPolicy: Never
40+
terminationGracePeriodSeconds: {{ $job.terminationGracePeriodSeconds }}
41+
serviceAccountName: {{ default (include "buzz.serviceAccountName" $root) $job.serviceAccountName }}
42+
automountServiceAccountToken: false
43+
enableServiceLinks: false
44+
securityContext:
45+
{{- toYaml $root.Values.relay.securityContext | nindent 12 }}
46+
{{- with $root.Values.image.pullSecrets }}
47+
imagePullSecrets:
48+
{{- toYaml . | nindent 12 }}
49+
{{- end }}
50+
containers:
51+
- name: deletion-drain
52+
image: {{ include "buzz.image" $root }}
53+
imagePullPolicy: {{ $root.Values.image.pullPolicy }}
54+
securityContext:
55+
{{- omit $root.Values.relay.containerSecurityContext "readOnlyRootFilesystem" | toYaml | nindent 16 }}
56+
readOnlyRootFilesystem: true
57+
command: ["/usr/local/bin/buzz-admin"]
58+
args: ["deletions", "drain"]
59+
env:
60+
- { name: BUZZ_S3_ENDPOINT, value: {{ required "s3.endpoint is required when operatorJobs.deletionDrain.enabled=true" (include "buzz.s3Endpoint" $root) | quote }} }
61+
- { name: BUZZ_S3_BUCKET, value: {{ required "s3.bucket is required when operatorJobs.deletionDrain.enabled=true" $root.Values.s3.bucket | quote }} }
62+
- { name: BUZZ_S3_REGION, value: {{ $root.Values.s3.region | quote }} }
63+
- { name: BUZZ_S3_ADDRESSING_STYLE, value: {{ $root.Values.s3.addressingStyle | quote }} }
64+
- name: DATABASE_URL
65+
valueFrom:
66+
secretKeyRef:
67+
name: {{ include "buzz.envSecretName" $root }}
68+
key: DATABASE_URL
69+
- name: REDIS_URL
70+
valueFrom:
71+
secretKeyRef:
72+
name: {{ include "buzz.envSecretName" $root }}
73+
key: REDIS_URL
74+
- name: BUZZ_S3_ACCESS_KEY
75+
valueFrom:
76+
secretKeyRef:
77+
name: {{ include "buzz.envSecretName" $root }}
78+
key: BUZZ_S3_ACCESS_KEY
79+
optional: true
80+
- name: BUZZ_S3_SECRET_KEY
81+
valueFrom:
82+
secretKeyRef:
83+
name: {{ include "buzz.envSecretName" $root }}
84+
key: BUZZ_S3_SECRET_KEY
85+
optional: true
86+
resources:
87+
{{- toYaml $job.resources | nindent 16 }}
88+
{{- else -}}
89+
{{- fail (printf "unsupported typed operator job %q" .type) -}}
90+
{{- end -}}
91+
{{- end -}}

‎deploy/charts/buzz/templates/_validate.tpl‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,13 @@ surface at template time regardless of which manifest helm renders first.
1818
{{- end -}}
1919
{{- end -}}
2020

21+
{{/* The deletion executor always uses Redis for tenant-scoped invalidation. */}}
22+
{{- if .Values.operatorJobs.deletionDrain.enabled -}}
23+
{{- if and (not .Values.redis.enabled) (not .Values.externalRedis.url) (not .Values.secrets.existingSecret) -}}
24+
{{- fail "operatorJobs.deletionDrain requires Redis. Enable redis.enabled=true, set externalRedis.url, or provide secrets.existingSecret with key REDIS_URL." -}}
25+
{{- end -}}
26+
{{- end -}}
27+
2128
{{/* Multiple replicas do NOT require ReadWriteMany git storage.
2229
2330
Git ref/object state is object-store-backed: every read and write hydrates
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{{- include "buzz.validate" . -}}
2+
{{- if .Values.operatorJobs.deletionDrain.enabled }}
3+
{{- include "buzz.operatorCronJob" (dict "root" . "type" "deletionDrain") }}
4+
{{- end }}

‎deploy/charts/buzz/templates/storage-accounting-cronjob.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
apiVersion: batch/v1
33
kind: CronJob
44
metadata:
5-
name: {{ include "buzz.fullname" . }}-storage-accounting
5+
name: {{ include "buzz.cronJobName" (dict "root" . "suffix" "storage-accounting") }}
66
labels:
77
{{- include "buzz.labels" . | nindent 4 }}
88
app.kubernetes.io/component: storage-accounting

0 commit comments

Comments
 (0)