cozy-default BackupClass S3 endpoint (internal seaweedfs-s3.tenant-root.svc) unreachable from network-isolated child tenants #3256

What happened

backupstrategy-controller renders every cozy-default backup Strategy (etcd, cnpg/postgres, mariadb, foundationdb…) with the internal system-bucket endpoint:

# packages/system/backupstrategy-controller/values.yaml
backupStorage:
  endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333

A backup/snapshot pod that runs in a child tenant's namespace cannot reach that ClusterIP: the tenant's Cilium network isolation (allow-to-* egress allowlist, plus spec.isolated on the root tenant that hosts the SeaweedFS) blocks cross-tenant traffic, so the pod gets dial tcp <clusterip>:8333: i/o timeout.

This surfaced via the etcd v1alpha2 adoption safety-snapshot (migration 50): the migration correctly stages cozy-backups-creds into each adopted namespace (credentials are fine), but the per-namespace snapshot Job's upload to S3 times out from a child tenant. The useSystemBucket schema on the app charts (clickhouse/postgres/…) documents that any tenant app can back up to the system cozy-backups bucket via the cozy-default BackupClass with S3_PATH=<ns>/<release> isolation — i.e. cross-tenant access to the system bucket is by design — but the network path to reach it isn't provisioned.

Impact

Any useSystemBucket backup (and the etcd-adoption snapshot) initiated from a network-isolated child tenant fails at S3 upload. Same-namespace backups (e.g. the root tenant's own etcd/apps, which live alongside the SeaweedFS) work, which hides it. Not etcd-specific.

Repro

On a cluster with an isolated root tenant and a child tenant, from a pod in the child tenant namespace:

nc -zv seaweedfs-s3.tenant-root.svc.cozy.local 8333   # timeout
nc -zv s3.<domain> 443                                  # open (ingress, reachable via allow-to-ingress)

Suggested fix

The layer that owns the system cozy-backups bucket / tenant networking should provision cross-tenant reachability — not the migration. Two options:

• (A, simplest) Point backupStorage.endpoint at the tenant-reachable S3 ingress the system bucket is already published on (https://s3.<domain>), consistent with how per-tenant backup strategies already reach S3. Downside: hairpin + TLS trust + requires the system bucket be ingress-published.
• (B, keeps traffic in-cluster) Keep the internal endpoint but ship a CiliumNetworkPolicy from the tenant chart (packages/apps/tenant/templates/networkpolicy.yaml) — an allow-to-backups egress to the system seaweedfs-s3 for every tenant, plus the matching ingress allow on the SeaweedFS side in the isolated root tenant.

Found during a 1.5.2 → 1.6.0-rc.1 upgrade / etcd-adoption test on a 3-node cluster.