Skip to content

Commit 5bcdb8c

Browse files
androndoclaude
androndo
and
claude
committed
fix(postgres): let in-place restore re-bootstrap instead of wedging
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the Postgres app under its own name. On stock v1.6.2 it never comes back up, for two independent reasons in the same restore patch - both surface only in place, because a to-copy restore targets a different app. 1. Archive-vs-recovery serverName collision. The driver set the restored cluster's WAL-archive serverName equal to the recovery source (bootstrap.serverName = bootstrap.oldName = sourceServerName) while the chart pinned the archiver serverName to the release name. In place those coincide, so the cluster archived onto the source's own WAL prefix and barman-cloud-check-wal-archive refused to start ("Expected empty archive"). Fixed by giving the restored cluster a fresh bootstrap.newServerName, keyed by the RestoreJob UID (unique per restore, stable across reconciles); the chart archives there while externalClusters still recovers from bootstrap.serverName. This is what the manual CNPG strategies hit (serverName defaults to the cluster name); the cozy-default flow dodged it (serverName is <ns>-<app>). 2. useSystemBucket guard. The restore populates explicit S3 coordinates (destinationPath / s3CredentialsSecret / endpointCA) but left backup.useSystemBucket=true, so the chart's "bootstrap.enabled is incompatible with useSystemBucket" guard failed the HelmRelease upgrade and the cluster never re-rendered. This is what the cozy-default (system-bucket) flow hits on an in-place restore. Fixed by clearing useSystemBucket in the restore patch, since the app is now on the explicit-creds path. The driver's app view (postgresapp.Backup) gains the field, without omitempty so a false survives the merge patch. Both are exercised only by an in-place restore; the postgres backup e2e drove only to-copy, so neither surfaced in CI. A step 50 in-place restore now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go unit assertion (a system-bucket in-place e2e needs the platform bucket in CI and is left as a follow-up). Signed-off-by: Andrey Kolkov <[email protected]> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 68d83cb commit 5bcdb8c

14 files changed

Lines changed: 194 additions & 34 deletions

File tree

‎api/apps/v1alpha1/postgresql/types.go‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎examples/backups/postgres/00-helpers.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ export BACKUPJOB_POSTMARKER_NAME="${BACKUPJOB_POSTMARKER_NAME:-pg-src-postmarker
3131
export RESTOREJOB_TOCOPY_NAME="${RESTOREJOB_TOCOPY_NAME:-pg-src-to-pg-target}"
3232
export RESTOREJOB_PITR_NAME="${RESTOREJOB_PITR_NAME:-pg-src-to-pg-target-pitr}"
3333
export RESTOREJOB_UNREACHABLE_NAME="${RESTOREJOB_UNREACHABLE_NAME:-pg-src-to-pg-target-unreachable}"
34+
export RESTOREJOB_INPLACE_NAME="${RESTOREJOB_INPLACE_NAME:-pg-src-in-place}"
3435
export PLAN_NAME="${PLAN_NAME:-pg-src-daily}"
3536
# App user password baked into 05-postgres-src.yaml (REPLACE_WITH_PASSWORD).
3637
export PG_PASSWORD="${PG_PASSWORD:-Xai7Wepo0aeThie8}"

‎examples/backups/postgres/cleanup.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@ done
2222

2323
log_substep "Deleting RestoreJobs..."
2424
kubectl -n "$NAMESPACE" delete restorejob.backups.cozystack.io \
25-
"$RESTOREJOB_TOCOPY_NAME" "$RESTOREJOB_PITR_NAME" "$RESTOREJOB_UNREACHABLE_NAME" --ignore-not-found
25+
"$RESTOREJOB_TOCOPY_NAME" "$RESTOREJOB_PITR_NAME" "$RESTOREJOB_UNREACHABLE_NAME" \
26+
"$RESTOREJOB_INPLACE_NAME" --ignore-not-found
2627

2728
log_substep "Deleting Plan + BackupJob + derived Backup artefacts..."
2829
kubectl -n "$NAMESPACE" delete plan.backups.cozystack.io "$PLAN_NAME" --ignore-not-found

‎examples/backups/postgres/run-all.sh‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,3 +308,22 @@ if [[ "$REASON" != "RecoveryTargetUnreachable" ]]; then
308308
exit 1
309309
fi
310310
log_success "Verified: recoveryTime ${UNREACHABLE_TIME} past the archive -> RestoreJob Failed with reason RecoveryTargetUnreachable."
311+
312+
# In-place restore last: it deletes pg-src and re-bootstraps from S3, so nothing
313+
# after it may depend on the source. Regression guard for the archive-vs-recovery
314+
# serverName collision - before bootstrap.newServerName the restored cluster
315+
# archived onto the source's own WAL prefix and wedged on "Expected empty archive".
316+
print_header "Step 50: in-place restore of '${PG_SRC_NAME}' from '${BACKUP_NAME}' and wait for Succeeded"
317+
kubectl -n "$NAMESPACE" apply -f "$SCRIPT_DIR/35-restorejob-in-place.yaml"
318+
wait_for_field restorejobs.backups.cozystack.io "$RESTOREJOB_INPLACE_NAME" \
319+
'{.status.phase}' Succeeded "$NAMESPACE" 1200 Failed
320+
wait_for_field clusters.postgresql.cnpg.io "$PG_SRC_CLUSTER" \
321+
'{.status.phase}' 'Cluster in healthy state' "$NAMESPACE" 600
322+
323+
print_header "Step 50 verify: the sentinel survived the in-place restore"
324+
GOT=$(psql_exec "$PG_SRC_CLUSTER" demo "SELECT token FROM e2e_sentinel WHERE id = 1;" | tr -d '[:space:]')
325+
if [[ "$GOT" != "$SENTINEL_TOKEN" ]]; then
326+
log_error "in-place restore sentinel mismatch: source has '${GOT}', expected '${SENTINEL_TOKEN}'"
327+
exit 1
328+
fi
329+
log_success "In-place restore verified: '${PG_SRC_NAME}' re-bootstrapped and archives to a fresh prefix (sentinel '${GOT}')."

‎hack/e2e-chainsaw/postgres/chainsaw-test.yaml‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -314,13 +314,14 @@ spec:
314314
# latest-point to-copy restore, the point-in-time restore (step 45), and
315315
# the negative unreachable-target restore (step 46, which sets a short
316316
# restoreTimeoutSeconds and must fail at that deadline with reason
317-
# RecoveryTargetUnreachable) (convention 6: justified in-line). The old 25m
318-
# budget predated the PITR phases; three restores plus the negative phase
319-
# need the larger window. run-all.sh fails fast on terminal
320-
# BackupJob/RestoreJob phases and stalled HelmReleases rather than polling
321-
# out the budget.
317+
# RecoveryTargetUnreachable), and the in-place restore (step 50, the
318+
# regression guard for the archive-vs-recovery serverName collision)
319+
# (convention 6: justified in-line). The old 25m budget predated the PITR
320+
# phases; four restores plus the negative phase need the larger window.
321+
# run-all.sh fails fast on terminal BackupJob/RestoreJob phases and stalled
322+
# HelmReleases rather than polling out the budget.
322323
- script:
323-
timeout: 40m
324+
timeout: 50m
324325
content: |
325326
set -eu
326327
cd ../../..

‎internal/backupcontroller/cnpgstrategy_controller.go‎

Lines changed: 37 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ package backupcontroller
33

44
import (
55
"context"
6+
"crypto/sha256"
7+
"encoding/hex"
68
"encoding/json"
79
"fmt"
810
"io"
@@ -865,7 +867,16 @@ func (r *RestoreJobReconciler) reconcileCNPGRestore(ctx context.Context, restore
865867
if err := r.setCNPGRestoreHRSuspended(ctx, target.Namespace, hrName, true); err != nil {
866868
return ctrl.Result{}, err
867869
}
868-
if err := r.patchPostgresAppForRestore(ctx, targetApp, sourceServerName, sourceDestinationPath, sourceEndpointURL, options.RecoveryTime, rendered.BarmanObjectStore.S3Credentials, rendered.BarmanObjectStore.EndpointCA, sourceDatabases, sourceUsers); err != nil {
870+
newServerName := restoredServerName(clusterName, restoreJob.UID)
871+
if newServerName == sourceServerName {
872+
// Practically unreachable (see restoredServerName), but if a digest
873+
// ever collides the restored cluster would archive onto the source's
874+
// prefix and CNPG would fail with "Expected empty archive" - log so
875+
// that recurrence is diagnosable as a hash collision, not a mystery.
876+
logger.Info("restored WAL-archive serverName collided with the recovery source",
877+
"cluster", clusterName, "serverName", newServerName)
878+
}
879+
if err := r.patchPostgresAppForRestore(ctx, targetApp, sourceServerName, newServerName, sourceDestinationPath, sourceEndpointURL, options.RecoveryTime, rendered.BarmanObjectStore.S3Credentials, rendered.BarmanObjectStore.EndpointCA, sourceDatabases, sourceUsers); err != nil {
869880
// Resume HR before terminal failure so an operator deleting
870881
// the failed RestoreJob does not leave the HR stuck.
871882
_ = r.setCNPGRestoreHRSuspended(ctx, target.Namespace, hrName, false)
@@ -1064,16 +1075,28 @@ func (r *RestoreJobReconciler) resolveCNPGRestoreTarget(restoreJob *backupsv1alp
10641075
func (r *RestoreJobReconciler) patchPostgresAppForRestore(
10651076
ctx context.Context,
10661077
app *postgresapp.Postgres,
1067-
sourceServerName, sourceDestinationPath, sourceEndpointURL, recoveryTime string,
1078+
sourceServerName, newServerName, sourceDestinationPath, sourceEndpointURL, recoveryTime string,
10681079
credsRef *strategyv1alpha1.S3CredentialsTemplate,
10691080
caRef *strategyv1alpha1.EndpointCARef,
10701081
sourceDatabases map[string]postgresapp.Database,
10711082
sourceUsers map[string]postgresapp.User,
10721083
) error {
1073-
patched := buildPostgresAppRestorePatch(app, sourceServerName, sourceDestinationPath, sourceEndpointURL, recoveryTime, credsRef, caRef, sourceDatabases, sourceUsers)
1084+
patched := buildPostgresAppRestorePatch(app, sourceServerName, newServerName, sourceDestinationPath, sourceEndpointURL, recoveryTime, credsRef, caRef, sourceDatabases, sourceUsers)
10741085
return r.Patch(ctx, patched, client.MergeFrom(app), client.FieldOwner(cnpgFieldManager))
10751086
}
10761087

1088+
// restoredServerName returns the WAL-archive serverName a restored cluster
1089+
// writes to. Keyed by the RestoreJob UID, it is unique per restore and never
1090+
// equals a source serverName, so the re-bootstrapped cluster archives into an
1091+
// empty prefix and barman-cloud-check-wal-archive passes even on an in-place
1092+
// restore (where the target cluster name equals the source's).
1093+
func restoredServerName(clusterName string, uid types.UID) string {
1094+
// 64 bits of the digest: enough that two RestoreJob UIDs colliding on the
1095+
// same cluster is not a practical concern, while keeping the prefix short.
1096+
sum := sha256.Sum256([]byte(uid))
1097+
return fmt.Sprintf("%s-restore-%s", clusterName, hex.EncodeToString(sum[:])[:16])
1098+
}
1099+
10771100
// buildPostgresAppRestorePatch returns a deep-copied Postgres application
10781101
// instance with restore-related fields overlaid. Pure: no cluster I/O. The
10791102
// wrapper above invokes the Kubernetes Patch.
@@ -1088,7 +1111,7 @@ func (r *RestoreJobReconciler) patchPostgresAppForRestore(
10881111
// anything not in spec) must see the source's exact map.
10891112
func buildPostgresAppRestorePatch(
10901113
app *postgresapp.Postgres,
1091-
sourceServerName, sourceDestinationPath, sourceEndpointURL, recoveryTime string,
1114+
sourceServerName, newServerName, sourceDestinationPath, sourceEndpointURL, recoveryTime string,
10921115
credsRef *strategyv1alpha1.S3CredentialsTemplate,
10931116
caRef *strategyv1alpha1.EndpointCARef,
10941117
sourceDatabases map[string]postgresapp.Database,
@@ -1098,10 +1121,20 @@ func buildPostgresAppRestorePatch(
10981121
patched.Spec.Bootstrap.Enabled = true
10991122
patched.Spec.Bootstrap.OldName = sourceServerName
11001123
patched.Spec.Bootstrap.ServerName = sourceServerName
1124+
// Archive under a fresh prefix distinct from the recovery source, or an
1125+
// in-place restore fails barman-cloud-check-wal-archive ("Expected empty archive").
1126+
patched.Spec.Bootstrap.NewServerName = newServerName
11011127
patched.Spec.Bootstrap.RecoveryTime = recoveryTime
11021128

11031129
patched.Spec.Backup.DestinationPath = sourceDestinationPath
11041130
patched.Spec.Backup.EndpointURL = sourceEndpointURL
1131+
// The restore populates explicit S3 coordinates (destinationPath /
1132+
// s3CredentialsSecret / endpointCA below), so the app is no longer on the
1133+
// system-bucket path. Clear useSystemBucket: the chart rejects
1134+
// bootstrap.enabled together with useSystemBucket (an unset-coordinates
1135+
// guard), and a system-bucket app restored in place would otherwise wedge
1136+
// its HelmRelease in that render error instead of re-bootstrapping.
1137+
patched.Spec.Backup.UseSystemBucket = false
11051138
// Switching to s3CredentialsSecret means inline keys must not survive
11061139
// on the CR .spec; otherwise tenants who switch credential modes leave
11071140
// cleartext keys behind in etcd and audit logs.

‎internal/backupcontroller/cnpgstrategy_controller_test.go‎

Lines changed: 57 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import (
1515
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
1616
"k8s.io/apimachinery/pkg/runtime"
1717
"k8s.io/apimachinery/pkg/runtime/schema"
18+
"k8s.io/apimachinery/pkg/types"
1819
"k8s.io/client-go/dynamic"
1920
dynamicfake "k8s.io/client-go/dynamic/fake"
2021
"k8s.io/client-go/kubernetes/scheme"
@@ -491,6 +492,7 @@ func TestBuildPostgresAppRestorePatch_ForwardsSecretRef_NotCleartext(t *testing.
491492
patched := buildPostgresAppRestorePatch(
492493
app,
493494
"pg-src",
495+
"postgres-pg-target-restore-deadbeef",
494496
"s3://bucket/pg-src/",
495497
"https://s3.example",
496498
"",
@@ -524,7 +526,7 @@ func TestBuildPostgresAppRestorePatch_ForwardsCustomKeyOverrides(t *testing.T) {
524526
SecretAccessKeyKey: "SECRET_KEY",
525527
}
526528

527-
patched := buildPostgresAppRestorePatch(app, "src", "s3://b/", "", "", creds, nil, nil, nil)
529+
patched := buildPostgresAppRestorePatch(app, "src", "pg-new", "s3://b/", "", "", creds, nil, nil, nil)
528530

529531
want := postgresapp.S3CredentialsSecret{
530532
Name: "creds",
@@ -542,7 +544,7 @@ func TestBuildPostgresAppRestorePatch_ForwardsCustomKeyOverrides(t *testing.T) {
542544
// Secret name and helm install would fail.
543545
func TestBuildPostgresAppRestorePatch_NoSecretRefIsSkipped(t *testing.T) {
544546
app := newPostgresApp("pg", "tenant")
545-
patched := buildPostgresAppRestorePatch(app, "src", "s3://b/", "", "", nil, nil, nil, nil)
547+
patched := buildPostgresAppRestorePatch(app, "src", "pg-new", "s3://b/", "", "", nil, nil, nil, nil)
546548
if got := patched.Spec.Backup.S3CredentialsSecret; got != (postgresapp.S3CredentialsSecret{}) {
547549
t.Errorf("spec.backup.s3CredentialsSecret must be zero when credsRef is nil; got %#v", got)
548550
}
@@ -700,7 +702,7 @@ func TestBuildPostgresAppRestorePatch_ReplacesTargetUsersAndDatabases(t *testing
700702
"appdb": {Extensions: []string{"hstore"}},
701703
}
702704

703-
patched := buildPostgresAppRestorePatch(app, "src", "s3://b/", "", "", nil, nil, sourceDatabases, sourceUsers)
705+
patched := buildPostgresAppRestorePatch(app, "src", "pg-new", "s3://b/", "", "", nil, nil, sourceDatabases, sourceUsers)
704706

705707
if _, ok := patched.Spec.Users["stale-target-user"]; ok {
706708
t.Errorf("stale target user survived restore; replace semantics regressed")
@@ -726,6 +728,7 @@ func TestBuildPostgresAppRestorePatch_ReplacesTargetUsersAndDatabases(t *testing
726728
func TestBuildPostgresAppRestorePatch_ScrubsStaleBackupSettings(t *testing.T) {
727729
app := newPostgresApp("pg-target", "tenant")
728730
app.Spec.Bootstrap.RecoveryTime = "2025-01-01T00:00:00Z"
731+
app.Spec.Backup.UseSystemBucket = true
729732
app.Spec.Backup.EndpointURL = "https://stale.example"
730733
app.Spec.Backup.S3AccessKey = "stale-ak"
731734
app.Spec.Backup.S3SecretKey = "stale-sk"
@@ -736,7 +739,7 @@ func TestBuildPostgresAppRestorePatch_ScrubsStaleBackupSettings(t *testing.T) {
736739

737740
// Restore with no recoveryTime, no endpointURL, no creds, no CA -
738741
// everything stale on the target must be wiped.
739-
patched := buildPostgresAppRestorePatch(app, "src", "s3://b/", "", "", nil, nil, nil, nil)
742+
patched := buildPostgresAppRestorePatch(app, "src", "pg-new", "s3://b/", "", "", nil, nil, nil, nil)
740743

741744
if got := patched.Spec.Bootstrap.RecoveryTime; got != "" {
742745
t.Errorf("stale recoveryTime survived; got %q", got)
@@ -756,6 +759,11 @@ func TestBuildPostgresAppRestorePatch_ScrubsStaleBackupSettings(t *testing.T) {
756759
if patched.Spec.Backup.EndpointCA != (postgresapp.EndpointCA{}) {
757760
t.Errorf("stale endpointCA survived; got %+v", patched.Spec.Backup.EndpointCA)
758761
}
762+
// A system-bucket source must be flipped to the explicit-creds path, or the
763+
// chart's bootstrap.enabled + useSystemBucket guard wedges the restore.
764+
if patched.Spec.Backup.UseSystemBucket {
765+
t.Errorf("useSystemBucket must be cleared on restore so the bootstrap guard passes")
766+
}
759767
}
760768

761769
// TestBuildPostgresAppRestorePatch_ForwardsEndpointCA verifies that a
@@ -770,14 +778,58 @@ func TestBuildPostgresAppRestorePatch_ForwardsEndpointCA(t *testing.T) {
770778
SecretRef: corev1.LocalObjectReference{Name: "pg-cnpg-backup-ca"},
771779
Key: "ca.crt",
772780
}
773-
patched := buildPostgresAppRestorePatch(app, "src", "s3://b/", "", "", nil, caRef, nil, nil)
781+
patched := buildPostgresAppRestorePatch(app, "src", "pg-new", "s3://b/", "", "", nil, caRef, nil, nil)
774782

775783
want := postgresapp.EndpointCA{Name: "pg-cnpg-backup-ca", Key: "ca.crt"}
776784
if got := patched.Spec.Backup.EndpointCA; got != want {
777785
t.Fatalf("endpointCA mismatch\n got: %#v\nwant: %#v", got, want)
778786
}
779787
}
780788

789+
// TestBuildPostgresAppRestorePatch_ArchiveServerNameDiffersFromSource guards the
790+
// in-place restore fix: the restored cluster archives under bootstrap.newServerName,
791+
// distinct from the recovery source (bootstrap.serverName / oldName).
792+
func TestBuildPostgresAppRestorePatch_ArchiveServerNameDiffersFromSource(t *testing.T) {
793+
app := newPostgresApp("pg", "tenant")
794+
// An in-place restore is the worst case: source serverName equals the
795+
// target cluster's own name.
796+
const source = "postgres-pg"
797+
patched := buildPostgresAppRestorePatch(app, source, "postgres-pg-restore-deadbeef", "s3://b/", "", "", nil, nil, nil, nil)
798+
799+
if got := patched.Spec.Bootstrap.ServerName; got != source {
800+
t.Errorf("recovery serverName must stay the source; got %q want %q", got, source)
801+
}
802+
if got := patched.Spec.Bootstrap.OldName; got != source {
803+
t.Errorf("recovery oldName must stay the source; got %q want %q", got, source)
804+
}
805+
if got := patched.Spec.Bootstrap.NewServerName; got == "" || got == source {
806+
t.Errorf("archive newServerName must be non-empty and differ from the source %q; got %q", source, got)
807+
}
808+
}
809+
810+
// TestRestoredServerName pins the archive-prefix generator: deterministic per
811+
// RestoreJob UID, distinct across UIDs, and never equal to the plain cluster
812+
// name (which is the source serverName on an in-place restore).
813+
func TestRestoredServerName(t *testing.T) {
814+
const cluster = "postgres-pg"
815+
a1 := restoredServerName(cluster, types.UID("3c2ee41b-346f-4ae2-98d2-5d629c96f744"))
816+
a2 := restoredServerName(cluster, types.UID("3c2ee41b-346f-4ae2-98d2-5d629c96f744"))
817+
b := restoredServerName(cluster, types.UID("9eb8feed-7d49-467e-b719-d1896c0602d5"))
818+
819+
if a1 != a2 {
820+
t.Errorf("not deterministic for a fixed UID: %q vs %q", a1, a2)
821+
}
822+
if a1 == b {
823+
t.Errorf("distinct UIDs must yield distinct serverNames; both %q", a1)
824+
}
825+
if a1 == cluster {
826+
t.Errorf("restored serverName must never equal the plain cluster name %q", cluster)
827+
}
828+
if !strings.HasPrefix(a1, cluster+"-restore-") {
829+
t.Errorf("restored serverName %q must carry the %q-restore- prefix", a1, cluster)
830+
}
831+
}
832+
781833
// TestMarshalUnmarshalCNPGBackupSnapshot round-trips the source-spec
782834
// snapshot persisted in Backup.status.underlyingResources, including the
783835
// BackupClassStrategy parameters that were in effect at backup time -

‎internal/backupcontroller/postgresapp/types.go‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -71,15 +71,20 @@ type PostgresSpec struct {
7171
}
7272

7373
type Bootstrap struct {
74-
Enabled bool `json:"enabled,omitempty"`
75-
OldName string `json:"oldName,omitempty"`
76-
ServerName string `json:"serverName,omitempty"`
77-
RecoveryTime string `json:"recoveryTime,omitempty"`
74+
Enabled bool `json:"enabled,omitempty"`
75+
OldName string `json:"oldName,omitempty"`
76+
ServerName string `json:"serverName,omitempty"`
77+
NewServerName string `json:"newServerName,omitempty"`
78+
RecoveryTime string `json:"recoveryTime,omitempty"`
7879
}
7980

8081
type Backup struct {
81-
DestinationPath string `json:"destinationPath,omitempty"`
82-
EndpointURL string `json:"endpointURL,omitempty"`
82+
DestinationPath string `json:"destinationPath,omitempty"`
83+
EndpointURL string `json:"endpointURL,omitempty"`
84+
// No omitempty: the restore patch must be able to set this back to false,
85+
// and a bool+omitempty would drop false from the merge patch, leaving the
86+
// server value at true.
87+
UseSystemBucket bool `json:"useSystemBucket"`
8388
S3AccessKey string `json:"s3AccessKey,omitempty"`
8489
S3SecretKey string `json:"s3SecretKey,omitempty"`
8590
S3CredentialsSecret S3CredentialsSecret `json:"s3CredentialsSecret,omitempty"`

0 commit comments

Comments
 (0)