Skip to content

Commit 18779dd

Browse files
fix(backups): report sizeBytes for MariaDB and CNPG backup artifacts (#5223)
## What this PR does Fixes #4316. Continues #5015 by androndo on a separate branch. All ten of their commits are kept as they were (authorship, messages and order, only rebased on current `main`). MariaDB and CNPG backups stored in S3 report `Backup.status.artifact.sizeBytes`, the size of the dump mariadb-operator uploaded or the total of the barman base backup, and the field stays unset when the size cannot be attributed to that one backup. Four commits are added on top: - `test(backups)`: operator Backup fixtures carry the condition mariadb-operator writes on success, `Complete=True` with reason `JobComplete` and message `Success`. Operator sets `Complete=True` for a failed Job too (reason `JobFailed`), so a fixture without reason describes a state that never happens. #5207 makes the driver check that reason, the tests pass with and without it. - `refactor(backups)`: the MariaDB matcher returns the matched dump (key, file name, timestamp from the name, size, LastModified) and tells "not found" from "ambiguous", and the Secret lookup and S3 client live on a small value any reconciler can build from its own client. A MariaDB restore needs this to restore the dump of the Backup it was given and not the newest one under the prefix. `sizeBytes` does not change here. - `fix(backups)`: the probe reads bucket, prefix, endpoint, credentials and TLS from the operator Backup's own `spec.storage.s3`, not from the strategy as it renders now, so editing the strategy during a backup does not send the probe somewhere else. - `fix(backups)`: a dump is attributed only when its ownership is proven. Before, a backup pod with a clock more than a minute behind, right after a previous run, got that run's dump size (666 bytes for a 123-byte dump in a test). Now the dump has to be the only one that either clock places in the operator Backup's lifetime, both clocks have to agree, and neither may place it in the lifetime of another finished operator Backup with the same bucket and prefix. Otherwise the size stays unset. Two cases are still not covered. First, this run's own dump is missing from the listing while the operator Backup of a neighbour was deleted or has not finished, then the neighbour's dump can be taken. Second, backup pod clock and object store clock are both off the same way by more than the minute of slack, which moves a finished neighbour's dump into this run's window, and nothing the clocks record tells them apart. A run whose Job failed has no dump of its own and reaches the probe only because the driver takes `Complete=True` for success, #5207 fixes that. Of the open threads on the original PR, the cache reads, the dump file name layout and the default AWS endpoint test were already handled by its later commits, the clock skew one is the last commit here and the failed Job case is #5207. `minio-go` is new to this tree, this PR adds it as a direct dependency and `internal/backupcontroller/artifact_size.go` is its only importer. The open Bucket strategy PR #4235 picks the same client at the same version. Each clock skew case, and a run whose own dump is gone while the neighbour's operator Backup still exists and has finished, have a unit test that fails on the old matcher, and every new guard has a mutant that turns a test red. I also ran the matcher against a real bucket after four MariaDB backups in a row. Runs that finished more than a minute apart mapped to the dump their operator Job logged, the two back to back came back unset instead of taking each other's dump, and with a run's own dump hidden from the listing the matcher refused the neighbour's dump that the old one returned. ### Screenshots N/A, no UI changes. ### Downstream repositories The only API change is the description of the existing `Backup.status.artifact.sizeBytes` field, no new field, value, default or enum, no package, values or tooling change, and neither cozystack/website nor cozystack/terraform-provider-cozystack references `sizeBytes`. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): MariaDB and CNPG (Postgres) backups stored in S3 now report `status.artifact.sizeBytes`: the size of the dump mariadb-operator uploaded, or the total size of the barman base backup (WAL archive excluded). A MariaDB backup takes the size of the one dump that falls inside its run on both the backup pod's and the object store's clock and inside no other finished run, and the size is left unset rather than set to 0 whenever it cannot be measured or attributed. Velero-based backups intentionally keep it unset. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup artifacts now report the stored size attributable to an individual MariaDB or CNPG backup when it can be determined. Shared or ambiguous storage may leave the size unset. * **Bug Fixes** * Improved handling of backup artifact creation and lookup errors, including retrying when an existing backup cannot be read and failing jobs when artifact creation is rejected. * **Documentation** * Clarified that an unset size means unknown; zero does not indicate an unknown size. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents e5bb3cd + 59f0013 commit 18779dd

14 files changed

Lines changed: 2174 additions & 10 deletions

‎api/backups/v1alpha1/backup_types.go‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,14 @@ type BackupArtifact struct {
3737
// For example: s3://bucket/prefix/file.tar.gz
3838
URI string `json:"uri"`
3939

40-
// SizeBytes is the size of the artifact in bytes, if known.
40+
// SizeBytes is the stored size in bytes of the files this backup owns,
41+
// if known: the dump of a MariaDB backup, for example, or the base backup
42+
// of a CNPG one, which leaves out the WAL archive the server's backups
43+
// share and a point-in-time restore also reads. Strategies that cannot
44+
// attribute stored bytes to a single backup, such as Velero's, whose
45+
// repository deduplicates across backups, leave it unset. Drivers never
46+
// write 0 for a size they cannot measure, so an absent value means
47+
// "unknown", not "empty".
4148
// +optional
4249
SizeBytes int64 `json:"sizeBytes,omitempty"`
4350

‎go.mod‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ require (
1616
github.com/go-logr/logr v1.4.4
1717
github.com/go-logr/zapr v1.3.0
1818
github.com/go-task/slim-sprig/v3 v3.0.0
19+
github.com/minio/minio-go/v7 v7.0.98
1920
github.com/onsi/ginkgo/v2 v2.33.0
2021
github.com/onsi/gomega v1.43.1
2122
github.com/prometheus/client_golang v1.24.1
@@ -63,6 +64,7 @@ require (
6364
github.com/fsnotify/fsnotify v1.10.1 // indirect
6465
github.com/fxamacker/cbor/v2 v2.9.4 // indirect
6566
github.com/gkampitakis/go-snaps v0.5.23 // indirect
67+
github.com/go-ini/ini v1.67.0 // indirect
6668
github.com/go-logr/stdr v1.2.2 // indirect
6769
github.com/go-openapi/jsonpointer v1.0.1 // indirect
6870
github.com/go-openapi/jsonreference v1.0.2 // indirect
@@ -93,18 +95,25 @@ require (
9395
github.com/inconshreveable/mousetrap v1.1.0 // indirect
9496
github.com/json-iterator/go v1.1.12 // indirect
9597
github.com/klauspost/compress v1.20.0 // indirect
98+
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
99+
github.com/klauspost/crc32 v1.3.0 // indirect
96100
github.com/kylelemons/godebug v1.1.0 // indirect
101+
github.com/minio/crc64nvme v1.1.1 // indirect
102+
github.com/minio/md5-simd v1.1.2 // indirect
97103
github.com/moby/spdystream v0.5.1 // indirect
98104
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
99105
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
100106
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
107+
github.com/philhofer/fwd v1.2.0 // indirect
101108
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
102109
github.com/prometheus/client_model v0.6.3 // indirect
103110
github.com/prometheus/common v0.71.0 // indirect
104111
github.com/prometheus/procfs v0.22.0 // indirect
105112
github.com/rogpeppe/go-internal v1.16.0 // indirect
113+
github.com/rs/xid v1.6.0 // indirect
106114
github.com/sirupsen/logrus v1.10.2 // indirect
107115
github.com/tidwall/match v1.2.0 // indirect
116+
github.com/tinylib/msgp v1.6.1 // indirect
108117
github.com/x448/float16 v0.8.4 // indirect
109118
go.etcd.io/etcd/api/v3 v3.7.0 // indirect
110119
go.etcd.io/etcd/client/pkg/v3 v3.7.0 // indirect

‎go.sum‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ github.com/gkampitakis/ciinfo v0.3.4 h1:5eBSibVuSMbb/H6Elc0IIEFbkzCJi3lm94n0+U7Z
6161
github.com/gkampitakis/ciinfo v0.3.4/go.mod h1:1NIwaOcFChN4fa/B0hEBdAb6npDlFL8Bwx4dfRLRqAo=
6262
github.com/gkampitakis/go-snaps v0.5.23 h1:okh5QR48zpUjpWtu65AtqxdCY8huJq+dEDuUzd1PuKg=
6363
github.com/gkampitakis/go-snaps v0.5.23/go.mod h1:uy3lVzCCRRsAwYqSocyw5fY8xRLCYEfqoOJNxr8HonM=
64+
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
65+
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
6466
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
6567
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
6668
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -141,6 +143,11 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
141143
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
142144
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
143145
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
146+
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
147+
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
148+
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
149+
github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM=
150+
github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw=
144151
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
145152
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
146153
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -151,6 +158,12 @@ github.com/maruel/natural v1.3.0 h1:VsmCsBmEyrR46RomtgHs5hbKADGRVtliHTyCOLFBpsg=
151158
github.com/maruel/natural v1.3.0/go.mod h1:v+Rfd79xlw1AgVBjbO0BEQmptqb5HvL/k9GRHB7ZKEg=
152159
github.com/mfridman/tparse v0.18.0 h1:wh6dzOKaIwkUGyKgOntDW4liXSo37qg5AXbIhkMV3vE=
153160
github.com/mfridman/tparse v0.18.0/go.mod h1:gEvqZTuCgEhPbYk/2lS3Kcxg1GmTxxU7kTC8DvP0i/A=
161+
github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
162+
github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
163+
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
164+
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
165+
github.com/minio/minio-go/v7 v7.0.98 h1:MeAVKjLVz+XJ28zFcuYyImNSAh8Mq725uNW4beRisi0=
166+
github.com/minio/minio-go/v7 v7.0.98/go.mod h1:cY0Y+W7yozf0mdIclrttzo1Iiu7mEf9y7nk2uXqMOvM=
154167
github.com/moby/spdystream v0.5.1 h1:9sNYeYZUcci9R6/w7KDaFWEWeV4LStVG78Mpyq/Zm/Y=
155168
github.com/moby/spdystream v0.5.1/go.mod h1:xBAYlnt/ay+11ShkdFKNAG7LsyK/tmNBVvVOwrfMgdI=
156169
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -165,6 +178,8 @@ github.com/onsi/ginkgo/v2 v2.33.0 h1:C8gBA6Uc2ZEubiV+SXiu5tZnMTwEmXHgkJwGozKtZf8
165178
github.com/onsi/ginkgo/v2 v2.33.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44=
166179
github.com/onsi/gomega v1.43.1 h1:vGIPFuYrIO6/0Z09s0I0QQQgFchiX4+tb1re3MScJYo=
167180
github.com/onsi/gomega v1.43.1/go.mod h1:e/C2HwaZ1DhvjzXXuFhcR7hY7Sh9pl7MmoWKEjzwcdA=
181+
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
182+
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
168183
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
169184
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
170185
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
@@ -180,6 +195,8 @@ github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
180195
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
181196
github.com/rogpeppe/go-internal v1.16.0 h1:O9DK+vNMDVGLr2BeZqmpLeMjiMNkuXfcqntWbZV6S5g=
182197
github.com/rogpeppe/go-internal v1.16.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
198+
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
199+
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
183200
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
184201
github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw=
185202
github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
@@ -206,6 +223,8 @@ github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
206223
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
207224
github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY=
208225
github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28=
226+
github.com/tinylib/msgp v1.6.1 h1:ESRv8eL3u+DNHUoSAAQRE50Hm162zqAnBoGv9PzScPY=
227+
github.com/tinylib/msgp v1.6.1/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
209228
github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 h1:6fotK7otjonDflCTK0BCfls4SPy3NcCVb5dqqmbRknE=
210229
github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75/go.mod h1:KO6IkyS8Y3j8OdNO85qEYBsRPuteD+YciPomcXdrMnk=
211230
github.com/vmware-tanzu/velero v1.18.3 h1:9uwN5yxTWQl9PsF5ljX4xiG+7VIq3I9mxOQ0DyyZ31A=
Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
package backupcontroller
3+
4+
import (
5+
"context"
6+
"crypto/x509"
7+
"errors"
8+
"fmt"
9+
"time"
10+
11+
"github.com/minio/minio-go/v7"
12+
"github.com/minio/minio-go/v7/pkg/credentials"
13+
corev1 "k8s.io/api/core/v1"
14+
apierrors "k8s.io/apimachinery/pkg/api/errors"
15+
apimeta "k8s.io/apimachinery/pkg/api/meta"
16+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
17+
"k8s.io/apimachinery/pkg/types"
18+
ctrl "sigs.k8s.io/controller-runtime"
19+
"sigs.k8s.io/controller-runtime/pkg/client"
20+
21+
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
22+
)
23+
24+
// artifactSizeProbeTimeout bounds the object-store round trips a driver
25+
// makes to measure Backup.status.artifact.sizeBytes. The probe runs inside
26+
// the reconcile that creates the Backup, and the BackupJob controller has a
27+
// single worker, so an unreachable endpoint stalls every other BackupJob for
28+
// this long. A variable so tests can shorten it.
29+
var artifactSizeProbeTimeout = 10 * time.Second
30+
31+
type s3ObjectInfo struct {
32+
Key string
33+
Size int64
34+
LastModified time.Time
35+
}
36+
37+
type s3ObjectLister interface {
38+
ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error)
39+
}
40+
41+
// s3Connection is what a driver resolves from its rendered strategy to
42+
// read the bucket with the same identity the backup was written with.
43+
type s3Connection struct {
44+
// Endpoint is host[:port], without a scheme.
45+
Endpoint string
46+
Secure bool
47+
Region string
48+
AccessKey string
49+
SecretKey string
50+
SessionToken string
51+
// CAPEM is an endpoint CA bundle. It is trusted on top of the system
52+
// roots, or instead of them when CAOnly is set: the probe must trust
53+
// what the writer of the backup trusted, no more.
54+
CAPEM []byte
55+
CAOnly bool
56+
}
57+
58+
// objectStore reads back what a backup wrote: the Secrets it was written
59+
// with and the bucket itself. It is a value rather than reconciler methods
60+
// so that the restore path, which has a reconciler of its own, can read
61+
// the same objects.
62+
type objectStore struct {
63+
// reader reads Secrets, which the manager's client fetches uncached,
64+
// and lists operator Backups, which it serves from the cache.
65+
reader client.Reader
66+
// newLister overrides the S3 client; nil means a real minio client.
67+
newLister func(conn s3Connection) (s3ObjectLister, error)
68+
}
69+
70+
func (s objectStore) lister(conn s3Connection) (s3ObjectLister, error) {
71+
if s.newLister != nil {
72+
return s.newLister(conn)
73+
}
74+
return newMinioObjectLister(conn)
75+
}
76+
77+
func (s objectStore) secretValue(ctx context.Context, namespace, name, key string) (string, error) {
78+
secret := &corev1.Secret{}
79+
if err := s.reader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil {
80+
return "", err
81+
}
82+
v, ok := secret.Data[key]
83+
if !ok {
84+
return "", fmt.Errorf("secret %s/%s has no key %q", namespace, name, key)
85+
}
86+
return string(v), nil
87+
}
88+
89+
func (r *BackupJobReconciler) objectStore() objectStore {
90+
return objectStore{reader: r.Client, newLister: r.newS3ObjectLister}
91+
}
92+
93+
func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) {
94+
return r.objectStore().lister(conn)
95+
}
96+
97+
type minioObjectLister struct {
98+
client *minio.Client
99+
}
100+
101+
func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) {
102+
opts := &minio.Options{
103+
Creds: credentials.NewStaticV4(conn.AccessKey, conn.SecretKey, conn.SessionToken),
104+
Region: conn.Region,
105+
Secure: conn.Secure,
106+
}
107+
if conn.Secure && len(conn.CAPEM) > 0 {
108+
// Start from minio's transport to keep its proxy and timeout settings.
109+
tr, err := minio.DefaultTransport(true)
110+
if err != nil {
111+
return nil, err
112+
}
113+
if tr.TLSClientConfig.RootCAs, err = endpointTrustPool(conn, tr.TLSClientConfig.RootCAs); err != nil {
114+
return nil, err
115+
}
116+
opts.Transport = tr
117+
}
118+
// minio.New refuses an endpoint with a path prefix, which it cannot
119+
// address, and accepts a bare trailing slash.
120+
client, err := minio.New(conn.Endpoint, opts)
121+
if err != nil {
122+
return nil, err
123+
}
124+
return &minioObjectLister{client: client}, nil
125+
}
126+
127+
// endpointTrustPool returns the roots to verify the endpoint against: the
128+
// CA bundle on top of system, or the CA bundle alone with CAOnly. A nil
129+
// system means the platform's roots.
130+
func endpointTrustPool(conn s3Connection, system *x509.CertPool) (*x509.CertPool, error) {
131+
pool := system
132+
switch {
133+
case conn.CAOnly:
134+
pool = x509.NewCertPool()
135+
case pool == nil:
136+
var err error
137+
if pool, err = x509.SystemCertPool(); err != nil {
138+
pool = x509.NewCertPool()
139+
}
140+
}
141+
if !pool.AppendCertsFromPEM(conn.CAPEM) {
142+
return nil, errors.New("no PEM certificates in the endpoint CA bundle")
143+
}
144+
return pool, nil
145+
}
146+
147+
func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) {
148+
return collectObjects(ctx, l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive}))
149+
}
150+
151+
// collectObjects drains a minio listing. minio-go ends a listing whose
152+
// context is cancelled between pages by closing the channel without an
153+
// error item, so a cancelled context is checked once the channel closes:
154+
// otherwise a timeout would pass a partial listing off as complete.
155+
func collectObjects(ctx context.Context, objects <-chan minio.ObjectInfo) ([]s3ObjectInfo, error) {
156+
var out []s3ObjectInfo
157+
for obj := range objects {
158+
if obj.Err != nil {
159+
return nil, obj.Err
160+
}
161+
out = append(out, s3ObjectInfo{Key: obj.Key, Size: obj.Size, LastModified: obj.LastModified})
162+
}
163+
if err := ctx.Err(); err != nil {
164+
return nil, err
165+
}
166+
return out, nil
167+
}
168+
169+
// errBackupNotReadBack reports that Create collided with an existing Backup
170+
// that could then not be read. The backup itself succeeded, so callers
171+
// retry the reconcile instead of failing the BackupJob.
172+
var errBackupNotReadBack = errors.New("the Backup already exists but could not be read back")
173+
174+
// failBackupArtifact handles an error from creating the Backup of a
175+
// completed backup: errBackupNotReadBack is retried through the controller's
176+
// backoff, with Ready=False telling why the BackupJob is still Running;
177+
// anything else fails the BackupJob.
178+
func (r *BackupJobReconciler) failBackupArtifact(ctx context.Context, j *backupsv1alpha1.BackupJob, err error) (ctrl.Result, error) {
179+
if errors.Is(err, errBackupNotReadBack) {
180+
apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
181+
Type: "Ready",
182+
Status: metav1.ConditionFalse,
183+
Reason: "BackupNotReadBack",
184+
Message: err.Error(),
185+
})
186+
if uerr := r.Status().Update(ctx, j); uerr != nil {
187+
getLogger(ctx).Info("cannot record that the Backup could not be read back", "backupjob", j.Name, "error", uerr)
188+
}
189+
return ctrl.Result{}, err
190+
}
191+
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to create Backup artifact: %v", err))
192+
}
193+
194+
// existingBackup returns the Backup a previous reconcile already created
195+
// for the job, so a retry skips the size probe whose result it would
196+
// discard. It reads past the cache: a retry typically follows a Create made
197+
// moments earlier, which the cache may not show yet. A failed read is not
198+
// fatal, because the Create that follows handles an existing Backup on its
199+
// own; probe is false then, since the Backup may well exist and a probe
200+
// would only be discarded, while a read that keeps failing would rerun it on
201+
// every retry.
202+
func (r *BackupJobReconciler) existingBackup(ctx context.Context, namespace, name string) (existing *backupsv1alpha1.Backup, probe bool) {
203+
existing = &backupsv1alpha1.Backup{}
204+
if err := r.apiReader().Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, existing); err != nil {
205+
if apierrors.IsNotFound(err) {
206+
return nil, true
207+
}
208+
getLogger(ctx).Info("cannot check for an existing Backup; creating it without measuring its size", "backup", name, "error", err)
209+
return nil, false
210+
}
211+
return existing, false
212+
}
213+
214+
func (r *BackupJobReconciler) secretValue(ctx context.Context, namespace, name, key string) (string, error) {
215+
return r.objectStore().secretValue(ctx, namespace, name, key)
216+
}

0 commit comments

Comments
 (0)