Skip to content

Commit 7fc5cc0

Browse files
mnenciaarmrugbartolini
committed
fix(major-upgrade): reset TimelineID to 1 after upgrade (#9830)
After a major version upgrade, pg_upgrade creates a new database system with a new System ID and resets the timeline to 1. However, cluster.Status.TimelineID retained its old value (e.g., timeline 2), causing replicas to restore incompatible timeline history files from object storage. This resulted in PostgreSQL fatal errors: "requested timeline 2 is not a child of this server's history" "Latest checkpoint in file backup_label is at 0/13000080 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/70226C8." The fix sets cluster.Status.TimelineID to 1 after major upgrade completion, ensuring validateTimelineHistoryFile() blocks incompatible timeline history files (e.g., 00000002.history). E2E tests verify this by performing a switchover to timeline 2 before the major upgrade, then confirming TimelineID is reset to 1 after upgrade completion. Closes #9764 Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Gabriele Bartolini <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> Co-authored-by: Gabriele Bartolini <[email protected]> (cherry picked from commit a2f9d47)
1 parent eee4466 commit 7fc5cc0

6 files changed

Lines changed: 294 additions & 3 deletions

File tree

‎docs/src/postgres_upgrades.md‎

Lines changed: 69 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -135,9 +135,10 @@ If the upgrade is successful, CloudNativePG:
135135
:::warning
136136
Re-cloning replicas can be time-consuming, especially for very large
137137
databases. Plan accordingly to accommodate potential delays. After completing
138-
the upgrade, it is strongly recommended to take a full backup. Existing backup
139-
data (namely base backups and WAL files) is only available for the previous
140-
minor PostgreSQL release.
138+
the upgrade, take a new base backup as soon as possible. Pre-upgrade backups
139+
and WAL files cannot be used for point-in-time recovery (PITR) across major
140+
version boundaries. See [Backup and WAL Archive Considerations](#backup-and-wal-archive-considerations)
141+
for more details.
141142
:::
142143

143144
:::warning
@@ -156,6 +157,71 @@ automatically decide the rollback.
156157
Ensure you monitor the process closely and take corrective action if needed.
157158
:::
158159

160+
### Backup and WAL Archive Considerations
161+
162+
When performing a major upgrade, `pg_upgrade` creates a new database system
163+
with a new *System ID* and resets the PostgreSQL timeline to 1. This has
164+
implications for backup and WAL archiving:
165+
166+
- **Timeline file conflicts**: New timeline 1 files may overwrite timeline 1
167+
files from the original cluster.
168+
- **Mixed version archives**: Without intervention, the archive will contain
169+
WAL files and backups from both PostgreSQL versions.
170+
171+
:::warning
172+
Point-in-time recovery (PITR) is not supported across major PostgreSQL version
173+
boundaries. You cannot use pre-upgrade backups to recover to a point in time
174+
after the upgrade. Take a new base backup as soon as possible after upgrading
175+
to establish a recovery baseline for the new major version.
176+
:::
177+
178+
How backup systems handle major upgrades depends on the plugin implementation.
179+
Some plugins may automatically manage archive separation during upgrades, while
180+
others require manual configuration to use different archive paths for each
181+
major version. Consult your backup plugin documentation for its specific
182+
behavior during major upgrades.
183+
184+
#### Example: Manual archive path separation with the Barman Cloud plugin
185+
186+
The Barman Cloud plugin does not automatically separate archives during major
187+
upgrades. To preserve pre-upgrade backups and keep archives clean, change the
188+
`serverName` parameter when you trigger the upgrade.
189+
190+
Before upgrade (PostgreSQL 16):
191+
192+
```yaml
193+
spec:
194+
imageName: ghcr.io/cloudnative-pg/postgresql:16-minimal-trixie
195+
plugins:
196+
- name: plugin-barman-cloud
197+
enabled: true
198+
parameters:
199+
destinationPath: s3://my-bucket/
200+
serverName: cluster-example-pg16
201+
```
202+
203+
To trigger the upgrade, change both `imageName` and `serverName` together:
204+
205+
```yaml
206+
spec:
207+
imageName: ghcr.io/cloudnative-pg/postgresql:17-minimal-trixie
208+
plugins:
209+
- name: plugin-barman-cloud
210+
enabled: true
211+
parameters:
212+
destinationPath: s3://my-bucket/
213+
serverName: cluster-example-pg17
214+
```
215+
216+
With this configuration, the old archive at `cluster-example-pg16` remains
217+
intact for pre-upgrade recovery, while the upgraded cluster writes to
218+
`cluster-example-pg17`.
219+
220+
:::info
221+
The deprecated in-tree `barmanObjectStore` implementation also requires manual
222+
`serverName` changes to separate archives during major upgrades.
223+
:::
224+
159225
### Example: Performing a Major Upgrade
160226

161227
Consider the following PostgreSQL cluster running version 16:

‎pkg/reconciler/majorupgrade/reconciler.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,9 @@ func majorVersionUpgradeHandleCompletion(
237237
return nil, err
238238
}
239239

240+
// Reset timeline ID to 1 after major upgrade to match pg_upgrade behavior.
241+
// This prevents replicas from restoring incompatible timeline history files
242+
// from the pre-upgrade cluster in object storage.
240243
if err := status.PatchWithOptimisticLock(
241244
ctx,
242245
c,
@@ -245,6 +248,7 @@ func majorVersionUpgradeHandleCompletion(
245248
Image: jobImage,
246249
MajorVersion: requestedMajor,
247250
}),
251+
status.SetTimelineID(1),
248252
); err != nil {
249253
contextLogger.Error(err, "Unable to update cluster status after major upgrade completed.")
250254
return nil, err

‎pkg/reconciler/majorupgrade/reconciler_test.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,9 @@ var _ = Describe("Major upgrade job status reconciliation", func() {
6868
Spec: apiv1.ClusterSpec{
6969
ImageName: "postgres:16",
7070
},
71+
Status: apiv1.ClusterStatus{
72+
TimelineID: 5, // Simulating pre-upgrade timeline
73+
},
7174
}
7275
pvcs := []corev1.PersistentVolumeClaim{
7376
buildPrimaryPVC(1),
@@ -107,6 +110,9 @@ var _ = Describe("Major upgrade job status reconciliation", func() {
107110
Expect(cluster.Status.PGDataImageInfo.Image).To(Equal("postgres:16"))
108111
Expect(cluster.Status.PGDataImageInfo.MajorVersion).To(Equal(16))
109112

113+
// the timeline ID has been reset to 1 to match pg_upgrade behavior
114+
Expect(cluster.Status.TimelineID).To(Equal(1))
115+
110116
// the job has been deleted
111117
var tempJob batchv1.Job
112118
err = fakeClient.Get(ctx, client.ObjectKeyFromObject(job), &tempJob)

‎pkg/resources/status/transactions.go‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,3 +76,10 @@ func SetPGDataImageInfo(imageInfo *apiv1.ImageInfo) Transaction {
7676
cluster.Status.PGDataImageInfo = imageInfo
7777
}
7878
}
79+
80+
// SetTimelineID is a transaction that sets the cluster timeline ID
81+
func SetTimelineID(timelineID int) Transaction {
82+
return func(cluster *apiv1.Cluster) {
83+
cluster.Status.TimelineID = timelineID
84+
}
85+
}
Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
/*
2+
Copyright © contributors to CloudNativePG, established as
3+
CloudNativePG a Series of LF Projects, LLC.
4+
5+
Licensed under the Apache License, Version 2.0 (the "License");
6+
you may not use this file except in compliance with the License.
7+
You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
17+
SPDX-License-Identifier: Apache-2.0
18+
*/
19+
20+
package status
21+
22+
import (
23+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
24+
25+
apiv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1"
26+
27+
. "github.com/onsi/ginkgo/v2"
28+
. "github.com/onsi/gomega"
29+
)
30+
31+
var _ = Describe("Status transactions", func() {
32+
Describe("SetClusterReadyCondition", func() {
33+
It("sets ready condition to true when phase is healthy", func() {
34+
cluster := &apiv1.Cluster{
35+
Status: apiv1.ClusterStatus{
36+
Phase: apiv1.PhaseHealthy,
37+
},
38+
}
39+
40+
SetClusterReadyCondition(cluster)
41+
42+
Expect(cluster.Status.Conditions).To(HaveLen(1))
43+
condition := cluster.Status.Conditions[0]
44+
Expect(condition.Type).To(Equal(string(apiv1.ConditionClusterReady)))
45+
Expect(condition.Status).To(Equal(metav1.ConditionTrue))
46+
Expect(condition.Reason).To(Equal(string(apiv1.ClusterReady)))
47+
Expect(condition.Message).To(Equal("Cluster is Ready"))
48+
})
49+
50+
It("sets ready condition to false when phase is not healthy", func() {
51+
cluster := &apiv1.Cluster{
52+
Status: apiv1.ClusterStatus{
53+
Phase: apiv1.PhaseMajorUpgrade,
54+
},
55+
}
56+
57+
SetClusterReadyCondition(cluster)
58+
59+
Expect(cluster.Status.Conditions).To(HaveLen(1))
60+
condition := cluster.Status.Conditions[0]
61+
Expect(condition.Type).To(Equal(string(apiv1.ConditionClusterReady)))
62+
Expect(condition.Status).To(Equal(metav1.ConditionFalse))
63+
Expect(condition.Reason).To(Equal(string(apiv1.ClusterIsNotReady)))
64+
Expect(condition.Message).To(Equal("Cluster Is Not Ready"))
65+
})
66+
67+
It("initializes conditions slice if nil", func() {
68+
cluster := &apiv1.Cluster{
69+
Status: apiv1.ClusterStatus{
70+
Phase: apiv1.PhaseHealthy,
71+
Conditions: nil,
72+
},
73+
}
74+
75+
SetClusterReadyCondition(cluster)
76+
77+
Expect(cluster.Status.Conditions).ToNot(BeNil())
78+
Expect(cluster.Status.Conditions).To(HaveLen(1))
79+
})
80+
})
81+
82+
Describe("SetPhase", func() {
83+
It("sets the cluster phase and reason", func() {
84+
cluster := &apiv1.Cluster{}
85+
86+
SetPhase(apiv1.PhaseMajorUpgrade, "Upgrading to version 17")(cluster)
87+
88+
Expect(cluster.Status.Phase).To(Equal(apiv1.PhaseMajorUpgrade))
89+
Expect(cluster.Status.PhaseReason).To(Equal("Upgrading to version 17"))
90+
})
91+
92+
It("overwrites existing phase and reason", func() {
93+
cluster := &apiv1.Cluster{
94+
Status: apiv1.ClusterStatus{
95+
Phase: apiv1.PhaseHealthy,
96+
PhaseReason: "All good",
97+
},
98+
}
99+
100+
SetPhase(apiv1.PhaseUpgrade, "Rolling update")(cluster)
101+
102+
Expect(cluster.Status.Phase).To(Equal(apiv1.PhaseUpgrade))
103+
Expect(cluster.Status.PhaseReason).To(Equal("Rolling update"))
104+
})
105+
})
106+
107+
Describe("SetImage", func() {
108+
It("sets the cluster image", func() {
109+
cluster := &apiv1.Cluster{}
110+
111+
SetImage("ghcr.io/cloudnative-pg/postgresql:17.2")(cluster)
112+
113+
Expect(cluster.Status.Image).To(Equal("ghcr.io/cloudnative-pg/postgresql:17.2"))
114+
})
115+
116+
It("overwrites existing image", func() {
117+
cluster := &apiv1.Cluster{
118+
Status: apiv1.ClusterStatus{
119+
Image: "ghcr.io/cloudnative-pg/postgresql:16.5",
120+
},
121+
}
122+
123+
SetImage("ghcr.io/cloudnative-pg/postgresql:17.2")(cluster)
124+
125+
Expect(cluster.Status.Image).To(Equal("ghcr.io/cloudnative-pg/postgresql:17.2"))
126+
})
127+
})
128+
129+
Describe("SetPGDataImageInfo", func() {
130+
It("sets the PGData image info", func() {
131+
cluster := &apiv1.Cluster{}
132+
imageInfo := &apiv1.ImageInfo{
133+
Image: "ghcr.io/cloudnative-pg/postgresql:17.2",
134+
MajorVersion: 17,
135+
}
136+
137+
SetPGDataImageInfo(imageInfo)(cluster)
138+
139+
Expect(cluster.Status.PGDataImageInfo).ToNot(BeNil())
140+
Expect(cluster.Status.PGDataImageInfo.Image).To(Equal("ghcr.io/cloudnative-pg/postgresql:17.2"))
141+
Expect(cluster.Status.PGDataImageInfo.MajorVersion).To(Equal(17))
142+
})
143+
144+
It("overwrites existing PGData image info", func() {
145+
cluster := &apiv1.Cluster{
146+
Status: apiv1.ClusterStatus{
147+
PGDataImageInfo: &apiv1.ImageInfo{
148+
Image: "ghcr.io/cloudnative-pg/postgresql:16.5",
149+
MajorVersion: 16,
150+
},
151+
},
152+
}
153+
newImageInfo := &apiv1.ImageInfo{
154+
Image: "ghcr.io/cloudnative-pg/postgresql:17.2",
155+
MajorVersion: 17,
156+
}
157+
158+
SetPGDataImageInfo(newImageInfo)(cluster)
159+
160+
Expect(cluster.Status.PGDataImageInfo.Image).To(Equal("ghcr.io/cloudnative-pg/postgresql:17.2"))
161+
Expect(cluster.Status.PGDataImageInfo.MajorVersion).To(Equal(17))
162+
})
163+
})
164+
165+
Describe("SetTimelineID", func() {
166+
It("sets the cluster timeline ID", func() {
167+
cluster := &apiv1.Cluster{
168+
Status: apiv1.ClusterStatus{
169+
TimelineID: 5,
170+
},
171+
}
172+
173+
SetTimelineID(10)(cluster)
174+
Expect(cluster.Status.TimelineID).To(Equal(10))
175+
})
176+
177+
It("resets timeline ID to 1 after major upgrade", func() {
178+
cluster := &apiv1.Cluster{
179+
Status: apiv1.ClusterStatus{
180+
TimelineID: 2,
181+
},
182+
}
183+
184+
SetTimelineID(1)(cluster)
185+
Expect(cluster.Status.TimelineID).To(Equal(1))
186+
})
187+
})
188+
})

‎tests/e2e/cluster_major_upgrade_test.go‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -326,6 +326,12 @@ var _ = Describe("Postgres Major Upgrade", Label(tests.LabelPostgresMajorUpgrade
326326
}
327327
}
328328

329+
verifyTimelineResetAfterUpgrade := func(ctx context.Context, client client.Client, cluster *apiv1.Cluster) {
330+
currentCluster, err := clusterutils.Get(ctx, client, cluster.Namespace, cluster.Name)
331+
Expect(err).ToNot(HaveOccurred())
332+
Expect(currentCluster.Status.TimelineID).To(Equal(1))
333+
}
334+
329335
BeforeEach(func() {
330336
if testLevelEnv.Depth < int(level) {
331337
Skip("Test depth is lower than the amount requested for this test")
@@ -353,6 +359,17 @@ var _ = Describe("Postgres Major Upgrade", Label(tests.LabelPostgresMajorUpgrade
353359
AssertClusterIsReady(cluster.Namespace, cluster.Name, testTimeouts[timeouts.ClusterIsReady],
354360
env)
355361

362+
By("Performing switchover to move to timeline 2")
363+
AssertSwitchover(cluster.Namespace, cluster.Name, env)
364+
365+
By("Verifying cluster is on timeline 2 before upgrade")
366+
Eventually(func(g Gomega) {
367+
currentCluster, err := clusterutils.Get(env.Ctx, env.Client, cluster.Namespace, cluster.Name)
368+
g.Expect(err).ToNot(HaveOccurred())
369+
g.Expect(currentCluster.Status.TimelineID).To(Equal(2))
370+
}).WithTimeout(time.Duration(testTimeouts[timeouts.NewPrimaryAfterSwitchover]) * time.Second).
371+
WithPolling(5 * time.Second).Should(Succeed())
372+
356373
By("Collecting the pods UUIDs")
357374
podList, err := clusterutils.ListPods(env.Ctx, env.Client, cluster.Name, cluster.Namespace)
358375
Expect(err).ToNot(HaveOccurred())
@@ -420,6 +437,9 @@ var _ = Describe("Postgres Major Upgrade", Label(tests.LabelPostgresMajorUpgrade
420437
By("Verifying the cluster is running the target version")
421438
verifyPostgresVersion(env, primary, oldStdOut, scenario.targetMajor)
422439

440+
By("Verifying timeline ID is reset to 1 after major upgrade")
441+
verifyTimelineResetAfterUpgrade(env.Ctx, env.Client, cluster)
442+
423443
// Expect temporary files to be deleted
424444
By("Checking no leftovers exist from the upgrade")
425445
verifyCleanupAfterUpgrade(env.Ctx, env.Client, primary)

0 commit comments

Comments
 (0)