Repository navigation
Replies: 3 comments 9 replies
|
Can you please share the startDelay, stopDelay and switchoverDelay options you have? |
|
Do you mind retrying - if you can recreate it - with the proposed new default values and provide feedback? See #2835 for details. |
values used for reference. Will try |
|
Im still able to reproduce this |
|
Can you please try reducing Then, also try again with the original spec, but without the WAL archive? The symptom you are seeing (file based replication) is just an effect, I don't think it is the cause (the replica cannot connect so Postgres retries indefinitely to fetch from the archive). |
|
Not proposing this as a fix, just curious to see if setting a small failoverDelay changes the outcome. |
|
Im still able to reproduce this behaviour.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-cluster
namespace: test
spec:
imageName: ghcr.io/cloudnative-pg/postgresql:15.1
instances: 3
minSyncReplicas: 2
maxSyncReplicas: 2
stopDelay: 1800
switchoverDelay: 3600
startDelay: 3600
storage:
storageClass: <your-storage-class>
size: 10Gi
affinity:
enablePodAntiAffinity: true
topologyKey: failure-domain.beta.kubernetes.io/zone
replicationSlots:
highAvailability:
enabled: true
updateInterval: 10
backup:
retentionPolicy: "30d"
barmanObjectStore:
destinationPath: s3://<your_s3_url>
endpointURL: https://<your_s3_endpoint>
s3Credentials:
accessKeyId:
name: s3-secret
key: ACCESS_KEY_ID
secretAccessKey:
name: s3-postgres-backup
key: SECRET_ACCESS_KEY
postgresql:
parameters:
shared_buffers: 256MB
max_connections: '200'
bootstrap:
initdb:
database: test_db
owner: test_user
secret:
name: postgresql-creds
import subprocess
DB_URL = "postgres://user:pw@localhost:5432/change-me"
def main():
query = "\dt"
result = subprocess.check_output(f"kubectl exec --namespace=change-me -it change-me-postgres-cluster-1 -- bash -c \"psql {DB_URL} -c '{query}'\"", shell=True)
if "test2" not in result.decode("utf-8"):
print("Database is empty")
print("creating test tables")
query = "CREATE TABLE test2 (id serial PRIMARY KEY, num integer);"
subprocess.call(f"kubectl exec --namespace=change-me -it change-me-postgres-cluster-1 -- bash -c \"psql {DB_URL} -c '{query}'\"", shell=True)
print("Database populated")
i = 0
while True:
query = f"INSERT INTO test(num) VALUES ({i});"
full_query = f"kubectl exec --namespace=change-me -it change-me-postgres-cluster-1 -- bash -c \"psql {DB_URL} -c '{query}'\""
print(full_query)
subprocess.call(f"kubectl exec --namespace=change-me -it change-me-postgres-cluster-1 -- bash -c \"psql {DB_URL} -c '{query}'\"", shell=True)
i += 1
print(f"Inserted {i} rows")
if __name__ == "__main__":
main()
kubectl delete -n change-me pod change-me-postgres-cluster-1 --grace-period=0 --forceNow one replica should become "file-based" and will loop wal recovery |
|
I guess this occurs because the primary is unable to write its WAL fully to the s3. The elected primary notices this and automatically corrects this mistake by discarding the missing WAL and changing the timeline. What is odd is that in this case the replica looping the wal recovery is the "old" primary. In my last 2 encounters this issue appeared with another replica 🤔. |
|
Can you try with? A grace period of 0 guarantees that the pod is deleted from the Kubernetes API server, but does not guarantee that the PID 1 process is terminated. So the operator doesn't see the pod, but the Postgres process might be still alive (this would not happen in case of a real failover). See: https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/ |
|
it does indeed work with |
|
I ran into a similar error. I suspect, that it was caused by a k3s upgrade plan where drain:-force was set. Can a |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hey,
as outlined in #2800 if a replica is not able to replicate due to a timeline change it will loop WAL recovery but does not crash.
The Operator considers this replica as healthy as it is technically accepting connections and is running.
As documented in the codebase
proposals:
I'd love to get your thoughts on this and I'd be happy to help implement them :D.
All reactions