Skip to content

feat: added default alert case for failing replication - #2869

Merged
leonardoce merged 28 commits into
cloudnative-pg:mainfrom
Lennart01:feat/ReplicationFailingAlert
Oct 4, 2023
Merged

leonardoce merged 28 commits into
cloudnative-pg:mainfrom
Lennart01:feat/ReplicationFailingAlert

Conversation

@Lennart01

Copy link
Copy Markdown
Contributor

Hey,
as long as #2800 is in the wild I thought it would be a good idea to add a default alert case to warn the user if this edge case occurs.
The alert was tested against a broken and a working cluster and performed as expected (also tested with single replica cluster)

@github-actions github-actions Bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.19 labels Sep 25, 2023
@github-actions

Copy link
Copy Markdown
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

Comment thread docs/src/samples/monitoring/alerts.yaml Resolved
@phisco
phisco force-pushed the feat/ReplicationFailingAlert branch from aaeced1 to 111b858 Compare September 28, 2023 06:43
@phisco

phisco commented Sep 28, 2023

Copy link
Copy Markdown
Contributor

@Lennart01, looks like the DCO step is failing, can you fix that? see https://github.com/cloudnative-pg/cloudnative-pg/pull/2869/checks?check_run_id=17206660410 for more details 🙏

@Lennart01
Lennart01 force-pushed the feat/ReplicationFailingAlert branch from 111b858 to 3fe4433 Compare September 28, 2023 08:27
@Lennart01

Copy link
Copy Markdown
Contributor Author

@phisco Sorry about that. Should be fixed 👍

Lennart01 and others added 12 commits September 30, 2023 17:24
add webhook validation for `shared_buffers`

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Gabriele Bartolini <[email protected]>
Signed-off-by: Tao Li <[email protected]>
Co-authored-by: Gabriele Bartolini <[email protected]>
Co-authored-by: Tao Li <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
Signed-off-by: Tao Li <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…-pg#2870)

The connectivity test were failing due to a dependency of a previous test
that should create the proper secret with the required certificate. Now,
we create the certificate every time in the test to keep the tests
independent.

Signed-off-by: Jonathan Gonzalez V <[email protected]>
Signed-off-by: Niccolò Fei <[email protected]>
Co-authored-by: Niccolò Fei <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…native-pg#2858)

Since the introduction of phaseWaitingForInstancesToBeActive, PhaseUpgrade/PhaseFailOver
(and other phases) are now properly being replaced by the newly introduced phase right after
the related operation is complete. Because these phases get superseded very quickly, we should
not rely passing through them during the E2E tests.

Signed-off-by: Niccolò Fei <[email protected]>
Co-authored-by: Niccolò Fei <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…tency (cloudnative-pg#2875)

Addresses two inconsistencies in the `ResourceName` sections of the generated roles:
- Ensures entries are sorted, preventing items from appearing out of order.
- Removes empty items

This is achieved using a new utility function, `cleanupResourceList`.

Signed-off-by: YanniHu1996 <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…2838)

This patch enhances the status section of a volume snapshot Backup
with Postgres metadata to match the status of an object store backup.

Closes cloudnative-pg#2813

Signed-off-by: Jaime Silvela <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
….20.1 (cloudnative-pg#2694)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Lennart Krauch <[email protected]>
Signed-off-by: GitHub <[email protected]>
Co-authored-by: postgres-versions-updater <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…2850)

The continuous-integration was only actually pushing images when a PR was
labeled with "always push images", but we should also produce the images whenever
a new commit is pushed on main or a release branch.
This will ensure that the `ghcr.io/cloudnative-pg/cloudnative-pg-testing` main and release-*
images are updated as soon as a new commit is pushed (up until now it was done only
once per day via the continuous-delivery workflow)

Signed-off-by: Niccolò Fei <[email protected]>
Signed-off-by: Gabriele Bartolini <[email protected]>
Co-authored-by: Gabriele Bartolini <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
jsilvela and others added 13 commits September 30, 2023 17:24
…native-pg#2854)

Previously, the volume snapshot code could fence a node
multiple times, leading to handling errors.
The new implementation fences nodes only when necessary.

Closes cloudnative-pg#2853

Signed-off-by: Jaime Silvela <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
Adds explanations of the persistence controls for volume snapshots and
volume snapshot contents.

Closes cloudnative-pg#2881

Signed-off-by: Gabriele Bartolini <[email protected]>
Signed-off-by: Jaime Silvela <[email protected]>
Co-authored-by: Jaime Silvela <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
)

Closes: cloudnative-pg#2863

Signed-off-by: YanniHu1996 <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…ain) (cloudnative-pg#2716)

This PR contains the following updates:

sigs.k8s.io/controller-runtime `v0.15.1` -> `v0.16.2`

Signed-off-by: Jonathan Gonzalez V <[email protected]>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Jonathan Gonzalez V <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
Minor changes to the main README and FAQ (which now includes a list of
other operators for PostgreSQL).

Signed-off-by: Gabriele Bartolini <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Jaime Silvela <[email protected]>
Co-authored-by: Philippe Scorsolini <[email protected]>
Co-authored-by: Jaime Silvela <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…oudnative-pg#2888)

Ensure that `standard_conforming_strings` is set to `on` in every
Prometheus connection transaction for metric collection. This is just in
case users disable that globally.

Closes cloudnative-pg#2887

Signed-off-by: Gabriele Bartolini <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…erator/pkg/apis/monitoring to v0.68.0 (main) (cloudnative-pg#2715)

This PR contains the following updates:

https://github.com/prometheus-operator/prometheus-operator `v0.67.1` -> `v0.68.0`

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Lennart Krauch <[email protected]>
…loudnative-pg#2630)

Updates the endpointURL for the Linode example in object_stores.md so
that it does not include the bucket name and moves the bucket name into
the S3 URL provided to destinationPath.

This is based on the recommendations in the current Linode documentation
at:

https://www.linode.com/docs/products/storage/object-storage/guides/urls/#cluster-url-s3-endpoint

Closes cloudnative-pg#2629

Signed-off-by: Michael Wallace <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…e-pg#2885)

Aligns the backup names created by the plugin's backup command and
by scheduled backups for clarity.

Closes cloudnative-pg#2883

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Jaime Silvela <[email protected]>
Co-authored-by: Niccolò Fei <[email protected]>
Co-authored-by: Jaime Silvela <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
Warning against using `kubectl delete pod --grace-period=0`.

Closes cloudnative-pg#2910

Signed-off-by: Gabriele Bartolini <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Jaime Silvela <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
…ve-pg#2879)

After the recovery process from a volume snapshot completes,
we need to cleanup the content of PGDATA, similarly to what
backup tools like Barman do. This is the list of files that we
remove from PGDATA:

- core.*
- pgsql_tmp*
- postgresql.auto.conf.tmp
- current_logfiles.tmp
- pg_internal.init
- postmaster.pid
- postmaster.opts
- recovery.conf
- recovery.signal
- standby.signal
- log/*
- pg_dynshmem/*
- pg_log/*
- pg_notify/*
- pg_replslot/*
- pg_serial/*
- pg_snapshots/*
- pg_stat_tmp/*
- pg_subtrans/*

Closes cloudnative-pg#2865

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Gabriele Bartolini <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Gabriele Bartolini <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Lennart Krauch <[email protected]>
@Lennart01
Lennart01 force-pushed the feat/ReplicationFailingAlert branch from 29e8f04 to 4d96e9c Compare September 30, 2023 15:24
@leonardoce leonardoce self-assigned this Oct 4, 2023
@leonardoce
leonardoce merged commit 9fa4330 into cloudnative-pg:main Oct 4, 2023
cnpg-bot pushed a commit that referenced this pull request Oct 4, 2023
Signed-off-by: Lennart Krauch <[email protected]>
(cherry picked from commit 9fa4330)
cnpg-bot pushed a commit that referenced this pull request Oct 4, 2023
Signed-off-by: Lennart Krauch <[email protected]>
(cherry picked from commit 9fa4330)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases

Projects

None yet

Development

Successfully merging this pull request may close these issues.