Repository navigation
Commit 324709c
fix(upgrade): break config uniformity deadlock during operator upgrades (#10342)
When upgrading from an operator version that includes cnpg.* parameters
in the PostgreSQL configuration hash to one that excludes them (change
introduced in #8868), pods running the old and new operator end up
computing different hashes for the same effective configuration. The
uniformity check in reconcilePods then blocks indefinitely, waiting for
all pods to report the same hash, but that can never happen because the
remaining pods still use the old algorithm. Neither the rolling update
nor the in-place instance manager upgrade can run, since both live
behind the blocked check.
The fix introduces `isConfigNonUniformityFromUpgrade()`, which groups
pods by their operator version and checks whether the hash split aligns
exactly with the version split. If every pod running the same operator
version agrees on its hash, but different versions disagree, the
non-uniformity is structural and will never self-resolve. In that case
we skip the wait and let `handleRollingUpdate` proceed.
The operator version key combines both the bootstrap controller image
(changes on rolling updates) and the executable hash (changes on
in-place upgrades), so the deadlock is detected regardless of the
upgrade strategy.
When the non-uniformity is from normal config propagation (pods on the
same operator version reporting different hashes), the function returns
false and the existing wait-for-uniformity behavior is preserved. This
keeps the fix scoped strictly to the operator upgrade deadlock scenario,
with no change in behavior during normal operations.
Closes: #10348
Assisted-by: Claude
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>1 parent 145a109 commit 324709c
3 files changed
Lines changed: 169 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1088 | 1088 | | |
1089 | 1089 | | |
1090 | 1090 | | |
1091 | | - | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
1092 | 1095 | | |
1093 | | - | |
1094 | | - | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
1095 | 1104 | | |
1096 | | - | |
1097 | 1105 | | |
1098 | 1106 | | |
1099 | 1107 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
336 | 336 | | |
337 | 337 | | |
338 | 338 | | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
339 | 380 | | |
340 | 381 | | |
341 | 382 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
820 | 820 | | |
821 | 821 | | |
822 | 822 | | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
823 | 939 | | |
824 | 940 | | |
825 | 941 | | |
| |||
0 commit comments