Skip to content

Make cluster meet reliable under link failures#461

Merged
madolson merged 11 commits intovalkey-io:unstablefrom
srgsanky:unstable
Jun 17, 2024
Merged

Make cluster meet reliable under link failures#461
madolson merged 11 commits intovalkey-io:unstablefrom
srgsanky:unstable

Conversation

@srgsanky
Copy link
Contributor

@srgsanky srgsanky commented May 8, 2024

When there is a link failure while an ongoing MEET request is sent the sending node stops sending anymore MEET and starts sending PINGs. Since every node responds to PINGs from unknown nodes with a PONG, the receiving node never adds the sending node. But the sending node adds the receiving node when it sees a PONG. This can lead to asymmetry in cluster membership. This changes makes the sender keep sending MEET until it sees a PONG, avoiding the asymmetry.

@srgsanky
Copy link
Contributor Author

srgsanky commented May 8, 2024

Posting this for initial comments. I can migrate the test based on the new framework once #442 is merged.

@codecov
Copy link

codecov bot commented May 8, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.20%. Comparing base (168da8b) to head (7ac84b6).
Report is 33 commits behind head on unstable.

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable     #461      +/-   ##
============================================
- Coverage     70.22%   70.20%   -0.02%     
============================================
  Files           109      109              
  Lines         59956    59967      +11     
============================================
- Hits          42104    42102       -2     
- Misses        17852    17865      +13     
Files Coverage Δ
src/cluster_legacy.c 86.52% <100.00%> (+0.07%) ⬆️
src/debug.c 54.07% <100.00%> (+0.13%) ⬆️

... and 9 files with indirect coverage changes

@srgsanky
Copy link
Contributor Author

srgsanky commented May 8, 2024

@madolson @hpatro @PingXie can one of you help review this change?

@hpatro
Copy link
Contributor

hpatro commented May 13, 2024

I think it's worth investing on this redis/redis#11095 to avoid this issue altogether.

@srgsanky
Copy link
Contributor Author

I think it's worth investing on this redis/redis#11095 to avoid this issue altogether.

Thanks, I wasn't aware of this linked issue. IMO these two issues can be solved independently. The linked issue tries to make the admin experience better for MEET command where as this PR tries to address a specific gap in MEET implementation.

  • With SYNC MEET, we will have to make changes to admin client timeout. This timeout can possible trickle up the stack in a control plane implementation.
  • If we choose to attempt handshake for a longer period of time, we either have to filter out nodes in handshake in cluster nodes output for a non-admin client or make the clients filter out the nodes with this new flag. This can require a client side change to avoid connecting to a node in handshake, experiencing availability issues.

The problem addressed in this PR (asymmetric cluster membership) can happen with SYNC MEET as well due to link failures. So, it is worth solving it. The handshake nodes will still be removed after the handshake timeout (same as node_timeout of 15s). Wdyt?

@madolson
Copy link
Member

The problem addressed in this PR (asymmetric cluster membership) can happen with SYNC MEET as well due to link failures. So, it is worth solving it. The handshake nodes will still be removed after the handshake timeout (same as node_timeout of 15s). Wdyt?

Yeah, I still believe this a problem even with the #11095.

@zuiderkwast
Copy link
Contributor

Awesome material for our next release which will be full of cluster improvements. Is it worth mentioning in release notes?

Btw @srgsanky you need to commit with -s. See the instructions on the DCO CI job's details page.

@madolson madolson added the release-notes This issue should get a line item in the release notes label May 13, 2024
@madolson
Copy link
Member

Awesome material for our next release which will be full of cluster improvements. Is it worth mentioning in release notes?

I would also be inclined to backport it.

@srgsanky
Copy link
Contributor Author

Awesome material for our next release which will be full of cluster improvements. Is it worth mentioning in release notes?

Btw @srgsanky you need to commit with -s. See the instructions on the DCO CI job's details page.

When I tried to merge the new changes into my fork, I ended up with a merge commit

* 2ff9879fa (HEAD -> unstable, origin/unstable, origin/HEAD) Moved test under unit and addressed other comments
*   b826ef77a Merge branch 'valkey-io:unstable' into unstable
|\
| * d52c8f30e Include stddef in zmalloc.h (#516)
| * dcc9fd4fe Resolve numtests counter error (#514)
...
| * 315b7573c Update server function's name to valkey (#456)
* | 49a884c06 Make cluster meet reliable under link failures
|/
* 4e944cede Migrate kvstore.c unit tests to new test framework. (#446)

I want to signoff just 49a884c, but the rebase is adding a signoff to all commits 315b757..d52c8f3 which are not made by me.

Do you have any recommendation to fix this?

As an alternate option, I can start fresh and add a new commit from the tip of unstable. I am not sure if I will be able to reuse this PR.

@srgsanky srgsanky force-pushed the unstable branch 2 times, most recently from d8aa71c to 2ff9879 Compare May 19, 2024 21:22
@zuiderkwast
Copy link
Contributor

I believe it's possible to undo a merge by git reset --hard 49a884c06 (the commit before the merge commit), then rebase to add the --signoff, then do git merge unstable again. The commit you added after merge commit can be cherry-picked after all this. Just remember the commit id.

If nothing works, then it's always possible to start from scratch with a new branch and cherry-pick all your commits into it. Then you can rename the branches and force-push to this PR's branch.

@madolson
Copy link
Member

madolson commented May 20, 2024

@srgsanky The commit missing the DCO is just the top one. You should just be able to do git commit -s --amend with a no-op and force push over what you have. It's the base commit, nevermind. I believe git rebase -i HEAD~3 should allow you to manually add the signature, maybe there is a better way to do it.

Copy link
Member

@madolson madolson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor nitpicks around the tests, it overall LGTM.

srgsanky added 3 commits May 19, 2024 21:07
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

Signed-off-by: Sankar <[email protected]>
@srgsanky
Copy link
Contributor Author

B isn't processing these right? It's just immediately dropping the first three and not processing them, it only ever processes the 4th one once correct?

Correct. It starts processing when we drop the filter - which can be 4th or later.

@srgsanky
Copy link
Contributor Author

I believe it's possible to undo a merge by git reset --hard 49a884c (the commit before the merge commit), then rebase to add the --signoff, then do git merge unstable again. The commit you added after merge commit can be cherry-picked after all this. Just remember the commit id.

This worked. Thanks!

It's the base commit, nevermind. I believe git rebase -i HEAD~3 should allow you to manually add the signature, maybe there is a better way to do it.

I tried this and all the commits in the other branch of the merge was also annotated with my signoff. So, I decided to ask you folks for the best approach.

btw is there any reasoning behind the requirement for the signoff?

@madolson
Copy link
Member

btw is there any reasoning behind the requirement for the signoff?

Technically we adopted it because it's an LF requirement, but it's also a good practice to force a trail of who committed what.

Copy link
Member

@PingXie PingXie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change LGTM overall.

srgsanky added 5 commits May 20, 2024 19:15
1. Reworked code comment
1. Added serverLogs
1. Renamed debug variable
1. Made close link filter to be directly coupled with drop filter

Signed-off-by: Sankar <[email protected]>
Multiple MEETs will be handled like a normal PING message.

Signed-off-by: Sankar <[email protected]>
@srgsanky
Copy link
Contributor Author

srgsanky commented May 29, 2024

The clang-format checker is currently failing due to changes introduced by another PR. Mentioned this in #118 (comment)

@LiiNen
Copy link
Contributor

LiiNen commented May 29, 2024

sorry. maybe i missed some. fixed #570

PingXie pushed a commit that referenced this pull request May 29, 2024
ref:
- #118 (my pervious change)
- #461 (issuing that clang
format checker fails due to my change)

There was an issue that clang-format cheker failed.
I don't know why I missed it and why it didn't catch.

just running `clang-format -i bitops.c` was all.

Signed-off-by: LiiNen <[email protected]>
Copy link
Member

@PingXie PingXie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@madolson madolson merged commit a81c320 into valkey-io:unstable Jun 17, 2024
@madolson
Copy link
Member

@PingXie Sorry, I saw your approval and had intended to merge this on Friday but ran out of time and missed that you had comments. They seemed like small comments, so @srgsanky feel free to address, I commented on one of them.

@PingXie
Copy link
Member

PingXie commented Jun 17, 2024

No worries. These can all be addressed incrementally.

PingXie pushed a commit to PingXie/valkey that referenced this pull request Jul 8, 2024
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

---------

Signed-off-by: Sankar <[email protected]>
PingXie pushed a commit to PingXie/valkey that referenced this pull request Jul 9, 2024
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

---------

Signed-off-by: Sankar <[email protected]>
PingXie pushed a commit to PingXie/valkey that referenced this pull request Jul 9, 2024
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

---------

Signed-off-by: Sankar <[email protected]>
Signed-off-by: Ping Xie <[email protected]>
PingXie pushed a commit that referenced this pull request Jul 10, 2024
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

---------

Signed-off-by: Sankar <[email protected]>
Signed-off-by: Ping Xie <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cluster release-notes This issue should get a line item in the release notes

Projects

Status: Done
Status: Done

Development

Successfully merging this pull request may close these issues.

7 participants