Skip to content

Gossiped node deletion #10861

@zuiderkwast

Description

@zuiderkwast

Typically you need to send a cluster forget to each node in a cluster to delete a node. If you don't do this fast enough, the node will be re-added through gossip. Ideally you just need to forget a node and it will eventually be forgotten throughout the cluster.

This is part of #8948.

Possible ways to implement this:

  1. Append the contents of the clusterBlacklistNodes (node id and timestamp) to all cluster ping and pong messages. A ping extension (mechanism introduced for hostnames) can be used for this. (The blacklist expire timestamp needs to be gossiped too, so that we don't end up with the node being re-forgotten and re-added to the blacklist by gossip after the blacklist TTL is passed.) This increases the size of the cluster bus messages while any node is blacklisted (1 minute), but if the blacklist is short and the ping extension is small, this might be acceptable.
  2. Append the contents of the clusterBlacklistNodes (node id and timestamp) to every pong message which is a response to a ping, but only when we detect that the ping's gossip section contains a forgotten node, i.e. a node in our blacklist. This minimizes the extra data sent over the cluster bus, but it is enough to make sure a node is forgotten throughout the cluster?
  3. When a CLUSTER FORGET command is received, broadcast this to the cluster, for example in a pong with the forgotten node in a ping extension. This is done only once and by the node which receives the command. If a client sends CLUSTER FORGET to all nodes in a cluster, there will be NxN pongs sent in a short period of time.

@madolson et.al. any preference among the above or other ideas?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions