When the cilium agent is down (due to a crash or an upgrade), it can
miss node events. Upon startup, live nodes are upserted, but when
deletions are missed, the agent fails to clean up node-related system
state. Examples of such state includes bpf map entries, xfrm states or
routes. In particular, the agent fails to clean up node IP to nodeID
mappings in the nodeid bpf map. Since K8s will happily recycle such IPs,
this can lead to breakage, as the agent associate the wrong nodeID with
IPs.

To avoid leaking this state, the node manager now dumps its view of the
current set of nodes to a file in the runtime state directory, which can
be read on restart of an agent. This is similar to how we restore other
state upon restart.

When reading this file, it's important to avoid resurrecting long-gone
nodes (as we don't know for how long the agent was down) - instead, we
merely take note of which nodes we knew of in the past, compare that to
the nodes we consider live (once synced to k8s), and delete the ones
which seem to have disappeared.

The motivation to build this reconciliation based on full state dumps to
disk is that downstream code generally assumes to have access to a full
node object in the deletion callbacks. This makes is infeasible to base
the pruning on just the information available in bpf maps. In an
alternative design, downstream subsystems are responsible for cleaning
up their own state based on just a node identifier, but current code
doesn't allow for this.

Signed-off-by: David Bimmler <[email protected]>