Delete ate.scheduler.eligible_workers - #1804

… and crash it when no durable snapshot remains

A PAUSED actor's local snapshot lives on the node its worker ran on, and
the Resume workflow pins the actor to that node (Constraints.RequiredNodes).
When the node had left the cluster -- a spot interruption, a scale-down, a
node roll -- nothing noticed: the scheduler found no worker on the node and
AssignWorker answered ResourceExhausted "no free workers available", the
message of a full pool. The ingress dataplane parks a request on that
status, so every resume waited out the whole parking budget for an error
no wait could clear, the consumer saw a timeout, and the actor stayed
PAUSED for good: a suspend dialled the atelet on the dead node and stayed
retryable, so the actor could not be suspended and -- a delete suspends
first -- not be deleted either (agent-substrate#398 names the
case; agent-substrate#397 the empty-node variant).

The scheduler now says what its pass over the fleet saw when a node
restriction was in effect: Schedule returns a NoCapacityError -- still
errors.Is ErrNoCapacity -- carrying the restricted nodes, the eligible
workers found on them (all full) and the ACTIVE workers of any pool that
report them. AssignWorker classifies it:

  - eligible workers on the node, all full: ResourceExhausted, "all
    eligible workers on those nodes are busy", the node named, parked as
    before;
  - the node still in the cluster but no eligible worker on it (a pool
    rolling, a pool moved off the node): ResourceExhausted with the node
    named, still parked -- a worker may come back;
  - the node gone from the cluster: the snapshot is lost. Without a
    durable snapshot to fall back to the actor is crashed with the reason
    LOCAL_SNAPSHOT_GONE (CRASHED, ate.actor.crashes labelled, deletable)
    and the caller gets DataLoss with the crash directive; with one, the
    actor is left as it is and the caller gets DataLoss naming the node.

A node counts as gone only when no ACTIVE worker of any pool reports it
and its Node object is absent, read from a Node informer ate-api-server
now runs (RBAC: nodes get/list/watch). A worker's absence alone is no
proof: a pool rolling, an ateom restarting or the atelet DaemonSet
updating leave a node without an ACTIVE worker for a while, with its disk
and the snapshot on it intact. Without a Node lister nothing is declared
lost.

The paused-origin suspend takes the same exit: an atelet missing on a node
that is gone crashes the actor (or fails fast when a durable snapshot
remains) instead of staying retryable forever, so the owner can delete
the actor.

DataLoss is the one status the dataplane never parks on -- it retries
Aborted, and while parking is enabled ResourceExhausted, FailedPrecondition
and Unavailable -- so the caller learns within one round trip. The atenet
router maps it to 410 with the cause in the body, the way it preserves the
capacity message.

The read side of agent-substrate#397: empty entries in
node_vms_with_local_snapshots are ignored when the constraints are built
and when the suspend picks the node to dial, so a record poisoned with
[""] by an older pause finalization is schedulable again (the write side
already refuses an unknown node).

Signed-off-by: Timo Derstappen <[email protected]>
Re-pin note: on v0.2.0-beta5 the terminal answer carried the AIP-193
reason LOCAL_SNAPSHOT_GONE and the crash directive; v0.3.0-alpha1
deleted the ateerrors taxonomy (agent-substrate#1817) and records a crash on the actor
(ActorStatus.crash, agent-substrate#1867), so the refusal is a plain DataLoss status
and the crash message names the lost snapshot and its nodes. The
scheduler's eligible-workers metric upstream deleted (agent-substrate#1804) is not
re-added; NoCapacityError keeps its counts.

(cherry picked from commit 68ec85b)