Add the crash reason and timestamp to the ActorStatus - #1867

…at the resume that crashed it

A golden actor whose image the registry refuses is crashed by the
resume at once (the control plane crashes an actor on every failed
atelet Run or Restore since agent-substrate#1220) and the
cause is recorded on the actor (ActorStatus.crash since agent-substrate#1867). The
ActorTemplate reconciler did not read it: a resume error was requeued
with backoff, the next pass found CRASHED and failed the template with
the fixed text "golden actor crashed before its snapshot was taken",
so a consumer watching the template (kagent reports Ready=False
ActorTemplateFailed) could not tell a missing image from anything else.

The reconciler now fails the template with the actor's recorded crash
message wherever it finds the golden actor CRASHED, and re-reads the
actor when its resume fails so a crash that resume caused fails the
template in the same pass, with the registry's answer in the message
(`GoldenActorCrashed: actor ate-golden/<uid> crashed: resume failed:
atelet Run: ... MANIFEST_UNKNOWN: manifest unknown`).

Re-pin note: on v0.2.0-beta5 this patch classified the registry's
final word in the image cache (ateerrors.ReasonFailedGetExternalObject)
and claimed it at atelet's Run and Restore boundaries; v0.3.0-alpha1
crashes on every atelet failure and deleted the ateerrors taxonomy
(agent-substrate#1817), so only the reconciler half remains.

(cherry picked from commit b1d759c)

… and crash it when no durable snapshot remains

A PAUSED actor's local snapshot lives on the node its worker ran on, and
the Resume workflow pins the actor to that node (Constraints.RequiredNodes).
When the node had left the cluster -- a spot interruption, a scale-down, a
node roll -- nothing noticed: the scheduler found no worker on the node and
AssignWorker answered ResourceExhausted "no free workers available", the
message of a full pool. The ingress dataplane parks a request on that
status, so every resume waited out the whole parking budget for an error
no wait could clear, the consumer saw a timeout, and the actor stayed
PAUSED for good: a suspend dialled the atelet on the dead node and stayed
retryable, so the actor could not be suspended and -- a delete suspends
first -- not be deleted either (agent-substrate#398 names the
case; agent-substrate#397 the empty-node variant).

The scheduler now says what its pass over the fleet saw when a node
restriction was in effect: Schedule returns a NoCapacityError -- still
errors.Is ErrNoCapacity -- carrying the restricted nodes, the eligible
workers found on them (all full) and the ACTIVE workers of any pool that
report them. AssignWorker classifies it:

  - eligible workers on the node, all full: ResourceExhausted, "all
    eligible workers on those nodes are busy", the node named, parked as
    before;
  - the node still in the cluster but no eligible worker on it (a pool
    rolling, a pool moved off the node): ResourceExhausted with the node
    named, still parked -- a worker may come back;
  - the node gone from the cluster: the snapshot is lost. Without a
    durable snapshot to fall back to the actor is crashed with the reason
    LOCAL_SNAPSHOT_GONE (CRASHED, ate.actor.crashes labelled, deletable)
    and the caller gets DataLoss with the crash directive; with one, the
    actor is left as it is and the caller gets DataLoss naming the node.

A node counts as gone only when no ACTIVE worker of any pool reports it
and its Node object is absent, read from a Node informer ate-api-server
now runs (RBAC: nodes get/list/watch). A worker's absence alone is no
proof: a pool rolling, an ateom restarting or the atelet DaemonSet
updating leave a node without an ACTIVE worker for a while, with its disk
and the snapshot on it intact. Without a Node lister nothing is declared
lost.

The paused-origin suspend takes the same exit: an atelet missing on a node
that is gone crashes the actor (or fails fast when a durable snapshot
remains) instead of staying retryable forever, so the owner can delete
the actor.

DataLoss is the one status the dataplane never parks on -- it retries
Aborted, and while parking is enabled ResourceExhausted, FailedPrecondition
and Unavailable -- so the caller learns within one round trip. The atenet
router maps it to 410 with the cause in the body, the way it preserves the
capacity message.

The read side of agent-substrate#397: empty entries in
node_vms_with_local_snapshots are ignored when the constraints are built
and when the suspend picks the node to dial, so a record poisoned with
[""] by an older pause finalization is schedulable again (the write side
already refuses an unknown node).

Signed-off-by: Timo Derstappen <[email protected]>
Re-pin note: on v0.2.0-beta5 the terminal answer carried the AIP-193
reason LOCAL_SNAPSHOT_GONE and the crash directive; v0.3.0-alpha1
deleted the ateerrors taxonomy (agent-substrate#1817) and records a crash on the actor
(ActorStatus.crash, agent-substrate#1867), so the refusal is a plain DataLoss status
and the crash message names the lost snapshot and its nodes. The
scheduler's eligible-workers metric upstream deleted (agent-substrate#1804) is not
re-added; NoCapacityError keeps its counts.

(cherry picked from commit 68ec85b)