ateapi: record a sync suspend's local snapshot copy and resume from it safely - #2883

…local

Since agent-substrate#2883 a synchronous suspend keeps a copy on the node after its upload,
so the resume after it reads that copy and is labeled local. The checkpoint
itself stays latest.

…s lookup

Following agent-substrate#2883 and agent-substrate#2881, every suspended actor with a retained local snapshot
has AssignedNode set and passes RequiredNodes = [assignedNode] into
scheduling.Schedule on ResumeActor. Without a node index, Schedule calls
s.source.Workers(), copying the entire cluster worker slice and scanning every
worker in O(W) time on each node-pinned resume.

Index cached workers by node name in workercache.Cache and use O(1) node lookup
in scheduling.Schedule when RequiredNodes is set:
- Add byNode index and WorkersByNode(node) in
  cmd/ateapi/internal/workercache/workercache.go, maintained across relist,
  Create/Update/Delete watch events, and Forget.
- Add workerNodeLookup in cmd/ateapi/internal/scheduling/scheduling.go and query
  nodeLookup.WorkersByNode(node) when Constraints.RequiredNodes is non-empty,
  avoiding full-fleet slice allocation and linear scanning.
- Add TestCache_WorkersByNode, TestScheduleRequiredNodesUsesNodeLookup, and
  BenchmarkScheduleRequiredNodes.

Benchmark results (go test -run='^$' -bench='BenchmarkScheduleRequiredNodes' -benchmem -count=5 ./cmd/ateapi/internal/scheduling/...):
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1727404     600.6 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1822207     623.1 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    2066503     626.2 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1887322     684.0 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    2061903     665.5 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     83085   15061 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     82180   17377 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     72000   14448 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     71266   14944 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     86091   14398 ns/op      4160 B/op   2 allocs/op

…s lookup

Following agent-substrate#2883 and agent-substrate#2881, every suspended actor with a retained local snapshot
has AssignedNode set and passes RequiredNodes = [assignedNode] into
scheduling.Schedule on ResumeActor. Without a node index, Schedule calls
s.source.Workers(), copying the entire cluster worker slice and scanning every
worker in O(W) time on each node-pinned resume.

Index cached workers by node name in workercache.Cache and use O(1) node lookup
in scheduling.Schedule when RequiredNodes is set:
- Add byNode index and WorkersByNode(node) in
  cmd/ateapi/internal/workercache/workercache.go, maintained across relist,
  Create/Update/Delete watch events, and Forget.
- Add workerNodeLookup in cmd/ateapi/internal/scheduling/scheduling.go and query
  nodeLookup.WorkersByNode(node) when Constraints.RequiredNodes is non-empty,
  avoiding full-fleet slice allocation and linear scanning.
- Add TestCache_WorkersByNode, TestScheduleRequiredNodesUsesNodeLookup, and
  BenchmarkScheduleRequiredNodes.

Benchmark results (go test -run='^$' -bench='BenchmarkScheduleRequiredNodes' -benchmem -count=5 ./cmd/ateapi/internal/scheduling/...):
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1802881     605.2 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    2008159     770.8 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1000000    1217 ns/op        96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1927760     839.7 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1705620     588.8 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     79581   17176 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     85873   14841 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     80368   15142 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     77570   14585 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     94237   17063 ns/op      4160 B/op   2 allocs/op

…s lookup

Following agent-substrate#2883 and agent-substrate#2881, every suspended actor with a retained local snapshot
has AssignedNode set and passes RequiredNodes = [assignedNode] into
scheduling.Schedule on ResumeActor. Without a node index, Schedule calls
s.source.Workers(), copying the entire cluster worker slice and scanning every
worker in O(W) time on each node-pinned resume.

Index cached workers by node name in workercache.Cache and use O(1) node lookup
in scheduling.Schedule when RequiredNodes is set:
- Add byNode index and WorkersByNode(node) in
  cmd/ateapi/internal/workercache/workercache.go, maintained across relist,
  Create/Update/Delete watch events, and Forget.
- Add workerNodeLookup in cmd/ateapi/internal/scheduling/scheduling.go and query
  nodeLookup.WorkersByNode(node) when Constraints.RequiredNodes is non-empty,
  avoiding full-fleet slice allocation and linear scanning.
- Add TestCache_WorkersByNode, TestScheduleRequiredNodesUsesNodeLookup, and
  BenchmarkScheduleRequiredNodes.

Benchmark results (go test -run='^$' -bench='BenchmarkScheduleRequiredNodes' -benchmem -count=5 ./cmd/ateapi/internal/scheduling/...):
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1802881     605.2 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    2008159     770.8 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1000000    1217 ns/op        96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1927760     839.7 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1705620     588.8 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     79581   17176 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     85873   14841 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     80368   15142 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     77570   14585 ns/op      4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     94237   17063 ns/op      4160 B/op   2 allocs/op

…s lookup

Following agent-substrate#2883 and agent-substrate#2881, every suspended actor with a retained local snapshot
has AssignedNode set and passes RequiredNodes = [assignedNode] into
scheduling.Schedule on ResumeActor. Without a node index, Schedule calls
s.source.Workers(), copying the entire cluster worker slice and scanning every
worker in O(W) time on each node-pinned resume.

Index cached workers by node name in workercache.Cache and use O(1) node lookup
in scheduling.Schedule when RequiredNodes is set:
- Add byNode index and WorkersByNode(node string) ([]*ateapipb.Worker, error) in
  cmd/ateapi/internal/workercache/workercache.go, maintained across relist,
  Create/Update/Delete watch events, and Forget, and returning an error when the
  cache is not ready.
- Add workerNodeLookup and workersForSchedule in
  cmd/ateapi/internal/scheduling/scheduling.go to query
  nodeLookup.WorkersByNode(node) when Constraints.RequiredNodes is non-empty,
  avoiding full-fleet slice allocation and linear scanning.
- Add TestCache_WorkersByNode, TestScheduleRequiredNodesUsesNodeLookup, and
  BenchmarkScheduleRequiredNodes.

Benchmark results (go test -run='^$' -bench='BenchmarkScheduleRequiredNodes' -benchmem -count=5 ./cmd/ateapi/internal/scheduling/...):
BenchmarkScheduleRequiredNodes/NodeIndex=true-8     849672    1203 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1000000    1157 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1086208    1237 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1000000    1248 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=true-8    1000124    1165 ns/op      96 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     35792   31085 ns/op    4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     37682   30128 ns/op    4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     39175   30108 ns/op    4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     36140   32745 ns/op    4160 B/op   2 allocs/op
BenchmarkScheduleRequiredNodes/NodeIndex=false-8     25226   80084 ns/op    4160 B/op   2 allocs/op

…nt-substrate#2486)

After a node is deleted, revert can still hang for a CRASHED actor or a
paused actor whose latest upload is PENDING. Delete can hang for any
actor still bound to that node. agent-substrate#2883 already lets revert and resume
forget an unreachable node when the actor has no worker assignment or
crash and the latest local snapshot has a COMPLETED durable copy.

When no atelet is indexed on the assigned node, `ensureAteletTerminated`
uses `NodeGone` to check the Node through the live Kubernetes API. It
skips termination only on NotFound, after checking again that no atelet
appeared. Existing nodes and lookup failures stay retryable. The API
server gains `get` permission on nodes.

External volumes still go through `DetachVolume`: nil and NotFound
succeed, other errors propagate. Kubernetes Node deletion does not fence
a machine that is still running; resources on such a machine may remain.

Rebased onto main `872ab2077`. The whole-workflow test comment now
identifies its paused actor's PENDING upload. Validation:
- Targeted revert/delete/detach and `TestNodeGone` tests pass with
`-race -count=3 -shuffle=on` and Docker-backed database tests enabled.
- The full race suite passes all 15 ateapi packages with tests. Overall
it reports 10,601 passing test/subtest events, 8 skipped, and two
failures in `cmd/ateworker-microvm/internal/child`:
`TestReapedWhenItDiesOnItsOwn` and `TestStop`. Both report `unsupported
operation` on macOS and reproduce from unchanged main `872ab2077`; the
non-Linux pidfd implementation returns `errors.ErrUnsupported`.
- `hack/verify-all.sh` passes (using local Python 3.13 and clang-format
23.1.3).
- A fresh independent review found no issues in the rebased patch. `git
diff --check` passes.

Earlier kind runs exercised real worker deletion, paused/crashed revert
and delete, and the kept-node control. Those live runs were not repeated
for this rebase. Current volume tests use a mock plugin; no real
CSI-driver validation is claimed.

Part of agent-substrate#2470. Resume without a completed latest upload remains
separate.

AI assistance: Claude Code helped write the original code and tests.
Codex clarified the fixture comment, rebased the branch, reviewed the
integration, and ran validation.

…substrate#2926)

Depends on agent-substrate#2883, which makes a synchronous suspend record `locality`
and keep `assigned_node`. The docs here describe that behavior after it
merges.

## Summary

Following the removal of `PauseActor` in agent-substrate#2813 and defaulting
`SuspendActor` to asynchronous mode in agent-substrate#2805, update documentation,
protobuf comments (and generated code), production Go comments, and
test/benchmark comments across the repository that still referred to
`PauseActor` or "paused" actors.

### Key Updates
- **Documentation (`docs/`)**:
- Remove the standalone `Pause` glossary entry in `docs/glossary.md` and
update `Actor`, `Checkpoint`, `Restore`, and `Suspend` definitions to
describe `SuspendActor`'s two durability modes
(`DURABILITY_MODE_SYNCHRONOUS` and `DURABILITY_MODE_ASYNCHRONOUS`, with
async as the default) and clarify that snapshots are always uploaded to
object storage.
- Update `docs/api-guide.md`, `docs/architecture.md`,
`docs/api-style-guide.md`, `docs/authorization.md`,
`docs/observability.md`, and `docs/metrics/registry/metrics.yaml` to
describe synchronous vs. asynchronous `SuspendActor` and node-local
snapshot pinning instead of `PauseActor`.
- **Protobuf & Generated Code (`pkg/proto/ateapipb/`,
`internal/proto/ateletpb/`)**:
- Update comments on `SuspendActor`, `RevertActor`, `DeleteSnapshotTag`,
`ActorStatus.assigned_node`, `Snapshot.locality`, and `ateletpb` RPCs;
regenerate `*.pb.go` and `*_grpc.pb.go`.
- **Production Code & Tests (`internal/`, `cmd/ateapi/`, `cmd/atelet/`,
`benchmarking/`)**:
- Update comments and error message hints referring to `PauseActor` or
"paused" actors to reflect asynchronous `SuspendActor` and node-local
snapshots.

…#3010)

agent-substrate#2883 lets resume forget an unreachable node when a paused actor's
latest snapshot is durable. When that upload is still PENDING, the actor
stays pinned. If the node was deleted, every resume returns
`Unavailable`, and nothing says the snapshot is lost.

Now, when resume finds no atelet on the pinned node and `NodeGone`
reports the Node object deleted, it reads the actor again. If the upload
was recorded during the lookup, it detaches as usual. Otherwise it
crashes the actor. The crash keeps `assigned_node` and the PENDING
snapshot. `RevertActor` then skips the terminate, deletes the pending
objects, and returns the actor to its last completed durable snapshot. A
node that still exists, or a failed Node lookup, keeps the current
retry.

There are two known limits. The check runs only after resume finds room
on another node, so with no room anywhere resume still returns
`RESOURCE_EXHAUSTED`. And a Node object deleted while the machine keeps
running crashes the actor even though its checkpoint is still on that
machine. That's the same trade agent-substrate#2486 makes.

A PENDING copy may already be fully uploaded if atelet died before
reporting it. Resume still crashes the actor, and revert deletes that
copy. Recovering it needs ate-api-server to check for the manifest,
which is a follow-up.

No proto changes. `ResumeActor` can now crash the actor and return
`FAILED_PRECONDITION` in this case, and `docs/api-guide.md` says so.

Part of agent-substrate#2470

Supersedes agent-substrate#3005. Its "upload completes during lookup" regression case
is adapted here.

AI use: Claude Code helped write the change and tests. I reviewed them.

- [ ] Tests pass
- [x] Docs updated