ate-setup: reconcile ate-api-server ConfigMaps and roll out on config changes - #3370

Bring `release-1.0` up to `main` at
5a5b192 with the 19 changes merged
since the previous batch in #3536. The final tree matches that commit
exactly.

**Please use Rebase and merge to preserve the individual commits and
author attribution.** All 19 applied cleanly in upstream order. Original
authors, author dates, co-author and sign-off trailers, and `-x` source
references are retained.

Upgrade notes:

- #3533: old gVisor MEMORY snapshots, including goldens, no longer
restore. Recreate them after upgrading. MicroVM MEMORY snapshots still
resume with their old mounts until recreated. `/tmp` is now always
tmpfs; ROOTFS snapshots no longer preserve its contents, and
image-provided files there are hidden. MicroVM `/tmp` counts against
actor memory and is capped at half the guest's memory. gVisor `/dev/shm`
is now a 64 MiB tmpfs. Mount a `durableDir` at either path when
persistence or more space is needed. `/run` now comes from the image
rootfs.
- #3447: remove `--actor-jwt-jwks-file` from custom credential-exchange
sidecar configs. Set `--actor-jwt-issuer-ca-file` only when the issuer
uses a private CA other than the service DNS CA.

- #3450: do not reuse a deleted atespace name. See the updated atespace
name-reuse guidance.

Includes #3505, which overlaps the open backport in #3547. GKE
provisioning and teardown now live in
[substrate-gke](https://github.com/ai-on-gke/substrate-gke) rather than
this repository.

<details>
<summary>Included PRs, in cherry-pick order</summary>

1. #3500: demos/{sandbox,jupyter,multi-template}: fix tutorial
prerequisites, walkthroughs, and counter volume
2. #3498: demos/counter: inline gVisor walkthrough and fix setup
commands
3. #3511: e2e tests: Extend the existing tests to cover tag actor.
4. #2775: Generate API reference documentation and publish it on
Substrate website
5. #3119: Document how people, pods and CI jobs should authenticate to
ate-api
6. #3505: tools: remove setup-gcp and hack/teardown.sh
7. #3147: e2e: resume fidelity after SandboxConfig version edits
8. #3370: ate-setup: reconcile ate-api-server ConfigMaps and roll out on
config changes
9. #1607: hack: reject breaking proto changes across each proto package
10. #3415: docs: update and add security diagrams
11. #3485: Resuming a microVM actor on the same node no longer reads its
rootfs upper back from disk
12. #3447: Verify actor JWTs in the credential exchange through OIDC
discovery
13. #3533: ocispec: mount tmpfs at /dev/shm, and /tmp on both runtimes
14. #3450: docs: Document that atespace names must not be reused
15. #3410: Document using more than one egress credential provider
16. #3486: atelet: register the device plugin on a new socket each time
17. #3553: ate-setup: wait for the demo namespace in delete demo
18. #3560: Fix multiple issues in claude-code-multiplex demo
19. #3517: docs: remove inaccurate early-stage architecture
documentation

</details>

Validation: every intermediate tree matches its upstream source commit,
all 19 authorship records and original attribution trailers are
preserved, and `git diff --check` passes. The original 18 backport
commits are unchanged.

AI use: AI-assisted cherry-picking, attribution checks, and review.

- [ ] Tests pass
- [x] Docs updated

---------

Signed-off-by: Mofi Rahman <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
Signed-off-by: Nayana Bidari <[email protected]>
Signed-off-by: Julian Gutierrez Oschmann <[email protected]>
Signed-off-by: Max Thompson <[email protected]>
Signed-off-by: Steven Shriver <[email protected]>
Signed-off-by: Grant McCloskey <[email protected]>
Signed-off-by: Shrutiya <[email protected]>
Signed-off-by: Haiyan Meng <[email protected]>
Signed-off-by: Greg Castle <[email protected]>
Signed-off-by: Benjamin Elder <[email protected]>
Signed-off-by: Anish Ramasekar <[email protected]>
Signed-off-by: Shruti Nair <[email protected]>
Signed-off-by: Haven Xia <[email protected]>
Co-authored-by: Mofi Rahman <[email protected]>
Co-authored-by: nybidari <[email protected]>
Co-authored-by: Julian Gutierrez Oschmann <[email protected]>
Co-authored-by: Max Thompson <[email protected]>
Co-authored-by: Grant McCloskey <[email protected]>
Co-authored-by: Steven Shriver <[email protected]>
Co-authored-by: shrutiyam-glitch <[email protected]>
Co-authored-by: haiyanmeng <[email protected]>
Co-authored-by: Greg Castle <[email protected]>
Co-authored-by: Benjamin Elder <[email protected]>
Co-authored-by: Anish Ramasekar <[email protected]>
Co-authored-by: Shruti Nair <[email protected]>
Co-authored-by: Haven Xia <[email protected]>
Co-authored-by: Adam Wathieu <[email protected]>

… changes (#3370)

`EnsureAPIServerPrerequisites` created
`ConfigMap/ate-api-authentication` and `ConfigMap/ate-api-config` only
when missing (unless `--ateapi-log-level` was passed), ignoring
`--ateapi-expected-jwt-issuer` on re-deploy and leaving legacy or
unknown `1.x` rollback fields in place where `ate-api-server`'s strict
YAML/prototext loaders would `CrashLoopBackOff`. Additionally,
`Deployment/ate-api-server`'s `ate.dev/env-hash` annotation only
digested environment variables rather than the mounted
`ate-api-authentication` and `ate-api-config` ConfigMaps.

Fixes #3271

## Summary

```text
EnsureAPIServerPrerequisites
  ensureAPIAuthenticationConfig -> ateapiauth.ParseAuthenticationConfig
    (rewrites when missing, when --ateapi-expected-jwt-issuer is set, or when stored YAML fails strict schema validation; preserves valid custom issuers and extra jwtProviders)
  ensureAPIConfig -> apiconfig.ParseTextProto
    (rewrites when missing, when --ateapi-log-level is set, or when stored prototext fails strict schema/level validation; preserves valid customized log levels)

DeployAPIServer / UpdateAPIAuthenticationConfig / UpdateAPIConfig
  apiServerEnvHash(env, ate-api-authentication, ate-api-config) -> annotates Deployment/ate-api-server pod template (ate.dev/env-hash)
```

## Evidence

- **Before:** Re-running `ate-setup deploy ate-system
--ateapi-expected-jwt-issuer <issuer>` skipped
`ConfigMap/ate-api-authentication`, and upgrading across schema removals
or rolling back `1.x -> 1.0.0` with unknown fields in
`authentication.yaml` or `apiconfig.textproto` left `ate-api-server` in
`CrashLoopBackOff`.
- **After:** Valid customized ConfigMaps (custom JWT issuers, extra
external `jwtProviders`, hand-edited log levels) survive `ate-setup
deploy` untouched when flags are unset; explicit flags or
invalid/unknown rollback fields rewrite the ConfigMaps cleanly and
update `ate.dev/env-hash`, rolling `Deployment/ate-api-server` to
`Ready`.

## Merge Danger

**Door:** Two-way

**Blast Radius:** Small (`internal/ateapiauth`, `internal/apiconfig`,
and `cmd/ate-setup` prerequisite reconciliation and `ate.dev/env-hash`
computation)

## Rationale

`api-server` crashes on any config field it doesn't recognize, and
`ate-setup` never touched those config maps once they existed. the PR
makes `ate-setup` rewrite the config maps when the server can't parse
it, and keeps valid custom ones.

However this need to be in `1.0,0` otherwise if `1.0.x` adds a config
field, then someone rollback from `1.0.x` to `1.0.0`, the old server
will fail the rollback as it cannot parse the yaml.

AI use: AI-assisted implementation and unit tests; reviewed and verified
by @moficodes on Kind and GKE.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Mofi Rahman <[email protected]>
(cherry picked from commit 671cb69)
Signed-off-by: Eitan Yarmush <[email protected]>