Document how people, pods and CI jobs should authenticate to ate-api - #3119

Bring `release-1.0` up to `main` at
5a5b192 with the 19 changes merged
since the previous batch in #3536. The final tree matches that commit
exactly.

**Please use Rebase and merge to preserve the individual commits and
author attribution.** All 19 applied cleanly in upstream order. Original
authors, author dates, co-author and sign-off trailers, and `-x` source
references are retained.

Upgrade notes:

- #3533: old gVisor MEMORY snapshots, including goldens, no longer
restore. Recreate them after upgrading. MicroVM MEMORY snapshots still
resume with their old mounts until recreated. `/tmp` is now always
tmpfs; ROOTFS snapshots no longer preserve its contents, and
image-provided files there are hidden. MicroVM `/tmp` counts against
actor memory and is capped at half the guest's memory. gVisor `/dev/shm`
is now a 64 MiB tmpfs. Mount a `durableDir` at either path when
persistence or more space is needed. `/run` now comes from the image
rootfs.
- #3447: remove `--actor-jwt-jwks-file` from custom credential-exchange
sidecar configs. Set `--actor-jwt-issuer-ca-file` only when the issuer
uses a private CA other than the service DNS CA.

- #3450: do not reuse a deleted atespace name. See the updated atespace
name-reuse guidance.

Includes #3505, which overlaps the open backport in #3547. GKE
provisioning and teardown now live in
[substrate-gke](https://github.com/ai-on-gke/substrate-gke) rather than
this repository.

<details>
<summary>Included PRs, in cherry-pick order</summary>

1. #3500: demos/{sandbox,jupyter,multi-template}: fix tutorial
prerequisites, walkthroughs, and counter volume
2. #3498: demos/counter: inline gVisor walkthrough and fix setup
commands
3. #3511: e2e tests: Extend the existing tests to cover tag actor.
4. #2775: Generate API reference documentation and publish it on
Substrate website
5. #3119: Document how people, pods and CI jobs should authenticate to
ate-api
6. #3505: tools: remove setup-gcp and hack/teardown.sh
7. #3147: e2e: resume fidelity after SandboxConfig version edits
8. #3370: ate-setup: reconcile ate-api-server ConfigMaps and roll out on
config changes
9. #1607: hack: reject breaking proto changes across each proto package
10. #3415: docs: update and add security diagrams
11. #3485: Resuming a microVM actor on the same node no longer reads its
rootfs upper back from disk
12. #3447: Verify actor JWTs in the credential exchange through OIDC
discovery
13. #3533: ocispec: mount tmpfs at /dev/shm, and /tmp on both runtimes
14. #3450: docs: Document that atespace names must not be reused
15. #3410: Document using more than one egress credential provider
16. #3486: atelet: register the device plugin on a new socket each time
17. #3553: ate-setup: wait for the demo namespace in delete demo
18. #3560: Fix multiple issues in claude-code-multiplex demo
19. #3517: docs: remove inaccurate early-stage architecture
documentation

</details>

Validation: every intermediate tree matches its upstream source commit,
all 19 authorship records and original attribution trailers are
preserved, and `git diff --check` passes. The original 18 backport
commits are unchanged.

AI use: AI-assisted cherry-picking, attribution checks, and review.

- [ ] Tests pass
- [x] Docs updated

---------

Signed-off-by: Mofi Rahman <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
Signed-off-by: Nayana Bidari <[email protected]>
Signed-off-by: Julian Gutierrez Oschmann <[email protected]>
Signed-off-by: Max Thompson <[email protected]>
Signed-off-by: Steven Shriver <[email protected]>
Signed-off-by: Grant McCloskey <[email protected]>
Signed-off-by: Shrutiya <[email protected]>
Signed-off-by: Haiyan Meng <[email protected]>
Signed-off-by: Greg Castle <[email protected]>
Signed-off-by: Benjamin Elder <[email protected]>
Signed-off-by: Anish Ramasekar <[email protected]>
Signed-off-by: Shruti Nair <[email protected]>
Signed-off-by: Haven Xia <[email protected]>
Co-authored-by: Mofi Rahman <[email protected]>
Co-authored-by: nybidari <[email protected]>
Co-authored-by: Julian Gutierrez Oschmann <[email protected]>
Co-authored-by: Max Thompson <[email protected]>
Co-authored-by: Grant McCloskey <[email protected]>
Co-authored-by: Steven Shriver <[email protected]>
Co-authored-by: shrutiyam-glitch <[email protected]>
Co-authored-by: haiyanmeng <[email protected]>
Co-authored-by: Greg Castle <[email protected]>
Co-authored-by: Benjamin Elder <[email protected]>
Co-authored-by: Anish Ramasekar <[email protected]>
Co-authored-by: Shruti Nair <[email protected]>
Co-authored-by: Haven Xia <[email protected]>
Co-authored-by: Adam Wathieu <[email protected]>

…3119)

Part of #1464.

- Rewrites `docs/guides/authentication.md` around who is calling
ate-api: a person running `kubectl ate`, a pod in the Substrate cluster
or another cluster, or a CI job, so people stop leaning on the shared
`ate-client` token.
- Adds pod certificates as an option for client pods (the benchmarks use
them), the `ate.dev/api-access` namespace label the ate-system
NetworkPolicy requires, and the Kubernetes RBAC `kubectl ate` needs to
reach ate-api.
- Warns where an identity is weaker than it looks: another cluster's
issuer can mint `ate-client`, GKE's Kubernetes Engine Developer and
basic Editor roles can act as any ServiceAccount, ate-system
port-forward rights reach loopback-only Envoy admin APIs that can stop
the data plane, and GitHub and Google tokens authenticate anyone.
- Moves the per-user ServiceAccount recipe and the signing-in and
install-time-provider gaps out of `authorization.md`, and drops its
claim that a Secret-based token works there (ate-api rejects its
issuer). The recipe and RBAC use #2851's names (`ate-users`,
`ate-api-client`).

For reviewers: #2851 (`--credential-plugin`) rewrites the same sections,
and its paragraph calling GitHub and Google tokens unsafe needs to match
this one.

Testing: checked each claim against the code on main (GitHub, GKE and
Kubernetes behavior against their docs or source) and that the relative
links resolve. None of the commands or manifests ran on a cluster.

AI use: AI-assisted.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Max Thompson <[email protected]>
(cherry picked from commit f1973c1)
Signed-off-by: Eitan Yarmush <[email protected]>