chore: optimize jira-log skill description and remove redundancy - #6

#6)

HyperShell authenticates callers with OIDC (Keycloak bearer tokens); client
mTLS is not required or supported. The gateway pod's TLS is server-side
transport encryption only. Remove client mTLS from the four clauses that
previously described it as a hard, mode-independent prerequisite:
- ingress overview (both modes converge on the same workload)
- ROKS route-mode cert-manager prerequisite note
- Requirement: Tenant Gateway Ingress via OpenShift Route (route mode)
- Requirement: cert-manager Is a Mode-Independent Prerequisite

cert-manager still mints the gateway's per-tenant server TLS + CA in every mode.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

#6)

HyperShell authenticates callers with OIDC (Keycloak bearer tokens); client
mTLS is not required or supported. The gateway pod's TLS is server-side
transport encryption only. Remove client mTLS from the four clauses that
previously described it as a hard, mode-independent prerequisite:
- ingress overview (both modes converge on the same workload)
- ROKS route-mode cert-manager prerequisite note
- Requirement: Tenant Gateway Ingress via OpenShift Route (route mode)
- Requirement: cert-manager Is a Mode-Independent Prerequisite

cert-manager still mints the gateway's per-tenant server TLS + CA in every mode.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* spec: add global architecture specification

Captures deployment patterns (single-node, global multi-region, multi-cloud),
tooling stack decisions (CNPG, ArgoCD, Tekton, Vault, Terraform, Prometheus),
namespace strategy, installer pipeline requirements, and monitoring architecture
from the Aug 10 architecture meeting.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* docs: add Ingress Architecture section detailing dual-ingress strategy

* docs: resolve open questions regarding IBM Gateway API availability, VPC LB scope, and control-plane reconciler bug

* docs: add shared gateway rationale to Design Decisions table

* docs(skills): add IBM Cloud Hub provisioning and shared-gateway ingress skills

Add two deploy skills and cross-link them from deploy-cluster and CLAUDE.md:

- cloud-hub-ingress-bootstrap: cloud-agnostic shared Gateway + wildcard DNS/TLS
  bootstrap (AWS reference / IBM parity). Encodes the OCP >= 4.19 requirement for
  the built-in openshift-default GatewayClass and the nlb-dns DNS+TLS path.
- ibm-cluster: ROKS VPC Gen2 provisioning mirroring the reference cluster, the
  cluster-create command (COS CRN required, not GUID), and a registry-storage
  decision table (emptyDir / PVC / COS) with PVC as the chosen persistent backend.
- deploy-cluster: add Cloud-Hub parameter overrides (registry host, ibmc-vpc-block
  storage class) and scope note pointing at the ingress bootstrap.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* spec: record IBM Gateway API version requirement in global architecture

The IBM Cloud Parity Plan now documents the root cause of the tenant-gateway
ingress gap: the built-in CIO-managed Gateway API (openshift-default GatewayClass)
is GA only on OCP >= 4.19, and the original hypershell-cluster ran 4.17. The fix is
a new >= 4.19 cluster (hysh-ibm-01, 4.21.27), not an in-place upgrade, cross-linked
to the ibm-cluster skill.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* feat(control-plane): environment-adaptive tenant ingress (Gateway API or Route)

Tenant-gateway ingress is now a selectable mode, chosen per environment by
configuration rather than hardcoded: emit Kubernetes Gateway API GRPCRoutes
where the Gateway API is available and functional, or OpenShift Routes
(HAProxy passthrough) where it is not.

Motivation: IBM Cloud ROKS is HyperShift-hosted and cannot run the CIO-managed
Istio (OSSM images unpullable, IDMS denied on the HostedCluster), yet ships the
Gateway API CRDs. Route passthrough preserves the gateway pod's per-tenant
self-signed TLS + client mTLS end-to-end, so it needs no shared Gateway,
wildcard cert, ClusterIssuer, or external DNS - it works on IBM's free
*.containers.appdomain.cloud wildcard.

Control plane:
- GATEWAY_INGRESS_MODE env var (gateway-api|route|none); auto-detects from
  opts.HasGatewayAPI/opts.IsOpenShift when unset. Explicit override wins, since
  ROKS's Gateway API CRDs are present-but-non-functional.
- reconcileRouteResources/deleteRouteResources (passthrough Route to
  openshell-gateway:8080 + openshell-gateway-allow-router NetworkPolicy +
  grpcs://<host>:443 address publish); shared deriveGatewayHostname/
  publishRouteAddress helpers; Route added to kindToResource and cleanup.
- Table tests for mode selection and hostname derivation.

Deploy:
- deploy/ibm kustomize overlay (on deploy/openshift) sets
  GATEWAY_INGRESS_MODE=route + base domain. Controller ClusterRole already
  grants route.openshift.io/routes.

Docs:
- global-architecture.spec.md: two first-class ingress modes with mode-aware
  requirements/scenarios and the ROKS Route-mode section.
- ibm-cluster / cloud-hub-ingress-bootstrap skills: Route mode is the ROKS path;
  do not run the shared-Gateway bootstrap there.
- Swept em dashes from tracked files; whitelisted pre-existing ACP mermaid nodes
  and rosa-vteam.yaml so make check passes.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* feat(control-plane): make gateway/supervisor/sandbox images env-overridable for registry mirrors

Gateway, supervisor, and sandbox default images were hardcoded to ghcr.io
with no override, so any gateway created without an explicit image failed
to pull on clusters whose nodes cannot reach ghcr.io (e.g. IBM ROKS).

- Add GATEWAY_IMAGE, GATEWAY_SUPERVISOR_IMAGE, and GATEWAY_SANDBOX_IMAGE
  env overrides (mirroring the existing HYPERSHELL_DATABASE_IMAGE pattern),
  with a new DefaultSandboxImage() so the sandbox base is resolved the same way.
- Substitute SANDBOX_IMAGE_PLACEHOLDER in the gateway configmap (ordered
  before IMAGE_PLACEHOLDER since the shorter token is a substring).
- Allow an optional host:port in image references so the in-cluster registry
  service address (image-registry.openshift-image-registry.svc:5000/...) validates.
- Apply supervisor_image on gateway PATCH.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(control-plane): add derived ingress hostname to gateway certificate SANs

When an ingress mode is active the gateway is reachable at an external
hostname (gw-<namespace>.<base-domain> or an explicit Route.Host), and both
ingress modes carry the gateway pod's TLS through unmodified (Route
passthrough / Gateway API BackendTLSPolicy). The server certificate must
therefore list that external hostname as a SAN, or clients fail verification.
The controller derives the hostname, so it injects it into the cert SANs
before cert-manager mints the certificate.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs(spec): expand global-architecture with ROKS ingress and image-mirror details

Extend the global architecture specification with the ingress-mode and
internal-registry-mirror behaviour exercised on IBM ROKS, and bump the
forbidden-terms whitelist line reference to track the moved example path.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs(skills): ROKS deploy guidance for ibm-cluster and deploy-cluster

Document the ROKS-specific deployment path: image mirroring to the internal
registry, raw operator installs, and the env overrides the control plane
needs on clusters that cannot reach ghcr.io.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test(pr-test): ROKS-adapted e2e-openshell script and ibm deploy overlay

Add e2e-openshell-roks.sh, a ROKS-layout copy of the canonical e2e that
targets the hypershell/hypershell-api route names, trusts the per-gateway
CA, uses the quoted SQL-like search grammar, and preserves a pre-existing
gateway on cleanup. Add the components/api-server/deploy/ibm overlay with
the controller cluster RBAC used on ROKS.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs(ibm-cluster): worker-SG egress fix + OIDC default-secure + cloud providers

Refine the registry-timeout explanation with its root cause (the kube-<clusterID>
worker Security Group is default-deny outbound) and the supported fix (add an
outbound 0.0.0.0/0:443 rule). Add section 5.7 covering Keycloak default-secure
gateway wiring, the correct OIDC `gateway add` command (not bare edge/cloud
mode), and the worker-egress requirement for cloud-model providers such as
google-vertex-ai. Correct the stale gateway-add note in 5.5.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs(ibm-cluster): document sandbox connect gateway-insecure env workaround

`openshell sandbox connect` execs the system ssh with a ProxyCommand that
re-execs `openshell ssh-proxy`, and the CLI omits `--gateway-insecure` from
that generated ProxyCommand. The child ssh-proxy therefore verifies the
self-signed passthrough gateway cert and fails `invalid peer certificate:
UnknownIssuer`; the flag on `connect` never reaches it. The child inherits
the environment, so `export OPENSHELL_GATEWAY_INSECURE=true` is the working
fix. Verified live on hysh-ibm-01 (sandbox woot).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* spec: inference-routing + ROKS credential-free sandbox agent runbook

Add openshell-inference-routing.spec.md documenting how sandbox agents reach
cloud models with no credential in the sandbox: the inference.local router
strips the client key and injects the provider token server-side, translating
/v1/messages -> Vertex :rawPredict. Covers the two credential paths (per-binary
sentinel rewrite vs router injection) and the request-shape compatibility
requirement. Register it in the spec index.

Add ibm-cluster skill section 5.8 with the ROKS runbook: `inference set`, the
required non-effort `--model claude-sonnet-4-5` workaround for Vertex's strict
vertex-2023-10-16 validation (adaptive-thinking / output_config.effort 400s),
sandbox connect, and the ~/.claude/settings.json wiring for bare `claude`.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs(spec): add --dangerously-skip-permissions to inference client example

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* style: replace em dashes with ASCII punctuation for forbidden-terms lint

main tightened scripts/check_forbidden_terms.py to reject U+2014; convert
the em dashes in files this PR authored to ASCII hyphens.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(control-plane): validate Route.Host and block cross-tenant route hijack

An explicit Gateway Route.Host was not validated: it bypassed the DNS-name
check applied to ServerDnsNames and flowed verbatim into the OpenShift Route
spec.host (with the controller holding routes/custom-host) and the gateway
certificate SANs. Under a shared wildcard base domain a tenant could set
Route.Host to another tenant's derived host (gw-<other>.<base-domain>) and
hijack its route, since OpenShift Route host claiming is first-come.

- ValidateGatewayConfig now DNS-validates Route.Host (hard fail).
- deriveGatewayHostname now requires an explicit host that falls under
  GATEWAY_API_BASE_DOMAIN to equal this tenant's own gw-<namespace>.<base>
  slot; foreign hosts under the shared wildcard are rejected (fail-closed:
  no Route is created). External/vanity hosts outside the base domain pass
  through unchanged.
- Table tests for the DNS validation and the hijack/own-slot/vanity cases.

Addresses Amber review finding #1 on PR #85.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs(spec): clarify control-plane role, terminology, ownership, and LB (amarin #1-5)

Address amarin's review comments on the global architecture spec:
- Broaden the control-plane operational role: it provisions the full set of
  OpenShell resources per tenant (namespaces, PKI, RBAC, ingress, CNPG, and
  supporting workloads), not just OpenShell Gateways.
- Terminology: add a canonical-names note and use fully-qualified names —
  "OpenShell Gateway" (workload), "Gateway API" (k8s API), "shared Gateway
  (Gateway API resource)" — instead of the overloaded bare "gateway".
- Make Sandboxes explicit in the Tier 3 gateway-workloads description.
- Source of truth: HyperShell's Cloud Hub PostgreSQL owns desired state for
  Fleet/Gateway/ManagedCluster; OpenShell Gateway runtime state (sandboxes,
  provider credentials, sessions) lives in the gateway's own database.
- Generalize the platform-services load balancer: describe the role generically
  with AWS NLB and IBM Cloud VPC LB as the concrete instances today.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs(spec): OIDC is the client auth mechanism; drop client mTLS (amarin #6)

HyperShell authenticates callers with OIDC (Keycloak bearer tokens); client
mTLS is not required or supported. The gateway pod's TLS is server-side
transport encryption only. Remove client mTLS from the four clauses that
previously described it as a hard, mode-independent prerequisite:
- ingress overview (both modes converge on the same workload)
- ROKS route-mode cert-manager prerequisite note
- Requirement: Tenant Gateway Ingress via OpenShift Route (route mode)
- Requirement: cert-manager Is a Mode-Independent Prerequisite

cert-manager still mints the gateway's per-tenant server TLS + CA in every mode.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* chore(deps): update OpenShell to v0.0.106; add self-reinforcing update-openshell skill

Add skills/tooling/update-openshell - a repeatable, self-reinforcing skill for
syncing HyperShell to upstream OpenShell releases (they ship ~daily). It bumps the
version-pin footprint, triages release notes for contract-affecting changes,
verifies build/config, and folds each run's lessons back into the skill + specs.

First run: update 0.0.101 -> 0.0.106.
- Bump defaultGatewayImage/defaultSupervisorImage (source of truth) and every copy
  across specs, the ROKS e2e script, kind lib, and the ibm-cluster skill.
- Normalize specs/platform/openshell-gateway-database.spec.md, which pinned the
  gateway image to a git SHA instead of a release tag.
- Preserve fixtures/historical refs (the validation_test.go regex fixture and the
  "v0.0.101 introduced credential drivers" historical sentence).

Triage 102-106: mechanically safe pin bump. One needs-decision item recorded as a
follow-up in the skill's learnings log - upstream v0.0.106 shipped a cert-manager
external issuer + OpenShift passthrough Route (NVIDIA/OpenShell#2468) that overlaps
HyperShell's hand-rolled per-tenant self-signed CA; evaluate adopting it separately.

Also repair pre-existing "make check" failures from earlier spec commits on this
branch: replace forbidden em dashes (U+2014) with " - " and refresh the
line-number-based forbidden-term whitelist (Mermaid ACP nodes + vteam path) after
the Terminology block shifted line numbers.

Register /update-openshell in CLAUDE.md.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(control-plane): make ROKS e2e pass on openshell 0.0.109

Bring the IBM Cloud ROKS (hysh-ibm-01) end-to-end path to 22/22 on openshell
0.0.109. components/pr-test/e2e-openshell-roks.sh now validates the full flow
(HyperShell API -> control plane -> per-tenant passthrough Route -> gateway ->
OIDC admin + developer -> sandbox create + exec) for both an admin and a
standard user.

Three fixes, each caught only by the live sandbox e2e (pin/schema diffing missed
all three):

- Sandbox client TLS: restore the openshell-client cert-manager Certificate and
  client_tls_secret_name so runners get OPENSHELL_TLS_CA to verify the gateway
  server cert (required by 0.0.109 combined topology). This is internal
  sandbox<->gateway TLS, distinct from external-client mTLS (external clients
  authenticate via OIDC over the Route; no client_ca_path).
- StatefulSet/Deployment collision: drop statefulset.yaml from the deploy order
  and remove the file so the gateway workload is a single Deployment (no orphaned
  crash-looping openshell-gateway-0).
- Workspace membership: the e2e's developer-RBAC step now has an admin grant the
  standard user 'user' membership on the 'default' workspace before sandbox
  create. openshell 0.0.109 enforces workspace membership as a second,
  non-claim-derived authz layer independent of the OIDC role.

The e2e defaults OPENSHELL to ~/.local/bin/openshell (>= 0.0.98, which has the
`workspace` subcommand); there is no downloadable 0.0.109 CLI.

Docs: ibm-cluster/SKILL.md gains a validation banner, section 5.9 (workspace
membership + CLI version), and section 5.5 notes on sandbox client TLS and the
Deployment-only workload; update-openshell/SKILL.md gains a 0.0.106 -> 0.0.109
learnings-log entry (v1beta1 confirmed against the running gateway).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(pr-test): REST-based gateway delete, stale-gateway re-provision, sandbox diagnostics

Three fixes to the ROKS e2e (components/pr-test/e2e-openshell-roks.sh), found
while diagnosing sandboxes that reached Running but never became Ready (every
runner crashlooped on "OPENSHELL_TLS_CA is required"):

- Delete via REST, not hsctl. hsctl exposes no `delete` subcommand (only
  create/get/list/login), so the cleanup trap's `hsctl delete gateway` silently
  no-op'd on every run - gateways and their tenant namespaces accumulated
  indefinitely. Added a delete_gateway helper that calls
  DELETE /api/hypershell/v1/gateways/{id} and use it in cleanup.

- Detect and re-provision stale gateways. A gateway provisioned by a controller
  predating the sandbox client-TLS fix has no openshell-client-tls secret and no
  client_tls_secret_name in gateway.toml, so its sandboxes crashloop. The script
  now checks those markers on an existing gateway and, if stale, deletes and
  re-provisions it instead of blindly reusing it.

- Surface sandbox failure root cause. dump_sandbox_diag prints the sandbox pod's
  phase/restart count and last container logs on any exec/not-ready/not-found
  failure, turning the CLI's generic "sandbox is not ready" into the actual
  cause.

Also correct ibm-cluster/SKILL.md 5.1: it wrongly claimed HyperShell no longer
issues an openshell-client certificate. It does - the client cert exists so
sandbox runners get OPENSHELL_TLS_CA to verify the gateway server cert (internal
sandbox->gateway TLS, not external mTLS).

Validated on hysh-ibm-01: 23/23 with a stale gateway present (detected, torn
down, re-provisioned; admin + developer sandbox exec both succeed), 22/22 on a
clean run with cleanup actually deleting the gateway (0 leftover e2e gateways).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

---------

Co-authored-by: user <[email protected]>
Co-authored-by: Claude <[email protected]>

Resolve the blocking and in-scope findings from the PR #528 code review:

- Reconciler self-heal now arms on fresh provisioning: a just-created
  ExternalSecret whose sync has not yet succeeded leaves the runtime in
  Provisioning and returns a retryable error so the watch queue requeues,
  instead of latching Running with no further event. Promotion to Running
  now requires every SecretSource to report Ready=True; a later sync
  failure is observed and degrades the runtime.
- UpdateStatus is now a column-scoped UPDATE of only the status columns
  (new dao.UpdateStatus) rather than a full-row Save, closing the
  lost-update race where a concurrent user spec PATCH was clobbered by the
  status write-back.
- secretFileMount preserves hash-suffix uniqueness when truncating volume
  names to <=63 chars (sanitizeDNS1123WithHash), so long equal base names
  cannot collide into duplicate pod volume names the CronJob API rejects.
- Remove the dead `active` dedup map from the AgentRuntime reconciler; the
  watch reconcile queue already serializes work per key.

Tests lock the fixed behavior: fresh-secrets requeue and self-heal to
Running, volume-name hash uniqueness, and a DB-backed lost-update test
asserting a concurrent spec PATCH survives the status write-back.

ExternalSecret GVR versioning (#4) and redundant self-event re-applies
(#6) are intentionally out of Wave B scope.

Refs #521

Co-Authored-By: Claude Opus 4.8 <[email protected]>

…s lifecycle (openshift-online#521 wave B) (openshift-online#528)

* feat(control-plane): AgentRuntime cluster-resource reconciler + status lifecycle (openshift-online#521 wave B)

Replaces the Wave A no-op AgentRuntimeReconciler with the real cluster-side
reconciler and the status lifecycle write-back (issue openshift-online#521 requirements #3 and
#5).

Control plane (requirement #3): on reconcile, idempotently create/update the
agent's Namespace, a ServiceAccount with automountServiceAccountToken=false, a
default-deny-ingress NetworkPolicy, one ExternalSecret per SecretSource (with the
SecretSource purpose driving the coordinator env var wiring), and the coordinator
CronJob (schedule from cron, image from coordinator_image). Deletion removes them
by deleting the namespace, which cascades. Stale ExternalSecrets whose SecretSource
was removed are pruned.

Status lifecycle (requirement #5): drives Pending -> Provisioning ->
Running/Degraded and writes Terminating on delete, back to the API server over a
new UpdateAgentRuntimeStatus gRPC RPC. Degraded surfaces the failing SecretSource
name in a new status_message field; a sync failure returns a retryable error so the
runtime self-heals once the secret syncs. A client- and server-side guard
suppresses no-op status writes to avoid a write-back loop.

Transport: the AgentRuntime gRPC watch/list payload now embeds the runtime's owned
SecretSources (resolved through a secretSources->agentRuntimes lookup seam, no
import cycle) so the reconciler needs no second round trip, and carries the new
status_message. These additions are gRPC-only (control-plane facing); REST/OpenAPI
exposure of status_message is deferred to keep the SDK unchanged.

Gateway-side workspace provisioning (requirement #4) is intentionally NOT included:
it is blocked on the gateway workspace API design (openshift-online#526). The Running transition
here reflects cluster-side success only and will additionally gate on the
AgentWorkspace once #4 lands.

Refs openshift-online#521

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(agent-runtime): address Wave B code-review findings (openshift-online#521)

Resolve the blocking and in-scope findings from the PR openshift-online#528 code review:

- Reconciler self-heal now arms on fresh provisioning: a just-created
  ExternalSecret whose sync has not yet succeeded leaves the runtime in
  Provisioning and returns a retryable error so the watch queue requeues,
  instead of latching Running with no further event. Promotion to Running
  now requires every SecretSource to report Ready=True; a later sync
  failure is observed and degrades the runtime.
- UpdateStatus is now a column-scoped UPDATE of only the status columns
  (new dao.UpdateStatus) rather than a full-row Save, closing the
  lost-update race where a concurrent user spec PATCH was clobbered by the
  status write-back.
- secretFileMount preserves hash-suffix uniqueness when truncating volume
  names to <=63 chars (sanitizeDNS1123WithHash), so long equal base names
  cannot collide into duplicate pod volume names the CronJob API rejects.
- Remove the dead `active` dedup map from the AgentRuntime reconciler; the
  watch reconcile queue already serializes work per key.

Tests lock the fixed behavior: fresh-secrets requeue and self-heal to
Running, volume-name hash uniqueness, and a DB-backed lost-update test
asserting a concurrent spec PATCH survives the status write-back.

ExternalSecret GVR versioning (#4) and redundant self-event re-applies
(openshift-online#6) are intentionally out of Wave B scope.

Refs openshift-online#521

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(control-plane): align ExternalSecret GVR to external-secrets.io/v1

The AgentRuntime reconciler hardcoded the ExternalSecret served version to
v1beta1, but every other ExternalSecret reference in the repo (deploy/components
manifests, README, specs/platform) uses external-secrets.io/v1 and no v1beta1
CRD is installed on target clusters. The reconciler therefore failed every
dynamic Get/Create/Update/List/Delete for any AgentRuntime with a SecretSource.
Align the GVR to v1; tests reference the constant symbolically and follow.

Refs openshift-online#521

Co-Authored-By: Claude Opus 4.8 <[email protected]>

---------

Co-authored-by: user <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>

Amber review (#660) findings:

- [#1] Credential-secrets migration now cleans the configured credential
  namespace (credential_driver.kubernetes_secrets.namespace when set, else the
  tenant namespace), matching where the old control plane wrote, so the
  custom-namespace upgrade path is covered.
- [#2] Migration cleanup inspects the Role and RoleBinding independently and
  deletes only objects carrying the exact legacy hypershell-control-plane
  ownership labels, leaving Helm-owned, foreign-owned, and unlabeled objects
  untouched and cleaning a partial stale state. Added a unit test for the
  foreign/helm/unlabeled/partial cases.
- [#3] P2.7 Vault qualification now puts E2E_VAULT_ADDR/E2E_VAULT_ROLE into the
  credential_driver.vault payload and asserts the rendered gateway.toml uses
  them, so it exercises the supplied backend rather than the convention default.
- [#7] P2.4 managedcluster delete/restore waits for Running before deleting the
  cluster, captures an observed-generation marker AFTER detachment, and requires
  the generation to advance past it (dropping the weak "or still Running"
  predicate) to prove a real post-re-registration reconcile.

Findings #4 (revoke sandbox-denial), #5 (scrape live console), and #6
(provider-create credentials) follow in subsequent commits.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

#7 (follow-up): the previous observed_generation > marker check was impossible -
the gateway is converged before deletion and re-registration changes no desired
field, so generation never bumps and observed_generation can never exceed it
(PLAT-DATA-06). Instead, patch a benign desired field (external_dns) WHILE the
cluster is absent - a partial patch leaving cluster_id untouched is not a
reassignment, so it is accepted despite the dangling cluster_id - which bumps
generation but cannot reconcile while detached. After re-registration, require
observed_generation to catch up to that pending generation, proving a real
post-restore reconcile.

#6: both e2e jobs now materialize scoped GCP ADC from the E2E_GCP_ADC_JSON secret
and export GOOGLE_APPLICATION_CREDENTIALS + ANTHROPIC_VERTEX_PROJECT_ID so P2.11
provider-create (E2E-COV-05.03) runs on the gate; it skips gracefully when the
credentials are not configured (forks). Requires a maintainer to set the
E2E_GCP_ADC_JSON secret and E2E_VERTEX_PROJECT_ID variable.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>