OpenShellSpawner: Close production gaps — skills, credentials, filesystem policy, MCP secrets

Context

The OpenShellSpawner has a working E2E flow (SDK migrated, parallel-safe, network policy auto-derived) but silently ignores several production features that other spawners implement. These features are passed by temporal_activities.py in every workflow and are critical for production use.

Cross-platform requirement: All features must work on both Podman and K8s deployments. No feature should silently fail on either target.

Gap Analysis

Feature	Current State	Proper Solution
`skills_image`	Silently ignored	`exec_stream()` to copy skills into sandbox post-creation
`credential_secret_name`	Silently ignored	`CreateProvider` + `AttachSandboxProvider` raw gRPC RPCs
`read_only`	Silently ignored	`SandboxPolicy.filesystem.read_only` path list
`mcp_secret_mounts`	Silently ignored	`_do_write_file()` to inject secret files post-creation
`tls_certs`	Silently ignored	Not needed — gateway provides transport security; log info + skip
`service_account`	Silently ignored	No OpenShell equivalent; log info + skip
JWT workaround	Stop/copy/restart hack	Upstream fix needed — document as known limitation

Plan

1. Skills image support (cross-platform)

Add _load_skills() method called inside _do_spawn() after sandbox creation.

Step 1 — Extract skills from image (platform-specific):

Podman mode (podman_cli set): podman run --rm -v /tmp/skills-<name>:/out <skills_image> cp -r <paths> /out/
K8s mode (podman_cli is None): Use crictl or spawn a short-lived K8s Job to extract skills to a temp dir

Step 2 — Stream into sandbox (platform-independent):

Primary: exec_stream(sandbox_id, ["tar", "xf", "-", "-C", "/app/skills"], stdin=tar_bytes)
Fallback (if exec_stream stdin not supported by SDK): _do_write_file() per-file with recursive directory walk

Pre-implementation: Verify exec_stream supports stdin bytes in the ExecSandboxRequest proto. If not, use _do_write_file() fallback from the start.

Test: Verify skills_paths subdirectory selection is preserved (matching K8s/Podman spawner contract).

2. Credential injection via OpenShell Provider API

Add _inject_credentials() using raw gRPC (same pattern as _expose_service()):

Derive credentials map from env dict: {credential_secret_name: env[credential_secret_name]}
CreateProvider(type="cloud-agents", credentials={...})
AttachSandboxProvider(sandbox=sandbox_name, provider=provider_id)
Track provider_id in self._provider_ids: dict[str, str] for cleanup
_do_destroy() calls DetachSandboxProvider and cleans up idempotently

Fallback chain (if Provider RPCs unavailable in proto):

Write credential files to /var/run/secrets/llm-credentials/ via _do_write_file()
Set matching env vars (GOOGLE_APPLICATION_CREDENTIALS=/var/run/secrets/llm-credentials/..., etc.)
For simple API key providers: set env var directly
Fail closed for providers that require file-backed credentials if file write fails — no silent degradation

Pre-implementation: Verify CreateProvider/AttachSandboxProvider exist in openshell._proto.openshell_pb2.

3. Read-only filesystem via SandboxPolicy

When read_only=True in _do_spawn():

Set spec.policy.filesystem.read_only = ["/"]
Set spec.policy.filesystem.read_write = ["/tmp", "/home/agent", "/var/log", "/app/skills/", "/var/secrets/mcp/", "/var/run/secrets/llm-credentials/"]
Set spec.policy.filesystem.include_workdir = True
Uses the same spec.policy object where _build_network_policy() already populates network rules

The read_write list includes all post-create injection targets so that skills loading, MCP secret injection, and credential writes succeed in advisory mode.

4. MCP secret mounts via file injection

Add _inject_mcp_secrets() method:

For each (secret_name, key, mount_path) tuple, resolve the secret value from runner env
Create parent dirs: exec_stream(sandbox_id, ["mkdir", "-p", mount_path])
Write to f"{mount_path}{key}" (mount_path is the directory, not the file)
Matches the existing contract where LIGHTSPEED_MCP_SERVERS headers reference file:/var/secrets/mcp/<server>/<key>

5. TLS certs — skip with info log

When tls_certs is not None, log info: "TLS certs not needed for OpenShell — gateway provides transport security." Use port 8080 (not 8443). Per spike doc analysis, the gateway terminates TLS at the gRPC boundary.

6. Service account — skip with info log

Log info that OpenShell manages identity. No K8s SA equivalent needed.

7. JWT workaround — document as known limitation

Keep _inject_podman_token() workaround. Add docstring noting upstream fix path: OpenShell Podman driver should use secret_env instead of file-based secret mounts to work around the Podman 5.8.x REST API bug. Not fixable in our code.

8. Tests

TestSkillsLoading — verify extraction + streaming, both Podman and K8s paths, skills_paths subdirectory selection preserved
TestCredentialInjection — verify gRPC CreateProvider + AttachSandboxProvider called, provider_id tracked for cleanup; test file-based fallback for Vertex
TestReadOnlyFilesystem — verify SandboxPolicy.filesystem populated with full read_write allowlist including injection targets
TestMCPSecretInjection — verify mkdir + write to f"{mount_path}{key}", not mount_path
TestTLSSkipped — verify info log, port remains 8080
TestServiceAccountSkipped — verify info log

Verification

Unit tests pass on all 8 items
E2E on EC2 with OpenShell gateway: workflow with skills_image + credentials
Read-only mode sets filesystem policy correctly (visible in gateway logs)
MCP secret files written to correct paths and readable by sandbox agent
Vertex-style file-backed credentials work end-to-end

Triage

Summary: Close production gaps in OpenShellSpawner — 7 features currently silently ignored need proper handling (skills, credentials, filesystem policy, MCP secrets, TLS, service account, JWT workaround).

Affected components:

src/cloud_agents/spawner/openshell_spawner.py — all 7 gaps are in _do_spawn() signature (lines 484-490) and _do_write_file() (line 734)
src/cloud_agents/workflow/temporal_activities.py — passes all features (lines 426-433)
tests/unit/spawner/test_openshell_spawner.py — new test classes needed

Related issues/PRs:

Migrate OpenShellSpawner to v0.0.78+ SDK API #95 — SDK migration (closed, foundation)
OpenShellSpawner: Use gateway ExposeService for HTTP routing instead of direct container access #98 — ExposeService routing (closed, foundation)
OpenShellSpawner: Auto-derive per-step network policy from workflow config #101 — Network policy auto-derive (closed, PR RSPEED-0000: Parallel-safe OpenShell spawner + auto-derive network policy #102)
PR RSPEED-0000: Parallel-safe OpenShell spawner + auto-derive network policy #102 — parallel-safe spawner (merged)

Suggested labels: enhancement, area/spawner

Plan review:

✅ Gap analysis table is accurate — all 7 features confirmed in spawner signature (_do_spawn lines 484-490) and activity layer (lines 426-433)
✅ _do_write_file() already exists at line 734 — reusable for MCP secret injection
✅ Raw gRPC pattern for Provider API matches existing _expose_service() approach
✅ Filesystem policy uses the same spec.policy object as _build_network_policy() — clean extension
✅ TLS/service_account skip-with-log is the right call per spike doc analysis
✅ Test plan covers all 7 items with specific test class names
⚠️ Skills loading via exec_stream + tar is the most complex item. The spawner would need to pull the skills image locally first (via Podman or container runtime), then stream contents into the sandbox. Worth specifying whether this uses the host's Podman CLI (like _inject_podman_token) or the OpenShell SDK.
⚠️ Provider API (CreateProvider/AttachSandboxProvider) — confirm these RPCs exist in the OpenShell proto package. The issue states "no Python SDK wrapper" but doesn't confirm the proto messages are available at openshell._proto.openshell_pb2.

Non-blocking ⚠️ only. Plan approved.

🤖 Triaged by beesarmy via Claude Code

One thing I'd tighten in the plan: mcp_secret_mounts currently carries (secret_name, key, mount_path) where mount_path is the directory /var/secrets/mcp/<server>/, while LIGHTSPEED_MCP_SERVERS encodes the header file ref as /var/secrets/mcp/<server>/<key>. So _inject_mcp_secrets() cannot just call _do_write_file() on mount_path; it needs to write to f"{mount_path}{key}" (or change the tuple contract), and the helper should create parent dirs first because those directories do not exist today.

Without that, the plan matches the high-level intent but misses the current activity/spawner contract details.

Related advisory-mode gotcha: if we implement post-create injection for skills and MCP secrets, the SandboxPolicy.filesystem.read_write allowlist probably needs to include the destinations we write after sandbox creation, not just /tmp, /home/agent, and /var/log.

Today the planned writes would target at least /app/skills and /var/secrets/mcp/...; if credential files end up under /var/run/secrets/llm-credentials/, that path likely belongs there too. Otherwise the copy/injection helpers may work in normal mode but fail specifically when read_only=True.

On credential providers, I think the plan needs one more contract decision before implementation. temporal_activities.py passes credential_secret_name=_to_k8s_secret_name(provider.get("credentials_secret")), but the actual secret value is only present in env under the original key (for example OPENAI_API_KEY). For Provider RPCs we likely need a concrete credentials map and maybe multiple keys, not just the normalized secret-name string.

I'd either (a) derive the provider payload explicitly from env in the spawner and document which keys are forwarded, or (b) change the spawn contract to pass a credential payload object instead of only credential_secret_name. Also worth tracking attached provider_ids separately so spawn-failure cleanup and _do_destroy() can detach/delete them idempotently.

Re-triage (author update)

Three good refinements that tighten the plan:

MCP secret mount path — correct. mount_path is the directory, file needs f"{mount_path}{key}". Parent dir creation needed. Will note for implementation.
Read-write allowlist for advisory mode — valid catch. When read_only=True, the filesystem.read_write list needs /app/skills/, /var/secrets/mcp/, and credential paths in addition to /tmp, /home/agent, /var/log. Otherwise post-create injection fails specifically in advisory mode.
Credential contract — the most architectural point. credential_secret_name is just the K8s secret name, but the Provider API needs actual key-value pairs. Two options:
- (a) Derive from env in spawner — spawner knows which env vars are credentials
- (b) Change spawn contract to pass credential payload object
Option (a) is more pragmatic (no ABC change), option (b) is cleaner long-term. Either way, provider_id tracking for cleanup is essential.

Plan remains approved with these refinements baked in before coding.

🤖 Triaged by beesarmy via Claude Code

Plan update — incorporating feedback

Refinement 1: MCP secret mount path (jameswnl)

mount_path is the directory (/var/secrets/mcp/<server>/), not the file. The write target is f"{mount_path}{key}". _inject_mcp_secrets() will:

Create parent directories via exec_stream(sandbox_id, ["mkdir", "-p", mount_path])
Write to f"{mount_path}{key}" (not mount_path)

Refinement 2: Read-write allowlist for advisory mode (jameswnl)

When read_only=True, filesystem.read_write must include all post-create injection targets:

/tmp, /home/agent, /var/log (agent workspace)
/app/skills/ (skills injection)
/var/secrets/mcp/ (MCP secret injection)
/var/run/secrets/llm-credentials/ (credential files, if applicable)

Otherwise post-create writes fail specifically in advisory mode.

Refinement 3: Credential contract (jameswnl + beesarmy)

Going with option (a) — derive from env in spawner:

_do_spawn already receives the full env dict which contains the actual credential values (e.g., OPENAI_API_KEY=sk-...)
credential_secret_name tells us which env key holds the credential
Spawner builds the Provider credentials map: {credential_secret_name: env[credential_secret_name]}
Track provider_id in self._provider_ids: dict[str, str] for cleanup
_do_destroy() calls DetachSandboxProvider + optionally deletes the provider

No ABC change needed — keeps the existing spawn contract.

beesarmy ⚠️: Skills loading host runtime

Skills image extraction will use the host's Podman CLI (same as _inject_podman_token):

podman run --rm -v /tmp/skills-<name>:/out <skills_image> cp -r <paths> /out/
Then exec_stream(sandbox_id, ["tar", "xf", "-", "-C", "/app/skills"], stdin=tar_bytes)
When podman_cli is None (K8s mode), fall back to _do_write_file() per-file

beesarmy ⚠️: Proto confirmation

Will verify CreateProvider / AttachSandboxProvider exist in openshell._proto.openshell_pb2 before implementation. If missing, fall back to env var injection with a logged warning.

🤖 Updated by Claude Code

Plan update — cross-platform requirement

Skills loading must work on both Podman and K8s deployments. Updated approach:

Cross-platform skills loading:

Runner extracts skills image content to a local temp dir:
- Podman mode (podman_cli set): podman run --rm -v /tmp/skills-<name>:/out <image> cp -r <paths> /out/
- K8s mode (podman_cli is None): Use crictl or spawn a short-lived K8s Job to extract, or bundle skills as a tar in the runner image
Create tarball from extracted content
Stream into sandbox via exec_stream(sandbox_id, ["tar", "xf", "-", "-C", "/app/skills"], stdin=tar_bytes)

Step 3 is identical regardless of deployment — only the extraction method (step 1) varies by platform. This is the same pattern as the existing spawner ABC where _do_spawn implementations differ but the contract is the same.

This cross-platform principle applies to all features in this issue — no feature should silently fail on either deployment target.

🤖 Updated by Claude Code

Re-review — plan-approved removed

Removing plan-approved per user request. The plan is missing K8s deployment considerations.

K8s deployment gaps

The entire plan assumes the OpenShell gateway runs with the Podman driver (DooD pattern on RHEL). But OpenShell also supports a K8s driver (line 77 of spike doc: "K8s driver requires agents.x-k8s.io CRD installed in the cluster"). The plan needs to address how each feature works when OpenShell runs on K8s:

Skills image — The plan proposes extracting skills via Podman CLI + exec_stream tar. On K8s, this Podman CLI approach won't work. The K8s spawner uses init containers (skills_image as an initContainer at kubernetes_spawner.py:167-185). OpenShell's K8s driver may support init containers or volume mounts natively — this needs investigation.
Credential injection — The K8s spawner mounts credentials via K8s Secrets projected into pods (kubernetes_spawner.py:222-241). The plan proposes raw gRPC CreateProvider/AttachSandboxProvider RPCs. Does the K8s driver handle Provider credentials differently than the Podman driver? Are secrets mounted as volumes or injected as env vars?
MCP secret mounts — The K8s spawner uses K8s Secret volume mounts. The plan proposes _do_write_file() post-creation injection. On K8s, secrets might be better handled via projected volumes (same pattern as credential_secret_name).
JWT workaround — The _inject_podman_token() workaround is correctly noted as "No-op on K8s path" (spike doc line 151). But the plan should explicitly state what happens on K8s: does the K8s driver handle JWT injection via projected SA tokens automatically?
Network policy — The _build_network_policy() uses SandboxPolicy. Does the K8s driver enforce network policy differently (e.g., via K8s NetworkPolicy resources instead of the sandbox policy object)?
Service account — The plan says "No K8s SA equivalent needed" but the K8s spawner passes service_account to pod specs (kubernetes_spawner.py:46). If OpenShell's K8s driver creates pods, it may need SA configuration.

What the plan should add

A section on K8s driver compatibility for each feature
Whether the OpenShell Python SDK or raw gRPC RPCs behave differently on K8s vs Podman drivers
Whether any features can leverage K8s-native primitives (init containers, projected volumes, NetworkPolicy) when the K8s driver is active
How the spawner detects which driver is active (or whether it should be driver-agnostic)

The plan is solid for Podman-only deployments but needs K8s driver coverage before approval.

🤖 Triaged by beesarmy via Claude Code

Re-review — K8s gaps assessment

The author's plan updates (04:25 and 04:28) were posted before my K8s re-review (04:29) — they already address most gaps. Evaluating:

K8s Gap	Status	Assessment
Skills image	✅ Addressed	Cross-platform: Podman CLI when `podman_cli` set, K8s fallback when not. Step 3 (`exec_stream` tar) is universal regardless of driver.
Credentials	✅ Addressed	Option (a) — derive from `env` dict. Driver-agnostic since `env` is always available. No K8s-specific projected volumes needed.
MCP secrets	✅ Addressed	`exec_stream` for mkdir + `_do_write_file`. Driver-agnostic — OpenShell's exec works identically on both drivers.
JWT workaround	✅ Already noted	"No-op on K8s path" per spike doc. K8s driver uses SA token exchange.
Network policy	✅ Driver-agnostic	`SandboxPolicy` is an OpenShell API object applied by the gateway — works identically regardless of K8s or Podman driver.
Service account	✅ Acceptable	OpenShell gateway manages pods, not our spawner — our SA config doesn't apply. Skip-with-log is correct.

The cross-platform principle ("no feature should silently fail on either deployment target") is the right framing.

Remaining minor gap: K8s skills extraction path is vague ("use crictl or spawn a short-lived K8s Job"). This can be refined during implementation — the architecture (extract → tar → exec_stream) is sound and the Podman path is well-specified.

Plan approved.

🤖 Triaged by beesarmy via Claude Code

Another plan-risk to pin down before implementation: the updated skills flow now depends on exec_stream(sandbox_id, ["tar", "xf", "-", "-C", "/app/skills"], stdin=tar_bytes), but this repo never uses exec_stream with stdin today. All current OpenShellSpawner call sites only do exec_stream(sandbox_id, cmd, env=...), and the spike docs only demonstrate output-oriented exec paths.

So I’d verify the Python SDK + raw proto layer actually supports streaming stdin in the sync-iterator API before committing to the tar approach. If not, the plan needs a different transport helper for archive upload. I’d also add a test that proves skills_paths survives the new path, since preserving subdirectory selection is part of the existing KubernetesSpawner / PodmanSpawner contract.

I’d also avoid the current fallback of "if Provider RPCs are missing, fall back to env var injection with a logged warning". That is probably acceptable for direct API-key providers, but it breaks the file-based credential contract used by Vertex in lightspeed-agentic-sandbox: GOOGLE_APPLICATION_CREDENTIALS is expected to point at /var/run/secrets/llm-credentials/GOOGLE_APPLICATION_CREDENTIALS, and the sandbox spec/docs call out that file mount as required.

So if Provider RPCs are unavailable, I think the safer fallback is either (a) write the credential files into /var/run/secrets/llm-credentials/ yourself and set the matching env vars, or (b) fail closed for providers that require file-backed credentials. A warning-only env fallback would violate the updated "no feature should silently fail on either deployment target" rule.

Plan update — addressing exec_stream stdin + credential fallback

1. exec_stream stdin verification

Valid concern. The current code only uses exec_stream(sandbox_id, cmd, env=...) for output-only paths. Before implementing the tar approach:

Pre-implementation spike: verify exec_stream supports stdin bytes in the SDK proto layer (ExecSandboxRequest message)
If stdin not supported: fall back to _do_write_file() per-file with recursive directory walk. Slower but uses the proven code path already at line 734
Test: add a test that skills_paths subdirectory selection is preserved (matching K8s/Podman spawner contract)

2. Credential fallback — file-based, not env-only

Agreed. Env-only fallback breaks Vertex (GOOGLE_APPLICATION_CREDENTIALS requires a file at /var/run/secrets/llm-credentials/). Updated fallback chain:

If Provider RPCs are available:

Use CreateProvider + AttachSandboxProvider (gateway-managed)

If Provider RPCs are unavailable:

Write credential files to /var/run/secrets/llm-credentials/ via _do_write_file()
Set matching env vars (GOOGLE_APPLICATION_CREDENTIALS=/var/run/secrets/llm-credentials/..., etc.)
For simple API key providers: set env var directly
Fail closed for providers that require file-backed credentials if file write fails — don't silently degrade

This satisfies the "no feature should silently fail" cross-platform rule.

🤖 Updated by Claude Code

Re-triage (latest updates)

Two good risk mitigations added:

exec_stream stdin — valid risk. Pre-implementation spike to verify ExecSandboxRequest supports stdin bytes. Fallback to _do_write_file() per-file is the right safety net (proven path at line 734). Skills path subdirectory test preserves K8s/Podman contract parity. ✅
Credential fallback — file-based, not env-only — correct call. Env-only breaks Vertex (GOOGLE_APPLICATION_CREDENTIALS requires file mount). The updated fallback chain (Provider RPCs → file write + env → fail closed for file-required providers) properly satisfies the cross-platform rule. ✅

Plan remains approved with these refinements.

🤖 Triaged by beesarmy via Claude Code

RSPEED-0000: Close OpenShell spawner production gaps — skills, creden…

RSPEED-0000: Skills image via Podman driver_config + fragility analys…

Navigation Menu

Repository navigation

OpenShellSpawner: Close production gaps — skills, credentials, filesystem policy, MCP secrets #104

Context

Gap Analysis

Plan

1. Skills image support (cross-platform)

2. Credential injection via OpenShell Provider API

3. Read-only filesystem via SandboxPolicy

4. MCP secret mounts via file injection

5. TLS certs — skip with info log

6. Service account — skip with info log

7. JWT workaround — document as known limitation

8. Tests

Verification

Triage

Re-triage (author update)

Plan update — incorporating feedback

Refinement 1: MCP secret mount path (jameswnl)

Refinement 2: Read-write allowlist for advisory mode (jameswnl)

Refinement 3: Credential contract (jameswnl + beesarmy)

beesarmy ⚠️: Skills loading host runtime

beesarmy ⚠️: Proto confirmation

Plan update — cross-platform requirement

Re-review — plan-approved removed

K8s deployment gaps

What the plan should add

Re-review — K8s gaps assessment

Plan update — addressing exec_stream stdin + credential fallback

1. exec_stream stdin verification

2. Credential fallback — file-based, not env-only

Re-triage (latest updates)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

OpenShellSpawner: Close production gaps — skills, credentials, filesystem policy, MCP secrets #104

Description

Context

Gap Analysis

Plan

1. Skills image support (cross-platform)

2. Credential injection via OpenShell Provider API

3. Read-only filesystem via SandboxPolicy

4. MCP secret mounts via file injection

5. TLS certs — skip with info log

6. Service account — skip with info log

7. JWT workaround — document as known limitation

8. Tests

Verification

Activity

beesarmy commented on Jul 11, 2026

Triage

jameswnl commented on Jul 11, 2026

jameswnl commented on Jul 11, 2026

jameswnl commented on Jul 11, 2026

beesarmy commented on Jul 11, 2026

Re-triage (author update)

jameswnl commented on Jul 11, 2026

Plan update — incorporating feedback

Refinement 1: MCP secret mount path (jameswnl)

Refinement 2: Read-write allowlist for advisory mode (jameswnl)

Refinement 3: Credential contract (jameswnl + beesarmy)

beesarmy ⚠️: Skills loading host runtime

beesarmy ⚠️: Proto confirmation

jameswnl commented on Jul 11, 2026

Plan update — cross-platform requirement

beesarmy commented on Jul 11, 2026

Re-review — plan-approved removed

K8s deployment gaps

What the plan should add

beesarmy commented on Jul 11, 2026

Re-review — K8s gaps assessment

jameswnl commented on Jul 11, 2026

jameswnl commented on Jul 11, 2026

jameswnl commented on Jul 11, 2026

Plan update — addressing exec_stream stdin + credential fallback

1. exec_stream stdin verification

2. Credential fallback — file-based, not env-only

beesarmy commented on Jul 11, 2026

Re-triage (latest updates)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions