feat(secrets): egress-time credential injection with process-local sentinels by steipete · Pull Request #102009 · openclaw/openclaw

steipete · 2026-07-08T04:00:55Z

What Problem This Solves

SecretRefs stop provider credentials from living in openclaw.json, but the resolved plaintext still travels through the whole model-call chain: resolveApiKeyForProvider → runtime auth storage → stream options → SDK client apiKey. Any log line, error dump, proxy-capture file, or introspection surface along that chain can leak the real key. Two packages/ai adapters (Anthropic, OpenAI responses) also constructed SDK clients without the host guarded fetch, so there was no in-repo interception point at all for those requests. The secrets docs punt the remaining risk to "an external credential proxy".

Fixes #102008.

Why This Change Was Made

Egress-time injection closes the gap inside the runtime instead of requiring an external broker:

Sentinels: SecretRef-managed model credentials become opaque per-process values (oc-sent-v1-<hex>) at model-auth resolution. Auth storage, stream options, SDK config, logs, and error objects only ever hold sentinels. Provider auth exchanges unwrap for the exchange and re-sentinel the result.
Egress swap: the guarded model fetch substitutes known sentinels in header values and the URL (query-param auth included) immediately before the SSRF-guarded send. Unknown sentinel-shaped values fail closed before any network activity.
Adapter convergence: every packages/ai provider now uses the latest injection point its SDK supports — custom fetch (Anthropic, OpenAI responses/completions, Azure, Mistral) or construction-time unwrap via a new inert host port (@google/genai exposes no fetch hook; evidence in adapter comments). Key-shape auth routing (e.g. sk-ant-oat detection) resolves through the host port without exposing values.
Exact-value redaction registry: every resolved secret value (and its percent-encoded form, which URL egress produces) is registered and masked by redactSensitiveText/redactSecrets/proxy-capture, even when pattern redaction is off — defense in depth if plaintext ever reaches a log path.
Also fixes a pre-existing unhandled-rejection race in capNonOkResponseBodyLazily (pipeThrough writer leak on cancellation/cap races) that the new tests exposed deterministically.

Plugin SDK auth surfaces (getApiKeyForModel / resolveApiKeyForProvider runtime methods) keep returning plaintext — shipped contract; plugins run their own HTTP clients. Internal call sites opt in via secretSentinels: true. Flipping the default once the plugin SDK grows a sentinel-aware helper is a named follow-up.

User Impact

No config changes. SecretRef-backed model credentials (auth profiles and models.providers.*.apiKey paths) are protected automatically.
OPENCLAW_SECRET_SENTINELS=off|0|false kill switch for incident response; exact-value redaction stays on.
Plain env credentials (no SecretRef) are unchanged and out of scope.
Docs: new "Egress-time injection (sentinels)" section in docs/gateway/secrets.md with the exact threat model (this is not process isolation; same-process memory still holds the value at the final adapter boundary).

Evidence

Focused Vitest (macOS, Node 24): src/secrets/sentinel.test.ts (10), src/logging/redact.test.ts (115), src/agents/provider-transport-fetch.test.ts (88, 3 consecutive runs, zero unhandled errors), integration test proving a local HTTP server receives the real header while the caller held only the sentinel and logs redact the exact value.
Full focused matrix (27 files, 900+ tests), corepack pnpm build, and pnpm check:changed green on Blacksmith Testbox during development.
Live test (real user path): isolated OPENCLAW_HOME/OPENCLAW_STATE_DIR, models.providers.anthropic.apiKey as a file SecretRef, openclaw agent --local -m "Reply with exactly the single word: pineapple" --model anthropic/claude-haiku-4-5 → reply "pineapple". Leak sweep: raw key present only in the staged key file; zero occurrences in the debug log, generated models.json, session/trajectory files, or any state artifact.
Structured review (Codex gpt-5.5) run to clean; its one accepted finding (percent-encoded URL form escaping exact-value redaction) is fixed in this PR (encoded-form registration + pre-swap URL in the model-fetch debug line, with regression test).

Adapter decision table (SDK fetch support verified against installed sources):

Adapter	Path
Anthropic, OpenAI responses/completions, Azure responses, Mistral	host guarded fetch (SDK `fetch`/`HTTPClient.fetcher`)
Google Gemini, Google Vertex	construction unwrap (`@google/genai` has no fetch hook)
ChatGPT responses (in-house transport), plugin-stream/harness handoffs	final core-owned unwrap
Cloudflare AI Gateway	delegated guarded fetch; composed `cf-aig-authorization` header covered by substring swap + test

Non-goals in this PR: channel tokens, MCP env/headers, skills env, sandbox egress proxying for tool traffic (tracked as follow-ups in the issue).

…ess-local sentinels SecretRef-managed model-provider credentials now travel as opaque oc-sent-v1 sentinels through auth storage, stream options, and SDK config; the guarded model fetch injects real values into headers and URLs immediately before the SSRF-guarded send and fails closed on unknown sentinels. packages/ai adapters converge on the host guarded fetch where the SDK supports custom fetch and unwrap at construction where it does not. Resolved values (and their percent-encoded forms) register for exact-value log redaction. Kill switch: OPENCLAW_SECRET_SENTINELS=off. Also fixes a pre-existing unhandled rejection race in capNonOkResponseBodyLazily (pipeThrough writer leak).

clawsweeper · 2026-07-08T04:02:39Z

Codex review: found issues before merge. Reviewed July 8, 2026, 4:07 AM ET / 08:07 UTC.

Summary
The branch adds process-local SecretRef sentinels, egress-time credential injection, exact-value redaction, provider adapter changes, docs, and tests for model-provider credentials.

PR surface: Source +1101, Tests +1597, Docs +15. Total +2713 across 60 files.

Reproducibility: yes. Source inspection shows PR-head sentinel headers are swapped into baseInit before ensureModelProviderLocalService, and the local-service probe sends those headers with global fetch().

Review metrics: 3 noteworthy metrics.

Env security knob: 1 added. OPENCLAW_SECRET_SENTINELS changes operator-facing security behavior and should be intentional before merge.
Local-service preflight path: 1 path affected. The health probe can reuse model request headers before the guarded provider egress path runs.
Provider SDK auth seams: 6 provider families touched. Anthropic, OpenAI, Azure, Mistral, Google, and ChatGPT paths have different fetch or construction-time credential semantics.

Stored data model
Persistent data-model change detected: persistent cache schema: src/agents/embedded-agent-runner/google-prompt-cache.test.ts, persistent cache schema: src/agents/embedded-agent-runner/google-prompt-cache.ts, serialized state: src/agents/embedded-agent-runner/google-prompt-cache.test.ts, serialized state: src/proxy-capture/runtime.test.ts, vector/embedding metadata: packages/ai/src/providers/google-client-auth.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #102008
Summary: The linked issue is the canonical SecretRef plaintext model-call-chain report, and this PR is its open fix candidate but remains blocked by the local-service egress-boundary flaw.

Members:

canonical: SecretRef credentials travel as plaintext through the model-call chain after resolution #102008 - The PR uses closing syntax for this issue and implements the sentinel/egress-injection design proposed there.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🐚 platinum hermit
Patch quality: 🧂 unranked krab
Result: blocked by patch quality or review findings.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P2] Fix the local-service health-probe header path so sentinels are not unwrapped before a guarded egress boundary.
[P2] Add focused regression coverage for sentinel-backed auth headers with localService health probes.
[P2] Get security/owner review after the probe fix because this changes cross-provider credential handling.

Risk before merge

[P1] Local-service providers can receive plaintext SecretRef-backed auth headers during the health probe before the guarded model request, so the egress-time injection boundary is incomplete.
[P1] The PR changes auth resolution, runtime auth storage, provider SDK construction, guarded fetch, redaction, and proxy capture across multiple providers; green CI does not settle upgrade behavior by itself.
[P1] The new OPENCLAW_SECRET_SENTINELS kill switch adds an operator-facing security-boundary env surface that needs deliberate maintainer acceptance.

Maintainer options:

Keep health probes sentinel-safe (recommended)
Fix the local-service preflight so SecretRef-backed credentials are not unwrapped into ensureModelProviderLocalService before the guarded provider egress path.
Accept sidecar exposure deliberately
Maintainers may decide local-service health probes are part of the trusted egress boundary, but that tradeoff should be documented and security-reviewed before merge.
Pause for staged design
If the local-service and SDK boundaries are too broad for one PR, pause this branch and split the hardening into narrower reviewed steps.

Next step before merge

[P2] The PR has a protected maintainer/security-boundary signal and needs author or maintainer follow-up on the local-service egress boundary before normal merge review can continue.

Maintainer decision needed

Question: Should this PR proceed with the process-local sentinel design after the local-service probe is made sentinel-safe, or should maintainers require a narrower staged hardening plan first?
Rationale: The linked issue is source-supported and this PR is a plausible core security fix, but it changes cross-cutting credential handling and currently leaves one concrete pre-guarded egress path.
Likely owner: joshavant — They have the strongest current history on SecretRef and auth-profile resolution, which is the first security boundary this decision changes.
Options:
- Fix probe boundary and continue (recommended): Require the local-service health-probe path to preserve sentinels or unwrap only inside a guarded egress helper, then continue reviewing this PR as the canonical fix.
- Stage a smaller hardening PR: Ask for exact-value redaction and guarded-fetch convergence first, leaving full sentinels until provider and local-service semantics are approved.
- Keep the current process boundary: Do not merge this sentinel approach now and keep the documented same-process plaintext limitation until a broader broker or plugin API design exists.

Security
Needs attention: The diff improves SecretRef handling broadly, but the guarded-fetch/local-service interaction leaves a concrete credential egress gap.

Review findings

[P1] Keep local-service probes behind the sentinel boundary — src/agents/provider-transport-fetch.ts:891

Review details

Best possible solution:

Keep this PR as the canonical candidate, but make local-service health probes sentinel-safe before merge and require maintainer security review of the cross-provider credential boundary.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows PR-head sentinel headers are swapped into baseInit before ensureModelProviderLocalService, and the local-service probe sends those headers with global fetch().

Is this the best way to solve the issue?

No, not yet. The process-local sentinel design is a plausible core fix, but this patch unwraps credentials early enough for local-service health probes to bypass the intended guarded egress boundary.

Full review comments:

[P1] Keep local-service probes behind the sentinel boundary — src/agents/provider-transport-fetch.ts:891
The swap at this line feeds unwrapped headers into baseInit, and baseInit?.headers is then passed to ensureModelProviderLocalService before fetchWithSsrFGuard. For a localService model using a SecretRef-backed Authorization header, the health probe sends plaintext with global fetch(), outside the guarded egress/capture boundary this PR documents.
Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 4ae8d735bf54.

Label changes

Label justifications:

P0: This PR is the candidate fix for a credential-exposure issue and currently leaves a high-severity SecretRef provider-auth egress gap in its own security boundary.
merge-risk: 🚨 compatibility: The new sentinel default and env kill switch change runtime auth behavior for SecretRef-backed model credentials and need upgrade/operator confidence.
merge-risk: 🚨 auth-provider: The patch changes provider credential resolution and SDK handoff paths across multiple model providers, including fail-closed sentinel handling.
merge-risk: 🚨 security-boundary: Merging as-is would claim egress-time credential injection while a local-service health probe can still receive unwrapped credentials before guarded fetch.
rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🐚 platinum hermit and patch quality is 🧂 unranked krab.
status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The PR body includes terminal proof of a real SecretRef-backed Anthropic run plus a leak sweep showing the raw key only in the staged key file, though that proof does not cover the local-service boundary.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal proof of a real SecretRef-backed Anthropic run plus a leak sweep showing the raw key only in the staged key file, though that proof does not cover the local-service boundary.

Evidence reviewed

PR surface:

Source +1101, Tests +1597, Docs +15. Total +2713 across 60 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	35	1276	175	+1101
Tests	24	1616	19	+1597
Docs	1	16	1	+15
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	60	2908	195	+2713

Security concerns:

[high] Plaintext SecretRef headers can reach local-service probes — src/agents/provider-transport-fetch.ts:891
baseInit.headers can contain unwrapped SecretRef credentials and is passed to ensureModelProviderLocalService before guarded fetch; the local-service probe then uses global fetch() with those headers.
Confidence: 0.88

What I checked:

Live PR state: This PR is open at head df1c8ce, carries the protected maintainer label, and links to the canonical SecretRef plaintext credential issue. (df1c8ce9f667)
Current main still has the original plaintext handoff: Current main returns resolved SecretRef profile material as apiKey: resolved.apiKey, so the central issue is not already implemented on main. (src/agents/model-auth.ts:1065, 4ae8d735bf54)
PR sentinelizes SecretRef-backed profile credentials: The PR changes profile auth resolution so callers that opt into secretSentinels receive a sentinelized API key for SecretRef-backed credentials. (src/agents/model-auth.ts:1185, df1c8ce9f667)
Blocking pre-guard unwrap: The PR swaps sentinels into plaintext URL/header values, stores the swapped headers in baseInit, and then passes those headers to local-service setup before fetchWithSsrFGuard runs. (src/agents/provider-transport-fetch.ts:891, df1c8ce9f667)
Local-service probe sends supplied headers with global fetch: ensureModelProviderLocalService builds health probe headers from model and request headers, then probeHealth sends them through global fetch(), outside the guarded provider egress seam. (src/agents/provider-local-service.ts:232, df1c8ce9f667)
Tests confirm both halves but not the unsafe composition: The new tests assert sentinel swaps reach guarded fetch params, and an existing local-service test asserts request headers are passed to health probes; there is no regression test preventing a sentinel-backed header from becoming plaintext before the probe. (src/agents/provider-transport-fetch.test.ts:190, df1c8ce9f667)

Likely related people:

joshavant: Recent history includes managed SecretRef provider auth and multiple SecretRef runtime/config commits, which are the first boundaries this PR changes. (role: recent auth and SecretRef contributor; confidence: high; commits: 9386d6214fec, 1769fb2aa1d6, 8944b75e1608; files: src/agents/model-auth.ts, src/secrets/runtime.ts, docs/gateway/secrets.md)
vincentkoc: Current blame and recent history cover the local-service probe and provider-transport paths involved in the blocking preflight issue. (role: recent provider transport and local-service contributor; confidence: high; commits: 5d6ac2308657, 3dbe98a1f737, e085fa1a3ffd; files: src/agents/provider-local-service.ts, src/agents/provider-transport-fetch.ts, packages/ai/src/providers/anthropic.ts)
steipete: They authored this PR and have prior merged work around the extracted AI provider package and provider transport surfaces touched here. (role: linked fix candidate author and adjacent provider contributor; confidence: medium; commits: df1c8ce9f667, 062f88e3e3af, 8ff61be8d621; files: src/agents/provider-transport-fetch.ts, packages/ai/src/providers/anthropic.ts, packages/ai/src/providers/openai-responses.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (3 earlier review cycles)

reviewed 2026-07-08T04:45:13.022Z sha df1c8ce :: found issues before merge. :: [P1] Keep local-service probes behind the sentinel boundary
reviewed 2026-07-08T05:31:00.370Z sha df1c8ce :: found issues before merge. :: [P1] Keep local-service probes behind the sentinel boundary
reviewed 2026-07-08T07:13:44.009Z sha df1c8ce :: found issues before merge. :: [P1] Keep local-service probes behind the sentinel boundary

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 84b522cb8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-08T11:54:23Z

+function redactCaptureText(value: string): string {
+  return redactRegisteredSecretValues(value, () => REDACTED_CAPTURE_HEADER_VALUE);
+}


Redact registered secrets from WebSocket captures too

With debug proxy capture enabled, the new exact-value redaction is only wired into the HTTP capture path; the WebSocket path (captureWsEvent) still persists params.url, params.payload, and params.errorText directly. That means any registered SecretRef value that reaches a WS URL/frame/error in no-fetch realtime/WebSocket provider paths can still be stored in capture_events or payload blobs in plaintext while equivalent HTTP captures are masked. Please apply the same redactCaptureUrl/redactCaptureText handling before recording WS events.

Useful? React with 👍 / 👎.

steipete · 2026-07-08T11:56:47Z

Merged via squash.

Prepared head SHA: 84b522cb8decab01e62aee192807e7e887529bf3
Landed commit: 4bf70be01a21bbec9dc4dbf60ebaeffb48ef74dd

…ntinels (openclaw#102009) * feat(secrets): resolve SecretRef model credentials at egress via process-local sentinels SecretRef-managed model-provider credentials now travel as opaque oc-sent-v1 sentinels through auth storage, stream options, and SDK config; the guarded model fetch injects real values into headers and URLs immediately before the SSRF-guarded send and fails closed on unknown sentinels. packages/ai adapters converge on the host guarded fetch where the SDK supports custom fetch and unwrap at construction where it does not. Resolved values (and their percent-encoded forms) register for exact-value log redaction. Kill switch: OPENCLAW_SECRET_SENTINELS=off. Also fixes a pre-existing unhandled rejection race in capNonOkResponseBodyLazily (pipeThrough writer leak). * test(plugin-sdk): update public surface budget

steipete requested a review from a team as a code owner July 8, 2026 04:00

openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime agents Agent runtime and tooling size: XL maintainer Maintainer-authored PR labels Jul 8, 2026

clawsweeper Bot mentioned this pull request Jul 8, 2026

SecretRef credentials travel as plaintext through the model-call chain after resolution #102008

Closed

steipete self-assigned this Jul 8, 2026

test(plugin-sdk): update public surface budget

84b522c

openclaw-barnacle Bot added extensions: lobster Extension: lobster scripts Repository scripts commands Command implementations channel: qqbot plugin: google-meet plugin: file-transfer extensions: diffs labels Jul 8, 2026

chatgpt-codex-connector Bot reviewed Jul 8, 2026

View reviewed changes

steipete merged commit 4bf70be into main Jul 8, 2026
214 of 218 checks passed

steipete deleted the feat/secret-sentinels branch July 8, 2026 11:56

clawsweeper Bot mentioned this pull request Jul 8, 2026

fix(cron): mirror runner env proxy in model preflight #102116

Open

github-actions Bot mentioned this pull request Jul 8, 2026

📡 Upstream Digest — 2026-07-08 13:44 UTC curtismercier/openclaw-mods#1238

Open

This was referenced Jul 8, 2026

[Feature]: Automatic Output Sanitization for Sensitive Data #43830

Open

Feature: Built-in pace-aware rate limiting for autonomous agents #45771

Open

clawsweeper Bot mentioned this pull request Jul 9, 2026

Add Vault SecretRef plugin #89255

Merged

amittell mentioned this pull request Jul 9, 2026

fix(agents): hard-deadline runtime auth refresh to prevent gateway deadlock #93952

Open

5 tasks

clawsweeper Bot mentioned this pull request Jul 9, 2026

fix(openai): surface chat-completions refusal as visible text #102334

Closed

This was referenced Jul 9, 2026

Debug capture leaks registered secrets and sentinels retain credential history #102388

Closed

fix(secrets): prevent capture and sentinel credential retention #102420

Merged

yetval mentioned this pull request Jul 11, 2026

fix(agents): protect provider auth exchange output by sensitivity, not input marker #104547

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(secrets): egress-time credential injection with process-local sentinels#102009

feat(secrets): egress-time credential injection with process-local sentinels#102009
steipete merged 2 commits into
mainfrom
feat/secret-sentinels

steipete commented Jul 8, 2026

Uh oh!

clawsweeper Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Uh oh!

Uh oh!

steipete commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

steipete commented Jul 8, 2026

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steipete commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

clawsweeper Bot commented Jul 8, 2026 •

edited

Loading