Skip to content

fix: prevent plaintext API key leak to models.json for exec/file-source SecretRefs (#34335)#34499

Closed
lml2468 wants to merge 3 commits intoopenclaw:mainfrom
lml2468:fix/secret-ref-key-leak
Closed

fix: prevent plaintext API key leak to models.json for exec/file-source SecretRefs (#34335)#34499
lml2468 wants to merge 3 commits intoopenclaw:mainfrom
lml2468:fix/secret-ref-key-leak

Conversation

@lml2468
Copy link
Copy Markdown
Contributor

@lml2468 lml2468 commented Mar 4, 2026

What

Prevent normalizeProviders from leaking plaintext API keys into models.json when the auth source is an exec or file SecretRef.

Why

Fixes #34335 — When a provider's API key is managed via an exec-source SecretRef (e.g. macOS Keychain), two bugs cause the plaintext key to persist on disk:

  1. resolveApiKeyFromProfiles() silently returns undefined for exec/file SecretRefs (sync function cannot resolve async sources), so normalizeProviders thinks no auth is configured.
  2. mergeWithExistingProviderSecrets() unconditionally preserves any existing plaintext apiKey from a previous models.json, defeating the SecretRef security model and breaking key rotation.

How

  • resolveApiKeyFromProfiles() now returns a sentinel value ("__redacted__") for exec/file-source SecretRefs instead of undefined, signaling that auth IS configured without leaking the actual secret.
  • mergeWithExistingProviderSecrets() skips preserving old plaintext keys when the new entry uses the sentinel.
  • getCustomProviderApiKey() filters the sentinel so it never reaches provider APIs. The runtime async path (resolveApiKeyForProvider) is unchanged and correctly resolves exec SecretRefs at request time.

Design alternatives considered

A cleaner long-term approach would be adding an apiKeySource field to the models.json schema to explicitly track auth provenance, avoiding sentinel values entirely. However, this would be a schema change with broader impact, better suited for a separate RFC. The sentinel approach is minimal, backward-compatible, and sufficient for this bug fix.

Testing

  • pnpm build passes
  • pnpm check passes (180 pre-existing tsgo errors from extension deps, unchanged)
  • 6 new tests for sentinel behavior + getCustomProviderApiKey filtering
  • 32 existing model-auth tests pass
  • AI-assisted (Claude, fully tested)

lml2468 added 2 commits March 4, 2026 21:39
…cretRefs (openclaw#34335)

When a provider's API key is managed via an exec-source SecretRef (e.g.
macOS Keychain), the synchronous resolveApiKeyFromProfiles() silently
returned undefined, causing two problems:

1. normalizeProviders thought no auth was configured and skipped setting
   apiKey, but mergeWithExistingProviderSecrets unconditionally preserved
   any existing plaintext key — defeating the SecretRef security model.

2. Key rotation via Keychain had no effect because models.json kept the
   stale old key via merge.

Fix:
- resolveApiKeyFromProfiles returns '__redacted__' sentinel for exec-source
  SecretRefs instead of undefined.
- mergeWithExistingProviderSecrets skips preserving old plaintext keys when
  the new entry uses the sentinel.
- getCustomProviderApiKey filters the sentinel so it never reaches providers.
- Runtime async path (resolveApiKeyForProvider) unchanged.

AI-assisted (Claude, fully tested)
Fixes openclaw#34335
- Add file-source to sentinel check (same sync limitation as exec)
- Add getCustomProviderApiKey sentinel filtering tests
- Fix lint (no-explicit-any → typed cast)
@openclaw-barnacle openclaw-barnacle bot added agents Agent runtime and tooling size: S labels Mar 4, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 4, 2026

Greptile Summary

This PR addresses a real security bug (#34335) where plaintext API keys could be leaked into models.json when a provider's credentials are managed via an exec- or file-source SecretRef. The fix introduces a sentinel constant to signal "auth is configured but cannot be resolved synchronously," wires it into resolveApiKeyFromProfiles, guards mergeWithExistingProviderSecrets against preserving stale plain-text keys, and filters the sentinel in getCustomProviderApiKey so it never reaches provider APIs.

The core approach is sound and the primary bug is fixed. Key issues found:

  • Logic bug in mergeWithExistingProviderSecrets (src/agents/models-config.ts): The new guard only checks that newEntry.apiKey is not the sentinel, but does not check whether existing.apiKey is the sentinel. If a user previously used an exec SecretRef (causing the sentinel to be written to models.json) and then switches to a plain API key, the old sentinel in existing.apiKey passes the guard, gets placed in preserved, and overwrites the new real key after the spread — silently breaking key rotation in the opposite direction.
  • Sentinel forwarded to discovery network calls: In resolveImplicitProviders, the sentinel value can flow into buildHuggingfaceProvider and buildVllmProvider as the apiKey argument, causing unnecessary failed network requests with the sentinel as a Bearer token.
  • Import ordering: REDACTED_API_KEY_SENTINEL is declared between two import blocks in models-config.providers.ts, which is non-standard and may trigger linter import-order rules.
  • Missing regression test: The test suite does not cover mergeWithExistingProviderSecrets where existing.apiKey is the sentinel and newEntry.apiKey is a real key.

Confidence Score: 3/5

  • The primary security bug is fixed, but a logic bug in the merge step can silently overwrite a newly configured real API key with the old sentinel when a user transitions away from an exec SecretRef.
  • The sentinel mechanism correctly prevents plaintext key leaks for the main bug scenario. However, mergeWithExistingProviderSecrets does not guard against existing.apiKey being the sentinel, meaning key rotation from SecretRef back to a plain key is silently broken — the key appears configured but all requests fail with an invalid credential. This is a functional regression that would be non-obvious to debug. The other findings are minor style and efficiency issues.
  • Pay close attention to src/agents/models-config.ts — specifically the mergeWithExistingProviderSecrets function where the sentinel guard is incomplete.

Last reviewed commit: 893ec3b

Comment on lines 3 to 12
import { coerceSecretRef } from "../config/types.secrets.js";
/**
* Sentinel value written to models.json instead of a real API key when the
* auth source is an exec/keychain SecretRef that cannot be resolved
* synchronously. The runtime async path (resolveApiKeyForProvider) handles
* actual credential resolution at request time. See #34335.
*/
export const REDACTED_API_KEY_SENTINEL = "__redacted__";

import { createSubsystemLogger } from "../logging/subsystem.js";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constant declared between import statements

REDACTED_API_KEY_SENTINEL is placed between two import blocks (after the coerceSecretRef import on line 3, before the createSubsystemLogger import on line 12). While ES modules hoist imports and this doesn't cause a runtime error, it is non-standard and will likely trigger import-order lint rules. The constant should be placed after all top-level imports.

Suggested change
import { coerceSecretRef } from "../config/types.secrets.js";
/**
* Sentinel value written to models.json instead of a real API key when the
* auth source is an exec/keychain SecretRef that cannot be resolved
* synchronously. The runtime async path (resolveApiKeyForProvider) handles
* actual credential resolution at request time. See #34335.
*/
export const REDACTED_API_KEY_SENTINEL = "__redacted__";
import { createSubsystemLogger } from "../logging/subsystem.js";
import { coerceSecretRef } from "../config/types.secrets.js";
import { createSubsystemLogger } from "../logging/subsystem.js";

Then declare the sentinel after all imports:

/**
 * Sentinel value written to models.json instead of a real API key when the
 * auth source is an exec/keychain SecretRef that cannot be resolved
 * synchronously.  The runtime async path (resolveApiKeyForProvider) handles
 * actual credential resolution at request time.  See #34335.
 */
export const REDACTED_API_KEY_SENTINEL = "__redacted__";
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/models-config.providers.ts
Line: 3-12

Comment:
**Constant declared between import statements**

`REDACTED_API_KEY_SENTINEL` is placed between two `import` blocks (after the `coerceSecretRef` import on line 3, before the `createSubsystemLogger` import on line 12). While ES modules hoist imports and this doesn't cause a runtime error, it is non-standard and will likely trigger import-order lint rules. The constant should be placed after all top-level imports.

```suggestion
import { coerceSecretRef } from "../config/types.secrets.js";
import { createSubsystemLogger } from "../logging/subsystem.js";
```
Then declare the sentinel after all imports:
```ts
/**
 * Sentinel value written to models.json instead of a real API key when the
 * auth source is an exec/keychain SecretRef that cannot be resolved
 * synchronously.  The runtime async path (resolveApiKeyForProvider) handles
 * actual credential resolution at request time.  See #34335.
 */
export const REDACTED_API_KEY_SENTINEL = "__redacted__";
```

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +166 to 172
if (
typeof existing.apiKey === "string" &&
existing.apiKey &&
newEntry.apiKey !== REDACTED_API_KEY_SENTINEL
) {
preserved.apiKey = existing.apiKey;
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sentinel can be propagated forward when user switches away from SecretRef

The guard added here prevents preserving a stale plain-text key when the new entry uses the sentinel. However, it does not prevent an old sentinel value in existing.apiKey (from a previous models.json write) from overwriting a real key in newEntry.apiKey.

Failing scenario:

  1. User configures an exec SecretRef → models.json is written with apiKey set to the sentinel.
  2. User switches to a plain API key.
  3. Next refresh: newEntry.apiKey is the real key, existing.apiKey is still the old sentinel.
  4. The current guard passes (new entry is not the sentinel), so preserved.apiKey is set to the old sentinel.
  5. { ...newEntry, ...preserved } results in the sentinel silently overwriting the real key.

The fix should also check existing.apiKey !== REDACTED_API_KEY_SENTINEL so that a stale sentinel is never carried forward.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/models-config.ts
Line: 166-172

Comment:
**Sentinel can be propagated forward when user switches away from SecretRef**

The guard added here prevents preserving a stale plain-text key when the *new* entry uses the sentinel. However, it does not prevent an *old* sentinel value in `existing.apiKey` (from a previous `models.json` write) from overwriting a real key in `newEntry.apiKey`.

Failing scenario:
1. User configures an exec SecretRef → `models.json` is written with `apiKey` set to the sentinel.
2. User switches to a plain API key.
3. Next refresh: `newEntry.apiKey` is the real key, `existing.apiKey` is still the old sentinel.
4. The current guard passes (new entry is not the sentinel), so `preserved.apiKey` is set to the old sentinel.
5. `{ ...newEntry, ...preserved }` results in the sentinel silently overwriting the real key.

The fix should also check `existing.apiKey !== REDACTED_API_KEY_SENTINEL` so that a stale sentinel is never carried forward.

How can I resolve this? If you propose a fix, please make it concise.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 4, 2026

Additional Comments (1)

src/agents/models-config.providers.ts
Sentinel forwarded into network-discovery calls

When resolveApiKeyFromProfiles returns the sentinel for a HuggingFace or vLLM exec/file SecretRef profile, the value propagates into buildHuggingfaceProvider and buildVllmProvider as the apiKey argument.

  • In buildHuggingfaceProvider: any non-empty string that is not an uppercase env-var name is used directly as a Bearer token, so discoverHuggingfaceModels is called with the sentinel as auth, sending an Authorization: Bearer <sentinel> header to the HuggingFace API.
  • In buildVllmProvider: discoverVllmModels similarly forwards the sentinel as a Bearer token.

No real credential leaks, but it produces unnecessary failed requests. The sentinel should be mapped to undefined before being passed into discovery builders.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/models-config.providers.ts
Line: 1117-1126

Comment:
**Sentinel forwarded into network-discovery calls**

When `resolveApiKeyFromProfiles` returns the sentinel for a HuggingFace or vLLM exec/file SecretRef profile, the value propagates into `buildHuggingfaceProvider` and `buildVllmProvider` as the `apiKey` argument.

- In `buildHuggingfaceProvider`: any non-empty string that is not an uppercase env-var name is used directly as a Bearer token, so `discoverHuggingfaceModels` is called with the sentinel as auth, sending an `Authorization: Bearer <sentinel>` header to the HuggingFace API.
- In `buildVllmProvider`: `discoverVllmModels` similarly forwards the sentinel as a Bearer token.

No real credential leaks, but it produces unnecessary failed requests. The sentinel should be mapped to `undefined` before being passed into discovery builders.

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 893ec3bf8f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +167 to +170
typeof existing.apiKey === "string" &&
existing.apiKey &&
newEntry.apiKey !== REDACTED_API_KEY_SENTINEL
) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip preserving old sentinel when key source changes

When a provider previously wrote "__redacted__" (exec/file SecretRef) and later switches to a resolvable key source, this merge path still preserves existing.apiKey because it only checks newEntry.apiKey !== REDACTED_API_KEY_SENTINEL. Since mergedProviders[key] = { ...newEntry, ...preserved }, the stale sentinel overwrites the new key and models.json cannot recover from the placeholder in merge mode, leaving downstream models.json consumers with an unusable credential.

Useful? React with 👍 / 👎.

…inel overwrite

- Move REDACTED_API_KEY_SENTINEL after all import statements (Greptile)
- Also filter existing.apiKey === sentinel in merge logic, preventing
  a stale sentinel from overwriting a real key when user switches back
  from SecretRef to inline key (Greptile + Codex)
- Add test for stale sentinel scenario
- Update JSDoc: exec/keychain → exec/file
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5960685c42

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

typeof existing.apiKey === "string" &&
existing.apiKey &&
existing.apiKey !== REDACTED_API_KEY_SENTINEL &&
newEntry.apiKey !== REDACTED_API_KEY_SENTINEL
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Derive sentinel before preserving existing apiKey

In merge mode this guard still preserves leaked plaintext keys for explicit providers that rely on exec/file SecretRefs, because resolveProvidersForMode() runs before normalizeProviders() in ensureOpenClawModelsJson, so newEntry.apiKey is still undefined here (not the sentinel) when the merge happens. In that common path, an old plaintext existing.apiKey from models.json is copied back and never replaced, so migrating a provider from inline key to SecretRef can continue to leave the real key on disk.

Useful? React with 👍 / 👎.

typeof existing.apiKey === "string" &&
existing.apiKey &&
existing.apiKey !== REDACTED_API_KEY_SENTINEL &&
newEntry.apiKey !== REDACTED_API_KEY_SENTINEL
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the analysis. I looked at this carefully — the concern is a false positive.

The flow is:

  1. resolveProvidersForModelsJson() calls resolveApiKeyFromProfiles() which returns REDACTED_API_KEY_SENTINEL for exec/file SecretRefs → newEntry.apiKey IS the sentinel at merge time
  2. resolveProvidersForMode() → calls mergeWithExistingProviderSecrets() with sentinel already set
  3. normalizeProviders() runs after merge

The Codex analysis seems to confuse normalizeProviders (which normalizes model definitions) with resolveProvidersForModelsJson (which resolves API keys). The sentinel is set in step 1 before merge, not in step 3 after.

For explicit providers with apiKey as a SecretRef object in config: hasConfiguredApiKey is true so it skips the profiles path entirely — the object flows through and is handled by normalizeOptionalSecretInput in step 3.

@joshavant
Copy link
Copy Markdown
Contributor

Thanks for the work on this PR and for contributing to this fix path.

Closing this as superseded by #38955 (merged), which integrated the relevant fixes from this thread and completed the broader hardening scope for the secrets/models persistence cluster.

The merged work now covers source-snapshot persistence, marker/provenance auth handling, audit/apply expansion (including models.json/header residues), and write-mode hardening/tests/docs in one end-to-end fix.

Appreciate the contribution here. If there’s a remaining unique change from this PR that should still land, please call it out in a follow-up issue and reference #38955.

@joshavant joshavant closed this Mar 7, 2026
@lml2468 lml2468 deleted the fix/secret-ref-key-leak branch March 8, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

normalizeProviders leaks plaintext API keys into models.json for exec-source SecretRefs

2 participants