Repository navigation
fix(adapter-utils): redact header-style secrets in command text - #12786
superbiche wants to merge 21 commits into
Conversation
The command redaction covered `Authorization: Bearer <value>`, shell `NAME=value` assignments, and common token shapes. It did not cover a credential passed in any other header. A `curl -H "X-API-Key: <token>"` command therefore kept the token in clear in a run log. A new rule redacts the value of any header whose name contains an api-key, token, secret, or auth hint. The rule keeps an optional auth scheme in the output, so `Authorization: Bearer <value>` produces the same text as before. `Authorization: Basic <value>` is now redacted too. The value ends at the first quote, backslash, or whitespace, so the rule stops at the end of one header argument. Claude-Session: https://claude.ai/code/session_01U9PF3d9SASC9tomDRjyeVt
|
The header rule stopped at the first whitespace or quote, so a multi-part credential such as a Digest or AWS SigV4 authorization value lost only its first token. It also matched any bare word carrying a credential hint, so prose and paths like `auth: failed` or `/v1/tokens:list` were redacted. The value is now bounded by its context: to the closing quote inside a quoted shell argument, and to the end of a comma-separated `key=value` list or a single token when unquoted. A header name must be hyphenated or underscored, or be the bare `authorization` or `apikey`; the `www-authenticate` and `proxy-authenticate` challenge headers are excluded. The recognized scheme list follows the IANA registry plus `AWS4-HMAC-SHA256` and `Token`. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
A double-quoted header value stopped at the first backslash, so a credential with an embedded escaped quote such as `"X-API-Key: abc\"def"` kept its tail in the recorded text. The double-quoted branch now consumes escape pairs and requires an unescaped opening quote, which keeps it off a serialized diagnostic where `\"` is the JSON escape. The single-quoted branch takes a backslash literally. Only the unquoted branch still stops at a backslash. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
…r value A backslash-newline continuation inside a double-quoted header argument ended the quoted match, so the unquoted fallback redacted only the part of the credential before the continuation. The double-quoted branch now treats the continuation as part of the value, with LF and CRLF line endings. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
…ed command A serialized command writes a double-quoted header argument with escaped quotes. The escaped opener fell through to the unquoted branch, which stops at the first backslash, so a multi-part credential such as a Digest value kept its later fields. A fourth branch mirrors the double-quoted one over `\"` delimiters and consumes the doubled escape sequences an embedded quote or backslash becomes. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
A header value is the rest of its shell word, which can concatenate unquoted, double-quoted, single-quoted, ANSI-C-quoted, and backslash-escaped segments. The rule now consumes every segment of that word before writing one placeholder, stops at whitespace and shell metacharacters so the next argument survives, and still redacts a run-log line truncated inside a quoted value. The recognized scheme list gains the registered `Concealed` scheme, and a quoted Digest parameter may carry HTTP quoted-pairs. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
…oken The first segment of an unquoted header value may open on an escape pair, so a value whose first byte is an escaped space is consumed, while an escaped-quote opener still falls to the caller's own rules. That first segment is bounded by whitespace only: a raw HTTP diagnostic carries an opaque credential the same way, so a shell metacharacter inside it is a credential byte. Only a continuation segment after a closing quote stops at a metacharacter, which keeps a following separator or command intact. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
…elimiters A value written with escaped quotes after the colon, the form an outer shell uses to pass quote syntax to `sh -c`, had no owner: the unquoted branch declines an escaped-quote opener so the server's own authorization rule keeps its shape. A dedicated branch now redacts that value and keeps the escaped quotes, so both rules agree on the same text. The unquoted branch also keeps a value's own delimiters when the value is quoted after the colon, which makes the rule idempotent across the log-writer and UI passes. The replace callback selects prefix, opener, and closer from the defined capture groups. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
…alization depth An escaped-quoted argument carries a backslash run before its quote that doubles and grows by one with every serialization layer, so a rule that requires exactly one backslash misses a command serialized twice. Both escaped-quote branches now capture the odd backslash run of the opener and close on the same run, with a tempered body that keeps deeper embedded quotes and a dangling trailing backslash inside the value. A scheme word may precede the escaped opener as well as follow it. Every capture group is named and the replace callback reads the groups object. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
…d keep a truncated string well formed A suffix segment adjacent to a serialized quoted header argument belongs to the same shell word, so the serialized branch now takes the shell continuation after its closer. A truncated serialized argument has no closer on its line; its value then stops before a bare quote, one preceded by an even run of backslashes, which can only be the enclosing serializer's delimiter, so that string stays parseable. The closer of an escaped-quoted value must itself be unescaped, so backtracking cannot read an escaped backslash as the closer. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
A shell escape pair doubles its backslash with every serialization layer, so the continuation and opening escape-pair segments now consume the whole backslash run with the character it escapes. An escaped-space segment adjacent to a header value inside a serialized command is therefore part of the value at any depth. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
…re quote An escape-pair segment is either an even backslash run followed by a character other than a quote, the form a serialized escaped space takes, or an odd run followed by any character, a true shell escape. An even run before a quote is an escaped backslash and the quote opens a further segment of the same word, which the value consumes. A truncated escaped-quoted value runs to the end of its line again: a bare quote there may be a further segment of the word, so it is redacted rather than kept as an enclosing delimiter. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
…trailing segment The even-run escape pair now requires a character that is neither a quote nor a backslash after the run, so a run of any even length is consumed by the odd alternative as pairs plus an escaped backslash and the following quote opens a segment of the same word. The continuation may end with one unterminated quoted segment that runs to the end of the line, so a log cut inside the last segment of a header word still redacts it. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
The header-secret rule was one composite regular expression with six branches. Command text arrives at an unknown serialization depth, and the same bytes mean opposite things at depth 0 and inside a JSON string, so every generalisation made for shell text broke serialized text or the reverse. Replace the whole thing with a bounded forward scanner. The scanner keeps the candidate detector as a regular expression: a header name, its colon, and an optional auth scheme are all a pattern can decide on its own. Everything after that is scanned in code, in one pass, linear in the length of the line. The contract is union over readings, never a guess. For each candidate the scanner enumerates the plausible readings of the surrounding text, scans the header's shell word once per reading, and redacts the longest span. A reading that disagrees can only lengthen the redaction, so disagreement over-redacts and never leaks. A reading is one number: the backslash run that spells a quote at that layer, with depth 0 as the run of length zero, which collapses the shell and serialized cases into one code path. Two leaks close. A serialized adjacent double-quoted segment whose first byte is a space is now consumed with the word it belongs to, at every depth. A truncated double-quoted tail carrying an escaped quote after a closed escaped value no longer leaves its remainder in the clear; bash reads those bytes as part of the credential-bearing word. One pass settles. The placeholder that replaces a value carries no quote, no backslash and no separator, so a second pass reads the output in a state the first pass never reached. The scanner consumes now whatever such a pass would consume, which keeps the caller's chain stable. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
Promote the round-eight review matrices into repository tables so the scanner's contract is checked here rather than in a harness under /tmp. Opener kind by value kind by suffix kind by serialization depth 0 to 3, including the adjacent double-quoted segment whose first byte is a space that leaked at depths 2 and 3. Truncated tails of each quote kind after each root kind, including the escaped-quote tail whose remainder leaked. Even backslash runs of 2, 4, 6 and 8 before a quote, a plain character, a space and a line end. Serialized rows assert the marker is gone, the output is stable, and the output still parses as the JSON string it arrived as and decodes to the redacted shell text. Truncated rows assert removal and stability only: a serialized string cut inside the header argument loses its outer delimiter, which is the contract's known over-redaction and never keeps a credential. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
The growth loop settles because the placeholder carries no byte that stops a scan: no quote, no backslash, no whitespace, no backtick, no metacharacter. Every caller passes `***REDACTED***`, so the assumption holds today, but it is an assumption about a parameter and it belongs next to the loop that depends on it. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
A single quote and an ANSI-C `$'` are what JSON serialization leaves alone, so unlike a double quote they name no layer. The reading set gave them depth 0 alone, so a serialized double-quoted segment adjacent to the closing quote was read as an escaped quote followed by a word-ending space, and its bytes stayed in the clear at every depth from 1 up. Bash joins them to the header word. Seed the tail after such an argument at every layer the unquoted reading set already carries and take the longest, which is what the union policy asks for. The body stays at depth 0 on purpose: these quotes delimit the same bytes at every layer, and reading the body deeper would take an ANSI-C escape for a plain backslash and close the value early. A backslash-newline is a line continuation. The token reader already named it, but the word scan returned at it as though it were a boundary, so the bytes on the next physical line stayed in the clear after an unquoted value and after a closed quoted argument. The shell removes the pair and joins the lines into one word, so the scan follows it now. Inside a quoted part the handling is unchanged. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
Add the two matrices whose absence hid the leaks: single-quoted and ANSI-C roots, full-header and value-only, with an adjacent serialized double-quoted segment at depths 0 to 3; and line continuations with LF and CRLF after an unquoted value, after each closed quoted argument, inside a double-quoted value, and on the value's first byte. Turn the permissive tables into transformation oracles. The truncated-tail, even-run space and line-end, and fixpoint tables now assert exact output where it is determinate. Rows stay on removal and stability only where the union makes the exact output a policy artifact rather than a fact about the credential: every serialized truncated row, which loses its outer delimiter, and the adjacent segment carrying an escaped quote, where scanning the tail at every layer runs a deeper reading to the line end. Each of those carries the reason inline. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
A raw line break inside a quoted part is an ordinary byte the shell keeps in the same word, but every body scanner returned before one, so a header value whose closing quote arrived on a later physical line left the rest of the credential in the clear. Single quotes were the strongest case, since the backslash is an ordinary byte there and the very next quote closes the part wherever it falls, but the double-quoted and ANSI-C bodies had it too. A body may now cross a line break, and only when its closer really arrives. Each scanner remembers the first break it crossed and falls back to it when no closer is found, so an argument cut mid-line by a run log still ends where its line does, and nothing outside quotes changes: a bare line break still ends the word. A line break carrying a backslash run in front of it is a continuation at whatever layer wrote that run. The reader already names the depth-0 spelling; a deeper layer spells the same pair as a run this reading may divide evenly, and a text that lost a layer of escaping on the break alone spells it as a run this reading cannot place at all. The shell joins the lines in every one of them, so the word scan does too. This widens redaction where a stray quote in a value finds a closing quote on a later line: the span can now reach that quote rather than stopping at the line. With no such quote the behaviour is unchanged. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
Exact output for a single-quoted and an ANSI-C body carrying a continuation and carrying a raw line break, a value-only single-quoted body, and a double-quoted body closed two lines later, with LF and CRLF at depths 0 to 3, each checked to still parse back through its serialization layers. The bound is pinned beside them: a quoted body whose closer never arrives still ends at its own line and the next line survives. One more row covers a line break whose backslash run belongs to a layer the reading cannot place. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
|
Want your agent to iterate on Greptile's feedback? Try greploops. |
The bearer-only rule stopped its value at a quote but not at a backslash, so inside a serialized command it consumed the backslash of the escaped closing quote and the stored JSON string no longer parsed (upstream paperclipai#11037). The header scanner already covers Authorization with any scheme: it reads the serialization layer off the delimiter, keeps the scheme word, and redacts the whole header word, so a value carrying a backslash or an escaped quote is redacted whole instead of cut. The bearer rule is removed; its byte-for-byte pin stays, and four tests pin serialized commands at depths 1 to 3, a JSON object carrying the command, a backslash inside the value, and a raw escaped quote. Claude-Session: https://claude.ai/code/session_01RYigf3eMFJjey9iKRApPGE
|
Ran the redaction at The controls redact in the same run, so the four are misses, not a broken harness. Same result through The file disagrees with itself about what a secret is. It's two patches, not one, and the obvious one only gets half. I applied both and ran the matrix:
So both are needed. The honest cost of the second one. Bare Scoping, so this isn't overstated. A cookie whose own name is secret-shaped —
|
Thinking Path
Linked Issues or Issue Description
No existing issue covers the header gap. Refs #11037: the bearer part of this change also fixes that issue's bearer class, because the bearer header is now redacted by the scanner (see What Changed). I searched the repository for open pull requests matching
command-redaction. The open neighbours: #12530 (JSON truncation when secrets are escaped-quoted; it covers the CLI-option and env-assignment classes with a lookahead, and its bearer change overlaps the bearer part here), #9999 (preserve serialized JSON during redaction), #11311 (env dumps and DSN passwords), #6038, and #6373. The problem followsbug_report.yml:What happened?
An agent ran a command that passed a credential in a request header. The command text was stored and displayed with the credential in clear:
redactCommandTextreturned that string unchanged. The same holds forx-api-key,Api-Key,X-Auth-Token, and any similar header name.Expected behavior
The stored command shows the header name and hides the value, in the same way it already does for
Authorization: Bearer <value>.Steps to reproduce
abc.server/src/redaction.ts, which calls this helper.Paperclip version or commit
Branched from
masterat06c0e883f.Deployment mode
Not deployment-specific. The gap is in a pure string function in
@paperclipai/adapter-utils.Agent adapter(s) involved
Not adapter-specific (core bug). Every adapter that records a command reaches this helper.
What Changed
This section describes the final state of the branch. An earlier push carried the same rule as one large regular expression. The independent review recommended replacing that shape, so the rule is now a bounded forward scanner.
packages/adapter-utils/src/command-redaction.ts.redactCommandTextcalls it between the existing bearer rule and the CLI-option rule.redactDiagnosticTexttherefore gets it too.X-API-Key,Api-Key,X-Auth-Token,X-Paperclip-Api-Key, andProxy-Authorizationall match. The bare namesAuthorizationandapikeyalso match. A bare hint word such asauth: faileddoes not match. A path such as/v1/tokens:listdoes not match. TheWWW-AuthenticateandProxy-Authenticatechallenge headers are excluded, because theirerror="invalid_token"parameters are diagnostics.0is depth-0 shell text.1,3,7, and15are JSON depths 1 to 4. The depth-0 reading is a member of every reading set, so a malformed serialized run always has the shell reading to fall back on. A reading that disagrees can only make the span longer. Disagreement over-redacts and never leaks.$'…'part, and a backslash escape pair. Whitespace or a shell metacharacter ends the word once a segment has closed, so the next argument survives.key=valueparameter list, the shape aDigest,Concealed, or AWS SigV4 credential takes, or a token bounded only by whitespace, a quote, a backtick, or a backslash. A raw HTTP diagnostic carries an opaque credential, so a;or|inside that token is a credential byte. Only a continuation segment stops at a metacharacter.Authorization:\"Digest …\". The scheme list follows the IANA HTTP Authentication Scheme Registry as of the RFC 9729Concealedaddition, plusAWS4-HMAC-SHA256,Hawk, andToken.Authorization: Bearer <value>produces the same text as before for a well-formed bearer header. A test pins that byte for byte.Authorization: Basic …,Digest …,Concealed …, andAWS4-HMAC-SHA256 …are now redacted. None of them was redacted before.curl -H "X-API-Key: abc" https://xfrom swallowing the URL.Digest, SigV4, andConcealedcredentials; serialized commands one, two, and three layers deep withJSON.parseon the result; line continuations and multi-line quoted bodies with LF and CRLF; negatives forContent-Type,auth: failed,/v1/tokens:list,WWW-Authenticate, and an empty quoted value; the byte-for-byte bearer pin; the already-redacted shape the server chain feeds in; and idempotency over every serialized case. Seven of the new blocks are table matrices over opener kind, value kind, suffix kind, and serialization depth.Verification
Commands run on this branch:
The server suites matter here.
server/src/redaction.tscallsredactCommandTextlast in its own chain, and it pins shapes such asprefix Authorization: ***REDACTED*** suffixkeeping its suffix.The new tests are real regression guards. The file holds 112 tests, of which 101 are new relative to
master. With the test file in place and the whole source change reverted to master's file, 89 of the 101 fail. The other 12 are regression pins that pass in both states by design: the negatives, the bearer comparisons, the already-redacted shapes the server chain feeds in, and the idempotency cases.Beyond Greptile and Superagent on this pull request, the change went through two independent adversarial review chains by GPT-5.6, read-only, against the real function. Eight rounds at its highest reasoning-effort tier examined the regex design and ended in the ruling to restructure it. Three further rounds at high effort examined the scanner and converged at pass-with-residuals, and one round judged the bearer fold. The reviewer's harness used a
bash -cargv oracle to decide which bytes belong to the header word, a 398,580-input serialization matrix at depths 1 to 3, timing from 10 KB to 1 MB, and a comparison againstupstream/masterfor every caller.On that matrix the scanner leaves 23,766 marker survivals, against 29,418 for the last regex revision. Every one of the 246 rows that the scanner keeps and the regex removed was put to the shell oracle one process per row. All 246 are separate shell words, so they are boundary preservation and not credential disclosure. The matrix reports 0 non-idempotent outputs.
The scanner is linear per candidate. Across the reviewer's adversaries from 10 KB to 1 MB, every input stays within the same order as the previous regex. The largest slowdown is a malformed parameter list at 1 MB, 16.6 ms against 28.2 ms. On dense-candidate inputs the scanner is two to three times faster. One pre-existing input is super-linear: a very long hyphenated name with no colon. It lives in the unchanged candidate-name pattern and is equally slow on
upstream/master.ROADMAP.md check: I grepped
ROADMAP.mdforredaction,redact,api-key,bridge, andsandbox callback. There were no matches. This is a bug fix in an existing subsystem, so it does not overlap planned core work.Duplicate search: I searched pull requests in this repository for
command-redactionandx-api-key. No open or merged pull request adds header-name redaction.Risks
Low risk, and the failure direction is safe. The rule can only replace more text with a placeholder. It can never reveal text that was hidden before. No credential survives any case below.
JSON.parserose from 25,740 to 163,612. I looked for a leak-free narrowing and did not find one. The same bytes are at once valid JSON and valid depth-0 shell. The natural exception opens a new leak, and the review's wider search found no leak-free narrowing either. I accepted the cost deliberately. All three callers display this text and none parses it:server/src/redaction.ts,packages/adapter-utils/src/server-utils.tsthroughserver/src/adapters/utils.ts, and the twoui/src/pages/AgentDetailpages, which import the helper asredactCommandSecretText.curl -H ""X-API-Key:abc123 https://example.testcan pull the following argument into the placeholder. This is the direct consequence of the union contract.secret-store: ok, or a path segment such as/api-keys:rotate, loses its value. This is unchanged from the first push.X-API-Key:abc;echo donebecomesX-API-Key:***REDACTED*** done. This is deliberate: the same text in a raw diagnostic is a credential containing;, and a leak there is worse than a lost separator here.Authorization: Bearer abc\"def https://xredacts toAuthorization: Bearer ***REDACTED***, because the depth-1 reading sees an opener with no closer and the union takes the longer span. The old bearer rule left\"def https://xvisible. Over-redaction, pinned as such.Model Used
Claude Opus 5 (
claude-opus-5), extended thinking at high effort, for the implementation; Claude Fable 5.1 (claude-fable-5-1), extended thinking at high effort, for the design, review triage, and this description. Independent adversarial review by GPT-5.6 (gpt-5.6-sol): eight rounds at its highest reasoning-effort tier on the regex design, three rounds at high effort on the scanner, read-only.Checklist
Fixes: #/Closes #/Refs #OR (b) described the issue in-PR following the relevant issue template#NNN/github.com/paperclipai/paperclipURLs)docs/...,fix/...) and contains no internal Paperclip ticket id or instance-derived details@superbiche · contributor · drafted with Claude Opus 5 and Claude Fable 5.1.