Skip to content

Commit 9f0f997

Browse files
thymikeeclaude
andauthored
fix(fill): redact the read-back of a sensitive fill (#4014)
* fix(fill): redact the read-back of a sensitive fill (#3392) A sensitive fill (--text-stdin, --record-as) redacted only the exact literal, so the altered forms a backend reads back still came back: the unconfirmed requested/before/after evidence, the Android failure details expected/actual and observed input text, and the iOS TEXT_ENTRY_MISMATCH message, which quoted both sides. The daemon now replaces those read-back fields with the placeholder for a sensitive fill, and the runner message quotes no field text. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * fix(fill): redact read-back before literal scrubbing and in settle output Redact a sensitive fill's read-back fields before the literal is scrubbed, so a short literal cannot rewrite the verification discriminator and leak the altered value. Keep the unconfirmed evidence keys structural. Replace the diff line text and tail labels of a --settle observation with the placeholder. Run the error read-back redaction for an empty sensitive value too. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * test(fill): pin the whole-text settle redaction and the exact literal Update the provider scenario to the new contract: a sensitive fill's settle text is replaced whole. Assert the exact literal is absent from the redacted result. Document the settle behavior. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * fix(cli): keep the pinned ref on a settle line whose text was redacted A sensitive fill replaces its settle line text whole, which dropped the ref prefix the CLI formatter pinned. Lead the line with the structured ref instead. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * test(cli): move the redacted settle line test under the fill formatter Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * refactor(fill): host the read-back redaction beside the fill payload redaction The new daemon module grew the eager closure of the daemon entries by one module. Move the helpers into parameterized-recorded-fill, which both callers already load. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> --------- Co-authored-by: Claude Sonnet 5.5 <[email protected]>
1 parent 61ce711 commit 9f0f997

14 files changed

Lines changed: 402 additions & 19 deletions

File tree

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'agent-device': patch
3+
---
4+
5+
Keep the read-back of a sensitive `fill` (`--text-stdin` or `--record-as`) out of responses and errors. An unconfirmed fill returns the redaction placeholder for `requested`, `before`, and `after`, and an Android fill failure does the same for `expected`, `actual`, and the observed input text. The iOS `TEXT_ENTRY_MISMATCH` message no longer quotes the expected or observed text. With `--settle`, a sensitive fill replaces the settle diff line text and tail labels whole, so text around the value in the field is no longer shown.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TypeExecution.swift‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -112,13 +112,11 @@ extension RunnerTests {
112112
)
113113
}
114114
if textResult.verified == false {
115-
let expected = textResult.expectedText ?? ""
116-
let observed = textResult.observedText ?? ""
117115
return Response(
118116
ok: false,
119117
error: ErrorPayload(
120118
code: "TEXT_ENTRY_MISMATCH",
121-
message: "text entry read-back did not match: expected \"\(expected)\", observed \"\(observed)\"",
119+
message: "text entry read-back did not match the requested text",
122120
hint: textEntryMode == .replacement
123121
? "The app may have formatted, filtered, or consumed the entry. Inspect the field or resulting screen before retrying. If it needs correction, retry fill with the full text and --delay-ms \(TextEntryTiming.recoveryDelayMilliseconds)."
124122
: nil

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+TextEntryConfirmationTests.swift‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,7 @@ extension RunnerTests {
302302
XCTAssertFalse(didRecordXCTestFailure(since: failures))
303303
XCTAssertFalse(response.ok)
304304
XCTAssertEqual(response.error?.code, "TEXT_ENTRY_MISMATCH")
305+
XCTAssertEqual(response.error?.message, "text entry read-back did not match the requested text")
305306
XCTAssertEqual(field.value as? String, "a")
306307
let counters = app.staticTexts["agent-device-text-entry-write-backs"].label.split(separator: " ")
307308
XCTAssertTrue(counters.contains("total-edits=11"))

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+TextTypingTests.swift‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,8 @@ extension RunnerTests {
459459
XCTAssertFalse(didRecordXCTestFailure(since: failureCountBefore))
460460
XCTAssertFalse(response.ok)
461461
XCTAssertEqual(response.error?.code, "TEXT_ENTRY_MISMATCH")
462+
// A sensitive fill's value must not come back, and the observed text is an altered form of it.
463+
XCTAssertEqual(response.error?.message, "text entry read-back did not match the requested text")
462464
XCTAssertEqual(app.textFields.count, 1)
463465
XCTAssertEqual(editableTextValue(for: app.textFields.element(boundBy: 0), treatingPlaceholderAsEmpty: true), "23456")
464466
}

‎packages/daemon/src/execution/interaction/internal/interaction-common.test.ts‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ test('parameterized fill scrubs backend and nested settle echoes at the response
110110
},
111111
settle: {
112112
diff: {
113-
lines: [{ kind: 'added', text: `TextField value=prefix${placeholder}suffix` }],
113+
lines: [{ kind: 'added', text: placeholder }],
114114
},
115115
hint: `Observed ${placeholder}4`,
116116
[`echo${placeholder}key`]: true,
@@ -127,6 +127,62 @@ test('parameterized fill scrubs backend and nested settle echoes at the response
127127
expect(JSON.stringify(recordedResult)).not.toContain(secret);
128128
});
129129

130+
test('a sensitive fill replaces the text its --settle observation read back and keeps refs and counts', () => {
131+
const sessionStore = makeSessionStore();
132+
const session = makeAuthoringSession('sensitive-settle');
133+
sessionStore.publish(session.name, session);
134+
const payload = {
135+
text: '000629177',
136+
settle: {
137+
settled: true,
138+
waitedMs: 1,
139+
captures: 2,
140+
quietMs: 1,
141+
timeoutMs: 100,
142+
diff: {
143+
summary: { additions: 1, removals: 1, unchanged: 3 },
144+
lines: [
145+
{ kind: 'removed', text: 'TextField value=000 62' },
146+
{ kind: 'added', text: 'TextField value=00 062 91 77', ref: 'e4' },
147+
],
148+
},
149+
tail: [{ ref: 'e5', role: 'button', label: 'Code 00 062 91 77' }],
150+
},
151+
};
152+
153+
const response = finalizeTouchInteraction({
154+
ref: sessionStore.lookup(session.name)!,
155+
sessionStore,
156+
command: 'fill',
157+
positionals: ['id="code"', '000629177'],
158+
flags: { recordAs: 'CODE' },
159+
result: payload,
160+
responseData: payload,
161+
actionStartedAt: 1,
162+
actionFinishedAt: 2,
163+
});
164+
165+
expect(response.ok).toBe(true);
166+
if (!response.ok) return;
167+
const settled = {
168+
settle: {
169+
settled: true,
170+
captures: 2,
171+
diff: {
172+
summary: { additions: 1, removals: 1, unchanged: 3 },
173+
lines: [
174+
{ kind: 'removed', text: '${CODE}' },
175+
{ kind: 'added', text: '${CODE}', ref: 'e4' },
176+
],
177+
},
178+
tail: [{ ref: 'e5', role: 'button', label: '${CODE}' }],
179+
},
180+
};
181+
expect(response.data).toMatchObject(settled);
182+
expect(session.actions[0]?.result).toMatchObject(settled);
183+
expect(JSON.stringify([response.data, session.actions])).not.toMatch(/000629177|062 91|000 62/);
184+
});
185+
130186
test('parameterized fill scrubs concatenated backend values and object keys through the handler route', async () => {
131187
const secret = 'TOKEN123';
132188
const placeholder = '${PASSWORD}';

‎packages/daemon/src/execution/interaction/internal/interaction-common.ts‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,10 @@ import {
1313
isSensitiveFillText,
1414
sensitiveFillPlaceholder,
1515
} from '@agent-device/ad-script';
16-
import { parameterizeRecordedFillPayload } from '@agent-device/selectors/parameterized-recorded-fill';
16+
import {
17+
parameterizeRecordedFillPayload,
18+
redactFillReadBack,
19+
} from '@agent-device/selectors/parameterized-recorded-fill';
1720
import type { InteractionFinalizationOperations } from '../../../core/interaction/internal/types.ts';
1821

1922
export function finalizeTouchInteraction(params: {
@@ -124,8 +127,7 @@ function parameterizeFillPayloads(params: {
124127
result: params.result,
125128
});
126129
const placeholder = sensitiveFillPlaceholder(params.flags);
127-
return [
128-
parameterizeRecordedFillPayload(params.result, literal, placeholder),
129-
parameterizeRecordedFillPayload(params.responseData, literal, placeholder),
130-
];
130+
const redact = (payload: Record<string, unknown>) =>
131+
parameterizeRecordedFillPayload(redactFillReadBack(payload, placeholder), literal, placeholder);
132+
return [redact(params.result), redact(params.responseData)];
131133
}

‎packages/daemon/src/server/__tests__/request-router-fill-text-stdin.test.ts‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
import { expect, test } from 'vitest';
88
import fs from 'node:fs';
99
import path from 'node:path';
10+
import { AppError } from '@agent-device/kernel/errors';
1011
import { createTestDeviceInventoryGateways } from '@agent-device/device-selection/inventory-gateways-fixtures';
1112
import {
1213
localRuntimeOwner,
@@ -38,25 +39,29 @@ function fillWithStdinText(
3839
flags: Record<string, unknown> = {},
3940
fillFails?: (text: string) => Error,
4041
text = SECRET,
42+
fillResult?: Record<string, unknown>,
4143
) {
4244
return sendToWebSession(
4345
session,
4446
{ command: 'fill', positionals: ['10', '20', text], flags: { textStdin: true, ...flags } },
4547
fillFails,
48+
fillResult,
4649
);
4750
}
4851

4952
function sendToWebSession(
5053
session: SessionState,
5154
request: Pick<DaemonRequest, 'command' | 'positionals' | 'flags'>,
5255
fillFails?: (text: string) => Error,
56+
fillResult?: Record<string, unknown>,
5357
) {
5458
const sessionStore = makeSessionStore('agent-device-router-fill-text-stdin-');
5559
sessionStore.publish(session.name, session);
5660
const typed: string[] = [];
5761
const fillPoint = async (input: FillPointInput) => {
5862
typed.push(input.text);
5963
if (fillFails) throw fillFails(input.text);
64+
return fillResult;
6065
};
6166
const owner = localRuntimeOwner('web');
6267
const baseFacts = createUnavailableRuntimeFactsForTest(WEB_DESKTOP_DEVICE, owner);
@@ -312,3 +317,162 @@ test('a whitespace-only --text-stdin value collapses the error text it appears i
312317
expect(result.ok).toBe(false);
313318
if (!result.ok) expect(result.error.message).toBe('[REDACTED]');
314319
});
320+
321+
const FILL_TARGET = {
322+
resourceId: 'otp',
323+
className: 'android.widget.EditText',
324+
packageName: 'com.test',
325+
rect: { x: 0, y: 0, width: 100, height: 40 },
326+
};
327+
328+
test.each([
329+
['an unrecorded', {}, '[REDACTED]', makeSession],
330+
['a --record-as', { recordAs: 'CODE' }, '${CODE}', makeAuthoringSession],
331+
])(
332+
'%s --text-stdin fill that settles on a formatted value returns no read-back of it',
333+
async (_name, flags, placeholder, sessionFor) => {
334+
const { response } = fillWithStdinText(
335+
sessionFor('web', { device: WEB_DESKTOP_DEVICE }),
336+
flags,
337+
undefined,
338+
'000629177',
339+
{
340+
verification: 'unconfirmed',
341+
requested: '000629177',
342+
before: '000 62',
343+
after: '00 062 91 77',
344+
target: FILL_TARGET,
345+
},
346+
);
347+
348+
const result = await response;
349+
expect(result.ok).toBe(true);
350+
if (!result.ok) return;
351+
expect(result.data).toMatchObject({
352+
verification: 'unconfirmed',
353+
requested: placeholder,
354+
before: placeholder,
355+
after: placeholder,
356+
target: FILL_TARGET,
357+
});
358+
expect(JSON.stringify(result)).not.toContain('062 91');
359+
expect(JSON.stringify(result)).not.toContain('000 62');
360+
},
361+
);
362+
363+
test('an unrecorded fill that settles on a formatted value still returns its read-back', async () => {
364+
const { response } = sendToWebSession(
365+
makeSession('web', { device: WEB_DESKTOP_DEVICE }),
366+
{ command: 'fill', positionals: ['10', '20', '000629177'] },
367+
undefined,
368+
{
369+
verification: 'unconfirmed',
370+
requested: '000629177',
371+
before: null,
372+
after: '00 062 91 77',
373+
target: FILL_TARGET,
374+
},
375+
);
376+
377+
const result = await response;
378+
expect(result.ok).toBe(true);
379+
if (result.ok) expect(result.data).toMatchObject({ after: '00 062 91 77' });
380+
});
381+
382+
const ALTERED_READ_BACK_FAILURE = (text: string) =>
383+
new AppError('COMMAND_FAILED', 'Android fill verification failed', {
384+
failureReason: 'text_mismatch',
385+
expected: text,
386+
actual: '00 062 91 77',
387+
targetInput: { text: '000 62', className: 'android.widget.EditText' },
388+
actualInput: { text: '00 062 91 77', className: 'android.widget.EditText' },
389+
});
390+
391+
test.each([
392+
['an unrecorded', {}, '[REDACTED]', makeSession],
393+
['a --record-as', { recordAs: 'CODE' }, '${CODE}', makeAuthoringSession],
394+
])(
395+
'%s --text-stdin fill failure that reports an altered read-back returns no part of it',
396+
async (_name, flags, placeholder, sessionFor) => {
397+
const { response } = fillWithStdinText(
398+
sessionFor('web', { device: WEB_DESKTOP_DEVICE }),
399+
flags,
400+
ALTERED_READ_BACK_FAILURE,
401+
'000629177',
402+
);
403+
404+
const result = await response;
405+
expect(result.ok).toBe(false);
406+
if (result.ok) return;
407+
expect(result.error.details).toMatchObject({
408+
failureReason: 'text_mismatch',
409+
expected: placeholder,
410+
actual: placeholder,
411+
targetInput: { text: placeholder, className: 'android.widget.EditText' },
412+
actualInput: { text: placeholder, className: 'android.widget.EditText' },
413+
});
414+
expect(JSON.stringify(result)).not.toContain('062 91');
415+
},
416+
);
417+
418+
test('an empty --record-as fill that fails to clear returns no part of the residual field text', async () => {
419+
const { response } = sendToWebSession(
420+
makeAuthoringSession('web', { device: WEB_DESKTOP_DEVICE }),
421+
{ command: 'fill', positionals: ['10', '20', ''], flags: { recordAs: 'CODE' } },
422+
() =>
423+
new AppError('COMMAND_FAILED', 'Android fill verification failed', {
424+
failureReason: 'text_mismatch',
425+
expected: '',
426+
actual: 'residual-4471',
427+
targetInput: { text: 'residual-4471', className: 'android.widget.EditText' },
428+
actualInput: { text: 'residual-4471', className: 'android.widget.EditText' },
429+
}),
430+
);
431+
432+
const result = await response;
433+
expect(result.ok).toBe(false);
434+
if (result.ok) return;
435+
expect(result.error.details).toMatchObject({
436+
actual: '${CODE}',
437+
targetInput: { text: '${CODE}' },
438+
actualInput: { text: '${CODE}' },
439+
});
440+
expect(JSON.stringify(result)).not.toContain('residual-4471');
441+
});
442+
443+
test.each([
444+
['a one-letter value', 'a'],
445+
['a value that is the verification discriminator', 'unconfirmed'],
446+
['a value that is a response key', 'target'],
447+
])(
448+
'a --record-as fill of %s keeps the unconfirmed structure and redacts its read-back',
449+
async (_name, text) => {
450+
const { response, sessionStore } = fillWithStdinText(
451+
makeAuthoringSession('web', { device: WEB_DESKTOP_DEVICE }),
452+
{ recordAs: 'CODE' },
453+
undefined,
454+
text,
455+
{
456+
verification: 'unconfirmed',
457+
requested: text,
458+
before: 'zq7',
459+
after: 'Q7z',
460+
target: FILL_TARGET,
461+
},
462+
);
463+
464+
const result = await response;
465+
expect(result.ok).toBe(true);
466+
if (!result.ok) return;
467+
const unconfirmed = {
468+
verification: 'unconfirmed',
469+
requested: '${CODE}',
470+
before: '${CODE}',
471+
after: '${CODE}',
472+
target: FILL_TARGET,
473+
};
474+
expect(result.data).toMatchObject(unconfirmed);
475+
expect(sessionStore.get('web')?.actions[0]?.result).toMatchObject(unconfirmed);
476+
expect(JSON.stringify(sessionStore.get('web')?.actions)).not.toMatch(/zq7|Q7z/);
477+
},
478+
);

‎packages/daemon/src/server/request-router.ts‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,10 @@ import {
6868
sensitiveFillPlaceholder,
6969
} from '@agent-device/ad-script';
7070
import { createPlatformRequestScope } from './platform-request-scope.ts';
71-
import { parameterizeSensitiveString } from '@agent-device/selectors/parameterized-recorded-fill';
71+
import {
72+
parameterizeSensitiveString,
73+
redactFillFailureReadBack,
74+
} from '@agent-device/selectors/parameterized-recorded-fill';
7275
import { isConfinedToAppLease, scopeRequestSession } from '../sessions/request-admission.ts';
7376
import { redactMacOsAppLeaseResponse } from '../resources/macos-app-lease.ts';
7477
import { resolveEffectiveSessionName } from '../sessions/session-routing.ts';
@@ -590,19 +593,25 @@ function registerParameterizedFillDiagnosticValue(req: DaemonRequest): void {
590593
const ERROR_TEXT_FIELDS: ReadonlySet<string> = new Set(['message', 'hint', 'cause', 'details']);
591594

592595
/**
593-
* A backend error can quote the text a sensitive fill failed to type. Only that fill's own text is
596+
* A backend error can quote the text a sensitive fill failed to type, or the altered text it read
597+
* back, which `redactFillFailureReadBack` replaces whole. Only that fill's own text is otherwise
594598
* redacted, and only in the error's text fields: `logPath`, `diagnosticId` and the other
595599
* daemon-owned fields come back as they are, and so does every other request's error.
596600
*/
597601
function redactSensitiveFillTextInError(req: DaemonRequest, error: DaemonError): DaemonError {
598602
const text = sensitiveFillText(req);
599-
if (!text) return error;
603+
if (text === undefined) return error;
600604
const placeholder = sensitiveFillPlaceholder(req.flags);
601605
const replace = (value: string) => parameterizeSensitiveString(value, text, placeholder);
602606
return Object.fromEntries(
603607
Object.entries(error).map(([field, value]) => [
604608
field,
605-
ERROR_TEXT_FIELDS.has(field) ? mapErrorStrings(value, replace) : value,
609+
ERROR_TEXT_FIELDS.has(field)
610+
? mapErrorStrings(
611+
field === 'details' ? redactFillFailureReadBack(value, placeholder) : value,
612+
replace,
613+
)
614+
: value,
606615
]),
607616
) as DaemonError;
608617
}

0 commit comments

Comments
 (0)