Skip to content

fix(cdp): decode the base64 postData a continueRequest carries - #874

Merged
SGavrl merged 4 commits into
h4ckf0r0day:mainfrom
ntdatt812:fix/569-continue-overrides
Oct 16, 2026
Merged

SGavrl merged 4 commits into
h4ckf0r0day:mainfrom
ntdatt812:fix/569-continue-overrides

Conversation

@ntdatt812

@ntdatt812 ntdatt812 commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Decode Fetch.continueRequest.postData from CDP base64 into raw bytes before sending the request. Keep those bytes through InterceptResolution::Continue.body and the HTTP transport. Binary overrides previously reached the server as base64 text.

Addresses the remaining continuation-body corruption in #716. Related to #569; this does not claim to resolve every navigation override issue.

Current head: 343e34e. Its source tree matches the locally validated tree 8acba1ac667f3127180683b82e32a7ec4469b65c.

  • Preserve the contributor's commits and update the branch against current main.
  • Use the existing base64 dependency. Invalid binary parameters return an error without consuming the paused request, allowing a corrected command to resume it.
  • Preserve Network.loadingFailed events, header and method overrides, and URL-rewrite safety checks.
  • Keep omitted and explicitly empty request bodies distinct. Apply the same decoding to the legacy domain handler; no production creator of that handler's paused records was found.
  • Document the Rust API change: Continue.body is now Option<Vec<u8>>. Text callers use text.into_bytes(); CDP clients need no configuration change.

Validation

CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 cargo nextest run --release --features render -E 'test(continue_request_) + test(fulfill_request_) + test(fetch_resolution_) + test(intercept) + test(binary)' --no-fail-fast
CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 cargo nextest run --release --features render --no-fail-fast
CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release -p obscura-cli --bins --features render
OBSCURA_BIN=/path/to/candidate python3 obstacle-course/run.py --json --runs 1 --warmup 0
  • Focused coverage: 32 passed.
  • Full render coverage: 2,332 passed, 5 existing skips.
  • New invalid-parameter and recovery regressions failed before the repair. Existing contributor regressions cover JSON, non-UTF-8 bodies and omitted bodies.
  • Exact release build passes. Pinned obstacle course 2d08d9a8e860e423b47209434a9fba9c75e6eee6: 33/33.
  • Ten real-client controls using Playwright 1.59.1: Chromium 10/10; current main 7/10; repaired branch 9/10. These are bounded local diagnostics, not upstream suite totals.
  • Binary continuation now preserves [0,128,255,16,195,40] instead of sending the text AID/EMMo. Normal fetch/XHR, inherited Request bodies, multipart files, fulfilled binary responses, omitted and empty overrides all pass. Same-session invalid-command rejection and retry now pass too.
  • The retained failure occurs when Fetch is enabled on a raw CDP session but the fetch is initiated through Playwright's separate managed session. The raw session receives no pause. This also fails on main; current routing selects the evaluation's session rather than the session that enabled Fetch. This PR does not fix that separate routing limitation.
  • Exact-head Policy, Core and Performance CI pass. Core reports render 2,332/2,332; no-render 31/31 library and 1,394/1,394 workspace tests; stealth networking 137/137 plus the opt-in guard test. Render, render+stealth, no-render and no-render+stealth builds pass.

Rendering

Not applicable.

Performance

Intercepted body rewrites now decode base64 once and move a byte vector into the existing HTTP request. The normal request path is unchanged.

Five alternating current-main/candidate pairs after compilation and functional servers stopped, with identical successful workloads:

Workload Main median wall Candidate Main median peak RSS Candidate
5,000 DOM insertions 147.5 ms 146.2 ms 54.7 MiB 54.8 MiB
2,000 storage writes and reads 58.5 ms 56.7 ms 41.9 MiB 43.3 MiB
MCP fill, type, click and snapshot 70.14 ms 66.41 ms 39,604 KiB 39,688 KiB

MCP median CPU is 60 ms for both revisions at 10 ms resolution. Changes are within the approximate 10% noise range; the smoke gate passes. No speedup claim or Chromium performance comparison. Comparing rewritten-body speed against main would be misleading because main sends the wrong data.

Remote paired performance CI also passes: both obstacle runs report 33/33, median per-stage latency ratio 1.00x, no stage above the reporting threshold. Five-pair smoke reports DOM latency +1.5%, RSS -0.2%; storage latency -3.3%, RSS unchanged. These changes remain inside the approximate 10% noise range.

Checklist

  • The change is focused and does not remove existing behavior without justification.
  • Tests cover the failure or feature.
  • Existing tests pass, including render and no-render configurations when affected.
  • I checked for CPU, latency, and memory regressions.
  • Public API or user-facing behavior changes are documented.

@yinnho

yinnho commented Sep 6, 2026

Copy link
Copy Markdown

Right fix — we shipped the same decode in our tree when wiring #569. Two contract notes from that implementation, since this PR routes through the shared decode_base64 helper:

One, the helper returns String via from_utf8_lossy, so a base64'd binary POST (a protobuf upload, a gzipped request body, a multipart file part) comes out with U+FFFD replacement characters before into_bytes() re-serializes them — the request-side twin of #754, which this repo fixed for responses by keeping bytes as bytes. We carry postData and fulfillRequest's body as Vec<u8> from decode to wire for the same reason; the asymmetry argument in the PR description applies to the end type too, not just the decoding.

Two, the helper decodes leniently: filter_map drops every non-alphabet character and zero-pads partial chunks, so raw text sent where base64 was expected becomes quietly mangled bytes with no error anywhere — the exact "no error anywhere said so" shape the description argues against. Chrome answers these with a protocol error; we reply continueRequest postData is not valid base64: <e> (mirrored for fulfillRequest), pinned by a test feeding "!!!not-base64" to both arms. The detail that makes strictness safe: the failed resolution doesn't consume the pause, so a valid answer on the same requestId afterwards still drives the page fetch — a client mistake becomes visible and recoverable instead of wedged.

@ntdatt812

ntdatt812 commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor Author

Both notes were right, and the first one lands squarely on my own test name — it asserts "the request body must be the bytes the client encoded" while the decode still went through a String. Pushed for both.

Bytes, not String. InterceptResolution::Continue.body is Option<Vec<u8>> now. It turns out to make the consumer simpler rather than wider: override_body at ops.rs:2408 is already Option<Vec<u8>>, so override_body = body.map(String::into_bytes) becomes override_body = body. Pinned by a test that carries AP/+ — the bytes 00 FF FE, which are not valid UTF-8 — and asserts they arrive byte for byte. Under the old path they came out as three U+FFFD.

Strict, and it does not wedge. decode_base64 decoded leniently and returned String; nothing calls it now, so it is deleted rather than left beside the new one. In its place decode_optional_binary(params, field, what) returns Ok(None) for an absent field, Ok(Some(bytes)), or an error naming the field, and both arms answer a bad value with -32602 and continueRequest postData is not valid base64: … (mirrored for fulfillRequest).

The recovery property you named needed a reorder, and it is worth spelling out because the old shape was one line away from a wedge: handle_fetch_resolution did intercepted_paused.remove(request_id) before parsing, so returning an error there would have taken the pause with it and left the request unanswerable. The resolution is now built first and the pause is only taken once it is Ok. The test drives exactly that: "!!!not-base64" is rejected, nothing is sent to the page, the entry is still in the map, and a corrected continueRequest on the same requestId afterwards resolves normally.

Padding is optional in the decoder — encoders disagree about it and the group length is unambiguous without it — but a character outside the alphabet, padding with anything after it, and a final group of one symbol are all refused.

One place I did not follow you, so it is not mistaken for an oversight: fulfillRequest's body is still a String at the resolution boundary. It is not the same one-line change as postData, because that body crosses a JSON reply at ops.rs:2430 ("body": b) where a Vec<u8> serialises as an array of numbers and the page-side consumer reads a string. Making it bytes end to end means changing that reply's shape too, which is a wider change than this PR and belongs with #754's response-side work rather than bolted onto a postData fix. The strictness half is mirrored today; the byte half is the one left, and I would rather it were its own change than a silent half-move here.

Verified in a Linux container on rustc 1.98: cargo nextest run -p obscura-cdp over these arms is 8 passed, 0 failed, and cargo fmt --check reports no diff in either changed file.

Each new claim was checked by breaking it. Putting the bytes back through String::from_utf8_lossy fails only the binary test, with the corruption spelled out:

assertion `left == right` failed: a binary body must reach the wire byte for byte
  left:  Some([0, 239, 191, 189, 239, 191, 189])
 right:  Some([0, 255, 254])

239, 191, 189 is U+FFFD twice, which is your first note as a number.

Taking the pause on the error path instead of leaving it fails only the two rejection tests, on the request must stay paused. Nothing else in the selection moves in either case.

`postData` is declared `binary` in the Fetch domain, so a client sends it
base64-encoded -- the same wire type as `fulfillRequest`'s `body` four
lines below, which is decoded. This arm passed the string through
unchanged and `op_fetch_url` turns it into bytes with `String::into_bytes`,
so the request body became the base64 text itself: a server reading a
rewritten POST got neither the JSON nor the form the client meant to send.

Decode it the same way. An absent `postData` still resolves to None, so a
continue that does not rewrite the body leaves it alone rather than
sending an empty one.
Two gaps the first pass left, both raised in review.

The decoded postData still travelled as a String, so a rewritten POST that
is a protobuf, a gzip stream or a multipart part had every non-UTF-8
sequence replaced with U+FFFD before reaching the wire. Continue's body is
Option<Vec<u8>> now; op_fetch_url's override_body was already Vec<u8>, so
the consumer loses a conversion rather than gaining one.

The decoder also dropped every character outside the base64 alphabet and
zero-padded partial groups, so raw text sent where base64 was expected
became quietly mangled bytes -- the same silence this change set out to
remove. decode_optional_binary refuses it instead, and both binary-carrying
arms answer -32602 naming the field.

Refusing needed the pause to outlive it: the resolution was built after
intercepted_paused.remove(), so an error would have taken the pause with
it and left the request unanswerable. It is built first now and the pause
is taken only once it is Ok, so a corrected command on the same requestId
still drives the fetch.

fulfillRequest's body stays a String at the resolution boundary: it
crosses op_fetch_url's JSON reply, where a Vec<u8> serialises as an array,
so making that end-to-end is a wider change than this one.
@ntdatt812
ntdatt812 force-pushed the fix/569-continue-overrides branch from ff831a8 to 3f9acc2 Compare September 24, 2026 02:30
@ntdatt812

Copy link
Copy Markdown
Contributor Author

Rebased onto main (1a3169d). Was CONFLICTING, now clean.

#912 landed in the meantime and changed what this had to be. It added body_base64 to Fetch.fulfillRequest, carrying the CDP body through unchanged so op_fetch_url can hand JS the exact bytes — which is precisely the lossy end this pull request's own comment said was out of scope. So your version of that arm is kept, not mine; body_base64 and the lossy body view both survive.

What is kept from here is the strict decode, now layered on top rather than replacing it:

  • Fetch.continueRequest reads postData through decode_optional_binary, so raw text in a base64 field is refused with -32602 instead of reaching the wire as mangled bytes. Ok(None) still means "leave this alone" rather than "make it empty".
  • Fetch.fulfillRequest keeps your shape and decodes raw_body to check it: a body that is not base64 would otherwise reach JS as the raw string through body_base64 anyway, so it is refused at the boundary.
  • decode_base64 stays, as a lossy wrapper over decode_base64_bytes, for domains/fetch.rs:164 which still wants the text view.

One thing I could not run: cargo build needs to create a symlink while unpacking the v8 crate, which Windows refuses without the privilege, so this is not compiled here. The file parses (rustfmt), and the one thing the rebase genuinely broke — the resolution match became a Result in this branch while main's fulfillRequest arm still returned the bare value — is fixed. CI compiles it.

@SGavrl
SGavrl merged commit bd95294 into h4ckf0r0day:main Oct 16, 2026
3 checks passed
whyparv pushed a commit to whyparv/obscura that referenced this pull request Nov 17, 2026
…errides

fix(cdp): decode the base64 postData a continueRequest carries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants