Skip to content

Latest commit

 

History

History
245 lines (193 loc) · 12 KB

README.md

File metadata and controls

245 lines (193 loc) · 12 KB

DataDome

DataDome protects a site by handing suspicious requests a challenge instead of the page. Solving it earns a datadome cookie; send that cookie and you get the real content.

the two challenge types

DataDome serves one of two things, and the rt field in the block page tells you which:

rt type what it looks like
i interstitial a "verifying your browser" page that resolves on its own. Solved by POSTing a payload.
c captcha a puzzle page. Solved by GETing /captcha/check with the payload in the query string.

An interstitial often escalates to a captcha when DataDome is unconvinced — idealista.ts exercises exactly that i -> c chain. Handle both.

recognising a challenge

A blocked response is a 403 whose body carries an inline dd object:

<script>var dd={'rt':'c','cid':'AHrlqAAAAAM…','hsh':'97DAF2A1CB…','t':'fe',
                's':51825,'e':'9cd1f431…','host':'geo.captcha-delivery.com',
                'cookie':'YWlkQoc1fsbfX4s8…'}</script>

Those fields are the challenge. cookie is the same value DataDome set in the datadome response cookie, and it is what you pass as ddCookie.

the HTTP flow

py-src/datadome/grainger_requests.py is this, end to end:

1.  GET  https://www.grainger.com/            -> 403 + var dd={…} + datadome cookie
2.  GET  geo.captcha-delivery.com/captcha/…   -> challenge document HTML
    GET  ct.captcha-delivery.com/…            -> each external script it loads
3.  POST <solver>/dd/solve       -> a prepared submission
4.  GET  geo.captcha-delivery.com/captcha/check?…  -> {"cookie":"datadome=…"}
5.  GET  https://www.grainger.com/  + cookie  -> 200, the real page

Step 3 sends dd, ddCookie, iframeData (the HTML from step 2), and a profile / js_profile pair describing the browser you are claiming to be.

Newer challenge documents load the DataDome collector as an external defer script on captcha-delivery.com instead of inlining it. The solver does not fetch it: send the body of every such script the document references as iframeData.externalScripts ([{ url, body }]; extractExternalScriptUrls in external-scripts.ts lists them). A missing one answers 422 dd.script.missing; one the document does not load answers 400. Documents with an inline collector need none. solver.ts takes the bodies from the browser's own responses; the Python clients fetch them through the same session, proxy and cookies as the document (collect_external_scripts in http_utils.py), and dev-resources/curl does the same with jq.

you send the submission, always

/dd/solve never submits for you — it returns a prepared submission. DataDome binds the clearance cookie to the IP that submitted it, so a cookie the solver earned would work from the solver, not from you: you would get a fresh 403 on your very next request and it would look like the solve failed.

Take the prepared submission and send it yourself from the IP you intend to scrape from.

keep the identity consistent

The profile you send the solver has to match the headers you actually put on the wire — user agent, sec-ch-ua, platform, language, timezone. DataDome cross-checks them. Changing the user agent in PROFILE without changing navigationHeaders is the most common way to get a solve rejected, and the error it produces looks nothing like the cause.

the browser flow

solver.ts exports one function:

import { solve } from '#src/datadome/solver.js';

const result = await solve(page, { proxy, solverApiKey, solverUrl, url });
// -> { cookie, responseStatus, url }

It attaches to a Playwright page, watches for a challenge, asks the solver for the sensor values, and lets Chrome own the actual submission — the native interstitial POST or captcha GET — so the request carries a real TLS fingerprint and the browser's own cookie jar. It handles the i -> c escalation and resolves once DataDome returns an accepted cookie.

For captchas it also measures the slider handle (#captcha__element div.slider) inside the challenge frame, in that frame's viewport CSS pixels, and sends it as iframeData.captchaLayout (built by captcha-layout.ts). Its viewport has to equal the js_profile.screen.innerWidth/innerHeight sent with it. It may not be needed: without it the solver reconstructs the slider geometry from the challenge document and its stylesheets, and falls back to its default geometry when it can't. If the measurement fails the client sends nothing and the solve goes ahead. The HTTP flow has no rendered frame to measure, so it leaves the field out.

Use it when the site needs a browser anyway. If all you want is a cookie, the HTTP flow is far cheaper.

examples

file what it shows
grainger.ts grainger.com through the browser bridge
idealista.ts an interstitial that escalates to a captcha
browser-target.ts the launch options, signal handling and screenshot capture every browser example shares
http-utils.ts the headers and block-page parsing the HTTP examples share
profile.ts the browser identity and DataDome endpoints, shared with solver.ts

The HTTP flow, in Python, under py-src/datadome/:

file what it shows
grainger_requests.py the HTTP flow with requests. --url= points it at any DataDome site.
grainger_httpx.py the same flow with httpx
grainger_urllib.py the same flow with only the standard library
http_utils.py the Python port of http-utils.ts (identity included)

the harder targets

Everything above is a demonstration. These six exist to be measured — they are the DataDome sites customers actually ask about, and they do not all clear reliably. The rt / t column is what an unauthenticated curl from a datacenter IP drew on 2026-08-25; a residential exit will often see something milder, and bv in particular is a property of the address rather than the site:

script target rt / t notes
alaska.ts my.alaska.gov c / fe the myAlaska login portal. labor.alaska.gov is not fronted and www.alaska.gov did not answer at all, so this is the alaska.gov worth pointing at
saks.ts www.saksfifthavenue.com c / bv captcha, and bv — banned visitor — from a datacenter address, which is a refusal rather than a puzzle. Needs residential
anthropologie.ts www.anthropologie.com i interstitial, on an origin that answers HTTP/1.1 while challenging
yelp.ts www.yelp.com/signup none no server-side block from a good IP — DataDome is present only as the client-side tag, so a clean address gets no challenge and the run times out rather than failing
etsy.ts www.etsy.com i scores first (x-datadome-riskscore on the block) and is the least forgiving here
bestwestern.ts www.bestwestern.com c / fe a captcha from the first request, with no interstitial in front of it
tripadvisor.ts www.tripadvisor.com i an interstitial that clears in one round. 3/3 headless on 2026-10-19, real homepage in the screenshot

They are deliberately not in npm test. Each is a headed browser run with a 30s hold, and several are expected to fail some fraction of the time, so gating master on them would be measuring the targets rather than the code. Measure them the way a comparison should be measured — a fixed number of runs each, verified by screenshot rather than by the solver's success flag:

npm run etsy -- --screenshot
node --env-file=.env src/loadtest.ts --script=src/datadome/etsy --iterations=3 --concurrency=1

Three things that decide the outcome more than the script does:

  • Headed, not headless. --headless on these draws recurrent challenges where the same identity headed clears first time. The flag is still there; it is just the wrong default for this set.
  • One residential exit IP per run, in the target's own country, held for the whole run. DataDome binds the clearance cookie to the submitting address, so a rotating proxy invalidates the thing you just earned. Put your provider's session token (or a {session} placeholder) in proxy= and pinSession will hold it; without one the load-test runner warns that every iteration shares an exit and the numbers mean nothing.
  • A screenshot is the result. A returned success flag is not proof the page rendered — a soft block answers 200. --screenshot writes what the page actually ended up showing to target/screenshots/, on failure as well as success.
./venv/bin/python py-src/datadome/grainger_requests.py --url=https://www.idealista.com/
node --env-file=.env src/datadome/idealista.ts --headless

lightpanda and plain HTTP

src/lightpanda.ts and src/mitm.ts drive a challenge with Lightpanda, a headless browser with no renderer, instead of Chrome. The example that runs them against grainger.com is dev-resources/lightpanda/datadome/grainger-lightpanda.ts.

There are also three examples that skip the browser entirely and drive http-utils.ts directly — undici, axios and fetch versions of the same four-request flow — in dev-resources/http/datadome/.

Neither Lightpanda nor a plain HTTP client is a real browser, and DataDome can tell: both sets solve reliably against a residential exit run locally but not against CI's proxyless runner, so they live in dev-resources/ instead of here and are excluded from the smoke suite and from npm run build — kept committed and runnable by hand, not part of what customers get or what CI gates on. See the comment above SCRIPTS in dev-resources/smoke.js for the full reasoning.

There is a shell version too, dev-resources/curl, which is the same flow in curl and jq. It is the clearest place to see the raw HTTP, and it doubles as a demonstration that TLS fingerprinting is a separate problem from solving: the solve always succeeds, but whether the site honours the cookie afterwards depends on how it treats a non-browser client. See the README for the site-by-site results.

picking a client

The three Python clients make the same requests, so copy whichever matches what you already use. Every one of them treats proxy= as optional — unset, the client is built without a proxy and the whole flow goes out from this machine. Two differences worth knowing:

  • requests and httpx both keep a cookie jar, and both need help with this one cookie. DataDome sets the clearance cookie with Domain=.grainger.com, but the response comes from geo.captcha-delivery.com, so the jar drops it as a domain mismatch. Worse, the block response already put a pre-solve datadome cookie in the jar for that domain — add the new one without removing it and both go out, the stale value wins, and it looks exactly like a failed solve. Replace, do not append.
  • urllib has no per-request proxy escape hatch, so build two openers: one with a ProxyHandler for the target traffic, and one with an empty ProxyHandler({}) for the call to your own solver. Credentials in the proxy URL become the Proxy-Authorization header on the CONNECT automatically.

known not to work yet

A couple of targets we tried aren't committed — kept locally, gitignored, rather than shipped as working examples.

gotchas

  • no challenge to solve — the site served your IP the page directly. Datacenter proxies frequently sail through; try a residential pool.
  • A 403 immediately after a solve — the cookie was earned on a different IP. Check session pinning, and that you sent the submission yourself.
  • The cookie is a full Set-Cookie string. /captcha/check returns datadome=abc…; Max-Age=31536000; Domain=.grainger.com; …. Split on ; and keep the value.
  • Cookies are per-registered-domain. One earned on grainger.com works across its subdomains, and nowhere else.