capture: 0 assets cataloged when the cookie banner click reloads the page #4923

On some sites hyperframes capture downloads no images at all. The JSON output has one warning:

"Asset catalog is empty — no images will be downloaded. The page may use non-standard image loading."

The images are plain <img src> tags, so that's not it. The page reloads in the middle of the capture.

Still happens on 0.8.113 (latest today), default flags, 5 runs on the same URL:

• 2 runs: "Asset catalog is empty"
• 1 run: capture failed with Cannot read properties of null (reading 'innerHTML'), which looks like the same reload hitting a different step
• 1 run: 15 images but no logo
• 1 run: everything, logo included

What I see (traced on 0.8.20, the consent click and the catalog step look the same in 0.8.113):

1. captureScrollScreenshots clicks consent buttons before scrolling. On the site I tested, [class*="consent"] button matches the close button of the cookie dialog (an "acconsento" banner), and clicking it reloads the page.
2. The next page.evaluate in the screenshot step throws because of the navigation. The error isn't a timeout, so it gets swallowed and the step returns nothing ("0 scroll screenshots captured").
3. catalogAssets then runs on the reloading page. I added a probe before each step: 21 <img> right after navigation, 0 when the catalog runs (body scrollHeight 0), 20 again a few seconds later at "Extracting HTML & CSS".

extracted/page.html comes out complete, which made this confusing at first. Only the catalog sees the empty page.

On 0.8.20 it's worse: 0 of 5 runs catalog anything. It's a race, and on this site the reload usually wins.

This one line before progress("design", "Cataloging assets...") fixes the catalog for me on 0.8.20, 33 assets in 5 of 5 runs (I haven't patched 0.8.113):

await page1.waitForNetworkIdle({ idleTime: 500, timeout: 15e3 }).catch(() => {});

Waiting for the navigation right after the consent clicks would probably be cleaner. I haven't tried that.

A second thing I hit on the same site, possibly separate: with the default 120 s --capture-budget, core extraction sometimes uses the whole budget. The page has four videos, one of them 4K, and my guess is the video download eats it, but I didn't measure that. The assets phase then reports degraded with remainingMs: 0 and downloads nothing, even after the fix above. With the fix plus --capture-budget 300000, 0.8.20 got images and the logo in 3 of 3 runs. It might help to download images before videos, since images are small and the logo is often the thing people want.

Repro:

npx [email protected] capture --json --skip-vision -o out <url>

I can share the URL privately if that helps. It's a client's site.