Skip to content

[Question] How to get shadow DOM content? #16722

Closed

Description

@otsaw

Hi! I'm using the Python sync bindings of Playwright and want to get the full HTML content of web pages. Some pages seem to use shadow DOM components, which are not included in the HTML returned by page.content. How do I get either the full HTML (preferrable) or the shadow DOM content separately?

Here's an example page with shadow DOM under popup-info and a minimal Python script to try to get that content:

https://mdn.github.io/web-components-examples/popup-info-box-web-component/

#!/usr/bin/env python3

import contextlib
import tempfile

from playwright.sync_api import sync_playwright

@contextlib.contextmanager
def get_browser_context():
    with tempfile.TemporaryDirectory() as user_data_dir:
        with sync_playwright() as playwright:
            yield playwright.chromium.launch_persistent_context(user_data_dir)

def get_content(browser, url):
    page = browser.new_page()
    response = page.goto(url)
    return response, page.content()

with get_browser_context() as browser:
    url = "https://mdn.github.io/web-components-examples/popup-info-box-web-component/"
    response, html = get_content(browser, url)
    print(response.status)
    print(html)

Activity

  1. aslushnikov commented on Aug 22, 2022

    @aslushnikov
    Contributor

    Osmo Salomaa (@otsaw) unfortunately we don't expose anything like this; you'll have to iterate shadow hosts manually in javascript land using page.evaluate().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions