Skip to content

feat(webview): per-asset custom HTTP headers for web assets - #3162

Merged
vpetersson merged 10 commits into
masterfrom
feat/2215-webpage-custom-headers
Jul 9, 2026
Merged

feat(webview): per-asset custom HTTP headers for web assets#3162
vpetersson merged 10 commits into
masterfrom
feat/2215-webpage-custom-headers

Conversation

@vpetersson

@vpetersson vpetersson commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What & why

Resolves the feature request for saving and sending custom HTTP headers on web (webpage) assets — for example an Authorization header carrying a Grafana service-account token, so a private dashboard renders on screen without having to be made public.

Adds a per-asset setting whose headers are injected into QtWebEngine when the asset is displayed. It works on both Qt5 (Pi 1–3) and Qt6 (Pi 5 / x86) through a single, non-version-gated code path.

How it works

Headers are stored in Asset.metadata['headers'] as a {name: value} object (no schema migration — mirrors the existing refresh_interval_s feature end to end). A QWebEngineUrlRequestInterceptor installed on the shared profile attaches them at request time, scoped to the asset's own host so a bearer token is never sent to a third-party CDN/analytics/font domain the page also talks to — while same-host XHR (which a dashboard needs to render its panels) still carries it.

The chain:

  • Model (app/models.py) — normalize_asset_headers / validate_asset_headers / parse_header_lines. Header names must be RFC 7230 tokens; CR/LF/NUL are rejected in values to prevent header/response splitting; count and length are capped.
  • v2 API (api/serializers/v2.py) — custom_headers as a read field (SerializerMethodField) and a strict write field on create/update, folded into metadata['headers'] (400 on malformed input).
  • Edit modal (_asset_modal.html, views.py, asset_filters.py, home.ts) — a webpage-only "Custom HTTP headers" textarea, one Name: Value per line, parsed and clamped server-side (the form forgives; the API 400s).
  • Viewer (anthias_viewer/__init__.py) — view_webpage(headers=...) sends the headers as JSON over a new setRequestHeaders D-Bus slot, version-skew-latched exactly like setReloadInterval; a header-only edit still forces a reload.
  • C++ webview (anthias_webview/src/*) — RequestHeaderInterceptor (mutex-guarded, since Chromium may call it off the UI thread on Qt5), wired through View / MainWindow.

Testing

  • Full non-integration suite: 1421 passed, 0 failed. ruff check + ruff format --check clean.
  • New tests: header-helper unit tests, v2 API round-trip / rejection / CRLF / count-cap tests, and viewer D-Bus tests (send, empty-clears, reload-on-header-change, version-skew latch).
  • The C++ isn't compiled on the dev host (no Qt WebEngine dev headers there); it builds in the webview Docker image.

Incidental fix: host-only test isolation

Running pytest src/anthias_server/api/tests/ in isolation produced 12 spurious 500s. The shared redis/dbus/env test isolation lived only in tests/conftest.py, which pytest never loads for the API test tree (no common ancestor), and ViewerPublisher / ReplyCollector are unreset singletons that leak a real redis client across tests. CI masked this because the redis service resolves there. Fixed by moving the shared isolation into a root conftest.py (both trees inherit it) and seeding those singletons with the per-test fake. Isolated API run: 12-failed/56s → 96-passed/5.8s; full suite unchanged.

Follow-up

  • Real-device E2E on Qt5 and Qt6 testbeds — Qt6: full end-to-end pass on x86 (header injected through the integrated stack). Qt5: compiled under Qt 5.15 and the interceptor is installed/driven at runtime on the armhf binary; the render-level hop can't run on the current 64-bit-KMS test fleet (armhf Qt5 needs the legacy 32-bit dispmanx GL stack), so a render pass on a 32-bit-Raspbian Pi is still recommended pre-release. See validation comments below.

🤖 Generated with Claude Code

Adds a per-asset setting that injects custom request headers (e.g. an
Authorization token for a private Grafana dashboard) into QtWebEngine
when a webpage asset is displayed. Works on both Qt5 and Qt6 via one
non-gated code path. Closes the request in the linked feature issue.

- Store headers in Asset.metadata['headers'] as {name: value}; no
  migration (mirrors the refresh_interval_s feature end-to-end)
- Validate/sanitise header names (RFC 7230 token) and reject CR/LF/NUL
  in values to prevent header splitting; cap count and length
- v2 API: custom_headers read + write fields folded into metadata
- Edit modal: webpage-only "Custom HTTP headers" textarea, one
  Name: Value per line, parsed + clamped in assets_update
- Viewer sends headers as JSON over a new setRequestHeaders D-Bus slot
  (version-skew latched like setReloadInterval); reloads on header change
- C++ RequestHeaderInterceptor on the shared QWebEngineProfile, scoped
  to the asset's own host so tokens never leak to third-party requests
- Fix host-only test isolation: move shared redis/dbus/env mocks to a
  root conftest.py so the api/tests tree inherits them, and reset the
  ViewerPublisher/ReplyCollector singletons per test
- Add unit, v2 API, and viewer D-Bus tests

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@vpetersson
vpetersson requested a review from a team as a code owner July 8, 2026 17:30
@vpetersson
vpetersson requested a review from Copilot July 8, 2026 17:30
Comment thread src/anthias_server/api/serializers/v2.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds end-to-end support for per-asset custom HTTP request headers on webpage assets, persisted in Asset.metadata['headers'] and injected at request time by the QtWebEngine webview (Qt5 + Qt6) with API/UI/viewer wiring and tests. Also refactors pytest configuration so API tests run in isolation with the same Redis/D-Bus/environment stubbing as the main test tree.

Changes:

  • Add model helpers + validations for header parsing/normalization and expose the feature via v2 API create/update fields.
  • Extend the viewer ↔ webview D-Bus interface with setRequestHeaders and implement a QtWebEngine request interceptor to attach headers for webpage assets.
  • Move shared pytest isolation to repo-root conftest.py and add/extend unit + API + viewer tests for the feature.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/test_viewer.py Adds viewer-side tests for sending/clearing headers, reload-on-header-change, and version-skew latch behavior.
tests/test_asset_headers.py New unit tests for header parsing/normalization/validation helpers.
tests/conftest.py Keeps only Playwright/marketing fixtures; shared isolation moved to repo root.
conftest.py New repo-root pytest isolation: ENV var, gi/pydbus stubs, fake Redis, singleton seeding/reset.
src/anthias_server/app/models.py Adds header validation/normalization/parsing helpers and limits.
src/anthias_server/api/serializers/v2.py Adds custom_headers read field + strict write validation and metadata merge behavior.
src/anthias_server/api/tests/test_assets.py Adds v2 API round-trip + rejection + sanitization tests for custom headers.
src/anthias_server/app/views.py Parses custom_headers textarea for the HTML form and stores into metadata['headers'].
src/anthias_server/app/templatetags/asset_filters.py Sanitizes metadata.headers before feeding the edit modal.
src/anthias_server/app/templates/_asset_modal.html Adds webpage-only “Custom HTTP headers” textarea to the edit modal.
src/anthias_server/app/static/src/home.ts Extends asset edit typing to include optional metadata.headers.
src/anthias_viewer/__init__.py Sends headers over new D-Bus slot, caches applied headers, and reloads when headers change.
src/anthias_webview/src/view.h Declares setRequestHeaders and stores staged headers + interceptor pointer.
src/anthias_webview/src/view.cpp Implements request interceptor + JSON parsing + per-load scoping before navigation.
src/anthias_webview/src/mainwindow.h Exposes setRequestHeaders as a D-Bus slot on MainWindow.
src/anthias_webview/src/mainwindow.cpp Forwards setRequestHeaders to View.
src/anthias_webview/src/main.cpp Updates D-Bus export comment to include setRequestHeaders.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/anthias_webview/src/view.cpp Outdated
Comment thread src/anthias_webview/src/view.cpp Outdated
Comment thread src/anthias_webview/src/view.cpp
vpetersson and others added 2 commits July 8, 2026 18:01
Hardware testing on x86/Qt6 surfaced a race: the first setRequestHeaders
D-Bus call after a cold webview spawn can lose to the webview's D-Bus
registration and fail transiently. On a single-asset playlist the
unchanged-URL short-circuit then skips loadPage forever, so the C++
interceptor never receives the staged headers and the page loads without
them.

- _apply_request_headers now returns whether the headers are in effect
- view_webpage only caches current_browser_headers on success, so a
  transient failure leaves the mismatch in place and the next asset_loop
  tick re-issues both setRequestHeaders and loadPage
- add a regression test for the transient-retry path

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The root conftest that carries the Redis/D-Bus/env isolation is also an
ancestor of tools/raspberry_pi_imager/tests, which the run-python-linter
job runs with -p no:django in a minimal venv (no Django, no pytz). Its
import-time connect_to_redis patch pulled in anthias_common.utils ->
pytz and crashed conftest collection for those app-free tests.

Gate every app-dependent hook (connect_to_redis patch, _mock_redis and
_ensure_assetdir autouse fixtures) on an import probe so the file
degrades to a no-op when the app isn't importable, while still providing
full isolation for the app test suites.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@vpetersson

Copy link
Copy Markdown
Contributor Author

Hardware validation (Qt5 + Qt6)

Built the viewer image from this branch for both a Qt6 and a Qt5 target and exercised the real compiled interceptor on testbeds (header injection observed at an echo server that logs request headers).

Qt6 — full E2E ✅ (x86 testbed)

Built anthias-viewer:*-x86 (Qt6, qt6-webengine-dev), deployed the viewer container, and created a webpage asset with custom_headers through the normal flow. The request to the asset URL carried both headers on the main document and same-host subresources:

GET /normal-retest
Authorization: Bearer ANTHIAS-QT6-TEST
X-Anthias-Test: probe

Path exercised end-to-end: Django form/API → metadata['headers'] → viewer view_webpage(headers=…)setRequestHeaders D-Bus → C++ RequestHeaderInterceptor → outbound request.

This surfaced (and I fixed, commit 41d06ca3) a cold-boot race where the first setRequestHeaders can lose to the just-spawned webview's D-Bus registration; on a single-asset playlist the unchanged-URL short-circuit would then never reapply it. view_webpage now only caches the applied headers on success, so a transient failure self-heals on the next tick.

Qt5 — compile + interceptor runtime ✅, final render blocked by headless HW ⚠️

Built anthias-viewer:*-pi3 (armhf/Qt 5.15). My interceptor (view.cpp etc.) cross-compiles cleanly against the Qt5.15 toolchain (-Wall -Wextra, no errors) — this is the meaningful Qt5-vs-Qt6 difference, since the interceptor is a single non-#if-gated code path. At runtime on the armhf binary the D-Bus slot is registered and callable (has setRequestHeaders: True, setRequestHeaders OK, loadPage OK, Type: Webpage / Loading web page …), so the interceptor is installed and driven.

The final HTTP-with-header hop could not be observed on the available hardware: the only free Qt5-capable board is headless, and armhf QtWebEngine hard-crashes creating a GL context there (OpenGL context creation failedBus error), both offscreen and via eglfs. That's an environment limit, not a code issue. Since the interceptor source is identical to the Qt6 build that passed full E2E, and it compiles + loads + is driven under Qt5.15, I'm confident in it — but a full render-level Qt5 pass on a Pi with an attached display is worth doing before release.

Both testbeds were restored to their prior images and all temporary state cleaned up.

Addresses PR review (Copilot + CodeQL):

- Scope custom headers to the asset's full origin (scheme + host + port)
  instead of host only, so an Authorization header can't ride an
  https->http downgrade or reach a different port on the same host.
- Parent the request interceptor to the process-lifetime default profile
  (not the View) and detach it in ~View, removing a potential
  use-after-free where the profile could outlive the interceptor.
- v2 custom_headers validation raises a fixed, self-authored message
  rather than echoing the caught exception text (CodeQL "information
  exposure through an exception").

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@vpetersson

Copy link
Copy Markdown
Contributor Author

Follow-up: Qt5 render attempt on a real Pi 3, and the origin-scoping re-check

After the review fixes (commit 7a2dc44f), I re-ran the hardware checks:

Qt6 (x86) — still full E2E ✅ with the origin-scoped interceptor: a same-origin webpage asset's request carried Authorization: Bearer … + X-Anthias-Test on the main document and same-host subresources, confirming the host→origin change didn't regress same-origin injection.

Qt5 render on a real Pi 3 — blocked by the test fleet's GPU stack ⚠️. I built anthias-viewer:*-pi3 (armhf/Qt 5.15) at this HEAD and deployed the full stack on a physical Pi 3. The Python side drives it correctly (Showing asset … / Current url is http://…/qt5-e2e), but AnthiasViewer crash-loops before the fetch with:

This plugin does not support createPlatformOpenGLContext!

Root cause: the armhf Qt5 WebView is built for the legacy 32-bit Raspbian brcm/dispmanx GL stack, but every board in the test fleet runs a 64-bit KMS OS, where that GL path can't bind — so QtWebEngine gets no GL context. Reproduced identically on Pi 4 and the Pi 3. This is an environment mismatch, not a code issue.

Net Qt5 evidence: compiles cleanly under Qt 5.15 (-Wall -Wextra, no errors) + the setRequestHeaders D-Bus slot and interceptor are installed and driven at runtime on the armhf binary. The interceptor source is identical to the Qt6 build that passes full E2E. A render-level Qt5 pass needs a 32-bit-Raspbian Pi (legacy dispmanx) — worth doing before release, but the code side is as validated as this fleet allows.

All three testbeds were restored to their prior images and locks released.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Comment thread src/anthias_viewer/__init__.py Outdated
…s dict

Copilot re-review: assigning current_browser_headers = headers aliased
the caller's dict, so a later in-place mutation could make the next
value comparison spuriously equal and skip a needed reload. Store
dict(headers) instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Comment thread src/anthias_viewer/__init__.py
Copilot re-review (security): on a transient setRequestHeaders failure
during an asset transition, view_webpage still issued loadPage. The C++
side then reused the PREVIOUS asset's staged headers and scoped them to
the new asset's origin — leaking e.g. a prior asset's Authorization
header to the next asset's host.

Now loadPage is skipped entirely when the header update didn't apply;
the URL/header cache is left stale so the next asset_loop tick retries
once the D-Bus call succeeds. Supersedes the earlier "cache only on
success" tweak.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Comment thread src/anthias_server/api/serializers/v2.py Outdated
Comment thread src/anthias_server/api/serializers/v2.py Outdated
Copilot re-review: DictField(child=CharField(...)) coerced numeric /
boolean JSON values to strings ({"X": 123} -> "123") before
validate_asset_headers could reject them, so the strict validation was
bypassed. Drop the CharField child (use the unvalidated child) on both
the create and update serializers so values reach validate_asset_headers
verbatim and its isinstance(str) gate 400s non-strings. Also avoids
CharField trimming a value we forward on the wire byte-for-byte.

Adds numeric / bool / null rejection cases to the API test.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 8 comments.

Comment thread src/anthias_webview/src/view.cpp Outdated
Comment thread src/anthias_webview/src/view.h Outdated
Comment thread src/anthias_webview/src/view.h
Comment thread src/anthias_webview/src/mainwindow.h Outdated
Comment thread src/anthias_viewer/__init__.py Outdated
Comment thread src/anthias_server/app/templates/_asset_modal.html
Comment thread src/anthias_server/app/templates/_asset_modal.html
Comment thread conftest.py Outdated
Copilot re-review (8 doc nits): the origin-scoping change left comments,
docstrings, and the edit-modal copy still describing "same-host" /
"same-site" scoping. Align all of them with the implementation
(same-origin = scheme + host + port), and correct the root conftest
comment to note ENVIRONMENT/dbus-stub setup still runs while only the
app-dependent hooks no-op when the app stack is absent. No logic change.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Comment thread conftest.py
Comment thread src/anthias_webview/src/view.cpp
…undary

Copilot re-review:
- conftest app-availability probe caught bare Exception, which would
  silently disable Redis isolation for the whole app suite on any real
  import error. Narrow to ImportError (the missing-pytz/Django case);
  let genuine app import failures surface.
- C++ setRequestHeaders forwarded arbitrary JSON names/values into
  info.setHttpHeader() unchecked. Add defensive re-validation at this
  trust-no-one D-Bus boundary (RFC 7230 name token, reject CR/LF/NUL
  values, cap count/lengths) mirroring validate_asset_headers, so a
  hostile/buggy caller can't put splitting bytes on the wire even though
  the server side is the primary gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Comment thread src/anthias_server/app/models.py Outdated
Comment thread src/anthias_server/app/models.py Outdated
Comment thread conftest.py
Copilot re-review:
- MAX_HEADER_VALUE_LEN now caps the UTF-8 byte length (value.encode),
  not the Python character count, matching the webview's byte-based cap
  (values go on the wire as UTF-8).
- models.py comment updated to same-origin, and to note the server side
  is the primary gate with the webview re-validating defensively.
- conftest docstring: correct the module path to
  src/anthias_viewer/__init__.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@vpetersson
vpetersson requested a review from Copilot July 8, 2026 21:04
@sonarqubecloud

sonarqubecloud Bot commented Jul 8, 2026

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

@vpetersson
vpetersson merged commit eaf6d9f into master Jul 9, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants