Update on what happened in WebKit in the week from March 23 to March 30.

This week comes with a mixed bag of new features, incremental improvements, and a new release with the ever important security issue fixes. Also: more blog posts!

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Releases 📦️

Community & Events 🤝

Update on what happened in WebKit in the week from March 10 to March 18.

The big ticket item in this week's update are the 2.52.0 releases, which include the work from the last six-month development period, and come with a security advisory. Meanwhile, WPE-Android also gets a release, and a number of featured blog posts.

WPE WebKit 📟

Releases 📦️

Community & Events 🤝

The WebKit team at Igalia is happy to announce a new release series of WPE WebKit. This is a summary of the most noteworthy changes from the latest release cycle.

Graphics improvements

WPE’s graphics support has seen numerous improvements with a positive impact in rendering performance, resource usage, and better rendering. Let’s have a look at some of the most significant changes:

• Compute the layers tile size, using a different strategy depending on whether GPU rendering is enabled. This optimizes resource usage depending on both hardware and software rendering mode.
• WPE now uses run-loop observers to properly schedule layer flushing and composition, which results in snappier and better performing rendering and animation.
• 2D-canvas acceleration has now improved performance, as operations are recorded for batched replay.
• Text rendering has better performance too.
• In non-composite mode, it’s now also possible to use damage propagation.
• Asynchronous scrolling has also seen performance improvements.

On top of this, as usual, many rendering issues have been fixed, making this release of WPE one of the best in terms of graphics support.

Multimedia improvements

WebRTC

When using GstWebRTC, WebRTC network access has been moved to the network process. This also requires librice, and building with the CMake USE_LIBRICE option. When this is enabled, it is still possible to choose the older libnice-based implementation at runtime by setting WEBKIT_GST_DISABLE_WEBRTC_NETWORK_SANDBOX=1 in the environment.

Having WebRTC network access in the network process is a security improvement, as it reduces the surface of attack in other more sensitive processes.

Other multimedia improvements

• Videos with BT2100-PQ colorspace are now tone-mapped to SDR, ensuring colours do not appear washed out.
• Support for the Audio Output Devices API, which allows Web content to enumerate audio devices and decide which one to use for output. This feature is disabled by default, and may be previewed using the ExposeSpeakers, ExposeSpeakersWithoutMicrophone, and PerElementSpeakerSelection feature flags.
• Many code improvements to the GStreamer backend that will result in a more stable multimedia experience.

WebXR

WebXR support through OpenXR has seen substantial development this cycle:

• New API has been added in order to support WebXR session permissions, querying whether a session is active and requesting to leave a session.
• Support for WebXR Hand Input module has been added for ports using OpenXR.
• Support for WebXR Hit Test Module is added with testable status, so it can be enabled at runtime (--features=+WebXRHitTestModule) or at build time passing -DENABLE_WEBXR_HIT_TEST=ON.

API Changes

The future of the WPE API

The traditional libwpe-based API remains in WPE for this release cycle, but we are planning to sunset it starting with the following one (2.54). This applies to Cog, which is no longer in active development and won’t have any more stable releases beyond the 0.18.x series. While both libwpe and Cog will remain available, we encourage developers to transition to the new WPEPlatform API, which will be considered stable by then.

This means that it is the perfect time to test the WPEPlatform API and provide feedback, as it’s still possible to make changes to it to better suit users’ needs.

WPEPlatform API changes

New platform APIs include:

• wpe_display_create_toplevel(). This way it’s possible to create a toplevel using the common API which allows the inspector to work when the application is handling toplevels.
• A new WPEDisplay::disconnected signal has been added, which allows platform implementations to notify when the native display gets “disconnected” and thus no longer usable. Applications can handle it to attempt recovery, or to know when they may free resources.
• A new WPEView::buffers-changed signal, alongside the associated WPEViewClass.buffers_changed virtual function, have been added. These may be used to know in advance which graphics buffers will be used for rendering the content for a given web view. This feature is mainly useful for platform implementations which may need to perform additional setup in advance, before updated web view contents are provided in the buffers configured by WebKit.
• Two new functions, wpe_clipboard_content_get_text() and wpe_clipboard_content_get_bytes(), allow applications to obtain the contents held in the clipboard.

The public API has received the following changes, which might require changes to existing platform implementations:

• Multiple callbacks are now supported for WPEScreenSyncObserver, the API has changed from wpe_screen_sync_observer_set_callback() to a pair of wpe_screen_sync_observer_add_callback()/_remove_callback() functions. The functions to start/stop the observer are no longer available, and instead the observer will be activated automatically when there are one or more callbacks attached to it.

The WPEPlatform API can now be used on Android.

Legacy API

The legacy libwpe-based API can be disabled at build time, by toggling the ENABLE_WPE_LEGACY_API CMake option. This allows removal of uneeded code when an application is exclusively using the new WPEPlatform API.

New WebKit API

• WebKitImage, an image abstraction that is used when taking snapshots of webviews.
• webkit_webview_get_snapshot() and webkit_webview_get_snapshot_finish() can be used to fetch a snapshot of a webview contents.

Web Standards support

As usual, this list is not exhaustive as WebKit continuously progresses in its support for new standards. Some of the highlights for this release are:

• Largest Contentful Paint is now enabled.
• Pointer and Touch Events now use more precise fractional coordinates.
• The Fetch API now accepts local connections.
• The Navigation API is now enabled.
• The Event Timing API is now enabled.
• The CSS random() function (in draft status) is now available.
• CSS field-sizing is now available.
• The Keyboard lock API is now available.
• The Origin API is now available.
• The Readable Byte Streams API is now available.
• Enabled CSS grid-lanes (a.k.a. Masonry layout), part of CSS Grid Layout Module Level 3.

Other notes

The Flatpak-based development SDK has been removed. Developers are encouraged to use the WebKit Container SDK instead.

Since 2022, my main focus has been working on the Wolvic browser, still the only open source WebXR-capable browser for Android/AOSP devices (Meta, Pico, Huawei, Lenovo, Lynx, HTC…) out there. That’s an effort that continues to this day (although to a much lesser extent nowadays). In early 2025, as a consequence of all that work in XR on the web, an opportunity emerged to implement WebXR support in WebKit for the WPE port, and we decided to take it.

WPE WebKit is a WebKit port optimized for embedded devices — think set-top boxes, digital signage, kiosk displays, and in-vehicle infotainment systems. It is developed by Igalia and powers web experiences on millions of devices worldwide, from set-top boxes to smart TVs and beyond.

WPE WebKit has recently introduced a brand-new platform API called WPEPlatform, which replaces the legacy libwpe + wpebackend-fdo stack. In this post, I will walk you through building a minimal WPE browser launcher using only the new WPEPlatform API, and demonstrate one of its newly available features: the Context Menu API — rendered entirely as an HTML overlay.

Why a New API? #

The legacy stack (libwpe + wpebackend-fdo + Cog platform plugins) had several pain points: nested Wayland compositor complexity, dependency on Mesa’s now-deprecated EGL_WL_bind_wayland_display extension, rigid C function-pointer tables, and platform code scattered across three libraries.

The new WPEPlatform API replaces all of this with a single, clean GObject-based layer — providing automatic backend creation, DMA-BUF direct buffer sharing, unified window management (fullscreen, maximize, resize, title), and easy language bindings via GObject Introspection.

Timeline: The stable release of WPEPlatform is planned for September 2026. At that point, the legacy API will be officially deprecated. We strongly recommend new projects to adopt the WPEPlatform API from the start.

WPEPlatform Launcher: A Minimal Browser in ~250 Lines #

To demonstrate the new API, I built WPEPlatformLauncher — a minimal but functional WPE WebKit browser that uses only the WPEPlatform API. No legacy libwpe, no wpebackend-fdo, no Cog — just the new API.

The full source code is available at: kate-k-lee/WebKit@aed6402

How Simple Is It? #

Here is the core of the launcher — creating a WebView with the new API:

/* WPEPlatform backend is created automatically — no manual setup needed */
auto* webView = WEBKIT_WEB_VIEW(g_object_new(WEBKIT_TYPE_WEB_VIEW,
    "web-context", webContext,
    "network-session", networkSession,
    "settings", settings,
    "user-content-manager", userContentManager,
    nullptr));

/* Get the WPEPlatform view — this is where the new API shines */
auto* wpeView = webkit_web_view_get_wpe_view(webView);
auto* toplevel = wpe_view_get_toplevel(wpeView);

/* Window management: fullscreen, resize, title — all built-in */
wpe_toplevel_fullscreen(toplevel);
wpe_toplevel_resize(toplevel, 1920, 1080);
wpe_toplevel_set_title(toplevel, "WPEPlatform Launcher");

/* Input events: just connect a GObject signal */
g_signal_connect(wpeView, "event", G_CALLBACK(onViewEvent), webView);

Compare this with the legacy API, which required:

1. Manually creating a WPEToolingBackends::ViewBackend
2. Wrapping it in a WebKitWebViewBackend with a destroy callback
3. Creating a C++ InputClient class and registering it
4. Having no window management (no maximize, minimize, title, etc.)

The new API handles backend creation, display detection, and input forwarding automatically.

Keyboard Shortcuts #

Handling keyboard events is straightforward with the WPEPlatform event system:

static gboolean onViewEvent(WPEView* view, WPEEvent* event, WebKitWebView* webView)
{
    if (wpe_event_get_event_type(event) != WPE_EVENT_KEYBOARD_KEY_DOWN)
        return FALSE;

    auto modifiers = wpe_event_get_modifiers(event);
    auto keyval = wpe_event_keyboard_get_keyval(event);

    /* Ctrl+Q: Quit */
    if ((modifiers & WPE_MODIFIER_KEYBOARD_CONTROL) && keyval == WPE_KEY_q) {
        g_application_quit(g_application_get_default());
        return TRUE;
    }

    /* F11: Toggle fullscreen via WPEToplevel */
    if (keyval == WPE_KEY_F11) {
        auto* toplevel = wpe_view_get_toplevel(view);
        if (wpe_toplevel_get_state(toplevel) & WPE_TOPLEVEL_STATE_FULLSCREEN)
            wpe_toplevel_unfullscreen(toplevel);
        else
            wpe_toplevel_fullscreen(toplevel);
        return TRUE;
    }

    return FALSE;
}

HTML-Based Context Menu: Solving the “No Native UI” Challenge #

WPE WebKit is designed for embedded environments where there is no native UI toolkit — no GTK, no Qt. This means features like context menus (right-click menus) that desktop browsers take for granted need to be implemented by the application.

The approach: intercept WebKit’s context-menu signal, read the menu items, and render them as an HTML/CSS overlay injected into the page DOM.

The Architecture #

User right-clicks
  → WebKit emits "context-menu" signal
  → onContextMenu() handler:
      1. Reads menu items via webkit_context_menu_get_items()
      2. Gets position via webkit_context_menu_get_position()
      3. Builds JavaScript that creates DOM elements
      4. Injects via webkit_web_view_evaluate_javascript()
      5. Returns TRUE (suppresses default menu)

User clicks a menu item
  → JS: window.webkit.messageHandlers.contextMenuAction.postMessage(actionId)
  → C: onContextMenuAction() receives the action ID
      → Executes: webkit_web_view_go_back(), execute_editing_command("Copy"), etc.

User clicks outside the menu
  → JS: overlay click handler removes the DOM elements

Reading Context Menu Items #

The Context Menu API provides everything we need:

static gboolean onContextMenu(WebKitWebView* webView,
    WebKitContextMenu* contextMenu, gpointer /* event */,
    WebKitHitTestResult* hitTestResult, gpointer)
{
    /* Save hit test result for link-related actions */
    savedHitTestResult = WEBKIT_HIT_TEST_RESULT(g_object_ref(hitTestResult));

    /* Iterate through menu items */
    GList* items = webkit_context_menu_get_items(contextMenu);
    for (GList* l = items; l; l = l->next) {
        auto* item = WEBKIT_CONTEXT_MENU_ITEM(l->data);

        if (webkit_context_menu_item_is_separator(item)) {
            /* Render as a horizontal line */
            continue;
        }

        const char* title = webkit_context_menu_item_get_title(item);
        auto action = webkit_context_menu_item_get_stock_action(item);
        /* Build HTML element with title and action ID */
    }

    /* Get position for menu placement */
    gint posX = 0, posY = 0;
    webkit_context_menu_get_position(contextMenu, &posX, &posY);

    return TRUE; /* Suppress default menu */
}

The HTML Menu: Dark Theme for Embedded #

The context menu is rendered with a dark theme CSS, designed for embedded/kiosk displays:

#__wpe_ctx_menu {
    position: fixed;
    min-width: 180px;
    background: #2b2b2b;
    border: 1px solid #505050;
    border-radius: 6px;
    padding: 4px 0;
    box-shadow: 0 8px 24px rgba(0,0,0,0.4);
    font-family: system-ui, sans-serif;
    font-size: 13px;
    color: #e0e0e0;
}

.__wpe_ctx_item:hover {
    background: #0060df;
    color: #ffffff;
}

Handling Actions via Script Message Handler #

Communication between the HTML menu and the C application uses WebKit’s script message handler mechanism:

/* Register message handler */
auto* ucm = webkit_user_content_manager_new();
webkit_user_content_manager_register_script_message_handler(
    ucm, "contextMenuAction", nullptr);
g_signal_connect(ucm, "script-message-received::contextMenuAction",
    G_CALLBACK(onContextMenuAction), nullptr);

// In the generated HTML menu item:
item.addEventListener('click', function() {
    window.webkit.messageHandlers.contextMenuAction.postMessage(actionId);
});

/* Handle the action in C */
static void onContextMenuAction(WebKitUserContentManager*, JSCValue* value, gpointer)
{
    int actionId = jsc_value_to_int32(value);

    switch (actionId) {
    case WEBKIT_CONTEXT_MENU_ACTION_RELOAD:
        webkit_web_view_reload(webView);
        break;
    case WEBKIT_CONTEXT_MENU_ACTION_COPY:
        webkit_web_view_execute_editing_command(webView, "Copy");
        break;
    case WEBKIT_CONTEXT_MENU_ACTION_OPEN_LINK:
        webkit_web_view_load_uri(webView,
            webkit_hit_test_result_get_link_uri(savedHitTestResult));
        break;
    /* ... more actions ... */
    }
}

Demo #

Here is the WPEPlatformLauncher in action, showing the HTML context menu with various actions:

Right-clicking shows the HTML context menu. Clicking “Reload” triggers an actual page reload.

Right-clicking a link shows link-specific actions like “Open Link” and “Copy Link Address”.

Building and Running #

I built and ran the WPEPlatformLauncher inside a container using the WebKit Container SDK, which provides a pre-configured development environment with all the dependencies needed to build WPE WebKit.

The WPEPlatformLauncher integrates into the WebKit build system:

# Build WPE WebKit with the launcher
Tools/Scripts/build-webkit --wpe --release

# Run
./WebKitBuild/WPE/Release/bin/WPEPlatformLauncher https://wpewebkit.org

# Run in fullscreen (kiosk mode)
./WebKitBuild/WPE/Release/bin/WPEPlatformLauncher --fullscreen https://your-app.com

The full source is a single main.cpp file (~600 lines including the context menu), integrated into the WebKit tree alongside MiniBrowser:

WebKit/Tools/
├── MiniBrowser/wpe/          ← Existing (supports both old + new API)
├── WPEPlatformLauncher/      ← New (WPEPlatform API only)
│   ├── main.cpp
│   └── CMakeLists.txt
└── PlatformWPE.cmake         ← Modified to add WPEPlatformLauncher

Summary #

The new WPEPlatform API makes building WPE WebKit applications significantly simpler:

• No manual backend setup — the platform is detected and configured automatically
• GObject-based — signals, properties, and ref counting instead of C function pointers
• DMA-BUF direct sharing — no dependency on Mesa’s deprecated EGL extensions
• Unified window management — fullscreen, maximize, minimize, resize, and title
• Language binding friendly — works with Python, JavaScript, and more via GObject Introspection

For embedded browser developers building kiosk UIs, set-top box interfaces, or digital signage with WPE WebKit — now is the time to adopt the new API. The stable release is coming in September 2026, and the legacy stack (libwpe, wpebackend-fdo, Cog) will be deprecated at that point.

Resources #

• WPE WebKit official site
• WPE WebKit API Reference (2.51.92)
• WebKitContextMenu API Reference
• WPEPlatformLauncher source code
• Igalia — WPE WebKit
• Cog — WPE launcher (legacy)

Update on what happened in WebKit in the week from March 2 to March 9.

As part of this week's handful of news, WebKitGTK and WPE WebKit now have support for Gamepad's "VibationActuator" property, the video decoding limit is now configurable at runtime in addition to build time, and an interesting fix that makes WebKit render fonts like other browsers by making it blend text incorrectly (!).

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Releases 📦️

Update on what happened in WebKit in the week from February 23 to March 2.

This installment of the periodical brings news about support for Qualcomm qtivdec2 and qtivenc2 on GStreamer, GPU texture atlas creation and replay substitution, enhancement of the scroll gesture in WPE, and two new releases: WebKitGTK 2.51.92 and WPE WebKit 2.51.92.

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Releases 📦️

Update on what happened in WebKit in the week from February 9 to February 23.

In this week we have a nice fix for video streams timestamps, a fix for a PDF rendering regression, support for rendering video buffers provided by Qualcomm video decoders, and a fix for a font selection issue. Also notable we had a new WPE Android release, and the libsoup 3.6.6 release.

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Releases 📦️

Update on what happened in WebKit in the week from February 2 to February 9.

The main event this week was FOSDEM (pun intended), which included presentations related to WebKit; but also we got a batch of stable and development releases, asynchronous scrolling work, OpenGL logging, cleanups, and improving the inspector for the WPE work.

Cross-Port 🐱

Graphics 🖼️

WebKitGTK 🖥️

WPE WebKit 📟

Releases 📦️

Community & Events 🤝

Update on what happened in WebKit in the week from January 26 to February 2.

A calm week for sure! The highlight this week is the fix for scrolling not starting when the main thread is blocked.

Cross-Port 🐱

Graphics 🖼️

Update on what happened in WebKit in the week from January 19 to January 26.

The main event this week has been the creation of the branch for the upcoming stable series, accompanied by the first release candidate before 2.52.0. But there's more: the WPE port gains hyphenation support and the ability to notify of graphics buffer changes; both ports get graphics fixes and a couple of new Web features, and WPE-Android also gets a new stable release.

Cross-Port 🐱

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Releases 📦️

Now that 2025 is over, it’s time to look back and feel proud of the path we’ve walked. Last year has been really exciting in terms of contributions to GStreamer and WebKit for the Igalia Multimedia team.

With more than 459 contributions along the year, we’ve been one of the top contributors to the GStreamer project, in areas like Vulkan Video, GstValidate, VA, GStreamer Editing Services, WebRTC or H.266 support.

In Vulkan Video we’ve worked on the VP9 video decoder, and cooperated with other contributors to push the AV1 decoder as well. There’s now an H.264 base class for video encoding that is designed to support general hardware-accelerated processing.

GStreaming Editing Services, the framework to build video editing applications, has gained time remapping support, which now allows to include fast/slow motion effects in the videos. Video transformations (scaling, cropping, rounded corners, etc) are now hardware-accelerated thanks to the addition of new Skia-based GStreamer elements and integration with OpenGL. Buffer pool tuning and pipeline improvements have helped to optimize memory usage and performance, enabling the edition of 4K video at 60 frames per second. Much of this work to improve and ensure quality in GStreamer Editing Services has also brought improvements in the GstValidate testing framework, which will be useful for other parts of GStreamer.

Regarding H.266 (VVC), full playback support (with decoders such as vvdec and avdec_h266, demuxers and muxers for Matroska, MP4 and TS, and parsers for the vvc1 and vvi1 formats) is now available in GStreamer 1.26 thanks to Igalia’s work. This allows user applications such as the WebKitGTK web browser to leverage the hardware accelerated decoding provided by VAAPI to play H.266 video using GStreamer.

Igalia has also been one of the top contributors to GStreamer Rust, with 43 contributions. Most of the commits there have been related to Vulkan Video.

In addition to GStreamer, the team also has a strong presence in WebKit, where we leverage our GStreamer knowledge to implement many features of the web engine related to multimedia. From the 1739 contributions to the WebKit project done last year by Igalia, the Multimedia team has made 323 of them. Nearly one third of those have been related to generic multimedia playback, and the rest have been on areas such as WebRTC, MediaStream, MSE, WebAudio, a new Quirks system to provide adaptations for specific hardware multimedia platforms at runtime, WebCodecs or MediaRecorder.

We’re happy about what we’ve achieved along the year and look forward to maintaining this success and bringing even more exciting features and contributions in 2026.

Update on what happened in WebKit in the week from December 26 to January 19.

We're back! The first periodical of 2026 brings you performance optimizations, improvements to the memory footprint calculation, new APIs, the removal of the legacy Qt5 WPE backend, and as always, progress on JSC's Temporal implementation.

Cross-Port 🐱

JavaScriptCore 🐟

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Update on what happened in WebKit in the week from December 16 to December 25.

Right during the holiday season 🎄, the last WIP installment of the year comes packed with new releases, a couple of functions added to the public API, cleanups, better timer handling, and improvements to MathML and WebXR support.

Cross-Port 🐱

Improved timers, by making some of them use the timerfd API. This reduces timer “lateness”—the amount of time elapsed between the configured trigger time, and the effective one—, which in turn improves the perceived smoothness of animations thanks to steadier frame delivery timings. Systems where the timerfd_create and timerfd_settime functions are not available will continue working as before.

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

WPE Android ↗ 🤖

Releases 📦️

Community & Events 🤝

This article is a continuation of the series on WPE performance considerations. While the previous article touched upon fairly low-level aspects of the DOM tree overhead, this one focuses on more high-level problems related to managing the application’s workload over time. Similarly to before, the considerations and conclusions made in this blog post are strongly related to web applications in the context of embedded devices, and hence the techniques presented should be used with extra care (and benchmarking) if one would like to apply those on desktop-class devices.

The workload #

Typical web applications on embedded devices have their workloads distributed over time in various ways. In practice, however, the workload distributions can usually be fitted into one of the following categories:

1. Idle applications with occasional updates - the applications that present static content and are updated at very low intervals. As an example, one can think of some static dashboard that presents static content and switches the page every, say, 60 seconds - such as e.g. a static departures/arrivals dashboard on the airport.
2. Idle applications with frequent updates - the applications that present static content yet are updated frequently (or are presenting some dynamic content, such as animations occasionally). In that case, one can imagine a similar airport departures/arrivals dashboard, yet with the animated page scrolling happening quite frequently.
3. Active applications with occasional updates - the applications that present some dynamic content (animations, multimedia, etc.), yet with major updates happening very rarely. An example one can think of in this case is an application playing video along with presenting some metadata about it, and switching between other videos every few minutes.
4. Active applications with frequent updates - the applications that present some dynamic content and change the surroundings quite often. In this case, one can think of a stock market dashboard continuously animating the charts and updating the presented real-time statistics very frequently.

Such workloads can be well demonstrated on charts plotting the browser’s CPU usage over time:

As long as the peak workload (due to updates) is small, no negative effects are perceived by the end user. However, when the peak workload is significant, some negative effects may start getting noticeable.

In case of applications from groups (1) and (2) mentioned above, a significant peak workload may not be a problem at all. As long as there are no continuous visual changes and no interaction is allowed during updates, the end-user is unable to notice that the browser was not responsive or missed some frames for some period of time. In such cases, the application designer does not need to worry much about the workload.

In other cases, especially the ones involving applications from groups (3) and (4) mentioned above, the significant peak workload may lead to visual stuttering, as any processing making the browser busy for longer than 16.6 milliseconds will lead to lost frames. In such cases, the workload has to be managed in a way that the peaks are reduced either by optimizing them or distributing them over time.

First step: optimization #

The first step to addressing the peak workload is usually optimization. Modern web platform gives a full variety of tools to optimize all the stages of web application processing done by the browser. The usual process of optimization is a 2-step cycle starting with measuring the bottlenecks and followed by fixing them. In the process, the usual improvements involve:

• using CSS containment,
• using shadow DOM,
• promoting certain parts of the DOM to layers and manipulating them with transforms,
• parallelizing the work with workers/worklets,
• using the visibility CSS property to separate painting from layout,
• optimizing the application itself (JavaScript code, the structure of the DOM, the architecture of the application),
• etc.

Second step: pre-rendering #

Unfortunately, in practice, it’s not uncommon that even very well optimized applications still have too much of a peak workload for the constrained embedded devices they’re used on. In such cases, the last resort solution is pre-rendering. As long as it’s possible from the application business-logic perspective, having at least some web page content pre-rendered is very helpful in situations when workload has to be managed, as pre-rendering allows the web application designer to choose the precise moment when the content should actually be rendered and how it should be done. With that, it’s possible to establish a proper trade-off between reduction in peak workload and the amount of extra memory used for storing the pre-rendered contents.

Pre-rendering techniques #

Nowadays, the web platform provides at lest a few widely-adapted APIs that provide means for the application to perform various kinds of pre-rendering. Also, due to the ways the browsers are implemented, some APIs can be purposely misused to provide pre-rendering techniques not necessarily supported by the specification. However, in the pursuit of good trade-offs, all the possibilities should be taken into account.

Before jumping into particular pre-rendering techniques, it’s necessary to emphasize that the pre-rendering term used in this article refers to the actual rendering being done earlier than it’s visually presented. In that sense, the resource is rasterized to some intermediate form when desired and then just composited by the browser engine’s compositor later.

Pre-rendering offline #

The most basic (and limited at the same time) pre-rendering technique is one that involves rendering offline i.e. before the browser even starts. In that case, the first limitation is that the content to be rendered must be known beforehand. If that’s the case, the rendering can be done in any way, and the result may be captured as e.g. raster or vector image (depending on the desired trade-off). However, the other problem is that such a rendering is usually out of the given web application scope and thus requires extra effort. Moreover, depending on the situation, the amount of extra memory used, the longer web application startup (due to loading the pre-rendered resources), and the processing power required to composite a given resource, it may not always be trivial to obtain the desired gains.

Pre-rendering using canvas #

The first group of actual pre-rendering techniques happening during web application runtime is related to Canvas and OffscreenCavas. Those APIs are really useful as they offer great flexibility in terms of usage and are usually very performant. However, in this case, the natural downside is the lack of support for rendering the DOM inside the canvas. Moreover, canvas has a very limited support for painting text — unlike the DOM, where CSS has a significant amount of features related to it. Interestingly, there’s an ongoing proposal called HTML-in-Canvas that could resolve those limitations to some degree. In fact, Blink has a functioning prototype of it already. However, it may take a while before the spec is mature and widely adopted by other browser engines.

When it comes to actual usage of canvas APIs for pre-rendering, the possibilities are numerous, and there are even more of them when combined with processing using workers. The most popular ones are as follows:

• rendering to an invisible canvas and showing it later,
• rendering to a canvas detached from the DOM and attaching it later,
• rendering to an invisible/detached canvas and producing an image out of it to be shown later,
• rendering to an offscreen canvas and producing an image out of it to be shown later.

When combined with workers, some of the above techniques may be used in the worker threads with the rendered artifacts transferred to the main for presentation purposes. In that case, one must be careful with the transfer itself, as some objects may get serialized, which is very costly. To avoid that, it’s recommended to use transferable objects and always perform a proper benchmarking to make sure the transfer is not involving serialization in the particular case.

While the use of canvas APIs is usually very straightforward, one must be aware of two extra caveats.

First of all, in the case of many techniques mentioned above, there is no guarantee that the browser will perform actual rasterization at the given point in time. To ensure the rasterization is triggered, it’s usually necessary to enforce it using e.g. a dummy readback (getImageData()).

Finally, one should be aware that the usage of canvas comes with some overhead. Therefore, creating many canvases or creating them often, may lead to performance problems that could outweigh the gains from the pre-rendering itself.

Pre-rendering using eventually-invisible layers #

The second group of pre-rendering techniques happening during web application runtime is limited to the DOM rendering and comes out of a combination of purposeful spec misuse and tricking the browser engine into making it rasterizing on demand. As one can imagine, this group of techniques is very much browser-engine-specific. Therefore, it should always be backed by proper benchmarking of all the use cases on the target browsers and target hardware.

In principle, all the techniques of this kind consist of 3 parts:

1. Enforcing the content to be pre-rendered being placed on a separate layer backed by an actual buffer internally in the browser,
2. Tricking the browser’s compositor into thinking that the layer needs to be rasterized right away,
3. Ensuring the layer won’t be composited eventually.

When all the elements are combined together, the browser engine will allocate an internal buffer (e.g. texture) to back the given DOM fragment, it will process that fragment (style recalc, layout), and rasterize it right away. It will do so as it will not have enough information to allow delaying the rasterization of the layer (as e.g. in case of display: none). Then, when the compositing time comes, the layer will turn out to be invisible in practice due to e.g. being occluded, clipped, etc. This way, the rasterization will happen right away, but the results will remain invisible until a later time when the layer is made visible.

In practice, the following approaches can be used to trigger the above behavior:

• for (1), the CSS properties such as will-change: transform, z-index, position: fixed, overflow: hidden etc. can be used depending on the browser engine,
• for (2) and (3), the CSS properties such as opacity: 0, overflow: hidden, contain: strict etc. can be utilized, again, depending on the browser engine.

The scrolling trick

While the above CSS properties allow for various combinations, in case of WPE WebKit in the context of embedded devices (tested on NXP i.MX8M Plus), the combination that has proven to yield the best performance benefits turns out to be a simple approach involving overflow: hidden and scrolling. The example of such an approach is explained below.

Suppose the goal of the application is to update a big table with numbers once every N frames — like in the following demo: random-numbers-bursting-in-table.html?cs=20&rs=20&if=59

With the number of idle frames (if) set to 59, the idea is that the application does nothing significant for the 59 frames, and then every 60th frame it updates all the numbers in the table.

As one can imagine, on constrained embedded devices, such an approach leads to a very heavy workload during every 60th frame and hence to lost frames and unstable application’s FPS.

As long as the numbers are available earlier than every 60th frame, the above application is a perfect example where pre-rendering could be used to reduce the peak workload.

To simulate that, the 3 variants of the approach involving the scrolling trick were prepared for comparison with the above:

• random-numbers-bursting-in-table-prerendered-1.html?cs=20&rs=20&if=59
• random-numbers-bursting-in-table-prerendered-2.html?cs=20&rs=20&if=59
• random-numbers-bursting-in-table-prerendered-3.html?cs=20&rs=20&if=59

In the above demos, the idea is that each cell with a number becomes a scrollable container with 2 numbers actually — one above the other. In that case, because overflow: hidden is set, only one of the numbers is visible while the other is hidden — depending on the current scrolling:

With such a setup, it’s possible to update the invisible numbers during idle frames without the user noticing. Due to how WPE WebKit accelerates the scrolling, changing the invisible numbers, in practice, triggers the layout and rendering right away. Moreover, the actual rasterization to the buffer backing the scrollable container happens immediately (depending on the tiling settings), and hence the high cost of layout and text rasterization can be distributed. When the time comes, and all the numbers need to be updated, the scrollable containers can be just scrolled, which in that case turns out to be ~2 times faster than updating all the numbers in place.

To better understand the above effect, it’s recommended to compare the mark views from sysprof traces of the random-numbers-bursting-in-table.html?cs=10&rs=10&if=11 and random-numbers-bursting-in-table-prerendered-1.html?cs=10&rs=10&if=11 demos:

While the first sysprof trace shows very little processing during 11 idle frames and a big chunk of processing (21 ms) every 12th frame, the second sysprof trace shows how the distribution of load looks. In that case, the amount of work during 11 idle frames is much bigger (yet manageable), but at the same time, the formerly big chunk of processing every 12th frame is reduced almost 2 times (to 11 ms). Therefore, the overall frame rate in the application is much better.

Results

Despite the above improvement speaking for itself, it’s worth summarizing the improvement with the benchmarking results of the above demos obtained from the NXP i.MX8M Plus and presenting the application’s average frames per second (FPS):

Clearly, the positive impact of pre-rendering can be substantial depending on the conditions. In practice, when the rendered DOM fragment is more complex, the trick such as above can yield even better results. However, due to how tiling works, the effect can be minimized if the content to be pre-rendered spans multiple tiles. In that case, the browser may defer rasterization until the tiles are actually needed. Therefore, the above needs to be used with care and always with proper benchmarking.

Conclusions #

As demonstrated in the above sections, when it comes to pre-rendering the contents to distribute the web application workload over time, the web platform gives both the official APIs to do it, as well as unofficial means through purposeful misuse of APIs and exploitation of browser engine implementations. While this article hasn’t covered all the possibilities available, the above should serve as a good initial read with some easy-to-try solutions that may yield surprisingly good results. However, as some of the ideas mentioned above are very much browser-engine-specific, they should be used with extra care and with the limitations (lack of portability) in mind.

As the web platform constantly evolves, the pool of pre-rendering techniques and tricks should keep evolving as well. Also, as more and more web applications are used on embedded devices, more pressure should be put on the specification, which should yield more APIs targeting the low-end devices in the future. With that in mind, it’s recommended for the readers to stay up-to-date with the latest specification and perhaps even to get involved if some interesting use cases would be worth introducing new APIs.

Update on what happened in WebKit in the week from December 8 to December 15.

In this end-of-year special have a new GMallocString helper that makes management of malloc-based strings more efficient, development releases, and a handful of advancements on JSC's implementation of Temporal, in particular the PlainYearMonth class.

Cross-Port 🐱

JavaScriptCore 🐟

Releases 📦️

Update on what happened in WebKit in the week from December 1 to December 8.

In this edition of the periodical we have further advancements on the Temporal implementation, support for Vivante super-tiled format, and an adaptation of the DMA-BUF formats code to the Android port.

Cross-Port 🐱

JavaScriptCore 🐟

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Some years ago I had mentioned some command line tools I used to analyze and find useful information on GStreamer logs. I’ve been using them consistently along all these years, but some weeks ago I thought about unifying them in a single tool that could provide more flexibility in the mid term, and also as an excuse to unrust my Rust knowledge a bit. That’s how I wrote Meow, a tool to make cat speak (that is, to provide meaningful information).

The idea is that you can cat a file through meow and apply the filters, like this:

cat /tmp/log.txt | meow appsinknewsample n:V0 n:video ht: \
ft:-0:00:21.466607596 's:#([A-za-z][A-Za-z]*/)*#'

which means “select those lines that contain appsinknewsample (with case insensitive matching), but don’t contain V0 nor video (that is, by exclusion, only that contain audio, probably because we’ve analyzed both and realized that we should focus on audio for our specific problem), highlight the different thread ids, only show those lines with timestamp lower than 21.46 sec, and change strings like Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp to become just AppendPipeline.cpp“, to get an output as shown in this terminal screenshot:

Cool, isn’t it? After all, I’m convinced that the answer to any GStreamer bug is always hidden in the logs (or will be, as soon as I add “just a couple of log lines more, bro” ).

Currently, meow supports this set of manipulation commands:

• Word filter and highlighting by regular expression (fc:REGEX, or just REGEX): Every expression will highlight its matched words in a different color.
• Filtering without highlighting (fn:REGEX): Same as fc:, but without highlighting the matched string. This is useful for those times when you want to match lines that have two expressions (E1, E2) but the highlighting would pollute the line too much. In those case you can use a regex such as E1.*E2 and then highlight the subexpressions manually later with an h: rule.
• Negative filter (n:REGEX): Selects only the lines that don’t match the regex filter. No highlighting.
• Highlight with no filter (h:REGEX): Doesn’t discard any line, just highlights the specified regex.
• Substitution (s:/REGEX/REPLACE): Replaces one pattern for another. Any other delimiter character can be used instead of /, it that’s more convenient to the user (for instance, using # when dealing with expressions to manipulate paths).
• Time filter (ft:TIME-TIME): Assuming the lines start with a GStreamer log timestamp, this filter selects only the lines between the target start and end time. Any of the time arguments (or both) can be omitted, but the - delimiter must be present. Specifying multiple time filters will generate matches that fit on any of the time ranges, but overlapping ranges can trigger undefined behaviour.
• Highlight threads (ht:): Assuming a GStreamer log, where the thread id appears as the third word in the line, highlights each thread in a different color.

The REGEX pattern is a regular expression. All the matches are case insensitive. When used for substitutions, capture groups can be defined as (?CAPTURE_NAMEREGEX).

The REPLACEment string is the text that the REGEX will be replaced by when doing substitutions. Text captured by a named capture group can be referred to by ${CAPTURE_NAME}.

The TIME pattern can be any sequence of numbers, : or . . Typically, it will be a GStreamer timestamp (eg: 0:01:10.881123150), but it can actually be any other numerical sequence. Times are compared lexicographically, so it’s important that all of them have the same string length.

The filtering algorithm has a custom set of priorities for operations, so that they get executed in an intuitive order. For instance, a sequence of filter matching expressions (fc:, fn:) will have the same priority (that is, any of them will let a text line pass if it matches, not forbidding any of the lines already allowed by sibling expressions), while a negative filter will only be applied on the results left by the sequence of filters before it. Substitutions will be applied at their specific position (not before or after), and will therefore modify the line in a way that can alter the matching of subsequent filters. In general, the user doesn’t have to worry about any of this, because the rules are designed to generate the result that you would expect.

Now some practical examples:

Example 1: Select lines with the word “one”, or the word “orange”, or a number, highlighting each pattern in a different color except the number, which will have no color:

$ cat file.txt | meow one fc:orange 'fn:[0-9][0-9]*'
000 one small orange
005 one big orange

Example 2: Assuming a pictures filename listing, select filenames not ending in “jpg” nor in “jpeg”, and rename the filename to “.bak”, preserving the extension at the end:

$ cat list.txt | meow 'n:jpe?g' \
   's:#^(?<f>[^.]*)(?<e>[.].*)$#${f}.bak${e}'
train.bak.png
sunset.bak.gif

Example 3: Only print the log lines with times between 0:00:24.787450146 and 0:00:24.790741865 or those at 0:00:30.492576587 or after, and highlight every thread in a different color:

$ cat log.txt | meow ft:0:00:24.787450146-0:00:24.790741865 \
   ft:0:00:30.492576587- ht:
0:00:24.787450146 739 0x1ee2320 DEBUG …
0:00:24.790382735 739 0x1f01598 INFO …
0:00:24.790741865 739 0x1ee2320 DEBUG …
0:00:30.492576587 739 0x1f01598 DEBUG …
0:00:31.938743646 739 0x1f01598 ERROR …

This is only the begining. I have great ideas for this new tool (as time allows), such as support for parenthesis (so the expressions can be grouped), or call stack indentation on logs generated by tracers, in a similar way to what Alicia’s gst-log-indent-tracers tool does. I might also predefine some common expressions to use in regular expressions, such as the ones to match paths (so that the user doesn’t have to think about them and reinvent the wheel every time). Anyway, these are only ideas. Only time and hyperfocus slots will tell…

By now, you can find the source code on my github. Meow!