This is something that was previously mentioned in #55106 and has been heavily explored in Media Experiments already (see for example this blog post for an overview of the latter). I believe the core logic at the heart of Media Experiments is a great foundation to implement such a feature, and Gutenberg is the right place to further develop it and bring it to more people.

Overview

Current image processing in WordPress relies on server-side resources and older image libraries, leading to potential performance issues and limited support for modern image formats such as AVIF. This results in a subpar user experience, particularly with resource-intensive tasks like resizing and compressing images. Additionally, the lack of modern compression tools like MozJPEG further hinders optimization efforts.

Client-side media processing offers a solution by leveraging the browser's capabilities to handle tasks like image resizing and compression. This approach not only alleviates the strain on server resources but also enables the use of more advanced image formats and compression techniques, ultimately improving website performance and user experience. By tapping into technologies like WebAssembly, WordPress can provide a more efficient and seamless media handling process for both new and existing content.

Resources

• Add query for LCP image formats and file sizes GoogleChromeLabs/wpp-research#97
• https://github.com/swissspidy/media-experiments

Roadmap

Given the many possibilities client-side media processing unlocks, this project can be split into multiple phases, each with their dedicated goals and success criteria. For example:

Phase 1

Goals

• Client-side sub-size image generation
  • Including PDF thumbnails
• Image compression during upload (for all sizes)
  • Scale down images according to big image size threshold setting
  • Always compress all images, which is different from what WordPress does today
  • Provide opt-out if needed (e.g. for photographers)
• A way in the block editor for compressing an existing image

Nice to have

• Converting HEIF/HEIC images to a web-safe format
• A way in the post editor for bulk optimization of all existing images in a post
  • Not to be confused with regenerating the whole media library

Non-goals

• Converting (transcoding) images to any other format (e.g. JPEG to WebP)
  • Except for HEIC which is a nice-to-have above
• Expand to other media types (e.g. video compression)

Phase 2

Goals

• Converting (transcoding) images to any other format (e.g. JPEG to WebP)
  • Including sub-sizes of course

Nice to have

• Create poster images for videos
• Convert animated GIFs to actual videos
  • Add quick setting to video panel for "GIF-like" playback #19730
• Expand to other media types (e.g. video compression)
• Provide additional client-side image editing capabilities such as cropping, rotation, and scaling
  • Makes it easier for users to make smaller images and thus keep file size down

Non-goals

• UI for regenerating/compressing the whole media library

Phase 3

Goals

• Generate sub-sizes in multiple different formats
  • Useful for <picture> , e.g. for using AVIF with a JPEG fallback
  • Alternatively or additionally, for each sub-size pick the format with the smallest file size
• Expand functionality from block editor to media library, in accordance with media library revamp as part of Gutenberg phase 3

Nice to have

• Expand to other media types (e.g. video compression)

Out of scope

Non-goals at the moment, or something for far out in the future:

• UI for regenerating/compressing the whole media library

Caveats and risks

Technical complexity

WASM-based image optimization requires SharedArrayBuffer support, which in turn requires cross-origin isolation. Implementing that in a robust way without breaking other parts of the editor is challenging. There are currently some known issues in Firefox and Safari due to these browsers not supporting credentialless iframe embeds. Embed previews in the editor currently do not work because of this, until those browsers add support for <iframe credentialless>.

Image compression by itself is rather complex due to the vast amount of different combinations of codecs and encoding options available. While the existing server-side image handling in WordPress has been proven over many years, doing everything client-side is new territory. There will be many edge cases that need to be handled.

As for overall architecture design, the existing Media Experiments project contains a very solid foundation that already solves many challenging problems and could be built upon.

Confidence level

Reception in the community so far has been extremely positive, including from WordPress leadership. There is clearly a desire for this kind of solution.

The biggest risk is about compatibility with other parts of the editor (plugins, embed previews) and certain environments because of the cross-origin isolation required for SharedArrayBuffer. Not breaking existing sites is very important. The risk can be mitigated through extensive testing, additional safeguards, and documentation. Lack of <iframe credentialless> support is not ideal as it degrades editor UX in some cases, but it does not break the authoring experience as a fallback exists.

While image optimization could theoretically be done through other means such as the canvas API, said API is extremely limited across browsers. For example, there is a lack of support for compression options (e.g. lossless or lossy) and mime types (no AVIF everywhere, no WebP in Safari).

Implementation status

• Add experiment for client-side media processing #64650
• Add safeguard to mediaUploadMiddleware #64843
• Media Utils: Add TypeScript support and export more utils #64784
• Add new Media section to preferences modal #64846
• Add new private vips package #64845
• Add new private upload-media package #66290
• Integrate vips into upload queue