Windows Media Foundation and FFMPEG sample code for streaming an application window to a web browser. The video is encoded as a H.264 stream inside a fragmented MPEG4 container that is Media Source Extensions (MSE) compatible, so that it can be received by modern web browsers with minimal client-side buffering.

Scope

This repo contains a reference implementation that demonstrates how MPEG4 transmit and receive can be implemented in a product.

Intended usage

• Lightweight environment for experimenting with video encoding settings.
• Starting point for developing product implementations.
• Independent implementation for compatibility testing.

Testing possibilities

The AppWebStream and StreamReceiver projects in this repo can be used together with web browsers, ffplay and VLC to test other apps ability to both transmit and receive streamed MPEG4 videos:

Getting started

• Open project in Visual Studio.
• Build project.
• Start WebAppStream.exe port [window handle]. You can use Spy++ (spyxx.exe distributed with Visual Studio) to determine window handles.
• Open http://localhost:port in web browser.

To build with FFMPG, you first need to download & unzip FFMPEG binaries to a folder pointed to by the FFMPEG_ROOT environment variable. Then, set the ENABLE_FFMPEG preprocessor define before building.

AppWebStream implementation details

Video metadata status

• All frames time-stamped with <0.1ms temporal accuracy against the MPEG4 1904 epoch
• Pixel-to-world coordinate transform metadata in mvhd atom
• Pixel spacing (DPI) metadata in stsd atom
• Stream restart to enable coordinate transform and DPI changes
• Freeze & resume frame time-stamps might lead to paused web browser playback (issue #24)

Media Foundation details

• 0 frame latency, except for the first 4 frames (frame N in, frame N out, frame N+1 in, frame N+1 out, frame N+2 in, frame N+2 out, ...)
• The MPEG4 container is manually modified as suggested in MFCreateFMPEG4MediaSink does not generate MSE-compatible MP4 to make it Media Source Extensions (MSE) compatible for streaming. The FFMPEG-based encoder is not affected by this issue.

HTTP and authentication

• Authentication is currently missing.
• The handcrafted HTTP communication should be replaced by a HTTP library (issue #33).

Frame grabbing method

The project is currently using the GDI BitBlt function to copy the content of the specified window handle to an offscreen HBITMAP object. This works fine for many applications, but single window capture doesn't work for apps that use GPU-accelerated drawing of the parent window. It would therefore probably be better to switch to the newer Desktop Duplication API for frame grabbing (sample).

Client support

The video stream adheres to the MPEG4 standard, so it should be supported by all client SW supporting H.264 encoded MPEG4 streams. The following clients have already been verified to work:

• Google Chrome
• Microsoft Edge
• Mozilla Firefox
• VLC media player: Open stream as http://localhost:port/movie.mp4
• ffplay: ffplay.exe -flags low_delay -i "http://localhost:port/movie.mp4"
• StreamReceiver project in this repo (latency issue #31)

Unsupported clients

• Doesn't yet work on iOS, due to incomplete ManagedMediaSource support (issue #25).

External resources

Tools for inspecting MPEG metadata:

• MP4Box.js / ISOBMFF Box Structure Viewer
• ExifTool can be used for extracting metadata if saving the stream to file.

External links:

• Windows Media Foundation samples on GitHub