Skip to content

Commit 3254d60

Browse files
authored
[camera_web] Re: Support for camera stream on web (#7950)
## Description Adds image streaming to `camera_web`, so `CameraController.startImageStream` works on web. `CameraPlugin.supportsImageStreaming()` now returns `true`, and `onStreamedFrameAvailable` returns a stream of RGBA `CameraImageData` frames. ### How frames are captured There are two capture paths, chosen per camera when it is created: | Browser support | Path | Frame source | |---|---|---| | `MediaStreamTrackProcessor` available (Chromium-based browsers) | `MediaStreamTrackProcessor` → `ReadableStream<VideoFrame>` → `VideoFrame.copyTo(format: 'RGBA')` | Every frame the camera track produces (`maxBufferSize: 1`, so the processor keeps only the newest frame when the consumer falls behind) | | Otherwise (e.g. Firefox, Safari) | `requestAnimationFrame` loop → draw the `<video>` element to an `OffscreenCanvas` (or a regular canvas if `OffscreenCanvas` is unavailable) → `getImageData` | Sampled at a 60 fps target | Both paths emit a single-plane RGBA image (`bytesPerPixel: 4`, `bytesPerRow` set to the stride the browser reports). `format` is `ImageFormatGroup.unknown` with `raw: 'rgba8888'`, because the platform interface has no RGBA group yet (see TODO, flutter/flutter#151193). ### Frame ownership Every emitted frame has its own buffer. The `copyTo` destination is allocated per frame, and `JSUint8Array.toDart` returns a view of it, not a second copy. This matches the native implementations: a consumer can keep a `CameraImage` past the callback, for example across an `await` or when sending it to an isolate, and later frames won't change its bytes. An earlier revision reused one buffer across frames. I benchmarked both approaches before switching (Chrome 155 with the fake camera; 640×480 to 1080p at 30, 60 and 90 fps; 3 runs per case): - **Throughput:** identical up to 60 fps. At 1080p90, buffer reuse was +1.3%, but the fake source itself tops out at about 85 fps there. - **CPU and GC:** buffer reuse used about 45–50% less renderer CPU and avoided the GC caused by per-frame allocation. GC pauses with per-frame buffers were at most 4.7 ms, there were no long tasks, and memory did not grow over time. - **Correctness:** with buffer reuse, 100% of frames held for 50 ms or longer had been overwritten by later frames (70/70). With per-frame buffers, 0/70 were. A buffer pool would need an explicit release API on the platform interface, so I've left it as a possible follow-up. ### Lifecycle and errors - The frame stream is a broadcast stream. Capture starts when the first listener subscribes and stops when the last one cancels. `Camera.dispose()` now closes the stream. - A `VideoFrame` is always closed after it is copied, including on errors. The reader lock is released when the loop ends. - Frames that can't be copied are skipped, and the stream keeps going (`unableToCloneFrame`, `missingPlaneLayout`). The stream closes without an error if the track reader closes (`videoTrackReaderClosed`). - Any other error is emitted on the stream before it closes. In `onStreamedFrameAvailable`, a `CameraWebException` is reported as a camera error event and rethrown as a `PlatformException`. - New `CameraErrorCode`s: `noVideoTrack`, `videoTrackReaderClosed`, `videoTrackReaderNotInitialized`, `cameraFrameDimensionsZero`, `unableToCloneFrame`, `missingPlaneLayout`. ### Refactors in `CameraService` - The `DOMException` → `CameraWebException` mapping now lives in `_domExceptionToCameraWebException`, shared by `getMediaStreamForOptions` and the new frame-stream methods. `DOMException`s are now detected with `isA<DOMException>()` instead of `on DOMException`, so the check behaves the same under dart2wasm. - New methods so the stream loop can be mocked in tests: `hasMediaStreamTrackProcessor`, `hasPropertyOffScreenCanvas`, `takeFrame`, `getMediaStreamVideoTrack`, `getMediaStreamTrackReader`, `readVideoTrack`, `copyVideoFrameToBufferAndGetStride`, `getCameraImageData`. ### Known limitations / follow-ups - **No frame-rate option yet.** The fallback path targets 60 fps, and the `MediaStreamTrackProcessor` path delivers whatever the track produces. `CameraImageStreamOptions` has no fps field (TODO, flutter/flutter#176148). - **No RGBA format group** in `ImageFormatGroup` (flutter/flutter#151193). - **Crop-and-scale resolutions aren't scaled on the `MediaStreamTrackProcessor` path.** When Chrome crops and scales the track to fit the requested resolution (e.g. `ResolutionPreset.low` = 320×240 on a 1080p camera), the `VideoFrame`'s `visibleRect` is the crop (1440×1080), and only `displayWidth/displayHeight` are 320×240. Frames are copied at `visibleRect` size, so they can be larger than `previewSize`. Scaling to the display size could be a follow-up. Fixes flutter/flutter#92460 ### Testing - New integration tests, all passing in Chrome (`flutter drive -d chrome` for each file in `example/integration_test/`): - `camera_web_stream_test.dart`: `supportsImageStreaming`, `onStreamedFrameAvailable`, and the `PlatformException` mapping. - `camera_test.dart`, in the `cameraFrameStream` group: - emitting frames; - closing with an error when reading a frame or building `CameraImageData` fails; - closing quietly when the reader closes; - closing on dispose; - continuing after `unableToCloneFrame`; - the fallback path's frame-rate target and RGBA byte length; - fallback errors. - `camera_service_test.dart`: each new `CameraService` method, including its error cases. - `camera_error_code_test.dart`: the new error codes. - Manually tested with `startImageStream` in Chrome using a fake camera (`--use-file-for-fake-video-capture`) at 640×480 to 1080p and 30–90 fps. That covered frame dimensions, stride and byte length, stopping and restarting the stream, background tabs, and dispose. ## Pre-Review Checklist
1 parent 2f2f117 commit 3254d60

13 files changed

Lines changed: 1852 additions & 248 deletions

‎packages/camera/camera_web/CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
## 0.3.6
2+
3+
* Adds support for camera image stream on web.
4+
* Updates minimum `web` package version to 1.0.0.
5+
16
## 0.3.5+7
27

38
* Fixes `availableCameras` failing when a video input device cannot be started, for example

‎packages/camera/camera_web/example/integration_test/camera_error_code_test.dart‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,13 +76,43 @@ void main() {
7676
expect(CameraErrorCode.notStarted.toString(), equals('cameraNotStarted'));
7777
});
7878

79+
testWidgets('noVideoTrack', (WidgetTester tester) async {
80+
expect(CameraErrorCode.noVideoTrack.toString(), equals('noVideoTrack'));
81+
});
82+
83+
testWidgets('cameraFrameDimensionsZero', (WidgetTester tester) async {
84+
expect(
85+
CameraErrorCode.cameraFrameDimensionsZero.toString(),
86+
equals('cameraFrameDimensionsZero'),
87+
);
88+
});
89+
90+
testWidgets('videoTrackReaderClosed', (WidgetTester tester) async {
91+
expect(CameraErrorCode.videoTrackReaderClosed.toString(), equals('videoTrackReaderClosed'));
92+
});
93+
94+
testWidgets('videoTrackReaderNotInitialized', (WidgetTester tester) async {
95+
expect(
96+
CameraErrorCode.videoTrackReaderNotInitialized.toString(),
97+
equals('videoTrackReaderNotInitialized'),
98+
);
99+
});
100+
79101
testWidgets('videoRecordingNotStarted', (WidgetTester tester) async {
80102
expect(
81103
CameraErrorCode.videoRecordingNotStarted.toString(),
82104
equals('videoRecordingNotStarted'),
83105
);
84106
});
85107

108+
testWidgets('missingPlaneLayout', (WidgetTester tester) async {
109+
expect(CameraErrorCode.missingPlaneLayout.toString(), equals('missingPlaneLayout'));
110+
});
111+
112+
testWidgets('unableToCloneFrame', (WidgetTester tester) async {
113+
expect(CameraErrorCode.unableToCloneFrame.toString(), equals('unableToCloneFrame'));
114+
});
115+
86116
testWidgets('unknown', (WidgetTester tester) async {
87117
expect(CameraErrorCode.unknown.toString(), equals('cameraUnknown'));
88118
});

‎packages/camera/camera_web/example/integration_test/camera_service_test.dart‎

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
// ignore_for_file: only_throw_errors
66

7+
import 'dart:async';
78
import 'dart:js_interop';
89
import 'dart:js_interop_unsafe';
910

@@ -793,5 +794,293 @@ void main() {
793794
);
794795
});
795796
});
797+
798+
group('hasMediaStreamTrackProcessor', () {
799+
setUp(() {
800+
cameraService.jsUtil = jsUtil;
801+
});
802+
803+
testWidgets('returns true if broswer has MediaStreamTrackProcessor '
804+
'otherwise false', (WidgetTester widgetTester) async {
805+
for (final supportsMediaStreamTrackProcessor in <bool>[true, false]) {
806+
when(
807+
jsUtil.hasProperty(window, 'MediaStreamTrackProcessor'.toJS),
808+
).thenReturn(supportsMediaStreamTrackProcessor);
809+
810+
final bool hasMediaStreamTrackProcessor = cameraService.hasMediaStreamTrackProcessor();
811+
812+
expect(
813+
hasMediaStreamTrackProcessor,
814+
supportsMediaStreamTrackProcessor ? isTrue : isFalse,
815+
);
816+
}
817+
});
818+
});
819+
820+
group('getMediaStreamVideoTrack', () {
821+
testWidgets('returns the first video track when tracks exist in the stream', (
822+
WidgetTester widgetTester,
823+
) async {
824+
final web.MediaStream mockStream = getVideoElementWithBlankStream(
825+
const Size(10, 10),
826+
).captureStream();
827+
828+
final web.MediaStreamTrack track = cameraService.getMediaStreamVideoTrack(mockStream);
829+
830+
expect(track, isNotNull);
831+
});
832+
833+
testWidgets('throw CameraWebException when no video tracks are found', (
834+
WidgetTester widgetTester,
835+
) async {
836+
final mockStream = createJSInteropWrapper(FakeMediaStream([])) as web.MediaStream;
837+
838+
expect(
839+
() => cameraService.getMediaStreamVideoTrack(mockStream, cameraId: 42),
840+
throwsA(
841+
isA<CameraWebException>().having((e) => e.code, 'code', CameraErrorCode.noVideoTrack),
842+
),
843+
);
844+
});
845+
});
846+
847+
group('getMediaStreamTrackReader', () {
848+
testWidgets('returns ReadableStreamDefaultReader of MediaStreamTrack', (
849+
WidgetTester tester,
850+
) async {
851+
final web.MediaStreamTrack mediaStreamTrack = getVideoElementWithBlankStream(
852+
const Size(10, 10),
853+
).captureStream().getTracks().toDart.first;
854+
855+
final web.ReadableStreamDefaultReader readableStreamDefaultReader = cameraService
856+
.getMediaStreamTrackReader(mediaStreamTrack);
857+
858+
expect(
859+
readableStreamDefaultReader.read().toDart,
860+
isA<Future<web.ReadableStreamReadResult>>(),
861+
);
862+
mediaStreamTrack.stop();
863+
});
864+
});
865+
866+
group('readVideoTrack', () {
867+
setUp(() {
868+
cameraService.isVideoFrameType = (_) => true;
869+
});
870+
871+
web.ReadableStreamDefaultReader mockMediaTrackReader({
872+
MockVideoFrame? mockFrame,
873+
bool done = false,
874+
}) {
875+
final mockReader = MockReadableStreamDefaultReader();
876+
mockReader.read = (() => Future.value(
877+
web.ReadableStreamReadResult(
878+
done: done,
879+
value: mockFrame == null ? null : createJSInteropWrapper(mockFrame) as JSAny,
880+
),
881+
).toJS).toJS;
882+
return createJSInteropWrapper(mockReader) as web.ReadableStreamDefaultReader;
883+
}
884+
885+
testWidgets('returns VideoFrame from ReadableStreamDefaultReader', (
886+
WidgetTester widgetTester,
887+
) async {
888+
final mockFrame = MockVideoFrame(width: 10, height: 10, bufferSize: 32);
889+
final web.ReadableStreamDefaultReader mockReader = mockMediaTrackReader(
890+
mockFrame: mockFrame,
891+
);
892+
893+
final web.VideoFrame videoFrame = await cameraService.readVideoTrack(mockReader);
894+
895+
expect(videoFrame.visibleRect, equals(mockFrame.visibleRect));
896+
expect(
897+
videoFrame.allocationSize(web.VideoFrameCopyToOptions()),
898+
equals(mockFrame.bufferSize),
899+
);
900+
expect(videoFrame.format, equals(mockFrame.format));
901+
});
902+
903+
void expectReadVideoFrameException({
904+
required web.ReadableStreamDefaultReader reader,
905+
required CameraErrorCode errorCode,
906+
}) {
907+
expect(
908+
() => cameraService.readVideoTrack(reader),
909+
throwsA(
910+
isA<CameraWebException>().having((CameraWebException e) => e.code, 'code', errorCode),
911+
),
912+
);
913+
}
914+
915+
testWidgets('throws CameraWebException if the video track reader is closed', (
916+
WidgetTester tester,
917+
) async {
918+
final web.ReadableStreamDefaultReader mockReader = mockMediaTrackReader(done: true);
919+
920+
expectReadVideoFrameException(
921+
reader: mockReader,
922+
errorCode: CameraErrorCode.videoTrackReaderClosed,
923+
);
924+
});
925+
926+
testWidgets('throws CameraWebException if the video track reader is not initialized', (
927+
WidgetTester tester,
928+
) async {
929+
final mockFrame = MockVideoFrame(width: 0, height: 0, bufferSize: 0)..visibleRect = null;
930+
final web.ReadableStreamDefaultReader mockReader = mockMediaTrackReader(
931+
mockFrame: mockFrame,
932+
);
933+
934+
expectReadVideoFrameException(
935+
reader: mockReader,
936+
errorCode: CameraErrorCode.videoTrackReaderNotInitialized,
937+
);
938+
});
939+
});
940+
941+
group('copyVideoFrameToBufferAndGetStride', () {
942+
web.VideoFrame stubVideoFrameCopyTo(JSPromise<JSArray<web.PlaneLayout>> Function() result) {
943+
final mockFrame = MockVideoFrame(width: 10, height: 10, bufferSize: 32);
944+
mockFrame.copyTo =
945+
((JSAny? destination, web.VideoFrameCopyToOptions options) => result()).toJS;
946+
return createJSInteropWrapper(mockFrame) as web.VideoFrame;
947+
}
948+
949+
testWidgets('returns first plane stride', (WidgetTester tester) async {
950+
const expectedStride = 640;
951+
final web.VideoFrame videoFrame = stubVideoFrameCopyTo(
952+
() => Future<JSArray<web.PlaneLayout>>.value(
953+
JSArray<web.PlaneLayout>()..add(web.PlaneLayout(offset: 0, stride: expectedStride)),
954+
).toJS,
955+
);
956+
957+
final int stride = await cameraService.copyVideoFrameToBufferAndGetStride(
958+
videoFrame,
959+
destination: JSUint8Array.withLength(100),
960+
copyOptions: web.VideoFrameCopyToOptions(),
961+
);
962+
963+
expect(stride, expectedStride);
964+
});
965+
966+
void expectCameraWebException(web.VideoFrame videoFrame, CameraErrorCode code) {
967+
expect(
968+
() => cameraService.copyVideoFrameToBufferAndGetStride(
969+
videoFrame,
970+
destination: JSUint8Array.withLength(100),
971+
copyOptions: web.VideoFrameCopyToOptions(),
972+
),
973+
throwsA(isA<CameraWebException>().having((CameraWebException e) => e.code, 'code', code)),
974+
);
975+
}
976+
977+
testWidgets('throws missingPlaneLayout when browser returns no planes', (
978+
WidgetTester tester,
979+
) async {
980+
final web.VideoFrame videoFrame = stubVideoFrameCopyTo(
981+
() => Future<JSArray<web.PlaneLayout>>.value(JSArray<web.PlaneLayout>()).toJS,
982+
);
983+
984+
expectCameraWebException(videoFrame, CameraErrorCode.missingPlaneLayout);
985+
});
986+
987+
testWidgets('throws unableToCloneFrame when DOMException occurs', (
988+
WidgetTester tester,
989+
) async {
990+
final web.VideoFrame videoFrame = stubVideoFrameCopyTo(
991+
() => throw web.DOMException('Frame is corrupted', 'InvalidStateError'),
992+
);
993+
994+
expectCameraWebException(videoFrame, CameraErrorCode.unableToCloneFrame);
995+
});
996+
});
997+
998+
group('getCameraImageData', () {
999+
testWidgets('returns CameraImageData with valid planes', (WidgetTester tester) async {
1000+
const mockWidth = 2;
1001+
const mockHeight = 2;
1002+
final mockBytes = Uint8List(mockWidth * mockHeight * 4);
1003+
1004+
final CameraImageData cameraImageData = cameraService.getCameraImageData(
1005+
width: mockWidth,
1006+
height: mockHeight,
1007+
bytes: mockBytes,
1008+
bytesPerRow: mockWidth * 4,
1009+
);
1010+
1011+
expect(cameraImageData.width, equals(mockWidth));
1012+
expect(cameraImageData.height, equals(mockHeight));
1013+
1014+
expect(cameraImageData.format.group, equals(ImageFormatGroup.unknown));
1015+
expect(cameraImageData.format.raw, equals('rgba8888'));
1016+
1017+
expect(cameraImageData.planes.length, equals(1));
1018+
final CameraImagePlane plane = cameraImageData.planes.first;
1019+
1020+
expect(plane.width, equals(mockWidth));
1021+
expect(plane.height, equals(mockHeight));
1022+
expect(plane.bytesPerPixel, equals(4));
1023+
expect(plane.bytesPerRow, equals(mockWidth * 4));
1024+
expect(plane.bytes, equals(mockBytes));
1025+
});
1026+
});
1027+
1028+
group('hasPropertyOffScreenCanvas', () {
1029+
setUp(() {
1030+
cameraService.jsUtil = jsUtil;
1031+
});
1032+
testWidgets('hasPropertyOffScreenCanvas returns true if broswer has OffscreenCanvas '
1033+
'otherwise false', (WidgetTester widgetTester) async {
1034+
for (final supportsOffscreenCanvas in <bool>[true, false]) {
1035+
when(
1036+
jsUtil.hasProperty(window, 'OffscreenCanvas'.toJS),
1037+
).thenReturn(supportsOffscreenCanvas);
1038+
final bool hasOffScreenCanvas = cameraService.hasPropertyOffScreenCanvas();
1039+
expect(hasOffScreenCanvas, supportsOffscreenCanvas ? isTrue : isFalse);
1040+
}
1041+
});
1042+
});
1043+
1044+
group('takeFrame', () {
1045+
testWidgets('returns Camera Image of Size '
1046+
'when videoElement is of Size '
1047+
'regardless of OffscreenCanvas support', (WidgetTester widgetTester) async {
1048+
const size = Size(10, 10);
1049+
final completer = Completer<void>();
1050+
final web.VideoElement videoElement = getVideoElementWithBlankStream(size)
1051+
..onLoadedMetadata.listen((_) {
1052+
completer.complete();
1053+
})
1054+
..load();
1055+
await completer.future;
1056+
1057+
for (final supportsOffscreenCanvas in <bool>[true, false]) {
1058+
when(
1059+
jsUtil.hasProperty(window, 'OffscreenCanvas'.toJS),
1060+
).thenReturn(supportsOffscreenCanvas);
1061+
1062+
final CameraImageData cameraImageData = cameraService.takeFrame(videoElement);
1063+
1064+
expect(size, Size(cameraImageData.width.toDouble(), cameraImageData.height.toDouble()));
1065+
}
1066+
});
1067+
1068+
testWidgets('throws CameraWebException when video element not loaded', (
1069+
WidgetTester widgetTester,
1070+
) async {
1071+
final web.VideoElement videoElement = getVideoElementWithBlankStream(const Size(10, 10));
1072+
1073+
expect(
1074+
() => cameraService.takeFrame(videoElement),
1075+
throwsA(
1076+
isA<CameraWebException>().having(
1077+
(CameraWebException e) => e.code,
1078+
'code',
1079+
CameraErrorCode.cameraFrameDimensionsZero,
1080+
),
1081+
),
1082+
);
1083+
});
1084+
});
7961085
});
7971086
}

0 commit comments

Comments
 (0)