Repository navigation
Commit 3254d60
authored
[camera_web] Re: Support for camera stream on web (#7950)
## Description
Adds image streaming to `camera_web`, so `CameraController.startImageStream` works on web. `CameraPlugin.supportsImageStreaming()` now returns `true`, and `onStreamedFrameAvailable` returns a stream of RGBA `CameraImageData` frames.
### How frames are captured
There are two capture paths, chosen per camera when it is created:
| Browser support | Path | Frame source |
|---|---|---|
| `MediaStreamTrackProcessor` available (Chromium-based browsers) | `MediaStreamTrackProcessor` → `ReadableStream<VideoFrame>` → `VideoFrame.copyTo(format: 'RGBA')` | Every frame the camera track produces (`maxBufferSize: 1`, so the processor keeps only the newest frame when the consumer falls behind) |
| Otherwise (e.g. Firefox, Safari) | `requestAnimationFrame` loop → draw the `<video>` element to an `OffscreenCanvas` (or a regular canvas if `OffscreenCanvas` is unavailable) → `getImageData` | Sampled at a 60 fps target |
Both paths emit a single-plane RGBA image (`bytesPerPixel: 4`, `bytesPerRow` set to the stride the browser reports). `format` is `ImageFormatGroup.unknown` with `raw: 'rgba8888'`, because the platform interface has no RGBA group yet (see TODO, flutter/flutter#151193).
### Frame ownership
Every emitted frame has its own buffer. The `copyTo` destination is allocated per frame, and `JSUint8Array.toDart` returns a view of it, not a second copy. This matches the native implementations: a consumer can keep a `CameraImage` past the callback, for example across an `await` or when sending it to an isolate, and later frames won't change its bytes.
An earlier revision reused one buffer across frames. I benchmarked both approaches before switching (Chrome 155 with the fake camera; 640×480 to 1080p at 30, 60 and 90 fps; 3 runs per case):
- **Throughput:** identical up to 60 fps. At 1080p90, buffer reuse was +1.3%, but the fake source itself tops out at about 85 fps there.
- **CPU and GC:** buffer reuse used about 45–50% less renderer CPU and avoided the GC caused by per-frame allocation. GC pauses with per-frame buffers were at most 4.7 ms, there were no long tasks, and memory did not grow over time.
- **Correctness:** with buffer reuse, 100% of frames held for 50 ms or longer had been overwritten by later frames (70/70). With per-frame buffers, 0/70 were.
A buffer pool would need an explicit release API on the platform interface, so I've left it as a possible follow-up.
### Lifecycle and errors
- The frame stream is a broadcast stream. Capture starts when the first listener subscribes and stops when the last one cancels. `Camera.dispose()` now closes the stream.
- A `VideoFrame` is always closed after it is copied, including on errors. The reader lock is released when the loop ends.
- Frames that can't be copied are skipped, and the stream keeps going (`unableToCloneFrame`, `missingPlaneLayout`). The stream closes without an error if the track reader closes (`videoTrackReaderClosed`).
- Any other error is emitted on the stream before it closes. In `onStreamedFrameAvailable`, a `CameraWebException` is reported as a camera error event and rethrown as a `PlatformException`.
- New `CameraErrorCode`s: `noVideoTrack`, `videoTrackReaderClosed`, `videoTrackReaderNotInitialized`, `cameraFrameDimensionsZero`, `unableToCloneFrame`, `missingPlaneLayout`.
### Refactors in `CameraService`
- The `DOMException` → `CameraWebException` mapping now lives in `_domExceptionToCameraWebException`, shared by `getMediaStreamForOptions` and the new frame-stream methods. `DOMException`s are now detected with `isA<DOMException>()` instead of `on DOMException`, so the check behaves the same under dart2wasm.
- New methods so the stream loop can be mocked in tests: `hasMediaStreamTrackProcessor`, `hasPropertyOffScreenCanvas`, `takeFrame`, `getMediaStreamVideoTrack`, `getMediaStreamTrackReader`, `readVideoTrack`, `copyVideoFrameToBufferAndGetStride`, `getCameraImageData`.
### Known limitations / follow-ups
- **No frame-rate option yet.** The fallback path targets 60 fps, and the `MediaStreamTrackProcessor` path delivers whatever the track produces. `CameraImageStreamOptions` has no fps field (TODO, flutter/flutter#176148).
- **No RGBA format group** in `ImageFormatGroup` (flutter/flutter#151193).
- **Crop-and-scale resolutions aren't scaled on the `MediaStreamTrackProcessor` path.** When Chrome crops and scales the track to fit the requested resolution (e.g. `ResolutionPreset.low` = 320×240 on a 1080p camera), the `VideoFrame`'s `visibleRect` is the crop (1440×1080), and only `displayWidth/displayHeight` are 320×240. Frames are copied at `visibleRect` size, so they can be larger than `previewSize`. Scaling to the display size could be a follow-up.
Fixes flutter/flutter#92460
### Testing
- New integration tests, all passing in Chrome (`flutter drive -d chrome` for each file in `example/integration_test/`):
- `camera_web_stream_test.dart`: `supportsImageStreaming`, `onStreamedFrameAvailable`, and the `PlatformException` mapping.
- `camera_test.dart`, in the `cameraFrameStream` group:
- emitting frames;
- closing with an error when reading a frame or building `CameraImageData` fails;
- closing quietly when the reader closes;
- closing on dispose;
- continuing after `unableToCloneFrame`;
- the fallback path's frame-rate target and RGBA byte length;
- fallback errors.
- `camera_service_test.dart`: each new `CameraService` method, including its error cases.
- `camera_error_code_test.dart`: the new error codes.
- Manually tested with `startImageStream` in Chrome using a fake camera (`--use-file-for-fake-video-capture`) at 640×480 to 1080p and 30–90 fps. That covered frame dimensions, stride and byte length, stopping and restarting the stream, background tabs, and dispose.
## Pre-Review Checklist1 parent 2f2f117 commit 3254d60
13 files changed
Lines changed: 1852 additions & 248 deletions
File tree
- packages/camera/camera_web
- example/integration_test
- helpers
- lib/src
- types
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
1 | 6 | | |
2 | 7 | | |
3 | 8 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
79 | 101 | | |
80 | 102 | | |
81 | 103 | | |
82 | 104 | | |
83 | 105 | | |
84 | 106 | | |
85 | 107 | | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
86 | 116 | | |
87 | 117 | | |
88 | 118 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
793 | 794 | | |
794 | 795 | | |
795 | 796 | | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
796 | 1085 | | |
797 | 1086 | | |
0 commit comments