RadCam 4K videos not being recorded/played on Electron/Browser#1822
RadCam 4K videos not being recorded/played on Electron/Browser#1822ArturoManzoli wants to merge 3 commits intobluerobotics:masterfrom
Conversation
20d2ed1 to
b320848
Compare
| :class="{ | ||
| 'blob red w-5 opacity-100 rounded-sm': isRecording, | ||
| 'blob red w-5 opacity-100 rounded-sm mr-[6px]': | ||
| isRecording || videoStore.preparingToRecord === nameSelectedStream, |
There was a problem hiding this comment.
preparingToRecord is a system logic state, while nameSelectedStream is a string reference.
From what I'm reading in the code, there's no reason for it not to be a boolean.
There was a problem hiding this comment.
This was made so when there are more than one MiniVideoRecorder instantiated, this status isn't shared between them.
Otherwise, when one video starts recording, the 'preparing' message would be displayed on both mini-widgets. We already have this issue with the isProcessingVideo variable being a boolean. When you press stop in one recorder, they all are set to 'Video processing' state.
There was a problem hiding this comment.
In this case, the variable should be renamed to something that indicates that it is not a boolean, but the name of the stream being prepared.
Also, since it only stores the name of one stream, what happens if there are two streams and the user clicks the "start recording all streams" action? It looks by the logic that only the last one will be on "preparing" state, while the first one will be overwritten.
src/stores/video.ts
Outdated
| activeStreams.value[streamName]!.mediaRecorder!.start(1000) | ||
| activeStreams.value[streamName]!.mediaRecorder!.start(500) |
There was a problem hiding this comment.
Is this change strictly needed? I ask it because it's a breaking change for any tool that is using the default size of 1 second for video chunks.
There was a problem hiding this comment.
I see. I'll try to make it 1000ms
There was a problem hiding this comment.
I still don't get it. In which kind of storage Is 600KB/ writing speed a problem?
There was a problem hiding this comment.
You were right. There was no limitation going on in this case, and the consequences of changing that would not be easy to deal with.
Switched back to 1000ms
src/stores/video.ts
Outdated
| const lastRenamedStreamName = ref('') | ||
| const preparingToRecord = ref<string | undefined>(undefined) | ||
|
|
||
| const options = { mimeType: 'video/webm; codecs=vp9' } |
There was a problem hiding this comment.
Is this change strictly needed? What does it change and why?
I ask it for the same reason as the chunk-size change.
There was a problem hiding this comment.
This is not great for performance: it will enforce a re-encoding from h264/h265 to vp9.
There was a problem hiding this comment.
Agreed, I'll double check to see if there is an alternative or if this is indeed an important change
b320848 to
9bc26ff
Compare
src/stores/video.ts
Outdated
| // Picks the best codec available for the stream | ||
| let streamCodecOptions | ||
| if (MediaRecorder.isTypeSupported('video/mp4;codecs=hvc1.1.6.L120.90')) { | ||
| // HEVC (H.265) | ||
| streamCodecOptions = { mimeType: 'video/mp4;codecs=hvc1.1.6.L120.90' } | ||
| } else if (MediaRecorder.isTypeSupported('video/mp4;codecs=avc1.42E01E')) { | ||
| // AVC (H.264) | ||
| streamCodecOptions = { mimeType: 'video/mp4;codecs=avc1.42E01E' } | ||
| } else { | ||
| // WebM/VP9 | ||
| streamCodecOptions = { mimeType: 'video/webm;codecs=vp9' } | ||
| } |
There was a problem hiding this comment.
This is tricky. I'm not completely sure about the behavior difference of the MediaRecorder when recording with or without specifying the mimeType.
In the absence of specification in the standard about this behavior, it's up to each browser implementation to decide whether it would re-encode or not. If we want to understand this, we could extensively test it or look at Chrome's source code to see what they are doing.
There is a possibility that when we don't specify the mimeType, it does not re-encode, and with this change, we lose this performance opportunity.
There was a problem hiding this comment.
The problem is that when mimeType is not specified, the video processing fails.
What I'm trying to do here is to get the stream's specs so there is no need to re-encode.
There was a problem hiding this comment.
The problem is that when mimeType is not specified, the video processing fails.
What I'm trying to do here is to get the stream's specs so there is no need to re-encode.
We could strategically try without setting the mimeType, with a fallback to setting it.
There was a problem hiding this comment.
The problem is that when mimeType is not specified, the video processing fails.
What I'm trying to do here is to get the stream's specs so there is no need to re-encode.
Are you sure? If that was true, people wouldn't be able to process at all, since we don't force the specification of the MimeType today...
There was a problem hiding this comment.
Oh, good to remind that the browser will only record the encodings it supports -- if the browser can't play H265, likely, it won't record H265.
There was a problem hiding this comment.
Are you sure? If that was true, people wouldn't be able to process at all, since we don't force the specification of the MimeType today...
Regular videos are processing as usual without specifying the mimeType, but 4k videos from the RadCam do not process at all.
Maybe is my camera setup.. Are you able to process any video from RadCam on Electron or even web using master branch?
There was a problem hiding this comment.
Yup. And Tony is also, as it uses it regularly. I suggest talking to him to better understand the problem. The original issue was only about subtitles generation, not video files (those were processing fine).
There was a problem hiding this comment.
When talking to Tony it became clear that the video processing as a whole wasn't ending normally. The chunk files were being stitched together but the processing was braking before being completed.
He was receiving the error Fail to process video file but was still able to get the video files from the folder (Electron version), even though the telemetry subtitles generation was being halted by the video processing failing.
A method that gets RTCP peer connection details from the Web RTC Manager was added on the last patch, so the appropriate codec is used on the stream being recorded. Now the processing issues seem to be solved and the correct codec should be always selected.
@rafaellehmkuhl can you test on your radcam if the videos and telemetry subtitles recorded with this modifications are processing fine?
9bc26ff to
fb0fe19
Compare
7d4f97d to
8f16c0a
Compare
src/stores/video.ts
Outdated
| const getRecorderOptions = async (streamData: StreamData): Promise<MediaRecorderOptions> => { | ||
| // Check on WebRTCManager to exposed RTCPeerConnection details. | ||
| const waitForPeerConnection = async (): Promise<RTCPeerConnection | null> => { | ||
| const until = Date.now() + 1_500 |
There was a problem hiding this comment.
1500 is a small number. No need for Literal underscores here.
src/stores/video.ts
Outdated
| const pc = | ||
| (streamData.webRtcManager as any).peerConnection ?? (streamData.webRtcManager as any).session?.peerConnection | ||
| if (pc) return pc as RTCPeerConnection | ||
| await new Promise((r) => setTimeout(r, 80)) |
There was a problem hiding this comment.
There's a sleep function in utils.ts.
src/stores/video.ts
Outdated
|
|
||
| const getRecorderOptions = async (streamData: StreamData): Promise<MediaRecorderOptions> => { | ||
| // Check on WebRTCManager to exposed RTCPeerConnection details. | ||
| const waitForPeerConnection = async (): Promise<RTCPeerConnection | null> => { |
There was a problem hiding this comment.
Since this function is very simple and only used here, can we make it not a function, and just code it inside the context?
src/stores/video.ts
Outdated
| const peerConnection = await waitForPeerConnection() | ||
| let detectedMime: string | null = null | ||
|
|
||
| if (peerConnection?.getStats) { | ||
| const stats = await peerConnection.getStats() | ||
| stats.forEach((r) => { | ||
| if (r.type === 'inbound-rtp' && r.kind === 'video' && r.codecId) { | ||
| const codec = stats.get(r.codecId) | ||
| if (codec?.mimeType) detectedMime = codec.mimeType.toLowerCase() | ||
| } | ||
| }) | ||
| if (detectedMime) console.debug('[Recorder] Detected codec via getStats →', detectedMime) | ||
| } | ||
| // SDP fallback – if getStats doesn't expose the coded yet, parse it out of the SDP | ||
|
|
||
| if (!detectedMime && peerConnection?.remoteDescription) { | ||
| const sdp = peerConnection.remoteDescription.sdp | ||
| const match = sdp.match(/^m=video .+$/m) | ||
| if (match) { | ||
| const pts = match[0].split(' ').slice(3) | ||
| for (const pt of pts) { | ||
| const rp = new RegExp(`a=rtpmap:${pt} ([A-Za-z0-9]+)/`).exec(sdp) | ||
| if (rp) { | ||
| detectedMime = `video/${rp[1].toLowerCase()}` | ||
| console.debug('[Recorder] Detected codec via SDP →', detectedMime) | ||
| break | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| if (detectedMime) { | ||
| if (MediaRecorder.isTypeSupported(detectedMime)) { | ||
| console.debug('[Recorder] Using detected codec.') | ||
| return { mimeType: detectedMime } | ||
| } | ||
|
|
||
| const codecOnly = detectedMime.replace(/^video\//, '') |
There was a problem hiding this comment.
There are a lot of cryptic strings here. Can we puts some comments here and there explaining exactly what is happening, so people reading this code in the future can better situate themselves?
| const namessAvailableAbstractedStreams = computed(() => { | ||
| return streamsCorrespondency.value.map((stream) => stream.name) | ||
| }) | ||
|
|
There was a problem hiding this comment.
No need to remove the line.
src/stores/video.ts
Outdated
| video.src = videoObjectUrl | ||
|
|
||
| video.addEventListener('error', () => { | ||
| URL.revokeObjectURL(videoObjectUrl) | ||
| reject('Error loading video') | ||
| }) | ||
|
|
||
| video.src = videoObjectUrl | ||
|
|
There was a problem hiding this comment.
Any reason for moving the code line to the top, since nothing happens to the object in the meanterm?
There was a problem hiding this comment.
It was a leftover from an earlier modification.
Returning to the original place..
src/stores/video.ts
Outdated
| const tryExtractFrame = (time: number): Promise<Blob> => { | ||
| return new Promise((resolveFrame, rejectFrame) => { | ||
| const onSeeked = (): void => { | ||
| try { | ||
| context.drawImage(video, 0, 0, width, height) | ||
| canvas.toBlob( | ||
| (blob: Blob | null) => { | ||
| video.removeEventListener('seeked', onSeeked) | ||
| if (blob) { | ||
| resolveFrame(blob) | ||
| } else { | ||
| rejectFrame('Failed to create blob') | ||
| } | ||
| URL.revokeObjectURL(videoObjectUrl) | ||
| }, | ||
| 'image/jpeg', | ||
| 0.6 | ||
| ) | ||
| } catch (err) { | ||
| video.removeEventListener('seeked', onSeeked) | ||
| URL.revokeObjectURL(videoObjectUrl) | ||
| rejectFrame(err) | ||
| } | ||
| } | ||
| resolve(blob) | ||
| URL.revokeObjectURL(videoObjectUrl) | ||
| } | ||
| canvas.toBlob(blobCallback, 'image/jpeg', 0.6) | ||
| video.addEventListener('seeked', onSeeked) |
There was a problem hiding this comment.
The canvas.toBlob call changed from on line to 12 lines. Can the code be written in a way that it is easier to read? This would be a better place to put a function instead.
There was a problem hiding this comment.
Agreed, fixed!
src/stores/video.ts
Outdated
| canvas.toBlob( | ||
| (fallbackBlob: Blob | null) => { | ||
| if (fallbackBlob) { | ||
| resolve(fallbackBlob) | ||
| } else { | ||
| resolve(new Blob()) | ||
| } | ||
| URL.revokeObjectURL(videoObjectUrl) | ||
| }, | ||
| 'image/jpeg', | ||
| 0.6 | ||
| ) |
There was a problem hiding this comment.
Same here.
Try to avoid those infinite indentation hell at all cost. It makes reading the code an homeric task.
There was a problem hiding this comment.
Agreed. The whole function could be improved.
A new approach for that was pushed on the last patch. I think it's way more clear now
| try { | ||
| ;(async () => { | ||
| const videoChunk = await tempVideoStorage.getItem(chunkName) | ||
| if (videoChunk) { | ||
| const firstChunkBlob = new Blob([videoChunk as Blob]) | ||
| const thumbnail = await extractThumbnailFromVideo(firstChunkBlob) | ||
| // Save thumbnail in the storage | ||
| await tempVideoStorage.setItem(videoThumbnailFilename(recordingHash), thumbnail) | ||
| unprocessedVideos.value = { ...unprocessedVideos.value, ...{ [recordingHash]: updatedInfo } } | ||
| } | ||
| } catch (error) { | ||
| console.error('Failed to extract thumbnail:', error) | ||
| } | ||
| if (!videoChunk) return | ||
| activeStreams.value[streamName]!.timeRecordingStart = new Date() | ||
| streamsPreparingToRecord.value = streamsPreparingToRecord.value.filter((name) => name !== streamName) | ||
| const firstChunkBlob = new Blob([videoChunk as Blob]) | ||
|
|
||
| extractThumbnailFromVideo(firstChunkBlob) | ||
| .then((thumbnail) => tempVideoStorage.setItem(videoThumbnailFilename(recordingHash), thumbnail)) | ||
| .catch((error) => { | ||
| console.warn('Failed to extract thumbnail:', error) | ||
| }) | ||
| })().catch((error) => { | ||
| console.error('Unexpected error extracting thumbnail:', error) | ||
| }) |
There was a problem hiding this comment.
Any reason for the await code to be replaced with then/catch?
There was a problem hiding this comment.
Both versions would work fine for the thumb extraction and the the change to then/catch was meant to isolate thumbnail‑extraction errors without turning the code into a sequence of additional try/ catch blocks.
71bcb17 to
d01662e
Compare
|
@ArturoManzoli could you add the explanation on the commits on what each one tries to solve, as we discussed previously? |
Yep, @rafaellehmkuhl after we spoke I did improve the description of the commits on their titles (1) and also added a long description on the PR's summary (2). (1) (2) 2- Since 4k recordings were demanding more time to write the first chunk on storage (sometimes up to 20 seconds). Stopping the recording before that first chunk, was causing the videos to fail processing; Also the recording timer was starting before the first chunk was written, making a difference between rec timer and real video duration, not noticeable when recording low res videos. (A separate issue #1823 was created to fix this preparation/timer-difference). 3- The absence of a mime type option on the MediaRecorder method was preventing the video from being processed normally. A method to extract this information from the Web RTC Manager was added to ensure the correct codec was selected. |
The descriptions (why it does what it does) that you quoted here are not on the commits. Only the titles are there:
|
…emetry processing The thumbnail extraction, when failing, was causing the whole video processing event to also fail. Quite often the thumbnails were being generated without the first chunk even being written on the memory. To fix that, the thumbnail extraction was put in parallel with the rest of the video processing logic. Signed-off-by: Arturo Manzoli <[email protected]>
…tate - Fix timer and video duration diff Since 4k recordings were demanding more time to write the first chunk on storage (sometimes up to 20 seconds). Stopping the recording before that first chunk, was causing the videos to fail processing; Also the recording timer was starting before the first chunk was written, making a difference between rec timer and real video duration, not noticeable when recording low res videos. (A separate issue bluerobotics#1823 was created to fix this preparation/timer-difference).
d01662e to
ffb092f
Compare
|
|
@ArturoManzoli I've tried to review this functionality yesterday, but my power supply is failing to supply enough current to the radcam. I'm getting a new power supply so I can properly test it. |
|
@ArturoManzoli my power supply arrived. I should be conducting the tests today. |
There was a problem hiding this comment.
Just tested it. The browser version is working fine, and is indeed solving the subtitles issue as well as the thumbnail one, but the Electron version seems to be downsampling the recorded video somehow.
This is a recording on the master branch:
Cockpit.May.21.2025.-.10.10.36.GMT-3.472c82c8.webm
This is a recording on this PR:
Cockpit.May.21.2025.-.09.59.18.GMT-3.682e4f40.webm
The stream is smooth, and my CPU and RAM usages are low.
Running on macOS 15.4.1.
The I'm currently working on solution based on a custom bitrate config. |
ffb092f to
7c4d31c
Compare
|
@rafaellehmkuhl The bandwidth ceiling was increased to 40 Mbits. The codec was the same on master and on this branch, the only difference was the bandwidth cap. |
Nice! Is there any problem on pushing to 60 Mbits? So we can also support 4K 60 FPS. There are some other people using high-fps 4K cameras, so if there's no harm, I would push for already supporting it. |
I think it's okay to do that. At least on high end machines running Cockpit-electron I'll crank up the values and we keep an eye on users feedback |
7c4d31c to
9cc8dac
Compare
|
@rafaellehmkuhl Done! Ready to review |
rafaellehmkuhl
left a comment
There was a problem hiding this comment.
With the latest changes, the lag apparently went away, but the bitrate is still low. Check the following:
Kapture.2025-05-27.at.11.49.37.mp4
Cockpit.May.27.2025.-.11.49.27.GMT-3.dbcad0d2.webm
9cc8dac to
7f6e87e
Compare
Signed-off-by: Arturo Manzoli <[email protected]>
7f6e87e to
1023375
Compare
|
On last patch:
Recorded on Electron: Cockpit.May.30.2025.-.15.41.11.GMT-3.c2f86202.webm |
|
Converting this PR to draft, as @joaoantoniocardoso is experimenting with important aspects of video streaming that are directly related to the issues it aims to resolve. |
|
@ArturoManzoli this one can probably be closed. |
|
Closed in favor of #2164. |



The video processing as a whole was failing on 4k RadCam streams, thus preventing the subtitles to be created.
This was due to three main reasons:
1- The thumbnail extraction, when failing, was causing the whole video processing event to also fail. Quite often the thumbnails were being generated without the first chunk even being written on the memory. To fix that, the thumbnail extraction was put in parallel with the rest of the video processing logic.
2- Since 4k recordings were demanding more time to write the first chunk on storage (sometimes up to 20 seconds). Stopping the recording before that first chunk, was causing the videos to fail processing; Also the recording timer was starting before the first chunk was written, making a difference between rec timer and real video duration, not noticeable when recording low res videos. (A separate issue #1823 was created to fix this preparation/timer-difference).
3- The absence of a mime type option on the MediaRecorder method was preventing the video from being processed normally. A method to extract this information from the Web RTC Manager was added to ensure the correct codec was selected.
Fix #1638
fix #1823