-
-
Notifications
You must be signed in to change notification settings - Fork 580
Resolve landmark jitter by adjusting frame copy timing #1311
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
For ImageReadMode.GPU, this makes sure the new texture contents are fully written before passing it on to MediaPipe.
|
Thank you!
I agree. It seems like a good idea to leave a FIXME comment. |
Assets/MediaPipeUnity/Samples/Scenes/Face Landmark Detection/FaceLandmarkerRunner.cs
Outdated
Show resolved
Hide resolved
Assets/MediaPipeUnity/Samples/Scenes/Face Landmark Detection/FaceLandmarkerRunner.cs
Outdated
Show resolved
Hide resolved
|
Done, let me know if this works for you. Completely unrelated, but I've noticed another thing. Currently, inference runs every frame even if the webcam didn't supply a new frame. If the app FPS is higher than the webcam FPS, this causes the same webcam frame to be sent to MediaPipe multiple times. You could check I have a webcam running at 30 FPS and the test app running at 60 FPS (testing on Windows). Implementing this cuts the CPU usage roughly in half, as you would expect:
If you'd like, I can open another PR for this later, implemented like this: 2f35f73 |
|
Thank you for your suggestion! |
homuler
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM

Potential fix for #1238
I've only implemented this for the
FaceLandmarkerRunnerfor now because I wanted to get your thoughts on it first.I've noticed that on my Samsung S21 when using
ImageReadMode.GPU, the video would "glitch" in a weird way. The detected landmarks would look like they jumped back in time a few frames randomly from time to time, which is also what's visible in the slo-mo video from #1238 . It gets especially bad when I turn onMultithreaded Renderingin the Android build settings.After some digging, I believe this is because the GPU operations that copy the frame data to the
TextureFrametexture aren't guaranteed to finish before sending it off to MediaPipe viataskApi.DetectAsync(...). This causes theTextureFramefrom the pool to potentially still have its previous texture content, which would be an old frame.One easy fix for that would be to move the
yield return waitForEndOfFrame;after the GPU copy operations. That way, the copy operations are finished before the frame is sent off. This fixes the issue entirely on my device.However, I'm not actually sure this is a proper fix as I don't think Unity technically guarantees that graphics operations will be finished within one frame. I was trying to use a
GraphicsFence, but apparently that's not supported on Android.Let me know what you think.
<--- before / after --->
comparison.mp4