Skip to content

Conversation

@DenchiSoft
Copy link
Contributor

Potential fix for #1238

I've only implemented this for the FaceLandmarkerRunner for now because I wanted to get your thoughts on it first.


I've noticed that on my Samsung S21 when using ImageReadMode.GPU, the video would "glitch" in a weird way. The detected landmarks would look like they jumped back in time a few frames randomly from time to time, which is also what's visible in the slo-mo video from #1238 . It gets especially bad when I turn on Multithreaded Rendering in the Android build settings.

After some digging, I believe this is because the GPU operations that copy the frame data to the TextureFrame texture aren't guaranteed to finish before sending it off to MediaPipe via taskApi.DetectAsync(...). This causes the TextureFrame from the pool to potentially still have its previous texture content, which would be an old frame.

One easy fix for that would be to move the yield return waitForEndOfFrame; after the GPU copy operations. That way, the copy operations are finished before the frame is sent off. This fixes the issue entirely on my device.

However, I'm not actually sure this is a proper fix as I don't think Unity technically guarantees that graphics operations will be finished within one frame. I was trying to use a GraphicsFence, but apparently that's not supported on Android.

Let me know what you think.

<--- before / after --->

comparison.mp4

For ImageReadMode.GPU, this makes sure the new texture contents are fully written before passing it on to MediaPipe.
@DenchiSoft DenchiSoft changed the title Moved waitForEndOfFrame yield after GPU operations. Resolve landmark jitter by adjusting frame copy timing Jan 18, 2025
@homuler homuler added the ci:run Run CI label Jan 18, 2025
@github-actions github-actions bot removed the ci:run Run CI label Jan 18, 2025
@homuler
Copy link
Owner

homuler commented Jan 18, 2025

Thank you!
I hadn't noticed the issue, so this really helps. I also understand the cause of the problem in #1238 now.

However, I'm not actually sure this is a proper fix as I don't think Unity technically guarantees that graphics operations will be finished within one frame.

I agree. It seems like a good idea to leave a FIXME comment.
To fix it properly, we might need to use GlSyncToken to make MediaPipe wait for the copy operation to complete (I am not confident that it will work, though).
ref. https://github.com/google-ai-edge/mediapipe/blob/cf6e4ec469e04e126cc36c1aebc51b48ffdd80c7/mediapipe/gpu/gl_texture_buffer.h#L129-L132

@DenchiSoft
Copy link
Contributor Author

DenchiSoft commented Jan 18, 2025

Done, let me know if this works for you.

Completely unrelated, but I've noticed another thing. Currently, inference runs every frame even if the webcam didn't supply a new frame. If the app FPS is higher than the webcam FPS, this causes the same webcam frame to be sent to MediaPipe multiple times.

You could check .updateCount on the texture you get from imageSource.GetCurrentTexture() and if the count wasn't increased, you just yield return null; continue; and check again in the next app frame.

I have a webcam running at 30 FPS and the test app running at 60 FPS (testing on Windows). Implementing this cuts the CPU usage roughly in half, as you would expect:

skip_frame_comparison
Of course it only improves performance if the webcam (or other video source) FPS is lower than the app FPS. Otherwise it makes no difference.

If you'd like, I can open another PR for this later, implemented like this: 2f35f73

@homuler homuler added the ci:run Run CI label Jan 19, 2025
@github-actions github-actions bot removed the ci:run Run CI label Jan 19, 2025
@homuler
Copy link
Owner

homuler commented Jan 19, 2025

Thank you for your suggestion!
You're absolutely right that this check would be valuable in a real-world application. However, since this is just a sample app, I believe we should focus on including only the code necessary to run MediaPipe. Therefore, I think it's fine to leave it as is.

Copy link
Owner

@homuler homuler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@homuler homuler merged commit 0e04cb8 into homuler:master Jan 19, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants