Skip to content

GestureRecognizer with more than one hand sometimes fails with "Packet isn't the sole owner of the holder" (ConcatenateTensorVectorCalculator) #6396

Open

Description

@opex-sys

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes. A Python application using the stock MediaPipe Tasks GestureRecognizer API (no graph or calculator changes), plus a small standalone repro script.

OS Platform and Distribution

Ubuntu 22.04 (aarch64), NVIDIA Jetson Orin

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

No response

Programming Language and version

Python 3.10

MediaPipe version

1.0.1 (pip). The relevant code looks the same in v1.2.2 and current master.

Bazel version

No response

Solution

Gesture Recognizer (MediaPipe Tasks), RunningMode.VIDEO, num_hands >= 2

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

When more than one hand is in view, recognize_for_video() occasionally fails after a few minutes with: RuntimeError: CalculatorGraph::Run() failed: Calculator::Process() for node "...singlehandgesturerecognizergraph__ConcatenateTensorVectorCalculator" failed: Packet isn't the sole owner of the holder. After that the graph stays broken and close() raises the same error. The only way to continue is to create a new recognizer. It never happens with a single hand.

Describe the expected behaviour

recognize_for_video() should keep working with any number of hands. A packet that is only copied for logging should not make ConcatenateTensorVectorCalculator fail.

Standalone code/steps you may have used to try to get what you need

Run the gesture recognizer repeatedly on an image that contains several hands:

import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

options = vision.GestureRecognizerOptions(
    base_options=python.BaseOptions(model_asset_path="gesture_recognizer.task"),
    running_mode=vision.RunningMode.VIDEO,
    num_hands=4,
)
image = mp.Image.create_from_file("multi_hands.jpg")  # picture with 2+ hands

with vision.GestureRecognizer.create_from_options(options) as recognizer:
    for i in range(200000):
        recognizer.recognize_for_video(image, i * 33)

Result: fails one or more times when more than one hand is detected, never with a single hand. It happens more often under CPU load, because the failure is a thread-timing race.
Higher the number of hands detecting with high CPU usage results in this error(detecting 9 hands in our test case)).

Other info / Complete Logs

Full error:

RuntimeError: CalculatorGraph::Run() failed:
Calculator::Process() for node "...singlehandgesturerecognizergraph__ConcatenateTensorVectorCalculator"
failed: Packet isn't the sole owner of the holder.
=== Source Location Trace: ===
third_party/mediapipe/framework/packet.h:749
third_party/mediapipe/calculators/core/concatenate_vector_calculator.h:103
third_party/mediapipe/framework/calculator_node.cc:1012


### What we found
We turned on VLOG 4 (which logs packet copies, consumes and queue adds) and caught the failure. Names are shortened: LANE__ is the single-hand gesture graph. Hands 1395–1399 are the hands of one frame, and the second column is the thread id.


.578741 1311 Calling Process() LANE__TensorConverterCalculator_3 timestamp: 1399
.578771 1311 Using copy constructor of Packet timestamp: 1399 vector<Tensor>
.578775 1311 Using copy constructor of Packet timestamp: 1395 vector<Tensor>   <- head of the queue copied
        ---- nothing from thread 1311 for 3.8 ms ----
.582244 1309 Calling Process() LANE__ConcatenateTensorVectorCalculator timestamp: 1395
.582254 1309 Consuming the data of Packet timestamp: 1395 vector<Tensor>       <- In(0) ok
.582270 1309 Consuming the data of Packet timestamp: 1395 vector<Tensor>       <- In(1) ok
.582315 1309 RecordError ... Packet isn't the sole owner of the holder.         <- In(2) fails
.582570 1311 Input stream:LANE____stream_10 has added packet at time: 1399     <- copy released


1. Thread 1311 copied hand 1395's packet at the front of the In(2) queue.
2. Thread 1311 then paused for about 3.8 ms.
3. During the pause, thread 1309 ran Concatenate for hand 1395 and failed on exactly that packet.
4. Thread 1311 released its copy only after the failure.

### Why we think this happens
ConcatenateTensorVectorCalculator must use Packet::Consume(), because Tensor can't be copied. Consume() requires use_count() == 1.

Every time a packet is added to a queue, InputStreamHandler::AddPackets() / MovePackets() call LogQueuedPackets():


Packet queue_head = stream->QueueHead();   // returns a copy of the head packet
if (!queue_head.IsEmpty()) {
  mediapipe::LogEvent(context->GetProfilingContext(),
                      event.set_packet_ts(queue_head.Timestamp()));
}


There are three problems with this:
- Only the timestamp is needed, but the whole packet is copied.
- The copy happens even when tracing is off, because the check is inside LogEvent.
- It runs on the adding thread while another thread may be consuming the head packet.

With several hands, one hand's piece can wait at the head of a Concatenate input queue while the next hand's piece is added behind it. With one hand, the queue is empty, so nothing gets copied.

### Question
- Is this analysis correct, or is something else going on?
- Is there a recommended fix or workaround for multi-hand use (a graph or executor setting, or a newer version where this is handled)?
- If a code change is needed, what approach would you prefer?

We can reproduce this reliably and are happy to test any proposed change.

### Possible fix (suggestion)
InputStreamManager already has MinTimestampOrBound(), which reads the head timestamp under the same lock without copying the packet. input_stream_handler.cc already uses it this way elsewhere:


-    Packet queue_head = stream->QueueHead();
-    if (!queue_head.IsEmpty()) {
-      mediapipe::LogEvent(context->GetProfilingContext(),
-                          event.set_packet_ts(queue_head.Timestamp()));
-    }
+    bool queue_empty = true;
+    const Timestamp queue_head_ts = stream->MinTimestampOrBound(&queue_empty);
+    if (!queue_empty) {
+      mediapipe::LogEvent(context->GetProfilingContext(),
+                          event.set_packet_ts(queue_head_ts));
+    }


The traced information stays the same, and the extra owner goes away. We're glad to open a PR if this direction looks right.

Code references (v1.2.2):
- LogQueuedPackets(): https://github.com/google-ai-edge/mediapipe/blob/v1.2.2/mediapipe/framework/input_stream_handler.cc#L300-L316
- QueueHead(): https://github.com/google-ai-edge/mediapipe/blob/v1.2.2/mediapipe/framework/input_stream_manager.cc#L71-L77
- MinTimestampOrBound(): https://github.com/google-ai-edge/mediapipe/blob/v1.2.2/mediapipe/framework/input_stream_manager.cc#L250-L261
- Packet::Consume(): https://github.com/google-ai-edge/mediapipe/blob/v1.2.2/mediapipe/framework/packet.h#L691-L710

Activity

  1. added
    type:bugBug in the Source Code of MediaPipe Solution
    on Nov 5, 2026
  2. akashvverma1995 commented on Nov 6, 2026

    @akashvverma1995

    Hi @opex-sys,

    Thank you for the detailed report, VLOG trace, and standalone reproduction script!

    Your analysis of the race condition—where stream->QueueHead() in InputStreamHandler::LogQueuedPackets() temporarily holds an extra packet reference while ConcatenateTensorVectorCalculator calls Packet::Consume()—makes complete sense.

    We are looking into this and reproducing it from our end, and we will update this thread once we have more details.

  3. added
    task:gesture recognitionIssues related to hand gesture recognition: Identify and recognize hand gestures
    os:linux-non-armIssues on linux distributions which run on x86-64 architecture. DOES NOT include ARM devices.
    on Nov 6, 2026
  4. opex-sys commented on Nov 6, 2026

    @opex-sys
    Author

    Thanks @akashvverma1995

    Note: For reproducing the race needs many hands (5+ is much more error prone) plus high CPU load). We hit an error most often with ~9 hands detected. Below is the script to run that. It needs 'gesture_recognizer.task' and an image with at least 5+ hands.

    import os, multiprocessing as mproc
    import mediapipe as mp
    from mediapipe.tasks import python
    from mediapipe.tasks.python import vision
    
    def burn():
        while True:
            pass
    
    def make():
        opts = vision.GestureRecognizerOptions(
            base_options=python.BaseOptions(model_asset_path="gesture_recognizer.task"),
            running_mode=vision.RunningMode.VIDEO,
            num_hands=9,
        )
        return vision.GestureRecognizer.create_from_options(opts)
    
    if __name__ == "__main__":
        # keep all cores busy to make thread pauses more likely
        for _ in range(os.cpu_count()):
            mproc.Process(target=burn, daemon=True).start()
    
        image = mp.Image.create_from_file("multi_hands.jpg")  # image with 5+ hands
        rec, ts, errors = make(), 0, 0
        # change the range as required
        for i in range(200000):
            ts += 33
            try:
                res = rec.recognize_for_video(image, ts)
                if i % 500 == 0:
                    print(f"iter {i}: hands={len(res.handedness)} errors={errors}")
            except RuntimeError as e:
                errors += 1
                print(f"iter {i}: ERROR #{errors}: {e}")
                rec, ts = make(), 0   # graph is broken after the error, recreate
        print("total errors:", errors)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

os:linux-non-armIssues on linux distributions which run on x86-64 architecture. DOES NOT include ARM devices.platform:pythonMediaPipe Python issuestask:gesture recognitionIssues related to hand gesture recognition: Identify and recognize hand gesturestype:bugBug in the Source Code of MediaPipe Solution

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions