[bug] skill_manage: parallel tool calls for the same (user, skill) deadlock forever on the sync client path (shared asyncio.Lock across per-call event loops) #5846

Problem summary

When a model issues two skill_manage tool calls for the same (user, skill) in one turn and the graph runs on the sync client path (DeerFlowClient.stream(), used by the TUI), the second call contends on the shared per-(user, skill) asyncio.Lock from a different event loop and never returns — the whole ToolNode run hangs permanently.

Affected area(s)

• Skills
• Agents / LangGraph (graph, prompts, langgraph.json)

What happened?

_skill_manage_impl guards every action with a shared per-(user_id, name) asyncio.Lock obtained from _get_lock (backend/packages/harness/deerflow/tools/skill_manage_tool.py:138), acquired at skill_manage_tool.py:142 (async with lock:). _skill_locks (skill_manage_tool.py:34) is a WeakValueDictionary keyed by (user_id, name), so both calls in the same turn get the same lock object.

On the sync client path, each tool call runs on its own event loop:

• skill_manage_tool.func is make_sync_tool_wrapper(_skill_manage_impl, "skill_manage") (skill_manage_tool.py:296).
• make_sync_tool_wrapper's run_coroutine executes asyncio.run(coro(*args, **kwargs)) per call (backend/packages/harness/deerflow/tools/sync.py:89) — a fresh event loop per call, on the calling thread.
• DeerFlowClient.stream() is "a parallel, sync, in-process consumer" (backend/packages/harness/deerflow/client.py:881), driven by the TUI (backend/packages/harness/deerflow/tui/runtime.py:107).
• The sync ToolNode._func maps the turn's tool calls over a ContextThreadPoolExecutor in parallel (installed langgraph/prebuilt/tool_node.py:823, executor.map(self._run_one, tool_calls, ...)), so two skill_manage calls for one skill run concurrently on separate threads, each inside its own asyncio.run().

An asyncio.Lock binds to whichever event loop first contends on it. Call A binds the lock to its loop, completes, and its asyncio.run() closes that loop. Call B, contending from its own loop, waits on a wakeup that is never delivered by any running loop — so call B hangs forever. Because executor.map waits for every call in the turn, the entire ToolNode run hangs permanently. skill_manage is registered when skill_evolution.enabled is true (backend/packages/harness/deerflow/tools/tools.py:144-148), and a model can naturally emit two skill_manage calls for one skill (e.g. patch + write_file) in a single turn.

Expected behavior

The second skill_manage call for the same (user, skill) should block only until the first finishes, then proceed (or raise its own validation error) — on any execution path, sync or async. The mutual exclusion must be safe under the sync client path's per-call event loops.

Basis in the repo: sibling module backend/packages/harness/deerflow/mcp/oauth.py:33-43 documents exactly this topology and uses a plain threading.Lock for it:

A plain threading.Lock, not asyncio.Lock: the embedded/TUI sync tool-call path (DeerFlowClient.stream() -> LangGraph ToolNode._func -> a ThreadPoolExecutor -> deerflow.tools.sync.make_sync_tool_wrapper's per-call asyncio.run()) invokes get_authorization_header from a fresh event loop on a fresh OS thread for every concurrent tool call. An asyncio.Lock binds to whichever loop first contends on it; a second caller's release/wake-up crossing loops without call_soon_threadsafe either deadlocks silently or raises "bound to a different event loop". threading.Lock has no loop affinity, so it is safe to share across however many event loops/threads call into the same server's lock.

PR #4240 (merged) made that exact fix for OAuth token refresh: "fix(mcp): use threading.Lock for OAuth token refresh to avoid cross-thread deadlock". skill_manage_tool.py's _get_lock predates that lesson and still holds an asyncio.Lock across the same call topology.

Steps to reproduce

No network or real storage needed. Only the storage layer is stubbed (with a 0.5s sleep to widen the contention window); the lock, _skill_manage_impl, and the production sync wrapper skill_manage_tool.func are the real code. action="bogus" still acquires the lock first — async with lock: at line 142 precedes the action dispatch — so any two same-(user, skill) calls contend.

"""Two parallel skill_manage tool calls in ONE turn (sync graph path)
contend on the shared per-(user, skill) asyncio.Lock from two event loops."""
import sys, threading, time

from deerflow.tools import skill_manage_tool as smt

class _StubStorage:
    def public_skill_exists(self, name):
        time.sleep(0.5)          # stand-in for the real disk lookup
        return False

smt.get_or_new_user_skill_storage = lambda user_id: _StubStorage()

class _RT:  # minimal Runtime stand-in: user scoping via runtime.context
    context = {"user_id": "repro-user", "thread_id": "repro-thread"}
    config = {}
    tool_call_id = "call_repro"

results = {}

def call(tag):
    try:
        results[tag] = smt.skill_manage_tool.func(
            runtime=_RT(), action="bogus", name="dup-skill"
        )
    except BaseException as e:
        results[tag] = f"{type(e).__name__}: {e}"

t0 = time.time()
ta = threading.Thread(target=call, args=("A",)); ta.start()
time.sleep(0.1)                                   # A now holds the lock
tb = threading.Thread(target=call, args=("B",), daemon=True); tb.start()
ta.join(10); tb.join(5)
print(f"elapsed: {time.time()-t0:.2f}s")
print("call A ->", results.get("A"))
print("call B ->", results.get("B") if "B" in results else f"NEVER RETURNED (thread alive={tb.is_alive()})")
sys.exit(0 if tb.is_alive() else 1)

Run against the backend environment at HEAD aa4e43a:

cd backend && uv run python /tmp/deerflow-repro/repro_skill_manage.py

Relevant logs

Verbatim output at HEAD aa4e43a2bcc8ffc3838b012c4583f561a0e199be, backend uv env, Python 3.12.14. The traceback is call A completing normally — it acquired the lock, hit the action dispatch, released the lock cleanly, and returned through sync.py:89. Call B is the second contender:

Error invoking tool 'skill_manage' via sync wrapper: Unsupported action 'bogus'.
Traceback (most recent call last):
  File ".../deerflow/tools/sync.py", line 89, in run_coroutine
    return asyncio.run(coro(*args, **kwargs))
  File ".../python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
  File ".../python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File ".../python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
  File ".../deerflow/tools/skill_manage_tool.py", line 259, in _skill_manage_impl
    raise ValueError(f"Unsupported action '{action}'.")
ValueError: Unsupported action 'bogus'.
elapsed: 5.52s
call A -> ValueError: Unsupported action 'bogus'.
call B -> NEVER RETURNED (thread alive=True)

Call B never returned after 5.52s with its thread still alive; the process only exits because B is a daemon thread. Two verifiers reproduced the same hang independently before this report, and I reproduced it again at the same HEAD (elapsed: 5.52s, call B -> NEVER RETURNED (thread alive=True)).

Root cause

backend/packages/harness/deerflow/tools/skill_manage_tool.py:142 — async with lock: holds a per-(user_id, name) asyncio.Lock (created in _get_lock, skill_manage_tool.py:37-42) across a code path that the sync tool wrapper invokes under a fresh asyncio.run() event loop per call (sync.py:89). The lock binds to the first call's loop; the second call's wakeup is scheduled on that (now closed) loop and is never delivered, so the call hangs forever.

Environment

• DeerFlow: main @ aa4e43a2bcc8ffc3838b012c4583f561a0e199be
• How are you running DeerFlow?: Other (backend harness; direct tool-call repro of the sync client path — DeerFlowClient.stream() / TUI is the affected surface, no gateway or sandbox involved)
• Operating system: macOS 26.6.2 (arm64, zsh)
• Python version: Python 3.12.14 (backend uv environment)
• Node.js / pnpm: not involved (backend-only repro)
• uv version: 0.12.8
• Git state: branch: main, commit: aa4e43a2bcc8ffc3838b012c4583f561a0e199be
• Support bundle: not applicable — a pure library-level concurrency bug, reproducible standalone with no config, sandbox, or gateway state involved

Proposed approach

Mirror oauth.py's approach: make the per-(user_id, name) mutual exclusion loop-agnostic — either a plain threading.Lock held around each storage-mutating section run via asyncio.to_thread, or key the existing lock registry by the running event loop. Happy to open a PR with that change if it's useful — say the word and I'll send it.

Related issues / PRs

• fix(mcp): use threading.Lock for OAuth token refresh to avoid cross-thread deadlock #4240 (merged) — the same fix for the same topology in mcp/oauth.py: threading.Lock instead of asyncio.Lock "to avoid cross-thread deadlock".
• Fix: 7 LOW findings from 2026-07 bug bash (fd, rename and _to_thread findings dropped per review) #4109 (open) — touches skill_manage_tool.py but only refactors _to_thread → asyncio.to_thread; does not touch _get_lock / _skill_locks.
• feat(skills): add guarded evolution host APIs #5645 (open) — touches skill_manage_tool.py (managed-read guards, read_custom_skill refactor) but keeps async with lock: / _get_lock as-is.
• No existing issue or PR found for this deadlock (searched skill_manage lock, skill_manage deadlock, _get_lock).

AI assistance disclosure

This issue was drafted with Claude Code assistance; the hang was independently reproduced by two verifiers and once more by the filer at the same commit, and a human has read, understands, and takes responsibility for this report.