Skip to content

Gateway leaks undici sockets on every streamed Anthropic API call (buildManagedResponse missing finalize on GC) #67461

Description

@jakedwyer

Summary

The openclaw-gateway process accumulates leaked IPv6 ESTABLISHED sockets to api.anthropic.com (via Cloudflare). After 5 hours of normal cron-driven agent-turn executions, we observed 11,507 leaked ESTABLISHED IPv6 sockets, eventually causing spawn EBADF errors when the process hits its fd limit. All exec tool calls fail at that point, breaking email-scan and other crons that shell out.

Reproduction

  1. Start the gateway normally via launchd (openclaw gateway --port 18789)
  2. Run several agent-turn cron jobs that use streaming (e.g., email-scan, calendar-sync, meeting-sync — all using claude-sonnet-4-6 or claude-haiku-4-5)
  3. After a few hours, check: lsof -p $(pgrep openclaw-gateway) | wc -l
  4. Observe socket count growing linearly with each streamed API call, never shrinking
  5. Eventually spawn EBADF errors appear in agent-turn output when fd limit is hit

Root cause

In dist/anthropic-vertex-stream-YyrYtvts.js at the buildManagedResponse function (around line 4429):

function buildManagedResponse(response, release) {
  if (!response.body) { release(); return response; }
  const source = response.body;
  let reader;
  let released = false;
  const finalize = async () => {
    if (released) return;
    released = true;
    await release().catch(() => void 0);
  };
  const wrappedBody = new ReadableStream({
    start() { reader = source.getReader(); },
    async pull(controller) {
      try {
        const chunk = await reader?.read();
        if (!chunk || chunk.done) { controller.close(); await finalize(); return; }
        controller.enqueue(chunk.value);
      } catch (error) { controller.error(error); await finalize(); }
    },
    async cancel(reason) {
      try { await reader?.cancel(reason); }
      finally { await finalize(); }
    }
  });
  return new Response(wrappedBody, { ... });
}

finalize() (which calls release() to return the undici dispatcher slot / socket) is only called from:

  • pull() when chunk.done is true (natural stream end)
  • cancel() if the consumer explicitly cancels

There is no release path if the consumer abandons the stream without fully draining it and without calling cancel(). The Anthropic SDK's SSE parser sees message_stop and may return from its stream loop without fully draining to done: true or calling body.cancel(), leaving the wrapper in limbo. The start() callback has already locked the source reader, so even GC of the outer Response can't free the inner undici socket.

Evidence

After 5 hours of normal operation (53 cron jobs, ~6 streaming agent-turns/hour):

$ lsof -p $(pgrep openclaw-gateway) | wc -l
12097

$ lsof -p $(pgrep openclaw-gateway) | awk '{print $5}' | sort | uniq -c | sort -rn | head -5
11508 IPv6
  419 unix
   91 IPv4
   38 REG
   28 KQUEUE

$ lsof -p $(pgrep openclaw-gateway) | grep IPv6 | grep ESTABLISHED | awk '{for(i=9;i<=NF;i++) if($i ~ /->/) print $i}' | sed 's/.*->//;s/:[0-9]*$//' | sort | uniq -c | sort -rn
4292 [2606:4700:20::681a:d73]:https    # Cloudflare (api.anthropic.com)
3619 [2606:4700:20::681a:c73]:https    # Cloudflare
3461 [2606:4700:20::ac43:466c]:https   # Cloudflare
 127 [2607:6bc0::10]:https             # Anthropic direct

After gateway restart: lsof count dropped from 12,097 to 69.

Suggested fix

Add a safety net so that release() is called even if the stream consumer abandons the wrapper:

Option A (FinalizationRegistry):

const registry = new FinalizationRegistry(async (release) => {
  await release().catch(() => void 0);
});
// In buildManagedResponse, after creating wrappedBody:
registry.register(wrappedBody, release);

Option B (idle timeout):

// In start(), set a watchdog timer that calls finalize() if pull() isn't called within 60s
let lastPull = Date.now();
const watchdog = setInterval(() => {
  if (Date.now() - lastPull > 60_000 && !released) { finalize(); clearInterval(watchdog); }
}, 30_000);
// In pull(), update lastPull = Date.now(); in finalize(), clearInterval(watchdog);

Option C (fix the consumer):
Ensure the Anthropic SDK consumer always calls body.cancel() after stream processing completes, even on the happy path. This is the most targeted fix but depends on the SDK's behavior.

Environment

  • openclaw v2026.4.14
  • macOS Darwin 25.2.0, arm64
  • Node (from /opt/homebrew/opt/node/bin/node)
  • undici 8.0.2 (from openclaw's package.json)
  • @anthropic-ai/vertex-sdk ^0.15.0

Workaround

Periodic gateway restart via launchctl kickstart -k gui/$UID/ai.openclaw.gateway. We've implemented an hourly fd watchdog that auto-restarts when fd count exceeds 2000.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions