Skip to content

Persistent PTY: every input lists all terminals, and the daemon sweeps /proc per terminal #60728

Open

Description

@isaacturnbull

Affects: opencode serve PersistentPty + opencode-pty daemon 0.3.1, protocol 7.

Summary

With ~25 persistent PTYs on one host, each keystroke takes ~160 ms inside opencode serve before the
daemon writes it, and the idle opencode-pty daemon uses ~28 % of a core, and every request waits up to 10 ms to be accepted. Three causes:

  1. serve calls list() before every op. PersistentPty.input (and write, resize, control, snapshot)
    lists every terminal to resolve one id. The daemon fills each terminal's foreground_process by
    sweeping /proc, ~7 ms a terminal, serially. A key costs O(terminals) of /proc sweeps.

  2. The daemon sweeps /proc per terminal on its own. Even with no requests, each terminal's thread
    walks all of /proc about every 10 s (28 sweeps in 10 s on a 28-PTY host, ~10 k reads a second).

  3. The daemon polls accept() every 10 ms. Every one-shot request (serve's included) waits 0–10 ms
    before the daemon reads it: on the live daemon, accept4 runs ~100 times a second alongside
    clock_nanosleep (~90/s) and sched_yield (~700/s) while idle. Requests phase-lock to a 10 ms grid
    (ping and input sent 37 ms apart all take 3.3 ms; spread out, p50 0.75 / p90 9.9 ms). A
    connection opened ≥10 ms before its request is written answers in 0.2 ms.

Measurements (Linux 7.0 Azure VM, 8 vCPU, 24–28 PTYs, eBPF + strace)

value
key first byte, client ws → serve → daemon 167 ms p50
key first byte, client → daemon socket directly 7 ms p50
daemon input op itself 0.1 ms
idle daemon CPU (60 s, no typing) 28.5 % (22.7 % sys)
serve list requests reaching the daemon, idle 1 every 5 s
ping/input on a fresh connection (1 PTY) p50 0.75 ms, p90 9.9 ms (10 ms accept poll)
same request on a connection opened 50 ms earlier 0.2 ms
key → echo, opencode TUI in a PTY, bare (sandbox, 364x91) p50 9–11 ms; the daemon's accept poll adds p50 5.6 / p90 9.4 ms

Adding 16 idle PTYs moved typing latency 136 → 18 ms (shell) and 141 → 30 ms (opencode TUI) once input
bypassed serve, so the cost scales with the terminal count, not the typing.

Suggested fix

  • serve: resolve a terminal by id without list() on input/write/resize/control/snapshot (keep a map
    from the attach, or call a per-id op on the daemon).
  • daemon: block in accept() (or poll the listener with the connections) instead of a 10 ms sleep loop,
    and serve several requests per connection so a client can keep one open.
  • daemon: find the foreground process with tcgetpgrp(master_fd) and read /proc/<pgid>/{stat,cmdline},
    instead of sweeping all of /proc; compute it lazily (only when list asks), not on a per-terminal timer.

Workaround in use

olympus sends keystrokes straight to the daemon's unix socket (protocol 7 input op), falling back to
serve's websocket. It depends on a private protocol and will be retired once this is fixed. It also keeps one spare
connection open ahead of the next key to dodge the accept poll.

Activity

  1. opencodejr commented on Oct 30, 2026

    @opencodejr
    Contributor

    Thanks for the detailed measurements. I could reproduce the main part of this.

    The script below creates persistent terminals through opencode serve and asks for the snapshot of one terminal. It counts the daemon's /proc opens for that single request:

    Terminals /proc opens (2.2.4, daemon 0.4.0) Median latency
    1 828 4 ms
    5 2,532 8 ms
    25 12,012 34 ms
    50 26,112 66 ms

    This still happens on the latest dev (171dbd6). PersistentPty.get(id) still sends the daemon's list op for each per-terminal operation, and list still sweeps all of /proc once per terminal to find its foreground process.

    The other two parts did not show up on the current daemon (0.4.0). With 50+ terminals and no requests, the idle daemon made no openat, accept4 or clock_nanosleep calls in 10 s. The polling was removed in #59701 and #59698, after the 0.3.1 release you measured. Upgrading should take care of the idle CPU and the 0–10 ms accept delay; the cost of each op still grows with the number of terminals.

    Reproduction script (repro.sh)
    #!/usr/bin/env bash
    # Reproduces #60728: every per-terminal op in `opencode serve` lists ALL terminals,
    # and the daemon's `list` walks /proc once per terminal, so op latency grows with
    # terminal count.
    # Usage: start `opencode serve --port 4199 --hostname 127.0.0.1` in a project dir,
    # then: PASS=<server password> ./repro.sh   (needs curl, strace, jq-less)
    set -euo pipefail
    URL=${URL:-http://127.0.0.1:4199}
    AUTH="opencode:${PASS:?set PASS to the server password}"
    SES=$(curl -s -u "$AUTH" -X POST -H 'content-type: application/json' -d '{}' "$URL/api/session" | grep -o '"id":"ses_[^"]*"' | head -1 | cut -d'"' -f4)
    create() {
      curl -s -u "$AUTH" -X POST -H 'content-type: application/json' -d '{"command":"/bin/sh","args":[],"title":"t","env":{},"cwd":"/tmp"}' \
        "$URL/api/experimental/session/$SES/terminal" | grep -o '"id":"pty_[^"]*"' | head -1 | cut -d'"' -f4
    }
    FIRST=$(create)
    DAEMON=$(pgrep -n -x opencode-pty)
    echo "session=$SES first=$FIRST daemon_pid=$DAEMON"
    measure() {
      local n=$1
      # one snapshot (per-terminal op) under strace: count /proc opens in the daemon
      timeout 5 strace -f -qq -e trace=openat -p "$DAEMON" -o /tmp/opencode/st.$n 2>/dev/null &
      local sp=$!; sleep 0.5
      local t=$(curl -s -o /dev/null -u "$AUTH" -w '%{time_total}' "$URL/api/experimental/persistent-pty/$FIRST/snapshot")
      sleep 0.5; kill $sp 2>/dev/null || true; wait $sp 2>/dev/null || true
      local procs=$(grep -c '"/proc/' /tmp/opencode/st.$n || true)
      # latency without strace, median of 10
      local lat=$(for i in $(seq 10); do curl -s -o /dev/null -u "$AUTH" -w '%{time_total}\n' "$URL/api/experimental/persistent-pty/$FIRST/snapshot"; done | sort -n | sed -n 5p)
      echo "terminals=$n  snapshot_of_ONE_terminal: daemon /proc opens=$procs  median latency=${lat}s"
    }
    measure 1
    for target in 5 25 50; do
      while [ "$(curl -s -u "$AUTH" "$URL/api/experimental/session/$SES/terminal" | grep -o '"id":"pty_' | wc -l)" -lt $target ]; do create >/dev/null; done
      measure $target
    done

    Run it: bash repro.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions