server (NixOS): watcher subscribes /nix/store as parent-directory watch for home-manager skill derivations, consuming ~523k inotify watches and causing system-wide ENOSPC #55367

Activity

1. opencode-agent commented on Oct 14, 2026

   

   opencode-agentBot

   on Oct 14, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the detailed report. I tried to reproduce this on v2.0.27 (opencode serve on Linux) and couldn't get the watch count past about 16. Here's what I tried:

   • A git project with .direnv/flake-inputs/<name> symlinked to a separate tree of ~5,000 directories, then requests to the server for that project (location, config, skills, file search, VCS status, sessions). The project directory only gets one-level, non-recursive watches, plus a single-file watch on .git/HEAD. Nothing went into the symlinked tree.
   • ~/.config/opencode set up home-manager style, with opencode.json and two skills/ entries as symlinks to other directories. One of the skills pointed to a directory with a SKILL.md plus a symlink to the big tree. The recursive watches on ~/.config/opencode, skills/ and the resolved skill directory did not follow symlinks into the big tree.

   So on 2.0.27 the watcher doesn't seem to follow symlinks in general. My guess is that on your machine one of the recursive watch roots resolves to a large store directory itself, for example a skill or plugin path pointing at a whole flake input source, or the home-manager-files tree. I couldn't confirm that, though. These would help:

   1. The watcher subscribe / watcher started lines from ~/.local/share/opencode/log/opencode.log for one service start, especially those with type=directory and backend=inotify, including the full paths (redact anything private).
   2. For each type=directory path, roughly how many directories it contains (find -L <path> -type d | wc -l and find <path> -type d | wc -l). This shows whether the size comes from following symlinks or from the directory itself.
   3. The rendered opencode.json (plugin entries and any skill or instruction paths), and what ls -la ~/.config/opencode ~/.config/opencode/skills shows.

   You can reply here or edit the issue. If you edit it, please also leave a comment so we know it's updated.

   Related issues:

   • watcher: recursive watcher follows Windows directory junctions and hangs sessions in workspaces with junction-mirrored trees #50463: the recursive watcher follows Windows directory junctions (similar symptom on another platform)
   • server: file watcher restarts in a loop on unmodified skill directories, dropping SSE and respawning the CLI #53416 / watcher: fs watcher subscribes/unsubscribes skill directories in a tight loop (100%+ CPU, macOS) #53811 / cli: service skill/compatibility watcher self-loops on .agents/skills, ~37% idle CPU #54114: watcher resubscribe loops on skill directories
   • tui: crashes with ENOSPC when inotify watch limit is exhausted #51208 / TUI crash: ENOSPC: no space left on device, watch '/home/davicompai/.config/opencode' #52776 / ENOSPC: no space left on device #55186: TUI crashes when inotify watches run out
2. opencode-agent

   added

   2.0

   on Oct 14, 2026
3. britter

   changed the title [-]server (NixOS): file watcher recursively watches /nix/store via home-manager and direnv symlinks, consuming ~523k inotify watches at startup and causing system-wide ENOSPC[/-] [+]server (NixOS): watcher subscribes /nix/store as parent-directory watch for home-manager skill derivations, consuming ~523k inotify watches and causing system-wide ENOSPC[/+] on Oct 14, 2026
4. britter commented on Oct 14, 2026

   

   britter

   on Oct 14, 2026

   Author

   More actions

   Correction and follow-up data as requested (@opencode-agent). I've also edited the issue body — the .direnv/flake-inputs theory in the original report was wrong. The watcher never follows symlinks; it subscribes /nix/store directly as a parent-directory watch for the resolved skill directories. That also explains why the repro attempts stayed at ~16 watches.

   1. Full watcher subscribe / watcher started lines for one service start (run=f678edfb, 2026-10-14T08:15:16Z, http.span fields stripped). The same pattern appears in every service start and on every config-reload re-subscribe:

   watcher subscribe path=/home/bene/.config/opencode type=directory ignores=3
   watcher started    path=/home/bene/.config/opencode type=directory backend=inotify
   watcher subscribe path=/home/bene/github/testlens-app/cli type=entries
   watcher started    path=/home/bene/github/testlens-app/cli type=entries backend=node
   watcher subscribe path=/home/bene/.config/opencode/plugins/nono-sandbox.ts type=file
   watcher started    path=/home/bene/.config/opencode/plugins/nono-sandbox.ts type=file backend=node
   watcher subscribe path=/home/bene/.config/opencode/AGENTS.md type=file
   watcher started    path=/home/bene/.config/opencode/AGENTS.md type=file backend=node
   watcher subscribe path=/home/bene/.config/opencode/skill type=file
   watcher started    path=/home/bene/.config/opencode/skill type=file backend=node
   watcher subscribe path=/home/bene/.config/opencode/skills type=directory
   watcher started    path=/home/bene/.config/opencode/skills type=directory backend=inotify
   watcher subscribe path=/nix/store/abhf9ln1xhwjc7vrhx7917f2j59j36ah-hunk-0.22.0/share/skills/hunk/hunk-review type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/home/bene/.config/nono/packages/nolabs-ai/opencode/skills/nono-sandbox type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/nix/store/0dj3vjkvs055qlfibpi4x416hghp7g1g-hm_expert type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/nix/store/zda04fwpaa5kk36dykj6cl9qfaf3kx24-hm_expert type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/nix/store/2qw2ilniapzykdwqm8x5vxw6wsj10cnr-hm_expert type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/nix/store/bvxp3157r27kgav6rl07ybygbz8h5jw4-hm_expert type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/nix/store/7wqnpr20jr5sfr3d5a6y892z45raj1pv-hm_tutor type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/nix/store/30lvqkjrmjrj1bw0hxwa0b2iqn0112ps-hm_tutor type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/nix/store/vfsyiqwjsalhgcna9817rb7f0ad07wqp-hm_tutor type=directory
   watcher started    ... backend=inotify
   watcher subscribe path=/nix/store type=directory ignores=0
   (no "watcher started" line for /nix/store)
   

   The /nix/store subscription was logged 15 times today — once per service start and again on every config-reload re-subscribe (the #53393 loop).

   2. Directory counts. /nix/store contains 1,416,513 directories (find /nix/store -type d | wc -l; find -L gives the same — the store has essentially no symlinked directories). Everything else subscribed is small: ~/.config/opencode ≈ 4k entries, project trees in the tens of thousands, the individual hm_* skill dirs are tiny. Notably, the subscribed skill directories are top-level store entries, so their common parent is the store root — which is what I believe the /nix/store subscription is: a parent-directory watch, deduplicated across all seven skills. On a normal filesystem the same watch would land on ~/.config/opencode/skills/, a few dozen entries.

   3. Rendered config and directory state.

   ~/.config/opencode:
     opencode.json -> /nix/store/vk3h...-home-manager-files/.config/opencode/opencode.json
     tui.json      -> /nix/store/vk3h...-home-manager-files/.config/opencode/tui.json
     node_modules/ (plain dir, one npm dep: @opencode-ai/plugin 1.17.18; no store symlinks)
     plugins/      (plain dir)
     skills/:
       hunk-review -> .../home-manager-files/.config/opencode/skills/hunk-review
       nono-sandbox -> /home/bene/.config/nono/packages/nolabs-ai/opencode/skills/nono-sandbox
       programming.java.expert        (plain dir)
       programming.java.gradle.expert (plain dir)
       programming.java.maven.expert  (plain dir)
       programming.nix.expert         (plain dir)
       programming.nix.tutor          (plain dir)
       programming.rust.tutor         (plain dir)
       programming.tutor              (plain dir)
       writing.avoid-ai-tropes        (plain dir)
   
   opencode.json:
     "plugin": ["/nix/store/<ponytail-hash>/.opencode/plugins/ponytail.mjs"]
     "instructions": ["~/.config/agents/host.md", ... 5 files]
   

   (No ~/.config/agents directory currently exists on disk, incidentally — those five instruction paths point at hm-files entries that this config generation no longer deploys. Possibly related to the resubscribe loop.)

   One hypothesis I can't confirm without reading the watcher source: the skill loader resolves each skill directory to its real path (hm_expert etc. in the log), and the watcher adds a parent-directory watch on the resolved path so replacement/deletion is noticed. On NixOS the parent of a top-level derivation is the store root, so that parent watch becomes a recursive watch on 1.4M directories. This started today, immediately after a home-manager switch that replaced the skills — before that, the skill paths presumably resolved somewhere harmless.

   The ENOSPC numbers from the report stand unchanged: 523,226 of 524,288 watches consumed within seconds of service start.
5. opencode-agent commented on Oct 14, 2026

   

   opencode-agentBot

   on Oct 14, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the corrected analysis and the logs: the /nix/store type=directory line was the missing piece. I can confirm it on v2.0.27, and it still happens on the latest dev (34d8ded).

   The script below reproduces it without Nix:

   • a ~5,000-directory tree stands in for /nix/store, with a skill file at its top level
   • ~/.config/opencode/skills/nix.expert/ is a plain directory whose SKILL.md is a symlink to that file, the way home-manager deploys it
   • opencode serve is started and the skills are requested once

   The server then logs watcher subscribe path=<store> type=directory and holds about 5,060 inotify watches, one per directory of the unrelated tree. On a real store with 1.4M directories, that is enough to use up max_user_watches.

   The cause matches your hypothesis. The skill loader resolves each SKILL.md to its real path. If that path is outside the directories it already watches, it adds a recursive watch on the file's parent directory, which here is the store root. OPENCODE_DISABLE_FILEWATCHER=true is still the workaround for now.

   Reproduction script (repro.sh)

   #!/usr/bin/env bash
   # Repro for #55367: a skill whose SKILL.md resolves to a file at the top level of
   # a large directory (like /nix/store) makes opencode recursively watch that whole
   # directory (watch(path.dirname(resolved), "directory") in config/plugin/skill.ts).
   #
   # Usage: bash repro.sh            -> runs `opencode` from PATH
   #        OC_DEV=/path/to/checkout bash repro.sh  -> runs the CLI from source
   set -u
   ROOT=$(mktemp -d /tmp/opencode/repro55367.XXXX)
   STORE=$ROOT/store            # stands in for /nix/store
   export XDG_CONFIG_HOME=$ROOT/config XDG_DATA_HOME=$ROOT/data XDG_STATE_HOME=$ROOT/state XDG_CACHE_HOME=$ROOT/cache
   PORT=${PORT:-4599}
   
   # 1. A "store" with ~5000 directories and a top-level skill file, like a nix store path.
   for i in $(seq 1 50); do for j in $(seq 1 100); do mkdir -p "$STORE/pkg$i/d$j"; done; done
   printf -- '---\nname: hm-expert\ndescription: test skill\n---\nbody\n' > "$STORE/abc123-hm_expert-SKILL.md"
   echo "store dirs: $(find "$STORE" -type d | wc -l)"
   
   # 2. home-manager style: a plain skill dir whose SKILL.md is a symlink into the store.
   mkdir -p "$XDG_CONFIG_HOME/opencode/skills/nix.expert"
   ln -s "$STORE/abc123-hm_expert-SKILL.md" "$XDG_CONFIG_HOME/opencode/skills/nix.expert/SKILL.md"
   
   # 3. A project and the server.
   PROJ=$ROOT/proj; mkdir -p "$PROJ"; cd "$PROJ" && git init -q
   if [ -n "${OC_DEV:-}" ]; then
     CMD=(bun --conditions=browser --preload="$OC_DEV/packages/tui/node_modules/@opentui/solid/scripts/preload.js" "$OC_DEV/packages/cli/src/index.ts")
   else
     CMD=(opencode)
   fi
   setsid "${CMD[@]}" serve --port "$PORT" > "$ROOT/serve.log" 2>&1 &
   for _ in $(seq 1 60); do grep -q "server password" "$ROOT/serve.log" && break; sleep 1; done
   PASS=$(grep "server password" "$ROOT/serve.log" | awk '{print $3}')
   curl -s -u "opencode:$PASS" -H "x-opencode-directory: $PROJ" "localhost:$PORT/api/skill" | head -c 200; echo
   sleep 8
   
   PID=$(pgrep -f "serve --port $PORT" | grep -v $$ | while read p; do [ -d /proc/$p/fdinfo ] && grep -l '^inotify' /proc/$p/fdinfo/* >/dev/null 2>&1 && echo $p; done | head -1)
   TOTAL=0
   for f in /proc/$PID/fdinfo/*; do c=$(grep -c '^inotify wd' "$f" 2>/dev/null); TOTAL=$((TOTAL + ${c:-0})); done
   echo "inotify watches held by server (pid $PID): $TOTAL"
   grep -hE 'watcher (subscribe|started)' "$XDG_DATA_HOME"/opencode/log/*.log 2>/dev/null | grep -F "path=$STORE " | sed 's/timestamp=[^ ]* //' | cut -c1-160
   kill "$PID" 2>/dev/null

   Run it: bash repro.sh
6. opencode-agent

   added

   reproduced

   on Oct 14, 2026
7. britter commented on Oct 14, 2026

   

   britter

   on Oct 14, 2026

   Author

   More actions

   Workaround for other NixOS/home-manager users: wrap the opencode package so the service inherits OPENCODE_DISABLE_FILEWATCHER=true (see also #53417 for this variable's documentation gap).

   With home-manager, wrap the package instead of setting the variable in your shell — the background service (serve --service) is spawned by the CLI, so it only picks up the variable if the package itself carries it:

   programs.opencode = {
     # Disable opencode's file watcher: on NixOS it exhausts the system-wide
     # inotify watch budget (this issue).
     package = pkgs.symlinkJoin {
       name = "opencode2";
       paths = [ pkgs.llm-agents.opencode2 ];
       nativeBuildInputs = [ pkgs.makeWrapper ];
       postBuild = ''
         wrapProgram $out/bin/opencode2 \
           --set OPENCODE_DISABLE_FILEWATCHER true
       '';
     };
     # ...
   };

   Setting the variable in home.sessionVariables is not sufficient: the managed service is (re)spawned by whatever CLI instance wins the service election, and its environment may come from a different, non-interactive context.

   With this in place, the service holds ~111 inotify watches at startup instead of ~523k, and system-wide ENOSPC failures (nh switch, new TUI instances) stop immediately. Cost: no config/skill hot-reload — changes require restarting opencode sessions, which is acceptable when config arrives via home-manager switch anyway.
8. britter

   added a commit that references this issue on Oct 14, 2026

   ai-agents: disable opencode file watching

   ...

   bcbfd62
9. Dante-dan commented on Oct 14, 2026

   

   Dante-dan

   on Oct 14, 2026

   Contributor

   More actions

   I’m taking this on. The fix watches an individually symlinked skill’s resolved file through the existing non-recursive file watcher, while keeping recursive discovery for directory symlinks. The skill tests pass, including target replacement, link retargeting, and discovering new skills behind a directory symlink.
10. rekram1-node

    closed this as completedin #55597on Oct 18, 2026