Skip to content

Multiple fff engines per process watch the same base path, multiplying inotify watches 2-3x #50966

Closed as not planned

Description

@connor-grady

Description

On Linux every inotify watch held by an opencode process belongs to fff, and a single process creates two or three independent fff engines for the same base path. Each engine opens its own inotify instance and registers one watch per indexed directory, so the same directory set is watched two or three times over. The kernel deduplicates watches by inode within one inotify instance but never across instances, so this multiplies the process's share of the per-user fs.inotify.max_user_watches budget with no functional benefit.

Measured across five live interactive sessions on one machine. engines counts fff-watcher-own threads, watches sums inotify wd: lines across all of /proc/<pid>/fdinfo/*, and distinct_dirs counts unique inodes in those same lines. Sessions A through C were opened on one repository, D and E on another.

session  engines  inotify_fds  watches  distinct_dirs  ratio
A        2        2            227      227            1.00
B        2        2            436      227            1.92
C        2        2            436      227            1.92
D        3        3            253       92            2.75
E        2        2            170       91            1.87

The count of inotify instances equals the count of fff-watcher-own threads in every case, so nothing else in the process contributes watches. In session D all three engines watch the same base path, and two of them hold byte-identical inode sets:

fd30 (84) vs fd32 (85): shared=83 only_a=1 only_b=2
fd30 (84) vs fd35 (84): shared=84 only_a=0 only_b=0
fd32 (85) vs fd35 (84): shared=83 only_a=2 only_b=1
union of all: 86   sum of sizes: 253

A headless opencode serve child serving a single directory holds exactly one engine, so this shows up on the interactive path.

This is a multiplier on an already large base. A session opened on a nixpkgs checkout indexes about 38k directories per worktree, and one opened on a bare-repo container holding eight worktrees indexes 304195. On this machine that saturated the limit at 522895 of 524288 watches and left 81 for everything else, at which point an unrelated vite dev server died with ENOSPC: System limit for number of file watchers reached. Removing the multiplier does not fix that case by itself, but it is the part that costs nothing to recover. Watches per tree are irreducible on Linux: notify's RecursiveMode::Recursive is emulated one watch per directory, and the O(1) whole-tree fanotify marks FAN_MARK_MOUNT and FAN_MARK_FILESYSTEM both require CAP_SYS_ADMIN per fanotify_mark(2).

Where it appears to come from: packages/core/src/filesystem/search.ts binds fffLayer through makeLocationNode, and packages/core/src/location-service-map.ts memoises it in a LayerMap keyed by Location.Ref. packages/schema/src/location.ts defines Ref as { directory, workspaceID? }, so two refs that resolve to the same directory each construct their own Fff.create({ basePath }), with their own index and their own inotify instance. Keying the engine by resolved base path would collapse them with no behaviour change, and would also drop the duplicated in-memory index (the large session above was holding 1.0 GB RSS).

Plugins

One local plugin is configured. I have not isolated whether it influences how many Location.Ref values get created.

OpenCode version

1.18.31

Steps to reproduce

  1. On Linux, open an interactive opencode session in a git worktree.
  2. Count engines: cat /proc/<pid>/task/*/comm | grep -c '^fff-watcher-own$'
  3. Count watches: cat /proc/<pid>/fdinfo/* | grep -c '^inotify wd:'
  4. Count distinct watched directories: cat /proc/<pid>/fdinfo/* | grep -oP 'ino:\K\S+' | sort -u | wc -l
  5. Step 3 exceeds step 4 whenever more than one engine resolves to the same directory.

Operating System

Linux 7.2.5 x86_64

Terminal

Ghostty

Activity

  1. Shalin-Shah-2002 commented on Oct 3, 2026

    @Shalin-Shah-2002

    I reproduced this and opened a fix: #52964. A location reload builds the replacement graph while running sessions still borrow the old one, so each held its own fff engine for the same directory. The PR shares one engine per directory until its last user closes.

  2. thdxr commented on Oct 11, 2026

    @thdxr
    Member

    Thanks for the report! OpenCode 2.0 is now out and development is focused on V2, so we're closing issues filed against OpenCode 1. Please upgrade (opencode upgrade --major, details: https://anoma.ly/notes/opencode-2-0), and if this is still relevant on 2.0, reopen this issue or open a new one with your 2.0 version.

    — from 𝕺𝖕𝖊𝖓𝕮𝖔𝖉𝖊

  3. connor-grady commented on Oct 11, 2026

    @connor-grady
    Author

    Filed the v2 follow-up as #54631. The duplication still happens on v2 through a different path: /reload and the other location invalidations keep the old graph's fff engine alive beside its replacement, and on every v2 release so far the old engine is never destroyed. The workspaceID cause proposed above does not apply on v2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions