Skip to content

UnmountAllUnder reads all of /proc/self/mountinfo on every call, which contends with mounts on dense gVisor nodes #2950

Open

Description

On a node running a few hundred gVisor actors, ateworker-gvisor threads spend a lot of time blocked in the kernel while reading /proc/self/mountinfo (they show m_start in /proc/<tid>/wchan).

imagecache.UnmountAllUnder reads the whole mount table on every call, and the gVisor worker calls it several times per actor teardown: resetRunscStateAndPidFileDirs (for the runsc state dir and the pidfile dir) and the bundle dir cleanup in main.go. With about 1,000 mounts in the worker's mount namespace, each read seems to hold the mount namespace lock for a while and contend with the mounts and unmounts that other actors' restores and suspends are doing.

Measured on one c3-highcpu-192-metal node (5035a6fa, one gVisor worker, sweperf astropy-7336, closed-loop users), sampling blocked worker threads once per second:

users worker threads blocked in m_start, per sample
150 15
225 62

Possible fix: find the mount points under the target directory with a walk that checks each entry with statx (STATX_ATTR_MOUNT_ROOT), without descending into mount points, and fall back to mountinfo if the walk is incomplete. In a prototype, the m_start waits went to 0 at 225 users and throughput improved about 6–8%. I'll send it as a PR referencing this issue.

Activity

  1. pmandewalkar commented on Oct 20, 2026

    @pmandewalkar
    ContributorAuthor

    being addressed by Besher in #2963

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions