Skip to content

Make Slurm epilog cleanup job-scoped #1407

New issue
New issue
Open
Open
Make Slurm epilog cleanup job-scoped#1407

Description

@dholt
dholt
opened on Sep 10, 2026

Track a focused follow-up for the existing Slurm job-end cleanup behavior.

40-lastuserjob-processes terminates processes by user, 42-lastuserjob-cleanup removes files by user ownership, and 50-lastuserjob-all-enroot-dirs removes configured Enroot paths. Those scopes do not necessarily belong only to the job that just ended.

Acceptance criteria:

  • Limit cleanup to processes and paths owned by the ending job.
  • Preserve unrelated login sessions, concurrent or suspended jobs, and shared paths used on other nodes.
  • Add regression tests for missing/erroring ownership information and overlapping workloads.
  • Preserve useful behavior; do not broadly disable tuning, SSH handling, or cleanup as a shortcut. Any necessary behavior change should be reviewed explicitly.

Related: #1391 and #1404. Historical #1315 concerns PID parsing; this follow-up concerns cleanup scope.

Keep this as a separate backlog item, not a prerequisite for unrelated fixes in the current PR queue.

Activity

  1. dholt
    added 3 commits that reference this issue on Sep 17, 2026

    fix(slurm): count suspended and configuring jobs in the last-user-job…

    e829637

    fix(slurm): stop the epilog file cleanup from deleting bystanders

    97c4795

    test(slurm): cover the epilog cleanup guards

    9f7959c
  2. dholt
    added a commit that references this issue on Oct 1, 2026

    docs: record owner decision (schedule B) in epilog design note (#1407)

    e872dc4
  3. dholt
    added a commit that references this issue on Oct 22, 2026

    docs(slurm): describe non-cgroup tracker fallback and processes start…

    7b529ca
  4. michael-balint
    added a commit that references this issue on Nov 4, 2026

    Merge pull request #1433 from NVIDIA/dholt/epilog-1407-job-scoped-r2

    433911f

  5. github-actions commented on Nov 10, 2026

    @github-actions
    github-actionsBot
    on Nov 10, 2026 – with GitHub Actions

    This issue is stale because it has been open for 60 days with no activity. Please update the issue or it will be closed in 7 days.

  6. github-actions
    added
    no-issue-activity
    on Nov 10, 2026

  7. dholt commented on Nov 10, 2026

    @dholt
    dholt
    on Nov 10, 2026
    ContributorAuthor

    Status update: #1433 (merged for 26.11) makes the Slurm epilog cleanup job-scoped by default. Processes are matched to the ending job through their cgroup, other running jobs of the same user are left alone, and unreadable ownership data defers the cleanup instead of removing anything. slurm_epilog_cleanup_scope: user keeps the previous behavior for now (deprecated). The tests under tests/slurm-epilog/ cover missing ownership data and overlapping jobs.

    Still open here:

    • Processes started outside Slurm (for example over SSH) while a job ends are not protected. They count as unassigned and can be removed after the user's last job ends. This is documented in docs/slurm-cluster/epilog-job-scoped-cleanup.md.
    • The overlap race has not been reproduced end to end through the scheduler, so the fix is backed by tests, not by a live reproduction of the original race.

    Keeping this issue open for those two items.

  8. github-actions
    removed
    no-issue-activity
    on Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions