Dynamic nodes stuck in IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING are permanently unschedulable, invisible to sinfo -R, and never remediated by clustermgtd #7577

This was a tough one for me to debug, had to have help from claude.ai. Here is the issue as summarized by claude.

Summary

A dynamic (cloud) compute node can end up in state
IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING. Slurm excludes such a node from scheduling, so it
is never selected for a job and ResumeProgram is never invoked — no EC2 API call is ever
made for it. clustermgtd does not recognize this state as unhealthy, so nothing ever clears
it. The node is invisible to normal monitoring: sinfo -R reports nothing, there is no
reason string, and the base state reads IDLE.

The result is silent, permanent capacity loss. On our cluster 250 of ~1720 dynamic nodes were
in this state across 30 compute resources, with 11 compute resources having zero usable
nodes. Two production queues had been running at half capacity for at least nine days. Jobs
pended with Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions, which points an operator at the wrong cause — nothing was DOWN or DRAINED.

Environment

• ParallelCluster 3.13.2 (aws-parallelcluster-cookbook-3.13.2)
• Slurm 24.05.8
• Dynamic nodes only. Compute resources spanning on-demand and spot, many instance families.

The effect is deterministic and easy to demonstrate

sbatch --test-only shows the node is not merely busy, it is not a candidate at all. Three
compute resources, same cluster, same moment:

# all 10 nodes in the latched state
$ sbatch --test-only -p od-c7a-2xl -c 1 --mem=1G --wrap="true"
allocation failure: Requested node configuration is not available

# nodes 1-2 clean, nodes 3-10 latched -> only a clean node is ever offered
$ sbatch --test-only -p od-m7a-xl -c 1 --mem=1G --wrap="true"
sbatch: Job ... to start at ... on nodes od-m7a-xl-dy-od-16-gb-4-cores-1

# all nodes clean
$ sbatch --test-only -p od-r7a-2xl -c 1 --mem=1G --wrap="true"
sbatch: Job ... to start at ... on nodes od-r7a-2xl-dy-od-64-gb-8-cores-2

Meanwhile the affected nodes look healthy by every normal check:

$ sinfo -R
REASON               USER      TIMESTAMP           NODELIST
                                                              # empty

$ scontrol show node od-r7i-12xl-dy-od-384-gb-24-cores-6
   State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
   Reason=                                                    # none
   BootTime=None SlurmdStartTime=None

Why nothing recovers it

Two independent gaps that compose:

1. Slurm cannot clear the flag. NOT_RESPONDING is cleared when a node answers a ping,
   and powered-down cloud nodes are deliberately never pinged — correctly, there is no
   instance to ping. For a powered-down node the flag is therefore cleared by an event that
   can never occur. Arguably the flag is also semantically meaningless for a node Slurm
   itself powered off.

2. clustermgtd does not consider it unhealthy. Its health checks key on base state and on
   EC2 reality. This node reads base state IDLE with no backing instance, which is exactly
   what a correctly powered-down dynamic node looks like. It is in none of the remediation
   sets. clustermgtd ran continuously for two months on our cluster, looping every 60
   seconds, and never flagged any of the 250 nodes.

Compounding this, ParallelCluster sets idle_on_node_suspend itself:

/opt/slurm/etc/slurm.conf:39
SlurmctldParameters=idle_on_node_suspend,power_save_min_interval=30,node_reg_mem_percent=75

That converts the base state from DOWN to IDLE on suspend. A node left at
DOWN+...+NOT_RESPONDING would have appeared in sinfo -R and in clustermgtd's unhealthy
list. Flipping it to IDLE removes the only signal an operator would have had, which is why
this is silent rather than merely annoying.

The state also survives a slurmctld restart, since it is restored from StateSaveLoc:

[2026-08-20T11:32:21.535] Recovered state of 1936 nodes
[2026-08-20T11:32:21.535] Down nodes: sp-r7i-l-dy-sp-16-gb-1-cores-2

1936 nodes recovered, one reported down, 250 silently unschedulable.

What I could not determine

I cannot tell you what creates the state, and I want to be upfront that on 3.13.2 every
path I traced self-heals correctly:

• Spot reclamation. Traced seven nodes end to end. clustermgtd logged no corresponding instance in EC2 / EC2 instance availability for node ... has timed out / Setting unhealthy dynamic nodes to down and power_down, and all seven landed at clean
  IDLE+CLOUD+POWERED_DOWN. No latch.
• Resume capacity failure. Produces update_node: node ... reason set to: (Code:UnfulfillableCapacity)Failure when resuming nodes then state set to DOWN — visible
  in sinfo -R, and reset by the insufficient-capacity timeout. Working as intended.
• slurmctld restart. See above, does not create it.

The affected nodes had zero events in six-plus weeks of retained logs (clustermgtd back
to Jul 8, slurm_suspend.log back to Jul 25, slurm_resume.log, slurmctld.log). So these
were most likely created by an older version or a one-off control-plane interruption and then
never swept up. My origin logs have long since rotated.

Why this is still actionable

Because the absence of a repro does not reduce the impact. Even if the bug that creates this
state is already fixed, there is no remediation for nodes already in it. It survives
cluster updates and daemon restarts, and it is invisible. A cluster affected before an upgrade
stays permanently degraded afterwards with no signal at all. Ours lost half of two queues for
at least nine days; one user's job waited 39.9 hours and then failed with ReqNodeNotAvail.

Requests

1. Remediate it. Have clustermgtd clear a stray NOT_RESPONDING flag on a powered-down
   dynamic node that has no backing instance. This is unambiguously safe: there is no
   instance, nothing is running, and the flag conveys no information about a machine that
   does not exist.
2. Failing that, surface it. A node that Slurm will not schedule should be visible
   somewhere — a reason string, a clustermgtd warning, anything. Silence is the expensive part.
3. Consider whether idle_on_node_suspend should apply to a node carrying
   NOT_RESPONDING, given it is what suppresses the DOWN state that would otherwise have
   made this observable.

Related in class, though a different trigger and a much louder symptom: #7470, where
clustermgtd likewise does not recognize a node state it did not create
(DOWN+CLOUD+MAINTENANCE+RESERVED+NOT_RESPONDING).

Workaround, for anyone who finds this

Detect:

scontrol show nodes --oneliner \
  | awk '{for(i=1;i<=NF;i++){if($i ~ /^NodeName=/)n=substr($i,10); if($i ~ /^State=/)s=substr($i,7)}
          if(s=="IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING") print n}'

Fix:

scontrol update nodename=<hostlist> state=power_down_force

The nodes cycle POWER_DOWN -> POWERING_DOWN -> clean IDLE+CLOUD+POWERED_DOWN within a
couple of clustermgtd passes, and pending jobs start once they land. On our cluster five jobs
that had been pending 11.5 hours started within a few minutes of the reset.

Note for anyone scripting this: match the state by whole +-delimited token, not substring —
POWERED_DOWN contains DOWN. And never reset a node in POWERING_UP, where
NOT_RESPONDING is legitimate and transient while the instance boots.