[Feature Request] Preserve the original drain reason when clustermgtd replaces an unhealthy node #7646

Summary

When clustermgtd replaces an unhealthy static node, it overwrites the node's existing Slurm reason with a fixed string, Static node maintenance: unhealthy node is being replaced. Any detailed reason an administrator or automation set when draining the node (for example GPU, NVLink or Xid diagnostics) is lost from scontrol show node / sinfo -R.

Many sites use drain reasons as the primary signal for alerting, incident response and hardware triage. Today there is no configuration option to keep that information once ParallelCluster takes over the node.

Current behavior

In src/slurm_plugin/clustermgtd.py (_handle_unhealthy_static_nodes), all nodes on the unhealthy replacement list are reset with the same reason:

reset_nodes(node_list, state="down", reason="Static node maintenance: unhealthy node is being replaced")

• v3.15.1: https://github.com/aws/aws-parallelcluster-node/blob/v3.15.1/src/slurm_plugin/clustermgtd.py
• Same code on develop: https://github.com/aws/aws-parallelcluster-node/blob/develop/src/slurm_plugin/clustermgtd.py

Dynamic nodes are handled the same way, using reason="Scheduler health check failed".

A DRAINED static node is treated as unhealthy (terminate_drain_nodes defaults to True), so the overwrite happens on the next clustermgtd loop, which runs every 60 seconds by default.

Steps to reproduce

Tested on ParallelCluster 3.15.1, Slurm scheduler, a queue with one static node.

1. Drain the static node with a detailed reason:

   scontrol update nodename=q1-st-cr1-1 state=drain reason="TEST: GPU0 Xid 79 fell off bus, job 12345"
   

   State=IDLE+CLOUD+DRAIN
   Reason=TEST: GPU0 Xid 79 fell off bus, job 12345 [root@2026-09-30T16:21:38]
   

2. Wait one clustermgtd loop. /var/log/parallelcluster/clustermgtd shows:

   16:22:20 Node state check: node q1-st-cr1-1 in DRAINED, node state: IDLE+CLOUD+DRAIN
   16:22:20 Setting unhealthy static nodes to DOWN
   16:22:20 Terminating instances backing unhealthy static nodes
   16:22:21 Launching new instances for unhealthy static nodes
   

3. The original reason is gone:

   State=DOWN+CLOUD+DRAIN+NOT_RESPONDING
   Reason=Static node maintenance: unhealthy node is being replaced [root@2026-09-30T16:22:20]
   

Expected behavior / proposal

reset_nodes updates all unhealthy nodes in a single scontrol update call (setting DOWN, resetting NodeAddr/NodeHostname and writing the reason), and Slurm requires a reason when setting a node DOWN. This explains why every node receives the same static string. The request is to keep the original reason visible while still providing a reason for DOWN, for example:

1. Append the previous reason to the replacement reason:
   Static node maintenance: unhealthy node is being replaced (previous reason: <original reason>)
2. Make the replacement reason format configurable in the clustermgtd config.

Since the node's current reason is already read from scontrol show nodes in each loop, this could be done by grouping nodes with the same existing reason into one reset_nodes call, or by issuing per-node updates for nodes that already carry a reason.

Option 1 keeps the current marker string, so existing tooling that matches on it keeps working, while preserving the diagnostic detail.

Current workaround

Slurm logs every reason set through scontrol update to slurmctld.log (also in the slurmctld CloudWatch log stream when CloudWatch logging is enabled), so the original reason can be recovered from the last update_node: node <name> reason set to: line before the Static node maintenance line:

[2026-09-30T16:21:38.025] update_node: node q1-st-cr1-1 reason set to: TEST: GPU0 Xid 79 fell off bus, job 12345
[2026-09-30T16:22:20.636] update_node: node q1-st-cr1-1 reason set to: Static node maintenance: unhealthy node is being replaced

This requires parsing logs rather than querying node state. It also only covers reasons set with scontrol update; drains Slurm makes on its own are logged differently.

Setting terminate_drain_nodes = False would keep the reason, but it also stops automatic replacement of drained nodes, so it doesn't solve this.

Environment

• ParallelCluster: 3.15.1 (behavior unchanged on develop)
• Scheduler: Slurm
• Node type: static (dynamic nodes affected similarly)