Graceful shutdowns for coder agents and shutdown scripts

In the ideal scenario where a workspace agent is connected to the coder server, we should initiate agent shutdown before we initiate Terraform provisioning to destroy/re-create the resource(s).

Why?

Providers behave differently, some may not initiate graceful shutdowns and we might not be able to control timeouts for them
Template authors may use the agent shutdown_script to perform a critical task that must complete successfully (e.g. backing up filesystem)
- The task/script may take a long time
We can leave the workspace running and allow debugging an agent that didn't successfully execute its shutdown_script

To consider:

At the end we must not exit the agent process
- Only applies when graceful shutdown is initiated by coder server!
- We should de-register signal handlers and wait "indefinitely"
- Let the next signal terminate the process
- Why? This prevents e.g. a systemd service from restarting the agent
Should behavior differ if a workspace is started/stopped?
What happens when an agent is disconnected and we can't tell it to shut down? -> Block or allow de-provision? Require the use of "force"?

Related: #4677, #5914, #6139

This is still relevant, so re-opening.

Hello, is there any update to this issue? Our organization has needs to run a script that would save state into something more permanent. It would be super useful if we were able to easily place a script in a predetermined directory or some other method that would allow it to run the script during shutdown.

@kaypeter87

Hi @kaypeter87, there are no real updates at this time (this issue is still relevant and hasn't changed).

However, what you're looking to achieve should be possible, with a caveat.

If you look over at #6174 you may find a solution for how to allow graceful shutdown to be extended depending on the provider you're using, caveat being that it can't be changed dynamically and you will have to pick a duration as the maximum grace time for shutting down.

Once you've ensured that stop only sends an interrupt signal to the workspace and waits for the agent to fully shut down, you can update the agent in your template to something like this:

resource "coder_agent" "main" {
  # ...
  shutdown_script = <<-EOT
    DIR=/path/to/shutdown.d
    if [ -d "$DIR" ]; then
      for script in "$DIR"/*.sh; do
        if [ -x "$script" ]; then
          "$script"
        fi
      done
    fi
  EOT
}

With that done, you can place executable shell files (.sh) into the directory specified in $DIR and have them run at shutdown.

Here's an example usage:

mkdir -p /path/to/shutdown.d
echo $'#!/bin/bash\necho Hi from shutdown\n' > /path/to/shutdown.d/hello.sh
chmod +x /path/to/shutdown.d/hello.sh

(The #!/bin/bash shebang is important here, since it's what makes the script executable in combination with the chmod, the shutdown script could alternatively be written differently if other use-cases need to be supported.)

Does this help?

NOTE: Providers usually have a short grace period, so this should work without any additional changes, but if you want to run long-running operations, please do take a look at the issue I linked earlier.

@spikecurtis

This has come up again. @spikecurtis identified additional failure scenarios:

Externally provisioned workspaces: We don't touch the underlying compute at all
Infrastructure timeout permissions: Template authors might not have permission to extend timeout settings
Terraform timing race: Terraform destroy might not wait for shutdown scripts, causing the stop build to complete before scripts finish

And clarified the solution:

Shutdown scope: Running scripts, terminating SSH and Reconnecting PTY connections, tearing down tailnet tunnels
Wait condition: When all agents are finished, disconnected, or hit configurable timeout, then kick off Terraform build

See #21538 (comment)

Navigation Menu

Repository navigation

Graceful shutdowns for coder agents and shutdown scripts #6175

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Graceful shutdowns for coder agents and shutdown scripts #6175

Description

Activity

github-actions commented on May 15, 2023

mafredri commented on Jan 9, 2024

1 remaining item

kaypeter87 commented on Sep 25, 2024

mafredri commented on Sep 26, 2024

mafredri commented on Jan 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions