Bug: Agent loses authorization before run_on_stop script completes #19467

Summary

The Coder agent is losing authorization and API connectivity before run_on_stop scripts complete execution, violating the documented behavior and preventing proper workspace shutdown operations.

Expected Behavior

According to the Terraform provider documentation:

run_on_stop (Boolean) This option defines whether or not the script should run when the agent stops. The script should exit when it is done to signal that the workspace can be stopped.

The agent would be expected to maintain full functionality (including external auth access and API connectivity) until the run_on_stop script completes and signals that the workspace can be stopped.

The alternative is to change the wording in the terraform article to 'should run after the agent stops'.

Actual Behavior

The agent loses authorization and API connectivity during run_on_stop script execution, causing:

1. coder external-auth access-token commands to fail with "Workspace agent not authorized"
2. API connectivity tests to fail
3. Git operations requiring authentication to fail

This reduces the ability of a shutdown script to perform actions.

Evidence

The example is from a shutdown script which is created and should perform GIT commands on shutdown.

Logs from run_on_stop script execution:

[2025-08-21 12:27:14] === On-stop demo script started ===
[2025-08-21 12:27:14] System information:
[2025-08-21 12:27:14] Hostname: coder-workspace-pod
[2025-08-21 12:27:14] Uptime:  12:27:14 up  4:31,  0 user,  load average: 0.19, 0.32, 0.33
[2025-08-21 12:27:14] Disk usage: overlay          61G   42G   17G  72% /
[2025-08-21 12:27:14] Workspace information:
[2025-08-21 12:27:14] Workspace name: test-workspace
[2025-08-21 12:27:14] Workspace owner: user
[2025-08-21 12:27:14] Workspace ID: [workspace-id]
[2025-08-21 12:27:14] Coder API connectivity: FAILED  ← API already failing
[2025-08-21 12:27:14] Looking for Git repositories in /home/coder/src/server...
[2025-08-21 12:27:14] Found Git repository at: /home/coder/src/server/repo
[2025-08-21 12:27:14] Git status:
 M TEST9.TXT
?? TEST10.TXT
[2025-08-21 12:27:14] Current branch: main
[2025-08-21 12:27:14] All branches: * main
  remotes/origin/HEAD -> origin/main
  remotes/origin/main
[2025-08-21 12:27:14] Remote branches:   origin/HEAD -> origin/main
  origin/main
[2025-08-21 12:27:14] Remote URL: https://github.com/user/repo
[2025-08-21 12:27:14] Last commit: abc123 Manual commit
[2025-08-21 12:27:14] Upstream branch: origin/main
[2025-08-21 12:27:14] Repository has full history
[2025-08-21 12:27:14] Created workspace status file
[2025-08-21 12:27:14] Found uncommitted changes, creating commit...
[main def456] Auto-save on workspace stop - Thu Aug 21 12:27:14 PM UTC 2025
 3 files changed, 3 insertions(+), 1 deletion(-)
 create mode 100644 TEST10.TXT
[2025-08-21 12:27:14] Attempting to get GitHub token directly...
[2025-08-21 12:27:14] ✗ Failed to get GitHub token directly  ← External auth failing
[2025-08-21 12:27:14] This may indicate that GitHub external auth is not configured or the agent cannot access it
[2025-08-21 12:27:14] Attempting push without authentication...
Encountered an error running "coder", see "coder --help" for more information
error: Trace=[get git token: ]
Workspace agent not authorized.  ← Agent unauthorized during script execution
Suggestion: Try logging in using 'coder login'.
error: unable to read askpass response from '/tmp/coder.XXXXXX/coder'
fatal: could not read Username for 'https://github.com': No such device or address
[2025-08-21 12:27:15] Saved workspace metadata to /tmp/workspace-metadata.json
[2025-08-21 12:27:15] Cleaning up temporary files...
[2025-08-21 12:27:15] === On-stop demo script completed successfully ===
[2025-08-21 12:27:15] Check /home/coder/shutdown.log for full execution details

Key Issues Observed:

1. API Connectivity Fails: curl -s -H "Coder-Session-Token: ${CODER_AGENT_TOKEN}" "${CODER_URL}/api/v2/workspaces" returns failure
2. External Auth Fails: coder external-auth access-token GH returns "Workspace agent not authorized"
3. Script Continues: The script continues to run and completes successfully, but without the expected agent capabilities

Impact

This bug prevents legitimate use cases such as:

• Auto-committing and pushing code changes on workspace stop
• Backing up workspace state to external services
• Performing authenticated cleanup operations
• Any operation requiring external auth or API access during shutdown

Similar Issues

• investigate why run_on_stop doesn't work consistently #17108
• Support for running a script on workspace delete #13387

Workaround

We've created a workaround that caches external auth tokens during workspace startup and uses them during shutdown:

resource "coder_script" "startup_auth" {
  agent_id     = coder_agent.main.id
  display_name = "Cache Auth Tokens"
  run_on_start = true
  
  script = <<-EOT
    #!/bin/bash
    mkdir -p ~/.cache/coder
    # Cache GitHub token for use in shutdown script
    if coder external-auth access-token GH > ~/.cache/coder/github_token 2>/dev/null; then
        chmod 600 ~/.cache/coder/github_token
        echo "Successfully cached GitHub token"
    fi
  EOT
}

resource "coder_script" "shutdown_script" {
  agent_id     = coder_agent.main.id
  display_name = "Auto-save on Stop"
  run_on_stop  = true
  
  script = <<-EOT
    #!/bin/bash
    # Use cached token instead of direct external-auth access
    if [[ -f ~/.cache/coder/github_token ]]; then
        GITHUB_TOKEN=$(cat ~/.cache/coder/github_token)
        # Use token for git operations...
    fi
  EOT
}

However, this workaround has security implications:

• Tokens are stored on disk in the workspace
• Tokens may expire during long-running workspaces (24 hours)
• It adds complexity and potential failure points

Environment

• Coder Version: Dev based off 2.24.X-2.25.1
• Platform: Kubernetes
• Template: Using coder_script resource with run_on_stop = true
• External Auth: GitHub configured and working during normal operation

Reproduction Steps

1. Create a template with a run_on_stop script that uses coder external-auth access-token
2. Create a workspace from the template
3. Stop the workspace
4. Observe that external auth fails during script execution

Expected Fix

The agent should maintain authorization and API connectivity until the run_on_stop script exits, as documented. The agent shutdown sequence should be:

1. Trigger run_on_stop scripts
2. Wait for all run_on_stop scripts to complete
3. Only then revoke agent authorization and stop the agent
4. Proceed with workspace shutdown

This would align with the documented behavior and enable proper workspace cleanup operations.