Repository navigation
Decide upon and document agent shutdown behavior and guarantees #6179
Description
If our users are to rely on the coder agent shutdown_script and graceful shutdown in general, we need to document how it behaves in different circumstances and what guarantees we provide.
Some initial thoughts on the matter:
- Should agent shutdown failure prevent de-provisioning?
- How long do we wait? Forever? Configuration options?
- What happens if an agent is disconnected and we can't initiate pre-provision shutdown request
- Do we wait for X time before de-provision?
- Do we wait indefinitely, but give users the option to shutdown anyway (with caveats of shutdown script may not be run, etc.)
- It's possible to
coder start wson an already started workspace, what should happen in this scenario?- Ideally, we would request the agent to shut down, but it may not be possible or may fail, what then?
- Should shutdown script run if startup script failed?
- Should startup script failure affect if we allow de-provisioning or not?
Activity
kconley-sq commentedon Feb 21, 2023
Here are my thoughts on those questions in the context of the use case we have in mind for this feature (#4677 (comment)):
Yes I think we would want the shutdown script failing to prevent de-provisioning in our use case since the shutdown script failing could mean that we failed to sync data in RAM to the data volume, so if Coder proceeded to tear down the EC2 instance before that data is synced from RAM to the EBS volume then we could lose data when we take an EBS snapshot of that EBS volume for persisting the Coder user's workspace home directory.
I think we would prefer to have a timeout configuration option. I think running something like
syncshouldn't take more than a few seconds, so I think we would want to have the Coder workspace stop operation fail if the shutdown script took longer than ~10 seconds to avoid the Coder user not being able to do anything with their workspace since we also disable their ability to cancel workspace builds.In our use case I don't think we would want the workspace to be shutdown immediately as usual due to the risk of data loss, but I think giving the users the option to shutdown anyway with a caveat warning would be good just in case they're willing to accept that data loss risk over getting their workspace shutdown sooner. Otherwise I'd like them to reach out to an admin on my team who has AWS console access to help them debug the issue and try to recover their workspace using the AWS console (e.g. we may be able to use the AWS serial console to access their EC2 instance and ensure data is synced).
But I don't think waiting for X time would be useful in our scenario since we haven't really seen a case where an agent being disconnected comes back online after a short period of time; if the agent is disconnected I'm not sure it's likely to come back online automatically.
Our Coder workspace template would end up tearing down and recreating the EC2 instance in this scenario so I think we would want the agent to run the shutdown script in this case to ensure we can sync any data in RAM to the EBS volume to be snapshotted.
If the shutdown script failed then I think we would want the overall workspace build to fail too in order to avoid data loss. I think in this case a user could then try to "stop" their workspace. Even if the shutdown script fails in this case, I'm thinking they could use the "override" button you proposed to force-stop it if they were willing to accept the risk of data loss.
To me it seems like the answer to this question would depend on what the startup and shutdown scripts do in a template, so I would vote for making this configurable in the workspace template.
I think in our use case we would still want to support de-provisioning a workspace even if the startup script failed since we don't do anything super important in ours (e.g. we use it to launch the
code-serverbut I don't think that failing should prevent stopping a workspace).But I can imagine that some startup scripts may be more critical than ours, so maybe this could be another configuration option that could be set in the workspace template?
github-actions commentedon May 23, 2023