update workspace button does not update agent to new version #8266
Description
in the screenshot below, the modal correctly says you can stop and start the workspace to update the agent. but the update workspace link at the bottom of the text does not update the agent as there's no tf resources changing.
it's possible we are new "Start" build, instead of a Stop & Start.
Activity
bpmct commentedon Aug 25, 2023
Interesting find. @mafredri - any idea why this is happening?
mafredri commentedon Aug 25, 2023
The assumption seems to be that a workspace should (re)start along with the update, which would update the agent. But that doesn't seem to be happening.
@ericpaulsen is this issue also reproducible via the CLI if you run
coder update workspace? I wonder if the behavior differs between CLI and dashboard (thoughts @BrunoQuaresma?)In general, I find it a bit weird that a workspace can be updated without a (re)start, so to speak. But perhaps that makes sense?
BrunoQuaresma commentedon Aug 25, 2023
In the FE, we send a stop/start build request to the API with the new template version so I would expect it to be updated. But, we definitely should have the API better endpoints for workspace operations.
mafredri commentedon Oct 24, 2023
@BrunoQuaresma it looks like the frontend is only performing a
starttransition these days when I press "Update" or change to a specific template version. Is it a regression?BrunoQuaresma commentedon Oct 24, 2023
Hm... in the FE we always "only updated" the workspace using the start transition pointing it to a version...not sure about the agents.
matifali commentedon Oct 24, 2023
One way to resolve this is to use stop/start for the "Update" workspace, as we do with the "Restart" workspace.
spikecurtis commentedon Oct 30, 2023
This is an example of our imperative UX clashing against Terraform's declarative design. In Terraform you should be able to just specify the final state -- there is nothing weird about going from (start, v1) -> (start, v2).
However, I do wonder whether we are correctly modeling the agent resource in Terraform. Is the agent resource reporting that it needs to be recreated? If not, then Terraform will just conclude that it's already in the desired state, even though it clearly isn't because we wanted the agent to get updated.
mafredri commentedon Oct 30, 2023
@spikecurtis that's exactly the issue we're dealing with. The problem is, our
coder_agentresource is basically just a representation of the database entry. Recreation of this resource means we simply add a new agent entry in the database. This invalidates the existing agent token but since the agent is managed/bootstrapped via the workspace, there is no signal to trigger workspace recreation.In essence, and in terraform spirit, we wouldn't even need to recreate the workspace, but this moves a lot of responsibility to the template authors as there can be many edge cases with restarting an agent in a dirty workspace.
Edge cases include:
Not to mention how are authors to ensure things are triggered correctly? Self-updates of the agent could be another solution, but has all of the aforementioned edge-cases (let's say the new template updated the startup script, now we have to re-run them).
Therefore, "restart" is a work-around that will always work, no matter the template. If we can coerce our terraform provider to trigger workspace re-creation on agent re-creation, that would be a better solution than doing
stop -> start(faster, if nothing else). But will that work in all instances? Probably not, so there might be added responsibility on template authors even in this case.Imagine if a workspace doesn't reference the agent (how do we know it's the "workspace"?), and instead the agent is downloaded and launched via null resources. Just a thought.
bpmct commentedon Oct 30, 2023
spikecurtis commentedon Oct 31, 2023
Unfortunately, this is not true. The template always matters. As an extreme example to illustrate the point, consider a template that simply ignores the start/stop signal from the Coder provider.
A more realistic example is one where a VM is created & the agent installed on the first build, but then start/stop just run (Terraform) provisioners that unpause and pause the VM.
I'll grant that it's more likely that a stop followed by a start will do what we want for a lot of templates we see in the wild, notably container templates.
One thing we might consider is making the
coder_agentresource itself (or it's key properties like startup script & token) contingent on stop/start, such that Terraform's planning logic will force template authors to deal with the fact that the agent shouldn't exist on a stop; that it's properties can change on rebuilds, and they need to deal with that in their template code.mafredri commentedon Oct 31, 2023
Great points @spikecurtis. I admit I hadn't considered use cases like the VM one, and I do feel it's a valid use-case, although it may be riddled with other gotchas with regards to propagating template updates (not just restarts of the agent). I will change my argument to be that a stop/start will handle most terraform templates. I think a lot of our users will use either Docker, Kubernetes or cloud providers, and in all these providers, I think the most common step is to destroy the workspace resource on stop (to save on resources).
How would we make it so that template authors are forced to deal with it? Even now, the
coder_agentresource is destroyed and re-created, but [this issue] persists.One trick we can employ is to e.g. modify the agent init script (
CODER_AGENT_ID=[placeholder]) on start. In some cases, this may trigger a workspace destroy/create because of how it's used (and provider behavior), but in other cases it may just update a file in the workspace. So we're still not at a complete solution.Here are my current thoughts on ways we can alleviate this, but none of this feels like a complete solution (and we definitely don't want to do all of these):
create,start,stop,start->start,deletefor the current template version to check that it behaves as expectedstart -> start, for instanceexec(Windows support is iffy)spikecurtis commentedon Oct 31, 2023
The
coder_agentresource is not currently destroyed at the terraform layer on a stop build. It's always present but just regenerates its token each build. If we made it so that you cannot actually reference the agent on a stop build, that might force template authors to deal with its absence.bpmct commentedon Nov 1, 2023
I suppose you're right, but think a restart will work for 90-99% of our users' cases. If a workspace ignores stop/start signals or doesn't download our agent from coder.example.com/bin, the best we can do is warn that an agent is outdated.
matifali commentedon May 8, 2024
This is still relevant and a valid discussion