Repository navigation
Can the Coder agent immediately realize it's disconnected? #5901
Description
With #5292, the experience is quite nice when I type sudo reboot or a pod gets rescheduled (e.g. on spot VMs). However, in the dashboard, the agent still thinks it's connected. Could this be instant and we display a message to the user saying the agent has disconnected? Eventually, this could work with #4680 or other messages in the terminal/IDEs to inform the user.
Metadata
Metadata
Assignees
Labels
No labels
Activity
mafredri commentedon Jan 28, 2023
This is something I could look at before tackling #4677 to add shutdown lifecycle states. It should be fairly minor:
feat(agent): Handle signals and shutdown gracefully
feat(agent): Handle signals and shutdown gracefully (#5914)
bpmct commentedon Feb 10, 2023
@mafredri is there another part to this? I noticed that the dashboard still reports the agent is online
Screen.Recording.2023-02-10.at.1.49.00.PM.mov
mafredri commentedon Feb 10, 2023
@bpmct if the agent is getting killed abruptly, then there’s not much we can do except wait for the timeout to detect the agent is disconnected. Recent changes should improve conditions to avoid being killed abruptly, but it’s still possible.
Could you share how the agent is being run? That could belp discern how it is being stopped. Logs from the agent might also help.
Ultimately, if the agent is receiving SIGINT or SIGTERM, then disconnection should work. You can try this via
kill -INT $agent_pid).bpmct commentedon Feb 10, 2023
It's being started via user-data, same as our aws-linux example.
What is the timeout for the agent at this point? How long would it take for the server to realize an agent looses connection after an agent is killed abruptly?
mafredri commentedon Feb 10, 2023
I can evaluate the relevant timeouts on Monday, don’t know off the top of my head (I believe they may also be configurable).
In the meantime, can you verify whether
InstanceInitiatedShutdownBehavioris set to Stop or Terminate? I’m hoping it’s Terminate, in which case changing to Stop might fix the issue.https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingInstanceInitiatedShutdownBehavior
mafredri commentedon Feb 13, 2023
@bpmct I'm trying to reproduce this issue (not on AWS), but each time I try to kill the agent, gracefully or abruptly (
kill -9), the disconnection status is immediately.That is, it should be fixed in 2dbe00a.
The thing is, I looked at your recording and it's showing your agent is based on commit 2dbe00a, and I have no idea why it's not working for you 🤔.
mafredri commentedon Feb 13, 2023
So after @bpmct helped me reproduce this issue (🙏🏻), it seems that we have an agent inactivity reporting issue.
As Ben observed, rebooting the server is an easy repro. This causes the agent to be considered disconnected only after like 20 minutes (Tailscale/WireGuard timeout I presume).
Another way to reproduce this is to stop networking:
However, simply killing the agent (either gracefully or immediately) does not reproduce this issue:
Why the difference? Since networking is still up, in the last case, we can immediately detect that other end went away. In the reboot case, the networking is shut down before the agent because it's not running as a systemd service (manually stopping the network mimics this case).
Typically an agent is considered disconnected after not talking to the server for 6 seconds, but for some reason, this is not happening here. We need to dig deeper into why this is and since we can't really reduce the Tailscale/WireGuard timeout, we need another path to discover that chatter has ceased.
fix(coderd): Detect agent disconnect via inactivity (#6528)
bpmct commentedon Mar 13, 2023
Nice!
Screen.Recording.2023-03-13.at.6.34.12.AM.mov