Escape sequences like DA responses can end up fragmented across multiple TCP/IP packets #19282

Activity

1. Michael Weiss (primeos-work)

   added the

   Bug

   issue type on Aug 26, 2025
2. Michael Weiss (primeos-work)

   added

   Issue-BugIt either shouldn't be doing this or needs an investigation.It either shouldn't be doing this or needs an investigation.

   Needs-TriageIt's a new issue that the core contributor team needs to triage at the next triage meetingIt's a new issue that the core contributor team needs to triage at the next triage meeting

   on Aug 26, 2025
3. similar-issues-ai commented on Aug 26, 2025

   

   similar-issues-aiBot

   on Aug 26, 2025 – with Similar Issues AI

   More actions

   We've found some similar issues:

   • Escape sequence leaking to terminal on starting/attaching tmux (over ssh) #17439 , similarity score: 84%

   If any of the above are duplicates, please consider closing this issue out and adding additional context in the original issue.

   Note: You can give me feedback by 👍 or 👎 this comment.

   👍React with 👍1Reacted by Michael Weiss
4. primeos-work commented on Aug 27, 2025

   

   Michael Weiss (primeos-work)

   on Aug 27, 2025

   Last edited by primeos-work

   Author

   More actions

   I originally reported this issue as PowerShell/Win32-OpenSSH#2275 but will close that issue in favor of this one as we noticed that the source of this issue should be found in the Windows Terminal source code instead (that analysis/discussion starts at PowerShell/Win32-OpenSSH#2275 (comment)) and with PowerShell/openssh-portable#771 this can probably be considered fixed on the Win32-OpenSSH side.

   There are quite a few duplicate issues here (and in other repos, forums, etc.) but they've been closed without correctly identifying the source of this issue:

   • ConPTY(?) inserts broken control codes into tmux sessions. #7185
   • Terminal putting ANSI Escape Code in terminal line when attaching tmux session #16384
   • Escape sequence leaking to terminal on starting/attaching tmux (over ssh) #17439
   • Printing specific sequence causes device attributes sequence to appear #19120 (might also be a result of the timing/fragmentation)
5. lhecker commented on Aug 27, 2025

   

   Leonard Hecker (lhecker)

   on Aug 27, 2025

   Last edited by lhecker

   Member

   More actions

   Referencing the analysis you linked:

   One thing Windows Terminal and ConPTY folks might want to check is if their device-attributes response is really being sent out via a single write() system call (as ESC sequences always should be).

   This doesn't make sense to me. First, we have been doing that for a long time.
   More importantly though, the PTY that underlies a terminal is essentially a binary ring buffer which has no idea about the boundaries of escape sequences in it. This is true on both Windows and Linux (and certainly on any other NIX). This means that all sorts of reasons can cause escape sequences to be chunked up, including into broken UTF8 sequences. Applications should handle it by implementing a proper finite state machine for parsing them. Many applications don't do this properly, including application you may assume to be of high quality. That's their fault though, because this aspect of PTYs is very old.

   If not, splitting such a response [...] that do not arrive within the 10 ms patience time that tmux-3.5 allows before it concludes that this might have been a manual press of the ESC key and not a response sequence.

   To my knowledge this issue also occurs with Linux terminals when using tmux via high latency SSH. As mentioned above, we do write the DA1 response atomically, so is there really anything else we can do here? I don't think so.

   It is one wide string L"?61;4;6;7;14;21;22;23;24;28;32;42c", except for the initial ESC [, which all may have to go through a UTF-8 encoder, and thus might end up in separate write() system calls and separate TCP packets?

   That's not how our system works. Responses get accumulated into the ControlCore::_pendingResponses buffer and then flushed out all at once here:
   

   terminal/src/cascadia/TerminalControl/ControlCore.cpp

   Lines 2174 to 2178 in 52262b0

   	if (!_pendingResponses.empty())
   	{
   	_sendInputToConnection(_pendingResponses);
   	_pendingResponses.clear();
   	}

   ---

   To my understanding there are two issues involved here:

   • OpenSSH used to do a bad job at reading stdin, which was "fixed" by Increase stdin chunksize from 16 to 1024 chars PowerShell/openssh-portable#771
     (Buffer size should be even higher, but the architecture is not amendable to such changes)
   • ConPTY (= our PTY) had a race condition: Fix a major stdin wakeup race condition #18816

   So, try using the latest Windows Terminal releases from yesterday and see if the issue still occurs with output such as 0;10;1c.

   ❤️React with ❤️3Reacted by Michael Weiss, Markus Kuhn and Radosław Piliszek
6. primeos-work commented on Aug 27, 2025

   

   Michael Weiss (primeos-work)

   on Aug 27, 2025

   Author

   More actions

   Huge thanks Leonard Hecker (@lhecker) for your reply/analysis! ❤️

   Many applications don't do this properly, including application you may assume to be of high quality.

   Yeah, I'm unfortunately not familiar with the standards but I'd also consider it a tmux issue as well. It should luckily be fixed with tmux/tmux@817b621 now but it'll take a long time until that's available on most SSH hosts.

   To my knowledge this issue also occurs with Linux terminals when using tmux via high latency SSH.

   Ok, so you've also experienced that issue on Linux systems?
   I don't remember ever running into that issue from my Linux laptops but it's certainly possible - I guess it depends on the terminal emulator (even though it's not necessarily a terminal emulator bug but it looks like terminal emulators can at least do a better or worse job at preventing it by ensuring the response ends up in a single network packet or at least using large enough buffer sizes - I'm not familiar at all with the involved details though).

   As mentioned above, we do write the DA1 response atomically, so is there really anything else we can do here? I don't think so.

   Ok, I was hoping there would be a way to significantly improve the chances of the response ending up in a single network packet but we can close this issue then if there's no reasonable improvement left.

   That's not how our system works. Responses get accumulated into the ControlCore::_pendingResponses buffer and then flushed out all at once here:

   Ok, thanks. If I followed the calls correctly it'll end up here?:

   terminal/src/terminal/adapter/InteractDispatch.cpp

   Lines 30 to 42 in 91c9a14

   	// Method Description:
   	// - Writes a collection of input to the host. The new input is appended to the
   	// end of the input buffer.
   	// If Ctrl+C is written with this function, it will not trigger a Ctrl-C
   	// interrupt in the client, but instead write a Ctrl+C to the input buffer
   	// to be read by the client.
   	// Arguments:
   	// - inputEvents: a collection of IInputEvents
   	void InteractDispatch::WriteInput(const std::span<const INPUT_RECORD>& inputEvents)
   	{
   	const auto& gci = ServiceLocator::LocateGlobals().getConsoleInformation();
   	gci.GetActiveInputBuffer()->Write(inputEvents);
   	}

   I guess one last thing that could theoretically be done to avoid this issue / improve the situation could be to let the active input buffer fully drain before appending such "atomic" escape sequences (or some alternative means to ensure the whole response ends up in a single 1024 bytes buffer) but I have no idea how well that would fit into the code and it would obviously have the drawback of slowing things down by not utilizing the full buffer size in those cases and (significantly) increasing complexity so it's most likely not worth it...

   I guess in the end it really is a tmux issue/problem (I also couldn't reproduce it with GNU Screen when I tried it just now but I didn't check if they also issue a DA request but I guess it's very likely the case) as Windows Terminal probably doesn't violate any standards here and I just incorrectly blamed it on Windows Terminal / Win32-OpenSSH as I didn't hit it with other setups (Linux systems/terminals). Sorry about that! (There are already quite a few reports though as mainly Windows Terminal users seem to run into this issue. I guess we at least have it documented/analyzed quite extensively now...)

   (Buffer size should be even higher, but the architecture is not amendable to such changes)

   Ok, that's a bit of a bummer but fair. With a higher buffer size the issue might already be gone almost entirely.

   So, try using the latest Windows Terminal releases from yesterday and see if the issue still occurs with output such as 0;10;1c.

   I'll give it a go but unfortunately I doubt it'll improve things as apparently your fix already got backported to version 1.22 and ea0ba56 is in version 1.22.12111.0 that I used for the fresh tests.
7. Michael Weiss (primeos-work)

   closed this as completedon Aug 27, 2025
8. lhecker commented on Aug 29, 2025

   

   Leonard Hecker (lhecker)

   on Aug 29, 2025

   Last edited by lhecker

   Member

   More actions

   Ok, so you've also experienced that issue on Linux systems?

   Not really. I've generally only administered servers that are either in a local DC or in my home. But I know it's an issue, which is why the escape-time setting exists at all.

   Ok, thanks. If I followed the calls correctly it'll end up here?:

   Sort of. That's the PTY side of things where input is written into the aforementioned ring buffer. The architecture is bad, so don't take the following as "it should be like this" but more like "it's currently like this":

   • Your app write DA1 into stdout (a pipe)
   • The PTY (aka ConPTY aka OpenConsole.exe aka conhost.exe) reads from stdout and
     • before v1.22:
       • Parses stdout into VT sequences
       • Reassembles them back into VT output
       • Sends it to the terminal (another pipe)
     • since v1.22:
       • Sends it to the terminal (another pipe)
   • The terminal reads from the PTY and
     • Parses the VT sequences
     • Sees the DA1
     • Writes the response into _pendingResponses
   • Now it writes _pendingResponses back to the PTY (a pipe)
   • The PTY reads from stdin (terminal) and
     • Parses stdin into VT sequences (like above before v1.22 == bad, needs a refactor)
     • Writes the input events into a "ring buffer" / queue
     • Signals availability of input to your app's stdin pipe
       • This is where Fix a major stdin wakeup race condition #18816 happened
   • Your app gets woken up because stdin is available and
     • Reads stdin from the PTY ring buffer

   I guess one last thing that could theoretically be done to avoid this issue / improve the situation could be to let the active input buffer fully drain before appending such "atomic" escape sequences

   I'd be against that. Applications should implement better VT parsers (which are robust against this), and terminals should implement the kitty keyboard protocol (including us; it turns Esc keys into the unmistakable sequence \x1b[27u). This fixes the issue from both sides IMO. Particularly the latter is a good fix, because your suggestion and any other current workaround rely on heuristics which is not great.

   There are already quite a few reports though as mainly Windows Terminal users seem to run into this issue.

   Windows Terminal has had (and still has) various major bugs with its VT handling, so the attribution is totally fair IMO. Most notably, before #17510 (= before v1.22) we would mangle VT sequences in stdout. We were the only PTY and terminal to do this, and it resulted in lots of issues. We also still somewhat mangle stdin but that's unrelated to this issue. I'm also intending to fix that in the near term.

   I'll give it a go but unfortunately I doubt it'll improve things as apparently your fix already got backported to version 1.22 and ea0ba56 is in version 1.22.12111.0 that I used for the fresh tests.

   Hmm that's weird then... Your second example looks like just the Esc character was mistaken for an Esc key so it printed the whole DA1 response to your prompt. But the first example looked exactly like how #18816 presented itself. We may have to investigate this, although I'm not sure how. (It doesn't happen to me so I can't debug it. I also could only figure out #18816 after I was able to reproduce it in wezterm.)