The FormatStringContinuation implementation uses the full lexer for comment handling. It expects only a limited set of tokens (String, Newline, NonLogicalNewline, Comment, Indent, Dedent) which is expected given that the lexing is performed only on the string literals.

Now, with PEP 701, the f-string literal will instead emit FStringStart, FStringMiddle, and FStringEnd. Not only that, it'll also emit other tokens for the expression part of the f-string. This poses a challenge as currently it panics for any unexpected token encountered: