see: #5400, {Add action to bring up a Cmd Palette with just sendInput actions}, and to a lesser extent #3121. Also #3158, #8166 et. al.

Okay, so the shell can tell the Terminal what the CWDs are. Terminal should store those in a queue, and then provide a way to hook them up to a cmdpal filled with them as sendInput actions. So hit enter on one of them in the cmdpal and presto, cd path/to/place/i/was/before↩ gets sent to the terminal to CD back to that directory.

This assumes you're in a context where cd some\path will work. Obviously, vim, emacs wouldn't work. Something like ssh gets tricky I suppose.

Maybe it doesn't work in the alt buffer, since a shell probably isn't active there?

iTerm2 has something like this. I think they use a side panel, but whatever. Docs: https://iterm2.com/documentation-shell-integration.html, under "Recent Directories"