This has caught me many times now. I'd xpanes onto all servers in my cluster (like 6), and then I go to zoom one of them to full screen, all panes exit. I learned to be careful and wait for a couple seconds after I press <leader> before I press z. However, that's not fool-proof since my laptop gets super slow and the gap I put in might not be enough. I think it might be interpreting my <leader>+z as <leader>+ctrl+z.

One thing to note is that I don't experience the same on my normal tmux panes. I do everything inside tmux and start xpanes from within it too.