Please follow this steps:

1. you make a normal Build+Upload of a blinky code: The first time it takes time, and it's normal since it's the first time
2. you make a a little modification in the code, then build : This time since it's an incremental build, it doesn't take too much time.
3. start a debug sessions: it takes time, and it's normal.
4. Stop debug session
5. Now upload again your code (even without any modification) and here is the problem ! : it recompile the whole code and it take much time again !!!! And for ESP-IDF it tooks 250 second for my PC just to build a simple blinky code !

And I think it's the same for PIO unit testing.

Suggestion:
I suggest the solution that PIO treats Debug builds, Realase builds and Unit test builds as different buids in different directories, in order to not recompiling the whole code each time. See by yourself