It'd be ideal to be able to not have to rely on timing but I also want to unblock flaky tests that others may be hitting on the PR and since this is a test we can always change it later if we are able to get a better way of testing what we want to test.

Yes I agree, we don't want to rely on the timing of the test class itself and typically running this test locally took < 10 seconds so I initially just picked 100 seconds for the token expiration, which on the other hand didn't seem enough for CI.
In the end, all we care about in this particular scenario is that there's only a single token left in the refresh scheduler's queue in order to make sure #7270 doesn't happen