For safety reasons, we have a sync before starting jobs on non-default streams (i.e., streams with index > 0) in MFIter. But the previous approach of delaying the sync until necessary is actually slower than performing a sync on stream 0 (i.e., the default stream in amrex, not to be confused with CUDA's legacy default stream) at the beginning. This is because stream sync is relatively cheap when there are no active jobs in that stream. For example, the sequence of sync, kernel, kernel, and sync is faster than the sequence of kernel, sync, kernel, and sync, where there are no previous active jobs.

I have done some testing on perlmutter and frontier, the new way is about 5-10% faster for the following code.

BoxArray ba(Box(IntVect(0), IntVect(255)));
ba.maxSize(128);
MultiFab mf(ba, DistributionMapping{ba}, 1, 0);
mf.setVal(1);
auto t0 = amrex::second;
for (int i = 0; i < 10; ++i)
{
    mf.setVal(t0);
}
auto t1 = amrex::second();