I can't get the macOS tests to pass without drastically increasing the timings, mac has very poor timing resolution (>100ms variation). But otherwise this looks reliable for Windows and Linux, the timings have been increased by about 5x.

Ok on Windows port on my machine

Perhaps we could use nice to set the tests the highest priority (-20) for the micropython process when running tests? I've done this successfully with MicroPython on embedded Linux to get more precise timing. It isn't 100% foolproof but makes a measurable difference. I assume that macOS has nice as well. I'm not sure what the Windows equivalent would be.

(Not relevant to this issue, but for anyone wanting to apply this to a different application with a single CPU, be aware that this will effectively make every other process run very slow if the MicroPython process is CPU bound, i.e. while True: pass.)

Ok on Windows port on my machine

Thanks for testing.

Perhaps we could use nice to set the tests the highest priority (-20) for the micropython process when running tests?

I've now tried this and it doesn't help the macOS tests, they still have a few hundred ms variation.

Codecov Report

Merging #8617 (bc44b8a) into master (0e7bfc8) will not change coverage.
The diff coverage is n/a.

❗ Current head bc44b8a differs from pull request most recent head c90f097. Consider uploading reports for the commit c90f097 to get more accurate results

@@           Coverage Diff           @@
##           master    #8617   +/-   ##
=======================================
  Coverage   98.22%   98.22%           
=======================================
  Files         154      154           
  Lines       20313    20313           
=======================================
  Hits        19953    19953           
  Misses        360      360           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e7bfc8...c90f097. Read the comment docs.

I've now tried this and it doesn't help the macOS tests, they still have a few hundred ms variation.

Bummer. I did some digging at it appears that nice does nothing on macos. 1 2 3 I also confirmed this with my own measurements (a simple C program with usleep).

It appears that to get accurate timing on macos we need to make threads realtime threads: https://developer.apple.com/library/archive/technotes/tn2169/_index.html

I tried the example code from the tech note and it does make a difference.