Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.39%. Comparing base (b2871e0) to head (7ef47ef).
⚠️ Report is 1 commits behind head on master.

@@           Coverage Diff           @@
##           master   #15909   +/-   ##
=======================================
  Coverage   98.39%   98.39%           
=======================================
  Files         171      171           
  Lines       22274    22274           
=======================================
  Hits        21916    21916           
  Misses        358      358           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

I've checked a few boards & ports with that test. Most work fine. SOme minor observations:

• nrf: Sometimes fails. When the test is repeated, it works.
• ESP32S3 at the CDC interface: Lots of timeouts. The UArt interface works fine.
• ESP32C3 at the CDC interface: Data IN runs at 0.09MBit/s, Data OUT at 0.66/~3 MBit/s. That's odd.
• ESP32S2: Locks often up in the DATA OUT, verify=1 sequence at larger buffer sizes
• STM32F411 Blackpill: Locks up DATA OUT, verify=1 sequence at larger buffer sizes

I've just tested on a esp32-S3 CDC with my shared/tinyusb migration PR: #15108
It works correctly without timeouts (@dpgeorge reported similar results previously on that PR)
DATA IN: 0.09 MBits/sec / DATA OUT: verify=0, bufsize=9999 1.14 MBits/sec

@robert-hh I've seen the same 0.09MBit/s DATA IN rates on renesas board, not sure why they're so low.

I also get lock-ups with the black pill board. I suspect it might be due to the 25mhz crystal causing instability - that is said to be the reason the built in DFU struggles to connect to usb more often than not.

On other stm32 boards though, I get much better speeds. WB55 for instance:

DATA IN: bufsize=256, read 32768 bytes in 63.21 msec = 506.22 kibytes/sec = 4.15 MBits/sec
DATA OUT: verify=0, bufsize=9999, wrote 639936 bytes in 1286.46 msec = 485.78 kibytes/sec = 3.98 MBits/sec

However for this I needed to increase the delay before the initial drain as the soft reboot took a little longer to complete.

def drain_input(ser):
    time.sleep(0.5)  # <---
    while ser.inWaiting() > 0:
        data = ser.read(ser.inWaiting())
        time.sleep(0.1)

rp2 (tinyusb)

DATA IN: bufsize=256, read 32768 bytes in 67.66 msec = 472.96 kibytes/sec = 3.87 MBits/sec
DATA OUT: verify=0, bufsize=9999, wrote 639936 bytes in 4998.38 msec = 125.03 kibytes/sec = 1.02 MBits/sec

Ah... some of the slow data rates are caused by UART!
On renesas-ra, while I'm using the USB port only, the stdout goes to all ports and uart is a blocking write. Disabling uart repl raised the DATA IN rates at least from 0.09 MBits/sec up to 3-5 MBits/sec. The DATA OUT rates are still slow though.

Ah... some of the slow data rates are caused by UART!

Oh, yes, that's bitten me a few times. You need to disable UART if you want to test USB throughput.

You could also use this tool to test UART reliability, if you connect up a USB-UART convertor.

I just noted some very interesting results testing a PICO2 on the DATA IN test for #17960.

So I think we're inadvertantly (and probably uncontrollably) benchmarking a few things here:

• Host Python performance (both these computers have CPython 3.13).
• Kernel I/O performance (time says none of these tests are running on CPU more than 250ms for the whole ~30 second run, about 85% user and 15% system).
• USB link quality (bad cable, noise issues, etc.)
• USB controller chipset quirks (I suspect that's the only thing that can explain my one very slow result). As I understand it, USB 3.x hubs implement an entire USB 2.0 controller inside them (and then encapsulate downstream packets as USB 3.x Superspeed), so plugging a device like a dev board into a USB 3.x hub depends a lot on the hub's controller.

Still seems like a good test, I think the thing to keep in mind is that it's only good for comparing between runs on the exact same hardware (even down to the port choice). 🤷

Interesting data! And interesting you got pretty much the same result as me for your laptop. For reference my laptop is an i7-8550U.

Apart from performance, this test is also useful to test link reliability. If a port can pass this test then it's REPL is most likely quite stable.

I've now updated this PR:

• renamed to tests/serial_test.py
• used argparse instead of hand crafting the CLI parsing
• used the same --test-instance argument as all the other tests to select the target device
• removed unused code
• added a small bit of documentation to tests/README.md

It's now ready to merge.

Note: the script now depends on tests/run-tests.py to get the helper function for the --test-instance argument. That means it's no longer a stand-alone test. Hopefully that's OK.