Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (d160759) to head (a43e385).
⚠️ Report is 5 commits behind head on master.

@@           Coverage Diff           @@
##           master   #17960   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22296    22296           
=======================================
  Hits        21937    21937           
  Misses        359      359           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +8 +0.001% RPI_PICO_W
       samd:    +8 +0.003% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@projectgus I built 6a61356, and will test when I have access to the boards tomorrow.

FWIW: my steps were

• checking out 6a61356
• cd ports/esp32
• make submodules
• make BOARD=...

If there is more to it, please kindly let me know, as I am not yet very familiar with the build process.

However, this still doesn't provide any tracking of when the ESP-IDF version has changed from whatever the current recommended version is... [...] Paying more attention to exact component versions means from now that we'll need to also pay attention to updating components (and the lock file) when it's relevant to do so.

Keep in mind that it's not an uncommon occurrence for Espressif to yank published component versions. That can make pro and con arguments both for keeping track of versions manually and for letting ESP-IDF resolve versions on its own so I won't weigh on this :)

Keep in mind that it's not an uncommon occurrence for Espressif to yank published component versions. That can make pro and con arguments both for keeping track of versions manually and for letting ESP-IDF resolve versions on its own

Although I agree there's pros and cons, I still think explicitly tracking dependencies is a win for this kind of scenario. We don't currently have any way to tell if someone is building with a local yanked component when they submit a bug report, but with this change they'd have to have explicitly chosen to do that.

Also, if we're building with a yanked component and this PR then CI should fail and we get the hint to go and update it.

The more complex case, as I see it, is that we won't automatically build MicroPython with bugfix releases of components, without manually running idf.py -B [...] update-components. In this aspect I agree it's a mixed bag, things won't quietly break but they won't quietly fix themselves either...

so I won't weigh on this :)

Overall I am very interested in your thoughts on this approach, both for this specific TinyUSB problem and generally - so feel free to weigh in if you think it's a step in the wrong direction.

If there is more to it, please kindly let me know, as I am not yet very familiar with the build process.

Thanks @gohai, those are the steps! To flash in the same step you can do make deploy BOARD=... PORT=... (more info in the README). Look forward to hearing if it changes anything (or not).

Although the lock files add complexity (at least, in the sense of adding more files) it sounds like it's worth it to keep proper track of which component versions are in use.

@projectgus This works fine on my USB-OTG-connected ESP32-S3-Zero. (This one had issues copying files earlier.)

However, I am still seeing reproducible repl hangs on the ESP32S3-S3-DEV-KIT-NXR8. See e.g. below (just starting repl made it hang):

@gohai So you get this immediately from simply running mpremote?

@projectgus I'd say I don't have a preference on how the tracking is performed, as long as progress is made :)

I think it all boils down on how much of a maintenance burden it can be to track versions manually. From my past experiences I can sum it all up as this (virtually none of these can be MicroPython's concern, I'm dumping them here in the hope they may help make a more informed decision): how often dependencies need to be updated, how much time it takes to validate those changes, and what's the turnaround time to get an emergency fix from Espressif into MicroPython (eg. potential security implications, that's yet another Pandora's box that should better stay shut) or how quickly a yanked component can be gotten rid of in the published repo lockfiles.

For emergency changes, if people in charge for releasing the Espressif components you rely on are based out of Shanghai, then you folks being a couple hours ahead of them is not a big deal, otherwise you may want to keep that in mind too.

Finally, I wonder if this approach is compatible with https://github.com/agatti/micropython-idf-component. Granted, it's up to me to work around your choices in this case if the end result is not compatible so I'm just talking out loud here.

Incidentally, @dpgeorge if you ever plan to release MicroPython as an ESP-IDF component and put it up into Espressif's registry, feel free to take as much code/ideas/whatnot from that repo. I've licensed that as MIT in order to make the process easier on your end if an official component is an option you want to consider.

However, I am still seeing reproducible repl hangs on the ESP32S3-S3-DEV-KIT-NXR8. See e.g. below (just starting repl made it hang):

Just to be extra sure, does that board with the official 1.26 release firmware and with a firmware with updated IDF components work if you connect to it from Linux, Windows, or a virtualised Linux (so the full USB device is passed through)?

@gohai So you get this immediately from simply running mpremote?

@projectgus It behaves identical to what I saw on 1.26 with that board... it sometimes hangs at startup, but if it does, running help() once or twice will definitely freeze it.

Failures/crashes were seen here too, but with some differences. I applied this PR to MicroPython v1.27.0-preview.61.g6a613568a on 2025-08-22; Generic ESP32S3 module with Octal-SPIRAM with ESP32S3

Tests were done with two ESP32-S3 N16R8 modules, one with an empty file system and the other with the file system 90% full.

The test consisted in copying a 10 kbyte binary file with mpremote to the microcontroller, always with a different target filename.

With the microcontroller with empty file system, I see 3 crashes in 200 attempts. On the USB COM port, I can see many queue_event 357: ASSERT FAILED messages and a reboot with Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1, similar as before. One of 500 file copies has the wrong length, i.e. the resulting file is 2048 bytes instead of 10 kbytes.

With the ESP32-S3 with a full file system and I see 5 failures every 10 attempts (50%) with the same reboot/guru meditation error, or with this mpremotemessage:

mpremote: Error with transport:
timeout waiting for first EOF reception

This might account for differences of behaviour between tests or boards. The full file system makes the file access slower, i.e. more flash reads/writes for the same transfer, probably making the occurrence of an error more likely.

I can't reproduce the ESP32-S3 crash on the REPL. For example help() works always ok. However mpremote cat of a 10kbyte ASCII file crashes sometimes.

@agatti Apologies that this is taking longer than I had hoped for. I don't have a PC or Linux machine readily available, but I'll try to get my hands on it this Wednesday or Thursday.

@agatti I managed to grab a Windows 11 machine at school. Here my results with the same board (ESP32S3-S3-DEV-KIT-NXR8).

This is it freezing e.g. after a dir() or help():

Thanks @gohai, that was extremely helpful! I was able to reproduce the problem here, and I believe I understand the root cause. I don't yet have a good fix for it, but looking into it...

These USB-CDC issues seem to be a combination of a few things:

• In feat(host) add endpoint close API and feat(HIL) improvement  hathach/tinyusb#3043 a new flag is added to the CDC config: tx_overwritabe_if_not_connected (sic). This isn't part of TinyUSB 0.18.0 release but is part of the latest Espressif TinyUSB releases. This flag also isn't part of MicroPython's CDC config, so defaults to 0 and changes the CDC behaviour from "FIFO always overwrites old bytes" to "FIFO never overwrite old bytes".

  This means that CDC write can get stuck in this loop forever with n2 == 0 and nothing ever processing the USB task: https://github.com/micropython/micropython/blob/master/shared/tinyusb/mp_usbd_cdc.c#L124 - which is the hang that @gohai can reproduce.

• However, there's the different bug where writes randomly lose bytes (i.e. lines of files go missing). This is because of the old TinyUSB "FIFO always overwrites old bytes" logic: if stdout writes faster than the USB host can read then some bytes are dropped. Updating to newer Espressif TinyUSB does fix this!

So, the solution is obviously to set the tx_overwritabe_if_not_connected flag and everything will be OK, right? No, because TinyUSB uses "DTR is set" as the signal that the host is connected (and will read bytes from the USB-CDC buffer). mpremote has a specific workaround where it clears DTR when connected to an Espressif device, to avoid confusing the bootloader reset. So TinyUSB will never think the host is connected, and would still always overwrite the FIFO and lose bytes sometimes 🤦 .

So, we need a new set of behaviour where:

• mp_usbd_cdc_tx_strn should always call mp_usbd_task() to attempt to free space in the FIFO.
• mp_usbd_cdc_tx_strn should never block for too long if the host isn't reading from USB-CDC (currently the timeout is 500ms, we obviously can't pause each stdout write for 500ms if the host isn't actively reading data).
• The mpremote ESP reset logic remains intact.

Fix for the issue @gohai saw with Windows is submitted at #17999. I don't think that change will do anything for macOS or Linux hosts, though. 😬

I've pushed another fix here for the tx_overwritabe_if_not_connected flag that Espressif added in TinyUSB component v0.18.0~3 (see updated description for more details).

@gohai Is there any chance that you could please test the latest version of this PR on your original MacOS install again, please?

@projectgus I built e29b383 and will test tomorrow with the two boards on macOS and Windows 11. Would you prefer if I test with, or without the mpremote changes in #17999?

@projectgus I built e29b383 and will test tomorrow with the two boards on macOS and Windows 11. Would you prefer if I test with, or without the mpremote changes in #17999?

For MacOS, can test with either. For Windows, can test with either but you might see some dropped bytes if you don't have the mpremote changes included (but you shouldn't see a hang any more).

@projectgus e29b383 fixes all issues I saw on macOS and Windows 11 with the ESP32S3-S3-DEV-KIT-NXR8 and an (USB-OTG) ESP32-S3-Zero. (I haven't done too much testing at this point, but at least none of my reproducers fail anymore.)

Tested: repl and file copying on macOS; repl on Windows [edit: this was with mpremote 1.25.0, fwiw]

I applied the changes of this PR to MicroPython v1.27.0-preview.76.ge29b383fc on 2025-08-29; Generic ESP32S3 module with Octal-SPIRAM with ESP32S3. I am newbie with git commands. I did the following to apply the changes in this PR before compiling a fresh clone of MicroPython. Please tell me if this is correct:

% git fetch origin pull/17960/head:tinyusb_pr
% git checkout tinyusb_pr

The basic test is copying repeatedly a 10_000 byte binary file to the ESP32-S3 with mpremote on a MAC with OS 14.6.1 (Sonoma). On the average there was about one crash every 100 copy commands. On the USB/COM port the messages were the same as in previous versions, i.e. many queue_event 357: ASSERT FAILED and finally a Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1).

Should I also apply other changes before testing?

I'm new to micropython but did manage to test e29b383.

I'm on MacOS 15.5 on an M1 Macbook with a ESP32-S3-Zero.

• With Version 1.26, I consistently had corrupted file transfers. Even with small main.py files (293 bytes), I had parts of it repeated always at the same position after uploading with mpremote cp main.py :main.py.
• Version 1.25 doesn't seem to support the flash file system on S3-Zero
• e29b383 works well.

Rather than duplicate the lockfile check in both CI and autobuild.sh, I added it to esp32 port CMakeLists.txt. To distinguish between a warning (for normal developer builds) or a fatal error (for CI and nightly builds where it indicates an ESP-IDF version mismatch), I've added an environment variable MICROPY_MAINTAINER_BUILD that's set in both CI and autobuild.sh.

We might be able to lean on this flag for other things as well (for example, disabling -Werror in normal builds to avoid breakage due to new compiler versions).

Thank you, when I run ``git describe --tags --dirty```, I get the correct result.

IDK if this is relevant, but when I apply the complete PR, the ESP32S3 crashes about once every 50 or 100 mpremote cp commands on a Mac.

However, when I apply the commit e29b383 only, I see no error.

IDK if this is relevant, but when I apply the complete PR, the ESP32S3 crashes about once every 50 or 100 mpremote cp commands on a Mac.

However, when I apply the commit e29b383 only, I see no error.

That's quite unexpected! Can you please copy and paste the contents of file ports/esp32/dependencies.lock on the build where you don't see any error?

I could not reproduce a version that does not fail. I did:

% git checkout master
% git fetch origin e29b383fc95fc3428f9eedbe98a431ff307a0185
% git checkout e29b383fc95fc3428f9eedbe98a431ff307a0185
% git describe --tags --dirty
v1.27.0-preview-76-ge29b383fc
% git branch
* (HEAD detached at e29b383fc)
  master
  tinyusb_pr

After recompiling MicroPython mpremote cp on the Mac crashes the microcontroller.

MicroPython version reported as MicroPython v1.27.0-preview.76.ge29b383fc on 2025-09-04; Generic ESP32S3 module with Octal-SPIRAM with ESP32S3

File ports/esp32/lockfiles/dependiencies.lock.esp32s3 contains:

dependencies:
  espressif/esp_tinyusb:
    component_hash: 96d232ced7afe1976119b62f7fbf1944a2a78b36228ff6f7b9318394ac1153cc
    dependencies:
    - name: idf
      require: private
      version: '>=5.0'
    - name: espressif/tinyusb
      registry_url: https://components.espressif.com
      require: public
      version: '>=0.14.2'
    source:
      registry_url: https://components.espressif.com/
      type: service
    version: 1.7.6~1
  espressif/mdns:
    component_hash: 46ee81d32fbf850462d8af1e83303389602f6a6a9eddd2a55104cb4c063858ed
    dependencies:
    - name: idf
      require: private
      version: '>=5.0'
    source:
      registry_url: https://components.espressif.com/
      type: service
    version: 1.1.0
  espressif/tinyusb:
    component_hash: aa65639878f27a44d349044afd9c3fc134a92bd560874fdac1d836019b5c07ca
    dependencies:
    - name: idf
      require: private
      version: '>=5.0'
    source:
      registry_url: https://components.espressif.com
      type: service
    targets:
    - esp32s2
    - esp32s3
    - esp32p4
    version: 0.18.0~4
  idf:
    source:
      type: idf
    version: 5.4.2
direct_dependencies:
- espressif/esp_tinyusb
- espressif/mdns
- idf
manifest_hash: 3b18b5bbac91c9fe5098d3759a37c84ed0828546d8cbc92e26e4c1698e689c8a
target: esp32s3
version: 2.0.0

This is not the first time I get inconsistent results. Probably I don't know enough about git commands.... :-(

This is not the first time I get inconsistent results. Probably I don't know enough about git commands.... :-(

@bixb922 Thanks for the extra info, the steps look right to me. I don't think this is necessarily your fault, these kind of issues have a lot of moving parts and can be hard to reproduce reliably.

I think we should still merge this PR, as it's confirmed to fix some of the issues reported and doesn't seem to introduce any new ones. Will continue to look into ways to debug the remaining issue on MacOS...

Thanks @projectgus for working on this, and everyone who tested it.

I did some basic testing of ESP32_GENERIC_S3 on my Arch Linux set up, and it works fine.

But, I also tested RPI_PICO2 and there is a regression in the USB CDC performance 😞 That's due to the subtle change in behaviour of mp_usbd_cdc_tx_strn().

I tested using #15909 and find that the "DATA IN" (in to the host PC) rate drops from about 586 kibytes/second down to about 540 kibytes/second. It's definitely noticeable using that test.

Doing some quick hacking, I can regain the performance with the following change:

--- a/shared/tinyusb/mp_usbd_cdc.c
+++ b/shared/tinyusb/mp_usbd_cdc.c
@@ -102,6 +102,10 @@ mp_uint_t mp_usbd_cdc_tx_strn(const char *str, mp_uint_t len) {
     size_t i = 0;
     while (i < len && (mp_uint_t)(mp_hal_ticks_ms() - last_write) < MICROPY_HW_USB_CDC_TX_TIMEOUT) {
         uint32_t n = len - i;
+        if (n > CFG_TUD_CDC_EP_BUFSIZE) {
+            n = CFG_TUD_CDC_EP_BUFSIZE;
+            mp_event_wait_ms(1);
+        }
 
         if (tud_cdc_connected()) {
             // Limit write to available space in tx buffer when connected.

I also get back the performance by making that new mp_event_wait_ms(1) unconditional, ie not wrapped in that if.

I suspect waiting a bit before attempting to write to the CDC lets it drain more in certain situations.

@projectgus what was the main reason you changed that function? Maybe we can just leave it as it was?

what was the main reason you changed that function? Maybe we can just leave it as it was?

The reason I changed it was to make two things unconditional each time around the loop:

• Callingmp_usbd_task()
• Evaluating the timeout

This isn't strictly necessary, but I thought it makes it less likely that another transient state mismatch leads to an infinite loop (I don't see a sequence of events in the old version of the code that could lead to that now the FIFO issue is patched, but it's hard to guess at every possible sequence of USB state changes.)

I'm not surprised there's a small speed regression, but I am a bit surprised at the patch you posted that reverses it. If it's OK with you then I'll have a further try at tweaking the new loop to fix the speed issue, but otherwise it should be safe to take the change out entirely.

If it's OK with you then I'll have a further try at tweaking the new loop to fix the speed issue

Yes, that's fine. Hopefully you can reproduce the performance measurements using #15909.

Hmm, I wasn't really able to reproduce those findings. I only see a few kibytes/sec variation between different versions (including the two patches mentioned above), it seems random enough to possibly be cache layout.

I did make a change that seems like it can't hurt (evaluating the timeout after writing to the FIFO), but the different speeds I was getting were all very similar. Here's my most recent run:

DATA IN: bufsize=256, read 32768 bytes in 48.64 msec = 657.87 kibytes/sec = 5.39 MBits/sec
DATA IN: bufsize=512, read 32768 bytes in 50.60 msec = 632.47 kibytes/sec = 5.18 MBits/sec
DATA IN: bufsize=1024, read 65536 bytes in 100.55 msec = 636.48 kibytes/sec = 5.21 MBits/sec
DATA IN: bufsize=2048, read 131072 bytes in 201.19 msec = 636.23 kibytes/sec = 5.21 MBits/sec
DATA IN: bufsize=4096, read 262144 bytes in 401.70 msec = 637.30 kibytes/sec = 5.22 MBits/sec
DATA IN: bufsize=8192, read 524288 bytes in 803.07 msec = 637.55 kibytes/sec = 5.22 MBits/sec
DATA IN: bufsize=16384, read 1048576 bytes in 1605.90 msec = 637.65 kibytes/sec = 5.22 MBits/sec

(The runs I did from the current master branch are +/-2 kibytes/sec of this.)

I've retested with a different set up: I had two USB hubs in the way (one was Octoprobe) and have now removed those. So now the RPI_PICO2 is connected directly to a root_hub.

Testing with master, this PR before your recent change, and with the most recent change (titled "Possible performance tweak") I get pretty much the same performance in all those three cases:

DATA IN: bufsize=256, read 32768 bytes in 52.67 msec = 607.52 kibytes/sec = 4.98 MBits/sec
DATA IN: bufsize=512, read 32768 bytes in 54.84 msec = 583.48 kibytes/sec = 4.78 MBits/sec
DATA IN: bufsize=1024, read 65536 bytes in 109.14 msec = 586.39 kibytes/sec = 4.80 MBits/sec
DATA IN: bufsize=2048, read 131072 bytes in 218.20 msec = 586.61 kibytes/sec = 4.81 MBits/sec
DATA IN: bufsize=4096, read 262144 bytes in 437.54 msec = 585.08 kibytes/sec = 4.79 MBits/sec
DATA IN: bufsize=8192, read 524288 bytes in 874.51 msec = 585.47 kibytes/sec = 4.80 MBits/sec
DATA IN: bufsize=16384, read 1048576 bytes in 1747.22 msec = 586.07 kibytes/sec = 4.80 MBits/sec

So... I think we are fine with this.

(It's interesting you get quite a bit faster rates, I guess it's due to a faster PC. Although that surprises me, the limiting factor should not be the PC!)

Do you want to squash down your latest commit? It looks good to me.

(Actually, maybe split the changes to shared/tinyusb/mp_usbd_cdc.c into the .tx_overwritabe_if_not_connected fix, and then a separate commit for the mp_usbd_cdc_tx_strn changes??)

(It's interesting you get quite a bit faster rates, I guess it's due to a faster PC. Although that surprises me, the limiting factor should not be the PC!)

Yeah, very surprising. I did some more tests and got even odder results, have left a comment at #15909 (comment)

(Actually, maybe split the changes to shared/tinyusb/mp_usbd_cdc.c into the .tx_overwritabe_if_not_connected fix, and then a separate commit for the mp_usbd_cdc_tx_strn changes??)

Good idea, done! Left some rationale in the commit message for future reference as well.