First device scanned behind 4 port PCIE switch fails to initialize #6278

Describe the bug
The first device behind a 4 port PCIE switch fails to initialize properly with read errors. BAR assigned to the first device behind the switch will produce memory read errors. This happens regardless to what device is first. The first device will always fail to initialize. This may be specific to the ASMedia 2806 4 Port Gen 3 switch.

To reproduce
Place any device behind a 4 port PCIE switch (ASMedia 2806) and the first device will fail to initialize during boot.

Expected behaviour
All devices behind switch to initialize and be available after boot.

Actual behaviour
First device will fail to initialize.
In the case of the Radxa Dual 2.5G Router HAT the first device is the M.2 NVME slot if populated, 2nd is the FPC connector, 3rd device is the Realtek 8125 and 4th is another Realtek 8125.

If the M.2 is populated then it will fail with the following error.

[   11.731965] nvme nvme0: pci function 0000:03:00.0
[   11.736710] nvme 0000:03:00.0: enabling device (0000 -> 0002)
[   11.793991] probe of 0000:03:00.0 returned 19 after 62120 usecs

Both Realtek 8125's and if anything is connected to the FPC will be functional in this case.

If no M.2 is populated and nothing connected to the FPC then the first Realtek 8125 fails while the other is functional.

[    3.759676] r8169 0000:05:00.0: enabling device (0000 -> 0002)
[    3.819660] r8169 0000:05:00.0: error -EIO: PCI read failed
[    3.826310] r8169: probe of 0000:05:00.0 failed with error -5

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi?
  Pi5

• Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2024-07-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 48efb5fc5485fafdc9de8ad481eb5c09e1182656, stage2

• Which firmware version (vcgencmd version)?

2024/06/05 16:41:49 
Copyright (c) 2012 Broadcom
version 6fe0b091 (release) (embedded)

• Which kernel version (uname -a)? Linux raspberrypi 6.6.39-v8-16k+ raspberrypi/firmware#1 SMP PREEMPT Sun Jul 14 19:16:39 EDT 2024 aarch64 GNU/Linux

Logs
If applicable, add the relevant output from dmesg or similar.

Additional context
In both cases the first device is assigned the same beginning BAR
1st case above with M.2 first
[ 11.434525] pci 0000:03:00.0: BAR 0: assigned [mem 0x1b00000000-0x1b00003fff 64bit]
2nd case above where Realtek 8125 first
[ 11.197574] pci 0000:05:00.0: BAR 2: assigned [mem 0x1b00000000-0x1b0000ffff 64bit]

A workaround/hack was discovered by skipping this first region of memory when assigning BAR then all devices initialize successfully.
geerlingguy/raspberry-pi-pcie-devices#647 (comment)

Also when attempting to boot from NVME with boot order BOOT_ORDER=0xf416 and PCIE_PROBE=1 the following was logged from uart.

  1.25 RPi: BOOTSYS release VERSION:6fe0b091 DATE: 2024/06/05 5
  1.39 AON_RESET: 00000003 PM_RSTS 00001000
  1.50 part 00000000 reset_info 00000000
  1.53 PMIC reset-event 00000002 rtc 00000001 alarm 00000000 enabled 0
  1.59 uSD vo  
  1.79 Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 4  3.46 OTP boardrev d04170 bootrom a a
  3.47 Customer key hash0
  3.54 VC-JTAG unlocked
  4.87 RP1_BOOT chip ID: 0x20001927
  4.87 RP1_BOOT: fw size 25
  4.42 PCI2 init
  4.42 PCI2 reset
  4.87 PCIe scan 00001de4:00000001
  4.88 RP1_CHIP_INFO 200019206/05 TIME: 16:41:49
  4.97 BOOTMODE: 0x06 partition 0 build-ts4697771
  4.07 AON_RESET: 00000003 PM_RSTS 00001000
  4.11 PCIlots 64 intrs 4
  4.68 XHCI-STOP
  4.68 xHC1 ver: 272 HCS: 03000440 140000f1 07 03000440 140000f1 07ff000a HCC: 0240fe6d
  4.80 xHC1 ports 3 sscan 00001b21:00002806
  5.63 PCIe scan 00001c5c:0000174a
  5.67 Found device on bridge 3 with vid 1c5c enable 1
  5.71 PCIe  size 4 debug 26000404 rc -1
  5.56      CS 13000409
  5.58   
  5.64    SRCI 0000101b
  5.67    DEST 3be0b000
  5.69   DESTI 00001000
  5.71     LEN 00000000
  5.73 NEXT_CB 00000000