igc driver not working on Pi5 #6134

Describe the bug

It appears that with an Intel i225 and i226 NIC in the Pi5 both of which use the igc driver it is not possible to read any address in the ioremapped address space. I should mention that the Intel i210 which uses the igb driver works without a hitch (the architecture of the i210 and the i225/226 is remarkably similar).

In igc_probe the call to pci_resource_start and pci_resource_len return the correct addresses of the memory also observed in lspci. The call to ioremap returns a valid address, however all calls to igc_rd32 always without exception return 0x0. Similarly a direct call to readl in the remapped space returns 0 no matter the register.

For fun I have also tried to map the pci resource using the pci_iomap call used in the igb driver but it doesn't make a difference.

Loading the driver fails with error -13 (IGC_ERR_SWFW_SYNC) which is essentially the first time the driver needs to read a set bit in a read call to acquire the software/firmware semaphore in igc_get_hw_semaphore_i225.

The same device (225/226 with igc) works fine on a CM4.

I've set the dtparam pciex1 and I've tried using the dtoverlay pcie_32bit_dma but it does not make a difference and the latter is not required on the CM4.

#help :-)

Steps to reproduce the behaviour

(with a 225/226 nic in the PCIe slot)
modprobe igc

Device (s)

Raspberry Pi 5

System

Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, f19ee211ddafcae300827f953d143de92a5c6624, stage4

2024/04/20 11:53:30
Copyright (c) 2012 Broadcom
version d1744d21 (release) (embedded)

Linux pi5-pcie 6.6.28-v8-16k+ #2 SMP PREEMPT Sun Apr 28 08:30:38 BST 2024 aarch64 GNU/Linux

Logs

Probably not worth a lot, but for the purposes of giving insight into the call stack:

[12587.426063] Intel(R) 2.5G Ethernet Linux Driver
[12587.426069] Copyright(c) 2018 Intel Corporation.
[12587.426171] igc 0000:01:00.0: PCIe PTM not supported by PCIe bus/controller
[12587.426244] igc_probe (1): 0x1b00000000-0x100000
[12587.426246] igc_probe (2): 0xffffc00083000000
[12587.426247] igc_probe (3): 0x0-0x0
[12587.426250] igc_get_invariants_base(1): 0
[12587.426251] igc_get_invariants_base(2): 0
[12587.426252] igc_get_invariants_base(3): 0
[12587.426253] igc_rd32 (1): 0x10! (0x83000000 8 1)
[12587.426256] igc_rd32 (2): 0x0 (4)
[12587.426257] igc_get_invariants_base(4): 0
[12587.426258] igc_rd32 (1): 0x10! (0x83000000 8 1)
[12587.426260] igc_rd32 (2): 0x0 (4)
[12587.426261] igc_rd32 (1): 0x8! (0x83000000 8 1)
[12587.426263] igc_rd32 (2): 0x0 (4)
[12587.426265] igc_rd32 (1): 0x5820! (0x83000000 8 1)
[12587.426267] igc_rd32 (2): 0x0 (4)
[12587.426268] igc_phy_hw_reset(1): 0
[12587.426269] igc_acquire_phy_base(1): 2i
[12587.426270] igc_get_hw_semaphore_i225(1): 123
[12587.426272] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426274] igc_rd32 (2): 0x0 (4)
[12587.426275] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426277] igc_rd32 (2): 0x0 (4)
[12587.426278] igc_get_hw_semaphore_i225(2): 0
[12587.426279] igc_get_hw_semaphore_i225(5): 65
[12587.426280] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426282] igc_rd32 (2): 0x0 (4)
[12587.426283] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426285] igc_rd32 (2): 0x0 (4)
[12587.426336] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426342] igc_rd32 (2): 0x0 (4)
...
[12587.430035] igc_rd32 (2): 0x0 (4)
[12587.430086] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.430096] igc_rd32 (2): 0x0 (4)
[12587.430097] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.430099] igc_rd32 (2): 0x0 (4)
[12587.430150] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.430156] igc_rd32 (2): 0x0 (4)
[12587.430157] igc_get_hw_semaphore_i225(6): 123
[12587.430158] igc_acquire_swfw_sync_i225(1): -13
[12587.430159] igc_phy_hw_reset(2): -13
[12587.430160] igc_init_phy_params_base(1): -13
[12587.430161] igc_get_invariants_base(5): -13
[12587.430162] err = ei->get_invariants(hw): -13
[12587.430173] igc: probe of 0000:01:00.0 failed with error -13

lspci -vvvvvvvvvvv :
0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-LM (rev 04)
Subsystem: Intel Corporation Ethernet Controller I226-LM
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 37
Region 0: Memory at 1b00000000 (32-bit, non-prefetchable) [virtual] [size=1M]
Region 3: Memory at 1b00200000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at 1b00100000 [virtual] [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable- Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Device Serial Number f0-b2-b9-ff-ff-11-b3-b8
Capabilities: [1c0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1f0 v1] Precision Time Measurement
PTMCap: Requester:+ Responder:- Root:-
PTMClockGranularity: 4ns
PTMControl: Enabled:- RootSelected:-
PTMEffectiveGranularity: Unknown
Capabilities: [1e0 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=55us PortTPowerOnTime=70us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=55us LTR1.2_Threshold=131072ns
L1SubCtl2: T_PwrOn=70us
Kernel modules: igc

Additional context