Multiple UEs data stopping suddenly #494

Activity

1. pirog-spb commented on Nov 24, 2023

   

   pirog-spb

   on Nov 24, 2023

   Collaborator

   More actions

   Hi @Sabreu

   Seems like a bug. I confirm. During deletion request prcessing FARs and QERs are released by wrong id.
2. pirog-spb commented on Nov 24, 2023

   

   pirog-spb

   on Nov 24, 2023

   Collaborator

   More actions

   @Sabreu FYI I've added a fix in the linked branch
3. pirog-spb

   self-assigned this

   on Nov 25, 2023
4. pirog-spb

   added

   bugSomething isn't workingSomething isn't working

   on Nov 25, 2023
5. Sabreu commented on Nov 27, 2023

   

   Sabreu

   on Nov 27, 2023

   Last edited by Sabreu

   Author

   More actions

   @pirog-spb

   cannot build anymore (was warning before I think, I have installed all updates so maybe build tools are newer or something):

   /home/14900k/git/eupf/cmd/ebpf/xdp/qer_program.c:39:16: error: call to undeclared function 'max'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
   __u64 ts = max(now - BURST, t_next);
   ^
   1 error generated.
   Error: can't execute clang: exit status 1

   Edit:
   Also getting this kind of error later:
   cmd/core/api.go:13:2: no required module provides package github.com/edgecomllc/eupf/cmd/docs; to add it:
   go get github.com/edgecomllc/eupf/cmd/docs

   $ go get github.com/edgecomllc/eupf/docs
   go: github.com/edgecomllc/eupf/docs: no matching versions for query "upgrade"

   Edit 2:
   I was able to build after copying from my own installation the docs folder...

   thanks
6. pirog-spb commented on Nov 27, 2023

   

   pirog-spb

   on Nov 27, 2023

   Collaborator

   More actions

   @Sabreu The old warning turned into an error. I've fixed and pushed it.

   Considering the second error. Please verify that you have installed swag utility. See build manual or Dockerfile commands for details.
7. Sabreu commented on Nov 27, 2023

   

   Sabreu

   on Nov 27, 2023

   Last edited by Sabreu

   Author

   More actions

   @pirog-spb

   I have installed swag earlier, I just pulled the new codes to different folder and tried to generate /cmd and build after that.

   EDIT:
   I tried to install swag also again and still seeing the same issues with cmd/docs
8. Sabreu commented on Nov 27, 2023

   

   Sabreu

   on Nov 27, 2023

   Author

   More actions

   @pirog-spb

   Correct FAR & QER deleted now and no data stopping. Thanks

   2023/11/27 10:16:07 INF Saving FAR info to session: 1, {Action:12 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:33819146 TransportLevelMarking:0}
   2023/11/27 10:16:07 DBG EBPF: Put FAR: internalId=9, qerInfo={Action:12 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:33819146 TransportLevelMarking:0}
   2023/11/27 10:16:07 INF WARN: No OuterHeaderCreation
   2023/11/27 10:16:07 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:33819146 TransportLevelMarking:0}
   2023/11/27 10:16:07 DBG EBPF: Put FAR: internalId=10, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:33819146 TransportLevelMarking:0}
   2023/11/27 10:16:07 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:4751 RemoteIP:50399754 LocalIP:33819146 TransportLevelMarking:0}
   2023/11/27 10:16:07 DBG EBPF: Put FAR: internalId=11, qerInfo={Action:2 OuterHeaderCreation:1 Teid:4751 RemoteIP:50399754 LocalIP:33819146 TransportLevelMarking:0}
   2023/11/27 10:16:07 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:1410065408 MaxBitrateDL:1410065408 StartUL:0 StartDL:0}
   2023/11/27 10:16:07 DBG EBPF: Put QER: internalId=3, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:1410065408 MaxBitrateDL:1410065408 StartUL:0 StartDL:0}
   2023/11/27 10:16:07 DBG EBPF: Put PDR Downlink: ipv4=10.25.0.216, pdrInfo={OuterHeaderRemoval:0 FarId:9 QerId:3 SdfFilter:<nil>}
   2023/11/27 10:16:07 DBG EBPF: Put PDR Uplink: teid=255776, pdrInfo={OuterHeaderRemoval:0 FarId:10 QerId:3 SdfFilter:<nil>}
   2023/11/27 10:16:07 DBG EBPF: Put PDR Uplink: teid=164640, pdrInfo={OuterHeaderRemoval:0 FarId:9 QerId:3 SdfFilter:<nil>}
   2023/11/27 10:16:07 DBG Error extracting PDR info: SDF Filter: bad formatting. Should be compatible with regexp: ^permit out (icmp|ip|tcp|udp) from (any|[\d.]+|[\da-fA-F:]+)(?:/(\d+))?(?: (\d+|\d+-\d+))? to ([\d.]+|[\da-fA-F:]+)(?:/(\d+))?(?: (\d+|\d+-\d+))?$
   2023/11/27 10:16:07 INF Session Establishment Request from 10.10.1.3 accepted.
   2023/11/27 10:16:07 DBG Received 69 bytes from 10.10.1.3:8805
   2023/11/27 10:16:07 DBG Handling PFCP message from 10.10.1.3:8805
   2023/11/27 10:16:07 INF Got Session Modification Request from: 10.10.1.3.
   
   2023/11/27 10:16:07 INF Finding association for 10.10.1.3
   2023/11/27 10:16:07 INF Finding session 5
   2023/11/27 10:16:07 INF
   Session Modification Request:
     UpdateFAR ID: 1
       Apply Action: [2 0]
       Update forwarding Parameters:
         Network Instance: 5gapn3
         Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:3221618979 IPv4Address:10.10.10.41 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
   
   2023/11/27 10:16:07 INF Updating FAR info: 1, {FarInfo:{Action:2 OuterHeaderCreation:1 Teid:3221618979 RemoteIP:688523786 LocalIP:33819146 TransportLevelMarking:0} GlobalId:9}
   2023/11/27 10:16:07 DBG EBPF: Update FAR: internalId=9, farInfo={Action:2 OuterHeaderCreation:1 Teid:3221618979 RemoteIP:688523786 LocalIP:33819146 TransportLevelMarking:0}
   2023/11/27 10:16:12 DBG Received 16 bytes from 10.10.1.3:8805
   2023/11/27 10:16:12 DBG Handling PFCP message from 10.10.1.3:8805
   2023/11/27 10:16:12 DBG Got Heartbeat Request with TS: 2023-11-24 13:15:18 +0200 EET, from: 10.10.1.3
   2023/11/27 10:16:12 DBG Sent Heartbeat Response to: 10.10.1.3
   [GIN] 2023/11/27 - 10:16:12 | 200 |    1.910357ms |       10.10.5.1 | GET      "/metrics"
   2023/11/27 10:16:17 DBG Sent Heartbeat Request to: 10.10.1.3
   2023/11/27 10:16:17 DBG Received 16 bytes from 10.10.1.3:8805
   2023/11/27 10:16:17 DBG Handling PFCP message from 10.10.1.3:8805
   2023/11/27 10:16:17 DBG Got Heartbeat Response with TS: 2023-11-24 13:15:18 +0200 EET, from: 10.10.1.3
   [GIN] 2023/11/27 - 10:16:22 | 200 |    1.778472ms |       10.10.5.1 | GET      "/metrics"
   2023/11/27 10:16:23 DBG Received 16 bytes from 10.10.1.3:8805
   2023/11/27 10:16:23 DBG Handling PFCP message from 10.10.1.3:8805
   2023/11/27 10:16:23 DBG Got Heartbeat Request with TS: 2023-11-24 13:15:18 +0200 EET, from: 10.10.1.3
   2023/11/27 10:16:23 DBG Sent Heartbeat Response to: 10.10.1.3
   2023/11/27 10:16:27 DBG Sent Heartbeat Request to: 10.10.1.3
   2023/11/27 10:16:27 DBG Received 16 bytes from 10.10.1.3:8805
   2023/11/27 10:16:27 DBG Handling PFCP message from 10.10.1.3:8805
   2023/11/27 10:16:27 DBG Got Heartbeat Response with TS: 2023-11-24 13:15:18 +0200 EET, from: 10.10.1.3
   2023/11/27 10:16:30 DBG Received 16 bytes from 10.10.1.3:8805
   2023/11/27 10:16:30 DBG Handling PFCP message from 10.10.1.3:8805
   2023/11/27 10:16:30 INF Got Session Deletion Request from: 10.10.1.3.
   
   2023/11/27 10:16:30 DBG EBPF: Delete PDR Downlink: ipv4=10.25.0.216
   2023/11/27 10:16:30 DBG EBPF: Delete PDR Uplink: teid=255776
   2023/11/27 10:16:30 DBG EBPF: Delete PDR Uplink: teid=164640
   2023/11/27 10:16:30 DBG EBPF: Delete FAR: intenalId=9
   2023/11/27 10:16:30 DBG EBPF: Delete FAR: intenalId=10
   2023/11/27 10:16:30 DBG EBPF: Delete FAR: intenalId=11
   2023/11/27 10:16:30 DBG EBPF: Delete QER: internalId=3
   2023/11/27 10:16:30 INF Deleting session: 5
   
9. pirog-spb

   linked a pull request that will close this issuefix: delete FAR and QER by correct id #498on Nov 27, 2023
10. pirog-spb

    closed this as completedin #498on Nov 28, 2023
11. Sabreu commented on Feb 14, 2025

    

    Sabreu

    on Feb 14, 2025

    Author

    More actions

    @pirog-spb
    I can see this similar kind of issue with main branch now. (:~/git/eupf$ git log
    commit 54ed069 (HEAD -> main, tag: v0.7.0, origin/main, origin/HEAD)

    As you can see 10.85.0.11 got these internal IDs and then 10.85.0.8 UE is released with those same IDs. Data stopped on 10.85.0.11 UE.

    Session Modification Request:
      UpdateFAR ID: 1
        Apply Action: [12 0]
        BAR ID: 1
    
    Session Establishment Request:
      CreatePDR ID: 1
        FAR ID: 1
        QER ID: 1
        URR ID: 1
        Source Interface: 1
        UE IPv4 Address: 10.85.0.11
      CreatePDR ID: 2
        Outer Header Removal: 0
        FAR ID: 2
        QER ID: 1
        Source Interface: 0
        TEID: 0
        Ipv4: <nil>
        Ipv6: <nil>
      CreatePDR ID: 3
        Outer Header Removal: 0
        FAR ID: 1
        Source Interface: 3
        TEID: 0
        Ipv4: <nil>
        Ipv6: <nil>
      CreatePDR ID: 4
        Outer Header Removal: 0
        FAR ID: 3
        Source Interface: 0
        TEID: 0
        Ipv4: <nil>
        Ipv6: <nil>
        SDF Filter: permit out 58 from ff02::2/128 to assigned
      CreateFAR ID: 1
        Apply Action: [12 0]
        BAR ID: 1
      CreateFAR ID: 2
        Apply Action: [2 0]
        Forwarding Parameters:
          Network Instance: 5gapn1
      CreateFAR ID: 3
        Apply Action: [2 0]
        Forwarding Parameters:
          Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:10 IPv4Address:10.10.6.3 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
      CreateQER ID: 1
        Gate Status DL: 0
        Gate Status UL: 0
        Max Bitrate DL: 10000000
        Max Bitrate UL: 10000000
        QFI: 1
      CreateURR ID: 1
        Measurement Method: 2
        Volume Threshold: &{Flags:1 TotalVolume:104857600 UplinkVolume:0 DownlinkVolume:0}
      CreateBAR ID: 1
    
    2025/02/14 14:35:51 INF Saving FAR info to session: 1, {Action:12 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:67504650 TransportLevelMarking:0}
    2025/02/14 14:35:51 DBG EBPF: Put FAR: internalId=1, farInfo={Action:12 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:67504650 TransportLevelMarking:0}
    2025/02/14 14:35:51 WRN No OuterHeaderCreation
    2025/02/14 14:35:51 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:67504650 TransportLevelMarking:0}
    2025/02/14 14:35:51 DBG EBPF: Put FAR: internalId=2, farInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:67504650 TransportLevelMarking:0}
    2025/02/14 14:35:51 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:10 RemoteIP:50727434 LocalIP:67504650 TransportLevelMarking:0}
    2025/02/14 14:35:51 DBG EBPF: Put FAR: internalId=3, farInfo={Action:2 OuterHeaderCreation:1 Teid:10 RemoteIP:50727434 LocalIP:67504650 TransportLevelMarking:0}
    2025/02/14 14:35:51 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:1410065408 MaxBitrateDL:1410065408 StartUL:0 StartDL:0}
    2025/02/14 14:35:51 DBG EBPF: Put QER: internalId=1, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:1410065408 MaxBitrateDL:1410065408 StartUL:0 StartDL:0}
    2025/02/14 14:35:51 INF Saving URR info to session: 1, {UplinkVolume:0 DownlinkVolume:0}
    2025/02/14 14:35:51 DBG EBPF: Put URR: internalId=1, urrInfo={UplinkVolume:0 DownlinkVolume:0}
    2025/02/14 14:35:51 DBG EBPF: Put PDR Downlink: ipv4=10.85.0.11, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:1 Urr1Id:1 Urr2Id:0 SdfFilter:<nil>}
    2025/02/14 14:35:51 DBG EBPF: Put PDR Uplink: teid=9, pdrInfo={OuterHeaderRemoval:0 FarId:2 QerId:1 Urr1Id:0 Urr2Id:0 SdfFilter:<nil>}
    2025/02/14 14:35:51 DBG EBPF: Put PDR Uplink: teid=10, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 Urr1Id:0 Urr2Id:0 SdfFilter:<nil>}
    2025/02/14 14:35:51 Matched groups: ["permit out 58 from ff02::2/128 to assigned" "58" "ff02::2" "128" "" "assigned" "" ""]
    2025/02/14 14:35:51 DBG EBPF: Put PDR Uplink: teid=9, pdrInfo={OuterHeaderRemoval:0 FarId:3 QerId:0 Urr1Id:0 Urr2Id:0 SdfFilter:{Protocol:4 SrcAddress:{Type:2 Ip:ff02::2 Mask:ffffffffffffffffffffffffffffffff} SrcPortRange:{LowerBound:0 UpperBound:65535} DstAddress:{Type:0 Ip:<nil> Mask:<nil>} DstPortRange:{LowerBound:0 UpperBound:65535}}}
    2025/02/14 14:35:51 INF Session Establishment Request from 10.10.6.3 accepted.
    
    2025/02/14 14:36:27 INF Got Session Deletion Request from: 10.10.6.3.
    2025/02/14 14:36:27 DBG EBPF: Delete PDR Uplink: teid=4
    2025/02/14 14:36:27 DBG EBPF: Delete PDR Uplink: teid=3
    2025/02/14 14:36:27 DBG EBPF: Delete PDR Downlink: ipv4=10.85.0.8
    2025/02/14 14:36:27 DBG EBPF: Delete FAR: intenalId=3
    2025/02/14 14:36:27 DBG EBPF: Delete FAR: intenalId=1
    2025/02/14 14:36:27 DBG EBPF: Delete FAR: intenalId=2
    2025/02/14 14:36:27 DBG EBPF: Delete QER: internalId=1
    2025/02/14 14:36:27 DBG EBPF: Delete URR: internalId=1
    2025/02/14 14:36:27 INF Deleting session: 4
    
12. Sabreu commented on Feb 19, 2025

    

    Sabreu

    on Feb 19, 2025

    Author

    More actions

    36fdd3a
    Here it was fixed
    d3e78b8#diff-fc56433772a8f2e727799bd4f2e218b1b556fb7c7630373773c4e7778212e2ed
    Next commit reverts all code back to non working.

    @pirog-spb