Skip to content

PFCP responses are silently dropped when a handler returns an error (Session Deletion Response lost after "delete: key does not exist") #647

Closed

Description

@zhangerfei-dp

Summary

PfcpHandlerMap.Handle() returns early when a handler reports an error and silently discards the response message the handler just built. The most visible case is the Session Deletion path: HandlePfcpSessionDeletionRequest() → deletePDR() hits Can't delete GTP PDR: delete: key does not exist (the datapath entry is already gone; cilium/ebpf surfaces ENOENT), the handler returns (SessionDeletionResponse(cause=RuleCreationModificationFailure), err), and the response is never transmitted.

Per PFCP (TS 29.244) the UP function must answer every received request. Without a response, the CP function (e.g. open5gs SMF) times out its PFCP transaction — on some open5gs versions this even aborts the entire SMF process (sess_5gc_timeout: Assertion 'stream' failed), so this can cascade into a core-network outage.

Environment / evidence

  • eUPF commit 241af932 — both defects are still present on main as of 2026-10-03
  • Session-churn scenario (~600 UEs, repeated attach/detach), Prometheus counters afterwards:
upf_pfcp_rx{message_name="Session Deletion Request"} 21
upf_pfcp_tx{message_name="Session Deletion Response"} 8
upf_pfcp_rx_errors{cause_code="RuleCreationModificationFailure",message_name="Session Deletion Request"} 13

i.e. 21 requests received, only 8 responses transmitted — 13 swallowed. Logs for those 13:

INF Got Session Deletion Request from: 172.22.0.7.
WRN Error handling PFCP message: Can't delete GTP PDR: delete: key does not exist

(There is no outgoing Session Deletion Response frame on the wire.)

Root cause

  1. cmd/core/pfcp_handlers.go — the response is dropped when the handler returns an error:
outgoingMsg, err := handler(conn, incomingMsg, stringIpAddr)
if err != nil {
    log.Warn().Msgf("Error handling PFCP message: %s", err.Error())
    return err                      // <-- outgoingMsg is discarded here
}
...
if outgoingMsg != nil {
    PfcpMessageTx.WithLabelValues(outgoingMsg.MessageTypeName()).Inc()
    return conn.SendMessage(outgoingMsg, addr)
}
  1. cmd/core/pdr_creation_context.go deletePDR() (and the FAR/QER delete loops in HandlePfcpSessionDeletionRequest) treat "entry already absent" (ENOENT / "key does not exist") as a hard failure, even though the desired end state of a delete is precisely “absent”.

Related but different root cause: #527 ("Can't delete PDR with v0.6.0", closed as fixed in v0.6.1) — the current main can still reproduce deletion failures + missing responses as above.

Suggested fix

  1. Always transmit outgoingMsg when non-nil, regardless of the handler's error (log the send error if any).
  2. Tolerate ENOENT on datapath deletes — treat "key does not exist" as success (optionally log at debug level).

Verification

We run a locally patched build with the two changes above. Same workload afterwards:

upf_pfcp_rx{Session Deletion Request} = 7
upf_pfcp_tx{Session Deletion Response} = 7      (all answered, all RequestAccepted)

No Error handling PFCP warnings in the log, and the downstream SMF no longer sees deletion timeouts. Happy to provide a PR with the patch.

Activity

  1. pirog-spb commented on Oct 3, 2026

    @pirog-spb
    Collaborator

    Hi @zhangerfei-dp ,

    Thanks for the detailed description. A PR would be very helpful.

    -- BR, Alex

  2. zhangerfei-dp commented on Oct 3, 2026

    @zhangerfei-dp
    ContributorAuthor

    Thanks Alex! The PR is up: #650

    Summary: Handle() now always transmits the handler's response when non-nil (send errors only logged), and deletePDR() / FAR / QER deletes treat ENOENT ("key does not exist") as success via a new isDataplaneKeyAbsent() helper.

    Verified on our churn workload: after the patch, every Session Deletion Request is answered (rx=7/tx=7, all RequestAccepted, no Error handling PFCP warnings). Happy to adjust if you'd prefer different semantics (e.g. keeping the failure cause but still sending the response).

  3. added 2 commits that reference this issue on Oct 4, 2026
    5daf001
    91be046
  4. added a commit that references this issue on Oct 9, 2026
    18d1ad7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions