Skip to content

fix(core): always send PFCP responses; treat datapath delete ENOENT as success - #650

Merged
pirog-spb merged 1 commit into
edgecomllc:mainfrom
zhangerfei-dp:fix/always-send-pfcp-response
Oct 11, 2026
Merged

pirog-spb merged 1 commit into
edgecomllc:mainfrom
zhangerfei-dp:fix/always-send-pfcp-response

Conversation

@zhangerfei-dp

@zhangerfei-dp zhangerfei-dp commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #647.

(Branch rewritten per review: the ENOENT handling moved into mapOperations (cmd/ebpf), ReadUrr is extracted, and HandlePfcpSessionModificationRequest is covered by the same change.)

What

  1. Handle() — always transmit the handler's response when non-nil. A send error is logged, and only propagated when the handler itself succeeded.
  2. mapOperations (cmd/ebpf) — delete operations absorb "key absent" (ebpf.ErrKeyNotExist / ENOENT) as success via isKeyAbsent(), so no caller needs to care about this error: the desired end state of a delete is that the entry is absent. This uniformly covers every delete call site, including HandlePfcpSessionModificationRequest (RemoveFAR / RemoveQER / RemoveURR).
  3. DeleteUrr() split — ReadUrr() now explicitly returns the URR usage data; DeleteUrr() has the conventional error-only signature (as suggested in review). The deletion/modification handlers read first, then delete.

Why

Measured on a session-churn workload:

upf_pfcp_rx{Session Deletion Request} = 21
upf_pfcp_tx{Session Deletion Response} = 8
upf_pfcp_rx_errors{cause=RuleCreationModificationFailure, ...} = 13

13 deletion requests received no response at all: the handler returned (failure-response, err) after Can't delete GTP PDR: delete: key does not exist, and Handle() returned before sending the response it had built. The CP side (open5gs SMF) then timed out its PFCP transaction — on some open5gs versions that even aborts the whole SMF process (related: open5gs/open5gs#4833).

Verification

Patched build, same workload afterwards: rx = 7 / tx = 7, all requests answered with RequestAccepted, no Error handling PFCP warnings in the log. go build ./... and go vet ./cmd/... (tests included) are clean, and the touched files are gofmt-clean.

@pirog-spb

Copy link
Copy Markdown
Collaborator

Hi @zhangerfei-dp,

Thanks for this PR. In general the idea is clear and implemented well.

I'd recommend to process ENOENT error inside mapOperations in order not to care this error in calling code.

The second point is that you have to apply fix for delete operations in HandlePfcpSessionModificationRequest.

If you agree to move ENOENT error inside mapOperations, take into account DeleteUrr. This method currently returns data for URR. It may be difficult to implement new behavior because of that. It would be better to extract ReadURR explicitly.

@zhangerfei-dp
zhangerfei-dp force-pushed the fix/always-send-pfcp-response branch from a9441a7 to 5daf001 Compare October 4, 2026 01:44
@zhangerfei-dp

Copy link
Copy Markdown
Contributor Author

Thanks for the review, Alex! Addressed in the rewritten branch (force-updated, single commit):

  1. ENOENT moved into mapOperations — all deletes in cmd/ebpf (DeletePdrUplink, DeletePdrDownlink, DeleteDownlinkPdrIp6, DeleteFar, DeleteQer, DeleteUrr) now absorb key-absent via a small isKeyAbsent() helper (ebpf.ErrKeyNotExist / ENOENT). The cmd/core helper from the first revision is gone.
  2. HandlePfcpSessionModificationRequest covered — RemoveFAR/RemoveQER no longer fail on absent entries thanks to the mapOperations change, and RemoveURR now uses ReadUrr + DeleteUrr.
  3. DeleteUrr split — new ReadUrr(internalId) (UrrInfo, error) returns the usage data explicitly; DeleteUrr(internalId) error is conventional. Callers (session deletion + modification handlers) read first, then delete; the test mock is updated accordingly.

Checked: go build ./..., go vet ./cmd/... clean (tests compile), touched files gofmt-clean. Behavior re-verified on our churn workload (all Session Deletion Requests answered; rx == tx). Happy to iterate further.

@pirog-spb

Copy link
Copy Markdown
Collaborator

Now error handling is perfectly incapsulated in mapOperations. But you removed changes in Handle():

Handle() — always transmit the handler's response when non-nil. A send error is logged, and only propagated when the handler itself succeeded.

Looks like this changes were OK.

-- BR, Alex

@zhangerfei-dp
zhangerfei-dp force-pushed the fix/always-send-pfcp-response branch from 5daf001 to 91be046 Compare October 6, 2026 11:06
@zhangerfei-dp

Copy link
Copy Markdown
Contributor Author

Restored the Handle() change — sorry it got lost in the rewrite, thanks for spotting it. Per your note: Handle() now always transmits the handler's response when non-nil; a send error is logged and only propagated when the handler itself succeeded (the handler's error is still returned as before).

Re-verified on the updated branch: go generate ./cmd/... + go build ./... + go vet ./... clean, go test ./cmd/core/... passes, and the churn workload keeps answering all Session Deletion Requests (rx == tx).

@pirog-spb pirog-spb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

…apOperations

Handle() returned early when a handler reported an error and silently
dropped the response message the handler had built. The visible case: a
Session Deletion Response was never transmitted when the datapath cleanup
hit "delete: key does not exist" (measured rx=21 vs tx=8 within one
workload), so the CP function (open5gs SMF) saw no response and timed out
its PFCP transaction — on some open5gs versions that aborts the whole SMF
process (open5gs/open5gs#4833).

Changes:
- Handle(): always transmit the handler's response when non-nil; a send
  error is logged and only propagated when the handler succeeded.
- mapOperations (cmd/ebpf): delete operations now treat "key absent"
  (ebpf.ErrKeyNotExist / ENOENT) as success via isKeyAbsent(), so calling
  code does not have to care about this error — the desired end state of a
  delete is that the entry is absent. This uniformly covers every delete
  call site, including HandlePfcpSessionModificationRequest.
- DeleteUrr() is split: ReadUrr() explicitly returns the URR usage data,
  and DeleteUrr() follows the conventional error-only signature.

Verified with a patched build on a session-churn workload: all deletion
requests answered (rx=7/tx=7, RequestAccepted), and no "Error handling
PFCP" warnings in the log.

Fixes edgecomllc#647

Signed-off-by: frank <[email protected]>
@pirog-spb
pirog-spb force-pushed the fix/always-send-pfcp-response branch from 91be046 to 18d1ad7 Compare October 9, 2026 07:48
@zhangerfei-dp

Copy link
Copy Markdown
Contributor Author

Thanks for the careful reviews and the guidance through the iterations, @pirog-spb — much appreciated!

This one came out of a real PFCP-session-churn scenario in our test rig, so it's great to see the fix land here. If anything else helps before merge (rebase, extra test evidence, etc.), just let me know — happy to assist.

@pirog-spb
pirog-spb merged commit bc686b3 into edgecomllc:main Oct 11, 2026
8 checks passed
pirog-spb pushed a commit that referenced this pull request Oct 31, 2026
…s success (#650)

Signed-off-by: frank <[email protected]>
(cherry picked from commit bc686b3)
Signed-off-by: pirog-spb <[email protected]>
pirog-spb pushed a commit that referenced this pull request Oct 31, 2026
…s success (#650)

Signed-off-by: frank <[email protected]>
(cherry picked from commit bc686b3)
Signed-off-by: pirog-spb <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PFCP responses are silently dropped when a handler returns an error (Session Deletion Response lost after "delete: key does not exist")

2 participants