Skip to content

Adding support for AWS ENI prefix delegation - IPv4 Only#18463

Merged
qmonnet merged 3 commits intocilium:masterfrom
DataDog:hemanthmalla/eni_pd
Mar 29, 2022
Merged

Adding support for AWS ENI prefix delegation - IPv4 Only#18463
qmonnet merged 3 commits intocilium:masterfrom
DataDog:hemanthmalla/eni_pd

Conversation

@hemanthmalla
Copy link
Copy Markdown
Member

@hemanthmalla hemanthmalla commented Jan 12, 2022 •
edited
Loading

AWS introduced support for assigning prefixes to EC2 network interfaces - prefix delegation (pd) . Some of the benefits of using PD are:

With aws-enable-prefix-delegation flag enabled operator can now allocate /28 prefixes to resolve deficit on nodes. Once allocated, operator will update the cilium node object with the corresponding 16 IPs. Agent will use these IPs just like private secondary IPs.

pd_arch

Refer to the RFC for additional details discussing the the design and key decisions.

Fixes: #16987

This PR needs changes from #18557 for running the e2e test.

Limitations :

  • PD is only supported on VMs based on nitro hypervisor.
  • Currently AWS only supports assigning fixed size prefixes. /28 for IPv4 and /80 for IPv6

Non goals :

  • IPv6 prefix support
  • Releasing unused prefixes when aws-excess-ip-release is enabled

Todo :

  • e2e tests for PD
  • Unit test for PD
  • Validate disabling prefix delegation works for existing PD nodes

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jan 12, 2022
@hemanthmalla hemanthmalla mentioned this pull request Jan 20, 2022
Closed
3 tasks
@hemanthmalla hemanthmalla force-pushed the hemanthmalla/eni_pd branch 4 times, most recently from 2ec8e60 to 81abb2a Compare January 21, 2022 13:54
@gandro gandro added area/eni Impacts ENI based IPAM. release-note/major This PR introduces major new functionality to Cilium. labels Jan 25, 2022
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jan 25, 2022
@dzoeteman
Copy link
Copy Markdown

dzoeteman commented Feb 17, 2022

Hi @hemanthmalla, can I assist in anyway with this PR?
I noticed aws-cni chaining mode is also affected by this (makes sense), and I would assume your code would fix it too for that use case.

@hemanthmalla
Copy link
Copy Markdown
Member Author

hemanthmalla commented Feb 22, 2022

Hi @dzoeteman, I'm waiting on ENI e2e changes from #18557 to update this PR with e2e tests for prefix delegation.

@aanm aanm mentioned this pull request Mar 3, 2022
Closed
@hemanthmalla hemanthmalla force-pushed the hemanthmalla/eni_pd branch 2 times, most recently from 9ddf219 to 3764354 Compare March 3, 2022 15:22
@hemanthmalla
Copy link
Copy Markdown
Member Author

hemanthmalla commented Mar 3, 2022

Removed e2e tests and marking as ready for review. Will follow up with another PR for e2e tests once the ENI e2e PR is merged.

@hemanthmalla hemanthmalla marked this pull request as ready for review March 3, 2022 16:13
@hemanthmalla hemanthmalla requested review from a team March 3, 2022 16:13
@hemanthmalla hemanthmalla requested review from a team as code owners March 3, 2022 16:13
@hemanthmalla
Copy link
Copy Markdown
Member Author

hemanthmalla commented Mar 28, 2022

@qmonnet possibly because my branch doesn't have commits from #19061 ?
Let's try again with a rebase ? Esp. because PD related code shouldn't affect this at all ?

@hemanthmalla
Adding support for tracking instance hypervisor type in ENI limits pkg
258870e 
Signed-off-by: Hemanth Malla <[email protected]>
@hemanthmalla
Adding support for prefix delegation in AWS ENI mode
67b0fb9 
AWS recently introduced support to assign IP prefixes to ENIs on nitro
instances.Adding support for this feature will allow for running roughly
16x pods. This commit adds necessary changes to the operator to allocate
IPv4 prefixes and update the cilium node CRD with corresponding IPs from
those prefixes.

Fixes: cilium#16987

Signed-off-by: Hemanth Malla <[email protected]>
@hemanthmalla
Updating helm chart to support prefix delegation
9d0f8b3 
Signed-off-by: Hemanth Malla <[email protected]>
@gandro
Copy link
Copy Markdown
Member

gandro commented Mar 28, 2022 •
edited by maintainer-s-little-helper bot
Loading

/test

Job 'Cilium-PR-K8s-GKE' failed:

Click to show.

Test Name

K8sDatapathConfig DirectRouting Check connectivity with direct routing and endpointRoutes

Failure Output

FAIL: Connectivity test between nodes failed

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-GKE so I can create one.

@qmonnet
Copy link
Copy Markdown
Member

qmonnet commented Mar 28, 2022

@qmonnet possibly because my branch doesn't have commits from #19061

Ah, thanks, I was looking for something like this 👍

? Let's try again with a rebase ? Esp. because PD related code shouldn't affect this at all ?

Yes agreed, I was a bit reluctant to merge because of the recurring issue, but it's true it's unlikely from your code change. I see you're rebased already, let's give it another try [oh but Sebastian was faster than me!]. Thanks!

@hemanthmalla
Copy link
Copy Markdown
Member Author

hemanthmalla commented Mar 28, 2022

Cilium-PR-K8s-1.23-kernel-net-next succeeded now, but looks like we hit another flake #17628 for Cilium-PR-K8s-GKE

@hemanthmalla
Copy link
Copy Markdown
Member Author

hemanthmalla commented Mar 28, 2022

@gandro / @qmonnet safe to ignore the GKE failure ?

@sayboras
Copy link
Copy Markdown
Member

sayboras commented Mar 29, 2022

/test-gke

@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Mar 29, 2022
@qmonnet qmonnet merged commit 6947ab1 into cilium:master Mar 29, 2022
@sayboras
Copy link
Copy Markdown
Member

sayboras commented Mar 29, 2022

safe to ignore the GKE failure ?

Yup, it was successful before, so I triggered re-run just to verify other changes from master branch (as jenkins is running with merge commit).

@joestringer joestringer mentioned this pull request Mar 29, 2022
Open
gandro added a commit to gandro/cilium that referenced this pull request Mar 30, 2022
@gandro
eni: Fix broken build due to unit test
27262ee 
PR cilium#18463 changed the signature of the CreateNetworkInterface and
NewNodeManager methods. This change was merged last week.  PR cilium#19096
added a unit test in TestNodeManagerENIExcludeInterfaceTags which calls
those methods with the old signature.

Because the latter PR was not rebased after a merge, we accidentally
merged a non-compilable unit test into master, even though CI was green
(as CI was run before the signature was changed).

Signed-off-by: Sebastian Wicki <[email protected]>
@gandro gandro mentioned this pull request Mar 30, 2022
Merged
pchaigno pushed a commit that referenced this pull request Mar 30, 2022
@gandro @pchaigno
eni: Fix broken build due to unit test
c667a3f 
PR #18463 changed the signature of the CreateNetworkInterface and
NewNodeManager methods. This change was merged last week.  PR #19096
added a unit test in TestNodeManagerENIExcludeInterfaceTags which calls
those methods with the old signature.

Because the latter PR was not rebased after a merge, we accidentally
merged a non-compilable unit test into master, even though CI was green
(as CI was run before the signature was changed).

Signed-off-by: Sebastian Wicki <[email protected]>
@mKeRix mKeRix mentioned this pull request May 23, 2022
Open
@hemanthmalla hemanthmalla mentioned this pull request Jun 2, 2022
Merged
@hemanthmalla hemanthmalla mentioned this pull request Jun 24, 2022
Merged
@BenHinthorne BenHinthorne mentioned this pull request Oct 26, 2022
Merged
@zhenggu zhenggu mentioned this pull request Mar 31, 2023
Closed
@tianhaopx tianhaopx mentioned this pull request Apr 28, 2024
Closed
3 tasks
This was referenced May 21, 2025
Merged
Merged
Merged
This was referenced Feb 27, 2026
Closed
Merged
Merged
@clubanderson clubanderson mentioned this pull request Mar 17, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gandro gandro gandro approved these changes

@ti-mo ti-mo ti-mo approved these changes

@christarazi christarazi christarazi approved these changes

@sayboras sayboras sayboras approved these changes

@qmonnet qmonnet qmonnet approved these changes

Assignees

No one assigned

Labels

area/eni Impacts ENI based IPAM. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/major This PR introduces major new functionality to Cilium.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ENI IPAM mode should add support for assigning IP prefixes to EC2 instance

7 participants

@hemanthmalla @dzoeteman @sayboras @gandro @qmonnet @ti-mo @christarazi