Skip to content

The invocation of CiliumIdentity changes is triggered after a random delay when modifying the namespace label.#38031

Merged
joestringer merged 2 commits intocilium:mainfrom
orange30:conntrol_delay
Jun 13, 2025
Merged

The invocation of CiliumIdentity changes is triggered after a random delay when modifying the namespace label.#38031
joestringer merged 2 commits intocilium:mainfrom
orange30:conntrol_delay

Conversation

@orange30
Copy link
Copy Markdown
Contributor

@orange30 orange30 commented Mar 6, 2025

Currently, in clusters where Cilium is deployed, changing the namespace label will instantly generate a number of ciliumidentity when the namespace have many pod who's label is different.
After these ciliumidentity events are pushed to the API server, they are fully distributed to each node.
These ciliumidentity changes will result in a large number of ciliumendpoint update events.
Therefore, when there are many pods who's lable is different in a namespace and the cluster has a large number of nodes, changing the namespace label can easily cause significant pressure on the API server and, under extreme circumstances, may lead to the API server crashing.

  1. In this patch, when modifying the namespace label, we delay the invocation of the CiliumIdentity change event by a random amount of time to alleviate pressure on the API server.
  2. The maximum delay time can be configured via the cilium-config ConfigMap.
  3. Modifying the namespace label is a relatively infrequent operation, so we believe that a delay should be acceptable in most cases.

Please ensure your pull request adheres to the following guidelines:

  • For first time contributors, read Submitting a pull request
  • All code is covered by unit and/or runtime tests where feasible.
  • All commits contain a well written commit description including a title,
    description and a Fixes: #XXX line if the commit addresses a particular
    GitHub issue.
  • If your commit description contains a Fixes: <commit-id> tag, then
    please add the commit author[s] as reviewer[s] to this issue.
  • All commits are signed off. See the section Developer’s Certificate of Origin
  • Provide a title or release-note blurb suitable for the release notes.
  • Are you a user of Cilium? Please add yourself to the Users doc
  • Thanks for contributing!

Fixes: #38030

A new option `--identity-max-jitter` adds a random delay to when Cilium reacts to namespace label updates. This jitter is intended to improve kube-apiserver resource usage at high scale.

@orange30 orange30 requested review from a team as code owners March 6, 2025 11:47
@orange30 orange30 requested review from derailed and thorn3r March 6, 2025 11:47
@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Mar 6, 2025
@orange30 orange30 requested review from tklauser and youngnick March 6, 2025 11:47
@github-actions github-actions bot added the kind/community-contribution This was a contribution made by a community member. label Mar 6, 2025
@orange30 orange30 changed the title Conntrol delay The invocation of CiliumIdentity changes is triggered after a random delay when modifying the namespace label. Mar 6, 2025
@joestringer joestringer added the release-note/minor This PR changes functionality that users may find relevant to operating Cilium. label Mar 7, 2025
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Mar 7, 2025
@orange30 orange30 force-pushed the conntrol_delay branch 7 times, most recently from 0ce2ffb to bec1317 Compare March 8, 2025 05:58
This was referenced Sep 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/community-contribution This was a contribution made by a community member. release-blocker/1.18 This issue will prevent the release of the next version of Cilium. release-note/minor This PR changes functionality that users may find relevant to operating Cilium.

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

High-Scale: Modifying the ns label may cause the apiserver to crash.

6 participants