[codex] Use pod-level CI resource variables#924
Conversation
This comment has been minimized.
This comment has been minimized.
9019784 to
fcefd69
Compare
fcefd69 to
79dce4c
Compare
There was a problem hiding this comment.
Pull request overview
Updates GitLab CI Kubernetes runner resource configuration to use pod-level CPU/memory budgets, avoiding double-counting resources across helper/build containers within the same CI pod.
Changes:
- Replaced container-level Kubernetes resource variables with pod-level equivalents (
KUBERNETES_POD_*). - Removed helper-container-specific resource variables and related commentary to reflect the new pod-level allocation model.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -7,19 +7,10 @@ stages: | |||
| .jobs-resource-allocation: | |||
| variables: | |||
| # Sized resources using https://app.datadoghq.com/dashboard/xff-wqx-4a2/ci-reliability-kubernetes-runner-sizing-and-diagnotic | |||
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 296753a797
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| KUBERNETES_POD_CPU_REQUEST: "6" | ||
| KUBERNETES_POD_CPU_LIMIT: "6" |
There was a problem hiding this comment.
On the CI nodes that use the static CPU policy described by the removed comment, moving the integer CPU request/limit entirely to spec.resources removes the build container's own CPU request. Kubernetes CPU Manager's static policy assigns exclusive CPUs only to containers with integer CPU requests (unless the alpha PodLevelResourceManagers path is enabled), so the build container now stays in the shared pool even though the pod schedules with a 6-CPU budget. That regresses the isolation these jobs had from noisy neighboring CI pods; keep the build container CPU request/limit while using the pod-level budget.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Job still runs ok and non-static CPU workload has been running extensively on prod according to compute.
Summary
Move the GitLab CI jobs to a single pod-level CPU and memory budget instead of defining separate build/helper container resources.
This reduces the total requested footprint of each CI pod. The helper container runs before the build container as part of the runner setup flow, so reserving a full helper budget in addition to the build budget overstates the resources that need to be available concurrently. With pod-level resources, the pod carries one shared budget while the build, helper, and init-permissions containers no longer define their own runner resource requests.
Validation
kubectl.spec.resourceswithcpu: 6andmemory: 12Girequests/limits.