Skip to content

Commit a5e5e1a

Browse files
authored
Reduce K8s test resource usage with single-node Kind clusters (#73998)
* Reduce K8s test resource usage with single-node Kind clusters * Add test for forwarded node port matching Kind config and NodePort service * Refactor Kubernetes configuration and tests to use centralized port rendering
1 parent 4969b1a commit a5e5e1a

6 files changed

Lines changed: 123 additions & 19 deletions

File tree

‎.github/workflows/kustomize-overlays-tests.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -102,13 +102,13 @@ jobs:
102102
# convention (sources come from the base
103103
# image, not the workspace).
104104
# 5. upload-k8s-image -> loads the now-built kubernetes-variant
105-
# image into every kind node, so kubelet
105+
# image into the kind node, so kubelet
106106
# can resolve the chart's image reference
107107
# without hitting ghcr (which is private).
108108
# 6. deploy-airflow
109109
# 7. smoke-test-overlay -> renders the overlay, auto-preloads the
110110
# overlay's own images into the kind
111-
# nodes (no per-overlay images list
111+
# node (no per-overlay images list
112112
# needed - discovery walks the rendered
113113
# manifest), applies, walks the
114114
# STATUS.yaml verify: block with
@@ -124,7 +124,7 @@ jobs:
124124
run: breeze k8s configure-cluster
125125
- name: "Build kubernetes-variant image on top of the restored PROD image"
126126
run: breeze k8s build-k8s-image --no-copy-local-sources
127-
- name: "Upload kubernetes-variant image to kind nodes"
127+
- name: "Upload kubernetes-variant image to the kind node"
128128
run: breeze k8s upload-k8s-image
129129
- name: "Deploy Airflow chart"
130130
run: breeze k8s deploy-airflow --executor CeleryExecutor

‎contributing-docs/testing/k8s_tests.rst‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -281,7 +281,6 @@ The typical session for tests with Kubernetes looks like follows:
281281
apiServerPort: 48366
282282
nodes:
283283
- role: control-plane
284-
- role: worker
285284
extraPortMappings:
286285
- containerPort: 30007
287286
hostPort: 18150
@@ -290,12 +289,11 @@ The typical session for tests with Kubernetes looks like follows:
290289
291290
Creating cluster "airflow-python-3.11-v1.24.2" ...
292291
✓ Ensuring node image (kindest/node:v1.24.2) 🖼
293-
✓ Preparing nodes 📦 📦
292+
✓ Preparing nodes 📦
294293
✓ Writing configuration 📜
295294
✓ Starting control-plane 🕹️
296295
✓ Installing CNI 🔌
297296
✓ Installing StorageClass 💾
298-
✓ Joining worker nodes 🚜
299297
Set kubectl context to "kind-airflow-python-3.11-v1.24.2"
300298
You can now use your cluster with:
301299
@@ -377,13 +375,11 @@ The typical session for tests with Kubernetes looks like follows:
377375
coredns-6d4b75cb6d-rwp9d 1/1 Running 0 71s
378376
coredns-6d4b75cb6d-vqnrc 1/1 Running 0 71s
379377
etcd-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s
380-
kindnet-ckc8l 1/1 Running 0 69s
381378
kindnet-qqt8k 1/1 Running 0 71s
382379
kube-apiserver-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s
383380
kube-controller-manager-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s
384-
kube-proxy-6g7hn 1/1 Running 0 69s
385381
kube-proxy-dwfvp 1/1 Running 0 71s
386-
kube-scheduler-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s
382+
kube-scheduler-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s
387383
388384
KinD Cluster API server URL: http://localhost:48366
389385
Connecting to localhost:18150. Num try: 1
@@ -446,7 +442,6 @@ The typical session for tests with Kubernetes looks like follows:
446442
Good version of helm installed: 3.9.2 in /Users/jarek/IdeaProjects/airflow/kubernetes-tests/.venv/bin
447443
Stable repo is already added
448444
Uploading Airflow image ghcr.io/apache/airflow/main/prod/python3.11-kubernetes to cluster airflow-python-3.11-v1.24.2
449-
Image: "ghcr.io/apache/airflow/main/prod/python3.11-kubernetes" with ID "sha256:fb6195f7c2c2ad97788a563a3fe9420bf3576c85575378d642cd7985aff97412" not yet present on node "airflow-python-3.11-v1.24.2-worker", loading...
450445
Image: "ghcr.io/apache/airflow/main/prod/python3.11-kubernetes" with ID "sha256:fb6195f7c2c2ad97788a563a3fe9420bf3576c85575378d642cd7985aff97412" not yet present on node "airflow-python-3.11-v1.24.2-control-plane", loading...
451446
452447
NEXT STEP: You might now deploy Airflow by:

‎dev/breeze/doc/05_test_commands.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -703,7 +703,7 @@ The runner is overlay-agnostic. For every overlay it:
703703

704704
* renders the overlay and substitutes ``RELEASE-NAME`` / ``NAMESPACE``,
705705
* **auto-preloads every ``image:`` referenced by the rendered manifest**
706-
into the kind nodes via ``docker pull`` (with retry on Docker Hub
706+
into the kind node via ``docker pull`` (with retry on Docker Hub
707707
rate limits) + ``kind load docker-image``, so the test does not flake
708708
on registry availability,
709709
* applies the overlay,

‎dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py‎

Lines changed: 28 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -379,6 +379,20 @@ def _get_kind_cluster_config_content(python: str, kubernetes_version: str) -> di
379379
return yaml.safe_load(config_path.read_text())
380380

381381

382+
# Must match nodePort in scripts/ci/kubernetes/nodeport.yaml.
383+
FORWARDED_NODE_PORT = 30007
384+
385+
386+
def _render_kind_cluster_config(forwarded_port_number: int, k8s_api_server_port: int) -> str:
387+
return (
388+
(SCRIPTS_CI_KUBERNETES_PATH / "kind-cluster-conf.yaml")
389+
.read_text()
390+
.replace("{{FORWARDED_NODE_PORT}}", str(FORWARDED_NODE_PORT))
391+
.replace("{{FORWARDED_PORT_NUMBER}}", str(forwarded_port_number))
392+
.replace("{{API_SERVER_PORT}}", str(k8s_api_server_port))
393+
)
394+
395+
382396
def set_random_cluster_ports(python: str, kubernetes_version: str, output: Output | None) -> None:
383397
"""
384398
Creates cluster config file and returns sockets keeping the ports bound.
@@ -390,24 +404,32 @@ def set_random_cluster_ports(python: str, kubernetes_version: str, output: Outpu
390404
f"[info]Random ports: K8S API: {k8s_api_server_port}, API Server: {forwarded_port_number}"
391405
)
392406
cluster_conf_path = get_kind_cluster_config_path(python=python, kubernetes_version=kubernetes_version)
393-
config = (
394-
(AIRFLOW_ROOT_PATH / "scripts" / "ci" / "kubernetes" / "kind-cluster-conf.yaml")
395-
.read_text()
396-
.replace("{{FORWARDED_PORT_NUMBER}}", str(forwarded_port_number))
397-
.replace("{{API_SERVER_PORT}}", str(k8s_api_server_port))
407+
config = _render_kind_cluster_config(
408+
forwarded_port_number=forwarded_port_number, k8s_api_server_port=k8s_api_server_port
398409
)
399410
cluster_conf_path.write_text(config)
400411
get_console(output=output).print(f"[info]Config created in {cluster_conf_path}:\n")
401412
get_console(output=output).print(config)
402413
get_console(output=output).print("\n")
403414

404415

416+
def _extract_forwarded_host_port(conf: dict[str, Any]) -> int:
417+
for node in conf["nodes"]:
418+
for port_mapping in node.get("extraPortMappings", []):
419+
if port_mapping["containerPort"] == FORWARDED_NODE_PORT:
420+
return port_mapping["hostPort"]
421+
raise ValueError(
422+
f"No node in the kind cluster config has an extraPortMappings entry for "
423+
f"containerPort {FORWARDED_NODE_PORT}. Delete and recreate the cluster."
424+
)
425+
426+
405427
def get_kubernetes_port_numbers(python: str, kubernetes_version: str) -> tuple[int, int]:
406428
conf = _get_kind_cluster_config_content(python=python, kubernetes_version=kubernetes_version)
407429
if not conf:
408430
return 0, 0
409431
k8s_api_server_port = conf["networking"]["apiServerPort"]
410-
api_server_port = conf["nodes"][1]["extraPortMappings"][0]["hostPort"]
432+
api_server_port = _extract_forwarded_host_port(conf)
411433
return k8s_api_server_port, api_server_port
412434

413435

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
from __future__ import annotations
19+
20+
from unittest.mock import patch
21+
22+
import pytest
23+
import yaml
24+
25+
from airflow_breeze.utils.kubernetes_utils import (
26+
FORWARDED_NODE_PORT,
27+
SCRIPTS_CI_KUBERNETES_PATH,
28+
_render_kind_cluster_config,
29+
get_kubernetes_port_numbers,
30+
)
31+
32+
# Rendered config of a cluster created before the switch to single-node clusters —
33+
# port extraction must keep working for clusters that already exist on disk.
34+
LEGACY_TWO_NODE_CONFIG = """\
35+
kind: Cluster
36+
apiVersion: kind.x-k8s.io/v1alpha4
37+
networking:
38+
ipFamily: ipv4
39+
apiServerAddress: "127.0.0.1"
40+
apiServerPort: 48366
41+
nodes:
42+
- role: control-plane
43+
- role: worker
44+
extraPortMappings:
45+
- containerPort: 30007
46+
hostPort: 18150
47+
listenAddress: "127.0.0.1"
48+
protocol: TCP
49+
"""
50+
51+
NO_FORWARDED_PORT_CONFIG = """\
52+
kind: Cluster
53+
apiVersion: kind.x-k8s.io/v1alpha4
54+
networking:
55+
ipFamily: ipv4
56+
apiServerAddress: "127.0.0.1"
57+
apiServerPort: 48366
58+
nodes:
59+
- role: control-plane
60+
"""
61+
62+
63+
@pytest.mark.parametrize(
64+
"config",
65+
[
66+
_render_kind_cluster_config(forwarded_port_number=18150, k8s_api_server_port=48366),
67+
LEGACY_TWO_NODE_CONFIG,
68+
],
69+
ids=["current-template", "legacy-two-node"],
70+
)
71+
@patch("airflow_breeze.utils.kubernetes_utils._get_kind_cluster_config_content", autospec=True)
72+
def test_get_kubernetes_port_numbers_extracts_ports_from_any_node(mock_get_content, config):
73+
mock_get_content.return_value = yaml.safe_load(config)
74+
assert get_kubernetes_port_numbers(python="3.10", kubernetes_version="v1.30.13") == (48366, 18150)
75+
76+
77+
@patch("airflow_breeze.utils.kubernetes_utils._get_kind_cluster_config_content", autospec=True)
78+
def test_get_kubernetes_port_numbers_raises_when_forwarded_port_mapping_is_missing(mock_get_content):
79+
mock_get_content.return_value = yaml.safe_load(NO_FORWARDED_PORT_CONFIG)
80+
with pytest.raises(ValueError, match="extraPortMappings"):
81+
get_kubernetes_port_numbers(python="3.10", kubernetes_version="v1.30.13")
82+
83+
84+
def test_nodeport_service_exposes_forwarded_node_port():
85+
node_port_service = yaml.safe_load((SCRIPTS_CI_KUBERNETES_PATH / "nodeport.yaml").read_text())
86+
assert [port["nodePort"] for port in node_port_service["spec"]["ports"]] == [FORWARDED_NODE_PORT]

‎scripts/ci/kubernetes/kind-cluster-conf.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,11 @@ networking:
2222
apiServerAddress: "127.0.0.1"
2323
apiServerPort: {{API_SERVER_PORT}}
2424
nodes:
25+
# A single node on purpose: kind drops the control-plane NoSchedule taint only for single-node
26+
# clusters, so the Airflow and KubernetesPodOperator pods are scheduled here.
2527
- role: control-plane
26-
- role: worker
2728
extraPortMappings:
28-
- containerPort: 30007
29+
- containerPort: {{FORWARDED_NODE_PORT}}
2930
hostPort: {{FORWARDED_PORT_NUMBER}}
3031
listenAddress: "127.0.0.1"
3132
protocol: TCP

0 commit comments

Comments
 (0)