Skip to content

Commit b58fd7b

Browse files
myasnikovdaniilmyasnikovdaniil
committed
feat(site-router): add routed site-to-site IPsec gateway app (Phase 1)
Adds `site-router`, a catalog app for routed (L3), source-IP-preserving tenant site-to-site connectivity over an IKEv2 IPsec tunnel terminated in a VyOS KubeVirt gateway VM. The chart materializes the gateway: VM + boot DataVolume, tunnel Service type LoadBalancer (UDP 500/4500, native loadBalancerClass), PSK and RBAC-isolated api-key Secrets, first-boot cloud-init, WorkloadMonitor, and two net-new Cilium policies (gateway egressDeny + gateway ingress). `site-router-controller` mediates what the chart cannot express: deny-set validation of the tunnel's remote networks, the kube-ovn return-route annotation, gateway-port `port_security` relaxation gated on the guest source filter, the live VyOS configuration push over the management API, tunnel/BGP observability, and status. There is no new tenant CRD — the `apps.cozystack.io/SiteRouter` app instance is the whole contract. A shared pure `internal/siterouter/denyset` validator backs both a SiteRouter-scoped admission check in `pkg/registry/apps/application` and the reconcile-time check, so a cluster-overlapping `remoteCIDR` is rejected identically at apply time and reconcile time. The admission runs fail-closed on create and on update. NAT/DNAT (a future `site-gateway`) and HA/VRRP are out of scope for Phase 1. The negative-security acceptance suite is authored but parked: its live run needs the published appliance image. Contents: - app chart packages/apps/site-router (10 helm-unittest suites) - site-router-controller: internal/controller/siterouter, cmd/site-router-controller, packages/system/site-router-controller - VyOS core library internal/vyos (client/parse/observation) and routed render internal/vyos/render - internal/siterouter/denyset + the admission check - the appliance build package packages/system/vyos-router-image - platform wiring: PackageSources for the application and the controller, both emitted by the naas bundle behind bundles.iaas.enabled (the gateway needs KubeVirt + CDI, which isp-hosted does not have) - docs: README, security-model, image-lifecycle, followups The e2e suite is committed parked as chainsaw-test.yaml.disabled, the way hack/e2e-chainsaw/backup/ is parked, because its appliance image is not published yet. Parking it as a file rather than as a name filtered out of hack/select-e2e.sh's output is deliberate: the selector decides coverage from the same enumeration, so a name-based filter leaves the graph walk still reaching the suite. site-router-application depends on cozystack.kubevirt-cdi, which would give kubevirt-cdi — and cozystack-basics, which reaches it — exactly one suite, stripped to an empty selection that both e2e lanes read as "skip Chainsaw" before posting the required "E2E Tests" status green (#3392). Parking the file keeps the suite out of the selector's universe and out of select-install.sh's round-trip validation at once, so those packages keep escalating to the full run as they do on main, and un-parking is renaming the file back. The appliance boot reference is committed digest-less (`…/vyos-router-disk:v0.0.0`). A first-party image that has never been built has no digest to commit, and nothing in the PR lane can create one: build-vyos ends at `git diff --binary HEAD > vyos.patch` and uploads it, so the stamp reaches e2e through pr.patch and never reaches git; only tags.yaml's post-merge `Prepare release` commits a digest. site-router-controller ships :v0.0.0 the same way in this PR, and securitygroup-controller sat at :v0.0.0 on main until a release build stamped it. An all-zero placeholder is strictly worse than no digest — it passes `required`, looks like a valid pin and resolves nowhere — so hack/image-refs-no-placeholder.bats keeps rejecting that shape untouched. While the ref is digest-less it is invisible to hack/lib/image-refs.sh, so promote/retag/mirror skip the appliance; tests/dv_test.yaml accepts either form and carries a TODO to re-tighten after the first release build. VYOS_BUILD_REF is pinned to e14a4895 (upstream `rolling` tip, 2026-08-11), whose kernel_version 6.18.44 is the only kernel the rolling mirror serves; the ref this started from asked for 6.18.41, which is gone, and live-build fails hard on that rather than building something stale. VYOS_VERSION tracks the ref's commit date so the label stays a function of the pin. The pin comment previously claimed Renovate kept these from aging silently — both custom managers do match the file, but Renovate only evaluates the default branch and neither the Makefile nor the managers exist there until this merges, so the pins were unwatched by construction; even after merge they raise a PR rather than holding the build up, while the breakage is on the mirror's kernel-retention clock. The comment and docs/image-lifecycle.md now say that. Two shared checks move with the app: cozystack-api's ClusterRole gains a name-scoped `get` on the cozystack ConfigMap that the deny-set admission reads (its pinned rule count goes 16 -> 17), and the site-router-controller chart ships the RBAC and deployment suites the sibling controller packages ship. Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent cc70cef commit b58fd7b

107 files changed

Lines changed: 15430 additions & 22 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/renovate.json‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,23 @@
3232
"matchStrings": ["clastix/kubectl:(?<currentValue>v[0-9]+\\.[0-9]+(?:\\.[0-9]+)?)@(?<currentDigest>sha256:[a-f0-9]+)"],
3333
"datasourceTemplate": "docker",
3434
"depNameTemplate": "docker.io/clastix/kubectl"
35+
},
36+
{
37+
"customType": "regex",
38+
"description": "Track the vyos-build container the VyOS router appliance is built in. It is pinned by digest behind the floating `current` tag in a Makefile variable, so neither the dockerfile manager (it is not a FROM line) nor gomod can see it, and the package's `make update` is a deliberate no-op because there is no chart to pull. Without this manager the build-tooling pin — the reproducibility lever for an image that ships as a tenant-facing gateway OS — would silently age.",
39+
"managerFilePatterns": ["/^packages/system/vyos-router-image/Makefile$/"],
40+
"matchStrings": ["VYOS_BUILD_IMAGE\\s*=\\s*vyos/vyos-build:(?<currentValue>\\S+)@(?<currentDigest>sha256:[a-f0-9]+)"],
41+
"datasourceTemplate": "docker",
42+
"depNameTemplate": "docker.io/vyos/vyos-build"
43+
},
44+
{
45+
"customType": "regex",
46+
"description": "Track the vyos-build git ref the VyOS router appliance flavor is built from. It is a bare commit SHA in a Makefile variable with no package manifest behind it, so only a regex manager can see it. Pinned to a commit rather than a branch on purpose — the appliance and the site-router cloud-init contract advance atomically — so this raises a PR when the upstream `current` branch moves and the bump stays a reviewed decision.",
47+
"managerFilePatterns": ["/^packages/system/vyos-router-image/Makefile$/"],
48+
"matchStrings": ["VYOS_BUILD_REF\\s*=\\s*(?<currentDigest>[0-9a-f]{40})"],
49+
"datasourceTemplate": "git-refs",
50+
"depNameTemplate": "https://github.com/vyos/vyos-build",
51+
"currentValueTemplate": "rolling"
3552
}
3653
],
3754
"packageRules": [

‎.github/workflows/pull-requests.yaml‎

Lines changed: 101 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ jobs:
7979
matrix: ${{ steps.p.outputs.matrix }}
8080
any: ${{ steps.p.outputs.any }}
8181
touched: ${{ steps.p.outputs.touched }}
82+
vyos: ${{ steps.p.outputs.vyos }}
8283
steps:
8384
# Claim the required context for THIS run before anything can be merged on
8485
# the strength of the last one. Commit statuses are latest-wins per (sha,
@@ -162,6 +163,17 @@ jobs:
162163
else
163164
echo "code=false" >> "$GITHUB_OUTPUT"
164165
fi
166+
# build-vyos runs a ~60-min privileged VyOS live-build whose image is
167+
# not consumed by the general test path (vm-default-images is an opt-in
168+
# bundle package), and a floating apt-repo failure in it would otherwise
169+
# block every unrelated PR. Gate it to the VyOS image pipeline and its
170+
# consumers (mirrors the docs-only `code` gate above: git-diff + grep);
171+
# finalize tolerates it being skipped.
172+
if grep -qE '^(packages/system/vyos-router-image/|packages/apps/site-router/|packages/system/vm-default-images/|\.github/workflows/pull-requests\.yaml$)' /tmp/changed.txt; then
173+
echo "vyos=true" >> "$GITHUB_OUTPUT"
174+
else
175+
echo "vyos=false" >> "$GITHUB_OUTPUT"
176+
fi
165177
# docs/agents/contributing.md carries the downstream trigger map, which
166178
# hack/downstream-trigger-map.bats asserts against: every path the map
167179
# cites still exists, and its repository list still matches the PR
@@ -522,6 +534,93 @@ jobs:
522534
path: _out/oci/*.oci.tar
523535
if-no-files-found: error
524536

537+
# VyOS is its own leg for the same reason Talos is: `make -C
538+
# packages/system/vyos-router-image image` runs a full VyOS live-build
539+
# (debootstrap + apt + squashfs) in a privileged container, far heavier than a
540+
# normal package image build, so it does not belong in the parallel matrix
541+
# (build-matrix.sh excludes it). It builds the qcow2, packages it as a
542+
# digest-pinned KubeVirt containerDisk pushed to the registry, and stamps the
543+
# ref@digest into packages/apps/site-router/images/vyos-router-disk.tag.
544+
# That stamp is captured as a patch fragment so finalize bundles the real digest
545+
# into the installer artifact (the deferred site-router e2e resolves it from
546+
# there). vm-default-images is an opt-in bundle package, so a non-opted-in e2e is
547+
# unaffected either way.
548+
#
549+
# Path-scoped (plan.outputs.vyos): this heavy, privileged build only runs when
550+
# the VyOS image pipeline or its consumers change (vyos-router-image /
551+
# site-router / vm-default-images / this workflow) — otherwise its image is never
552+
# consumed by the run and a floating apt-repo failure would needlessly block
553+
# unrelated PRs. When skipped it must not fail the gate: finalize accepts a
554+
# skipped result (see its `if`).
555+
build-vyos:
556+
name: Build VyOS
557+
needs: ["plan"]
558+
if: |
559+
needs.plan.outputs.code == 'true'
560+
&& needs.plan.outputs.vyos == 'true'
561+
&& !contains(github.event.pull_request.labels.*.name, 'release')
562+
runs-on: oracle-vm-4cpu-16gb-x86-64
563+
timeout-minutes: 60
564+
permissions:
565+
contents: read
566+
packages: write
567+
steps:
568+
- name: Checkout code
569+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
570+
with:
571+
fetch-depth: 0
572+
fetch-tags: true
573+
- name: Set up Docker config
574+
run: |
575+
if [ -d ~/.docker ]; then
576+
cp -r ~/.docker "${{ runner.temp }}/.docker"
577+
fi
578+
- name: Login to GitHub Container Registry
579+
if: ${{ !github.event.pull_request.head.repo.fork }}
580+
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
581+
with:
582+
username: ${{ secrets.OCIR_USER }}
583+
password: ${{ secrets.OCIR_TOKEN }}
584+
registry: iad.ocir.io
585+
env:
586+
DOCKER_CONFIG: ${{ runner.temp }}/.docker
587+
# buildkitd-config routes docker.io base-image pulls through mirror.gcr.io
588+
# (hack/buildkitd.toml); see the build job for the full rationale. The
589+
# docker-container builder is used for the containerDisk buildx push; the
590+
# heavy VyOS live-build itself runs as a privileged `docker run` on the
591+
# runner's default docker daemon (it needs --privileged + /dev for the
592+
# loop/kpartx disk operations, exactly like the Talos imager).
593+
- name: Set up Buildx (docker-container driver, docker.io mirror)
594+
id: buildx
595+
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
596+
with:
597+
driver: docker-container
598+
buildkitd-config: hack/buildkitd.toml
599+
env:
600+
DOCKER_CONFIG: ${{ runner.temp }}/.docker
601+
- name: Build VyOS router image (qcow2 + containerDisk)
602+
run: make -C packages/system/vyos-router-image image
603+
env:
604+
DOCKER_CONFIG: ${{ runner.temp }}/.docker
605+
BUILDER: ${{ steps.buildx.outputs.name }}
606+
IMAGE_TAG: pr-${{ github.event.pull_request.number }}-${{ github.event.pull_request.head.sha }}
607+
PUBLISH_VERSIONED: '0'
608+
PUBLISH_FLOATING: '0'
609+
- name: Capture VyOS patch fragment
610+
run: |
611+
mkdir -p _out/fragments
612+
# Captures the stamped apps/site-router/images/vyos-router-disk.tag.
613+
# Artifact only — this never commits or pushes, so the committed .tag
614+
# keeps whatever it holds until a release build stamps it.
615+
git diff --binary HEAD > _out/fragments/vyos.patch
616+
echo "VyOS fragment:"; git --no-pager diff --stat HEAD
617+
- name: Upload VyOS patch fragment
618+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
619+
with:
620+
name: pr-patch-fragment-vyos
621+
path: _out/fragments/vyos.patch
622+
if-no-files-found: error
623+
525624
# Reassemble the unified pr.patch from every fragment, then build the installer
526625
# LAST: `make -C packages/core/installer image` runs `flux push artifact
527626
# --path=packages`, which bundles the WHOLE digest-patched tree into the OCI
@@ -533,14 +632,15 @@ jobs:
533632
# to e2e (which needs finalize), so a red unit test skips the expensive
534633
# install+e2e path instead of wasting ~1h on it. Restores the gating the old
535634
# in-build-job tests gave, without re-coupling the cheap parallel builds.
536-
needs: ["plan", "checks", "build", "build-talos"]
635+
needs: ["plan", "checks", "build", "build-talos", "build-vyos"]
537636
if: |
538637
!cancelled()
539638
&& needs.plan.outputs.code == 'true'
540639
&& !contains(github.event.pull_request.labels.*.name, 'release')
541640
&& needs.checks.result == 'success'
542641
&& (needs.build.result == 'success' || needs.build.result == 'skipped')
543642
&& needs.build-talos.result == 'success'
643+
&& (needs.build-vyos.result == 'success' || needs.build-vyos.result == 'skipped')
544644
runs-on: oracle-vm-4cpu-16gb-x86-64
545645
timeout-minutes: 30
546646
permissions:

‎Makefile‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,12 @@ build: build-deps
3333
make -C packages/system/bucket image
3434
make -C packages/system/objectstorage-controller image
3535
make -C packages/system/securitygroup-controller image
36+
make -C packages/system/site-router-controller image
3637
make -C packages/system/grafana-operator image
3738
make -C packages/system/redis-operator image
3839
make -C packages/core/testing image
3940
make -C packages/core/talos image
41+
make -C packages/system/vyos-router-image image
4042
make -C packages/core/platform image
4143
make -C packages/core/installer image
4244
make manifests

‎api/apps/v1alpha1/siterouter/types.go‎

Lines changed: 124 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)