Skip to content

test(e2e): lightweight E2E cluster (kind / Talos-in-Docker) for diff-scoped PR runs #3238

New issue
New issue
Closed
Closed
test(e2e): lightweight E2E cluster (kind / Talos-in-Docker) for diff-scoped PR runs#3238
Assignees
myasnikovdaniil
Labels
area/ciIssues or PRs related to CI workflows, GitHub Actions, automationarea/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/featureCategorizes issue or PR as related to a new featurepriority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completetriage/acceptedIndicates an issue is ready to be actively worked on

Description

@myasnikovdaniil
myasnikovdaniil
opened on Jul 8, 2026

Problem

Every PR E2E run provisions a full Talos cluster of three nested QEMU/KVM VMs inside a privileged Docker container, on the largest VM runner class. This is the heaviest and slowest part of PR CI, and it is paid in full even for a PR that only touches a single app and needs nothing from the storage / networking / virtualization stack that the heavy cluster exists to exercise.

Sibling epic #2847 (Track A) makes the install lighter — a test-minimal bundle + hack/select-install.sh install only the packages a PR's diff needs. This issue tackles the orthogonal axis #2847 explicitly left out of scope: the cluster the install runs on. The two compose: a minimal install on a minimal cluster.

Current heavy approach (evidence)

  • .github/workflows/pull-requests.yaml — the e2e job runs on the 24 vCPU / 96 GB VM runner class with a 180-minute timeout.
  • Makefile: prepare-env → packages/core/testing apply + prepare-cluster.
  • packages/core/testing/Makefile: apply — docker run --privileged launches the e2e-sandbox container.
  • hack/e2e-prepare-cluster.bats — inside that container it boots 3 QEMU VMs (-smp 8 -m 24576, i.e. 8 vCPU + 24 GB each), each with a 50 GB system disk cloned from the Talos nocloud-amd64.raw image plus a 200 GB data disk, wires a bridge + masquerade, generates Talos machine-config, and waits on the Talos API before install.
  • A dedicated build-talos job builds and uploads the nocloud disk (make -C packages/core/talos talos-nocloud) on every PR to feed the above.

Net: full Talos + nested virtualization + a per-PR disk build, for all PRs.

Proposed lighter approach

Add a lightweight cluster provisioner as an alternative to the QEMU-VM path in hack/e2e-prepare-cluster.bats, selected per-PR (e.g. reuse the existing diff/TIA signal from #2559 / #2847 so only diffs that don't touch storage, CNI dataplane, or virtualization opt into it). Two candidate backends:

  1. kind (Kubernetes-in-Docker) — nodes are containers; no nocloud disk, no nested VMs, runs on a small runner. Closest to the original Add simple test with kind #98 idea but re-scoped to the current Package/bundle install model and to diff-scoped selection rather than a fixed paas-hosted bundle.
  2. Talos-in-Docker (talosctl cluster create --provisioner docker) — keeps Talos as the distro (so machine-config, the cozystack API, and the platform bundle behave as in production) while dropping QEMU and the raw-disk build. Higher fidelity than kind; still no kernel-module / KVM surface.

Either way the heavy Talos-on-QEMU path stays as the default and is always used for storage / networking / virtualization diffs and for every release run.

Constraints / non-goals

  • A containerized cluster cannot load out-of-tree kernel modules (DRBD / LINSTOR / ZFS), cannot run KubeVirt VMIs, and does not exercise the real CNI dataplane. So the lightweight path is only valid for diff-scoped tests that need none of those; anything touching system/linstor, system/kubeovn/CNI, or apps/virtual-machine etc. must fall back to full Talos. Release E2E always runs full Talos.
  • Not replacing the BATS harness or the app-test suite — only the cluster provisioning step.

Open questions

  • kind vs Talos-in-Docker: is the extra fidelity of Talos-in-Docker worth it, or is kind enough for the app-only tests that would use this path?
  • Selection signal: reuse the e2e-tia / select-e2e.sh diff classification, or add a distinct e2e-light-cluster opt-in label?
  • Which app E2E tests actually pass on a containerized cluster today? Needs an audit of hack/e2e-apps/*.bats for module/VM/dataplane assumptions.
  • Runner sizing: what runner class can host the light cluster, and what is the measured wall-time / cost delta vs the current 24 vCPU / 96 GB path?
  • How does the minimal install (bundles.testMinimal, [Epic] Diff-driven & scenario-based E2E testing for Cozystack #2847) behave on a containerized cluster — any packages in the minimal baseline that still assume Talos primitives?

Relation to other work

Activity

  1. myasnikovdaniil
    added
    kind/featureCategorizes issue or PR as related to a new feature
    on Jul 8, 2026
  2. myasnikovdaniil
    self-assigned this
    on Jul 8, 2026
  3. myasnikovdaniil
    added
    area/ciIssues or PRs related to CI workflows, GitHub Actions, automation
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    on Jul 8, 2026

  4. myasnikovdaniil commented on Jul 9, 2026

    @myasnikovdaniil
    myasnikovdaniil
    on Jul 9, 2026
    ContributorAuthor

    Survival audit — all 22 hack/e2e-apps/*.bats on kind / Talos-in-Docker

    Verdict Count Files
    SURVIVES (only needs a default StorageClass) 15 bucket, clickhouse, etcd, external-dns, foundationdb, harbor, kafka, kuberture, mariadb, mongodb, openbao, postgres, qdrant, redis, serviceexposure
    CONDITIONAL (survives iff CNI=Cilium; no virt/kernel deps) 2 gateway, securitygroup
    NEEDS-REAL-CLUSTER (nested KVM virt and/or DRBD-replicated storage) 5 vminstance, kubernetes-latest, kubernetes-previous, kubernetes-oidc-system, kubernetes-oidc-customconfig

    SURVIVES (15): all set storageClass: "" (or need no storage) and provision single replicas — no DRBD/replication. kind's local-path or any Talos-in-Docker default SC satisfies them. bucket/harbor additionally need the SeaweedFS+COSI platform stack (a provisioning prerequisite, not a kernel blocker). serviceexposure deliberately uses the externalIPs backend to avoid any MetalLB/Cilium pool or L2 reachability (serviceexposure.bats:5–11).

    CONDITIONAL (2) — no nested-virt/kernel deps, but require Cilium as CNI (fine on Talos-in-Docker w/ Cilium, fail on vanilla kindnet):

    • gateway — GatewayClass cilium, cilium-gateway-* LB Service, Programmed needs Cilium dataplane + LB IPAM.
    • securitygroup — round-trips SecurityGroup ↔ cilium.io/v2 CiliumNetworkPolicy CRD.

    NEEDS-REAL-CLUSTER (5) — hard blockers, nested KVM virt and/or DRBD:

    • vminstance — KubeVirt VM/VMI + CDI DataVolume + storageClass: replicated.
    • kubernetes-latest / -previous — Kamaji CP + CAPK KubeVirt worker VMs + CDI + DRBD-replicated worker disks (run-kubernetes.sh:68,402-438). (latest's second test, verify_storageclass_fallback_default, renders against the mgmt cluster and would survive alone.)
    • kubernetes-oidc-system / -customconfig — tenant k8s with storageClass: replicated.

    Recommendation

    A lightweight lane (Talos-in-Docker + stock Cilium CNI + a default StorageClass) can run 17 of 22 tests — the 15 SURVIVES plus the 2 CONDITIONAL. Only 5 (vminstance + the four kubernetes-*) truly need a bare-metal-like cluster, each for the same reason: nested KVM for KubeVirt VMs + DRBD storage. Suggested split: run the 17 non-virt tests on the fast containerized lane for PR feedback; keep the 5 virt/DRBD tests on the existing nested-QEMU lane. Reclaims most of the suite's wall-clock with no loss of coverage that actually depends on the heavyweight environment.

  5. myasnikovdaniil commented on Jul 9, 2026

    @myasnikovdaniil
    myasnikovdaniil
    on Jul 9, 2026
    ContributorAuthor

    Follow-up: this is an enabledPackages profile + a storage/CNI swap, not a new bundle

    Correcting my earlier "someone has to author a bundle" — cozystack already has the granular mechanism. All four bundles (system / iaas / paas / naas) have per-bundle enabled flags, and on top of that bundles.enabledPackages / bundles.disabledPackages give per-package control — which is exactly what cozypkg add/del drives. So the lightweight path is:

    core-only platform (all bundles enabled: false) → cozypkg add only the packages that survive.

    The catch: core-only has no CNI and no StorageClass

    Both normally arrive via the system bundle:

    • cozystack.networking = kube-ovn + cilium (OVS kernel module)
    • cozystack.linstor = DRBD + ZFS

    Neither runs in talos-in-docker. So the profile must substitute two things — this is the only part that isn't just cozypkg add:

    1. Default StorageClass → local-path/hostpath (mandatory: the 15 survivors all use storageClass: "" = default SC and need some PV).
    2. CNI → cilium-only (drop kube-ovn/OVS). Cilium runs on the host kernel (eBPF), and this is also what makes the 2 CONDITIONAL tests (gateway, securitygroup) pass. Note: today's networking package only ships kubeovn-cilium and kubeovn-cilium-generic variants — a cilium-only variant is the one genuinely new piece of work here.

    Result

    With storage + CNI substituted, the e2e values just set bundles.enabledPackages to the surviving app set:

    • Enable: the 15 SURVIVES packages (+ gateway, securitygroup once cilium-only is in).
    • Never enable the iaas bundle → KubeVirt/CDI + the CAPK-VM tenant clusters stay off, so the 5 NEEDS-REAL tests (vminstance + the four kubernetes-*) drop out for free — no nested KVM, no DRBD.

    Net: 17/22 tests on a container cluster, driven by an enabledPackages list + a local-path default SC + a cilium-only networking variant. No new bundle, no per-component conditional logic.

  6. github-actions
    added
    triage/acceptedIndicates an issue is ready to be actively worked on
    on Aug 12, 2026
  7. myasnikovdaniil
    added 2 commits that reference this issue on Sep 10, 2026

    docs(e2e): add lightweight container-cluster e2e lane proposal

    9f0e3ca

    feat(system/local-storage): OpenEBS LocalPV hostpath default StorageC…

    c17a81a
  8. Aleksei Sviridkin (lexfrei)
    added
    priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete
    on Sep 23, 2026

  9. myasnikovdaniil commented on Oct 1, 2026

    @myasnikovdaniil
    myasnikovdaniil
    on Oct 1, 2026
    ContributorAuthor

    done via #4020

  10. myasnikovdaniil
    closed this as completedon Oct 1, 2026
  11. Aleksei Sviridkin (lexfrei)
    added a commit that references this issue on Oct 7, 2026

    feat(local-storage): add opt-in OpenEBS LocalPV hostpath package (#3266)

    6192d58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/ciIssues or PRs related to CI workflows, GitHub Actions, automationarea/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/featureCategorizes issue or PR as related to a new featurepriority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completetriage/acceptedIndicates an issue is ready to be actively worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions