Repository navigation
docs(design-proposals): add the platform and catalogs split proposal - #106
myasnikovdaniil wants to merge 4 commits into
Conversation
Split Cozystack into the platform and two catalogs, virtualization and services, with contracts K1-K6 between them, a contribution policy for new applications and a rollout in four steps. Supersedes community#43 and community#46. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
|
Strong proposal. Monorepo-first with separate repositories as the last step is the right call, the contracts are written out concretely instead of hand-waved, and the cut is backed by data. The boundary lint with a measurable zero-forbidden-edges criterion is the right spine to hang it on. Most of what follows is about sequencing against work already in flight, plus a few failure modes the contracts don't cover yet. I checked the current-state claims against Blocking (would want these resolved before it leaves Draft)
Non-blocking but worth addressing
Smaller
Questions
|
…ve them to authors A separate organization for community applications would have no access to the project's e2e runners and no CNCF rules, while still carrying the project's name. An application either meets the bar for one of our catalogs and lives in this tree until the catalogs move, or its author publishes it as their own catalog. An interim repository would only have to be moved again once the catalogs get their own. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
Where a new application goes is decided together with whether the project takes it, and that is a proposal. An e2e requirement would mostly re-test the upstream operator, and a fixed number of users is easy to game. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
| | **platform** | Core (operator, CRDs, `cozystack-api`, `cozystack-controller`, lineage webhook). System packages (network with multus as its option, storage, the host GPU stack, cert-manager, monitoring, Keycloak). Tenant and tenant modules. Shared operators (`postgres-operator`, `objectstorage-controller`). Object storage (the SeaweedFS tenant module, the COSI controller, the Bucket application). The backup framework as optional component. The dashboard | | ||
| | **virtualization** | VMInstance, VMDisk, VirtualPrivateCloud, tenant Kubernetes, kubernetes-nodes, computeplane. KubeVirt with its GPU device wiring, CDI, CAPI, Kamaji, forklift. gpu-operator and HAMi as addons inside tenant clusters. VM and worker images. The VM backup driver and Velero. CSI for tenant clusters | |
There was a problem hiding this comment.
The network layer can be separate from the virtualization just as much as storage.
|
|
||
| **Where a new component goes.** If it exists for VMs or for what runs on VMs, it belongs to virtualization, VM networks like VPC included. Multus stays in the platform as option of the network provider: it runs on top of the CNI config and has to match its plugin versions, and the cozyplane network builds VPC without it ([cozystack#4508](https://github.com/cozystack/cozystack/pull/4508)). Same for GPUs: the host GPU stack prepares nodes and is a platform setting, but handing GPUs to VMs and GPU addons of tenant clusters belong to virtualization. If it is finished software a tenant consumes, it belongs to services. If everything needs it, it belongs to the platform. | ||
|
|
||
| **Object storage stays in the platform.** The backup framework depends on COSI and the system bucket, the tenant chart enables SeaweedFS as tenant module, and both catalogs consume buckets: virtualization through Velero, services through Harbor, ClickHouse, CNPG and PSMDB. Moving it to services would add a platform → services edge. The bucket class names, `_namespace.seaweedfs` and the `cozy-backups-creds` secret become part of K2 and the backup SDK, so SeaweedFS stays the default provider but not the only one. |
There was a problem hiding this comment.
The backup framework doesn't have to depend on object storage. Some of the backup strategies that Cozystack ships make use of object storage and/or Velero, but if anyone writes a bespoke BackupStrategy to deliver a Backup to some other place, by some other protocol. they sidestep these dependencies completely.
| | 2 | The tenant-admin role lists every catalog kind, and the platform grants rights and network policies for KubeVirt | `packages/system/cozystack-basics/templates/clusterroles.yaml`, `packages/system/cozystack-api/templates/rbac.yaml`, `packages/apps/tenant/templates/networkpolicy.yaml` | Each `*-rd` chart ships the role for its kind through the existing aggregation labels. Virtualization ships own roles and policies | | ||
| | 3 | The tenant chart renders computeplane by name | `packages/apps/tenant/templates/computeplane.yaml` | Generic list of tenant modules driven by `ApplicationDefinition` | | ||
| | 4 | Option providers in core Go, 8 of 14 for virtualization | `pkg/registry/core/option/providers.go` | Declarative option sources shipped by the catalog | | ||
| | 5 | Warnings and name limits for Kubernetes, Kafka, Postgres and MariaDB in `cozystack-api` | `pkg/registry/apps/application/rest.go` | Remove, or a `ValidatingAdmissionPolicy` with `Warn` in the application chart | |
There was a problem hiding this comment.
Already done, the most problematic remaining limit is for kubernetes-nodes.
| | 4 | Option providers in core Go, 8 of 14 for virtualization | `pkg/registry/core/option/providers.go` | Declarative option sources shipped by the catalog | | ||
| | 5 | Warnings and name limits for Kubernetes, Kafka, Postgres and MariaDB in `cozystack-api` | `pkg/registry/apps/application/rest.go` | Remove, or a `ValidatingAdmissionPolicy` with `Warn` in the application chart | | ||
| | 6 | WorkloadMonitor knows CDI `DataVolume` | `internal/controller/workloadmonitor_controller.go` | Generic readiness gates declared by the application | | ||
| | 7 | VM import: an API group, a controller and a binary in the root Go module | `api/migration/v1alpha1/`, `internal/migrationcontroller/`, `cmd/migration-controller/` | Moves to virtualization whole | |
There was a problem hiding this comment.
That's interesting, the API group was properly renamed to forklift and it is very much VM-specific. The broader problem is that several binaries are also part of the root module, e.g. kubeovn-plunger.
|
|
||
| ### Contracts | ||
|
|
||
| The contracts are the seam of the split. Today the platform and applications rely on each other through names, keys and conventions written down nowhere. They hold only because everything ships at one version. Once the platform and catalogs have own versions, the dangerous failures are the silent ones: Helm ignores an unknown values key, a `Capabilities` check switches a template branch off, a `lookup` returns nothing, a webhook lets an object through without labels. Each contract below lists what it covers, so every such assumption is either declared and checked, or removed. |
There was a problem hiding this comment.
The contracts are the seam of the split.
I'm sorry, this isn't something written for humans. It took another LLM to understand that this probably means
The split requires explicit, versioned contracts between the platform and catalogs to ensure they remain compatible as they evolve independently.
| The contract: | ||
|
|
||
| - **Smaller first.** Helpers with one consumer move to that consumer: `keda`, `strings.hexToInt`, most of `_cozyconfig.tpl`. Helpers with no consumer are removed: `ipv4-cluster-cidr`, `ipv4-service-cidr` and `ipv4-join-cidr` read `_cluster` keys nothing writes, and only the library's own test chart calls them. What stays is the shared core: the tenant RBAC subject model, resource presets, image references, hostname and `required` helpers from K2. That is the public interface, listed in the contract. | ||
| - **Delivery.** `cozy-lib` is published as an OCI Helm library chart with every platform release, under the platform version. A catalog pins it in `Chart.yaml` `dependencies`, and `helm dependency build` puts the pinned version into the catalog artifact at build time. `cozypkg validate` checks the pin, `cozypkg init` scaffolds it. The cluster keeps copying what is in the artifact, so on the cluster side nothing changes. |
There was a problem hiding this comment.
Have you prototyped this? I recall attempting something along these lines last year and found that this makes local development quite a bit harder. The dependencies listed in Helm's Chart.yaml are either a local filesystem reference or an http URL. A reference that works for one mode (e.g. local development) would break on a live cluster. Or an http URL that works in one Cozystack installation breaks in e.g. an airgap installation.
| #### K6. Dashboard | ||
|
|
||
| What a catalog declares so the console shows its applications without a console release. | ||
|
|
||
| | Item | Contract | | ||
| |---|---| | ||
| | `AD.spec.dashboard` | the CRD schema. The console reads 5 of its 12 fields today (`category`, `description`, `icon`, `keysOrder`, `module`). The rest are read or declared deprecated. `tabs` becomes the typed list from step 1, `visibility` comes from [community#39](https://github.com/cozystack/community/pull/39) | | ||
| | schema extensions | `x-cozystack-options`, the new `x-cozystack-widget`, and the option source annotations, in the K1 extension schema | | ||
| | `core.cozystack.io` resources the console uses | options, tenant namespaces, tenant modules, tenant secrets, taps, and their served OpenAPI | | ||
| | application labels and release name | from K1 | | ||
| | sidebar section object | a CRD with a title, a list of CRDs and a placement | | ||
| | RBAC | each catalog grants tenants read access to its CRD schemas and its subresources through the tenant aggregation labels | | ||
|
|
||
| **Check:** the console has unit tests that render tabs, actions and sections from fixture ADs of every platform version it supports. |
There was a problem hiding this comment.
The spirit is right, but this is worth a design proposal of its own.
| - Application joins our catalog when all of these hold: a maintainer or reviewer takes it in CODEOWNERS, licenses of the application and its operator are acceptable for a CNCF project, it has tests for what the chart adds on top of the upstream operator, its upstream operator is maintained, and users need it, as the proposal shows. | ||
| - Until step 3 our catalogs live in `cozystack/cozystack`, so accepted application goes into its catalog directory there, under the coupling rule below, and moves together with the catalog later. We don't create interim repository for it. | ||
| - Any other application is published by its author as a catalog from author's own repository and installed through the marketplace. The project does not host such catalogs and has no separate organization for them. | ||
| - Recommended layout for such repository is `PackageSource` with `cozypkg`, the same format our own catalogs use after step 2. The older layout from `cozystack/external-apps-example`, a Git repository with a bootstrap `HelmRelease`, keeps working but is not recommended for new catalogs. |
There was a problem hiding this comment.
How to add a compatible app to Cozystack is a contract in itself and should be documented. Off the top of my head it is indeed a PackageSource, a "system" package like newapp-rd with the ApplicationDefinition and an underlying Helm chart along the lines of packages/apps/newapp. Along the way we should also document how someone running an airgap installation or simply an installation with stricter security requirements could vendor a third-party app, so their Cozystack never explicitly reaches out to the third-party author's catalog, but only to their vendoring repo.
| **10. The API watches ApplicationDefinitions** (K1). | ||
| - Today `cozystack-api` reads all ADs once at start, and the controller restarts the API Deployment when an AD changes. | ||
| - The API watches ADs and adds or removes kinds at runtime. An error in one AD disables that kind with a status condition, and step 0 already stops it from taking down the whole API. The controller stops restarting the API. | ||
| - The core derives `chartRef` from the source and component instead of each AD copying the artifact name formula. | ||
| - Done when: installing or upgrading a catalog changes the served kinds without restarting `cozystack-api`. |
There was a problem hiding this comment.
This work seems unrelated to the broader goals of the proposal. The API server is restarted on AD changes today because it's a fork of sample-apiserver that carries a lot of initialization code that noone ever untangled to start supporting runtime reconfiguration. This improvement, subject to the availability of manpower can be done at any time.
| - a new application needs a platform release for tenant RBAC | ||
| - a new VM tab needs a console release | ||
|
|
||
| We would pay the full cost of the split (CI, versions, release sync) and reach none of the goals. Rejected. |
There was a problem hiding this comment.
You'd be able to stop running the massive E2E suite though. That seems like a pretty significant win.
This PR adds design proposal for splitting cozystack into platform and two catalogs, virtualization and services. It supersedes community#43 and community#46, what is taken from each is listed in the Scope section, both can be closed after this one is accepted.
Main part for review is contracts K1-K6 between platform and catalogs, they are the seam of the split and the rest depends on them. Contracts have no versions of their own, catalog declares minimum platform version and the properties it needs from network, storage and GPU.
Proposal also sets contribution policy for new applications: they go to separate catalog repositories and connect through
PackageSourceandcozypkg(marketplace from 1.7). Step 0 work starts before acceptance and will be tracked in an epic in cozystack/cozystack.Before review
decisions/directory, or says below why none is needed.New proposal, it does not revise a merged one.
DCO
git commit --signoff).