Skip to content

Commit c958f39

Browse files
committed
design-proposal: cozymarketplace supplementary — backend, private sources, publication validation
Supplementary to #18 by @kvaps. Accepts the repository-centric model and fills three concrete pieces that #18 lists as open questions or leaves implicit: in-cluster backend for the dashboard marketplace view, secretRef on PackageSourceRef for one-command private taps, and a cozypkg validate subcommand plus CI gate for publication validation. Per-package version pinning remains out of scope, siding with #18. Signed-off-by: IvanHunters <[email protected]>
1 parent fbfc6ba commit c958f39

1 file changed

Lines changed: 149 additions & 0 deletions

File tree

  • design-proposals/cozymarketplace-supplementary
Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Cozymarketplace: backend, private sources, and publication validation (supplementary to #18)
2+
3+
- **Title:** `Cozymarketplace — Phase 1 backend, private repository support, and publication validation`
4+
- **Author(s):** `@IvanHunters`
5+
- **Date:** `2026-06-25`
6+
- **Status:** Draft
7+
- **Supplements:** [`cozystack/community#18`](https://github.com/cozystack/community/pull/18) by `@kvaps`
8+
- **Related code:** [`cozystack/cozystack#2472`](https://github.com/cozystack/cozystack/pull/2472), [`cozystack/cozystack#2455`](https://github.com/cozystack/cozystack/issues/2455)
9+
10+
## Overview
11+
12+
This proposal is supplementary, not competing, to `#18`. It accepts the repository-centric model — the install and version unit is the External-Apps repository as a versioned OCI artifact — and fills in three concrete pieces that `#18` lists as open questions or leaves implicit: the in-cluster backend the dashboard talks to, the `PackageSourceRef` change that makes private repositories work in one command, and the publication validation that gates submissions to the meta-index. Per-package version pinning remains out of scope, siding with `#18`.
13+
14+
## Scope and related proposals
15+
16+
`#18` defines the meta-index, the repository-as-unit model, and the `cozypkg` repository commands. This proposal does not modify any of them. It specifies the backend that the dashboard view in `#18` implies, the credential plumbing private taps require, and the CI gate that lets the meta-index accept community submissions safely.
17+
18+
## Context
19+
20+
`#18` already covers how Cozystack ships External Apps today (`PackageSource` + `Package` + Flux `HelmRelease`) and how the repository-as-unit model layers on top. Three pieces in that picture remain underspecified.
21+
22+
First, `#18` says the catalog should be visible inside Cozystack so operators can install apps from there, with the dashboard handling enable/disable in Phase 1. It does not specify how the dashboard obtains the catalog data — walking k8s resources from the browser is impractical, and an aggregating server-side component is required.
23+
24+
Second, `#18` lists private repositories as an open question. The current `PackageSourceRef` CRD has no `secretRef` field, so tapping a private repository requires creating a `Secret`, then a `GitRepository`/`OCIRepository` with `secretRef` set, then a `PackageSource` referencing it — three out-of-band steps that defeat the one-command tap UX.
25+
26+
Third, `#18` lists publication validation as an open question. Without a gate, a community-submitted index entry can point at an artifact that does not pull, contains malformed `marketplace.yaml`, or ships a chart that fails `helm lint`, and the failure surfaces only at install time in someone else's cluster.
27+
28+
## Goals
29+
30+
Provide an in-cluster backend that powers the dashboard marketplace view: a small set of endpoints in `cozystack-api`, a `TapIndex` cache controller, and well-defined RBAC for connecting and disconnecting taps.
31+
32+
Add a `secretRef` field to the `PackageSourceRef` CRD so that connecting a private repository becomes a single command, with the reconciler materializing the underlying Flux source with the same credential.
33+
34+
Provide a `cozypkg validate` subcommand that lints a candidate marketplace repository offline, and reuse it in a GitHub Actions workflow that gates PRs to the meta-index repository.
35+
36+
Keep the existing External-Apps pipeline unchanged. All additions are additive: existing public installs see zero behaviour change when the new fields are left empty.
37+
38+
## Non-goals
39+
40+
Per-package version pinning. Out of scope, deferred per `#18`.
41+
42+
A dynamic external catalog backend. The external browse surface — a public site that lists all submitted repositories — is designed as a static site generated from the meta-index. No database, no runtime API, no telemetry. The publication CI is a GitHub Actions workflow, not a service.
43+
44+
Commercial / paid-operator marketplace. Acknowledged in `#18` as a later, separate marketplace.
45+
46+
Cross-tap dependency resolution with version constraints. Within a single tapped repository `dependsOn` already works; cross-tap version-constrained resolution is not addressed here.
47+
48+
## Design
49+
50+
### Backend endpoints in `cozystack-api`
51+
52+
The dashboard reads marketplace state through a small set of endpoints layered into `cozystack-api`, which already mediates dashboard access to platform CRs and existing auth/RBAC. No new component, no new CRDs — the marketplace state is fully derived from `PackageSource` plus the parsed artifact contents.
53+
54+
| Method | Path | Purpose |
55+
|---|---|---|
56+
| `GET` | `/marketplace/taps` | List connected taps with metadata. |
57+
| `GET` | `/marketplace/taps/{name}/packages` | Packages exposed by one tap. |
58+
| `GET` | `/marketplace/search?q=<query>` | Search across all taps by name, tag, description. |
59+
| `POST` | `/marketplace/taps` | Connect a tap; creates `PackageSource` and, when given, the `Secret`. |
60+
| `DELETE` | `/marketplace/taps/{name}` | Disconnect a tap. Lifecycle of installed `Package` CRs is an open question (see below). |
61+
62+
A `TapIndex` cache controller in the same binary watches `OCIRepository.status.artifact.revision`, pulls the parsed `marketplace.yaml` on each revision change, and serves the GET endpoints from memory. Without it, every search would hit OCI per request. Cluster-admin is required for `POST` and `DELETE`, matching the existing `Package` cluster-scoped model. `GET` is open to any authenticated user so tenant-admins can browse.
63+
64+
### Private repository support — `SecretRef` in `PackageSourceRef`
65+
66+
The new field is additive and nil-default:
67+
68+
```go
69+
type PackageSourceRef struct {
70+
Kind, Name, Namespace, Path string
71+
SecretRef *corev1.LocalObjectReference // NEW; nil preserves current behaviour
72+
}
73+
```
74+
75+
The `packagesource-reconciler` sets `spec.secretRef` on the materialized Flux source when `SecretRef != nil`. Secret format depends on source kind, matching what Flux source-controller already documents: `kubernetes.io/dockerconfigjson` for OCI; Opaque with `username`+`password` or `bearerToken` for Git over HTTPS; Opaque with `identity` (PEM private key) and `known_hosts` for Git over SSH. The Secret must exist in `cozy-system` before the reconciler runs; otherwise Flux reports a failed condition until it appears. This is symmetric with the platform-source change already in flight in `cozystack/cozystack#2472` — that PR closed the gap for the bootstrap platform source; this CRD field closes the same gap for every user-tapped repository.
76+
77+
### Publication validation — `cozypkg validate` and CI gate
78+
79+
A new `cozypkg validate <repository-url>[@<tag>]` subcommand pulls the artifact (or fails); parses `marketplace.yaml` against the published JSON schema; for each declared `PackageSource`, runs `helm lint` on every `Component.Path`; verifies that every `dependsOn` resolves either inside the same repository or in a known cozystack-shipped source; flags components with `install.Privileged: true` so the operator sees a privileged badge in the dashboard before install; and, with `--require-signature`, performs cosign verification.
80+
81+
The same logic runs in a GitHub Actions workflow in the meta-index repository, triggered on PRs that add or modify an entry. The workflow resolves `source.url` and `source.tag` from the diff, runs the validator, annotates the PR with the report, and blocks merge on hard failures. It does not replace maintainer review; it lowers the cost of that review by surfacing structural failures up front.
82+
83+
### External catalog (design only, deferred)
84+
85+
The public browse surface — a site at e.g. `marketplace.cozystack.io` — is designed as a static site generated from the meta-index on every meta-index merge. No runtime backend, no database. Hugo, Astro, or MkDocs all fit. Client-side search via Pagefind. Icons, screenshots, READMEs are served by reference into the OCI artifacts produced by repository authors. Implementation is explicitly deferred; flagged here only to confirm no dynamic backend is required.
86+
87+
## User-facing changes
88+
89+
CLI: a new `cozypkg validate` subcommand; existing `tap` / `add` / `list` semantics unchanged.
90+
91+
CRD: a new optional `PackageSourceRef.secretRef` field; nil preserves current behaviour.
92+
93+
Dashboard: a marketplace view backed by the new `/marketplace/*` endpoints. Tapped repositories are visible, packages browsable, installs flow through the existing Package-creation path.
94+
95+
Meta-index repository: a new `validate.yaml` GitHub Actions workflow.
96+
97+
## Upgrade and rollback compatibility
98+
99+
Strictly additive. `PackageSourceRef.SecretRef = nil` produces the same Flux source manifest as today, so existing tapped public repositories see no change. The new endpoints are net-new paths under `/marketplace/*`; no existing route changes shape. The `TapIndex` cache starts empty and populates from existing `PackageSource` resources at startup; nothing else relies on its presence. Rolling back to a `cozystack-api` version without the marketplace endpoints leaves the cluster fully functional — only the dashboard marketplace view goes blank. No migration script is required for the `secretRef` field alone.
100+
101+
## Security
102+
103+
The trust boundary `#18` already describes — tapping a third-party repository runs that repository's charts in the operator's cluster — is preserved. The publication CI gate is the first line of defence on the meta-index side, but it does not endorse content; it only checks structural validity. Maintainer review remains required for every PR.
104+
105+
Privileged components are surfaced both at validation time (the CI workflow emits a warning and labels the entry) and at install time (`cozypkg add` prompts for confirmation unless `--allow-privileged` is passed).
106+
107+
The new `secretRef` references a Secret by name; the controller never reads or logs the credentials. The Secret is consumed by Flux source-controller under its existing RBAC.
108+
109+
## Failure and edge cases
110+
111+
`marketplace.yaml` malformed inside the artifact → cache controller logs the parse error, marks the tap as `Degraded`, and returns the last-known-good payload. Operator sees a clear error in the dashboard.
112+
113+
Secret removed while still referenced → reconcile produces the source with a broken `secretRef`; Flux surfaces the failed pull condition, which the marketplace endpoint forwards.
114+
115+
Tap removed while packages from it remain installed → see open question. Default behaviour proposed: leave installed `Package` resources in place (they carry `helm.sh/resource-policy: keep`) and mark them as `OrphanedSource` in the dashboard until the operator re-taps or removes them with `cozypkg del`.
116+
117+
`POST /marketplace/taps` with a `secretRef` pointing at a non-existent Secret → endpoint accepts the request (Flux can recover later when the Secret appears), but the dashboard shows the `Secret not found in cozy-system` condition sourced from Flux.
118+
119+
## Testing
120+
121+
Unit: new endpoints against a fake k8s client; cache controller refresh logic against a fake OCI source; `cozypkg validate` golden-output tests against fixture repositories, both valid and intentionally broken. Integration: end-to-end test that taps a fixture private OCI repository with a `dockerconfigjson` Secret on a kind cluster and verifies the dashboard endpoint surfaces the parsed packages. CI workflow is exercised on a fixture entry pointing at `cozystack/external-apps-example`.
122+
123+
## Rollout
124+
125+
Four independent PRs in cozystack, two of them landable in parallel. `secretRef` in `PackageSourceRef` (~50 LOC plus a migration script if needed) and `cozypkg validate` plus the CI workflow (~600 LOC Go and ~200 LOC YAML/shell) can land before or after `#18`. The marketplace endpoints in `cozystack-api` (~400 LOC Go) and the `TapIndex` cache controller (~300 LOC Go) consume the `marketplace.yaml` shape introduced by `#18` and depend on it landing first. The external static catalog is a separate proposal under Aenix maintenance; designed-only here.
126+
127+
## Open questions
128+
129+
Tap-remove lifecycle. When a tap is disconnected and packages installed from it remain, do we orphan them with a dashboard marker, cascade-delete, or block deletion until the operator confirms? Default proposed: orphan with marker.
130+
131+
Endpoint hosting. Marketplace endpoints inside `cozystack-api` versus a new `cozystack-marketplace-controller`. Default proposed: extend `cozystack-api`.
132+
133+
External catalog hostname. `marketplace.cozystack.io`, `apps.cozystack.io`, `hub.cozystack.io`. Aenix-side decision.
134+
135+
Privileged-tap policy. Should a cluster be configurable to refuse tapping any repository whose components declare `Privileged: true`, regardless of operator approval? Deferred to a follow-up.
136+
137+
Verified-vs-community labelling. Who maintains the verified allowlist and on what criteria. Maintainer-side governance question.
138+
139+
## Alternatives considered
140+
141+
Building the marketplace surface entirely in the dashboard frontend against the existing `cozystack-api` CR proxy. Rejected because the catalog requires aggregation across multiple artifacts plus parsed `marketplace.yaml` content; doing this in the browser per page load is too slow and would re-implement caching client-side.
142+
143+
A dedicated CRD for marketplace state (`Tap`, `TapEntry`, ...). Rejected because all of the necessary state is derivable from `PackageSource` plus the parsed artifact; new CRDs raise migration cost without changing capability.
144+
145+
A dynamic external catalog backend (database plus API). Rejected as Phase 1 over-engineering. The meta-index is already a single source of truth; a static generator is sufficient and dramatically lower-cost.
146+
147+
Inline credentials in `PackageSourceRef` (e.g. `username`/`password` directly on the CR field). Rejected on principle — credentials live in Secrets, never on CR fields, to remain encryptable at rest and to flow through existing RBAC.
148+
149+
Per-package version pinning as part of this proposal. Out of scope to stay aligned with `#18`. A separate proposal exists in the author's internal notes and may be revived if a concrete operational case emerges (CVE in a single package of a large repository, partner publishing on independent cadence).

0 commit comments

Comments
 (0)