|
| 1 | +# Cozymarketplace: backend, private sources, and publication validation (supplementary to #18) |
| 2 | + |
| 3 | +- **Title:** `Cozymarketplace — Phase 1 backend, private repository support, and publication validation` |
| 4 | +- **Author(s):** `@IvanHunters` |
| 5 | +- **Date:** `2026-06-25` |
| 6 | +- **Status:** Draft |
| 7 | +- **Supplements:** [`cozystack/community#18`](https://github.com/cozystack/community/pull/18) by `@kvaps` |
| 8 | +- **Related code:** [`cozystack/cozystack#2472`](https://github.com/cozystack/cozystack/pull/2472), [`cozystack/cozystack#2455`](https://github.com/cozystack/cozystack/issues/2455) |
| 9 | + |
| 10 | +## Overview |
| 11 | + |
| 12 | +This proposal is supplementary, not competing, to `#18`. It accepts the repository-centric model — the install and version unit is the External-Apps repository as a versioned OCI artifact — and fills in three concrete pieces that `#18` lists as open questions or leaves implicit: the in-cluster backend the dashboard talks to, the `PackageSourceRef` change that makes private repositories work in one command, and the publication validation that gates submissions to the meta-index. Per-package version pinning remains out of scope, siding with `#18`. |
| 13 | + |
| 14 | +## Scope and related proposals |
| 15 | + |
| 16 | +`#18` defines the meta-index, the repository-as-unit model, and the `cozypkg` repository commands. This proposal does not modify any of them. It specifies the backend that the dashboard view in `#18` implies, the credential plumbing private taps require, and the CI gate that lets the meta-index accept community submissions safely. |
| 17 | + |
| 18 | +## Context |
| 19 | + |
| 20 | +`#18` already covers how Cozystack ships External Apps today (`PackageSource` + `Package` + Flux `HelmRelease`) and how the repository-as-unit model layers on top. Three pieces in that picture remain underspecified. |
| 21 | + |
| 22 | +First, `#18` says the catalog should be visible inside Cozystack so operators can install apps from there, with the dashboard handling enable/disable in Phase 1. It does not specify how the dashboard obtains the catalog data — walking k8s resources from the browser is impractical, and an aggregating server-side component is required. |
| 23 | + |
| 24 | +Second, `#18` lists private repositories as an open question. The current `PackageSourceRef` CRD has no `secretRef` field, so tapping a private repository requires creating a `Secret`, then a `GitRepository`/`OCIRepository` with `secretRef` set, then a `PackageSource` referencing it — three out-of-band steps that defeat the one-command tap UX. |
| 25 | + |
| 26 | +Third, `#18` lists publication validation as an open question. Without a gate, a community-submitted index entry can point at an artifact that does not pull, contains malformed `marketplace.yaml`, or ships a chart that fails `helm lint`, and the failure surfaces only at install time in someone else's cluster. |
| 27 | + |
| 28 | +## Goals |
| 29 | + |
| 30 | +Provide an in-cluster backend that powers the dashboard marketplace view: a small set of endpoints in `cozystack-api`, a `TapIndex` cache controller, and well-defined RBAC for connecting and disconnecting taps. |
| 31 | + |
| 32 | +Add a `secretRef` field to the `PackageSourceRef` CRD so that connecting a private repository becomes a single command, with the reconciler materializing the underlying Flux source with the same credential. |
| 33 | + |
| 34 | +Provide a `cozypkg validate` subcommand that lints a candidate marketplace repository offline, and reuse it in a GitHub Actions workflow that gates PRs to the meta-index repository. |
| 35 | + |
| 36 | +Keep the existing External-Apps pipeline unchanged. All additions are additive: existing public installs see zero behaviour change when the new fields are left empty. |
| 37 | + |
| 38 | +## Non-goals |
| 39 | + |
| 40 | +Per-package version pinning. Out of scope, deferred per `#18`. |
| 41 | + |
| 42 | +A dynamic external catalog backend. The external browse surface — a public site that lists all submitted repositories — is designed as a static site generated from the meta-index. No database, no runtime API, no telemetry. The publication CI is a GitHub Actions workflow, not a service. |
| 43 | + |
| 44 | +Commercial / paid-operator marketplace. Acknowledged in `#18` as a later, separate marketplace. |
| 45 | + |
| 46 | +Cross-tap dependency resolution with version constraints. Within a single tapped repository `dependsOn` already works; cross-tap version-constrained resolution is not addressed here. |
| 47 | + |
| 48 | +## Design |
| 49 | + |
| 50 | +### Backend endpoints in `cozystack-api` |
| 51 | + |
| 52 | +The dashboard reads marketplace state through a small set of endpoints layered into `cozystack-api`, which already mediates dashboard access to platform CRs and existing auth/RBAC. No new component, no new CRDs — the marketplace state is fully derived from `PackageSource` plus the parsed artifact contents. |
| 53 | + |
| 54 | +| Method | Path | Purpose | |
| 55 | +|---|---|---| |
| 56 | +| `GET` | `/marketplace/taps` | List connected taps with metadata. | |
| 57 | +| `GET` | `/marketplace/taps/{name}/packages` | Packages exposed by one tap. | |
| 58 | +| `GET` | `/marketplace/search?q=<query>` | Search across all taps by name, tag, description. | |
| 59 | +| `POST` | `/marketplace/taps` | Connect a tap; creates `PackageSource` and, when given, the `Secret`. | |
| 60 | +| `DELETE` | `/marketplace/taps/{name}` | Disconnect a tap. Lifecycle of installed `Package` CRs is an open question (see below). | |
| 61 | + |
| 62 | +A `TapIndex` cache controller in the same binary watches `OCIRepository.status.artifact.revision`, pulls the parsed `marketplace.yaml` on each revision change, and serves the GET endpoints from memory. Without it, every search would hit OCI per request. Cluster-admin is required for `POST` and `DELETE`, matching the existing `Package` cluster-scoped model. `GET` is open to any authenticated user so tenant-admins can browse. |
| 63 | + |
| 64 | +### Private repository support — `SecretRef` in `PackageSourceRef` |
| 65 | + |
| 66 | +The new field is additive and nil-default: |
| 67 | + |
| 68 | +```go |
| 69 | +type PackageSourceRef struct { |
| 70 | + Kind, Name, Namespace, Path string |
| 71 | + SecretRef *corev1.LocalObjectReference // NEW; nil preserves current behaviour |
| 72 | +} |
| 73 | +``` |
| 74 | + |
| 75 | +The `packagesource-reconciler` sets `spec.secretRef` on the materialized Flux source when `SecretRef != nil`. Secret format depends on source kind, matching what Flux source-controller already documents: `kubernetes.io/dockerconfigjson` for OCI; Opaque with `username`+`password` or `bearerToken` for Git over HTTPS; Opaque with `identity` (PEM private key) and `known_hosts` for Git over SSH. The Secret must exist in `cozy-system` before the reconciler runs; otherwise Flux reports a failed condition until it appears. This is symmetric with the platform-source change already in flight in `cozystack/cozystack#2472` — that PR closed the gap for the bootstrap platform source; this CRD field closes the same gap for every user-tapped repository. |
| 76 | + |
| 77 | +### Publication validation — `cozypkg validate` and CI gate |
| 78 | + |
| 79 | +A new `cozypkg validate <repository-url>[@<tag>]` subcommand pulls the artifact (or fails); parses `marketplace.yaml` against the published JSON schema; for each declared `PackageSource`, runs `helm lint` on every `Component.Path`; verifies that every `dependsOn` resolves either inside the same repository or in a known cozystack-shipped source; flags components with `install.Privileged: true` so the operator sees a privileged badge in the dashboard before install; and, with `--require-signature`, performs cosign verification. |
| 80 | + |
| 81 | +The same logic runs in a GitHub Actions workflow in the meta-index repository, triggered on PRs that add or modify an entry. The workflow resolves `source.url` and `source.tag` from the diff, runs the validator, annotates the PR with the report, and blocks merge on hard failures. It does not replace maintainer review; it lowers the cost of that review by surfacing structural failures up front. |
| 82 | + |
| 83 | +### External catalog (design only, deferred) |
| 84 | + |
| 85 | +The public browse surface — a site at e.g. `marketplace.cozystack.io` — is designed as a static site generated from the meta-index on every meta-index merge. No runtime backend, no database. Hugo, Astro, or MkDocs all fit. Client-side search via Pagefind. Icons, screenshots, READMEs are served by reference into the OCI artifacts produced by repository authors. Implementation is explicitly deferred; flagged here only to confirm no dynamic backend is required. |
| 86 | + |
| 87 | +## User-facing changes |
| 88 | + |
| 89 | +CLI: a new `cozypkg validate` subcommand; existing `tap` / `add` / `list` semantics unchanged. |
| 90 | + |
| 91 | +CRD: a new optional `PackageSourceRef.secretRef` field; nil preserves current behaviour. |
| 92 | + |
| 93 | +Dashboard: a marketplace view backed by the new `/marketplace/*` endpoints. Tapped repositories are visible, packages browsable, installs flow through the existing Package-creation path. |
| 94 | + |
| 95 | +Meta-index repository: a new `validate.yaml` GitHub Actions workflow. |
| 96 | + |
| 97 | +## Upgrade and rollback compatibility |
| 98 | + |
| 99 | +Strictly additive. `PackageSourceRef.SecretRef = nil` produces the same Flux source manifest as today, so existing tapped public repositories see no change. The new endpoints are net-new paths under `/marketplace/*`; no existing route changes shape. The `TapIndex` cache starts empty and populates from existing `PackageSource` resources at startup; nothing else relies on its presence. Rolling back to a `cozystack-api` version without the marketplace endpoints leaves the cluster fully functional — only the dashboard marketplace view goes blank. No migration script is required for the `secretRef` field alone. |
| 100 | + |
| 101 | +## Security |
| 102 | + |
| 103 | +The trust boundary `#18` already describes — tapping a third-party repository runs that repository's charts in the operator's cluster — is preserved. The publication CI gate is the first line of defence on the meta-index side, but it does not endorse content; it only checks structural validity. Maintainer review remains required for every PR. |
| 104 | + |
| 105 | +Privileged components are surfaced both at validation time (the CI workflow emits a warning and labels the entry) and at install time (`cozypkg add` prompts for confirmation unless `--allow-privileged` is passed). |
| 106 | + |
| 107 | +The new `secretRef` references a Secret by name; the controller never reads or logs the credentials. The Secret is consumed by Flux source-controller under its existing RBAC. |
| 108 | + |
| 109 | +## Failure and edge cases |
| 110 | + |
| 111 | +`marketplace.yaml` malformed inside the artifact → cache controller logs the parse error, marks the tap as `Degraded`, and returns the last-known-good payload. Operator sees a clear error in the dashboard. |
| 112 | + |
| 113 | +Secret removed while still referenced → reconcile produces the source with a broken `secretRef`; Flux surfaces the failed pull condition, which the marketplace endpoint forwards. |
| 114 | + |
| 115 | +Tap removed while packages from it remain installed → see open question. Default behaviour proposed: leave installed `Package` resources in place (they carry `helm.sh/resource-policy: keep`) and mark them as `OrphanedSource` in the dashboard until the operator re-taps or removes them with `cozypkg del`. |
| 116 | + |
| 117 | +`POST /marketplace/taps` with a `secretRef` pointing at a non-existent Secret → endpoint accepts the request (Flux can recover later when the Secret appears), but the dashboard shows the `Secret not found in cozy-system` condition sourced from Flux. |
| 118 | + |
| 119 | +## Testing |
| 120 | + |
| 121 | +Unit: new endpoints against a fake k8s client; cache controller refresh logic against a fake OCI source; `cozypkg validate` golden-output tests against fixture repositories, both valid and intentionally broken. Integration: end-to-end test that taps a fixture private OCI repository with a `dockerconfigjson` Secret on a kind cluster and verifies the dashboard endpoint surfaces the parsed packages. CI workflow is exercised on a fixture entry pointing at `cozystack/external-apps-example`. |
| 122 | + |
| 123 | +## Rollout |
| 124 | + |
| 125 | +Four independent PRs in cozystack, two of them landable in parallel. `secretRef` in `PackageSourceRef` (~50 LOC plus a migration script if needed) and `cozypkg validate` plus the CI workflow (~600 LOC Go and ~200 LOC YAML/shell) can land before or after `#18`. The marketplace endpoints in `cozystack-api` (~400 LOC Go) and the `TapIndex` cache controller (~300 LOC Go) consume the `marketplace.yaml` shape introduced by `#18` and depend on it landing first. The external static catalog is a separate proposal under Aenix maintenance; designed-only here. |
| 126 | + |
| 127 | +## Open questions |
| 128 | + |
| 129 | +Tap-remove lifecycle. When a tap is disconnected and packages installed from it remain, do we orphan them with a dashboard marker, cascade-delete, or block deletion until the operator confirms? Default proposed: orphan with marker. |
| 130 | + |
| 131 | +Endpoint hosting. Marketplace endpoints inside `cozystack-api` versus a new `cozystack-marketplace-controller`. Default proposed: extend `cozystack-api`. |
| 132 | + |
| 133 | +External catalog hostname. `marketplace.cozystack.io`, `apps.cozystack.io`, `hub.cozystack.io`. Aenix-side decision. |
| 134 | + |
| 135 | +Privileged-tap policy. Should a cluster be configurable to refuse tapping any repository whose components declare `Privileged: true`, regardless of operator approval? Deferred to a follow-up. |
| 136 | + |
| 137 | +Verified-vs-community labelling. Who maintains the verified allowlist and on what criteria. Maintainer-side governance question. |
| 138 | + |
| 139 | +## Alternatives considered |
| 140 | + |
| 141 | +Building the marketplace surface entirely in the dashboard frontend against the existing `cozystack-api` CR proxy. Rejected because the catalog requires aggregation across multiple artifacts plus parsed `marketplace.yaml` content; doing this in the browser per page load is too slow and would re-implement caching client-side. |
| 142 | + |
| 143 | +A dedicated CRD for marketplace state (`Tap`, `TapEntry`, ...). Rejected because all of the necessary state is derivable from `PackageSource` plus the parsed artifact; new CRDs raise migration cost without changing capability. |
| 144 | + |
| 145 | +A dynamic external catalog backend (database plus API). Rejected as Phase 1 over-engineering. The meta-index is already a single source of truth; a static generator is sufficient and dramatically lower-cost. |
| 146 | + |
| 147 | +Inline credentials in `PackageSourceRef` (e.g. `username`/`password` directly on the CR field). Rejected on principle — credentials live in Secrets, never on CR fields, to remain encryptable at rest and to flow through existing RBAC. |
| 148 | + |
| 149 | +Per-package version pinning as part of this proposal. Out of scope to stay aligned with `#18`. A separate proposal exists in the author's internal notes and may be revived if a concrete operational case emerges (CVE in a single package of a large repository, partner publishing on independent cadence). |
0 commit comments