Skip to content

Bound or restructure releases/manifest.json growth (append-forever history) #1310

Description

@Aaronontheweb

Summary

feeds/scripts/generate-release-manifest.sh preserves every prior release in the releases[] array forever. The live manifest.json is already 68 KB across 64 releases (~1 KB/release) and grows monotonically. This isn't urgent, but it's append-forever by design and worth a deliberate strategy before it gets large.

Surfaced while designing the beta channel (#1027); we deliberately left it as-is there to keep that PR focused. The beta channel will accelerate growth (multiple x.y.z-beta.n entries per stable), which is why it's worth tracking now.

Why it matters (eventually, not yet)

  • install.sh no-jq fallback slurps the entire manifest (tr '\n' ' ' + grep) — gets slower/more fragile as the file grows.
  • C# update-check (UpdateCheckService.FetchVerifiedManifestAsync) downloads + signature-verifies + deserializes the whole file on each check (24h-cached, so low impact — but still the whole blob).
  • Monotonic growth is a smell; better to pick a bounded model on our terms.

Current numbers

  • 64 releases → 68 KB. Projected: ~200 releases ≈ 0.2 MB, ~500 ≈ 0.5 MB. With betas, the entry count climbs faster.

Ideas (not yet decided)

  1. Bound releases[] to a recent window (e.g. keep last N≈25), always retaining whatever latest and latestPrerelease reference even if they'd age out. Cheapest — fits inside the existing generator's Python. Binaries stay in R2 (netclaw-releases/{version}/…) and GitHub Releases permanently, so nothing is lost; the only effect is an old NETCLAW_VERSION=<ancient> pin can't resolve its sha256 from the hot manifest.
  2. Split hot vs cold: a small manifest.json (pointers + referenced entries + recent window) for the install/update hot path, plus a cold manifest-history.json (or per-version sidecars) fetched only on explicit old-version pins.
  3. Per-version checksum sidecars ({version}/checksums.json) so a pinned install verifies against the sidecar and the main manifest shrinks to pointers + minimal recent entries.
  4. Harden the no-jq parse path regardless of which option, so it doesn't depend on whole-file slurping.

Constraints to preserve

  • Keep schemaVersion: 1 backward-compatible (older clients ignore additive fields).
  • Keep minisign signature verification intact.
  • Don't break explicit pins for recent versions; immutable binaries remain available.

Related: #1027

Metadata

Metadata

Assignees

No one assigned

    Labels

    cleanupCode quality improvements and tech debt reductionenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions