Skip to content

fix(api): a malformed ApplicationDefinition keeps cozystack-api from starting #4976

Open

Description

@myasnikovdaniil

Summary

resourceConfigFrom (start.go L284-L298) builds one resource per ApplicationDefinition and returns first error it hits. Error comes from buildResourceFromCRD, which fails when release.cozystack.io/* Helm timeout annotation doesn't parse (config.ParseHelmTimeoutAnnotation, start.go L362-L380), and Complete passes it up, so API server doesn't start.

This is on purpose. Comment above the parsing says bad annotation should surface as loud startup failure instead of silent drop to defaults (v1.6.4 start.go L277-L296), and same failure is in v1.6.4, so it needs a backport to release-1.6 too. Problem is blast radius, one definition takes API down for every tenant.

Definition controller hashes release.cozystack.io/ annotations and rolls cozystack-api when hash changes (applicationdefinition_controller.go L135-L190). Bad definition triggers rollout, and new pod fails at startup.

Second path is openAPISchema. When it is not valid JSON patchSpec returns error (openapi.go L156-L157) and both call sites pass it up instead of skipping the kind (L342-L344, L477-L479). I did not trace whether API server treats that error as fatal at startup.

Same mechanism is described in a comment on #4359, but no issue tracks it.

Impact

Creating ApplicationDefinition used to need admin rights, so at worst it was operator mistake. Since #4308 connecting a tapped repository installs registration chart that creates ApplicationDefinitions from repository content, so publisher can ship definition that stops API for whole cluster. Platform chart with the same typo would do the same.

I read the failing path in code, not reproduced on a cluster. How old pod behaves during rollout depends on Deployment strategy and I did not test it. Any later restart of API (node drain, OOM) takes it down anyway.

Suggested fix

  1. Validate release.cozystack.io/* annotations, and schema fields API reads, at admission. Validating endpoint added in #4424 (internal/lineagecontrollerwebhook/appdef_validator.go) is the place. It has failurePolicy: Ignore (validatingwebhookconfiguration.yaml L26-L30), so it narrows the window but can't be the only guard.
  2. In resourceConfigFrom skip definition that fails buildResourceFromCRD, log it and surface it (event on definition or status condition) instead of returning the error. At least for definitions not from platform, whether broken platform definition should still fail hard is for implementer to decide.

Related

#3448 (dynamic API group registration, open draft) and #1960 (hot reload for ApplicationDefinition changes, open) touch same startup path. Going by their descriptions neither covers single broken definition failing startup.

Acceptance

  • Unit test next to start_owner_test.go: one definition with unparseable timeout annotation, others valid. resourceConfigFrom returns config for valid ones and reports the broken one.
  • Admission endpoint refuses definition with unparseable release.cozystack.io/* annotation.
  • Broken definition no longer prevents cozystack-api restart.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    area/apiIssues or PRs related to the cozystack-api aggregated API server
    on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiIssues or PRs related to the cozystack-api aggregated API serverkind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions