api: an Application event can report another WorkloadMonitor's status from a lagging cache #5325

Where

pkg/registry/apps/application/rest.go: the Application conversion used by REST.Watch (and by GET and LIST), which builds the WorkloadsReady condition from all WorkloadMonitors of the application.

Problem

An application can have several WorkloadMonitors. When the Application watch converts an event, the monitor that triggered it is taken from the event itself, but every other monitor's status comes from the informer cache of the aggregated API server. That cache follows its own watch and can lag behind the monitor watch the event came from.

Sequence:

1. Monitors A and B belong to one application. B's status changes (say, to not operational), and the watch delivers an event that reflects it.
2. A's status changes shortly after. The event for A takes A from the event and B from the cache, which has not yet seen B's change.
3. That event reports B's old status, so the client's WorkloadsReady flips back.
4. The client stays with the regressed status until another event for the application is built from a cache that has caught up.

Impact

Rare, needs two monitors of one application changing within the cache lag. When it happens, WorkloadsReady can show a stale value with no error, until the next event for that application.

Possible directions

• Build the monitor part of the status from a read that is at least as new as the triggering event, for example a live list of the application's monitors when the cache is behind the event's revision.
• Merge the triggering monitor into the cached set only when the cached copies of the other monitors are known to be at least as new, otherwise read them live.