fix: prevent plugin declaration cache eviction loop from spinning forever - #784

Description

memCache.set in pkg/utils/cache/helper/combined.go could enter an infinite loop while holding the cache write lock, blocking every reader of the plugin declaration cache until the process was restarted.

Closes #783

Root cause

memCache tracked its size in a hand-maintained itemSize counter kept alongside c.items. Let D = itemSize - len(c.items).

DeletePluginDeclarationCache removed an entry from the map without decrementing itemSize, so every call raised D by one, permanently. Once D >= maxMemCacheSize, the eviction loop in set could no longer terminate — it drained the map, leastKey stayed "", nothing was deleted, and itemSize never fell below the threshold. The loop then spun under c.Lock(), pinning a core while every reader piled up on RLock.

Note on the reported cause

The issue attributes the drift to itemSize++ firing unconditionally on overwrites. That is a real accounting bug, but it cannot by itself reach the terminal condition, so a fix that only makes the increment conditional would not stop the hang.

Tracking D through each operation:

An overwrite requires the key to be present, so len(c.items) >= 1; and eviction runs immediately before the add whenever itemSize >= maxMemCacheSize, exiting at exactly maxMemCacheSize - 1. Inductively that pins D <= maxMemCacheSize - 1 — one short of the terminal condition, indefinitely. DeletePluginDeclarationCache is the only source outside that rebalancing, and is what actually drives D over the line.

This is confirmed by the tests in this PR run against the unfixed code: the overwrite-only test passes, while the test that interleaves set with an external delete hangs.

Changes

• Drive eviction off len(c.items) and drop itemSize entirely. This removes the whole class of drift, including the DeletePluginDeclarationCache source, rather than patching one contributor.
• break out of the eviction loop when there is nothing left to evict, so no future accounting error can reintroduce a non-terminating loop.
• Collapse get to a single write lock. It read item.lastAccess after releasing RLock while other goroutines mutated that field under Lock — a data race the detector flags. The read lock bought no concurrency anyway, since every cache hit already took the write lock to bump accessCount.

Net effect on the implementation file is 19 insertions, 36 deletions.

Type of Change

• Bug fix
• New feature
• Refactor
• Performance improvement
• Other

Essential Checklist

Testing

• I have tested the changes locally and confirmed they work as expected
• I have added unit tests where necessary and they pass successfully

Bug Fix (if applicable)

• I have used GitHub syntax to close the related issue (e.g., Fixes #123 or Closes #123)

Additional Information

Verification

Each failure was reproduced against the unfixed code first, rather than assumed.

The new tests wrap the work in a timeout so a regression fails the suite rather than hanging CI.

go build ./... reports pattern PRIVATE_KEY.pem: no matching files found. That is pre-existing on main, unrelated to this change, and unaffected by it.

Not addressed

Point 3 of the issue — the O(n) scan per evicted item — is left alone. The O(n*k) behaviour it describes was a consequence of the drift: with len(c.items) driving the loop, set evicts exactly one entry per call in steady state, so k = 1. Converting to a heap or LRU list is a worthwhile but separate design change.