Steps to reproduce

1. Create many projects with all backends configured (more backends => higher memory usage).
2. Request offers for every project.
3. Observe server memory usage growing at ~100MB per project and never decreasing.

For example, 10 active projects increases server memory usage from ~400MB to 1400MB and stays at that level.

This is very problematic in most production deployments having many projects.

Actual behaviour

This is due to per-project backends cache that keeps backends in cache without ttl. Most backends store large offers cache, which has ttl, but items are evicted only on cache update. Thus, the cache stores offers for all projects that requested offers at least once in the past.

A solution is to set a ttl for backend cache and actively expire it.

Expected behaviour

No response

dstack version

master

Server logs

Additional information

No response