Skip to content

fix: implement model cache refresh to prevent stale disk cache #9478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mrubens merged 3 commits into from
Nov 21, 2025
Merged

fix: implement model cache refresh to prevent stale disk cache #9478

mrubens merged 3 commits into from
Nov 21, 2025

Conversation

@daniel-lxs
Copy link
Member

@daniel-lxs daniel-lxs commented Nov 21, 2025 •
edited
Loading

Problem

The on-disk model cache was never refreshed after initial creation, causing:

  • Stale model info (pricing, context windows, capabilities)
  • New models from providers not appearing
  • Cache persisting indefinitely across extension restarts

Previously, flushModels() only cleared the memory cache (5-minute TTL), but the disk cache remained untouched. After memory cache expiration, getModelsFromCache() would reload from stale disk cache, preventing any fresh API calls.

Solution

1. Add refreshModels() function

  • Forces fresh API fetch, bypassing all cache checks
  • Updates both memory and disk cache atomically
  • Gracefully degrades to existing cache on error

2. Add initializeModelCacheRefresh()

  • Runs 2 seconds after extension activation (non-blocking)
  • Automatically refreshes public providers (OpenRouter, Glama, Vercel AI Gateway)
  • Includes 500ms delays between refreshes to avoid rate limits

3. Update flushModels() signature

  • Add optional refresh parameter
  • When refresh=true, triggers background API fetch after flush

4. Update all manual refresh triggers

  • Ollama, LMStudio, and Roo "Refresh Models" buttons now actually refresh
  • Auth state changes trigger refresh for Roo models

Cache Freshness

With this fix:

  • Memory cache: 5 minutes (unchanged)
  • Disk cache: Refreshed on every extension load (~daily for most users)
  • Manual refresh: Actually fetches fresh data from API
  • Auth changes: Immediate refresh for Roo

Testing

All existing tests pass. The cache refresh is:

  • Non-blocking (does not delay extension activation)
  • Atomic (cache always available during refresh)
  • Error-tolerant (falls back to existing cache on failure)

Changes

  • src/api/providers/fetchers/modelCache.ts: Add refresh functions
  • src/extension.ts: Initialize background refresh on activation
  • src/core/webview/webviewMessageHandler.ts: Update manual refresh handlers
  • src/api/providers/fetchers/lmstudio.ts: Simplify with new refresh API

@daniel-lxs
fix: implement model cache refresh to prevent stale disk cache
effc83e 
- Add refreshModels() function to force fresh API fetch bypassing cache
- Add initializeModelCacheRefresh() for background refresh on extension load
- Update flushModels() with optional refresh parameter
- Refresh public providers (OpenRouter, Glama, Vercel AI Gateway) on startup
- Update manual refresh triggers to actually fetch fresh data
- Use atomic writes to keep cache available during refresh

Previously, the disk cache was written once and never refreshed, causing
stale model info (pricing, context windows, new models) to persist
indefinitely. This fix ensures the cache is refreshed on extension load
and whenever users manually refresh models.
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Nov 21, 2025
@roomote
Copy link
Contributor

roomote bot commented Nov 21, 2025 •
edited
Loading

Rooviewer Clock   See task on Roo Cloud

Review complete. The previously identified issue has been resolved.

  • Race condition in flushModels() when refresh=true - memory cache is cleared before refresh completes, potentially causing unnecessary API calls
Previous reviews

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

roomote[bot]
roomote bot reviewed Nov 21, 2025
View reviewed changes
@hannesrudolph hannesrudolph added the Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. label Nov 21, 2025
@daniel-lxs
fix: prevent race condition in flushModels with refresh
77eb3c2 
Keep old cache in memory when refresh=true to avoid gap in cache
availability. This prevents getModels() from triggering a fresh API
fetch if called immediately after flushModels(router, true).

The refreshModels() function will atomically replace the memory cache
when the refresh completes, maintaining graceful degradation.
@daniel-lxs daniel-lxs moved this from Triage to PR [Needs Review] in Roo Code Roadmap Nov 21, 2025
@hannesrudolph hannesrudolph added PR - Needs Review and removed Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. labels Nov 21, 2025
mrubens
mrubens reviewed Nov 21, 2025
View reviewed changes
src/api/providers/fetchers/modelCache.ts Outdated
try {
// Force fresh API fetch - skip getModelsFromCache() check
switch (provider) {
case "openrouter":
Copy link
Collaborator

@mrubens mrubens Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little nervous about having this list of providers here in a way that we forget to update when we add a new provider. Any ideas on how to prevent that from happening?

Copy link
Member Author

@daniel-lxs daniel-lxs Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, let me fix that

Copy link
Collaborator

@mrubens mrubens Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there at least a way to combine with the getModels above?

@daniel-lxs
refactor: extract fetchModelsFromProvider to avoid duplication
04af1c9 
Extract the provider switch statement into a shared fetchModelsFromProvider()
function to eliminate duplication between getModels() and refreshModels().

This ensures we only maintain the provider list in one place, making it
easier to add new providers without forgetting to update both functions.
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Nov 21, 2025
@mrubens mrubens merged commit 842a4af into main Nov 21, 2025
13 checks passed
@mrubens mrubens deleted the fix/model-cache-refresh branch November 21, 2025 22:32
@github-project-automation github-project-automation bot moved this from PR [Needs Review] to Done in Roo Code Roadmap Nov 21, 2025
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap Nov 21, 2025
mini2s added a commit to zgsm-ai/costrict that referenced this pull request Nov 22, 2025
@mini2s @github-actions
@cte @daniel-lxs @mrubens @roomote @roomote @hannesrudolph @heyseth @sebastiand-cerebras @dleffel @ajjuaire @brunobergher @ellipsis-dev @renovate @jr @AlexKer
Roo to main (#689