Skip to content

Conversation

@pwilkin
Copy link
Contributor

@pwilkin pwilkin commented Jul 17, 2025

Related GitHub Issue

Closes: #5075

Description

This changes the models query to use the LM Studio custom API which adds context-size support and adds a hook upon the first model query that reloads the models cache (due to LM Studio only showing real model size when the model is loaded)

Test Procedure

  1. Add LM Studio provider model
  2. Start a task
  3. Send the first instructions
  4. Max context size should be equal to the one set in LM Studio

Pre-Submission Checklist

  • Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
  • Scope: My changes are focused on the linked issue (one major feature/fix per PR).
  • Self-Review: I have performed a thorough self-review of my code.
  • Testing: New and/or updated tests have been added to cover my changes (if applicable).
  • Documentation Impact: I have considered if my changes require documentation updates (see "Documentation Updates" section below).
  • Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Documentation Updates

Does this PR necessitate updates to user-facing documentation?

  • No documentation updates are required.
  • Yes, documentation updates are required. (Please describe what needs to be updated or link to a PR in the docs repository).

Get in Touch

ilintar on Discord


Important

Enhances LM Studio model handling by adding context-size support and ensuring model cache updates, with UI adjustments for model selection.

  • Behavior:
    • Updates getLmStudioModels in lm-studio.ts to fetch models with context-size support.
    • Adds cache flushing and re-fetching logic in LmStudioHandler in lm-studio.ts to ensure model info is up-to-date.
    • Modifies webviewMessageHandler in webviewMessageHandler.ts to handle requestRouterModels for LM Studio.
  • UI:
    • Updates LMStudio component in LMStudio.tsx to display available models and handle model selection.
    • Adjusts ApiOptions in ApiOptions.tsx to trigger model fetching for LM Studio.
  • Misc:
    • Removes requestLmStudioModels message type from WebviewMessage.ts.

This description was created by Ellipsis for 61d4387. You can customize this summary. It will automatically update as commits are pushed.

@pwilkin pwilkin requested review from cte, jr and mrubens as code owners July 17, 2025 13:35
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Jul 17, 2025
@daniel-lxs daniel-lxs moved this from Triage to PR [Needs Prelim Review] in Roo Code Roadmap Jul 17, 2025
@daniel-lxs
Copy link
Member

Hey @pwilkin Thank you for taking the time to solve this issue.

It seems like some unit tests are failing, can you take a look?

Let me know if you have any questions!

@daniel-lxs daniel-lxs moved this from PR [Needs Prelim Review] to PR [Changes Requested] in Roo Code Roadmap Jul 17, 2025
@pwilkin
Copy link
Contributor Author

pwilkin commented Jul 17, 2025

Yeah, sorry, I did those unit tests on an old version, then saw they were deleted, didn't reallize they just moved to a new place :> will commit.

@pwilkin
Copy link
Contributor Author

pwilkin commented Jul 17, 2025

BTW: any idea why the webView tests are duplicated? The ClineProvider and webViewMessageHandler tests for the requestRouterModels are virtually the same.

@daniel-lxs daniel-lxs moved this from PR [Changes Requested] to PR [Needs Prelim Review] in Roo Code Roadmap Jul 18, 2025
Copy link
Member

@daniel-lxs daniel-lxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @pwilkin Thank you for your contributions, I took a look at your implementation and it looks good, I left a couple of observations, can you take a look?

Let me know what you think!

@daniel-lxs daniel-lxs moved this from PR [Needs Prelim Review] to PR [Changes Requested] in Roo Code Roadmap Jul 19, 2025
@daniel-lxs daniel-lxs moved this from PR [Changes Requested] to PR [Needs Prelim Review] in Roo Code Roadmap Jul 22, 2025
Copy link
Member

@daniel-lxs daniel-lxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @pwilkin Thank you for answering my previous review, I noticed a couple of potential issues with the way the cached models are refreshed.

Let me know what you think!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that the model info fetching happens after the streaming has already started. This means the first message in a conversation will still use the default context window (128,000) from openAiModelInfoSaneDefaults instead of the actual model's context window.

I know that you mentioned that LMStudio uses JIT model loading, the concern is that this initial request might overwhelm the context window if that initial request has a lot of tokens, any idea on how to handle this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change we could request the LMStudio models using the requestLmStudioModels, using requestRouterModels will fetch all the router models and it could be inefficient.

I this change necessary or can we keep the old message to only request LMStudio models?

@daniel-lxs daniel-lxs moved this from PR [Needs Prelim Review] to PR [Changes Requested] in Roo Code Roadmap Jul 22, 2025
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jul 22, 2025
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Jul 22, 2025
@pwilkin
Copy link
Contributor Author

pwilkin commented Jul 22, 2025

@daniel-lxs I think at this point I'm pretty confident this was the wrong route to take :)

Instead, I think the proper approach to solve this is:

  • store info about whether a model's state was updated when loaded somehere (/api/providers/fetchers/lmstudio.ts maybe? basically we need to distinguish if we loaded from LLMInstanceInfo vs LLMInfo)
  • in useSelectedModel.ts, on the getSelectedModel call, force a live reload of the model data if we don't have its live data yet (that means basically doing:
const client = new LMStudioClient({ baseUrl: lmsUrl })
const model = await client.llm.model(apiConfiguration.lmStudioModelId)
const modelInfo = await model.getModelInfo()

and then updating the cache + returning the new modelInfo processed via parseLMStudioModel.

This way, we don't wait until the first request is made to get the data and we don't need to do all that hacking + refreshing of the state.

The only thing I'm afraid about is a case when getSelectedModel is called in a weird UI state, since the operation will be blocking and loading a model in LM Studio can take a looooong time (on slow setups + big models I'd say even 2-3 minutes). The alternative is doing this on upsertProviderProfile in ClineProvider.ts when provider is LM Studio - but that means the user won't have a current view of the model's context size when selecting models in the LM Studio configurator.

Let me know what you think!

@pwilkin
Copy link
Contributor Author

pwilkin commented Jul 24, 2025

@daniel-lxs Added proper fix in #6183, closing this one.

@pwilkin pwilkin closed this Jul 24, 2025
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap Jul 24, 2025
@github-project-automation github-project-automation bot moved this from PR [Changes Requested] to Done in Roo Code Roadmap Jul 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working PR - Changes Requested size:L This PR changes 100-499 lines, ignoring generated files.

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Roo does not correctly detect context length for LM Studio models

3 participants