-
Notifications
You must be signed in to change notification settings - Fork 2.8k
fix: proper context window loading for LMStudio (fixes #5075) #5814
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Hey @pwilkin Thank you for taking the time to solve this issue. It seems like some unit tests are failing, can you take a look? Let me know if you have any questions! |
|
Yeah, sorry, I did those unit tests on an old version, then saw they were deleted, didn't reallize they just moved to a new place :> will commit. |
|
BTW: any idea why the webView tests are duplicated? The ClineProvider and webViewMessageHandler tests for the requestRouterModels are virtually the same. |
daniel-lxs
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @pwilkin Thank you for your contributions, I took a look at your implementation and it looks good, I left a couple of observations, can you take a look?
Let me know what you think!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @pwilkin Thank you for answering my previous review, I noticed a couple of potential issues with the way the cached models are refreshed.
Let me know what you think!
src/api/providers/lm-studio.ts
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I noticed that the model info fetching happens after the streaming has already started. This means the first message in a conversation will still use the default context window (128,000) from openAiModelInfoSaneDefaults instead of the actual model's context window.
I know that you mentioned that LMStudio uses JIT model loading, the concern is that this initial request might overwhelm the context window if that initial request has a lot of tokens, any idea on how to handle this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before this change we could request the LMStudio models using the requestLmStudioModels, using requestRouterModels will fetch all the router models and it could be inefficient.
I this change necessary or can we keep the old message to only request LMStudio models?
…emove outdated lm-studio fetcher
…nfo instead of description hack
|
@daniel-lxs I think at this point I'm pretty confident this was the wrong route to take :) Instead, I think the proper approach to solve this is:
and then updating the cache + returning the new modelInfo processed via This way, we don't wait until the first request is made to get the data and we don't need to do all that hacking + refreshing of the state. The only thing I'm afraid about is a case when Let me know what you think! |
|
@daniel-lxs Added proper fix in #6183, closing this one. |
Related GitHub Issue
Closes: #5075
Description
This changes the models query to use the LM Studio custom API which adds context-size support and adds a hook upon the first model query that reloads the models cache (due to LM Studio only showing real model size when the model is loaded)
Test Procedure
Pre-Submission Checklist
Documentation Updates
Does this PR necessitate updates to user-facing documentation?
Get in Touch
ilintar on Discord
Important
Enhances LM Studio model handling by adding context-size support and ensuring model cache updates, with UI adjustments for model selection.
getLmStudioModelsinlm-studio.tsto fetch models with context-size support.LmStudioHandlerinlm-studio.tsto ensure model info is up-to-date.webviewMessageHandlerinwebviewMessageHandler.tsto handlerequestRouterModelsfor LM Studio.LMStudiocomponent inLMStudio.tsxto display available models and handle model selection.ApiOptionsinApiOptions.tsxto trigger model fetching for LM Studio.requestLmStudioModelsmessage type fromWebviewMessage.ts.This description was created by
for 61d4387. You can customize this summary. It will automatically update as commits are pushed.