Add note on local inference alternative to HfApiModel#327
Conversation
|
Hi @HKanoje, this could be a nice addition if we could scope this a bit more uniform. Could you create a uniform section on this within the onboarding? I think we can just mention local serving and ideally use similar API classed (the HF ones) but explain how to redirect the base URL that relies on OpenAIAPI spec for something like ollama. WDYT? |
|
@davidberenstein1957 Yes I think you're right. It makes more sense to put in as a different section because HfApiModel() is been used everywhere what do you suggest where should I make this section? During the onboarding steps? |
HKanoje
force-pushed
the
add-local-inference-note
branch
from
ae82b7b to
2139435
Compare
|
Hey, @davidberenstein1957 I’ve added the local inference section to the onboarding as you suggested. Let me know if there’s anything you'd like me to adjust or improve! Check here #327 |
burtenshaw
left a comment
burtenshaw
left a comment
There was a problem hiding this comment.
Thanks! This is a really nice addition to the course. Do you have a twitter or linkedin profile? I'd like to communicate about it.
|
Thanks for the approval and kind words, @burtenshaw I’m glad you like the addition. This is my twitter: https://x.com/hrithik_kanoje0 and this is my LinkedIn: https://www.linkedin.com/in/hrithikkanoje/ Feel free to connect, and I’d be happy to discuss further! |
Add note on local inference alternative to HfApiModel
3 participants
This PR adds a note in the docs explaining how users can run models locally using
Ollama + LiteLLMas an alternative toHfApiModel()in case they hit Hugging Face API credit limits.It includes a sample code snippet using:
LiteLLMModelwithmodel_id="ollama_chat/qwen2.5:7b"It closes huggingface/smolagents#967 Issues