Skip to content

docs: add llama.cpp local model configuration (#2931) - #5688

Merged
WillemJiang merged 1 commit into
bytedance:mainfrom
JY-M666:docs/2931-llamacpp-config
Sep 22, 2026
Merged

WillemJiang merged 1 commit into
bytedance:mainfrom
JY-M666:docs/2931-llamacpp-config

Conversation

@JY-M666

@JY-M666 JY-M666 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • document how to connect DeerFlow to a local llama.cpp llama-server
  • add OpenAI-compatible Qwen examples to config.example.yaml and the English/Chinese configuration pages
  • document Docker networking and tool-calling/chat-template requirements

Why

Issue #2931 asks how to configure a Qwen model served locally by llama.cpp in config.yaml.

DeerFlow already supports OpenAI-compatible endpoints through langchain_openai:ChatOpenAI, but current main does not include a llama.cpp configuration example. A previous attempt in #2960 was closed without being merged, and the documentation gap remains.

What changed

The examples use llama.cpp's OpenAI-compatible /v1/chat/completions endpoint:

models:
  - name: qwen-llamacpp
    use: langchain_openai:ChatOpenAI
    model: qwen-local
    api_key: no-key
    base_url: http://localhost:8080/v1
    request_timeout: 600.0
    max_retries: 2
    supports_vision: false

The docs also explain:

  • using --alias qwen-local for a stable API model name
  • using a container-reachable host instead of localhost when DeerFlow runs in Docker
  • replacing the placeholder API key when llama-server --api-key is enabled
  • ensuring the model chat template supports function/tool calls; current llama.cpp enables its Jinja chat-template engine by default

Validation

  • verified the branch is based on the current upstream main
  • verified the diff is limited to three documentation/configuration files
  • verified current main contains no llama.cpp configuration example
  • checked DeerFlow's current ModelConfig / model factory behavior for ChatOpenAI and base_url
  • checked current llama.cpp server documentation for OpenAI-compatible chat completions, model aliases, Jinja chat templates, and function/tool calling
  • repository CI will provide the authoritative formatting/build/test result for this PR

Closes #2931

@CLAassistant

CLAassistant commented Sep 22, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions Bot added area:docs Documentation and Markdown only area:frontend Next.js frontend under frontend/ risk:medium Medium risk: regular code changes size/S PR changes 20-100 lines labels Sep 22, 2026
@WillemJiang
WillemJiang merged commit 1e6fda4 into bytedance:main Sep 22, 2026
19 checks passed
@WillemJiang WillemJiang added this to the 2.2.0 milestone Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Documentation and Markdown only area:frontend Next.js frontend under frontend/ risk:medium Medium risk: regular code changes size/S PR changes 20-100 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[runtime] 如何在config.yaml中配置LLama.cpp本地部署的qwen3.6-35b-a3b模型

3 participants