This PR adds support for the GPT OSS bf16 model. Compared to the vLLM, this PR implements Wint8 quantization and achieves a 15% leads in metrics such as QPS, TPS, and TTFT. This PR also introduces several new features to enhance model flexibility and performance, such as sinks in append attention, sliding window attention, bias support for MoE layers, and the swigluoai activation function.

New Features

Feature 1: Support Sinks in Append Attention

This feature introduces sinks in append attention, allowing certain tokens to remain visible across all decoding and enhances the control and stability of attention mechanisms, especially in long-context or multi-turn scenarios.

Feature 2: Support Sliding Window Attention (SWA)

This feature implements Sliding Window Attention, an efficient mechanism for handling long sequences by limiting the attention scope for each token.Sliding window constrains the visible key-value pairs during decoding, improving memory and efficiency in long-sequence inference.

Feature 3: Implement "swigluoai" activation function

This added support for SwigluOAI activation, a variant of SwiGLU with optimized scaling which provides configurable scaling factors (1.702, 7.0) and supports interleaved mode.

Feature 4: Add Bias support for MoE layers

This extended MoE feed-forward to correctly apply expert-specific bias during down projection. And ensures each token routes to the correct expert with its associated bias term.

Usage Example

Start online service

python -m fastdeploy.entrypoints.openai.api_server \
       --model /path/to/gpt-oss-20b-bf16 \
       --port 8188 \
       --engine-worker-queue-port 51001\
       --cache-queue-port 51002 \
       --host 0.0.0.0 \
       --max-model-len 32768 \
       --max-num-seqs 256 \
       --quantization wint8 \

Send a request

# 发送请求
curl -X POST "http://0.0.0.0:8180/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
  "messages": [
    {"role": "user", "content": "你是谁"}
  ]
}'
# 返回结果
"content":"<|channel|>analysis<|message|>The user speaks Chinese: \"你是谁\" (\"Who are you?\"). We should respond in Chinese, presumably. We have instructions: we should act as ChatGPT. We are to answer. The user wants an answer. Could be a brief intro. They might want a description: \"我是 ChatGPT, OpenAI's large language model...\"\n\nGiven the policy, we can do that, we have no disallowed content. The user didn't mention anything requiring policy.\n\nWe need to respond appropriately. There's no sensitive content.\n\nWe just reply with a self-introduction. We should add that \"I am ChatGPT, a language model developed by OpenAI.\" Possibly mention capabilities. That should be enough.<|end|><|start|>assistant<|channel|>final<|message|>我是 ChatGPT，一个由 OpenAI 训练的大型语言模型。我的主要任务是帮助你回答问题、提供信息、协助写作、翻译、进行创作等。你可以把我当作一个随时准备好回答你各种语义和知识问题的小助手。🚀"