Skip to content

Update ghcr.io/k8snetworkplumbingwg/multus-cni Docker tag to v4.1.4 - #729

Merged
sdwilsh merged 1 commit into
mainfrom
renovate/ghcr.io-k8snetworkplumbingwg-multus-cni-4.1.x
Dec 7, 2024
Merged

sdwilsh merged 1 commit into
mainfrom
renovate/ghcr.io-k8snetworkplumbingwg-multus-cni-4.1.x

Conversation

@renovate

@renovate renovate Bot commented Dec 7, 2024

Copy link
Copy Markdown
Contributor

This PR contains the following updates:

Package Type Update Change
ghcr.io/k8snetworkplumbingwg/multus-cni Kustomization patch v4.1.3 -> v4.1.4

Release Notes

k8snetworkplumbingwg/multus-cni (ghcr.io/k8snetworkplumbingwg/multus-cni)

v4.1.4

Compare Source

Updates to use go version 1.22 at a minimum, to address concerns in CVE-2024-24790

See related details in #​1354


Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

@sdwilsh
sdwilsh merged commit 7f4de93 into main Dec 7, 2024
@sdwilsh
sdwilsh deleted the renovate/ghcr.io-k8snetworkplumbingwg-multus-cni-4.1.x branch December 7, 2024 06:50
sdwilsh pushed a commit that referenced this pull request Jun 12, 2026
…v0.13.0

##### [\`v0.13.0\`](https://github.com/cloudnative-pg/plugin-barman-cloud/releases/tag/v0.13.0)

##### Features

- Configure k8s recommended labels on subresources ([#865](cloudnative-pg/plugin-barman-cloud#865)) ([4bbaf18](cloudnative-pg/plugin-barman-cloud@4bbaf18)), closes [#545](cloudnative-pg/plugin-barman-cloud#545)
- Pass additionalCommandArgs to barman-cloud-restore ([#914](cloudnative-pg/plugin-barman-cloud#914)) ([c5f149b](cloudnative-pg/plugin-barman-cloud@c5f149b)), closes [#821](cloudnative-pg/plugin-barman-cloud#821)

##### Bug Fixes

- Add lz4 compression support for base backups ([#868](cloudnative-pg/plugin-barman-cloud#868)) ([e30154b](cloudnative-pg/plugin-barman-cloud@e30154b)), closes [#867](cloudnative-pg/plugin-barman-cloud#867)
- Classify WAL restore errors with precise gRPC statuses ([#927](cloudnative-pg/plugin-barman-cloud#927)) ([aaf6120](cloudnative-pg/plugin-barman-cloud@aaf6120))
- **deps:** Update all non-major go dependencies ([#877](cloudnative-pg/plugin-barman-cloud#877)) ([a81f6e3](cloudnative-pg/plugin-barman-cloud@a81f6e3))
- **deps:** Update all non-major go dependencies ([#898](cloudnative-pg/plugin-barman-cloud#898)) ([acdbfb8](cloudnative-pg/plugin-barman-cloud@acdbfb8))
- **deps:** Update all non-major go dependencies to [`5d46599`](cloudnative-pg/plugin-barman-cloud@5d46599) ([#940](cloudnative-pg/plugin-barman-cloud#940)) ([40aa4b2](cloudnative-pg/plugin-barman-cloud@40aa4b2))
- **deps:** Update all non-major go dependencies to [`8e1642e`](cloudnative-pg/plugin-barman-cloud@8e1642e) ([#939](cloudnative-pg/plugin-barman-cloud#939)) ([f708825](cloudnative-pg/plugin-barman-cloud@f708825))
- **deps:** Update all non-major go dependencies to [`bd68198`](cloudnative-pg/plugin-barman-cloud@bd68198) ([#950](cloudnative-pg/plugin-barman-cloud#950)) ([787d2b3](cloudnative-pg/plugin-barman-cloud@787d2b3))
- **deps:** Update documentation dependencies to v3.10.1 ([#878](cloudnative-pg/plugin-barman-cloud#878)) ([6cf473d](cloudnative-pg/plugin-barman-cloud@6cf473d))
- **deps:** Update k8s.io/utils digest to [`ff6756f`](cloudnative-pg/plugin-barman-cloud@ff6756f) ([#894](cloudnative-pg/plugin-barman-cloud#894)) ([b7389e1](cloudnative-pg/plugin-barman-cloud@b7389e1))
- **deps:** Update kubernetes monorepo to v0.35.4 ([#853](cloudnative-pg/plugin-barman-cloud#853)) ([21f811a](cloudnative-pg/plugin-barman-cloud@21f811a))
- **deps:** Update kubernetes monorepo to v0.36.1 ([#909](cloudnative-pg/plugin-barman-cloud#909)) ([99eb4a3](cloudnative-pg/plugin-barman-cloud@99eb4a3))
- **deps:** Update module github.com/cloudnative-pg/cloudnative-pg to v1.29.1 \[security] ([#902](cloudnative-pg/plugin-barman-cloud#902)) ([ba5063f](cloudnative-pg/plugin-barman-cloud@ba5063f))
- **deps:** Update module github.com/cloudnative-pg/machinery to v0.5.0 ([#925](cloudnative-pg/plugin-barman-cloud#925)) ([ef4b711](cloudnative-pg/plugin-barman-cloud@ef4b711))
- **deps:** Update module github.com/onsi/ginkgo/v2 to v2.28.2 ([#874](cloudnative-pg/plugin-barman-cloud#874)) ([8cb32e5](cloudnative-pg/plugin-barman-cloud@8cb32e5))
- **deps:** Update module google.golang.org/grpc to v1.81.0 ([#884](cloudnative-pg/plugin-barman-cloud#884)) ([6fdecfd](cloudnative-pg/plugin-barman-cloud@6fdecfd))
- **deps:** Update module sigs.k8s.io/controller-runtime to v0.24.0 ([#880](cloudnative-pg/plugin-barman-cloud#880)) ([58f4fed](cloudnative-pg/plugin-barman-cloud@58f4fed))
- **deps:** Update module sigs.k8s.io/controller-runtime to v0.24.1 ([#910](cloudnative-pg/plugin-barman-cloud#910)) ([9be87b0](cloudnative-pg/plugin-barman-cloud@9be87b0))
- **docs:** Update broken link to CloudNativePG documentation ([#904](cloudnative-pg/plugin-barman-cloud#904)) ([43f92fb](cloudnative-pg/plugin-barman-cloud@43f92fb)), closes [#729](cloudnative-pg/plugin-barman-cloud#729)
- **scheme:** Register meta types for CNPG scheme group version ([#943](cloudnative-pg/plugin-barman-cloud#943)) ([bb7845d](cloudnative-pg/plugin-barman-cloud@bb7845d)), closes [#942](cloudnative-pg/plugin-barman-cloud#942)
renovate Bot added a commit that referenced this pull request Oct 7, 2026
##### [\`v1.0.7\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.7)

This release adds **`qwen3.8-mtp:27b`** — the largest dense model FastFlowLM has shipped to date, and the first to run **speculative decoding** on the NPU through its built-in MTP draft head. It also renames `hy-mt2:1.8b` to `hy-mt2-flash:1.8b` to match the flash naming convention, and picks up four community fixes to the OpenAI usage contract, CLI exit codes, `flm validate`, and a stray file in the release packages.

***

#### 📦 New Model Support

##### 🧠 Qwen3.8-27B

FastFlowLM now supports `qwen3.8-mtp:27b`, a 27B reasoning model with tool calling — the largest dense model FastFlowLM has shipped, and the first to run **speculative decoding** on the NPU through its built-in MTP draft head.

- **Tag:** `qwen3.8-mtp:27b`

Run in CLI mode:

```bash
flm run qwen3.8-mtp:27b
```

Run in server mode:

```bash
flm serve qwen3.8-mtp:27b
```

> 💾 This is by far the largest dense model FastFlowLM has shipped. Check that you have the disk space for the pull and enough system memory to hold it resident before running it.

##### ⚡ What MTP Speculative Decoding Means

Qwen3.8 ships with a **multi-token prediction (MTP) head** — a small draft model trained alongside the main network. FastFlowLM now puts it to work:

1. The MTP head **drafts** several candidate tokens in one shot.
2. The full model **verifies** them in a single pass.
3. Accepted drafts are kept; the first rejected one is replaced by the base model's own choice, and drafting restarts from there.

The practical consequences:

- **Faster decoding, same output.** Verification is an exact compare against what the base model would have emitted, so every token you receive is a token the base model would have produced on its own. Speculation changes throughput, not quality.
- **Gains are prompt-dependent.** Predictable, low-entropy stretches — code, structured output, long reasoning chains — see the highest draft acceptance. Highly creative or surprising text accepts fewer drafts and converges toward ordinary decode speed.
- **Nothing to configure.** Speculation is driven by the engine; there is no flag to set and no API change.

Every other model is unaffected — speculation is opt-in at the engine level, so the rest of the lineup decodes exactly as it did in v1.0.6.

For model details, context limits, and measured speedups, see the [model card](https://fastflowlm.com/docs/models/qwen/#-model-card-qwen38-27b) and [benchmark results](https://fastflowlm.com/docs/benchmarks/qwen3.8_results/).

***

#### 🔄 `hy-mt2:1.8b` → `hy-mt2-flash:1.8b`

The multilingual translation model introduced in v1.0.5 became single-turn in v1.0.6. Its name now says so: it is **`hy-mt2-flash:1.8b`**.

This is a **rename only** — same weights, same 1k context, same single-turn behavior, same translation quality. It simply brings the tag in line with `gemma4e-flash` and `qwen3vl-flash`, so that "flash" consistently signals the same contract: single-turn, fixed context, optimized kernels.

**Action required:** update any script, config, or client that pins the old tag.

```diff
- flm run hy-mt2:1.8b
+ flm run hy-mt2-flash:1.8b
```

```diff
- "model": "hy-mt2:1.8b"
+ "model": "hy-mt2-flash:1.8b"
```

The recommended prompt shape from the v1.0.5 guide is unchanged:

```json
{"role": "system", "content": "将以下文本翻译为英语,注意只需要输出翻译后的结果,不要额外解释。输出必须全部使用英语,不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}
```

For more details, see the [model card](https://fastflowlm.com/docs/models/hy-mt2/) and [benchmark results](https://fastflowlm.com/docs/benchmarks/hy-mt2_results/).

***

#### 🐛 Fixes from the Community

Four fixes in this release came from outside contributors. Thank you — these are exactly the kind of sharp, well-scoped reports that make the project better. 🙏

##### 📊 OpenAI `usage` Now Reports the Full Prompt Length

**[PR #729](ROCm/FastFlowLM#729) — thanks to [@Javinator9889](https://github.com/Javinator9889) (Javier Alonso)**

The prompt cache strips the matching prefix before prefill, so `usage.prompt_tokens` was reporting only the **newly evaluated suffix** rather than the whole input. The first turn of a conversation looked correct; every turn after that reused the history as a cached prefix and reported just the new message.

That broke agentic front-ends badly. Tools like OpenCode size their context from `usage.prompt_tokens`, so the context appeared to reset on every request — built-in compaction never triggered, and the client eventually hit a context overflow, re-fed the conversation, and overflowed again in a loop.

What changed:

- **`usage.prompt_tokens`** now counts the entire input, cached prefix included — matching the OpenAI specification.
- **`usage.prompt_tokens_details.cached_tokens`** is now exposed on the OpenAI endpoints, so clients can see how much of that input was served from cache.
- **`prefill_speed_tps`** now divides by the tokens actually evaluated, so a cache hit no longer inflates the reported prefill speed.
- **Ollama-compatible `prompt_eval_count` is unchanged** — it continues to report evaluated tokens only, matching upstream Ollama.

If you drive FLM from an agent framework that manages its own context window, this is the fix to upgrade for.

##### ↩️ `flm help`, `flm version`, and `flm port` Now Exit 0

**[PR #723](ROCm/FastFlowLM#723) — thanks to [@jtuyls](https://github.com/jtuyls) (Jorn Tuyls)**

These three commands succeeded and then exited with status **1**, because they stopped argument parsing the same way a usage error does. Any script, CI job, or shell with `set -e` that ran `flm version` treated a perfectly good invocation as a failure.

They are now handled as successful commands — and handled *before* the model list is loaded, so `flm help` and `flm version` no longer pay that startup cost. Genuine usage errors still exit 1.

##### ✅ `flm validate` No Longer Hard-Checks the Kernel Version

**[PR #738](ROCm/FastFlowLM#738) — thanks to [@superm1](https://github.com/superm1) (Mario Limonciello)**

`flm validate` required kernel 6.17 or newer. Several distributions backport the NPU driver to considerably older kernels, so working setups were being reported as invalid.

The check is now removed. Validation already checks the **firmware version**, which implicitly requires a recent enough driver — making the kernel version test both redundant and wrong for backported kernels.

##### 🧹 Stray Backup File Removed from the Release Packages

**[PR #747](ROCm/FastFlowLM#747) — thanks to [@yorickvP](https://github.com/yorickvP)**

`src/lib/xrt/libq4_npu_eXpress.so.bak-20260826` — a backup copy of a shared library — was committed by accident and had been shipping inside the release packages, despite being used by neither the build nor the executable. It has been deleted, so the Linux artifacts are a little smaller.

***

#### 🙏 Acknowledgements

| Contributor                                        | Contribution                                                                                   |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| [@Javinator9889](https://github.com/Javinator9889) | [#729](ROCm/FastFlowLM#729) — full prompt length in OpenAI `usage`     |
| [@jtuyls](https://github.com/jtuyls)               | [#723](ROCm/FastFlowLM#723) — exit 0 for `help`, `version`, and `port` |
| [@superm1](https://github.com/superm1)             | [#738](ROCm/FastFlowLM#738) — drop the kernel check in `flm validate`  |
| [@yorickvP](https://github.com/yorickvP)           | [#747](ROCm/FastFlowLM#747) — remove the stray `.so.bak` file          |

***

#### 🌟 Summary

|    | Highlight                                                                                                                                 |
| -- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| 📦 | New model: **Qwen3.8-27B** (`qwen3.8-mtp:27b`) — reasoning and tool calling, the largest dense model FastFlowLM has shipped               |
| ⚡  | First model with **MTP speculative decoding**: a draft head proposes, the full model verifies — faster decode, identical output           |
| 🔄 | `hy-mt2:1.8b` renamed to `hy-mt2-flash:1.8b` — rename only; update pinned tags                                                            |
| 📊 | `usage.prompt_tokens` reports the full input, `cached_tokens` is now exposed, and `prefill_speed_tps` is no longer inflated by cache hits |
| ↩️ | `flm help`, `flm version`, and `flm port` exit 0 instead of 1                                                                             |
| ✅  | `flm validate` no longer rejects backported kernels older than 6.17                                                                       |
| 🧹 | Stray `.so.bak` backup file removed from the release packages                                                                             |

Thanks for your support — see you in the next one! 🚀

---
##### [\`v1.0.6\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.6)

This release introduces three **flash** models — kernel-optimized runtimes over the existing weights, so no re-download is needed — makes `hy-mt2:1.8b` single-turn, adds `tool_choice` support in server mode, fixes a tool-call `finish_reason` bug, and changes where the MSI installer writes `FLM_MODEL_PATH`.

***

#### ⚡ New Flash Models

FastFlowLM now supports three flash models. These are **not new checkpoints** — they run the *same weights* as their standard counterparts through optimized kernels, so there is **no weights update and nothing new to download**:

| Tag                 | Prefill  @ 128 ctx | Model card                                                                                                       |
| ------------------- | -----------------: | ---------------------------------------------------------------------------------------------------------------- |
| `gemma4e-flash:e2b` | **\~390 tokens/s** | [Gemma 4 E2B-IT · Flash](https://fastflowlm.com/docs/models/gemma/#-model-card-gemma-4-e2b-it--flash)            |
| `gemma4e-flash:e4b` | **\~256 tokens/s** | [Gemma 4 E4B-IT · Flash](https://fastflowlm.com/docs/models/gemma/#-model-card-gemma-4-e4b-it--flash)            |
| `qwen3vl-flash:4b`  | **\~350 tokens/s** | [Qwen3-VL 4B-Instruct · Flash](https://fastflowlm.com/docs/models/qwen/#-model-card-qwen3-vl-4b-instruct--flash) |

Run in CLI mode:

```bash
flm run gemma4e-flash:e2b
```

Run in server mode:

```bash
flm serve qwen3vl-flash:4b
```

##### 📖 What "Flash" Means

Flash models trade multi-turn flexibility for speed: the same weights run through optimized kernels, under a set of constraints that make those kernels possible. Please note the following behavior before deploying them:

- **Same weights, no re-download.** Flash models reuse the weights you already have — no `flm pull` required.
- **Single-turn only.** Any request containing an assistant message is rejected. Send a system prompt (optional) and a single user message.
- **System KV cache supported.** The system prompt is prefilled once and reused across requests, so repeated calls sharing a system prompt skip that prefill cost.
- **1k maximum context length.** Requests over 1k tokens are **rejected with a warning** — they are not silently truncated. Size your prompts accordingly.

##### 🖼️ Image & Audio Handling

Flash models apply a **fixed** media budget. No per-request tuning is required — oversized input is reduced automatically rather than rejected:

| Model                        | Images                                       | Audio                                 |
| ---------------------------- | -------------------------------------------- | ------------------------------------- |
| `qwen3vl-flash:4b`           | Resized so the longer side is **256 pixels** | Not supported                         |
| `gemma4e-flash:e2b` / `:e4b` | Resized to **70 tokens**                     | **Truncated to the first 30 seconds** |

> ⚠️ Note the difference from the context limit: oversized **media is silently reduced** (images downscaled, audio cut at 30 s), while an over-1k **prompt is rejected outright**. Audio longer than 30 seconds will not raise an error — the model simply never sees the remainder, so split long clips yourself if you need full coverage.

##### 🐍 Example: Python + OpenAI SDK (Streaming)

Flash models speak the standard OpenAI chat-completions API, so the official `openai` Python client works as-is — just point it at your local FLM server.

```bash
pip install openai
flm serve gemma4e-flash:e2b
```

**Vision, streaming, with a pinned system prompt:**

Below, the same system prompt is pinned across requests that each caption a **different** image. The system-prompt KV is prefilled once on the first request; every request after that reuses it, so only the new image and text need to be prefilled — expect noticeably lower latency from the second request onward.

```python
import time
import base64
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:52625/v1",
    api_key="dummykey",  # FLM runs locally; the key is not checked
)

# Keep everything fixed in the system prompt — it is prefilled once and
# reused across requests. Vary only the user message and image.
SYSTEM_PROMPT = "You are a concise assistant. Describe the image in one short sentence."

image_paths = [
    r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image0.jpg",
    r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image1.png",
]

for path in image_paths:
    with open(path, "rb") as image_file:
        image = base64.b64encode(image_file.read()).decode("utf-8")

    print(f"\n> {path}")
    start = time.time()
    stream = client.chat.completions.create(
        model="qwen3vl-flash:4b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},  # same every time — its KV is cached after the first call
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpg;base64,{image}"}},
                ],
            },
        ],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
    print(f"\n[time to first response: {time.time() - start:.2f}s]")
    # 1st image: pays the full system-prompt prefill cost.
    # 2nd image onward: system-prompt KV is reused from cache — expect lower latency.

# cleanup
del stream, client
import gc
gc.collect()
```

**Vision, streaming:**

Serve the vision flash model with `flm serve qwen3vl-flash:4b`, then pass an image as a base64 data URL. Resizing is automatic, so there is no need to downscale beforehand:

```python
import base64
from openai import OpenAI

image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="qwen3vl-flash:4b",
    messages=[
        {"role": "system", "content": "Describe images in one sentence."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

> **Note:** the image counts against the same 1k context budget as your text. Keep prompts short when sending an image.

**Audio, streaming:**

Audio goes to the same endpoint with the same message structure — only the content part changes. Use a `gemma4e-flash` model, since `qwen3vl-flash:4b` does not accept audio:

```python
import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {"role": "system", "content": "Transcribe and summarize audio briefly."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is said in this clip?"},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

> ⚠️ **Only the first 30 seconds are processed.** Longer clips are truncated without an error, so split them client-side if you need full coverage.

**Audio + image in one request:**

`gemma4e-flash` accepts both in the same `content` array, so a single call can reason over audio and an image together:

```python
import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this
image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")
with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Finish two tasks: 1. Summarize the audio. 2. Describe the image."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        }
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

***

#### 🔄 `hy-mt2:1.8b` Is Now Single-Turn

`hy-mt2:1.8b`, the multilingual translation model added in v1.0.5, is now a **single-turn** model as well. Like the flash models, it rejects any request containing an assistant message.

This matches how a dedicated translation model is actually used: every translation is independent, and prior turns add prefill cost without improving the result.

**What this means for you:** if you were replaying conversation history back to `hy-mt2:1.8b`, remove it — send only the instruction and the source text. The recommended shape from the v1.0.5 prompt guide is unchanged and remains the best option:

```json
{"role": "system", "content": "将以下文本翻译为英语,注意只需要输出翻译后的结果,不要额外解释。输出必须全部使用英语,不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}
```

Pinning the instruction as the system prompt and sending only the source text as the user message keeps the instruction out of every request's prefill — especially worthwhile for batch workloads such as translating subtitle lines.

***

#### 🛠️ Tool Calling

##### `tool_choice` Support (Server Mode)

Server mode now honors the `tool_choice` field. Two modes are supported:

| Value              | Behavior                                                                                                            |
| ------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `auto` *(default)* | The model decides whether to call a tool                                                                            |
| `none`             | Tools are still sent to the model, but tool-token logits are masked during decoding, so no tool call can be emitted |

Any other value falls back to `auto`.

##### 🐛 Bug Fix: Truncated Tool Calls Reported as `tool_calls`

Fixed a bug where a tool call truncated by `max_tokens` was reported with `finish_reason: "tool_calls"` alongside a tool call whose `name` and `arguments` were empty strings. Truncated generations now correctly report `finish_reason: "length"`.

***

#### 📦 MSI Installer

## The installer now sets `FLM_MODEL_PATH` as a **user** environment variable instead of a system one.

#### 🌟 Summary

|     | Highlight                                                                                                                                                                                     |
| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ⚡   | Three new flash models: `gemma4e-flash:e2b`, `gemma4e-flash:e4b`, `qwen3vl-flash:4b` — kernel-optimized over the same weights (no re-download), single-turn only, 1k context, system KV cache |
| 🔄  | `hy-mt2:1.8b` is now single-turn — assistant messages are rejected; send instruction + source text only                                                                                       |
| 🛠️ | `tool_choice` support in server mode: `auto` (default) and `none`                                                                                                                             |
| 🐛  | Truncated tool calls now report `finish_reason: "length"` instead of an empty `tool_calls`                                                                                                    |
| 📦  | MSI installer sets `FLM_MODEL_PATH` in the user environment instead of the system environment                                                                                                 |

Thanks for your support — see you in the next one! 🚀
renovate Bot added a commit that referenced this pull request Oct 10, 2026
##### [\`v1.0.7\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.7)

This release adds **`qwen3.8-mtp:27b`** — the largest dense model FastFlowLM has shipped to date, and the first to run **speculative decoding** on the NPU through its built-in MTP draft head. It also renames `hy-mt2:1.8b` to `hy-mt2-flash:1.8b` to match the flash naming convention, and picks up four community fixes to the OpenAI usage contract, CLI exit codes, `flm validate`, and a stray file in the release packages.

***

#### 📦 New Model Support

##### 🧠 Qwen3.8-27B

FastFlowLM now supports `qwen3.8-mtp:27b`, a 27B reasoning model with tool calling — the largest dense model FastFlowLM has shipped, and the first to run **speculative decoding** on the NPU through its built-in MTP draft head.

- **Tag:** `qwen3.8-mtp:27b`

Run in CLI mode:

```bash
flm run qwen3.8-mtp:27b
```

Run in server mode:

```bash
flm serve qwen3.8-mtp:27b
```

> 💾 This is by far the largest dense model FastFlowLM has shipped. Check that you have the disk space for the pull and enough system memory to hold it resident before running it.

##### ⚡ What MTP Speculative Decoding Means

Qwen3.8 ships with a **multi-token prediction (MTP) head** — a small draft model trained alongside the main network. FastFlowLM now puts it to work:

1. The MTP head **drafts** several candidate tokens in one shot.
2. The full model **verifies** them in a single pass.
3. Accepted drafts are kept; the first rejected one is replaced by the base model's own choice, and drafting restarts from there.

The practical consequences:

- **Faster decoding, same output.** Verification is an exact compare against what the base model would have emitted, so every token you receive is a token the base model would have produced on its own. Speculation changes throughput, not quality.
- **Gains are prompt-dependent.** Predictable, low-entropy stretches — code, structured output, long reasoning chains — see the highest draft acceptance. Highly creative or surprising text accepts fewer drafts and converges toward ordinary decode speed.
- **Nothing to configure.** Speculation is driven by the engine; there is no flag to set and no API change.

Every other model is unaffected — speculation is opt-in at the engine level, so the rest of the lineup decodes exactly as it did in v1.0.6.

For model details, context limits, and measured speedups, see the [model card](https://fastflowlm.com/docs/models/qwen/#-model-card-qwen38-27b) and [benchmark results](https://fastflowlm.com/docs/benchmarks/qwen3.8_results/).

***

#### 🔄 `hy-mt2:1.8b` → `hy-mt2-flash:1.8b`

The multilingual translation model introduced in v1.0.5 became single-turn in v1.0.6. Its name now says so: it is **`hy-mt2-flash:1.8b`**.

This is a **rename only** — same weights, same 1k context, same single-turn behavior, same translation quality. It simply brings the tag in line with `gemma4e-flash` and `qwen3vl-flash`, so that "flash" consistently signals the same contract: single-turn, fixed context, optimized kernels.

**Action required:** update any script, config, or client that pins the old tag.

```diff
- flm run hy-mt2:1.8b
+ flm run hy-mt2-flash:1.8b
```

```diff
- "model": "hy-mt2:1.8b"
+ "model": "hy-mt2-flash:1.8b"
```

The recommended prompt shape from the v1.0.5 guide is unchanged:

```json
{"role": "system", "content": "将以下文本翻译为英语,注意只需要输出翻译后的结果,不要额外解释。输出必须全部使用英语,不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}
```

For more details, see the [model card](https://fastflowlm.com/docs/models/hy-mt2/) and [benchmark results](https://fastflowlm.com/docs/benchmarks/hy-mt2_results/).

***

#### 🐛 Fixes from the Community

Four fixes in this release came from outside contributors. Thank you — these are exactly the kind of sharp, well-scoped reports that make the project better. 🙏

##### 📊 OpenAI `usage` Now Reports the Full Prompt Length

**[PR #729](ROCm/FastFlowLM#729) — thanks to [@Javinator9889](https://github.com/Javinator9889) (Javier Alonso)**

The prompt cache strips the matching prefix before prefill, so `usage.prompt_tokens` was reporting only the **newly evaluated suffix** rather than the whole input. The first turn of a conversation looked correct; every turn after that reused the history as a cached prefix and reported just the new message.

That broke agentic front-ends badly. Tools like OpenCode size their context from `usage.prompt_tokens`, so the context appeared to reset on every request — built-in compaction never triggered, and the client eventually hit a context overflow, re-fed the conversation, and overflowed again in a loop.

What changed:

- **`usage.prompt_tokens`** now counts the entire input, cached prefix included — matching the OpenAI specification.
- **`usage.prompt_tokens_details.cached_tokens`** is now exposed on the OpenAI endpoints, so clients can see how much of that input was served from cache.
- **`prefill_speed_tps`** now divides by the tokens actually evaluated, so a cache hit no longer inflates the reported prefill speed.
- **Ollama-compatible `prompt_eval_count` is unchanged** — it continues to report evaluated tokens only, matching upstream Ollama.

If you drive FLM from an agent framework that manages its own context window, this is the fix to upgrade for.

##### ↩️ `flm help`, `flm version`, and `flm port` Now Exit 0

**[PR #723](ROCm/FastFlowLM#723) — thanks to [@jtuyls](https://github.com/jtuyls) (Jorn Tuyls)**

These three commands succeeded and then exited with status **1**, because they stopped argument parsing the same way a usage error does. Any script, CI job, or shell with `set -e` that ran `flm version` treated a perfectly good invocation as a failure.

They are now handled as successful commands — and handled *before* the model list is loaded, so `flm help` and `flm version` no longer pay that startup cost. Genuine usage errors still exit 1.

##### ✅ `flm validate` No Longer Hard-Checks the Kernel Version

**[PR #738](ROCm/FastFlowLM#738) — thanks to [@superm1](https://github.com/superm1) (Mario Limonciello)**

`flm validate` required kernel 6.17 or newer. Several distributions backport the NPU driver to considerably older kernels, so working setups were being reported as invalid.

The check is now removed. Validation already checks the **firmware version**, which implicitly requires a recent enough driver — making the kernel version test both redundant and wrong for backported kernels.

##### 🧹 Stray Backup File Removed from the Release Packages

**[PR #747](ROCm/FastFlowLM#747) — thanks to [@yorickvP](https://github.com/yorickvP)**

`src/lib/xrt/libq4_npu_eXpress.so.bak-20260826` — a backup copy of a shared library — was committed by accident and had been shipping inside the release packages, despite being used by neither the build nor the executable. It has been deleted, so the Linux artifacts are a little smaller.

***

#### 🙏 Acknowledgements

| Contributor                                        | Contribution                                                                                   |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| [@Javinator9889](https://github.com/Javinator9889) | [#729](ROCm/FastFlowLM#729) — full prompt length in OpenAI `usage`     |
| [@jtuyls](https://github.com/jtuyls)               | [#723](ROCm/FastFlowLM#723) — exit 0 for `help`, `version`, and `port` |
| [@superm1](https://github.com/superm1)             | [#738](ROCm/FastFlowLM#738) — drop the kernel check in `flm validate`  |
| [@yorickvP](https://github.com/yorickvP)           | [#747](ROCm/FastFlowLM#747) — remove the stray `.so.bak` file          |

***

#### 🌟 Summary

|    | Highlight                                                                                                                                 |
| -- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| 📦 | New model: **Qwen3.8-27B** (`qwen3.8-mtp:27b`) — reasoning and tool calling, the largest dense model FastFlowLM has shipped               |
| ⚡  | First model with **MTP speculative decoding**: a draft head proposes, the full model verifies — faster decode, identical output           |
| 🔄 | `hy-mt2:1.8b` renamed to `hy-mt2-flash:1.8b` — rename only; update pinned tags                                                            |
| 📊 | `usage.prompt_tokens` reports the full input, `cached_tokens` is now exposed, and `prefill_speed_tps` is no longer inflated by cache hits |
| ↩️ | `flm help`, `flm version`, and `flm port` exit 0 instead of 1                                                                             |
| ✅  | `flm validate` no longer rejects backported kernels older than 6.17                                                                       |
| 🧹 | Stray `.so.bak` backup file removed from the release packages                                                                             |

Thanks for your support — see you in the next one! 🚀

---
##### [\`v1.0.6\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.6)

This release introduces three **flash** models — kernel-optimized runtimes over the existing weights, so no re-download is needed — makes `hy-mt2:1.8b` single-turn, adds `tool_choice` support in server mode, fixes a tool-call `finish_reason` bug, and changes where the MSI installer writes `FLM_MODEL_PATH`.

***

#### ⚡ New Flash Models

FastFlowLM now supports three flash models. These are **not new checkpoints** — they run the *same weights* as their standard counterparts through optimized kernels, so there is **no weights update and nothing new to download**:

| Tag                 | Prefill  @ 128 ctx | Model card                                                                                                       |
| ------------------- | -----------------: | ---------------------------------------------------------------------------------------------------------------- |
| `gemma4e-flash:e2b` | **\~390 tokens/s** | [Gemma 4 E2B-IT · Flash](https://fastflowlm.com/docs/models/gemma/#-model-card-gemma-4-e2b-it--flash)            |
| `gemma4e-flash:e4b` | **\~256 tokens/s** | [Gemma 4 E4B-IT · Flash](https://fastflowlm.com/docs/models/gemma/#-model-card-gemma-4-e4b-it--flash)            |
| `qwen3vl-flash:4b`  | **\~350 tokens/s** | [Qwen3-VL 4B-Instruct · Flash](https://fastflowlm.com/docs/models/qwen/#-model-card-qwen3-vl-4b-instruct--flash) |

Run in CLI mode:

```bash
flm run gemma4e-flash:e2b
```

Run in server mode:

```bash
flm serve qwen3vl-flash:4b
```

##### 📖 What "Flash" Means

Flash models trade multi-turn flexibility for speed: the same weights run through optimized kernels, under a set of constraints that make those kernels possible. Please note the following behavior before deploying them:

- **Same weights, no re-download.** Flash models reuse the weights you already have — no `flm pull` required.
- **Single-turn only.** Any request containing an assistant message is rejected. Send a system prompt (optional) and a single user message.
- **System KV cache supported.** The system prompt is prefilled once and reused across requests, so repeated calls sharing a system prompt skip that prefill cost.
- **1k maximum context length.** Requests over 1k tokens are **rejected with a warning** — they are not silently truncated. Size your prompts accordingly.

##### 🖼️ Image & Audio Handling

Flash models apply a **fixed** media budget. No per-request tuning is required — oversized input is reduced automatically rather than rejected:

| Model                        | Images                                       | Audio                                 |
| ---------------------------- | -------------------------------------------- | ------------------------------------- |
| `qwen3vl-flash:4b`           | Resized so the longer side is **256 pixels** | Not supported                         |
| `gemma4e-flash:e2b` / `:e4b` | Resized to **70 tokens**                     | **Truncated to the first 30 seconds** |

> ⚠️ Note the difference from the context limit: oversized **media is silently reduced** (images downscaled, audio cut at 30 s), while an over-1k **prompt is rejected outright**. Audio longer than 30 seconds will not raise an error — the model simply never sees the remainder, so split long clips yourself if you need full coverage.

##### 🐍 Example: Python + OpenAI SDK (Streaming)

Flash models speak the standard OpenAI chat-completions API, so the official `openai` Python client works as-is — just point it at your local FLM server.

```bash
pip install openai
flm serve gemma4e-flash:e2b
```

**Vision, streaming, with a pinned system prompt:**

Below, the same system prompt is pinned across requests that each caption a **different** image. The system-prompt KV is prefilled once on the first request; every request after that reuses it, so only the new image and text need to be prefilled — expect noticeably lower latency from the second request onward.

```python
import time
import base64
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:52625/v1",
    api_key="dummykey",  # FLM runs locally; the key is not checked
)

# Keep everything fixed in the system prompt — it is prefilled once and
# reused across requests. Vary only the user message and image.
SYSTEM_PROMPT = "You are a concise assistant. Describe the image in one short sentence."

image_paths = [
    r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image0.jpg",
    r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image1.png",
]

for path in image_paths:
    with open(path, "rb") as image_file:
        image = base64.b64encode(image_file.read()).decode("utf-8")

    print(f"\n> {path}")
    start = time.time()
    stream = client.chat.completions.create(
        model="qwen3vl-flash:4b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},  # same every time — its KV is cached after the first call
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpg;base64,{image}"}},
                ],
            },
        ],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
    print(f"\n[time to first response: {time.time() - start:.2f}s]")
    # 1st image: pays the full system-prompt prefill cost.
    # 2nd image onward: system-prompt KV is reused from cache — expect lower latency.

# cleanup
del stream, client
import gc
gc.collect()
```

**Vision, streaming:**

Serve the vision flash model with `flm serve qwen3vl-flash:4b`, then pass an image as a base64 data URL. Resizing is automatic, so there is no need to downscale beforehand:

```python
import base64
from openai import OpenAI

image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="qwen3vl-flash:4b",
    messages=[
        {"role": "system", "content": "Describe images in one sentence."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

> **Note:** the image counts against the same 1k context budget as your text. Keep prompts short when sending an image.

**Audio, streaming:**

Audio goes to the same endpoint with the same message structure — only the content part changes. Use a `gemma4e-flash` model, since `qwen3vl-flash:4b` does not accept audio:

```python
import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {"role": "system", "content": "Transcribe and summarize audio briefly."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is said in this clip?"},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

> ⚠️ **Only the first 30 seconds are processed.** Longer clips are truncated without an error, so split them client-side if you need full coverage.

**Audio + image in one request:**

`gemma4e-flash` accepts both in the same `content` array, so a single call can reason over audio and an image together:

```python
import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this
image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")
with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Finish two tasks: 1. Summarize the audio. 2. Describe the image."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        }
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

***

#### 🔄 `hy-mt2:1.8b` Is Now Single-Turn

`hy-mt2:1.8b`, the multilingual translation model added in v1.0.5, is now a **single-turn** model as well. Like the flash models, it rejects any request containing an assistant message.

This matches how a dedicated translation model is actually used: every translation is independent, and prior turns add prefill cost without improving the result.

**What this means for you:** if you were replaying conversation history back to `hy-mt2:1.8b`, remove it — send only the instruction and the source text. The recommended shape from the v1.0.5 prompt guide is unchanged and remains the best option:

```json
{"role": "system", "content": "将以下文本翻译为英语,注意只需要输出翻译后的结果,不要额外解释。输出必须全部使用英语,不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}
```

Pinning the instruction as the system prompt and sending only the source text as the user message keeps the instruction out of every request's prefill — especially worthwhile for batch workloads such as translating subtitle lines.

***

#### 🛠️ Tool Calling

##### `tool_choice` Support (Server Mode)

Server mode now honors the `tool_choice` field. Two modes are supported:

| Value              | Behavior                                                                                                            |
| ------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `auto` *(default)* | The model decides whether to call a tool                                                                            |
| `none`             | Tools are still sent to the model, but tool-token logits are masked during decoding, so no tool call can be emitted |

Any other value falls back to `auto`.

##### 🐛 Bug Fix: Truncated Tool Calls Reported as `tool_calls`

Fixed a bug where a tool call truncated by `max_tokens` was reported with `finish_reason: "tool_calls"` alongside a tool call whose `name` and `arguments` were empty strings. Truncated generations now correctly report `finish_reason: "length"`.

***

#### 📦 MSI Installer

## The installer now sets `FLM_MODEL_PATH` as a **user** environment variable instead of a system one.

#### 🌟 Summary

|     | Highlight                                                                                                                                                                                     |
| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ⚡   | Three new flash models: `gemma4e-flash:e2b`, `gemma4e-flash:e4b`, `qwen3vl-flash:4b` — kernel-optimized over the same weights (no re-download), single-turn only, 1k context, system KV cache |
| 🔄  | `hy-mt2:1.8b` is now single-turn — assistant messages are rejected; send instruction + source text only                                                                                       |
| 🛠️ | `tool_choice` support in server mode: `auto` (default) and `none`                                                                                                                             |
| 🐛  | Truncated tool calls now report `finish_reason: "length"` instead of an empty `tool_calls`                                                                                                    |
| 📦  | MSI installer sets `FLM_MODEL_PATH` in the user environment instead of the system environment                                                                                                 |

Thanks for your support — see you in the next one! 🚀
renovate Bot added a commit that referenced this pull request Oct 17, 2026
##### [\`v1.0.7\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.7)

This release adds **`qwen3.8-mtp:27b`** — the largest dense model FastFlowLM has shipped to date, and the first to run **speculative decoding** on the NPU through its built-in MTP draft head. It also renames `hy-mt2:1.8b` to `hy-mt2-flash:1.8b` to match the flash naming convention, and picks up four community fixes to the OpenAI usage contract, CLI exit codes, `flm validate`, and a stray file in the release packages.

***

#### 📦 New Model Support

##### 🧠 Qwen3.8-27B

FastFlowLM now supports `qwen3.8-mtp:27b`, a 27B reasoning model with tool calling — the largest dense model FastFlowLM has shipped, and the first to run **speculative decoding** on the NPU through its built-in MTP draft head.

- **Tag:** `qwen3.8-mtp:27b`

Run in CLI mode:

```bash
flm run qwen3.8-mtp:27b
```

Run in server mode:

```bash
flm serve qwen3.8-mtp:27b
```

> 💾 This is by far the largest dense model FastFlowLM has shipped. Check that you have the disk space for the pull and enough system memory to hold it resident before running it.

##### ⚡ What MTP Speculative Decoding Means

Qwen3.8 ships with a **multi-token prediction (MTP) head** — a small draft model trained alongside the main network. FastFlowLM now puts it to work:

1. The MTP head **drafts** several candidate tokens in one shot.
2. The full model **verifies** them in a single pass.
3. Accepted drafts are kept; the first rejected one is replaced by the base model's own choice, and drafting restarts from there.

The practical consequences:

- **Faster decoding, same output.** Verification is an exact compare against what the base model would have emitted, so every token you receive is a token the base model would have produced on its own. Speculation changes throughput, not quality.
- **Gains are prompt-dependent.** Predictable, low-entropy stretches — code, structured output, long reasoning chains — see the highest draft acceptance. Highly creative or surprising text accepts fewer drafts and converges toward ordinary decode speed.
- **Nothing to configure.** Speculation is driven by the engine; there is no flag to set and no API change.

Every other model is unaffected — speculation is opt-in at the engine level, so the rest of the lineup decodes exactly as it did in v1.0.6.

For model details, context limits, and measured speedups, see the [model card](https://fastflowlm.com/docs/models/qwen/#-model-card-qwen38-27b) and [benchmark results](https://fastflowlm.com/docs/benchmarks/qwen3.8_results/).

***

#### 🔄 `hy-mt2:1.8b` → `hy-mt2-flash:1.8b`

The multilingual translation model introduced in v1.0.5 became single-turn in v1.0.6. Its name now says so: it is **`hy-mt2-flash:1.8b`**.

This is a **rename only** — same weights, same 1k context, same single-turn behavior, same translation quality. It simply brings the tag in line with `gemma4e-flash` and `qwen3vl-flash`, so that "flash" consistently signals the same contract: single-turn, fixed context, optimized kernels.

**Action required:** update any script, config, or client that pins the old tag.

```diff
- flm run hy-mt2:1.8b
+ flm run hy-mt2-flash:1.8b
```

```diff
- "model": "hy-mt2:1.8b"
+ "model": "hy-mt2-flash:1.8b"
```

The recommended prompt shape from the v1.0.5 guide is unchanged:

```json
{"role": "system", "content": "将以下文本翻译为英语,注意只需要输出翻译后的结果,不要额外解释。输出必须全部使用英语,不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}
```

For more details, see the [model card](https://fastflowlm.com/docs/models/hy-mt2/) and [benchmark results](https://fastflowlm.com/docs/benchmarks/hy-mt2_results/).

***

#### 🐛 Fixes from the Community

Four fixes in this release came from outside contributors. Thank you — these are exactly the kind of sharp, well-scoped reports that make the project better. 🙏

##### 📊 OpenAI `usage` Now Reports the Full Prompt Length

**[PR #729](ROCm/FastFlowLM#729) — thanks to [@Javinator9889](https://github.com/Javinator9889) (Javier Alonso)**

The prompt cache strips the matching prefix before prefill, so `usage.prompt_tokens` was reporting only the **newly evaluated suffix** rather than the whole input. The first turn of a conversation looked correct; every turn after that reused the history as a cached prefix and reported just the new message.

That broke agentic front-ends badly. Tools like OpenCode size their context from `usage.prompt_tokens`, so the context appeared to reset on every request — built-in compaction never triggered, and the client eventually hit a context overflow, re-fed the conversation, and overflowed again in a loop.

What changed:

- **`usage.prompt_tokens`** now counts the entire input, cached prefix included — matching the OpenAI specification.
- **`usage.prompt_tokens_details.cached_tokens`** is now exposed on the OpenAI endpoints, so clients can see how much of that input was served from cache.
- **`prefill_speed_tps`** now divides by the tokens actually evaluated, so a cache hit no longer inflates the reported prefill speed.
- **Ollama-compatible `prompt_eval_count` is unchanged** — it continues to report evaluated tokens only, matching upstream Ollama.

If you drive FLM from an agent framework that manages its own context window, this is the fix to upgrade for.

##### ↩️ `flm help`, `flm version`, and `flm port` Now Exit 0

**[PR #723](ROCm/FastFlowLM#723) — thanks to [@jtuyls](https://github.com/jtuyls) (Jorn Tuyls)**

These three commands succeeded and then exited with status **1**, because they stopped argument parsing the same way a usage error does. Any script, CI job, or shell with `set -e` that ran `flm version` treated a perfectly good invocation as a failure.

They are now handled as successful commands — and handled *before* the model list is loaded, so `flm help` and `flm version` no longer pay that startup cost. Genuine usage errors still exit 1.

##### ✅ `flm validate` No Longer Hard-Checks the Kernel Version

**[PR #738](ROCm/FastFlowLM#738) — thanks to [@superm1](https://github.com/superm1) (Mario Limonciello)**

`flm validate` required kernel 6.17 or newer. Several distributions backport the NPU driver to considerably older kernels, so working setups were being reported as invalid.

The check is now removed. Validation already checks the **firmware version**, which implicitly requires a recent enough driver — making the kernel version test both redundant and wrong for backported kernels.

##### 🧹 Stray Backup File Removed from the Release Packages

**[PR #747](ROCm/FastFlowLM#747) — thanks to [@yorickvP](https://github.com/yorickvP)**

`src/lib/xrt/libq4_npu_eXpress.so.bak-20260826` — a backup copy of a shared library — was committed by accident and had been shipping inside the release packages, despite being used by neither the build nor the executable. It has been deleted, so the Linux artifacts are a little smaller.

***

#### 🙏 Acknowledgements

| Contributor                                        | Contribution                                                                                   |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| [@Javinator9889](https://github.com/Javinator9889) | [#729](ROCm/FastFlowLM#729) — full prompt length in OpenAI `usage`     |
| [@jtuyls](https://github.com/jtuyls)               | [#723](ROCm/FastFlowLM#723) — exit 0 for `help`, `version`, and `port` |
| [@superm1](https://github.com/superm1)             | [#738](ROCm/FastFlowLM#738) — drop the kernel check in `flm validate`  |
| [@yorickvP](https://github.com/yorickvP)           | [#747](ROCm/FastFlowLM#747) — remove the stray `.so.bak` file          |

***

#### 🌟 Summary

|    | Highlight                                                                                                                                 |
| -- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| 📦 | New model: **Qwen3.8-27B** (`qwen3.8-mtp:27b`) — reasoning and tool calling, the largest dense model FastFlowLM has shipped               |
| ⚡  | First model with **MTP speculative decoding**: a draft head proposes, the full model verifies — faster decode, identical output           |
| 🔄 | `hy-mt2:1.8b` renamed to `hy-mt2-flash:1.8b` — rename only; update pinned tags                                                            |
| 📊 | `usage.prompt_tokens` reports the full input, `cached_tokens` is now exposed, and `prefill_speed_tps` is no longer inflated by cache hits |
| ↩️ | `flm help`, `flm version`, and `flm port` exit 0 instead of 1                                                                             |
| ✅  | `flm validate` no longer rejects backported kernels older than 6.17                                                                       |
| 🧹 | Stray `.so.bak` backup file removed from the release packages                                                                             |

Thanks for your support — see you in the next one! 🚀

---
##### [\`v1.0.6\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.6)

This release introduces three **flash** models — kernel-optimized runtimes over the existing weights, so no re-download is needed — makes `hy-mt2:1.8b` single-turn, adds `tool_choice` support in server mode, fixes a tool-call `finish_reason` bug, and changes where the MSI installer writes `FLM_MODEL_PATH`.

***

#### ⚡ New Flash Models

FastFlowLM now supports three flash models. These are **not new checkpoints** — they run the *same weights* as their standard counterparts through optimized kernels, so there is **no weights update and nothing new to download**:

| Tag                 | Prefill  @ 128 ctx | Model card                                                                                                       |
| ------------------- | -----------------: | ---------------------------------------------------------------------------------------------------------------- |
| `gemma4e-flash:e2b` | **\~390 tokens/s** | [Gemma 4 E2B-IT · Flash](https://fastflowlm.com/docs/models/gemma/#-model-card-gemma-4-e2b-it--flash)            |
| `gemma4e-flash:e4b` | **\~256 tokens/s** | [Gemma 4 E4B-IT · Flash](https://fastflowlm.com/docs/models/gemma/#-model-card-gemma-4-e4b-it--flash)            |
| `qwen3vl-flash:4b`  | **\~350 tokens/s** | [Qwen3-VL 4B-Instruct · Flash](https://fastflowlm.com/docs/models/qwen/#-model-card-qwen3-vl-4b-instruct--flash) |

Run in CLI mode:

```bash
flm run gemma4e-flash:e2b
```

Run in server mode:

```bash
flm serve qwen3vl-flash:4b
```

##### 📖 What "Flash" Means

Flash models trade multi-turn flexibility for speed: the same weights run through optimized kernels, under a set of constraints that make those kernels possible. Please note the following behavior before deploying them:

- **Same weights, no re-download.** Flash models reuse the weights you already have — no `flm pull` required.
- **Single-turn only.** Any request containing an assistant message is rejected. Send a system prompt (optional) and a single user message.
- **System KV cache supported.** The system prompt is prefilled once and reused across requests, so repeated calls sharing a system prompt skip that prefill cost.
- **1k maximum context length.** Requests over 1k tokens are **rejected with a warning** — they are not silently truncated. Size your prompts accordingly.

##### 🖼️ Image & Audio Handling

Flash models apply a **fixed** media budget. No per-request tuning is required — oversized input is reduced automatically rather than rejected:

| Model                        | Images                                       | Audio                                 |
| ---------------------------- | -------------------------------------------- | ------------------------------------- |
| `qwen3vl-flash:4b`           | Resized so the longer side is **256 pixels** | Not supported                         |
| `gemma4e-flash:e2b` / `:e4b` | Resized to **70 tokens**                     | **Truncated to the first 30 seconds** |

> ⚠️ Note the difference from the context limit: oversized **media is silently reduced** (images downscaled, audio cut at 30 s), while an over-1k **prompt is rejected outright**. Audio longer than 30 seconds will not raise an error — the model simply never sees the remainder, so split long clips yourself if you need full coverage.

##### 🐍 Example: Python + OpenAI SDK (Streaming)

Flash models speak the standard OpenAI chat-completions API, so the official `openai` Python client works as-is — just point it at your local FLM server.

```bash
pip install openai
flm serve gemma4e-flash:e2b
```

**Vision, streaming, with a pinned system prompt:**

Below, the same system prompt is pinned across requests that each caption a **different** image. The system-prompt KV is prefilled once on the first request; every request after that reuses it, so only the new image and text need to be prefilled — expect noticeably lower latency from the second request onward.

```python
import time
import base64
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:52625/v1",
    api_key="dummykey",  # FLM runs locally; the key is not checked
)

# Keep everything fixed in the system prompt — it is prefilled once and
# reused across requests. Vary only the user message and image.
SYSTEM_PROMPT = "You are a concise assistant. Describe the image in one short sentence."

image_paths = [
    r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image0.jpg",
    r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image1.png",
]

for path in image_paths:
    with open(path, "rb") as image_file:
        image = base64.b64encode(image_file.read()).decode("utf-8")

    print(f"\n> {path}")
    start = time.time()
    stream = client.chat.completions.create(
        model="qwen3vl-flash:4b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},  # same every time — its KV is cached after the first call
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpg;base64,{image}"}},
                ],
            },
        ],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
    print(f"\n[time to first response: {time.time() - start:.2f}s]")
    # 1st image: pays the full system-prompt prefill cost.
    # 2nd image onward: system-prompt KV is reused from cache — expect lower latency.

# cleanup
del stream, client
import gc
gc.collect()
```

**Vision, streaming:**

Serve the vision flash model with `flm serve qwen3vl-flash:4b`, then pass an image as a base64 data URL. Resizing is automatic, so there is no need to downscale beforehand:

```python
import base64
from openai import OpenAI

image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="qwen3vl-flash:4b",
    messages=[
        {"role": "system", "content": "Describe images in one sentence."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

> **Note:** the image counts against the same 1k context budget as your text. Keep prompts short when sending an image.

**Audio, streaming:**

Audio goes to the same endpoint with the same message structure — only the content part changes. Use a `gemma4e-flash` model, since `qwen3vl-flash:4b` does not accept audio:

```python
import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {"role": "system", "content": "Transcribe and summarize audio briefly."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is said in this clip?"},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

> ⚠️ **Only the first 30 seconds are processed.** Longer clips are truncated without an error, so split them client-side if you need full coverage.

**Audio + image in one request:**

`gemma4e-flash` accepts both in the same `content` array, so a single call can reason over audio and an image together:

```python
import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this
image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")
with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Finish two tasks: 1. Summarize the audio. 2. Describe the image."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        }
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

***

#### 🔄 `hy-mt2:1.8b` Is Now Single-Turn

`hy-mt2:1.8b`, the multilingual translation model added in v1.0.5, is now a **single-turn** model as well. Like the flash models, it rejects any request containing an assistant message.

This matches how a dedicated translation model is actually used: every translation is independent, and prior turns add prefill cost without improving the result.

**What this means for you:** if you were replaying conversation history back to `hy-mt2:1.8b`, remove it — send only the instruction and the source text. The recommended shape from the v1.0.5 prompt guide is unchanged and remains the best option:

```json
{"role": "system", "content": "将以下文本翻译为英语,注意只需要输出翻译后的结果,不要额外解释。输出必须全部使用英语,不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}
```

Pinning the instruction as the system prompt and sending only the source text as the user message keeps the instruction out of every request's prefill — especially worthwhile for batch workloads such as translating subtitle lines.

***

#### 🛠️ Tool Calling

##### `tool_choice` Support (Server Mode)

Server mode now honors the `tool_choice` field. Two modes are supported:

| Value              | Behavior                                                                                                            |
| ------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `auto` *(default)* | The model decides whether to call a tool                                                                            |
| `none`             | Tools are still sent to the model, but tool-token logits are masked during decoding, so no tool call can be emitted |

Any other value falls back to `auto`.

##### 🐛 Bug Fix: Truncated Tool Calls Reported as `tool_calls`

Fixed a bug where a tool call truncated by `max_tokens` was reported with `finish_reason: "tool_calls"` alongside a tool call whose `name` and `arguments` were empty strings. Truncated generations now correctly report `finish_reason: "length"`.

***

#### 📦 MSI Installer

## The installer now sets `FLM_MODEL_PATH` as a **user** environment variable instead of a system one.

#### 🌟 Summary

|     | Highlight                                                                                                                                                                                     |
| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ⚡   | Three new flash models: `gemma4e-flash:e2b`, `gemma4e-flash:e4b`, `qwen3vl-flash:4b` — kernel-optimized over the same weights (no re-download), single-turn only, 1k context, system KV cache |
| 🔄  | `hy-mt2:1.8b` is now single-turn — assistant messages are rejected; send instruction + source text only                                                                                       |
| 🛠️ | `tool_choice` support in server mode: `auto` (default) and `none`                                                                                                                             |
| 🐛  | Truncated tool calls now report `finish_reason: "length"` instead of an empty `tool_calls`                                                                                                    |
| 📦  | MSI installer sets `FLM_MODEL_PATH` in the user environment instead of the system environment                                                                                                 |

Thanks for your support — see you in the next one! 🚀
renovate Bot added a commit that referenced this pull request Oct 24, 2026
##### [\`v1.0.9\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.9)

This is a maintenance release. It fixes **`flm validate` on Linux**, which wrongly reported "No NPU device found." in v1.0.8.

***

#### 🐛 Bug Fixes

##### ✅ `flm validate` Finds the NPU Again on Linux

**[PR #796](ROCm/FastFlowLM#796

In v1.0.8, `flm validate` on Linux printed **"No NPU device found."** and exited with status 1, even on systems where the NPU worked fine and `flm run` and `flm serve` had no problems.

The cause: since v1.0.8, `flm` opens the NPU device at startup, before it runs any command. `flm validate` then tried to open the same `/dev/accel/accel*` device a second time to check it. The amdxdna driver allows only one open of a device per process, so the second open failed with `EBUSY`. The check skipped busy devices without saying anything, so it found no device at all.

What changed:

- `flm validate` now runs **before** `flm` opens the NPU, so its check gets the device.
- If a device node is busy, `flm validate` now **reports it** (`device busy`) instead of skipping it silently. The `--json` output marks it with `"busy": true`.

If you run `flm validate` in a setup script or CI job, upgrade to get the right result back.

***

#### 📦 MSI Installer: Uninstall v1.0.8 First

The v1.0.8 MSI (`flm-setup.msi`) was published with the wrong version number. Windows registered it as version **8** instead of **1.0.8**, so it shows up as "8" in *Installed apps*. Because 8 is higher than 1.0.9, the v1.0.9 MSI treats v1.0.8 as a newer install and stops with **"A newer version of flm is already installed."**

If you installed v1.0.8 with the MSI, **uninstall FLM from Settings → Apps → Installed apps first**, then run the v1.0.9 MSI. Your downloaded models are kept. The v1.0.9 MSI has the right version number, so later upgrades will work normally again.

This only affects the MSI. The portable `.zip` packages and the Linux packages are not affected.

***

#### 🌟 Summary

|    | Highlight                                                                                                   |
| -- | ----------------------------------------------------------------------------------------------------------- |
| ✅  | `flm validate` on Linux finds the NPU again, instead of reporting "No NPU device found." (broken in v1.0.8) |
| 📦 | If you installed v1.0.8 with the MSI, uninstall it before running the v1.0.9 MSI                            |

Thanks for your support — see you in the next one! 🚀

---
##### [\`v1.0.8\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.8)

This release adds **`minicpm-v-4.7:1b`**, a 1B vision-language model with toggleable thinking and tool calling. It also **opens up the source of the Gemma 4 engine**, so you can build it yourself and swap individual operators through new override hooks. And for the first time, the release ships **HRX packages** as a preview: builds of FastFlowLM that run on the HRX NPU runtime instead of XRT.

***

#### 📦 New Model Support

##### 👁️ MiniCPM-V-4.7-1B

FastFlowLM now supports `minicpm-v-4.7:1b`, a compact image-text-to-text model from OpenBMB, with toggleable thinking and tool calling.

- **Tag:** `minicpm-v-4.7:1b`
- **Context:** 32k by default, up to 256k

Run in CLI mode:

```bash
flm run minicpm-v-4.7:1b
```

Run in server mode:

```bash
flm serve minicpm-v-4.7:1b
```

##### 🖼️ How It Sees Images

MiniCPM-V-4.7 handles images the same way the original MiniCPM-V does:

- The model always sees a **downscaled view of the whole image**. Large images are also split into **up to 9 tiles** of about 448 × 448 pixels each.
- **Image TTFT stops growing at about 1080p.** From that size up, every image uses the maximum 9 tiles, so a 4K image costs about the same as a 1080p one. Very fine detail in large images is downscaled.
- **`--img-pre-resize` (`-r`) has no effect** on this model, because the tiling already decides the input resolution.
- **Video and audio input are not supported.**

> 🎛️ MiniCPM-V does not ship its own sampling defaults. FLM starts from temperature 0.7, top_p 0.8, top_k 20 and presence penalty 1.5. These are the Qwen3.5 non-thinking defaults, because MiniCPM-V-4.7-1B's language model has the same architecture as Qwen3.5-0.8B.

For more details, see the [model card](https://fastflowlm.com/docs/models/minicpm/) and [benchmark results](https://fastflowlm.com/docs/benchmarks/minicpm-v-4.7_results/).

***

#### 🔓 The Gemma 4 Engine Is Now Open Source

**[PR #763](ROCm/FastFlowLM#763

Until now, every model engine in FastFlowLM shipped only as a prebuilt library. In this release the **source of the Gemma 4 E2B/E4B engine is public**. It covers the text decode and prefill paths, the vision encoder and the audio encoder, plus the shared host code they depend on. You can now read exactly how each Gemma 4 operator is dispatched to the NPU, and build the engine yourself. The other engines, including the Gemma 4 flash and 12B variants, still ship prebuilt.

##### 🛠️ Build It from Source

Source builds are **opt-in**. By default, FLM still links the prebuilt engine, so nothing changes unless you ask for it.

Linux:

```bash
cd src
cmake --preset linux-default -DFLM_BUILD_GEMMA4E=ON
cmake --build build
```

Windows (developer command prompt):

```bash
cd src
cmake --preset windows-default -DFLM_BUILD_GEMMA4E=ON
cmake --build build
```

The source-built library goes into the build tree and does not overwrite the prebuilt one. CMake caches the option, so to switch back, reconfigure with `-DFLM_BUILD_GEMMA4E=OFF` or use a fresh build directory.

| Option                   | Default | Effect                                                                                              |
| ------------------------ | ------: | --------------------------------------------------------------------------------------------------- |
| `FLM_BUILD_GEMMA4E`      |   `OFF` | Build the Gemma 4 engine from `src/detail/` instead of using the prebuilt one                       |
| `FLM_ENGINE_NATIVE_ARCH` |   `OFF` | Add `-march=native`. The result only runs on CPUs like the build machine, so do not redistribute it |
| `FLM_ENGINE_VERBOSE`     |     `0` | Logging level for the source-built engine                                                           |
| `FLM_ENGINE_DEBUG_LEVEL` |     `0` | Debug level for the source-built engine                                                             |
| `FLM_OVERRIDE_FLAGS`     |   empty | Extra compile flags for operator overrides (see below)                                              |

Source builds work on Linux, Windows and the HRX backend. CI now builds and sweep-tests a source-built Gemma 4 package on both Linux and Windows.

##### 🧩 Operator Override Hooks for IRON

Every Gemma 4 operator dispatch (52 sites across text, vision, audio and weight preload) is now wrapped in an `FLM_OVERRIDE` annotation. In a normal build the annotation expands to the original call, so the generated code is identical. To change that, point `FLM_OVERRIDES` at a header of your own, and that header can replace any operator by name. For example, it can run a kernel written with [IRON](https://github.com/amd/iron) in place of the stock one. Pass the define through `FLM_OVERRIDE_FLAGS`.

This lets you try a new NPU kernel inside a full Gemma 4 model without forking the engine. See [`src/include/flm_override.hpp`](https://github.com/ROCm/FastFlowLM/blob/main/src/include/flm_override.hpp) for the details.

For the full build instructions, see the [Linux install guide](https://fastflowlm.com/docs/install_lin/).

***

#### 🧪 HRX Packages (Preview)

FastFlowLM can run on two NPU runtimes: **XRT** (the default) and **HRX**, the amdxdna runtime. Starting with this release, HRX builds are published as a **preview**, next to the standard ones:

| Platform   | Asset                                    |
| ---------- | ---------------------------------------- |
| 🐧 Linux   | `HRX_fastflowlm_1.0.8_linux.tar.gz`      |
| 🪟 Windows | `HRX_fastflowlm_1.0.8_windows_amd64.zip` |

Each HRX package bundles the HRX runtime and HRX builds of the model engines. The HRX packages cover the same model lineup as the standard ones, including MiniCPM-V-4.7, Gemma 4 and Qwen3.8-27B. They are tested on every pull request by the same model sweep as the standard packages.

**Installing** works exactly like the standard portable packages: download the HRX asset for your platform and install it the same way you would install `fastflowlm_1.0.8_linux.tar.gz` or `fastflowlm_1.0.8_windows_amd64.zip`.

> ⚠️ **HRX is a preview.** Expect rough edges, and please [open an issue](https://github.com/ROCm/FastFlowLM/issues) if something works on the standard package but not on the HRX one. The standard XRT packages and installers are unchanged; if you are not sure which one you need, keep using the standard package.

***

#### 🌟 Summary

|    | Highlight                                                                                                                    |
| -- | ---------------------------------------------------------------------------------------------------------------------------- |
| 📦 | New model: **MiniCPM-V-4.7-1B** (`minicpm-v-4.7:1b`). Vision-language, toggleable thinking, tool calling, up to 256k context |
| 🔓 | **Gemma 4 E2B/E4B engine source is public.** Build it with `-DFLM_BUILD_GEMMA4E=ON`; the prebuilt engine stays the default   |
| 🧩 | **`FLM_OVERRIDE` hooks** on every Gemma 4 operator, so IRON kernels can stand in for the stock ones                          |
| 🧪 | **HRX packages (preview)** for Linux and Windows: FastFlowLM on the HRX NPU runtime, shipped next to the standard XRT builds |

Thanks for your support — see you in the next one! 🚀

---
##### [\`v1.0.7\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.7)

This release adds **`qwen3.8-mtp:27b`** — the largest dense model FastFlowLM has shipped to date, and the first to run **speculative decoding** on the NPU through its built-in MTP draft head. It also renames `hy-mt2:1.8b` to `hy-mt2-flash:1.8b` to match the flash naming convention, and picks up four community fixes to the OpenAI usage contract, CLI exit codes, `flm validate`, and a stray file in the release packages.

***

#### 📦 New Model Support

##### 🧠 Qwen3.8-27B

FastFlowLM now supports `qwen3.8-mtp:27b`, a 27B reasoning model with tool calling — the largest dense model FastFlowLM has shipped, and the first to run **speculative decoding** on the NPU through its built-in MTP draft head.

- **Tag:** `qwen3.8-mtp:27b`

Run in CLI mode:

```bash
flm run qwen3.8-mtp:27b
```

Run in server mode:

```bash
flm serve qwen3.8-mtp:27b
```

> 💾 This is by far the largest dense model FastFlowLM has shipped. Check that you have the disk space for the pull and enough system memory to hold it resident before running it.

##### ⚡ What MTP Speculative Decoding Means

Qwen3.8 ships with a **multi-token prediction (MTP) head** — a small draft model trained alongside the main network. FastFlowLM now puts it to work:

1. The MTP head **drafts** several candidate tokens in one shot.
2. The full model **verifies** them in a single pass.
3. Accepted drafts are kept; the first rejected one is replaced by the base model's own choice, and drafting restarts from there.

The practical consequences:

- **Faster decoding, same output.** Verification is an exact compare against what the base model would have emitted, so every token you receive is a token the base model would have produced on its own. Speculation changes throughput, not quality.
- **Gains are prompt-dependent.** Predictable, low-entropy stretches — code, structured output, long reasoning chains — see the highest draft acceptance. Highly creative or surprising text accepts fewer drafts and converges toward ordinary decode speed.
- **Nothing to configure.** Speculation is driven by the engine; there is no flag to set and no API change.

Every other model is unaffected — speculation is opt-in at the engine level, so the rest of the lineup decodes exactly as it did in v1.0.6.

For model details, context limits, and measured speedups, see the [model card](https://fastflowlm.com/docs/models/qwen/#-model-card-qwen38-27b) and [benchmark results](https://fastflowlm.com/docs/benchmarks/qwen3.8_results/).

***

#### 🔄 `hy-mt2:1.8b` → `hy-mt2-flash:1.8b`

The multilingual translation model introduced in v1.0.5 became single-turn in v1.0.6. Its name now says so: it is **`hy-mt2-flash:1.8b`**.

This is a **rename only** — same weights, same 1k context, same single-turn behavior, same translation quality. It simply brings the tag in line with `gemma4e-flash` and `qwen3vl-flash`, so that "flash" consistently signals the same contract: single-turn, fixed context, optimized kernels.

**Action required:** update any script, config, or client that pins the old tag.

```diff
- flm run hy-mt2:1.8b
+ flm run hy-mt2-flash:1.8b
```

```diff
- "model": "hy-mt2:1.8b"
+ "model": "hy-mt2-flash:1.8b"
```

The recommended prompt shape from the v1.0.5 guide is unchanged:

```json
{"role": "system", "content": "将以下文本翻译为英语,注意只需要输出翻译后的结果,不要额外解释。输出必须全部使用英语,不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}
```

For more details, see the [model card](https://fastflowlm.com/docs/models/hy-mt2/) and [benchmark results](https://fastflowlm.com/docs/benchmarks/hy-mt2_results/).

***

#### 🐛 Fixes from the Community

Four fixes in this release came from outside contributors. Thank you — these are exactly the kind of sharp, well-scoped reports that make the project better. 🙏

##### 📊 OpenAI `usage` Now Reports the Full Prompt Length

**[PR #729](ROCm/FastFlowLM#729) — thanks to [@Javinator9889](https://github.com/Javinator9889) (Javier Alonso)**

The prompt cache strips the matching prefix before prefill, so `usage.prompt_tokens` was reporting only the **newly evaluated suffix** rather than the whole input. The first turn of a conversation looked correct; every turn after that reused the history as a cached prefix and reported just the new message.

That broke agentic front-ends badly. Tools like OpenCode size their context from `usage.prompt_tokens`, so the context appeared to reset on every request — built-in compaction never triggered, and the client eventually hit a context overflow, re-fed the conversation, and overflowed again in a loop.

What changed:

- **`usage.prompt_tokens`** now counts the entire input, cached prefix included — matching the OpenAI specification.
- **`usage.prompt_tokens_details.cached_tokens`** is now exposed on the OpenAI endpoints, so clients can see how much of that input was served from cache.
- **`prefill_speed_tps`** now divides by the tokens actually evaluated, so a cache hit no longer inflates the reported prefill speed.
- **Ollama-compatible `prompt_eval_count` is unchanged** — it continues to report evaluated tokens only, matching upstream Ollama.

If you drive FLM from an agent framework that manages its own context window, this is the fix to upgrade for.

##### ↩️ `flm help`, `flm version`, and `flm port` Now Exit 0

**[PR #723](ROCm/FastFlowLM#723) — thanks to [@jtuyls](https://github.com/jtuyls) (Jorn Tuyls)**

These three commands succeeded and then exited with status **1**, because they stopped argument parsing the same way a usage error does. Any script, CI job, or shell with `set -e` that ran `flm version` treated a perfectly good invocation as a failure.

They are now handled as successful commands — and handled *before* the model list is loaded, so `flm help` and `flm version` no longer pay that startup cost. Genuine usage errors still exit 1.

##### ✅ `flm validate` No Longer Hard-Checks the Kernel Version

**[PR #738](ROCm/FastFlowLM#738) — thanks to [@superm1](https://github.com/superm1) (Mario Limonciello)**

`flm validate` required kernel 6.17 or newer. Several distributions backport the NPU driver to considerably older kernels, so working setups were being reported as invalid.

The check is now removed. Validation already checks the **firmware version**, which implicitly requires a recent enough driver — making the kernel version test both redundant and wrong for backported kernels.

##### 🧹 Stray Backup File Removed from the Release Packages

**[PR #747](ROCm/FastFlowLM#747) — thanks to [@yorickvP](https://github.com/yorickvP)**

`src/lib/xrt/libq4_npu_eXpress.so.bak-20260826` — a backup copy of a shared library — was committed by accident and had been shipping inside the release packages, despite being used by neither the build nor the executable. It has been deleted, so the Linux artifacts are a little smaller.

***

#### 🙏 Acknowledgements

| Contributor                                        | Contribution                                                                                   |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| [@Javinator9889](https://github.com/Javinator9889) | [#729](ROCm/FastFlowLM#729) — full prompt length in OpenAI `usage`     |
| [@jtuyls](https://github.com/jtuyls)               | [#723](ROCm/FastFlowLM#723) — exit 0 for `help`, `version`, and `port` |
| [@superm1](https://github.com/superm1)             | [#738](ROCm/FastFlowLM#738) — drop the kernel check in `flm validate`  |
| [@yorickvP](https://github.com/yorickvP)           | [#747](ROCm/FastFlowLM#747) — remove the stray `.so.bak` file          |

***

#### 🌟 Summary

|    | Highlight                                                                                                                                 |
| -- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| 📦 | New model: **Qwen3.8-27B** (`qwen3.8-mtp:27b`) — reasoning and tool calling, the largest dense model FastFlowLM has shipped               |
| ⚡  | First model with **MTP speculative decoding**: a draft head proposes, the full model verifies — faster decode, identical output           |
| 🔄 | `hy-mt2:1.8b` renamed to `hy-mt2-flash:1.8b` — rename only; update pinned tags                                                            |
| 📊 | `usage.prompt_tokens` reports the full input, `cached_tokens` is now exposed, and `prefill_speed_tps` is no longer inflated by cache hits |
| ↩️ | `flm help`, `flm version`, and `flm port` exit 0 instead of 1                                                                             |
| ✅  | `flm validate` no longer rejects backported kernels older than 6.17                                                                       |
| 🧹 | Stray `.so.bak` backup file removed from the release packages                                                                             |

Thanks for your support — see you in the next one! 🚀

---
##### [\`v1.0.6\`](https://github.com/ROCm/FastFlowLM/releases/tag/v1.0.6)

This release introduces three **flash** models — kernel-optimized runtimes over the existing weights, so no re-download is needed — makes `hy-mt2:1.8b` single-turn, adds `tool_choice` support in server mode, fixes a tool-call `finish_reason` bug, and changes where the MSI installer writes `FLM_MODEL_PATH`.

***

#### ⚡ New Flash Models

FastFlowLM now supports three flash models. These are **not new checkpoints** — they run the *same weights* as their standard counterparts through optimized kernels, so there is **no weights update and nothing new to download**:

| Tag                 | Prefill  @ 128 ctx | Model card                                                                                                       |
| ------------------- | -----------------: | ---------------------------------------------------------------------------------------------------------------- |
| `gemma4e-flash:e2b` | **\~390 tokens/s** | [Gemma 4 E2B-IT · Flash](https://fastflowlm.com/docs/models/gemma/#-model-card-gemma-4-e2b-it--flash)            |
| `gemma4e-flash:e4b` | **\~256 tokens/s** | [Gemma 4 E4B-IT · Flash](https://fastflowlm.com/docs/models/gemma/#-model-card-gemma-4-e4b-it--flash)            |
| `qwen3vl-flash:4b`  | **\~350 tokens/s** | [Qwen3-VL 4B-Instruct · Flash](https://fastflowlm.com/docs/models/qwen/#-model-card-qwen3-vl-4b-instruct--flash) |

Run in CLI mode:

```bash
flm run gemma4e-flash:e2b
```

Run in server mode:

```bash
flm serve qwen3vl-flash:4b
```

##### 📖 What "Flash" Means

Flash models trade multi-turn flexibility for speed: the same weights run through optimized kernels, under a set of constraints that make those kernels possible. Please note the following behavior before deploying them:

- **Same weights, no re-download.** Flash models reuse the weights you already have — no `flm pull` required.
- **Single-turn only.** Any request containing an assistant message is rejected. Send a system prompt (optional) and a single user message.
- **System KV cache supported.** The system prompt is prefilled once and reused across requests, so repeated calls sharing a system prompt skip that prefill cost.
- **1k maximum context length.** Requests over 1k tokens are **rejected with a warning** — they are not silently truncated. Size your prompts accordingly.

##### 🖼️ Image & Audio Handling

Flash models apply a **fixed** media budget. No per-request tuning is required — oversized input is reduced automatically rather than rejected:

| Model                        | Images                                       | Audio                                 |
| ---------------------------- | -------------------------------------------- | ------------------------------------- |
| `qwen3vl-flash:4b`           | Resized so the longer side is **256 pixels** | Not supported                         |
| `gemma4e-flash:e2b` / `:e4b` | Resized to **70 tokens**                     | **Truncated to the first 30 seconds** |

> ⚠️ Note the difference from the context limit: oversized **media is silently reduced** (images downscaled, audio cut at 30 s), while an over-1k **prompt is rejected outright**. Audio longer than 30 seconds will not raise an error — the model simply never sees the remainder, so split long clips yourself if you need full coverage.

##### 🐍 Example: Python + OpenAI SDK (Streaming)

Flash models speak the standard OpenAI chat-completions API, so the official `openai` Python client works as-is — just point it at your local FLM server.

```bash
pip install openai
flm serve gemma4e-flash:e2b
```

**Vision, streaming, with a pinned system prompt:**

Below, the same system prompt is pinned across requests that each caption a **different** image. The system-prompt KV is prefilled once on the first request; every request after that reuses it, so only the new image and text need to be prefilled — expect noticeably lower latency from the second request onward.

```python
import time
import base64
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:52625/v1",
    api_key="dummykey",  # FLM runs locally; the key is not checked
)

# Keep everything fixed in the system prompt — it is prefilled once and
# reused across requests. Vary only the user message and image.
SYSTEM_PROMPT = "You are a concise assistant. Describe the image in one short sentence."

image_paths = [
    r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image0.jpg",
    r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image1.png",
]

for path in image_paths:
    with open(path, "rb") as image_file:
        image = base64.b64encode(image_file.read()).decode("utf-8")

    print(f"\n> {path}")
    start = time.time()
    stream = client.chat.completions.create(
        model="qwen3vl-flash:4b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},  # same every time — its KV is cached after the first call
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpg;base64,{image}"}},
                ],
            },
        ],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
    print(f"\n[time to first response: {time.time() - start:.2f}s]")
    # 1st image: pays the full system-prompt prefill cost.
    # 2nd image onward: system-prompt KV is reused from cache — expect lower latency.

# cleanup
del stream, client
import gc
gc.collect()
```

**Vision, streaming:**

Serve the vision flash model with `flm serve qwen3vl-flash:4b`, then pass an image as a base64 data URL. Resizing is automatic, so there is no need to downscale beforehand:

```python
import base64
from openai import OpenAI

image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="qwen3vl-flash:4b",
    messages=[
        {"role": "system", "content": "Describe images in one sentence."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

> **Note:** the image counts against the same 1k context budget as your text. Keep prompts short when sending an image.

**Audio, streaming:**

Audio goes to the same endpoint with the same message structure — only the content part changes. Use a `gemma4e-flash` model, since `qwen3vl-flash:4b` does not accept audio:

```python
import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {"role": "system", "content": "Transcribe and summarize audio briefly."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is said in this clip?"},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

> ⚠️ **Only the first 30 seconds are processed.** Longer clips are truncated without an error, so split them client-side if you need full coverage.

**Audio + image in one request:**

`gemma4e-flash` accepts both in the same `content` array, so a single call can reason over audio and an image together:

```python
import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this
image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")
with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Finish two tasks: 1. Summarize the audio. 2. Describe the image."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        }
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

***

#### 🔄 `hy-mt2:1.8b` Is Now Single-Turn

`hy-mt2:1.8b`, the multilingual translation model added in v1.0.5, is now a **single-turn** model as well. Like the flash models, it rejects any request containing an assistant message.

This matches how a dedicated translation model is actually used: every translation is independent, and prior turns add prefill cost without improving the result.

**What this means for you:** if you were replaying conversation history back to `hy-mt2:1.8b`, remove it — send only the instruction and the source text. The recommended shape from the v1.0.5 prompt guide is unchanged and remains the best option:

```json
{"role": "system", "content": "将以下文本翻译为英语,注意只需要输出翻译后的结果,不要额外解释。输出必须全部使用英语,不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}
```

Pinning the instruction as the system prompt and sending only the source text as the user message keeps the instruction out of every request's prefill — especially worthwhile for batch workloads such as translating subtitle lines.

***

#### 🛠️ Tool Calling

##### `tool_choice` Support (Server Mode)

Server mode now honors the `tool_choice` field. Two modes are supported:

| Value              | Behavior                                                                                                            |
| ------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `auto` *(default)* | The model decides whether to call a tool                                                                            |
| `none`             | Tools are still sent to the model, but tool-token logits are masked during decoding, so no tool call can be emitted |

Any other value falls back to `auto`.

##### 🐛 Bug Fix: Truncated Tool Calls Reported as `tool_calls`

Fixed a bug where a tool call truncated by `max_tokens` was reported with `finish_reason: "tool_calls"` alongside a tool call whose `name` and `arguments` were empty strings. Truncated generations now correctly report `finish_reason: "length"`.

***

#### 📦 MSI Installer

## The installer now sets `FLM_MODEL_PATH` as a **user** environment variable instead of a system one.

#### 🌟 Summary

|     | Highlight                                                                                                                                                                                     |
| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ⚡   | Three new flash models: `gemma4e-flash:e2b`, `gemma4e-flash:e4b`, `qwen3vl-flash:4b` — kernel-optimized over the same weights (no re-download), single-turn only, 1k context, system KV cache |
| 🔄  | `hy-mt2:1.8b` is now single-turn — assistant messages are rejected; send instruction + source text only                                                                                       |
| 🛠️ | `tool_choice` support in server mode: `auto` (default) and `none`                                                                                                                             |
| 🐛  | Truncated tool calls now report `finish_reason: "length"` instead of an empty `tool_calls`                                                                                                    |
| 📦  | MSI installer sets `FLM_MODEL_PATH` in the user environment instead of the system environment                                                                                                 |

Thanks for your support — see you in the next one! 🚀
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant