Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

Qwen3.8-Flash-Next on an Intel Arc B580 (12 GB) in a Proxmox LXC

Run an 84 GB mixture-of-experts model on a 12 GB GPU at about 30 to 35 tokens per second with a 256K context, inside an unprivileged Proxmox container, using Strata. Plus a 103-item coding benchmark comparing two quantized builds of the model.

Companion blog post: An 84 GB Model on a 12 GB GPU: Running Qwen3.8-Flash-Next on an Intel Arc B580 in Proxmox.

Results at a glance

ISTA-DASLab Coder IQ1_M DavidAU IQ3_S (abliterated)
Benchmark, all 103 graded items 96 (93%) 100 (97%)
Coding, structured spec / casual prompt 45/48, 21/24 46/48, 23/24
Hardest tasks at xhigh reasoning 4/5 5/5
Tool calling / agent repos 16/16, 10/10 16/16, 10/10
Generation speed (coding) 34.2 tok/s 30.2 tok/s
256K needle test (5 facts in 243K tokens) 5/5 5/5
Experts in VRAM 1,896 of 12,288 (15%) 1,742 of 24,576 (7%)
Pinned host RAM ~23 GiB ~47 GiB
Host RAM left (64 GB host) ~31 GiB ~8 GiB

Verdict: DavidAU IQ3_S is the default, because it is more robust to casual prompts and to SQL/Bash work. The Coder build is the better choice if you need the RAM. Full analysis: docs/12-results.md.

Other measured results:

  • MTP speculative decoding: 24.1 to 34.1 tok/s (1.41x) with 51/51 graded answers unchanged (docs/07-mtp.md)
  • Context: 128K and 256K are stable, with all 5 needles found at both. 512K (YaRN) ran the host out of memory (docs/08-context-scaling.md)
  • Prompt cache: a 3.4K-token prefix went from 6.0 s cold to 0.49 s warm (docs/10-api-usage.md)
  • Thermals under hours of load: GPU 65 to 70 C, about 110 W, no throttling (docs/01-hardware-and-host.md)

The setup

Proxmox VE 9.2 host (Ryzen 7 5700X, 64 GB RAM, Arc B580 12 GB, xe driver stays on the host)
└── unprivileged LXC, Ubuntu 26.04, 8 cores, /dev/dri/renderD128 + card0 passed in, memlock unlimited
    ├── strata.service        :8080  DavidAU IQ3_S   256K ctx, MTP, lazy load, 20 min idle unload
    └── strata-coder.service  :8081  Coder IQ1_M     256K ctx, MTP, lazy load, 20 min idle unload
        (a before_load hook unloads the other model first, so only one is ever on the GPU)

How it fits: Strata keeps the dense weights and a cache of hot experts in VRAM, mirrors every other expert in pinned host RAM, and the GPU reads missing experts over PCIe. The KV cache for long contexts also lives in pinned RAM (--kv-resident), which leaves more VRAM for experts.

Documentation

01. Hardware, host, and why GPU passthrough was dropped The VFIO reset that froze the whole host, and thermals
02. The LXC container Sizing, /dev/dri passthrough, and the memory accounting trap
03. Installing Strata on Arc What the installer does not do on Intel, and how to run it as a service
04. The run config, flag by flag Every engine argument and server key explained
05. Building the pack by hand Fixing native_experts.txt does not match the model
06. The two models GSQ-RCO explained, file sizes, the shared shard
07. MTP speculative decoding Fetching the draft layer by hand, and an A/B quality test
08. Context scaling 32K to 512K Needle tests, memory math, and the OOM incident
09. On-demand loading, two models one GPU lazy_load, idle_unload_s, and the unload hook
10. Using the API OpenAI Chat/Responses, Anthropic Messages, prompt caching
11. Benchmark methodology 103 items, code-graded, with fairness checks
12. Results Every number and every failure explained
13. Troubleshooting Every error hit, with its fix
14. Lessons learned

Quick start

These steps assume a Proxmox host with an Arc card on the xe driver and at least 64 GB of RAM. Read docs/02 and docs/08 before you load anything: the model's pinned memory is not limited by the container.

# on the Proxmox host (edit CTID, storage names and GIDs first)
scripts/setup/01-create-ct.sh

# inside the container
scripts/setup/02-ct-gpu-runtime.sh     # Level Zero + OpenCL, check clinfo sees the B580
scripts/setup/03-install-strata.sh     # Strata 0.2.3
scripts/setup/04-download-models.sh    # ~114 GB, pinned revisions, sha256-checked
scripts/setup/05-build-packs.sh        # seconds
scripts/setup/06-mtp-draft.sh          # ~5.2 GB download, MTP draft layer
# put your API key into configs/strata/*.json, then:
scripts/setup/07-install-services.sh

Then:

curl http://<ct-ip>:8080/v1/chat/completions -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model":"x","messages":[{"role":"user","content":"hello"}],"reasoning_effort":"low"}'

The first request loads the model, which takes about a minute.

Repository layout

docs/          the write-ups above
configs/       proxmox CT config, Strata run configs (key removed), systemd units, unload hook
scripts/       setup/ (numbered, in order) and monitor/ (host RAM, GPU/CPU, Strata status)
benchmark/     codebench/ (the 103-item suite) and earlier/ (MTP, context, reasoning tests)
results/       codebench/ raw JSONL + summary, earlier-tests/ raw JSON

Versions

Proxmox VE 9.2.21 (kernel 7.0.14), Ubuntu 26.04.1 in the container, Strata 0.2.3 (engine 0.1.48 SYCL), intel-opencl-icd / libze-intel-gpu1 26.05.37020.3, libze1 1.28.2, Node 22 and Python 3.14 for the benchmark.

Credits

  • Strata by Niko1221: the engine that makes this possible
  • ISTA-DASLab: the GSQ-RCO quantizations and the expert-pruned Coder build
  • DavidAU and SC117: the abliterated IQ3_S build
  • Qwen team: Qwen3.8-Flash-Next

License

Code (benchmark/, scripts/, configs/) is MIT. Docs and results are CC BY 4.0. See LICENSE. Model weights and Strata are under their own licenses and are not included.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages