Skip to content

About

Evidence-backed Qwen3.8-27B comparison across independent machine studies; Apple Silicon remains explicitly unmeasured.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Qwen3.8-27B multi-machine benchmark comparison

This repository is the comparison layer for independent Qwen3.8-27B GGUF machine studies. It currently compares two completed Windows/CUDA measurements and reserves an explicitly empty Apple Silicon/macOS row.

Current evidence status: RTX 4090 measured; RTX 4070 Ti measured; Apple Silicon not measured. No Mac result is estimated or inferred.

4K decode throughput comparison

Headline measured results

Under the shared 4K baseline operating point (llama.cpp b10448, 84 prompt tokens, 256 completion tokens, one slot, q8_0 K/V, MTP off), the RTX 4090 measured:

Quantization RTX 4090 RTX 4070 Ti RTX 4090 uplift
UD-IQ2_XXS 73.885 tok/s 43.643 tok/s 69.294%
UD-Q2_K_XL 65.898 tok/s 38.030 tok/s 73.279%

The IQ2 context ladder also used matching 3,231-, 6,423-, and 12,831-token prompts:

IQ2 context throughput comparison

At the 16K configuration, the RTX 4090 produced 65.434 tok/s versus 39.201 tok/s on the RTX 4070 Ti, a 66.919% uplift.

What the comparison does and does not say

  • The 4K IQ2 and Q2 rows are the cleanest cross-machine performance comparison because their runtime, model identity, prompt, sampling controls, cache types, slot count, and full 66/66 CUDA layer placement match.
  • The context chart is a matched IQ2 synthetic workload comparison, not a long-context reasoning benchmark.
  • Objective-suite scores are retained in the normalized data for auditability, but they are not treated as GPU performance. IQ2 scored 8/24 on the RTX 4090 replication and 9/24 on the RTX 4070 Ti study; this one-task difference is not a hardware-quality claim.
  • IQ4_XS is not charted across the GPUs: the RTX 4090 used 66/66 CUDA layers, while the RTX 4070 Ti used a hybrid 45/66 operating point. Comparing those values as though only the GPU changed would be misleading.
  • MTP speedups are workload-specific and remain opt-in evidence. Both studies kept MTP off by default because at least one paired prose output changed.
  • The Mac track stays not_measured with null results until Phase 0 and later approved measurement phases run on the actual Apple Silicon machine.

Reproduce the comparison

Only Python's standard library is required:

python scripts/generate_comparison.py
python scripts/verify_comparison.py

The generator reads the normalized machine records in data/machines, writes data/comparison.json, and regenerates the SVG figures. CI rejects stale generated files and invented Mac measurements.

See METHODOLOGY.md for inclusion rules and SOURCES.md for pinned upstream evidence.

Independent study repositories

Licensed under Apache-2.0.

About

Evidence-backed Qwen3.8-27B comparison across independent machine studies; Apple Silicon remains explicitly unmeasured.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages