Float8WeightOnlyConfig weight-only quantization via torchao.quantization.quantize_() results in dramatically worse energy efficiency compared to FP16 baseline on RTX 5090 (Blackwell, sm_100). The penalty escalates with model size: +158% at 0.5B → +701% at 7B parameters. At 7B, power draw reaches 448W (near the 575W TDP) while throughput collapses to 10.5 tok/s (vs 70 tok/s FP16).

For comparison, bitsandbytes NF4 (4-bit) achieves −11.5% energy savings at the same 7B scale.

Environment

Minimal Reproduction

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from torchao.quantization import quantize_, Float8WeightOnlyConfig

model_id = "Qwen/Qwen2.5-7B-Instruct"  # any HF causal LM

# Load FP16 baseline
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Apply FP8 weight-only quantization
quantize_(model, Float8WeightOnlyConfig())

# Generate — observe high power draw, low throughput
inputs = tokenizer("Hello", return_tensors="pt").to("cuda")
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Running the above on RTX 5090 produces correct output, but power monitoring (NVML 10Hz sampling) shows ~448W average and only ~10.5 tok/s throughput.

Benchmark Data (RTX 5090, Qwen2.5-7B, batch_size=1, 256 tokens, greedy)

FP8 is the worst of all 5 precision modes tested, despite achieving the expected ~46% memory reduction (14.19 → 7.61 GB).

Full Model-Size Sweep (FP8 only)

The energy penalty escalates with model size, suggesting the overhead is per-weight rather than fixed.

Suspected Causes

1. C++ extension incompatibility: We observed torchao C++ extensions failing to compile against the nightly PyTorch build (torch 2.12.0.dev vs torchao 0.17.0.dev), likely forcing Python-only fallback code paths for FP8 dequantization.

2. No fused inference kernels: Float8WeightOnlyConfig appears optimized for training-oriented FP8 (compute scaling), not inference-oriented weight-only dequantization. The per-tensor scaling overhead without kernel fusion leads to extreme compute inefficiency.

3. Power anomaly: FP8 draws more power than FP16 (448W vs 375W at 7B), which is unexpected for a method that reduces memory traffic. This suggests the GPU is spending cycles on dequantization compute rather than idling on memory.

Questions for Maintainers

1. Is Float8WeightOnlyConfig intended to work efficiently for inference on Blackwell, or is it primarily a training-oriented API?
2. Are there known issues with C++ extensions on torch 2.12.0.dev + cu128?
3. Would Float8DynamicActivationFloat8WeightConfig or a different FP8 path be more appropriate for inference throughput?
4. Is there a recommended way to verify whether the fused CUDA kernels are actually being used (vs Python fallback)?

Context

This data is from a systematic energy efficiency study of quantization methods across Ada Lovelace and Blackwell architectures. Full dataset (113+ measurements) and benchmark code available at:

• Code: https://github.com/hongping-zh/ecocompute-ai
• Benchmark script: bench_fp8_vs_all.py

We would be happy to run additional diagnostics or test a patched version if the team can suggest one.