E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring

Official code for E-PMQ — expert-guided post-merge quantization (PMQ) for merged models.

News

• 2026-05-16: Paper released on arXiv and Hugging Face Papers.

Paper overview

Model merging integrates multiple task experts into one full-precision checkpoint; post-merge quantization then compresses that merged model for deployment. Ordinary PTQ assumes the full-precision network is a trustworthy reconstruction target. After merging, quantization deviation (low-bit reconstruction error) and expert-relative merging deviation (differences between merged weights and source experts) are coupled: naive PMQ that reconstructs only the merged weights can accumulate both errors.

E-PMQ addresses this with:

1. Expert-guided calibration targets — layer-wise objectives use source expert weights to form informative output targets on the calibration streams (aligned with the GPTQ-style sequential solver).
2. Merged-weight anchoring — a ridge-style penalty keeps quantized weights close to the merged checkpoint $W_m$, preserving integrated merged behavior and avoiding collapse when the anchor is removed.

Layer-wise, the continuous relaxation takes the form

$$ \min_Q ;\sum_{i=1}^{K} |Q X_i^{\ell} - W_i^{\ell} X_i^{\ell}|_F^2 + \lambda_{\ell} |Q - W_m^{\ell}|_F^2 , $$

where expert statistics and the anchor appear in the effective normal equations used by the implementation.

For figures and extended tables see the paper PDF in assets/.

The implementation keeps a single-model inference path: same parameter count and deployment form as standard weight-only PTQ after quantization finishes.

Paper highlights

Broader tables (per-task vision, extended CLIP scales, anchor and bit-width analyses) are in the paper PDFs under assets/. Representative headline numbers:

Setting	Baseline (avg.)	+ E-PMQ (avg.)
CLIP ViT-B/32, 8 tasks, Task Arithmetic, 4-bit GPTQ	65.0	73.6
CLIP ViT-B/32, 8 tasks, TIES-Merging, 4-bit GPTQ	69.1	74.8
CLIP ViT-L/14, 20 tasks, Task Arithmetic, 4-bit GPTQ	34.8	76.7
FLAN-T5-base GLUE, Task Arithmetic, 4-bit GPTQ	78.26	83.34
Llama-3.1-3B MergeBench-style mix, Task Arithmetic, 4-bit GPTQ	58.71	60.27
Llama-3.1-8B (same protocol), Task Arithmetic, 4-bit GPTQ	61.66	62.91

Scores follow the paper’s reporting conventions (top-1 accuracy % for CLIP, GLUE metrics / STS-B Spearman for T5, benchmark mixes for LLMs). Small differences can arise from dependency versions, checkpoint paths, and evaluation harness versions.

Repository layout

Path	Description
CLIP/	Merged CLIP vision quantization with the feature-matching GPTQ E-PMQ solver; RTN / GPTQ / AWQ baselines.
T5/	Standalone Flan-T5 / GLUE quantization for rtn, gptq, awq, and E-PMQ.
LLM/	Causal LM fake-quant export (rtn, gptq, awq, epmq) and driver scripts for merged-expert quantization; optional vLLM eval via external APC scripts.
assets/E-PMQ_arxiv.pdf	Paper PDF (arXiv-style preprint in this bundle).
assets/E-PMQ.pdf	Alternate paper PDF copy.

Generated experiment outputs, cached datasets, and Python bytecode are not checked in. New runs write outputs under each module’s results/ directory (see per-module README files).

Experimental defaults (paper-aligned)

The paper’s main quantized experiments use 4-bit weight-only quantization, 256 calibration samples per task (so $256K$ total samples for $K$ merged tasks on CLIP), forward-order layer-wise calibration, and evaluation only on test splits after quantization. Appendix G of the bundled PDF states group size 128, calibration batch size 32, and evaluation batch size 128. Anchor strength $\alpha$ is set per merging family (e.g. $\alpha = 0.01$ for Simple Averaging, Task Arithmetic, and TIES-Merging in the reported CLIP setup). Adjust these in each module’s registry scripts to match your checkpoints.

Environment

The code expects PyTorch and the Hugging Face stack. Typical dependencies include:

• torch
• transformers
• datasets
• tqdm
• numpy

CUDA is recommended for large runs. LLM evaluation optionally uses ${APC_ROOT}/script-llm/eval.sh (see LLM/README.md).

If downloads should stay inside this repo, set standard Hugging Face cache variables before running:

export HF_HOME=$PWD/.cache/huggingface
export HF_DATASETS_CACHE=$HF_HOME/datasets
export HF_HUB_CACHE=$HF_HOME/hub

Use HF_ENDPOINT or your usual proxy/mirror settings if your network requires them; nothing is hard-coded here.

Clone

git clone https://github.com/wwjzhy/E-PMQ.git
cd E-PMQ

If you are working from a fork or mirror, clone that URL instead and treat assets/ as the local copy of the paper PDFs.

Quick start

List configured CLIP merged models:

bash CLIP/scripts/run_alpha_baseline.sh --list-models

List configured T5 merged models:

bash T5/scripts/run_t5_quant_baseline.sh --list-models

List configured LLM merged checkpoints (slug → path):

bash LLM/scripts/run_llm_quant_baseline.sh --list-models

Smoke tests

Run T5 baselines without full evaluation:

METHODS="rtn" TASKS="sst2" N_CALIBRATION=4 SKIP_EVAL=true \
  bash T5/scripts/run_t5_quant_baseline.sh

Use --dry-run or --skip-eval on the CLIP batch scripts for cheap pipeline checks (see CLIP/README.md).

Full reproduction

Before reproducing full paper-scale runs, update checkpoint roots and the model matrix in the corresponding scripts/experiment_registry*.sh files (LLM/scripts/experiment_registry_llm*.sh for causal LM). Submodule READMEs document the exact drivers:

• CLIP: CLIP/README.md — run_alpha_eval.sh, run_alpha_baseline.sh
• T5: T5/README.md — run_t5_quant_eval.sh, run_t5_quant_baseline.sh
• LLM: LLM/README.md — run_llm_quant_eval.sh, run_llm_quant_baseline.sh, run_llm_quant_all.sh

Implementation notes

• Naive PMQ baselines (rtn, gptq, awq on the merged checkpoint) reconstruct the merged weights directly.
• E-PMQ uses the same GPTQ-style sequential rounding loop but replaces the Hessian/right-hand side so they encode expert-guided targets plus merged-weight anchoring (paper Eq. (15) and Appendix B–C).
• Default T5 experts load from tanganke/flan-t5-base_glue-{task} unless overridden (--expert_model_template when calling t5_quant_eval.py directly).

Citation

If you find this repository useful, please cite the paper (update with the official arXiv record once available):

@misc{wang2026epmqexpertguidedpostmergequantization,
      title={E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring}, 
      author={Wenjun Wang and Yanggan Gu and Shuo Cai and Yuanyi Wang and Pengkai Wang and Jianmin Wu and Hongxia Yang},
      year={2026},
      eprint={2605.16882},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.16882}, 
}

Corresponding author (paper): Hongxia Yang ([email protected]). Contact on the title page: [email protected].