Skip to content

Commit fa6c830

Browse files
IlyasMoutawwakilCopilotArthurZuckersywangyiclaude
authored
DeepGEMM BF16 + mixed FP8/FP4 + MegaMoE + refactor (#45634)
* init * style * full support * support EP better using offsets ! * comments * get rid of neutralize_ep_sentinels * remove deepgemm stuff * fix * prefix * move * fix * remove comment * fix unintilized outputs leaking * revert unnecessary changes * more unnecessary changes * revert downcast * keep it simple * guard deepgemm cuda version * fix style * update * add deepgemm testing * moe sentinel support * fix * compilable sonicmoe * mega moe kernel support attempt * use package for now * skip ep router and experts pre/post processing * simpler * fix Co-authored-by: Copilot <[email protected]> * fix * fix * dtensor support * more dtensor * simpler * remove comment * revert * bc order * revert extra indent * revert unnecessary change * update * less defensive * allow all kernels * alow all kernels * hub only * fix * fix * test * test * sync * check nvcc * probe * fix * test psum * test * test * probe * fix * test * nan issue * repro * repro * fix * simplifications * fix * fix * fix * fix * fix * fix * fix * empty * simplify * test deepseek * dsv4 only * download dsv4 * fix test * push * test * fix * fix ep plan * fix attempt * debug * attempt * debug * fixes in modeling * more modeling changes * more modeling changes * initial megamoe works but wrong output, deepgemm fully functional * mega moe works ! * simplification * comments * fix merge * fix rank divergence * use _keep_in_fp32_modules * simplify quantizer * introduce indexerscorer * remove testing file * style * weight rename for scorer * style * address review comments * style * align with laguna * stale comments and styling * use grouped finegrained fp8 for grouped linear Co-authored-by: sywangyi [email protected] * comments and docs * Revert "[DeepSeek V4] Fix MoE converter substring-matching FP8 scale companions (#45930)" This reverts commit ecc7e32. * fix quantizer and use upstream deepgemm * remove force al kernels * [deepseek_v4] fix FP8 save-side substring match + fp32 gate through softmax * core_model_loading: WeightConverter.reverse_transform now appends a (?=\.|$) token-boundary lookahead to each reversed source so they can't substring-match longer sibling tokens. Re-fixes #45794 (the FP8Experts scale companion `mlp.experts.gate_up_proj_scale_inv` was being routed into the weight-only [Chunk, SplitModulelist] ops on save) without needing `$` on every model-specific converter — that was #45930's approach and broke load via `process_source_pattern`. * Add `ConversionOps.supports_round_trip` opt-in flag; the on-the-fly `quantization_operation` check in reverse_transform now uses it instead of a blanket bail-out. Flagged the ops with audited reverse pairs (Chunk/Concatenate, MergeModulelist/SplitModulelist, Transpose, Conv3d↔Linear, Ernie pair, _IdentityOp, Fp8Quantize/Fp8Dequantize). * deepseek_v4 compressors (HCA / Indexer / CSA): drop the trailing `.to(chunk_gate.dtype)` so the gate stays fp32 from the `+ position_bias` through the softmax — matches the reference Compressor.forward which upcasts x to fp32 at entry. * deepseek_v4 config: only set `attention_factor=1.0` when the compress group resolves to `rope_type="yarn"`. Suppresses the noisy "Unrecognized keys in rope_parameters for rope_type=default" warning on the second post_init pass (where yarn params have already been consumed). * Refresh slow integration test expectations to match the fp32-gate output. Verified on the published DeepSeek-V4-Flash checkpoint: RUN_SLOW=1 pytest tests/models/deepseek_v4 -k Integration -> 2 passed. * always floats scal inv with triton * minimal changes i quantizer * force stable abi build for now * support transposed * fix yarn * unnecessary change * add flash base test and fix the incorrect and crash issue during enab… (#46076) * fix incorrect output in tests/models/deepseek_v4/test_modeling_deepseek_v4.py::DeepseekV4IntegrationTest::test_v4_flash_fp8_generation and add test for flash-base Signed-off-by: Wang, Yi <[email protected]> * update Signed-off-by: Wang, Yi <[email protected]> --------- Signed-off-by: Wang, Yi <[email protected]> * merge * better errors across deepgemm * more better errors for deepgemm * ep tests * Apply suggestions from code review Co-authored-by: Arthur <[email protected]> * scope supports_round_trip to quantization ops only Drop the flag from the regular shape ops (Chunk, Concatenate, MergeModulelist, SplitModulelist, Transpose, Conv3d↔Linear, _IdentityOp, the Ernie pair) — those reverse themselves via their own `reverse_op` property, no flag needed. The flag only gates `WeightConverter.reverse_transform`'s check on the converter's `quantization_operation`, so today it's just Fp8Quantize / Fp8Dequantize. Other quant backends (bnb, torchao, eetq, mxfp4, quanto, ...) stay opt-out until their round-trip is audited. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * Apply suggestions from code review Co-authored-by: Arthur <[email protected]> * Update src/transformers/core_model_loading.py * style: ruff format Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * custom ep/tp plan for megamoe * process megamoe after weight loading * fix megamoe process * up * the actual fix??? * up * nit comment for futur * style: drop unused exception variable in Fp8Dequantize._dequantize_one * up * style: apply make fix-repo docstring formatting * simplify after merge * review comments * revert unncessary * avoid name contraction * fixes * dynamic tp/ep plan swap * fix deepgemm single error and device assertion docs * properly support bias in grouped linear * simplify * fix dist error handling and min cuda compatbility * fix least cuda compat * get alignment from deepgemm and cache it --------- Signed-off-by: Wang, Yi <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Arthur Zucker <[email protected]> Co-authored-by: Wang, Yi <[email protected]> Co-authored-by: Arthur <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
1 parent a46a732 commit fa6c830

16 files changed

Lines changed: 1627 additions & 486 deletions

‎src/transformers/conversion_mapping.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -329,6 +329,10 @@ def _build_checkpoint_conversion_mapping():
329329
source_patterns=r"^layers\.(\d+)\.self_attn\.indexer\.",
330330
target_patterns=r"layers.\1.self_attn.compressor.indexer.",
331331
),
332+
WeightRenaming(
333+
source_patterns=r"^layers\.(\d+)\.self_attn\.compressor\.indexer\.weights_proj\.",
334+
target_patterns=r"layers.\1.self_attn.compressor.indexer.scorer.weights_proj.",
335+
),
332336
WeightRenaming(
333337
source_patterns=r"^layers\.(\d+)\.self_attn\.compressor\.norm\.",
334338
target_patterns=r"layers.\1.self_attn.compressor.kv_norm.",

‎src/transformers/integrations/deepgemm.py‎

Lines changed: 748 additions & 0 deletions
Large diffs are not rendered by default.

‎src/transformers/integrations/finegrained_fp8.py‎

Lines changed: 374 additions & 400 deletions
Large diffs are not rendered by default.

‎src/transformers/integrations/hub_kernels.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -288,7 +288,7 @@ def register_kernel_mapping_transformers(*args, **kwargs):
288288
"mamba-ssm": {"repo_id": "kernels-community/mamba-ssm", "version": 1},
289289
"falcon_mamba-ssm": {"repo_id": "kernels-community/mamba-ssm", "version": 1},
290290
"finegrained-fp8": {"repo_id": "kernels-community/finegrained-fp8", "version": 1},
291-
"deep-gemm": {"repo_id": "kernels-community/deep-gemm", "version": 1},
291+
"deep-gemm": {"repo_id": "kernels-community/deep-gemm", "version": 2},
292292
"sonic-moe": {"repo_id": "kernels-community/sonic-moe", "revision": "ep-support"},
293293
}
294294

@@ -376,7 +376,7 @@ def lazy_load_kernel(kernel_name: str, mapping: dict[str, ModuleType | None] = _
376376
repo_id = _HUB_KERNEL_MAPPING[kernel_name]["repo_id"]
377377
revision = _HUB_KERNEL_MAPPING[kernel_name].get("revision", None)
378378
version = _HUB_KERNEL_MAPPING[kernel_name].get("version", None)
379-
kernel = get_kernel(repo_id, revision=revision, version=version)
379+
kernel = get_kernel(repo_id, revision=revision, version=version, allow_all_kernels=ALLOW_ALL_KERNELS)
380380
mapping[kernel_name] = kernel
381381
except FileNotFoundError as e:
382382
mapping[kernel_name] = None

‎src/transformers/integrations/moe.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
is_torch_less_or_equal,
2525
is_torchdynamo_compiling,
2626
)
27+
from .deepgemm import deepgemm_bf16_experts_forward
2728
from .sonicmoe import sonicmoe_experts_forward
2829

2930

@@ -133,7 +134,8 @@ def batched_mm_experts_forward(
133134
# Clamp EP sentinels so `gate_up_proj[expert_ids]` stays in-bounds. Routing weights are already
134135
# zero at sentinel slots (RouterParallel masks them at dispatch), so the weighted mul drops
135136
# those contributions — we pay the wasted GEMM compute because batched_mm has no offset to skip.
136-
expert_ids.clamp_(0, self.num_experts - 1)
137+
# Out-of-place to avoid mutating the caller's routing tensor (a contiguous `reshape(-1)` aliases it).
138+
expert_ids = expert_ids.clamp(0, self.num_experts - 1)
137139

138140
# Select gate_up or just up projection weights and biases
139141
if self.has_gate:
@@ -483,6 +485,7 @@ class ExpertsInterface(GeneralInterface):
483485
"""Interface for registering custom experts forward functions."""
484486

485487
_global_mapping = {
488+
"deepgemm": deepgemm_bf16_experts_forward,
486489
"batched_mm": batched_mm_experts_forward,
487490
"grouped_mm": grouped_mm_experts_forward,
488491
"sonicmoe": sonicmoe_experts_forward,

‎src/transformers/integrations/sonicmoe.py‎

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828

2929
from ..utils import logging
3030
from .hub_kernels import lazy_load_kernel
31+
from .tensor_parallel import to_local
3132

3233

3334
logger = logging.get_logger(__name__)
@@ -169,18 +170,10 @@ def sonicmoe_experts_forward(
169170
# already zero (RouterParallel masks them at dispatch), so the per-token reduction
170171
# contributes nothing for sentinel slots.
171172

172-
# FSDP2 / EP wraps weights as DTensors but the kernel takes raw CUTLASS / CuteDSL pointers,
173-
# so unwrap to local shards before reshaping. `to_local()` is autograd-aware — backward
174-
# will rewrap the gradient as a DTensor matching each parameter's placements.
175-
w1 = self.gate_up_proj
176-
w2 = self.down_proj
177-
b1 = self.gate_up_proj_bias if self.has_bias else None
178-
b2 = self.down_proj_bias if self.has_bias else None
179-
if isinstance(w1, torch.distributed.tensor.DTensor):
180-
w1 = w1.to_local()
181-
w2 = w2.to_local()
182-
b1 = b1.to_local() if b1 is not None else None
183-
b2 = b2.to_local() if b2 is not None else None
173+
w1 = to_local(self.gate_up_proj)
174+
w2 = to_local(self.down_proj)
175+
b1 = to_local(self.gate_up_proj_bias) if self.has_bias else None
176+
b2 = to_local(self.down_proj_bias) if self.has_bias else None
184177

185178
# Map activation function
186179
act_name = getattr(self.config, "hidden_act", "silu").lower()

‎src/transformers/integrations/tensor_parallel.py‎

Lines changed: 80 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,21 @@
3737
logger = logging.get_logger(__name__)
3838

3939

40+
def to_local(t):
41+
"""Unwrap a `DTensor` to its local shard if needed; pass through otherwise.
42+
43+
Custom kernels (CUTLASS, CuteDSL, Triton) take raw tensor pointers and don't
44+
understand `DTensor`, so weights wrapped by FSDP2 / EP need this unwrap before
45+
they can be fed to the kernel. ``to_local()`` is autograd-aware on the train
46+
path: backward rewraps the gradient as a DTensor matching each parameter's
47+
placements.
48+
"""
49+
if hasattr(torch.distributed, "tensor") and hasattr(torch.distributed.tensor, "DTensor"):
50+
if isinstance(t, torch.distributed.tensor.DTensor):
51+
return t.to_local()
52+
return t
53+
54+
4055
def initialize_tensor_parallelism(
4156
tp_plan: str | dict[str, str] | None, tp_size: int | None = None, device_mesh=None, device_map=None
4257
):
@@ -773,6 +788,25 @@ def _backward_hook(mod, grad_input, grad_output, mesh=device_mesh):
773788
module.register_full_backward_hook(_backward_hook)
774789

775790

791+
class AllReduceParallel(TensorParallelLayer):
792+
"""All-reduce a module's forward output across the TP mesh. Use as a declarative
793+
sync point at the boundary of a multi-arg module whose compute ends in a partial
794+
sum (e.g. the lightning indexer's score sum before its top-k).
795+
"""
796+
797+
def _prepare_input_fn(self, mod, inputs, device_mesh):
798+
return inputs
799+
800+
def _prepare_output_fn(self, mod, outputs, device_mesh):
801+
return all_reduce_forward(outputs, device_mesh)
802+
803+
def shard_tensor(self, param, tensor_idx=None, device=None, dtype=None):
804+
return param[...].to(device=device, dtype=dtype)
805+
806+
def prepare_module_tp(self, module, device_mesh, **kwargs):
807+
distribute_module(module, device_mesh, output_fn=self._prepare_output_fn)
808+
809+
776810
class MlaKvAProjParallel(TensorParallelLayer):
777811
"""
778812
For MLA attention used in DeepSeek-V2 style models (deepseek_v2, longcat_flash, glm_moe_dsa, glm4_moe_lite):
@@ -1095,7 +1129,7 @@ def __init__(self, **kwargs):
10951129
super().__init__(**kwargs)
10961130

10971131
def _prepare_input_fn(self, mod, inputs, device_mesh):
1098-
return inputs[0] if inputs else inputs
1132+
return inputs
10991133

11001134
def _prepare_output_fn(self, mod, outputs, device_mesh):
11011135
"""
@@ -1173,6 +1207,19 @@ def shard_tensor(
11731207
return param[...].to(device=device, dtype=dtype)
11741208

11751209

1210+
class RouterParallelMegaMoe(RouterParallel):
1211+
"""Router TP plan used with DeepGEMM Mega MoE.
1212+
1213+
Mega MoE handles EP dispatch inside the kernel and wants raw global expert ids
1214+
with unmasked routing weights, so the router doesn't pre-shard per EP rank like
1215+
`RouterParallel._prepare_output_fn` does. The quantizer's `update_tp_plan` swaps
1216+
`"ep_router"` → `"megamoe_router"` when `experts_implementation == "deepgemm_megamoe"`.
1217+
"""
1218+
1219+
def _prepare_output_fn(self, mod, outputs, device_mesh):
1220+
return outputs
1221+
1222+
11761223
class MoeTensorParalellExperts(TensorParallelLayer):
11771224
"""
11781225
Note: For tensor parallel, the MoEExpertsParallel TP layer handles gradient sync:
@@ -1198,7 +1245,7 @@ def _prepare_input_fn(self, mod, inputs, device_mesh):
11981245
# and partial_expert_output is different on each GPU before all-reduce
11991246
top_k_weights = all_reduce_backward(top_k_weights, device_mesh)
12001247

1201-
return (hidden_states, top_k_index, top_k_weights)
1248+
return hidden_states, top_k_index, top_k_weights
12021249

12031250
def _prepare_output_fn(self, mod, outputs, device_mesh):
12041251
# all_reduce_forward to sum partial expert outputs across GPUs
@@ -1212,6 +1259,32 @@ def shard_tensor(
12121259
return param[...].to(device=device, dtype=dtype)
12131260

12141261

1262+
class MoeTensorParalellMegaMoeExperts(TensorParallelLayer):
1263+
"""TP layer for DeepGEMM Mega MoE experts.
1264+
1265+
Mega MoE is inference-only (the kernel has no backward) and handles EP dispatch +
1266+
combine + per-rank token sharding internally — so we skip the gradient-sync hooks
1267+
that the regular `MoeTensorParalellExperts` would apply, and we forward the EP
1268+
`process_group` into the module so the symm-buffer rendezvous can run on first
1269+
forward. The quantizer's `update_tp_plan` swaps the experts plan key from
1270+
`"moe_tp_experts"` to `"megamoe_experts"` when
1271+
`from_pretrained(..., experts_implementation="deepgemm_megamoe")`.
1272+
"""
1273+
1274+
def _prepare_input_fn(self, mod, inputs, device_mesh):
1275+
hidden_states, top_k_index, top_k_weights = inputs[0], inputs[1], inputs[2]
1276+
return hidden_states, top_k_index, top_k_weights, device_mesh.get_group()
1277+
1278+
def _prepare_output_fn(self, mod, outputs, device_mesh):
1279+
# Kernel returned the fully-combined gathered output; no further reduction.
1280+
return outputs
1281+
1282+
def shard_tensor(
1283+
self, param: torch.Tensor, tensor_idx: int | None = None, device=None, dtype=None
1284+
) -> torch.Tensor:
1285+
return param[...].to(device=device, dtype=dtype)
1286+
1287+
12151288
class MoeIdentityExpertParallel(TensorParallelLayer):
12161289
"""
12171290
TP class for zero/identity experts in MoE layers.
@@ -1250,10 +1323,13 @@ class ParallelInterface(GeneralInterface):
12501323
"sequence_parallel": SequenceParallel(),
12511324
"grouped_gemm": GroupedGemmParallel(),
12521325
"ep_router": RouterParallel(),
1326+
"megamoe_router": RouterParallelMegaMoe(),
12531327
"moe_tp_experts": MoeTensorParalellExperts(),
1328+
"megamoe_experts": MoeTensorParalellMegaMoeExperts(),
12541329
"moe_identity_expert": MoeIdentityExpertParallel(),
12551330
"replicated_with_grad_allreduce": ReplicatedWithGradAllReduce(),
12561331
"mla_kv_a_proj": MlaKvAProjParallel(),
1332+
"all_reduce": AllReduceParallel(),
12571333
}
12581334
if is_torch_available() and _torch_distributed_available
12591335
else {}
@@ -1274,6 +1350,7 @@ class ParallelInterface(GeneralInterface):
12741350
"sequence_parallel": None,
12751351
"replicated_with_grad_allreduce": None,
12761352
"mla_kv_a_proj": None,
1353+
"all_reduce": None,
12771354
}
12781355

12791356
# Bias sharding: colwise shards bias, rowwise doesn't (bias is replicated and all-reduced)
@@ -1289,6 +1366,7 @@ class ParallelInterface(GeneralInterface):
12891366
"sequence_parallel": None,
12901367
"replicated_with_grad_allreduce": None,
12911368
"mla_kv_a_proj": None,
1369+
"all_reduce": None,
12921370
}
12931371

12941372
@classmethod

‎src/transformers/modeling_utils.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2212,6 +2212,18 @@ def set_experts_implementation(self, experts_implementation: str | dict):
22122212
else experts_implementation.get("", self.config._experts_implementation)
22132213
)
22142214

2215+
# MegaMoE is locked at load time: its TP plan is baked into `base_model_tp_plan`
2216+
# by `update_tp_plan` (and isn't re-evaluated) and `setup_megamoe_weights`
2217+
# mutates the expert weights into UTCCP layout on first forward. Either side of
2218+
# a switch would silently produce garbage, so reject it with a clear pointer.
2219+
current = self.config._experts_implementation
2220+
if "deepgemm_megamoe" in (current, requested_implementation) and current != requested_implementation:
2221+
raise RuntimeError(
2222+
f"Cannot switch experts implementation from {current!r} to {requested_implementation!r} "
2223+
"at runtime: `deepgemm_megamoe` is a load-time choice. Reload via "
2224+
"`from_pretrained(..., experts_implementation=...)` to switch."
2225+
)
2226+
22152227
if requested_implementation != self.config._experts_implementation:
22162228
requested_implementation = self._check_and_adjust_experts_implementation(requested_implementation)
22172229
# Apply the change (on the internal attr, to avoid setting it recursively)

‎src/transformers/models/deepseek_v4/configuration_deepseek_v4.py‎

Lines changed: 26 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -112,20 +112,28 @@ class DeepseekV4Config(PreTrainedConfig):
112112
"norm": (["hidden_states"], ["hidden_states"]),
113113
}
114114
base_model_ep_plan = {
115-
# EP-only by default, same shape as gpt-oss: route on the gate, run the
116-
# routed experts as a grouped-GEMM kernel sharded along the expert axis,
117-
# and wrap the experts module with `moe_tp_experts` so its output gets
118-
# all-reduced across ranks. Attention stays replicated (V4 is shared-KV
119-
# MQA + a CSA / HCA compressor branch — both broadcast a single KV head
120-
# across all attention heads via `repeat_kv`, so colwise-sharding
121-
# `q_b_proj` would leave KV replicated and `repeat_kv` would no longer
122-
# match the rank-local query head count). The shared MLP also stays
123-
# replicated — it's small and not worth TP-ing. There's deliberately
124-
# no `base_model_tp_plan` for V4: we don't ship a pure-TP plan, only EP.
115+
# V4 ships EP only (no `base_model_tp_plan` — the runtime picks one plan or
116+
# the other, never both, and V4 is MoE so EP is the only sensible config).
117+
# MoE parallelism: route on the gate, run the routed experts as a grouped-GEMM
118+
# kernel sharded along the expert axis, and wrap the experts module with
119+
# `moe_tp_experts` so its output gets all-reduced across ranks. Same shape as
120+
# gpt-oss. Main attention stays replicated: V4 is shared-KV MQA + a CSA / HCA
121+
# compressor branch — both broadcast a single KV head across all attention
122+
# heads via `repeat_kv`, so colwise-sharding `q_b_proj` would leave KV
123+
# replicated and `repeat_kv` would no longer match the rank-local query head
124+
# count. The shared MLP also stays replicated — it's small and not worth
125+
# sharding. The Lightning Indexer is the one carve-out: its keys are
126+
# replicated (own compressor at index_head_dim fed by replicated
127+
# hidden_states), so head-sharding is well-formed; `q_b_proj` and the
128+
# `scorer.weights_proj` go colwise, and the `scorer` output is all-reduced
129+
# so every rank sees the same `index_scores` and picks the same top-k.
125130
"layers.*.mlp.gate": "ep_router",
126131
"layers.*.mlp.experts.gate_up_proj": "grouped_gemm",
127132
"layers.*.mlp.experts.down_proj": "grouped_gemm",
128133
"layers.*.mlp.experts": "moe_tp_experts",
134+
"layers.*.self_attn.compressor.indexer.q_b_proj": "colwise",
135+
"layers.*.self_attn.compressor.indexer.scorer.weights_proj": "colwise",
136+
"layers.*.self_attn.compressor.indexer.scorer": "all_reduce",
129137
}
130138

131139
vocab_size: int = 129280
@@ -283,12 +291,14 @@ def __post_init__(self, **kwargs):
283291
)
284292
self.qk_rope_head_dim = int(self.head_dim * self.partial_rotary_factor)
285293

286-
# yarn is applied ONLY to layers with a
287-
# compressor (CSA/HCA); pure sliding-window layers use plain RoPE with
288-
# `theta=rope_theta` (10000) and no scaling. Compress layers use
289-
# `theta=compress_rope_theta` (160000) with yarn factor=16, and the reference
290-
# does NOT multiply cos/sin by the yarn mscale — force `attention_factor=1.0`
291-
# so transformers' `_compute_yarn_parameters` doesn't apply `0.1·log(16)+1`.
294+
# yarn is applied ONLY to layers with a compressor (CSA/HCA); pure
295+
# sliding-window layers use plain RoPE with `theta=rope_theta` (10000) and no
296+
# scaling. Compress layers use `theta=compress_rope_theta` (160000) with yarn
297+
# factor=16. For a YaRN compress layer we force `attention_factor=1.0`: the V4
298+
# reference does NOT multiply cos/sin by YaRN's computed mscale, and leaving the
299+
# key unset lets `_compute_yarn_parameters` derive `0.1·log(factor)+1 ≈ 1.277`.
300+
# We only inject the key when `rope_type="yarn"` — the default-rope validator
301+
# rejects it (and `compute_default_rope_parameters` hardcodes 1.0 anyway).
292302
rp = self.rope_parameters or {}
293303
if isinstance(rp.get("main"), dict) and isinstance(rp.get("compress"), dict):
294304
# Already nested — drop any leftover top-level keys.

0 commit comments

Comments
 (0)