Skip to content

Commit f9e9d73

Browse files
author
Afloat
authored
fix(vla_jepa): pair rotary frequencies with adjacent coordinates (#4835)
* fix(vla_jepa): pair rotary frequencies with adjacent coordinates * docs(vla_jepa): trace the upstream rotary correction and reduce test diff
1 parent 2e88044 commit f9e9d73

2 files changed

Lines changed: 11 additions & 2 deletions

File tree

‎docs/source/policy_vla_jepa_README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,12 @@ Converted from [ginwind/VLA-JEPA](https://huggingface.co/ginwind/VLA-JEPA).
1616

1717
At inference time only the Qwen backbone and action head are used; the world model is not needed.
1818

19+
### World-model training compatibility
20+
21+
The rotary frequency expansion comes from [V-JEPA 2](https://github.com/facebookresearch/vjepa2/blob/main/src/models/utils/modules.py). [Upstream PR #15](https://github.com/facebookresearch/vjepa2/pull/15) by echosprint proposed the pairwise correction; [PR #75](https://github.com/facebookresearch/vjepa2/pull/75) documented it while keeping the original expansion for released-checkpoint compatibility. This VLA-JEPA change applies that known correction to the training-only world predictor.
22+
23+
The correction to pairwise rotary frequencies keeps the world predictor's parameter names and shapes unchanged. Existing checkpoints continue to load, but predictor outputs and world-model training losses change relative to the earlier frequency-block repetition. Pretrained world-model quality needs separate evaluation when continuing training.
24+
1925
---
2026

2127
## Citation

‎src/lerobot/policies/vla_jepa/world_model.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,11 @@ def rotate_queries_or_keys(x: torch.Tensor, pos: torch.Tensor) -> torch.Tensor:
4848
omega /= dim / 2.0
4949
omega = 1.0 / 10000**omega
5050
freqs = torch.einsum("..., f -> ... f", pos, omega)
51-
emb_sin = freqs.sin().squeeze(-1).repeat(1, 1, 1, 2)
52-
emb_cos = freqs.cos().squeeze(-1).repeat(1, 1, 1, 2)
51+
# Each adjacent coordinate pair is rotated by one frequency.
52+
# Upstream proposal: https://github.com/facebookresearch/vjepa2/pull/15
53+
# Checkpoint compatibility notes: https://github.com/facebookresearch/vjepa2/pull/75
54+
emb_sin = freqs.sin().repeat_interleave(2, dim=-1)
55+
emb_cos = freqs.cos().repeat_interleave(2, dim=-1)
5356

5457
y = x.unflatten(-1, (-1, 2))
5558
y1, y2 = y.unbind(dim=-1)

0 commit comments

Comments
 (0)