[Olmo3] different RoPE per layer type by zucchini-nlp · Pull Request #46911 · huggingface/transformers

zucchini-nlp · 2026-06-26T08:39:46Z

What does this PR do?

Reverts back per layer-type RoPE in Olmo which was removed in #39847

zucchini-nlp · 2026-06-26T08:42:13Z

+    def test_real_model_7b_greedy_generation(self):
+        expectations = Expectations(
+            {
+                ("cuda", None): 'system\nYou are a helpful function-calling AI assistant. You do not currently have access to any functions. <functions></functions>\nuser\nWho would win in a fight - a dinosaur or a cow named Moo Moo?\nassistant\nThis is a fun and imaginative question! Let’s break it down:\n\n### 1. **A Dinosaur (General Case)**\nDinosaurs were a huge and diverse group, spanning from tiny feathered raptors to massive sauropods like *Brachiosaurus* or *Tyrannosaurus rex',
+            }
+        )  # fmt: skip
+


there were no integration tests with official ckpt, somehow it redirects to someone's personal repo 🫠

I added tests with ckpt that are supposed to use rope scaling, and we should see a difference now . Adding test for long seq beyond sliding window in a sec

i didn't delete existing slow tests, not sure if that repo is supposed to be tested. LMK if you think we can delete everything to not waste resources on running them

imo, best to keep but move/copy to internal testing. personal repos is not so nice

no need to save on resources, rather have something non broken

HuggingFaceDocBuilderDev · 2026-06-26T09:01:42Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

zucchini-nlp · 2026-06-26T09:57:46Z

run-slow: olmo3, olmo_hybrid

zucchini-nlp · 2026-06-26T09:58:45Z

+        # Released ckpt don't use any ROPE and have  it set to `None`
        self.rotary_emb = (
            OlmoHybridRotaryEmbedding(config=config)
            if getattr(config, "rope_parameters", None) is not None


ig we can't delete the module, since some users might have added rope in non-official ckpts

yep rather keep it now, that's why it's hard when official ckpts release after integration :/

zucchini-nlp · 2026-06-26T11:00:11Z

run-slow: olmo3, olmo_hybrid

github-actions · 2026-06-26T11:01:42Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/olmo3", "models/olmo_hybrid"]
quantizations: []

github-actions · 2026-06-26T11:11:52Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	de5cfa59	workflow commit (merge commit)
PR	a6dfe8d0	branch commit (from PR)
main	ed7d6c8d	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

vasqu

I'm only a bit hesitant re that non downcasting on the rope of olmo3 - any source for that?

But other than that agree with most points, just nits/smaller comments

vasqu · 2026-06-26T11:29:19Z

+    def test_real_model_7b_greedy_generation(self):
+        expectations = Expectations(
+            {
+                ("cuda", None): 'system\nYou are a helpful function-calling AI assistant. You do not currently have access to any functions. <functions></functions>\nuser\nWho would win in a fight - a dinosaur or a cow named Moo Moo?\nassistant\nThis is a fun and imaginative question! Let’s break it down:\n\n### 1. **A Dinosaur (General Case)**\nDinosaurs were a huge and diverse group, spanning from tiny feathered raptors to massive sauropods like *Brachiosaurus* or *Tyrannosaurus rex',
+            }
+        )  # fmt: skip
+


imo, best to keep but move/copy to internal testing. personal repos is not so nice

no need to save on resources, rather have something non broken

vasqu · 2026-06-26T11:31:09Z

-        config, _ = self.model_tester.prepare_config_and_inputs_for_common()
+    @is_tensor_parallel_test
+    def test_tp_generation_quantized(self):
+        # If model uses rope-theta 50k (default value), the test fails


that is surprising, wondering whether it also affects other models 👀

ikr, very weird but I didn't want to dig yet. Most models init with 10k by default except for really weird ones, I will check a bit later and open an issue/another PR if necessary

hmm seems like it is an error accumulation from QK norm over the whole hidden-dim and higher theta, I'll just leave it there as there are more models skipping this particular test with todo: investigate
maybe we'd need to raise tols for certain cases

vasqu · 2026-06-26T11:32:34Z

+        # Released ckpt don't use any ROPE and have  it set to `None`
        self.rotary_emb = (
            OlmoHybridRotaryEmbedding(config=config)
            if getattr(config, "rope_parameters", None) is not None


yep rather keep it now, that's why it's hard when official ckpts release after integration :/

vasqu · 2026-06-26T11:36:12Z

            self.num_key_value_heads = self.num_attention_heads
        super().__post_init__(**kwargs)

+    def convert_rope_params_to_dict(self, **kwargs):


not on you but we should avoid it whenever we can

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

github-actions · 2026-06-26T12:28:53Z

CI Dashboard: View test results in Grafana

github-actions · 2026-06-29T08:41:59Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: olmo3, olmo_hybrid

Cyrilvallez · 2026-07-28T06:24:11Z

cc @zucchini-nlp @vasqu why are the cos/sin not downcasted at the end here like any other Rope module? Any reason? This means any later ops will be upcasted in the model...

zucchini-nlp · 2026-07-28T08:32:42Z

@Cyrilvallez yeas, it is done on purpose to follow official ref impl. QK are then recasted back to model dtype after rotating

transformers/src/transformers/models/olmo3/modeling_olmo3.py

Lines 121 to 126 in dff4572

    
           q_type, k_type = q.dtype, k.dtype 
        
           cos = cos.unsqueeze(unsqueeze_dim) 
        
           sin = sin.unsqueeze(unsqueeze_dim) 
        
           q_embed = (q * cos) + (rotate_half(q) * sin) 
        
           k_embed = (k * cos) + (rotate_half(k) * sin) 
        
           return q_embed.to(q_type), k_embed.to(k_type)

Cyrilvallez · 2026-07-28T08:45:08Z

All right, thanks for confirming!

* fix maybe, needs checking * fix * oops * fix repo * fix rope tests * why was it added if ckpt has no rope? * fix the TP test * adjust expectations with runners * Update tests/models/olmo3/test_modeling_olmo3.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * spit out personal repo test in a new class --------- Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

) A lens encodes one forward pass, and the library version decides what that pass is. transformers 5.0 to 5.12 applied OLMo-3's YaRN rope_scaling to its sliding-window layers too; 5.13.0 (huggingface/transformers#46911) restored per-layer-type RoPE. Both OLMo-3 lenses were fitted on 5.11.0 and are served on 5.14.1, so they do not match the model (#235). - pyproject: transformers>=5.13.0; relock (5.11.0 -> 5.17.0). - run-all-fit-lens.py: write an `environment` block (transformers, torch, accelerate, datasets, python, jlens_commit) into config.yaml and its header. - fit_lens.py: store the same facts as a `provenance` dict inside the .pt, in the JacobianLens.save layout so upstream load() and our lens_loader read it unchanged. torch.__version__ is coerced to str because the weights_only unpickler refuses TorchVersion. - backfill-config-environment.py: add the block to configs already on the Hub without touching weights, and open a PR there (neuronpedia/jacobian-lens#5). It also marks the two OLMo-3 lenses and rewrites qwen3-32b's results from its own CSV, flagging that .pt as a partial checkpoint (Hub discussion #3). Fixes #235. Co-authored-by: Cursor <cursoragent@cursor.com>

zucchini-nlp added 3 commits June 25, 2026 17:43

fix maybe, needs checking

5989c48

fix

20b842d

oops

23c2ad4

zucchini-nlp commented Jun 26, 2026

View reviewed changes

zucchini-nlp added 3 commits June 26, 2026 11:07

fix repo

f2a7a64

fix rope tests

07977af

why was it added if ckpt has no rope?

72940c0

zucchini-nlp commented Jun 26, 2026

View reviewed changes

Comment thread tests/causal_lm_tester.py Resolved

zucchini-nlp commented Jun 26, 2026

View reviewed changes

zucchini-nlp added 2 commits June 26, 2026 12:46

fix the TP test

9f3e779

adjust expectations with runners

a6dfe8d

huggingface deleted a comment from github-actions Bot Jun 26, 2026

zucchini-nlp requested a review from vasqu June 26, 2026 10:58

vasqu approved these changes Jun 26, 2026

View reviewed changes

zucchini-nlp and others added 2 commits June 26, 2026 13:44

Update tests/models/olmo3/test_modeling_olmo3.py

367404a

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

spit out personal repo test in a new class

5f7f39c

zucchini-nlp added this pull request to the merge queue Jun 29, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 29, 2026

Merge branch 'main' into olmo3-rope

12b9e1d

zucchini-nlp added this pull request to the merge queue Jun 29, 2026

Merged via the queue into huggingface:main with commit 326683d Jun 29, 2026
5 checks passed

zucchini-nlp deleted the olmo3-rope branch June 29, 2026 08:52

LukaIhn mentioned this pull request Jul 6, 2026

Fix OLMo3 sliding attention RoPE parameters vllm-project/vllm#43419

Draft

hmellor mentioned this pull request Jul 8, 2026

Bump Transformers version to 5.13.1 vllm-project/vllm#47867

Merged

yihao-liang mentioned this pull request Jul 14, 2026

Add native OLMo3 (Olmo3ForCausalLM) support by reusing the Olmo2 implementation sgl-project/sglang#31175

Closed

zucchini-nlp mentioned this pull request Jul 16, 2026

Fix OLMo 3 scaled RoPE handling for sliding attention #45945

Closed

This was referenced Sep 5, 2026

OLMo-3 lenses fitted with transformers < 5.13 use the wrong RoPE on sliding-window layers anthropics/jacobian-lens#15

Open

olmo-3-1025-7b Jacobian lens was fitted under transformers 5.11.0, which applies YaRN to the wrong layers hijohnnylin/neuronpedia#235

Closed

hijohnnylin mentioned this pull request Sep 20, 2026

jlens: require transformers >= 5.13 and record the fit environment hijohnnylin/neuronpedia#247

Merged

Conversation

zucchini-nlp commented Jun 26, 2026

What does this PR do?

Uh oh!

zucchini-nlp Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Jun 26, 2026

Uh oh!

Uh oh!

zucchini-nlp commented Jun 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

CI Results

Commit Info

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

Uh oh!

Cyrilvallez commented Jul 28, 2026

Uh oh!

zucchini-nlp commented Jul 28, 2026

Uh oh!

Cyrilvallez commented Jul 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zucchini-nlp Jun 26, 2026 •

edited

Loading