You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There has been some effort in hf/transformers to extract and share RoPE types across models, so we should be able to unify more, especially the loading/conversion parts. Some of the improvements are still in progress. A few relevant PRs:
They added a single object that encapsulates RoPE config, including rope base, rope percentage and rope scaling. They have attributes from all scaling strategies flat in the config, but we can stick to our :scaling_strategy discriminated union.
We should also unify it under a single top-level config key, such as rotary_embedding_options or similar. We need to load both the new format and old ones for backward compatibility.
Some models have multiple RoPE configs. For example Gemma3 has a list of layer_types and then a map with layer_type -> rope_parameters. So we will need to update our implementation accordingly.
Activity
jonatanklosko commentedon Jan 5, 2026
There are more significant changes coming in v5:
They added a single object that encapsulates RoPE config, including rope base, rope percentage and rope scaling. They have attributes from all scaling strategies flat in the config, but we can stick to our
:scaling_strategydiscriminated union.We should also unify it under a single top-level config key, such as
rotary_embedding_optionsor similar. We need to load both the new format and old ones for backward compatibility.Some models have multiple RoPE configs. For example Gemma3 has a list of
layer_typesand then a map withlayer_type -> rope_parameters. So we will need to update our implementation accordingly.