Repository navigation
[Q] Non-causal / bidirectional attention in decoder models for sentence embeddings #478
Description
While working on sentence transformers support in Bumblebee, I hit an issue running popular decoder-based text embedding models (such as Alibaba-NLP/gte-Qwen2-7B-instruct or fine-tuned Llama/Mistral embedding backbones). These models require non-causal (bidirectional) attention across input tokens.
In #468, :use_bidirectional_attention (default false) was introduced for Gemma3Text, following Gemma3TextConfig.use_bidirectional_attention in huggingface/transformers.
However, in checkpoints for other decoder architectures (like Qwen, Llama, and Mistral), Hugging Face models typically set "is_causal": false in config.json instead of using use_bidirectional_attention.
Before opening a PR, I'd like to ask how you'd prefer to model this in Bumblebee:
- Expose
:is_causal(default:true):
Use:is_causaldirectly on the spec, or accept both and validate against conflicting values in Gemma - https://github.com/huggingface/transformers/blob/469230357aab0f2b303b0d638c1f8d06edb14184/src/transformers/models/gemma3/modeling_gemma3.py#L317. - Unify on
:use_bidirectional_attention(default:false):
Add:use_bidirectional_attentiontoShared.common_options/1across Llama, Mistral, Qwen3, and Gemma 3, map"is_causal": falsefrom HF configs, and validate against conflicting values in shared options.
Activity
jonatanklosko commentedon Oct 5, 2026
We generally try to use consistent naming across models, so we can have
:use_bidirectional_attentionacross models.Note that the
Alibaba-NLP/gte-Qwen2-7B-instructexample you gave they actually use "auto_map" with custom .py implementation of the model, which we don't support.is_causalis not typically in the config, since models like Llama, Mistral generally are causal, so I would rather add it on case by case basis.Also, I wouldn't necessarily add it to Llama and Mistral, unless there is a specific checkpoint that has this flag in the config and is relevant, since those models are typically causal. In other words, I would do it on case by case basis.