fix(template): align multimodal fields and handle text-only inputs

[taking-lying-flat]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Multimodal token fields can become misaligned during encoding and truncation. For example, truncating a Gemma sample leaves token_type_ids at its original length, while applying a one-dimensional mask directly to ERNIE-VL's [1, sequence_length] token-type tensor raises an indexing error.

• Truncate token_type_ids with the same retained-position mask as the other per-token fields. Slice tensors along their sequence dimension while preserving their dtype, device, and leading batch dimension so the collator can handle them correctly.
• Correct PaliGemma's prompt/answer boundary: the last prompt token remains type 0, the first answer token becomes type 1, and fully masked labels do not cause an out-of-range lookup.
• Extend CogVLM/CogAgent/Cog2Video loss_scale alongside inserted visual tokens, using zero weight for those tokens.
• Initialize SailVL labels and text embeddings for text-only samples, including the non-DeepSpeed path.
• Preserve CogVLM/Cog2Video images when mixing multimodal and text-only rows, independent of row order. CogAgent's cross-attention requires an image for each batch row; reject unsupported mixed batches explicitly instead of dropping image data or raising a KeyError.

Local regression tests exercise the actual PaliGemma, Gemma3, Gemma3n, Molmo2, CogVLM, CogVLM2, CogAgent Chat/VQA, and Cog2Video template encoders and collators with lightweight tokenizer/processor/model-method stubs. Coverage includes image, audio and video expansion, exact token-type values (including Gemma3n audio type 3), labels, non-binary loss scales, left/right truncation, both batch orders, padding, and media tensors. ERNIE-VL's token-type assembly and generic list/tensor truncation also remain covered.

Validation

The results below were obtained at commit 30fdde5, before restoring test files to the upstream versions. The regression tests are retained as a local patch and are not included in the final PR diff. Production fixes are unchanged.

• All 247 registered templates (130 implementation classes) pass left/right truncation contract tests: 494 cases. These exercise each actual truncation implementation with already encoded fields; they do not validate every model-specific processor or pretrained model.
• The focused template regression suite passes 729 tests and 23 subtests, including token-type encoders/collators, MOSS-VL, MiniCPM-O, tool schemas, serialized messages and media paths.
• 8 CPU tests run a small, randomly initialized Transformers PaliGemma model through forward passes and training backward passes after actual template encoding, truncation and collation. They cover both truncation/padding directions and mixed image/text batches; tokenization is stubbed and pretrained weights are not used.
• SailVL's new tests reproduced four failures before the fix and pass all 10 cases afterward, covering image/text encoding and embedding/vision gradients.
• pre-commit run --all-files and git diff --check: passed.

Broader testing was attempted and is not fully green:

• Existing template integration/alignment tests (including MOSS processor alignment): 8 passed, 1 skipped, 248 failed, 11 setup errors. Of the 259 unsuccessful cases, 257 stop at missing model/processor caches in offline mode; the remaining two require vllm and qwen_vl_utils. A local launcher supplies the real imports normally executed by standalone scripts and the Transformers backend argument; models and assertions are not mocked.
• All tests/general: 820 passed, 265 subtests passed, 21 failed, 7 setup errors. Remaining failures include missing model/dataset resources, a dataset clone timeout, the existing LISA callback's TrainingArguments.max_epochs compatibility error, and a ModelScope-only ID tested with the HF download channel. Counts overlap the focused suite and must not be added together.

Validation used Python 3.11, torch 2.14 and transformers 5.16.1. Full pretrained-model forward/training across all templates remains unverified and requires the appropriate models, optional dependencies and accelerator environment.

[tastelikefeet]

sometimes token_type_ids is a 2-d tensor, which will raise error here.
Check: ERNIE-VL please