[BUG] Error when SFT Emu3 model on any-to-any dataset

[fangliancheng]

Required prerequisites

• I have read the documentation https://align-anything.readthedocs.io.
• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

What version of align-anything are you using?

0.0.1.dev0

System information

How the library was installed: pip
Python version: 3.11.0
Align-anything version: 0.0.1.dev0

Problem description

I was trying to SFT Emu3 model following instructions here:
https://align-anything.readthedocs.io/en/latest/training/any_to_any.html

I copy the example dataset to a file train.json and created the following .sh script to run sft:

# Initialize variables
MODEL_NAME_OR_PATH="BAAI/Emu3-Chat"
PROCESSOR_NAME_OR_PATH="BAAI/Emu3-VisionTokenizer"
TRAIN_DATASETS="/fs/scratch/rb_bd_dlp_rng-dl01_cr_AID_employees/activities/aid_005/fal1syv/align-anything/projects/emu3/example"
TRAIN_DATA_FILE="train.json"
OUTPUT_DIR="../outputs/any2any"

# Source the setup script
source ./setup.sh

# Execute deepspeed command
deepspeed \
    --master_port ${MASTER_PORT} \
    --module align_anything.trainers.any_to_any.sft \
    --model_name_or_path ${MODEL_NAME_OR_PATH} \
    --processor_name_or_path ${PROCESSOR_NAME_OR_PATH} \
    --train_datasets ${TRAIN_DATASETS} \
    --train_data_file ${TRAIN_DATA_FILE} \
    --train_template Any2Any \
    --train_split train \
    --output_dir ${OUTPUT_DIR}

When run the above script, I got the following error:

Reproducible example code

The Python snippets:

 self.model, _, _ = load_pretrained_models(
            self.cfgs.model_cfgs.model_name_or_path,
            processor_name_or_path=self.cfgs.model_cfgs.processor_name_or_path,
            model_max_length=self.cfgs.model_cfgs.model_max_length,
            padding_side='right',
            trust_remote_code=True,
            processor_kwargs=self.cfgs.train_cfgs.processor_kwargs,
        )

Command lines:

bash emu3/emu3_sft.sh

Traceback

[rank0]: Traceback (most recent call last):
[rank0]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank0]:   File "<frozen runpy>", line 88, in _run_code
[rank0]:   File "/fs/scratch/rb_bd_dlp_rng-dl01_cr_AID_employees/activities/aid_005/fal1syv/align-anything/align_anything/trainers/any_to_any/sft.py", line 111, in <module>
[rank0]:     sys.exit(main())
[rank0]:              ^^^^^^
[rank0]:   File "/fs/scratch/rb_bd_dlp_rng-dl01_cr_AID_employees/activities/aid_005/fal1syv/align-anything/align_anything/trainers/any_to_any/sft.py", line 105, in main
[rank0]:     trainer = SuperviseTrainer(cfgs=cfgs, ds_cfgs=ds_cfgs)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/fs/scratch/rb_bd_dlp_rng-dl01_cr_AID_employees/activities/aid_005/fal1syv/align-anything/align_anything/trainers/text_to_text/sft.py", line 55, in __init__
[rank0]:     self.init_models()
[rank0]:   File "/fs/scratch/rb_bd_dlp_rng-dl01_cr_AID_employees/activities/aid_005/fal1syv/align-anything/align_anything/trainers/any_to_any/sft.py", line 54, in init_models
[rank0]:     self.model, _, _ = load_pretrained_models(
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: TypeError: load_pretrained_models() got an unexpected keyword argument 'processor_name_or_path'

Expected behavior

No response

Additional context

No response

[htlou]

Sorry for the late reply! I'm currently looking into this issue and trying to reproduce it on my end. I'll keep updating this thread with my progress as I investigate. Thanks for raising this bug report!