[DamonFool]

Thanks @fairydreaming we can run T5 models with llama.cpp.
However, current implementation seems to only support equal encoder-decoder layers.
And we've seen T5 models with different encoder-decoder layers which fail to run with llama.cpp.

Since HF transformers can run T5 models with different encoder-decoder layers very well, it would be better to also support them in llama.cpp.

Thanks.

[DamonFool]

This PR is an extension of #8141 which adds support of T5 and FLAN-T5 model families with different encoder-decoder layers.

[CISC]

Sooo, I started reviewing this, but when checking HF for encoder-only T5 models it turns out that all of them (the ones I could find at least) has num_decoder_layers set to the same value as num_layers and even is_encoder_decoder set to true, however they do not have any decoder layers!

I think the proper solution here is to just make the decoder layers not required and then simply check that they are set before doing the decoder part.

[DamonFool]

Sooo, I started reviewing this, but when checking HF for encoder-only T5 models it turns out that all of them (the ones I could find at least) has num_decoder_layers set to the same value as num_layers and even is_encoder_decoder set to true, however they do not have any decoder layers!

I think the proper solution here is to just make the decoder layers not required and then simply check that they are set before doing the decoder part.

Thanks @CISC for your review.

We have LLM_ARCH_T5ENCODER for encoder-only T5 models and LLM_ARCH_T5 for encoder-decoder T5 models.
This pr aims at extending the support of LLM_ARCH_T5 for encoder-decoder T5 models.
It shouldn't change the runtime behavior of encoder-only T5 models after this patch (except that num_decoder_layers may be added in the gguf file).

The issue you mentioned above is about LLM_ARCH_T5ENCODER for encoder-only T5 models.
I would suggest fixing it in a separate PR if it really maters.

What do you think?
Thanks.

[CISC]

The issue you mentioned above is about LLM_ARCH_T5ENCODER for encoder-only T5 models. I would suggest fixing it in a separate PR if it really maters.

I see, I'll continue the review then, do you have links to some example models?

[DamonFool]

I see, I'll continue the review then, do you have links to some example models?

Sorry, I didn't find a T5 model with different encoder-decoder layers in the open source world.

[CISC]

🤣 Poor CI (it's faster/easier to add changes to batch, then commit).

[DamonFool]

Thanks @CISC for your excellent review!
All your improvements have been applied.

And it seem much better now.

I will do some check and feed back here.

[CISC]

I will do some check and feed back here.

Thank you.

[DamonFool]

I will do some check and feed back here.

I tested t5-small, flan-t5-small and a local t5 model (4-encoder-layer & 2-decoder-layer).
All passed.

[DamonFool]

Thanks @CISC and @ggerganov for your help.