Failed to use --fp16 and --use_dml_attn simultaneously in Whisper pytorch-directml

[XciciciX]

I tried to run Whisper with torch-directml. It can run with either --fp16 flag or --use_dml_attn flag. However, I want to run with both flags. It failed with the following information:

[joshjkim]

@XciciciX Thank you for reporting this issue. Could you please try with the latest Whisper sample? Multilingual models preserve past_key_value tensors when detecting language which causes the parameter to be incorrect when running transcription inference on audio. The sample has been updated to remove DML attention in AudioEncoder as there's no significant perf gains in the encoder