Use models as Seq2Seq model

### System Info - `transformers` version: 4.40.0 - Platform: Linux-6.1.58+-x86_64-with-glibc2.35 - Python version: 3.10.12 - Huggingface_hub version: 0.22.2 - Safetensors version: 0.4.3 - Accelerate version: 0.29.3 - Accelerate config: not found - PyTorch version (GPU?): 2.2.1+cu121 (False) - Tensorflow version (GPU?): 2.15.0 (False) - Flax version (CPU?/GPU?/TPU?): 0.8.2 (cpu) - Jax version: 0.4.26 - JaxLib version: 0.4.26 - Using GPU in script?: No - Using distributed or parallel set-up in script?: No ### Who can help? @ArthurZucker @muellerzr @stevhliu ### Information - [ ] The official example scripts - [X] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [X] My own task or dataset (give details below) ### Reproduction There is this snippet in many model documentations: To be used in a Seq2Seq model, the model needs to initialized with both is_decoder=True and bidirectional=False argument as well as add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass. I try it like this for the MEGA model: ``` from transformers import MegaConfig # config for a small seq2seq model like in the MEGA paper config = MegaConfig( vocab_size=vocabulary_size, max_position_embeddings=context_length, is_decoder=True, bidirectional=False, add_cross_attention=True ) from transformers import AutoTokenizer, MegaModel,MegaForCausalLM model = MegaModel(config=config) # only the causalLM as decoder-only seems to run #model = MegaForCausalLM(config=config) ``` The following error occurs, when training it with `Seq2SeqTrainer`,`Seq2SeqTrainingArguments` and `DataCollatorForSeq2Seq`: ``` [transformers/models/mega/modeling_mega.py](https://localhost:8080/#) in forward(self, hidden_states, attention_mask, causal_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions, use_cache) 1271 if self.cross_attn is not No…