GPT-neo generate is ignoring passed position ids

### System Info ```shell python version: 3.9 transformers version: 4.18 ``` ### Who can help? @patil-suraj @patrickvonplaten ### Information - [ ] The official example scripts - [X] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [X] My own task or dataset (give details below) ### Reproduction ``` from transformers import AutoTokenizer, AutoModelForCausalLM import torch text = "hi there: " modelname = "EleutherAI/gpt-neo-125M" model = AutoModelForCausalLM.from_pretrained(modelname) tokenizer = AutoTokenizer.from_pretrained(modelname) inputs = tokenizer(text, return_tensors ="pt") inputs["position_ids"] = inputs["attention_mask"].cumsum(-1) print(inputs) a = model.generate(**inputs) inputs["position_ids"] = inputs["position_ids"] + 10 print(inputs) b = model.generate(**inputs) # a and b should be different because the position ids are different print(a) print(b) ``` the result ``` a = tensor([[5303, 612, 25, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220]]) b = tensor([[5303, 612, 25, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220, 220]]) ``` ### Expected behavior ```shell The outputs a and b should (almost always) be different because different position ids should be passed to the model's forward function, resulting in different activations. The issue seems to be here: https://github.com/huggingface/transformers/blob/ee393c009a243bbb86fa11d2efa771a1704d85cf/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L699 Specifically, the else statement is such that if both an attention mask and position ids are passed, the position ids are erased. In such a scenario, the default position ids in the model's forward function (https://github.com/huggingface/transformers/blob/ee393c009a243bbb86fa11d2efa771a1704d85cf/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L537) are…