Nougat-small VisionEncoderDecoderModel failed when max_new_tokens > 3584, Index out of range in self

### System Info # System Info - transformers == 4.35.0 - torch == 2.0.1+cu117 - Error shows on both cpu and cuda. # Task Use nougat to parse text from large documents. # Issue NougatTokenizerFast. model_max_length = 3584 . Setting max_new_tokens=3585 (or larger) in model.generate works well on most pages, but fails on pages which output should be '[MISSING_PAGE_POST]'. Once the nougat fails, parsing all the following pages will fail unless kernel restart. # Error IndexError: index out of range in self # What I Tried - Setting `nougat_model.config.decoder.pad_token_id = nougat_image_processor.tokenizer.eos_token_id` doesn't work. - I suspect one special token is generated outside of vocabulary, which prevents from end-of-senescence detection. - Nougat_base can generate an empty string with max_new_tokens=3585 on the same page, without errors. ### Who can help? _No response_ ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [ ] My own task or dataset (give details below) ### Reproduction **Sample Document**  **Code** ``` nougat_image_processor = NougatProcessor.from_pretrained("facebook/nougat-small") nougat_model = VisionEncoderDecoderModel.from_pretrained("facebook/nougat-small") pixel_values = nougat_image_processor(image, return_tensors="pt").pixel_values device = 'cpu'#"cuda" if torch.cuda.is_available() else "cpu" if device == 'cuda': pixel_values = pixel_values.cuda() nougat_model = nougat_model.cuda() outputs = nougat_model.generate( pixel_values.to(device), min_length=1, max_new_tokens=3585, bad_words_ids=[[nougat_image_processor.tokenizer.unk_token_id]], ) generated = nougat_image_processor.batch_decode(outputs[0], skip_special_tokens=True)[0] generated = nougat_image_processor.post_process_…