Detr models crashes when changing the num_queries parameter in the config

### System Info - `transformers` version: 4.36.2 - Platform: Linux-5.15.133+-x86_64-with-glibc2.35 - Python version: 3.10.10 - Huggingface_hub version: 0.20.2 - Safetensors version: 0.4.1 - Accelerate version: 0.26.1 - Accelerate config: not found - PyTorch version (GPU?): 2.1.2+cu121 (True) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: Yes, Tesla T4 - Using distributed or parallel set-up in script?: No ### Who can help? @amyeroberts ### Information - [ ] The official example scripts - [X] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [X] My own task or dataset (give details below) ### Reproduction 1. Load the model with a custom `num_queries` hyperparameter. ``` id2label = {0: 'Test'} label2id = {'Test': 0} model_name = "facebook/detr-resnet-50" image_processor = AutoImageProcessor.from_pretrained(model_name) detr = DetrForObjectDetection.from_pretrained( model_name, id2label=id2label, label2id=label2id, ignore_mismatched_sizes=True, num_queries=5 ) ``` 2. Train (or just run the forward pass with an input containing `labels`) I got the following error ``` ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ in <module>:1 │ │ │ │ ❱ 1 trainer.train() │ │ 2 │ │ │ │ /home/jovyan/obj-detection/.v…