DETR: IndexError: Caught IndexError in replica 0 on device 0. IndexError: index 8 is out of bounds for dimension 0 with size 8

### System Info - `transformers` version: 4.37.1 - Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.37 - Python version: 3.10.12 - Huggingface_hub version: 0.20.3 - Safetensors version: 0.4.2 - Accelerate version: 0.26.1 - Accelerate config: not found - PyTorch version (GPU?): 2.1.2+cu121 (True) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: Yes - Using distributed or parallel set-up in script?: Not explicitly, but Trainer is picking up 2 GPUs ### Who can help? @amyeroberts Hi. I'm getting the error in the title trying to reproduce [this example](https://huggingface.co/docs/transformers/tasks/object_detection). The error is real. I don't know what caused it, but I've narrowed the cause to DETR receiving `BatchSize x NumGPUs` number of targets, but expecting only `BatchSize` which causes the overflow. If I limit the amount of GPUs to 1 (via `CUDA_VISIBLE_DEVICES=0`, for example), it runs ok. Here's the stack trace: ``` Traceback (most recent call last): File "/home/mgruner/cellphones-in-the-wild/./train.py", line 116, in <module> trainer.train() File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train return inner_training_loop( File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 1869, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 2768, in training_step loss = self.compute_loss(model, inputs) File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 2791, in compute_loss outputs = model(**inputs) File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/m…