Change Device Allocation and Improve Shape Handling in Trainer evaluation loop

### Feature request ## Context Currently when setting the [compute_metrics](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L360) parameter as non-null value it sets the [prediction_loss_only](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3402) param as `False`. As `prediction_loss_only` is `False`, [self.prediction_step](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3586) will return non-null value for the `logits` variable. This variable will be used here to be [padded](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3608), then [concatenated](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3612) to previously stored logits. ## What bothers me ### Device Allocation If the logits are tensors, a new tensor will be created using the same device as logits' one ([cf](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_pt_utils.py#L91)). Meaning that if `logits` is on `cuda:0`, the resulting tensor will also be on `cuda:0`. This unnecessarily consumes VRAM as after the tensor will be moved back to CPU ([cf](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3626)) without further operations. IMO the resulting tensor should be on CPU from the beginning to avoid consuming VRAM unnecessarily. But let me know if I miss something :) ### Handling of Shapes It seems that the concatenation is only done across `logits` ([cf](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3608)), not taking into account the shape of previously registered logits in `preds_host`. Meaning that at each iteration, the `logits` should have the same max length otherwise it will raise the following error : `RuntimeError: Sizes of tensors must match except in dimension 0. Expected size [SHAPE] but got size [ANOTHER SHAPE] for tensor num…