Detr Models cannot be loaded with `device_map="auto"`

### System Info - `transformers` version: 4.28.1 - Platform: macOS-13.1-x86_64-i386-64bit - Python version: 3.9.2 - Huggingface_hub version: 0.12.1 - Safetensors version: not installed - PyTorch version (GPU?): 1.13.1 (False) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: no - Using distributed or parallel set-up in script?: no ### Who can help? _No response_ ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [ ] My own task or dataset (give details below) ### Reproduction ```python from transformers import pipeline p = pipeline( "object-detection", model="facebook/detr-resnet-50", image_processor="facebook/detr-resnet-50", device_map="auto" ) ``` ### Expected behavior This does not work because the `transformers.models.detr.modeling_detr.DetrConvEncoder` model init involves copy weights from `nn.BatchNorm2d` to `DetrFrozenBatchNorm2d` which is not allowed when on a meta device. ``` File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/pipelines/__init__.py", line 779, in pipeline framework, model = infer_framework_load_model( File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/pipelines/base.py", line 262, in infer_framework_load_model model = model_class.from_pretrained(model, **kwargs) File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 471, in from_pretrained return model_class.from_pretrained( File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2629, in from_pretrained model = cls(config, *model_args, **model_kwargs) File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/models/detr/modeling_detr.py", line 1373, in…