Pytorch can have its default dtype permanently set to the "wrong" value if there is an exception when loading a model

### System Info I just ran into the most head-scratching issue. My data collator was crashing because a tensor it made was in half precision (fp16). I couldn't figure out why, but then I realized my `torch.get_default_dtype()` was `torch.float16`! Then I realized it's because my model code threw an exception in a previous run of a notebook cell. And if you look at this code PreTrainedModel:_from_config : [def _from_config(cls, config, **kwargs):](https://github.com/huggingface/transformers/blob/995a7ce9a80b80062ccfe0b2d78857fb17351e27/src/transformers/modeling_utils.py#L1256-L1294) You can see that it tries to set the dtype back to the original value, but doesn't do so in a `finally` block: ```python # override default dtype if needed dtype_orig = None if torch_dtype is not None: dtype_orig = cls._set_default_torch_dtype(torch_dtype) # do some stuff here....maybe throw an exception... # restore default dtype if it was modified (assuming we get to this line) if dtype_orig is not None: torch.set_default_dtype(dtype_orig) return model ``` This would of course leave my torch default dtype in whatever it was in when I was trying to load the model. We could sprinkle some `finally` blocks around, or we could write a class like this: ```python class temporily_set_default_torch_dtype: def __init__(self, dtype): self.new_dtype = dtype if dtype is not None: self.original_dtype = torch.get_default_dtype() else: # try to make this a no-op self.original_dtype = None def __enter__(self): if self.new_dtype is not None: torch.set_default_dtype(self.new_dtype) def __exit__(self, exc_type, exc_val, exc_tb): if self.original_dtype is not None: torch.set_default_dtype(self.original_dtype) ``` And use it like so: ```python torch.set_default_dtype(torch.float32) print(f"default dtype is…