The results of run_mae.py pre-training were abnormal

### System Info - `transformers` version: 4.39.0.dev0 - Platform: Linux-5.10.0-60.18.0.50.oe2203.x86_64-x86_64-with-glibc2.34 - Python version: 3.9.18 - Huggingface_hub version: 0.21.3 - Safetensors version: 0.4.2 - Accelerate version: 0.27.2 - Accelerate config: not found - PyTorch version (GPU?): 2.0.1+cu117 (True) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: <fill in> - Using distributed or parallel set-up in script?: <fill in> ### Who can help? @NielsRogge. ### Information - [X] The official example scripts - [ ] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [X] My own task or dataset (give details below) ### Reproduction 1. ViT_MAE_visualization_demo: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/ViTMAE/ViT_MAE_visualization_demo.ipynb 2. run_mae.py: https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining This is the result of loading the pre-trained model `facebook/vit-mae-huge` directly from huggingface.  This is the result of my continued pre_training using `facebook/vit-mae-huge`  I don't know why it's getting worse，The loss is decreasing from loss=1.7 at the beginning to loss=0.13. ### Expected behavior a