Multi-node training with Deepspeed hangs when `full_determinism = True`

Hey, as I've described below, I think there are problems training Deepspeed in a multi-node setting when `full_determinism = True` in the `TrainingArguments`. I've replicated this on multiple hardware configurations (i.e. different nodes and GPU types — specifically A6000, V100, RTX 3090 — on the same large cluster system). Please take a look, thank you very much! ### System Info ### `transformers-cli env` - `transformers` version: 4.27.3 - Platform: Linux-3.10.0-1160.76.1.el7.x86_64-x86_64-with-glibc2.10 - Python version: 3.8.1 - Huggingface_hub version: 0.13.3 - PyTorch version (GPU?): 2.0.0+cu117 (True) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: Yes - Using distributed or parallel set-up in script?: Yes ### Additional info - `deepspeed` version: 0.8.3 - gcc: 10.2 - cuda: 11.7.1 - pdsh: 2.34 ### Who can help? @sgugger @stas00 ### Information - [ ] The official example scripts - [X] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [X] My own task or dataset (give details below) ### Reproduction ### Shell Please set the following environment variables appropriately: ```bash export NODELIST="gpu1504 gpu1505" export NUM_NODES=2 export GPUS_PER_NODE=1 export MASTER_ADDR=gpu1504 export MASTER_PORT=9901 ``` Create `train.py` from the snippet below, then run with the following commands: ```bash conda create -n ds-trainer python==3.8.1 conda activate ds-trainer pip install transformers[deepspeed] echo "PATH=$PATH" > .deepspeed_env cat /dev/null >| hostfile for i in $NODELIST; do echo "$i slots=$GPUS_PER_NODE" >> hostfile; done deepspeed --num_gpus $GPUS_PER_NODE --num_nodes $NUM_NODES --master_addr $MASTER_ADDR --master_port $MASTER_PORT --hostfile hostfile train.py ``` ### `train.py` ```python import torch from torch.utils.data import Dataset from transfor…