an inplace operation preventing TorchDistributor training

### System Info databricks ### Who can help? @ArthurZucker @younesbelkada Hi team, I got an error message by using TorchDistributor. I have checked in the class BertEmbeddings (url as below), line 238, embeddings += position_embeddings is an inplace operation, would you be able to change to embeddings = embeddings + position_embeddings, to allow TOrchDistributor? BertEmbeddings url: https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py TorchDistributor sample code: https://docs.databricks.com/_extras/notebooks/source/deep-learning/torch-distributor-notebook.html Thank you very much! Ling ### Information - [X] The official example scripts - [ ] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [ ] My own task or dataset (give details below) ### Reproduction ``` single_node_single_gpu_dir = create_log_dir() print("Data is located at: ", single_node_single_gpu_dir) def train_one_epoch(model, device, data_loader, optimizer, epoch): torch.autograd.set_detect_anomaly(True) model.train() for batch_idx, (data, labels) in enumerate(data_loader): inputs1, inputs2 = data[0], data[1] inputs1 = {key: val.to(device) for key, val in inputs1.items()} inputs2 = {key: val.to(device) for key, val in inputs2.items()} # labels = labels.float().to(device) labels = labels.to(device) optimizer.zero_grad() # Compute embeddings embeddings1 = model(inputs1)['sentence_embedding'] embeddings2 = model(inputs2)['sentence_embedding'] # Compute loss loss = cosine_similarity_loss(embeddings1, embeddings2, labels) loss.backward() optimizer.step() if batch_idx % log_interval == 0: print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format( epoch, batch_idx * len(data), len(data_loader) * len(data), 100. * batch_idx / len(data_loader), loss.item())) if int(os.environ["RANK"…