How to run Trainer + DeepSpeed + Zero3 + PEFT

### System Info - `transformers` version: 4.34.0.dev0 - Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34 - Python version: 3.11.4 - Huggingface_hub version: 0.16.4 - Safetensors version: 0.3.3 - Accelerate version: 0.24.0.dev0 - Accelerate config: not found - PyTorch version (GPU?): 2.0.1+cu117 (True) ### Who can help? @ArthurZucker and @younesbelkada and @pacman100 and @muellerzr ### Information - [ ] The official example scripts - [X] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [ ] My own task or dataset (give details below) ### Reproduction [This script](https://gist.github.com/BramVanroy/f2abb3940111b73ae8923822ef6096dd) is a modification of the official run_clm script. The only additions are the BNB config and PEFT. Yet, I cannot get it to work with a [deepspeed zero3 config](https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/configs/ds_falcon_180b_z3.json). Requirements to install: ``` accelerate >= 0.12.0 torch >= 1.3 datasets >= 1.8.0 sentencepiece != 0.1.92 protobuf evaluate scikit-learn trl peft bitsandbytes ``` In the past I have had issues with low_cpu_mem_usage but neither a true/false value seem to get this to work: Command 1: ```sh deepspeed --include="localhost:0,1" run_clm.py \ --model_name_or_path facebook/opt-125m\ --dataset_name wikitext\ --dataset_config_name wikitext-2-raw-v1\ --per_device_train_batch_size 2\ --per_device_eval_batch_size 2\ --do_train\ --do_eval\ --output_dir /tmp/test-clm\ --deepspeed deepspeed_configs/ds_config_zero3.json\ --low_cpu_mem_usage true ``` ==> `ValueError: DeepSpeed Zero-3 is not compatible with `low_cpu_mem_usage=True` or with passing a `device_map`.` Command 2: ```sh deepspeed --include="localhost:0,1" run_clm.py \ --model_name_or_path facebook/opt-125m\ --dataset_name wikitext\ --dataset_config_name wikitext-2-raw-v1\ --per_device_train_batch_size 2\ …