Flaky Whisper PT-TF & PT-Flax Equivalence Test

### System Info transformers 4.29.0 dev ### Who can help? @ArthurZucker @sanchit-gandhi @ydshieh ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [ ] My own task or dataset (give details below) ### Reproduction Flaky test, so not reproducible. Example run where the error occurred: * https://app.circleci.com/pipelines/github/huggingface/transformers/64100/workflows/b4463c5d-b3dc-4b00-a7cd-19acd096cb07/jobs/792381 * https://app.circleci.com/pipelines/github/huggingface/transformers/64111/workflows/dc9092c4-0673-46c7-b89f-f805bc20128c/jobs/792557 * https://app.circleci.com/pipelines/github/huggingface/transformers/64230/workflows/e2d42ca4-f367-4a85-9054-a0ea99e49849/jobs/794534 Occasionally, the PT-TF and PT-Flax whisper equivalence test fails. The tolerance was increased in #23257 and #23288 but the reason for recent failures has not yet been found. ### Expected behaviour Equivalence tests reliably pass.