CLIPTokenizer (and others based on the same telephoned OpenAI code) incorrect tokenize 1138 out of 34483 words that have an exact match in vocab

### System Info - `transformers` version: 4.36.0 - Platform: Linux-5.15.120+-x86_64-with-glibc2.35 - Python version: 3.10.12 - Huggingface_hub version: 0.19.4 - Safetensors version: 0.4.1 - Accelerate version: not installed - Accelerate config: not found - PyTorch version (GPU?): 2.1.0+cu118 (False) - Tensorflow version (GPU?): 2.14.0 (False) - Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu) - Jax version: 0.4.20 - JaxLib version: 0.4.20 - Using GPU in script?: no - Using distributed or parallel set-up in script?: no ### Who can help? @ArthurZucker and @younesbelkada ### Information - [X] The official example scripts - [X] My own modified scripts ### Tasks - [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [X] My own task or dataset (give details below) ### Reproduction Visit https://colab.research.google.com/drive/18I0mYxTV-UCDjKWTxfuaR3P6o00w18Q9?usp=sharing for a reproduction. ``` from transformers import CLIPProcessor tokenizer = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32").tokenizer # match whole words whole_words = {k: v for k, v in tokenizer.get_vocab().items() if k.endswith("</w>")} to_trim = len("<w/>") missed = 0 for token_str, token_int in whole_words.items(): tokenized = tokenizer.tokenize(token_str[:-to_trim]) if len(tokenized) != 1: missed += 1 print(f"transformers {missed} words out of {len(whole_words)} incorrectly tokenized ({missed/len(whole_words)*100})%") ``` this prints `transformers 1138 words out of 34483 incorrectly tokenized (3.3001768987617086)%` I see that everyone copied OpenAI's buggy tokenization code. Besides this issue there is also https://github.com/openai/CLIP/issues/343. The code in that repository was obviously not used for training, so this could explain a lot of misses / poor performance in CLIP based models. ### Expected behavior tokenization of a word that exactly matches an entry in the vocab file should return exactly 1 token