[Efficiency] Decoding can be made faster by not converting special tokens to ids for each token.

### System Info - `transformers` version: 4.29.0.dev0 - Platform: macOS-14.0-arm64-arm-64bit - Python version: 3.11.4 - Huggingface_hub version: 0.13.3 - Safetensors version: not installed - PyTorch version (GPU?): 2.0.0 (False) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: <fill in> - Using distributed or parallel set-up in script?: <fill in> ### Who can help? @ArthurZucker ### Information - [X] The official example scripts - [X] My own modified scripts ### Tasks - [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [X] My own task or dataset (give details below) ### Reproduction The following function is being called for each token while using decoding function. ```python from transformers import T5Tokenizer tokenizer = T5Tokenizer.from_pretrained(TOKENIZER_PATH) beams = tokenizer.batch_decode( outputs, skip_special_tokens=True ) ``` ```python @property def all_special_ids(self) -> List[int]: """ `List[int]`: List the ids of the special tokens(`'<unk>'`, `'<cls>'`, etc.) mapped to class attributes. """ all_toks = self.all_special_tokens all_ids = self.convert_tokens_to_ids(all_toks) return all_ids ``` ### Expected behavior all_special_ids should not be called for each token while decoding at the time of inferencing.