[Feature request] FastV: Plug-and-play inference acceleration for large vision language models

### Feature request FastV is a plug-and-play inference acceleration method for large vision language models relying on visual tokens. It could reach 45% theoretical FLOPs reduction without harming the performance through pruning redundant visual tokens in deep layers. It is originally implemented in https://github.com/pkunlp-icler/FastV , with [paper](https://arxiv.org/abs/2403.06764) and [demo](https://www.fastv.work/)  *aokvqa results* | Model | Score | latency / first output token (A100 80G) | GPU Memory | | ----------------------------- | ----- | --------------------------------------- | ---------- | | 7B Vanilla Decoding | 76.8 | 0.138s | 18G | | 13B Vanilla Decoding | 81.9 | 0.203s | 33G | | \- 13B FastV (K=2 R=25%) | 81.8 | 0.181s | 29G | | \- 13B FastV (K=2 R=50%) | 81.3 | 0.155s | 28G | | \- 13B FastV (K=2 R=75%) | 80.9 | **0.124s** | 27G | | 13B Vanilla Decoding 4Bit | 81.5 | 0.308s | 12G | | \- 13B FastV 4Bit (K=2 R=25%) | 81.7 | 0.277s | 11G | | \- 13B FastV 4Bit (K=2 R=50%) | 81.1 | 0.275s | 10G | | \- 13B FastV 4Bit (K=2 R=75%) | 80.3 | 0.245s | **9G** | ### Motivation I want to merge the FastV feature to speed up the inference of HF's multimodal models (Llava is the first one). Since Llava is based on Llama, I need to add features to both the Llama and Llava model. I would add a fastv_config parameters to the modeling_llava.py from llava and add a new forward function named fastv_forward from llama. I want to kn…