Add Prismatic VLMs to Transformers

### Model description Hi! I'm the author of ["Prismatic VLMs"](https://github.com/TRI-ML/prismatic-vlms), our upcoming ICML paper that introduces and ablates design choices of visually-conditioned language models that are similar to LLaVa or InstructBLIP, introducing ~50 new VLMs at the 3B/7B/13B scale that are trained with: - Different Visual Representations (`CLIP`, `SigLIP`, `DINOv2`, fusions thereof like `SigLIP + DINOv2`) - Different LLM Backbones (`LLaMa2`, `Vicuña v1.5`, `Mistral v0.1`, `Mistral v0.1 Instruct`, `Phi-2`, etc.) - Different Data (e.g., the LLaVa v1.5 Data, LVIS-Instruct-4V, and more upcoming!) Our best models outperform LLaVa v1.5 given the same data/same scale on a wide spectrum of different evaluation tasks; furthermore, we're seeing a lot of folks adopt our code for their research into new data mixtures, scaling to different LLM/Vision backbones, new projection mechanisms, and more. I think it'd be awesome to support these in `transformers` -- especially to tap into existing tooling for loading quantized versions of models, using `PEFT` and other tools in the HF ecosystem for adaptation/fine-tuning, and general usability of our trained models. While we have 50+ checkpoints (all open-sourced, and loadable in our library), all currents models share a pretty common interface of using some pretrained visual extractor from `timm`, a `XXXForCausalLM` from `transformers`, and a lightweight `nn.Module` to project visual features into the LLM embedding space. As such, I'm hoping to contribute a general `modeling_prismatic.py` class that implements `PrismaticPretrainedModel` and `PrismaticForConditionalGeneration` that properly instantiates the appropriate VLM instance using the dependencies already in `transformers`. --- I'm happy to get started with this, following the [instructions here](https://huggingface.co/docs/transformers/add_new_model#how-to-add-a-model-to--transformers), but would love help/advice on clean ways to support all the various im…