VLMs Processors are not fully consistent in the inputs formats they accept

### Summary of accepted inputs for image-text-to-text-models | Model | Uses Image Tokens | Accepts URL | Accepts PIL | Accepts Path | Accepts Non-Nested Batched Images (one per text) | Accepts Nested Images (one per text) | Accepts Nested Images (Multiple images per text, same number for each text) | Accepts Nested Images (Multiple images per text, different number for each text) | |---|---|---|---|---|---|---|---|---| | Pix2Struct | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | BLIP | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | Donut | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | UDOP | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | GIT | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | KOSMOS-2 | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | Idefics | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | FUYU | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | | Llava | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | VipLlava | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | BLIP2 | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | | Qwen2-VL | ✅ | ❌ | ✅ | ❌ | ✅* crash when padding set to True | ❌ | ✅ | ✅ | | Chameleon | ✅ | ❌ | ✅ | ❌ | ❌* works when padding manually set to True | ✅ | ✅ | ✅ | | Pixtral | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | | Paligemma | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | | Llava-Next | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | | LLaVA-OneVision | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | | Mllama | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | | InstructBLIP | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | | Idefics2 | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | | Idefics3 | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ### Reproduction For instance, with these inputs: ```python import requests from PIL import Image image_ny = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" image_chicago = "https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg" url_pokemon = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/pokemon.png" image_pokemon = Image.open(requests.get(url_pokemon, stream=True).raw) text_batched_image_tokens = [ "<image> <image> Are these the same cities? If not what cities are these?", …