MMVP Benchmark Datacard

Basic Information

Title: MMVP Benchmark

Description: The MMVP (Multimodal Visual Patterns) Benchmark focuses on identifying “CLIP-blind pairs” – images that are perceived as similar by CLIP despite having clear visual differences. MMVP benchmarks the performance of state-of-the-art systems, including GPT-4V, across nine basic visual patterns. It highlights the challenges these systems face in answering straightforward questions, often leading to incorrect responses and hallucinated explanations.

Dataset Details

• Content Types: Images (CLIP-blind pairs)
• Volume: 300 images
• Source of Data: Derived from ImageNet-1k and LAION-Aesthetics
• Data Collection Method: Identification of CLIP-blind pairs through comparative analysis