VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models
This research introduces VisGraphVar, a benchmark generator that evaluates Large Vision-Language Models' (LVLMs) capabilities across seven graph-related visual tasks. Testing of six LVLMs using 990 generated graph images reveals significant performance variations based on visual attributes and imperfections, highlighting current limitations compared to human analysis. The findings emphasize the need for comprehensive testing beyond reasoning tasks and guide development of more robust visual analysis systems.
Home page: here.
Our research, detailed in the paper "VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models", demonstrates the effectiveness of this approach.
Clone this repository and install the required dependencies:
git clone [email protected]:camilochs/visgraphvar.git
cd visgraphvar
pip3 -r requirements.txt
To review the dataset generated with VisGraphVar in the paper you can find it in Huggingface.
To generate a benchmark of visual graphs you must run the specify module task. The configuration for each task is in config.yaml.
>> cat visgraphvar/detection/config.yaml
>> python3.11 -m visgraphvar.detection.main
Folders with images are generated inside the task folder.
- The images generated by each task (folder
visgraphvar/), that would be thebechmark_pathinevaluator/config.yaml. - The evaluator is in the
evaluator/folder. - Check the configuration in
evaluator/config.yaml. - In the
evaluator/utils/config.yamlfolder you must add the OpenRouter API.
In the evaluator/main.py file you can choose which task to run. The results for each LVLM should be in the evaluations folder inside each evaluator task. For example, the result for task 1 (detection), will be found in evaluator/tasks/detection/evaluations/.
To execute:
>> python3.11 -m evaluator.main
All the results of the experiments presented in our paper can be found in the supplementary material folder.
@ARTICLE{10855899,
author={Sartori, Camilo Chacón and Blum, Christian and Bistaffa, Filippo},
journal={IEEE Access},
title={VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models},
year={2025},
volume={},
number={},
pages={1-1},
keywords={Visualization;Benchmark testing;Image edge detection;Layout;Cognition;Computational modeling;Generators;Image color analysis;Complexity theory;Image segmentation;benchmark;computer vision;graph theory;large vision-language models},
doi={10.1109/ACCESS.2025.3535837}
}