Entropy-Gradient Grounding: Training-Free Evidence Retrieval
in Vision-Language Models

Marcel Gropl^1* · Jaewoo Jung^3* · Seungryong Kim³ · Marc Pollefeys¹ · Sunghwan Hong^1,2

¹ETH Zurich, ²ETH AI Center, ³KAIST

^*Equal contribution

Paper | Project Page

We propose a training-free, model-intrinsic grounding method for vision-language models that uses uncertainty as supervision. Our approach backpropagates entropy of the next-token distribution to visual token embeddings, yielding relevance maps without auxiliary detectors or attention-map heuristics — enabling robust grounding on detail-critical and high-resolution settings across seven benchmarks and four VLM architectures.

🚀 What to Expect

• Inference code for LLaVA.
  
• Inference code for Qwen 2.5.
  
• Inference code for InternVL 3.5.
  
• Single image demo.
  

Installation

Our code is developed based on Python 3.10, PyTorch 2.1.2, and CUDA 12.1.

We recommend using conda for installation:

git clone https://github.com/merci00001/entropy-gradient-grounding.git
cd entropy-gradient-grounding

conda create -n egrounding python=3.10
conda activate egrounding

pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

Method Overview

Given a query, Entropy-Gradient Grounding frames visual grounding as test-time evidence retrieval:

1. Entropy-gradient relevance maps — compute the entropy of the model's next-token distribution and backpropagate it to visual token embeddings to produce a grounding signal without any auxiliary detectors or attention-map heuristics.
2. Multi-evidence ranking — extract and rank multiple coherent regions to support compositional and multi-clue queries.
3. Iterative zoom-and-reground — refine localization iteratively with a spatial-entropy stopping rule to prevent over-refinement.

Inference

LLaVA 1.6 (Mistral)

python inference_llava.py \
  --model-path liuhaotian/llava-v1.6-mistral-7b \
  --image-folder /path/to/images \
  --question-file /path/to/questions.jsonl \
  --answers-file /path/to/answers.jsonl \
  --conv-mode mistral_instruct \
  --is15 False

LLaVA 1.5

python inference_llava.py \
  --model-path liuhaotian/llava-v1.5-7b \
  --image-folder /path/to/images \
  --question-file /path/to/questions.jsonl \
  --answers-file /path/to/answers.jsonl \
  --conv-mode vicuna_v1 \
  --is15 True

The --question-file follows the standard LLaVA JSONL format, where each line contains an image filename and a query.

Key arguments:

Argument	Description
--model-path	HuggingFace model ID or local path
--question-file	Input questions in LLaVA JSONL format
--answers-file	Output file for model predictions
--conv-mode	mistral_instruct for Mistral-based models, vicuna_v1 otherwise
--is15	Set True for LLaVA 1.5, False for LLaVA 1.6
--to_run	Number of disjoint regions to keep track of

Qwen 2.5 / InternVL 3.5

Coming soon.

Citation

@article{gropl2025egg,
  title={Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models},
  author={Gropl, Marcel and Jung, Jaewoo and Kim, Seungryong and Pollefeys, Marc and Hong, Sunghwan},
  journal={arXiv preprint},
  year={2025}
}

Acknowledgement

We thank the authors of LLaVA and LocalizationHeads for their excellent work and code, which served as the foundation for this project.