BLIP-2 Image Captioning and Visual QA

Generate detailed image descriptions and answer visual questions using Salesforce BLIP-2. Enables text-based image search through auto-generated captions. Provides an alternative approach to CLIP embedding-based visual search by converting images to rich text descriptions that can be searched with standard text retrieval methods.

Overview

This project implements a complete image captioning and visual search pipeline using BLIP-2 (Bootstrapping Language-Image Pre-training). The system generates detailed captions for image collections, indexes them in Elasticsearch for text-based retrieval, and supports visual question answering for interactive image exploration.

Alternative Approach Note

This is an alternative to the CLIP-based visual search approach. BLIP-2 generates rich text captions from images, enabling search through standard text retrieval (BM25, semantic search over captions). CLIP was chosen for the primary project because direct embedding-based similarity is faster at query time and does not require caption generation during indexing. However, BLIP-2 captions provide human-readable descriptions, support more nuanced text queries, and enable attribute-based filtering that embedding similarity cannot easily express.

Key Features

• Image Captioning: Generate concise or detailed descriptions of any image
• Visual QA: Answer natural language questions about image content
• Attribute Extraction: Structured extraction of objects, colors, materials, and styles
• Caption-Based Search: BM25 and semantic search over generated captions
• Hybrid Search: Combine text search with attribute filtering and VQA reranking
• Batch Processing: Efficient batch captioning for large image collections

Architecture

Image Upload --> BLIP-2 Captioner --> Captions + Attributes
                                           |
                                     Elasticsearch Index
                                           |
Text Query --> BM25/Semantic Search --> Ranked Results
                                           |
                                     VQA Reranking --> Final Results

Project Structure

blip2-image-captioning/
├── src/
│   ├── captioner.py          # BLIP-2 model wrapper for captioning and VQA
│   ├── caption_search.py     # Search engine over generated captions
│   └── api.py                # FastAPI serving endpoints
├── configs/
│   └── config.yaml           # Model and search configuration
├── requirements.txt
└── README.md

Quick Start

pip install -r requirements.txt

# Start API server
uvicorn src.api:app --host 0.0.0.0 --port 8000

# Generate caption for an image
curl -X POST http://localhost:8000/caption \
  -F "[email protected]"

# Visual question answering
curl -X POST http://localhost:8000/vqa \
  -F "[email protected]" \
  -F "question=What color is the car?"

# Index a directory
curl -X POST http://localhost:8000/index \
  -H "Content-Type: application/json" \
  -d '{"directory": "/path/to/images"}'

# Search
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "sunset over mountains", "top_k": 10}'

API Endpoints

Method	Endpoint	Description
POST	/caption	Generate caption for uploaded image
POST	/vqa	Visual question answering
POST	/index	Index directory of images
POST	/search	Search images by text query
GET	/health	Service health check

Model Options

Model Variant	Parameters	Speed	Quality
Salesforce/blip2-opt-2.7b	3.7B	Fast	Good
Salesforce/blip2-opt-6.7b	7.8B	Medium	Better
Salesforce/blip2-flan-t5-xl	4.1B	Medium	Better
Salesforce/blip2-flan-t5-xxl	12.1B	Slow	Best

Comparison with CLIP Search

Aspect	BLIP-2 (Caption)	CLIP (Embedding)
Index Time	Slow (generate captions)	Fast (single forward pass)
Query Time	Fast (text search)	Fast (cosine sim)
Query Expressiveness	High (natural language)	Medium (short phrases)
Interpretability	High (readable captions)	Low (vector space)
Storage	Text + attributes	Dense vectors

License

MIT License