CLIP-Powered Visual Search Engine

A zero-shot image search and visual similarity system built on OpenAI CLIP. Enables text-to-image search, image-to-image similarity retrieval, automated attribute tagging, near-duplicate detection, and embedding-based clustering across large image libraries using FAISS for efficient nearest-neighbor lookup.

Features

• Text-to-Image Search -- Find images matching natural language descriptions without any training data or fine-tuning.
• Image-to-Image Search -- Retrieve visually similar images from an indexed collection.
• Zero-Shot Auto-Tagging -- Automatically assign attribute labels (color, style, material, etc.) to images using configurable category vocabularies.
• Near-Duplicate Detection -- Identify and group near-duplicate images based on embedding similarity thresholds.
• Embedding Clustering -- K-means clustering on CLIP embeddings to discover visual themes in a collection.
• FAISS-Backed Index -- Scalable vector search supporting FlatIP, IVFFlat, and IVFPQ index types with persistence.
• REST API -- FastAPI service for indexing, searching, tagging, and deduplication.

Project Structure

clip-visual-search/
├── configs/
│   └── config.yaml
├── src/
│   ├── clip_encoder.py
│   ├── index_manager.py
│   ├── search_engine.py
│   └── api.py
├── requirements.txt
├── .gitignore
└── README.md

Quick Start

Installation

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Index a Directory

from src.search_engine import VisualSearchEngine

engine = VisualSearchEngine(config_path="configs/config.yaml")
engine.index_directory("/path/to/images")

Text Search

results = engine.text_search("a red sports car on a highway", top_k=10)
for path, score in results:
    print(f"{score:.3f}  {path}")

Image Similarity Search

results = engine.image_search("query_image.jpg", top_k=10)

Auto-Tag Images

tags = engine.auto_tag("/path/to/images", categories={
    "color": ["red", "blue", "green", "black", "white"],
    "style": ["modern", "vintage", "minimalist"],
})

Find Duplicates

groups = engine.deduplicate(threshold=0.95)
for group in groups:
    print(f"Duplicate group: {group}")

Start the API

uvicorn src.api:app --host 0.0.0.0 --port 8000

API Endpoints

Method	Endpoint	Description
POST	/index	Index a directory of images
POST	/search/text	Text-to-image search
POST	/search/image	Image-to-image similarity search
POST	/tag	Auto-tag images with attribute labels
POST	/deduplicate	Find near-duplicate image groups
GET	/stats	Index statistics
GET	/health	Health check

Configuration

Edit configs/config.yaml to configure the CLIP model, FAISS index type, search parameters, tagging categories, and deduplication thresholds.

Supported FAISS Index Types

• FlatIP -- Exact inner product search. Best accuracy, linear scan time.
• IVFFlat -- Inverted file index with flat quantizer. Good accuracy/speed tradeoff.
• IVFPQ -- Inverted file with product quantization. Best for very large collections.

Requirements

• Python 3.9+
• GPU recommended for encoding but not required
• See requirements.txt for full dependency list

License

MIT License