Concept Selection Models

This is the code repository of "Understanding Multimodal Deep Neural Networks: A Concept Selection View". The paper is accepted by CogSci 2024.

Setup

• Use helper/prepare_concept_bank.ipynb to establish the concept library.
• Use helper/image_representation.py to get the image representations.
• Use helper/clip_label.py to annote the concepts by CLIP.

Run

• Use rough_selection.ipynb to conduct the greedy rough selection.

• To run experiments, you can refer the command:

bash scripts/example.sh

--algorithm can be chosen from from lp, cbm and mask.

• Use fine_selection.ipynb to conduct the mask fine selection.

Citation

If you find this code useful, please consider citing our paper:

@inproceedings{shang2023understanding,
  title={Understanding Multimodal Deep Neural Networks: A Concept Selection View},
  author={Shang, Chenming and Zhang, Hengyuan and Wen, Hao and Yang, Yujiu},
  booktitle={Proceedings of the Annual Meeting of the Cognitive Science Society},
  volume={46},
  year={2023}
}