visual-reasoning-rationalization

Code associated with the "Natural Language Rationales with Full-Stack Visual Reasoning" Findings of EMNLP 2020 paper

Citation

@inproceedings{marasovic-et-al-2020-rationalization,
    title = "{{Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs}}",
    author = "Marasovi\'{c}, Ana and Bhagavatula, Chandra and Park, Jae Sung and Le Bras, Ronan and Smith, Noah A. and Choi, Yejin",
    booktitle = "Findings of EMNLP",
    year = "2020",
    url = "https://arxiv.org/abs/2010.07526"
}

Installation

conda env create -f environment.yml
conda activate rationalization

Trained models

https://visual-reasoning-rationalization.s3-us-west-2.amazonaws.com/models_vqa.zip

https://visual-reasoning-rationalization.s3-us-west-2.amazonaws.com/models_vcr.zip

https://visual-reasoning-rationalization.s3-us-west-2.amazonaws.com/models_e_snli_ve.zip

Extract in an empty directory named models using tar -xzvf.

Downloading data

https://visual-reasoning-rationalization.s3-us-west-2.amazonaws.com/data.zip

Download features

https://visual-reasoning-rationalization.s3-us-west-2.amazonaws.com/features.zip

Example commands

Training

export FEATURES=textual_objects
python scripts/run_finetuning.py -e ${FEATURES} 

The value of FEATURES can be one of the following: text_only, textual_objects, embedding_objects, textual_situation, embedding_situation, textual_viscomet, textual_viscomet.

Decoding

python scripts/run_generation.py -e ${FEATURES}_eval --model_name_or_path /models/vcr_gen/q_a_to_r/

Templates for human evaluation

HTML files can be found at human_eval_templates