This repo implements the CIF Aware Missing Modality Imagination Network (CIF-MMIN) for the following paper: "Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities"
python 3.8.0
pytorch >= 1.8.0
First you should change the data folder path in data/config and preprocess your data follwing the code in preprocess/.
The preprocess of feature was done handcrafted in several steps, we will make it a automatical running script in the next update. You can download the preprocessed feature to run the code.
-
For Training CIF-MMIN on IEMOCAP:
First training a model self-supervise model with all audio, visual and lexical modality as the pretrained encoder.
bash scripts/CAP_utt_self_supervise.sh AVL [num_of_expr] [GPU_index]
Then
bash scripts/our/CAP_CIF_MMIN.sh [num_of_expr] [GPU_index]
-
For Training CIF-MMIN on MSP-improv:
bash scripts/MSP_utt_self_supervise.sh AVL [num_of_expr] [GPU_index]
bash scripts/our/MSP_CIF_MMIN.sh [num_of_expr] [GPU_index] -
For Training CIF-MMIN on MOSI:
bash scripts/MOSI_utt_self_supervise.sh AVL [num_of_expr] [GPU_index]
bash scripts/our/MOSI_CIF_MMIN.sh [num_of_expr] [GPU_index]
Note that you can run the code with default hyper-parameters defined in shell scripts, for changing these arguments, please refer to options/get_opt.py and the modify_commandline_options method of each model you choose.
Baidu Yun Link IEMOCAP A V L modality Features 链接：https://pan.baidu.com/s/1i4_ZKFwGUE4cVrxi20dKPg?pwd=id33 提取码：id33
MSP-IMPROV A V L modality Features 链接：https://pan.baidu.com/s/1UzyiC2idpXM8pz0RU5YOCQ?pwd=pcel 提取码：pcel
CMU-MOSI A V L modality Features 链接：https://pan.baidu.com/s/1N8fl8gQC5HTWXuESPUz1pg?pwd=vm51 提取码：vm51
MIT license.
Copyright (c) 2023 S2Lab, School of Inner Mongolia University.