|
| 1 | +# Current Tasks |
| 2 | + |
| 3 | +> () indicates the task name in the lmms_eval. The task name is also used to specify the dataset in the configuration file. |
| 4 | +> The following is manually updated documentation. You could use `lmms_eval task --list` to list all supported tasks and their task names. |
| 5 | +
|
| 6 | +- AI2D (ai2d) |
| 7 | +- ChartQA (chartqa) |
| 8 | +- CMMMU (cmmmu) |
| 9 | + - CMMMU Validation (cmmmu_val) |
| 10 | + - CMMMU Test (cmmmu_test) |
| 11 | +- COCO Caption (coco_cap) |
| 12 | + - COCO 2014 Caption (coco2014_cap) |
| 13 | + - COCO 2014 Caption Validation (coco2014_cap_val) |
| 14 | + - COCO 2014 Caption Test (coco2014_cap_test) |
| 15 | + - COCO 2017 Caption (coco2017_cap) |
| 16 | + - COCO 2017 Caption MiniVal (coco2017_cap_val) |
| 17 | + - COCO 2017 Caption MiniTest (coco2017_cap_test) |
| 18 | +- [ConBench](https://github.com/foundation-multimodal-models/ConBench) (conbench) |
| 19 | +- DOCVQA (docvqa) |
| 20 | + - DOCVQA Validation (docvqa_val) |
| 21 | + - DOCVQA Test (docvqa_test) |
| 22 | +- Ferret (ferret) |
| 23 | +- Flickr30K (flickr30k) |
| 24 | + - Ferret Test (ferret_test) |
| 25 | +- GQA (gqa) |
| 26 | +- HallusionBenchmark (hallusion_bench_image) |
| 27 | +- Infographic VQA (info_vqa) |
| 28 | + - Infographic VQA Validation (info_vqa_val) |
| 29 | + - Infographic VQA Test (info_vqa_test) |
| 30 | +- LLaVA-Bench (llava_in_the_wild) |
| 31 | +- LLaVA-Bench-COCO (llava_bench_coco) |
| 32 | +- MathVerse (mathverse) |
| 33 | + - MathVerse Text Dominant (mathverse_testmini_text_dominant) |
| 34 | + - MathVerse Text Only (mathverse_testmini_text_only) |
| 35 | + - MathVerse Text Lite (mathverse_testmini_text_lite) |
| 36 | + - MathVerse Vision Dominant (mathverse_testmini_vision_dominant) |
| 37 | + - MathVerse Vision Intensive (mathverse_testmini_vision_intensive) |
| 38 | + - MathVerse Vision Only (mathverse_testmini_vision_only) |
| 39 | +- MathVista (mathvista) |
| 40 | + - MathVista Validation (mathvista_testmini) |
| 41 | + - MathVista Test (mathvista_test) |
| 42 | +- MMBench (mmbench) |
| 43 | + - MMBench English (mmbench_en) |
| 44 | + - MMBench English Dev (mmbench_en_dev) |
| 45 | + - MMBench English Test (mmbench_en_test) |
| 46 | + - MMBench Chinese (mmbench_cn) |
| 47 | + - MMBench Chinese Dev (mmbench_cn_dev) |
| 48 | + - MMBench Chinese Test (mmbench_cn_test) |
| 49 | +- MME (mme) |
| 50 | +- MMMU (mmmu) |
| 51 | + - MMMU Validation (mmmu_val) |
| 52 | + - MMMU Test (mmmu_test) |
| 53 | +- MMUPD (mmupd) |
| 54 | + - MMUPD Base (mmupd_base) |
| 55 | + - MMAAD Base (mmaad_base) |
| 56 | + - MMIASD Base (mmiasd_base) |
| 57 | + - MMIVQD Base (mmivqd_base) |
| 58 | + - MMUPD Option (mmupd_option) |
| 59 | + - MMAAD Option (mmaad_option) |
| 60 | + - MMIASD Option (mmiasd_option) |
| 61 | + - MMIVQD Option (mmivqd_option) |
| 62 | + - MMUPD Instruction (mmupd_instruction) |
| 63 | + - MMAAD Instruction (mmaad_instruction) |
| 64 | + - MMIASD Instruction (mmiasd_instruction) |
| 65 | + - MMIVQD Instruction (mmivqd_instruction) |
| 66 | +- MMVet (mmvet) |
| 67 | +- Multi-DocVQA (multidocvqa) |
| 68 | + - Multi-DocVQA Validation (multidocvqa_val) |
| 69 | + - Multi-DocVQA Test (multidocvqa_test) |
| 70 | +- NoCaps (nocaps) |
| 71 | + - NoCaps Validation (nocaps_val) |
| 72 | + - NoCaps Test (nocaps_test) |
| 73 | +- OKVQA (ok_vqa) |
| 74 | + - OKVQA Validation 2014 (ok_vqa_val2014) |
| 75 | +- POPE (pope) |
| 76 | +- RefCOCO (refcoco) |
| 77 | + - refcoco_seg_test |
| 78 | + - refcoco_seg_val |
| 79 | + - refcoco_seg_testA |
| 80 | + - refcoco_seg_testB |
| 81 | + - refcoco_bbox_test |
| 82 | + - refcoco_bbox_val |
| 83 | + - refcoco_bbox_testA |
| 84 | + - refcoco_bbox_testB |
| 85 | +- RefCOCO+ (refcoco+) |
| 86 | + - refcoco+_seg |
| 87 | + - refcoco+_seg_val |
| 88 | + - refcoco+_seg_testA |
| 89 | + - refcoco+_seg_testB |
| 90 | + - refcoco+_bbox |
| 91 | + - refcoco+_bbox_val |
| 92 | + - refcoco+_bbox_testA |
| 93 | + - refcoco+_bbox_testB |
| 94 | +- RefCOCOg (refcocog) |
| 95 | + - refcocog_seg_test |
| 96 | + - refcocog_seg_val |
| 97 | + - refcocog_bbox_test |
| 98 | + - refcocog_bbox_val |
| 99 | +- ScienceQA (scienceqa_full) |
| 100 | + - ScienceQA Full (scienceqa) |
| 101 | + - ScienceQA IMG (scienceqa_img) |
| 102 | +- ScreenSpot (screenspot) |
| 103 | + - ScreenSpot REC / Grounding (screenspot_rec) |
| 104 | + - ScreenSpot REG / Instruction Generation (screenspot_reg) |
| 105 | +- SeedBench (seedbench) |
| 106 | +- SeedBench 2 (seedbench_2) |
| 107 | +- ST-VQA (stvqa) |
| 108 | +- TextCaps (textcaps) |
| 109 | + - TextCaps Validation (textcaps_val) |
| 110 | + - TextCaps Test (textcaps_test) |
| 111 | +- TextVQA (textvqa) |
| 112 | + - TextVQA Validation (textvqa_val) |
| 113 | + - TextVQA Test (textvqa_test) |
| 114 | +- VizWizVQA (vizwiz_vqa) |
| 115 | + - VizWizVQA Validation (vizwiz_vqa_val) |
| 116 | + - VizWizVQA Test (vizwiz_vqa_test) |
| 117 | +- VQAv2 (vqav2) |
| 118 | + - VQAv2 Validation (vqav2_val) |
| 119 | + - VQAv2 Test (vqav2_test) |
| 120 | +- WebSRC (websrc) |
| 121 | + - WebSRC Validation (websrc_val) |
| 122 | + - WebSRC Test (websrc_test) |
0 commit comments