Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Eval] multiple_choice.py: E->Z #41

Merged
merged 7 commits into from
Jan 8, 2024
Merged

[Eval] multiple_choice.py: E->Z #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
74158d5
E to Z
kennymckormick Jan 4, 2024
9fec824
update
kennymckormick Jan 5, 2024
8d517f9
update multiple_choice eval
kennymckormick Jan 5, 2024
1e29442
update
kennymckormick Jan 5, 2024
0a0e7af
Merge branch 'main' of github.com:open-compass/VLMEvalKit into EtoZ
kennymckormick Jan 8, 2024
dd0b416
update MMMU results
kennymckormick Jan 8, 2024
c635245
update MathVista Performance
kennymckormick Jan 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 30 additions & 29 deletions results/MMMU.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# MMMU Evaluation Results

> - In MMMU Evaluation, we evaluate the `dev` (150 samples) and `validation` (900 samples) set of MMMU.
> - For MMMU, we support the evaluation of the `dev` (150 samples) and `validation` (900 samples) set. Here we only report the results on the `validation` set.
> - **Answer Inference:**
> - For models with `interleave_generate` interface (accept interleaved images & texts as inputs), all testing samples can be inferred. **`interleave_generate` is adopted for inference.**
> - For models without `interleave_generate` interface, samples with more than one images are skipped (42 out of 1050, directly count as wrong). **`generate` is adopted for inference.**
Expand All @@ -11,31 +11,32 @@

### MMMU Scores

| Model | Overall<br>(Val) | Art & Design<br>(Val) | Business<br>(Val) | Science<br>(Val) | Health & Medicine<br>(Val) | Humanities & Social Science<br>(Val) | Tech & Engineering<br>(Val) | Overall<br>(Dev) |
|:------------------------------|-------------------:|------------------------:|--------------------:|-------------------:|-----------------------------:|---------------------------------------:|------------------------------:|-------------------:|
| GPT-4v (detail: low) | 53.8 | 66.7 | 60 | 46 | 54.7 | 71.7 | 36.7 | 52.7 |
| GeminiProVision | 48.4 | 59.2 | 36 | 42 | 52 | 66.7 | 42.9 | 54 |
| Qwen-VL-Chat | 37.6 | 49.2 | 36 | 28 | 32.7 | 55.8 | 31.9 | 30 |
| CogVLM-17B-Chat | 37.1 | 51.7 | 33.3 | 35.3 | 34.7 | 42.5 | 31.4 | 41.3 |
| LLaVA-InternLM-7B (LoRA) | 37 | 44.2 | 32 | 29.3 | 38.7 | 47.5 | 34.8 | 43.3 |
| LLaVA-v1.5-13B | 36.8 | 49.2 | 23.3 | 36 | 34 | 51.7 | 33.3 | 42 |
| ShareGPT4V-7B | 36.7 | 50 | 27.3 | 26.7 | 37.3 | 50 | 34.8 | 30 |
| TransCore-M | 36.6 | 54.2 | 32 | 27.3 | 32 | 49.2 | 32.4 | 38.7 |
| SharedCaptioner | 36.6 | 44.2 | 28.7 | 30 | 37.3 | 46.7 | 36.2 | 40 |
| LLaVA-v1.5-7B | 36.1 | 45.8 | 25.3 | 34 | 32 | 48.3 | 35.7 | 38.7 |
| InternLM-XComposer-VL | 35.7 | 45.8 | 28.7 | 22.7 | 30.7 | 53.3 | 37.6 | 36.7 |
| LLaVA-v1.5-13B (LoRA, XTuner) | 35.1 | 40.8 | 30.7 | 26.7 | 35.3 | 45 | 35.2 | 43.3 |
| mPLUG-Owl2 | 34.6 | 47.5 | 26 | 21.3 | 37.3 | 50 | 31.9 | 40.7 |
| LLaVA-v1.5-7B (LoRA, XTuner) | 33.7 | 48.3 | 23.3 | 30 | 32.7 | 46.7 | 28.6 | 37.3 |
| instructblip_13b | 32.9 | 37.5 | 29.3 | 32 | 28.7 | 37.5 | 33.8 | 30 |
| PandaGPT-13B | 32.7 | 42.5 | 35.3 | 30 | 29.3 | 45.8 | 21.9 | 26.7 |
| LLaVA-v1-7B | 32.1 | 31.7 | 24.7 | 31.3 | 32 | 37.5 | 35.2 | 33.3 |
| InstructBLIP-7B | 30.4 | 38.3 | 28 | 22 | 30.7 | 39.2 | 28.6 | 24 |
| VisualGLM | 28.9 | 30 | 24 | 28 | 28 | 40.8 | 26.2 | 28.7 |
| Qwen-VL | 28.8 | 43.3 | 18.7 | 25.3 | 32.7 | 42.5 | 19.5 | 29.3 |
| OpenFlamingo v2 | 28.2 | 27.5 | 30 | 28.7 | 28 | 33.3 | 24.3 | 21.3 |
| MiniGPT-4-v1-13B | 26.2 | 33.3 | 19.3 | 28.7 | 26 | 34.2 | 21 | 23.3 |
| IDEFICS-80B-Instruct | 25.1 | 39.2 | 17.3 | 23.3 | 24 | 48.3 | 11.4 | 23.3 |
| MiniGPT-4-v2 | 24.6 | 27.5 | 22.7 | 21.3 | 28 | 33.3 | 19 | 32 |
| MiniGPT-4-v1-7B | 23 | 32.5 | 27.3 | 18.7 | 17.3 | 15 | 26.2 | 19.3 |
| IDEFICS-9B-Instruct | 19.6 | 22.5 | 11.3 | 20.7 | 23.3 | 31.7 | 13.3 | 20 |
| Model | Overall | Art & Design | Business | Science | Health & Medicine | Humanities & Social Science | Tech & Engineering |
|:------------------------------|----------:|---------------:|-----------:|----------:|--------------------:|------------------------------:|---------------------:|
| GPT-4v (detail: low) | 53.8 | 67.5 | 59.3 | 46 | 54.7 | 70.8 | 37.1 |
| GeminiProVision | 48.9 | 59.2 | 36.7 | 42.7 | 52 | 66.7 | 43.8 |
| CogVLM-17B-Chat | 37.3 | 51.7 | 34 | 36 | 35.3 | 41.7 | 31.4 |
| Qwen-VL-Chat | 37 | 49.2 | 35.3 | 28 | 31.3 | 54.2 | 31.9 |
| LLaVA-InternLM-7B (LoRA) | 36.9 | 44.2 | 32 | 29.3 | 38.7 | 46.7 | 34.8 |
| LLaVA-v1.5-13B | 36.9 | 49.2 | 24 | 37.3 | 33.3 | 50.8 | 33.3 |
| TransCore-M | 36.9 | 54.2 | 32.7 | 28 | 32 | 48.3 | 33.3 |
| ShareGPT4V-7B | 36.6 | 50 | 28.7 | 26 | 37.3 | 49.2 | 34.3 |
| SharedCaptioner | 36.3 | 44.2 | 28.7 | 29.3 | 37.3 | 45.8 | 36.2 |
| LLaVA-v1.5-7B | 36.2 | 45.8 | 26 | 34 | 32.7 | 47.5 | 35.7 |
| InternLM-XComposer-VL | 35.6 | 45.8 | 28.7 | 22.7 | 30.7 | 52.5 | 37.6 |
| LLaVA-v1.5-13B (LoRA, XTuner) | 35.2 | 40.8 | 30.7 | 27.3 | 35.3 | 44.2 | 35.7 |
| mPLUG-Owl2 | 34.7 | 47.5 | 26 | 21.3 | 38 | 50 | 31.9 |
| LLaVA-v1.5-7B (LoRA, XTuner) | 33.7 | 48.3 | 23.3 | 30.7 | 32.7 | 45.8 | 28.6 |
| InstructBLIP-13B | 33.2 | 37.5 | 30 | 32.7 | 30 | 36.7 | 33.8 |
| PandaGPT-13B | 32.9 | 42.5 | 36 | 30.7 | 30 | 43.3 | 22.9 |
| LLaVA-v1-7B | 32.3 | 31.7 | 26 | 31.3 | 32.7 | 35.8 | 35.7 |
| InstructBLIP-7B | 30.6 | 38.3 | 28.7 | 22 | 30.7 | 39.2 | 28.6 |
| VisualGLM | 29.9 | 30.8 | 27.3 | 28.7 | 29.3 | 40.8 | 26.2 |
| Qwen-VL | 29.6 | 45 | 18.7 | 26.7 | 32.7 | 42.5 | 21 |
| OpenFlamingo v2 | 28.8 | 27.5 | 30.7 | 29.3 | 28.7 | 33.3 | 25.2 |
| MiniGPT-4-v1-13B | 26.3 | 31.7 | 20.7 | 28 | 25.3 | 35 | 21.9 |
| Frequent Choice | 25.8 | 26.7 | 28.4 | 24 | 24.4 | 25.2 | 26.5 |
| MiniGPT-4-v2 | 25 | 27.5 | 23.3 | 22 | 27.3 | 32.5 | 21 |
| IDEFICS-80B-Instruct | 24 | 39.2 | 18 | 20 | 22 | 46.7 | 11 |
| MiniGPT-4-v1-7B | 23.6 | 33.3 | 28.7 | 19.3 | 18 | 15 | 26.2 |
| IDEFICS-9B-Instruct | 18.4 | 22.5 | 11.3 | 17.3 | 21.3 | 30 | 13.3 |
48 changes: 24 additions & 24 deletions results/MathVista.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,31 +11,31 @@

| Model | ALL | SCI | TQA | NUM | ARI | VQA | GEO | ALG | GPS | MWP | LOG | FQA | STA |
|:------------------------------|------:|------:|------:|------:|------:|------:|------:|------:|------:|------:|------:|------:|------:|
| **Human (High School)** | 60.3 | 64.9 | 63.2 | 53.8 | 59.2 | 55.9 | 51.4 | 50.9 | 48.4 | 73 | 40.7 | 59.7 | 63.9 |
| GPT-4v (detail: low) | 47.5 | 62.3 | 67.1 | 22.9 | 45.6 | 38.5 | 50.2 | 53.4 | 50 | 57 | 16.2 | 33.5 | 45.5 |
| GeminiProVision | 45.7 | 57.4 | 60.8 | 27.1 | 41.9 | 40.2 | 39.7 | 42.3 | 38.5 | 45.7 | 10.8 | 46.1 | 52.5 |
| CogVLM-17B-Chat | 34.6 | 50.8 | 44.9 | 23.6 | 31.2 | 36.3 | 26.4 | 28.1 | 26 | 27.4 | 16.2 | 39 | 42.2 |
| Qwen-VL-Chat | 33.5 | 41 | 39.2 | 24.3 | 28 | 32.4 | 28.5 | 30.2 | 29.8 | 25.8 | 13.5 | 39 | 40.9 |
| InternLM-XComposer-VL | 29.3 | 36.9 | 37.3 | 27.8 | 28.3 | 34.1 | 31.4 | 28.1 | 28.8 | 29 | 13.5 | 21.9 | 21.9 |
| SharedCaptioner | 28.9 | 37.7 | 37.3 | 34.7 | 28.3 | 33 | 25.9 | 23.8 | 22.1 | 36 | 16.2 | 21.6 | 20.9 |
| LLaVA-v1.5-13B | 26.5 | 37.7 | 38.6 | 22.9 | 24.9 | 32.4 | 22.6 | 24.2 | 22.6 | 18.8 | 21.6 | 23.8 | 23.9 |
| LLaVA-InternLM-7B (LoRA) | 26.4 | 32 | 34.8 | 20.1 | 22.1 | 29.6 | 27.6 | 28.1 | 27.9 | 21 | 24.3 | 21.9 | 20.3 |
| LLaVA-v1.5-13B (LoRA, XTuner) | 26.3 | 44.3 | 39.9 | 20.1 | 24.1 | 32.4 | 20.9 | 22.4 | 21.6 | 18.8 | 18.9 | 23 | 22.9 |
| **Human (High School)** | 60.3 | 64.9 | 63.2 | 53.8 | 59.2 | 55.9 | 51.4 | 50.9 | 48.4 | 73 | 40.7 | 59.7 | 63.9 |
| GPT-4v (detail: low) | 47.8 | 63.9 | 67.1 | 22.9 | 45.9 | 38.5 | 49.8 | 53 | 49.5 | 57.5 | 18.9 | 34.6 | 46.5 |
| GeminiProVision | 45.8 | 58.2 | 60.8 | 27.1 | 41.9 | 40.2 | 39.7 | 42.3 | 38.5 | 45.7 | 10.8 | 46.5 | 52.8 |
| CogVLM-17B-Chat | 34.7 | 51.6 | 44.3 | 23.6 | 30.9 | 36.3 | 26.8 | 28.1 | 26.4 | 26.9 | 16.2 | 39.8 | 42.9 |
| Qwen-VL-Chat | 33.8 | 41.8 | 39.2 | 24.3 | 28.3 | 33 | 28.5 | 30.2 | 29.8 | 25.8 | 13.5 | 39.8 | 41.5 |
| InternLM-XComposer-VL | 29.5 | 37.7 | 37.3 | 27.8 | 28.6 | 34.1 | 31.8 | 28.1 | 28.8 | 29.6 | 13.5 | 22.3 | 22.3 |
| SharedCaptioner | 29 | 37.7 | 37.3 | 35.4 | 28.3 | 33 | 25.9 | 23.8 | 22.1 | 36.6 | 16.2 | 21.6 | 20.9 |
| LLaVA-v1.5-13B | 26.4 | 37.7 | 38.6 | 22.9 | 24.9 | 32.4 | 22.6 | 24.2 | 22.6 | 18.8 | 21.6 | 23.4 | 23.6 |
| LLaVA-InternLM-7B (LoRA) | 26.3 | 32 | 34.8 | 20.8 | 22.4 | 30.2 | 27.6 | 28.1 | 27.9 | 21 | 24.3 | 21.2 | 19.6 |
| IDEFICS-80B-Instruct | 26.2 | 37.7 | 34.8 | 22.2 | 25.2 | 33 | 23.4 | 22.8 | 23.1 | 21.5 | 18.9 | 22.3 | 21.3 |
| LLaVA-v1.5-13B (LoRA, XTuner) | 26.2 | 44.3 | 39.2 | 20.1 | 24.1 | 32.4 | 21.3 | 22.4 | 22.1 | 18.8 | 18.9 | 22.7 | 22.6 |
| ShareGPT4V-7B | 25.8 | 41 | 38.6 | 19.4 | 25.5 | 36.3 | 19.7 | 21.4 | 20.2 | 16.1 | 13.5 | 22.3 | 21.6 |
| IDEFICS-80B-Instruct | 25.7 | 37.7 | 34.8 | 22.9 | 25.5 | 33.5 | 20.9 | 20.6 | 20.2 | 21.5 | 18.9 | 22.3 | 21.3 |
| TransCore-M | 25.4 | 41 | 44.3 | 19.4 | 24.4 | 34.1 | 21.3 | 24.6 | 20.7 | 17.2 | 13.5 | 17.8 | 18.9 |
| TransCore-M | 25.4 | 41 | 44.3 | 19.4 | 24.4 | 34.1 | 20.9 | 24.2 | 20.2 | 17.2 | 13.5 | 18.2 | 19.3 |
| mPLUG-Owl2 | 25.3 | 44.3 | 41.8 | 18.8 | 23.5 | 31.8 | 18.8 | 20.3 | 17.8 | 16.7 | 13.5 | 23 | 23.9 |
| PandaGPT-13B | 24.5 | 35.2 | 30.4 | 17.4 | 21 | 27.4 | 23.8 | 23.8 | 25.5 | 18.8 | 16.2 | 22.3 | 21.6 |
| LLaVA-v1.5-7B (LoRA, XTuner) | 24.1 | 39.3 | 35.4 | 17.4 | 22.1 | 30.2 | 21.3 | 21 | 21.6 | 16.1 | 24.3 | 20.8 | 20.3 |
| LLaVA-v1-7B | 23.8 | 32.8 | 34.2 | 13.9 | 20.7 | 28.5 | 22.2 | 24.6 | 24 | 13.4 | 10.8 | 21.6 | 20.3 |
| InstructBLIP-7B | 23.6 | 32.8 | 31.6 | 13.9 | 23.5 | 29.6 | 19.7 | 20.6 | 20.2 | 15.6 | 13.5 | 23 | 20.9 |
| PandaGPT-13B | 24.6 | 36.1 | 30.4 | 17.4 | 21 | 27.4 | 23.8 | 23.8 | 25.5 | 18.8 | 16.2 | 22.7 | 21.9 |
| LLaVA-v1.5-7B (LoRA, XTuner) | 24.2 | 39.3 | 36.1 | 17.4 | 22.1 | 30.2 | 21.3 | 21.4 | 21.6 | 16.1 | 24.3 | 20.8 | 20.3 |
| InstructBLIP-7B | 23.7 | 33.6 | 31.6 | 13.9 | 23.5 | 29.6 | 19.7 | 20.6 | 20.2 | 15.6 | 13.5 | 23.4 | 21.3 |
| LLaVA-v1-7B | 23.7 | 32.8 | 34.2 | 13.9 | 20.7 | 28.5 | 22.2 | 24.6 | 24 | 13.4 | 10.8 | 21.2 | 19.9 |
| LLaVA-v1.5-7B | 23.6 | 33.6 | 36.7 | 11.1 | 21 | 28.5 | 18.8 | 23.1 | 19.2 | 14.5 | 13.5 | 22.3 | 21.6 |
| MiniGPT-4-v2 | 23 | 31.1 | 32.9 | 13.2 | 17 | 25.7 | 22.2 | 26.3 | 24 | 10.8 | 16.2 | 23 | 20.6 |
| VisualGLM | 21.6 | 37.7 | 29.7 | 15.3 | 18.4 | 30.2 | 22.2 | 22.4 | 24 | 7.5 | 2.7 | 19 | 18.9 |
| MiniGPT-4-v2 | 22.9 | 29.5 | 32.3 | 13.2 | 17 | 25.7 | 22.6 | 26.7 | 24.5 | 10.8 | 16.2 | 22.7 | 20.3 |
| VisualGLM | 21.5 | 36.9 | 29.7 | 15.3 | 18.1 | 30.2 | 22.2 | 22.8 | 24 | 7 | 2.7 | 19 | 18.6 |
| InstructBLIP-13B | 21.5 | 28.7 | 27.8 | 19.4 | 21.5 | 31.8 | 17.6 | 18.5 | 18.3 | 13.4 | 13.5 | 19 | 17.9 |
| IDEFICS-9B-Instruct | 20.8 | 29.5 | 31 | 13.2 | 17.8 | 29.6 | 16.7 | 20.3 | 17.8 | 8.1 | 13.5 | 20.1 | 18.6 |
| MiniGPT-4-v1-13B | 20.8 | 27 | 24.7 | 9 | 18.4 | 27.9 | 21.8 | 23.5 | 23.6 | 9.7 | 10.8 | 19.3 | 17.3 |
| MiniGPT-4-v1-7B | 20.7 | 27.9 | 29.1 | 8.3 | 17.3 | 23.5 | 21.3 | 23.5 | 22.6 | 13.4 | 8.1 | 17.5 | 17.3 |
| OpenFlamingo v2 | 19 | 22.1 | 24.7 | 5.6 | 16.4 | 24 | 21.8 | 24.2 | 24 | 8.1 | 10.8 | 16 | 14.3 |
| **Random Chance** | 17.9 | 15.8 | 23.4 | 8.8 | 13.8 | 24.3 | 22.7 | 25.8 | 24.1 | 4.5 | 13.4 | 15.5 | 14.3 |
| Qwen-VL | 14.5 | 34.4 | 29.7 | 10.4 | 12.2 | 22.9 | 6.7 | 7.8 | 5.8 | 5.4 | 16.2 | 13 | 10.3 |
| IDEFICS-9B-Instruct | 20.4 | 29.5 | 31 | 13.2 | 17.8 | 29.6 | 15.1 | 18.9 | 15.9 | 8.1 | 13.5 | 20.1 | 18.6 |
| MiniGPT-4-v1-13B | 20.4 | 27 | 24.7 | 9 | 18.1 | 27.4 | 20.9 | 22.8 | 22.6 | 9.7 | 10.8 | 19 | 16.9 |
| MiniGPT-4-v1-7B | 20.2 | 27 | 29.1 | 7.6 | 16.7 | 21.8 | 20.9 | 23.1 | 22.1 | 14 | 5.4 | 16.7 | 17.3 |
| OpenFlamingo v2 | 18.6 | 22.1 | 24.7 | 5.6 | 16.4 | 24 | 21.3 | 23.8 | 23.6 | 8.1 | 10.8 | 14.9 | 13.3 |
| **Random Chance** | 17.9 | 15.8 | 23.4 | 8.8 | 13.8 | 24.3 | 22.7 | 25.8 | 24.1 | 4.5 | 13.4 | 15.5 | 14.3 |
| Qwen-VL | 15.5 | 34.4 | 29.7 | 10.4 | 12.2 | 22.9 | 9.6 | 10.7 | 9.1 | 5.4 | 16.2 | 14.1 | 11.6 |
Loading