A Toolkit for Evaluating Large Vision-Language Models.

📊Datasets & Models • 🏗️Quickstart • 🛠️Support New • 🎯Goal • 🖊️Citation

VLMEvalKit (the python package name is vlmeval) is an open-source evaluation toolkit of large vision-language models (LVLMs). It enables one-command evaluation of LVLMs on various benchmarks, without the heavy workload of data preparation under multiple repositories. In VLMEvalKit, we adopt generation-based evaluation for all LVLMs (obtain the answer via generate / chat interface), and provide the evaluation results obtained with both exact matching and LLM(ChatGPT)-based answer extraction.

🆕 News

• [2023-12-26] We support QwenVLPlus API now. The evaluation result is on the road. 🔥🔥🔥
• [2023-12-26] We support MMMU now (Dataset Name: MMMU_DEV_VAL). The evaluation result is available at MMMU. 🔥🔥🔥
• [2023-12-24] We support two VQA Datasets: OCRVQA (Dataset Name: OCRVQA_TEST, OCRVQA_TESTCORE) and TextVQA (Dataset Name: TextVQA_VAL). The evaluation is undergoing. 🔥🔥🔥
• [2023-12-23] We update the performance of GPT-4v and GeminiProVision on all existing benchmarks, check the result. 🔥🔥🔥
• [2023-12-20] We support a new benchmark: COCO Caption (Dataset Name: COCO_VAL). The evaluation is undergoing. 🔥🔥🔥

📊 Datasets, Models, and Evaluation Results

Supported Dataset

Dataset	Dataset Names (for run.py)	Inference	Evaluation	Results
MMBench Series: MMBench, MMBench-CN, CCBench	MMBench-DEV-[EN/CN] MMBench-TEST-[EN/CN] CCBench	✅	✅	MMBench Series
MME	MME	✅	✅	MME
SEEDBench_IMG	SEEDBench_IMG	✅	✅	SEEDBench_IMG
MM-Vet	MMVet	✅	✅	MM-Vet
MMMU	MMMU_DEV_VAL	✅	✅	MMMU
COCO Caption	COCO_VAL	✅	✅
OCRVQA	OCRVQA_TESTCORE, OCRVQA_TEST	✅	✅
TextVQA	TextVQA_VAL	✅	✅
Core-MM	CORE_MM	✅

Supported API Models

GPT-4-Vision-Preview	GeminiProVision	QwenVLPlus

Supported PyTorch / HF Models

IDEFICS-9B-Instruct🎞️🚅, IDEFICS-80B-Instruct🎞️🚅	InstructBLIP-[7B/13B]	LLaVA-[v1-7B/v1.5-7B/v1.5-13B]	MiniGPT-4-[v1-7B/v1-13B/v2-7B]	mPLUG-Owl2🎞️
OpenFlamingo-v2🎞️	PandaGPT-13B	Qwen-VL🎞️🚅, Qwen-VL-Chat🎞️🚅	VisualGLM-6B🚅	InternLM-XComposer-7B🎞️🚅
ShareGPT4V-7B🚅	TransCore-M	LLaVA (XTuner)🚅

🎞️: Support multiple images as inputs, via the multi_generate interface.

🚅: Model can be used without any additional configuration / operation.

# Demo
from vlmeval.config import supported_VLM
model = supported_VLM['idefics_9b_instruct']()
# Forward Single Image
ret = model.generate('assets/apple.jpg', 'What is in this image?')
print(ret)  # The image features a red apple with a leaf on it.
# Forward Multiple Images
ret = model.multi_generate(['assets/apple.jpg', 'assets/apple.jpg'], 'How many apples are there in the provided images? ')
print(ret)  # There are two apples in the provided images.

🏗️ QuickStart

Before running the evaluation script, you need to configure the VLMs and set the model_paths properly.

After that, you can use a single script run.py to inference and evaluate multiple VLMs and benchmarks at a same time.

Step0. Installation

git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit
pip install -e .

Step1. Configuration

VLM Configuration: All VLMs are configured in vlmeval/config.py, for some VLMs, you need to configure the code root (MiniGPT-4, PandaGPT, etc.) or the model_weight root (LLaVA-v1-7B, etc.) before conducting the evaluation. During evaluation, you should use the model name specified in supported_VLM in vlmeval/config.py to select the VLM. For MiniGPT-4 and InstructBLIP, you also need to modify the config files in vlmeval/vlm/misc to configure LLM path and ckpt path.

Following VLMs require the configuration step:

Code Preparation & Installation: InstructBLIP (LAVIS), LLaVA (LLaVA), MiniGPT-4 (MiniGPT-4), mPLUG-Owl2 (mPLUG-Owl2), OpenFlamingo-v2 (OpenFlamingo), PandaGPT-13B (PandaGPT), TransCore-M (TransCore-M), LLaVA-XTuner (XTuner).

Manual Weight Preparation & Configuration: InstructBLIP, LLaVA-v1-7B, MiniGPT-4, OpenFlamingo-v2, PandaGPT-13B

Step2. Evaluation

We use run.py for evaluation. To use the script, you can use $VLMEvalKit/run.py or create a soft-link of the script (to use the script anywhere):

Arguments

• --data (list[str]): Set the dataset names that are supported in VLMEvalKit (defined in vlmeval/utils/data_util.py).
  • including: MME, SEEDBench_IMG, MMBench_DEV_EN, MMBench_TEST_EN, MMBench_DEV_CN, MMBench_TEST_CN, CCBench, Core_MM, MMVet
• --model (list[str]): Set the VLM names that are supported in VLMEvalKit (defined in supported_VLM in vlmeval/config.py).
• --mode (str, default to 'all', choices are ['all', 'infer']): When mode set to "all", will perform both inference and evaluation; when set to "infer", will only perform the inference.
• --nproc (int, default to 4): The number of threads for OpenAI API calling.
• --verbose (bool, store_true)

Command

You can run the script with python or torchrun:

# When running with `python`, only one VLM instance is instantiated, and it might use multiple GPUs (depending on its default behavior). 
# That is recommended for evaluating very large VLMs (like IDEFICS-80B-Instruct).

# IDEFICS-80B-Instruct on MMBench_DEV_EN, MME, and SEEDBench_IMG, Inference and Evalution
python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose 
# IDEFICS-80B-Instruct on MMBench_DEV_EN, MME, and SEEDBench_IMG, Inference only
python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose --mode infer

# When running with `torchrun`, one VLM instance is instantiated on each GPU. It can speed up the inference. 
# However, that is only suitable for VLMs that consume small amounts of GPU memory. 

# IDEFICS-9B-Instruct, Qwen-VL-Chat, mPLUG-Owl2 on MMBench_DEV_EN, MME, and SEEDBench_IMG. On a node with 8 GPU. Inference and Evaluation.
torchrun --nproc-per-node=8 run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct qwen_chat mPLUG-Owl2 --verbose
# Qwen-VL-Chat on MME. On a node with 2 GPU. Inference and Evaluation. 
torchrun --nproc-per-node=2 run.py --data MME --model qwen_chat --verbose

The evaluation results will be printed as logs, besides. Result Files will also be generated in the directory $YOUR_WORKING_DIRECTORY/{model_name}. Files ending with .csv contain the evaluated metrics.

🛠️ Custom Benchmark or VLM

To implement a custom benchmark or VLM in VLMEvalKit, please refer to Custom_Benchmark_and_Model.

🎯 The Goal of VLMEvalKit

The codebase is designed to:

1. Provide an easy-to-use, opensource evaluation toolkit to make it convenient for researchers & developers to evaluate existing LVLMs and make evaluation results easy to reproduce.
2. Make it easy for VLM developers to evaluate their own models. To evaluate the VLM on multiple supported benchmarks, one just need to implement a single generate function, all other workloads (data downloading, data preprocessing, prediction inference, metric calculation) are handled by the codebase.

The codebase is not designed to:

1. Reproduce the exact accuracy number reported in the original papers of all 3rd party benchmarks. The reason can be two-fold:
   1. VLMEvalKit mainly uses generation-based evaluation for all VLMs (and optionally with LLM-based answer extraction). Meanwhile, some benchmarks may use different metrics (SEEDBench uses PPL-based evaluation, eg.). For these benchmarks, we will compare both scores in the corresponding README file. We encourage developers to support other different evaluation paradigms in the codebase.
   2. By default, we use the same prompt template for all VLMs to evaluate on a benchmark. Meanwhile, some VLMs may have their specific prompt templates (some may not covered by the codebase at this time). We encourage VLM developers to implement their own prompt template in VLMEvalKit, if that is not covered currently. That will help to improve the reproducibility.

🖊️ Citation

If you use VLMEvalKit in your research or wish to refer to the published OpenSource evaluation results, please use the following BibTeX entry and the BibTex entry corresponding to the specific VLM / benchmark you used.

@article{MMBench,
    author  = {Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, Dahua Lin},
    journal = {arXiv:2307.06281},
    title   = {MMBench: Is Your Multi-modal Model an All-around Player?},
    year    = {2023},
}

@misc{2023opencompass,
    title={OpenCompass: A Universal Evaluation Platform for Foundation Models},
    author={OpenCompass Contributors},
    howpublished = {\url{https://github.com/open-compass/opencompass}},
    year={2023}
}

💻 Other Projects in OpenCompass

• opencompass: An LLM evaluation platform, supporting a wide range of models (LLaMA, LLaMa2, ChatGLM2, ChatGPT, Claude, etc) over 50+ datasets.
• MMBench: Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"
• BotChat: Evaluating LLMs' multi-round chatting capability.
• LawBench: Benchmarking Legal Knowledge of Large Language Models.