Mia-Bench: Towards a better instruction after the multimodal LLM evaluation

We introduce Mia Bench, a new reference point designed to evaluate large multimodal language models (MLLM) about its ability to strictly adhere to complex instructions. Our reference point comprises a diverse set of 400 image Prompt, each designed to challenge the fulfillment of the models with the layer instructions to generate precise responses that satisfy specific patterns requested. The results of the evaluation of a wide range of latest MLLM reveal significant variations in performance, highlighting the areas to improve the loyalty of the instruction. In addition, we create additional training data and explore the supervised fine adjustment to improve the ability of models to strictly follow instructions without compromising performance in other tasks. We hope that this reference point not only serves as a tool to measure MLLM fulfillment to the instructions, but also guide future developments in MLLM training methods.