Each time a cutting-edge model is released, the AI community focuses on a few familiar report cards.
MMLU-Pro, MMMU, MMMU-Pro... These names might sound foreign to ordinary users, but for model companies and researchers, they have almost become the "standard subjects." GPT, Claude, Gemini, Llama, Qwen, DeepSeek continuously submit their answers on these benchmarks.
"The proof is in the pudding." How good a model is often needs to be proven by these scores.
Many performance comparison charts in model launch presentations rely on them; some leaderboards on HuggingFace are also built upon these evaluation systems. It could even be said that today, when the AI industry discusses model capabilities, they are already using a common language defined by these benchmarks.
But interestingly, almost everyone focuses on the scores, yet few know who sets the questions. Behind MMLU-Pro, MMMU, and MMMU-Pro, the same name can be seen—Wenhu Chen.
He is an Assistant Professor in the Department of Computer Science at the University of Waterloo in Canada. On Google Scholar, his papers have been cited over 30,000 times.
He is also the founder of TIGERLab. The English full name of this lab is Text and Image GEnerative Research Lab. Because the Chinese word for "tiger" is in his name, Wenhu Chen gave it a very distinctive Chinese name—Hutou Bang (Tiger Head Gang).
01
After the Old Exam Papers Lost Their Effectiveness
Wenhu Chen first caught wider attention because of MMLU-Pro.
MMLU was once one of the most commonly used benchmark evaluations for assessing the capabilities of large language models. It was like a comprehensive test paper, covering multiple subjects, used to measure a model's performance in knowledge understanding and reasoning tasks.
Early on, this paper was very useful. It could distinguish between models through scores, and the industry could also use it to observe whether large language models were truly improving.
But problems soon emerged.
As model capabilities continuously improved, MMLU gradually became "insufficiently challenging." The scores of cutting-edge models got higher and higher, and the gaps between them became smaller and smaller.
After OpenAI released o3, this problem became even more apparent. The accuracy of o3 on MMLU was already close to 100%, and other cutting-edge models also successively submitted scores approaching full marks.
This might sound like good news, but for evaluation, it actually meant trouble.
If everyone can get close to full marks on an exam paper, it becomes very difficult to continue judging who is stronger and where their strengths lie. It can still prove that models possess certain capabilities, but it is no longer suitable for measuring new progress.
The AI industry needed a harder, less easily "fooled" exam paper.
In 2024, Wenhu Chen and his team launched MMLU-Pro.
MMLU-Pro revamped this exam paper rather than simply expanding the question bank.
It contains 12,032 questions, covering 14 fields including mathematics, physics, chemistry, law, engineering, psychology, and health. Compared to the original MMLU, it expands the options from 4 to 10, reducing the probability of models guessing correctly. It also incorporates more reasoning-oriented questions and cleans up the original question bank of questions that were relatively simple, ambiguous, or lacked sufficient discriminative power.
The effect was direct.
The paper's results showed that model accuracy on MMLU-Pro decreased by 16% to 33% compared to the original MMLU. When the same model was tested under 24 different prompt styles, the score variation also decreased from 4% to 5% in the original MMLU to about 2%.
In other words, this new paper is not only harder but also more stable.
It reopened the gaps between models that all seemed excellent on the old exam paper. It also became easier to tell whether a model truly understands reasoning or is just better at handling old-style questions.
02
Usable Benchmark Evaluations
MMLU-Pro was quickly adopted by the industry.
MMLU-Pro later entered the NeurIPS 2024 Datasets and Benchmarks track and was also integrated into EleutherAI's lm-evaluation-harness framework. For the open-source model community, this meant it was no longer just a dataset in a paper but had entered the common evaluation toolchain.
Many models began reporting MMLU-Pro scores upon release. Some leaderboards on HuggingFace also incorporated it into their evaluation systems.
If MMLU-Pro solved the problem of the "old exam paper losing effectiveness" in language model evaluation, then MMMU pushed Wenhu Chen and TIGERLab to the center of multimodal evaluation.
The problems with multimodal models are more complex.
Language models answer questions, mainly processing text. Multimodal models, however, have to simultaneously process information in different forms like images, charts, diagrams, maps, tables, musical scores, chemical structures, etc. They not only need to understand the question stem but also truly comprehend the content in the images, and reason by integrating visual information, textual information, and domain knowledge.
The MMMU benchmark contains 11,500 multimodal questions sourced from university exams, quizzes, and textbooks, covering six major domains: Arts & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Technology & Engineering, further subdivided into 30 subjects and 183 subfields.
These questions are not simply asking the model "what's in the picture." They require the model to combine image information with domain knowledge, much like a student tackling a professional problem.
When MMMU was released, the research team tested 14 open-source multimodal models, as well as representative closed-source models like GPT-4V and Gemini Ultra. Even the strongest closed-source models at the time, GPT-4V and Gemini Ultra, only achieved accuracy rates of 56% and 59% respectively.
These numbers indicate that while multimodal models appear to be progressing rapidly, they still have significant room for improvement when it comes to problems requiring genuine professional understanding and reasoning.
Later, Wenhu Chen's team released MMMU-Pro, further plugging the gaps that allowed models to bypass visual information. It filters out questions that could be answered by text-only models, expands answer choices, and introduces a vision-only setting where questions are embedded within images, requiring the model to perform both visual reading and text comprehension simultaneously.
Simply put, it prevents the model from "guessing the answer just by looking at the text."
This kind of work might sound somewhat tedious, but it is crucial. Because future multimodal models need to enter scenarios like healthcare, education, scientific research, design, and engineering; merely being able to describe a picture is not enough. They must be able to judge, reason, explain, and find the truly useful parts within complex visual information.
03
The People Behind the "Exam Papers"
Wenhu Chen's later work on MMLU-Pro and MMMU stems from his long-standing research direction.
His research interests have always been related to complex information understanding, knowledge question answering, and reasoning.
He earned his bachelor's degree from Huazhong University of Science and Technology, then pursued a master's at RWTH Aachen University in Germany, followed by a Ph.D. in Computer Science from the University of California, Santa Barbara. During his Ph.D., he had already begun research in areas like complex question answering, table reasoning, and knowledge evidence localization.
These tasks share a common characteristic: the answer often does not lie within a single piece of text.
It might be hidden in a table, require combining a piece of text and an image, or might need the model to first retrieve information, then integrate, calculate, and reason. The model cannot just be good at reciting existing knowledge.
Projects Wenhu Chen participated in, such as HybridQA, TabFact, Program of Thoughts, and MAmmoTH, are all related to this line of work.
This also explains his sensitivity to loopholes in model evaluation.
A good benchmark evaluation is not simply about making questions increasingly difficult, but about anticipating where models are most likely to "guess correctly" or "appear competent."
A model might memorize the question bank, guess answers based on options, or use text to bypass visual information... Good evaluation needs to patch these loopholes well.
After his Ph.D., Wenhu Chen joined Google Research and later participated in the development and evaluation of Google DeepMind's Gemini multimodal model from 2021 to 2025. This experience was also important. Long-term exposure to cutting-edge model development gave him a clearer understanding of how model capabilities grow and made it easier to see potential biases and blind spots in evaluation.
In the fall of 2022, Wenhu Chen joined the David R. Cheriton School of Computer Science at the University of Waterloo as an Assistant Professor. The same year, he was selected as a Canada CIFAR AI Chair. Subsequently, he founded "TIGERLab" (aka Hutou Bang), continuing research focused on foundation models, multimodal capabilities, and benchmark evaluations.
Hutou Bang doesn't just work on benchmark evaluations; they also conduct model and system research.
In the video direction, UniVideo attempts to place video understanding, generation, and editing within the same framework, allowing the model not only to generate a sequence of frames but also to understand content, respond to instructions, and complete edits. Vamba targets long video understanding, addressing the memory, computation, and training efficiency challenges posed by hour-long videos. MoCha, a collaboration with Meta's Generative AI team, focuses on talking virtual character generation, producing high-quality character videos from voice and text descriptions.
An exam setter who never takes tests themselves cannot set good questions. Building models themselves, in turn, makes them more suitable for evaluation.
Because truly good evaluation often comes from an understanding of model capability boundaries. Only by knowing how models are built and what problems they encounter in real tasks can one more easily design questions that can differentiate performance and expose weaknesses.
Now, Wenhu Chen has joined Meta's Superalignment Lab, where his work continues to focus on multimodal pretraining data and evaluation, serving Meta's foundation models.
The AI industry does not lack visible figures. Typically, the spotlight falls on entrepreneurs, star researchers, and heads of large model companies. New product launches, funding news, open-source models, and team adjustments often attract the most external attention, making these names more visible to the public.
But today, the participation of Chinese talent in the AI field extends far beyond these most conspicuous positions.
This article is from the WeChat public account "Letters AI", author: Jin Ya
