Behind the AI Scorecards Lies a Chinese 'Question Setter'
Behind the AI scorecards that dominate industry discussions—benchmarks like MMLU-Pro, MMMU, and MMMU-Pro—stands a Chinese-Canadian researcher: Wenhu Chen. As an assistant professor at the University of Waterloo and founder of the TIGER Lab, Chen has become a key "exam-setter" for evaluating large language and multimodal models.
Chen first gained broader recognition with MMLU-Pro, a more challenging and stable update to the popular MMLU benchmark. As top models like OpenAI’s o3 began achieving near-perfect scores on the original MMLU, it became difficult to distinguish their true capabilities. MMLU-Pro introduced more complex reasoning questions, expanded answer choices, and filtered out ambiguous or simple items, effectively reintroducing differentiation among state-of-the-art models.
His work on MMMU addressed the evaluation of multimodal models, requiring them to integrate visual information (like charts, diagrams, or tables) with textual knowledge across diverse academic subjects. Even the strongest models initially scored only around 56-59%, highlighting significant room for improvement in genuine multimodal reasoning. MMMU-Pro further refined this by preventing models from bypassing visual cues.
Chen’s research focus has long been on complex information understanding and reasoning. His background—including a PhD at UC Santa Barbara, research at Google/DeepMind on Gemini, and now a role in Meta’s superintelligence lab—provides deep insight into model development and their potential weaknesses. His TIGER Lab also builds models (e.g., for video understanding and generation), ensuring his evaluation benchmarks are grounded in practical challenges.
While AI headlines often spotlight company leaders and product launches, Chen’s work exemplifies the critical, behind-the-scenes contributions of researchers crafting the rigorous standards that define and drive progress in AI capabilities.
marsbit06/19 09:18
