# From KMMLU-Redux to Pro: A Professional Korean Benchmark Suite for LLM Evaluation

Authors: Seokhee Hong, SunKyoung Kim, Guijin Son, Soyeon Kim, Yeonjung Hong, Jinsik Lee, Qiancheng Zhang, Qihao Wang, Qin-Feng Zhu, Qiushi Chen, R. Du, R. L. Chen, R. Jin, Ruisong Ge, Ruizhe Zhang, Runji Pan, Runxin Wang, Ruoyu Xu, Ruyi Zhang, S. Chen, Shanghao Li, Shangyan Lu, Shanhuang Zhou, Shaoqing Chen, Shengfeng Wu, Shengfeng Ye, Shi-yi Ye, Shiyu Ma, Shuang Wang, Shuiping Zhou, Shunfeng Yu, Shuting Zhou, T. Pan, Tao Wang, Tian Yun, Tian Pei, W. Sun, W. Xiao, Wanjia Zeng, Wei Zhao, Wen An, Wenfeng Liu, Wenjun Liang, Wen-Wan Gao, Wentao Yu, X. Q. Zhang, Xiangyue Li, Xianzu Jin, Xiao Wang, Xiaodong Bi, Xiaohan Liu, Xiaojin Wang, Xi-aokang Shen, Xiaokang Chen, Xiaotao Chen, Xiaowen Nie, Xiaoxi-ang Sun, Xin Wang, C. Xin, Xin Liu, Xingchao Xie, Xingkai Liu, Xinnan Yu, Xinxia Song, Xinyi Shan, Xinyu Zhou, Xinyuan Yang, Xuecheng Li, Xuheng Su, Y. Lin, Q. LiY., Y. Wang, Y. X. Wei, Yang Zhu, Yan-hong Zhang, Yanhong Xu, Yanping Xu, Yao Huang, Yaohui Li, Yaofeng Zhao, Yao-wei Sun, Yaohui Li, Yi Wang, Yu Yi, Yichao Zheng, Yifan Zhang, Yiliang Shi, Xiong Ying, Ying He, Yishi Tang, Yisong Piao, Yixuan Wang, Yi-Xuan Tan, Yiyuan Ma, Yongqiang Liu, Guo Yu, Yuan Wu, Yuchen Ou, Yuduan Zhu, Yue Wang, Yuheng Gong, Yujia Zou, Yukun He, Yunfan Zha, Yunxian Xiong, Yuting Ma, Yuxiang Yan, Yuxi-ang Luo, Yuxuan You, Yuyang Liu, Z. Zhou, Z. Wu, Z. Ren, Zhangli Ren, Zhengbu Sha, Zhe Fu, Xu Zhen, Zhen Huang, Zhenda Zhang, Zhengyan Xie, Zhewen Zhang, Zhi Hao, Z. Gou, Zhiqiang Ma, Zhihong Yan, Zhipeng Shao, Zhiyu Xu, Zhuoshu Zhang, Zihui Li, Zijia Gu, Zhu Zijun, Zilin Liu, Ziwei Li, Xie, Aryo Pradipta, Joshua Ong, J. Leang, Gi-won Hong, Alessio Devoto, Alberto Carlo, Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xia Du, Mohammad Reza Ghasemi Madani, C. Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, P. Minervini, Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schel-ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, A. Hartshorn, Aobo Yang, Archi Mitra, A. Sravankumar, A. Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro-driguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, B. Biron, Binh Tang, Bob-bie Chern, Char-lotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, C. Touret, Chunyang Wu, C. Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Al-lonsius, D. Song, D. Pintz, D. Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, E. Lobanova, Emily Dinan, E. Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, G. Lewis, Govind Thattai, G. Nail, Grégoire Mi-alon, Guan Pang, Guillem Cu-curell, Hailey Nguyen, Han-nah Korevaar, Hu Xu, Hugo Touvron, Imanol Iliyan Zarov, Arrieta Ibarra, Is-abel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, J. V. D. Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, J. Spisak, Jongsoo Park, Joseph Rocca, J. Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Keqian Li, Kenneth Heafield, Kevin R. Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, K. Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, L. Malo, Lukas Blecher, Lukas Landzaat, L. Oliveira, Madeline Muzzi, Ma-hesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, M. Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam-badur, Mike Lewis, Mitesh Min Si, Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bash-lykov, Nikolay Bo-goychev, Niladri S. Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Petar Pengwei Li, Peter Weng, Prajjwal Bhargava, P. Dubal, Punit Praveen Krishnan, S. Koura, Puxin Xu, Q. He, Qingxiao Dong, R. Srinivasan, R. Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, R. Raileanu, R. Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ron-nie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, R. Hou, Rui Wang, S. Hosseini, Sa-hana Chennabasappa, Sanjay Singh, Sean Bell, Seo-hyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharath Narang, S. Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Van-denhende, Soumya Batra, Spencer Whit-man, Sten Sootla, S. Collot, Suchin Gururangan, S. Borodinsky, Tamar Herman, T. Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petro-vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Zoe Chen, Aaditya Papakipos, Aayushi Singh, Abha Sri-vastava, A. Jain, A. Kelsey, S. Adithya, Adolfo Gangidi, Ahuva Victoria, G. Ajay, A. Menon, Alex Sharma, Alex Boesenberg, Allie Baevski, Amanda Feinstein, Amit Kallet, Amos San-gani, Anam Teo, Andrei Yunus, A. Lupu, Andrew Alvarado, A. Caples, Andrew Gu, Andrew Ho, Andrew Poulton, A. Ryan, Annie Dong, Annie Franco, Anuj Goyal, Apara-jita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin R. Bharambe, Assaf Eisenman, Azadeh Yaz-dan, Beau James, Ben Maurer, B. Leonhardi, Bernie Huang, Beth Loyd, Beto de Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, B. Ni, Braden Han-cock, Bram Wasti, Brandon Spence, B. Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, C. Tindal, Christoph Fe-ichtenhofer, Cynthia Gao, Damon Civin, D. Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, E. Arcaute, Evan Dunbar, Evan Smoth-ers, Fei Sun, F. Kreuk, Feng Tian, Filippos Kokkinos, Firat Oz-genel, Francesco Caggioni, Frank J. Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grigory Sizov, Guangyi, Guna Zhang, Hakan Lakshmi-narayanan, Hamid Inan, Shojanazeri Han, Han Zou, Hanwen Wang, Haroun Zha, Harrison Habeeb, Helen Rudolph, H. Suk, Hunter As-pegren, Hongyuan Goldman, I. Zhan, Igor Damlaj, Igor Molybog, Ilias Tufanov, Leon-tiadis, Jennifer Tang, J. Chan, J. Zhen, Jeremy Reizen-stein, Jessica Teboul, Jian Zhong, Jingyi Jin, Joe Yang, Jon Cummings, J. Carvill, S. Jonathan, J. McPhie, J. Torres, Ginsburg Junjie, Kai Wang, K. Wu, Hou Karan, Kartikay Sax-ena, Katayoun Khandelwal, Kathy Zand
Venue: Conference on Empirical Methods in Natural Language Processing (2025)

## Abstract
The development of Large Language Models (LLMs) requires robust benchmarks that encompass not only academic domains but also industrial fields to effectively evaluate their applicability in real-world scenarios. In this paper, we introduce two Korean expert-level benchmarks. KMMLU-Redux, reconstructed from the existing KMMLU, consists of questions from the Korean National Technical Qualification exams, with critical errors removed to enhance reliability. KMMLU-Pro is based on Korean National Professional Licensure exams to reflect professional knowledge in Korea. Our experiments demonstrate that these benchmarks comprehensively represent industrial knowledge in Korea. We release our dataset publicly available.

## Links
- arXiv: https://arxiv.org/abs/2507.08924
- PDF: https://arxiv.org/pdf/2507.08924
- HTML: https://arxiv.org/html/2507.08924
- Hugging Face: https://huggingface.co/papers/2507.08924
- alphaXiv: https://www.alphaxiv.org/abs/2507.08924
- DOI: https://doi.org/10.18653/v1/2025.findings-emnlp.1038
- Semantic Scholar: https://www.semanticscholar.org/paper/280150318