Dingjie Song
Welcome! I am a Ph.D. student in the Department of Computer Science and Engineering at Lehigh University, advised by Prof. Lichao Sun. Previously, I was a research assistant with the CUHK-Shenzhen NLP group, mentored by Dr. Benyou Wang. I obtained my M.E. from the Software Institute and Natural Language Processing Group at Nanjing University, under the guidance of Dr. Xinyu Dai and Dr. Jidong Ge. Before that, I completed my B.E. at the Software Institute of Nanjing University.
Email: dingjiesong.cs@gmail.com
Links: Research Overview / Updates / Awards / Papers
Research Overview
My research interests are in Natural Language Processing, especially intelligent interactive systems 🤖 and Domain-specific LLMs 👨🏻⚕️ and the following directions:
- Multimodal LLM [COLM 2024], [COLING 2025], [NAACL 2025], [LongLLaVA], [MM-Detect]
- Medical LLM: [NAACL 2024], [COLM 2024]
- Multilingual LLM: [NAACL 2024]
- Task-oriented dialogue systems: [NLPCC 2023 Oral], [JCST 2023]
Updates
Jan 2025: 🎉🎉 MLLM-Bench was accepted to NAACL’25 main conference!
Dec 2024: 🎉🎉 TRIM was accepted to COLING’25 main conference!
Nov 2024: MM-Detect 🕵️ released! MM-Detect is the first Data Contamination Detection Framework for MLLMs! More information can be found in 📃 paper and the GitHub.
Sep 2024: TRIM ✂️ released! TRIM is a simple yet effective Image Token Reduction Method for efficient MLLMs! More information can be found in 📃 paper, 🤗 HuggingFace and the GitHub.
Sep 2024: LongLLaVA 🐍🦙 released! LongLLaVA is the first MLLM with hybrid architecture that can handle up to 1000 images! More information can be found in 📃 paper, 🤗 HuggingFace and the GitHub. 🥈#2 Paper of the day on Huggingface Daily Paper.
July 2024: 🎉🎉 Two papers MileBench and HuatuoGPT2 were accepted to COLM’24 main conference!
April 2024: MileBench 🛣️ released! MileBench is a pioneering benchmark designed to rigorously test the MultImodal Long-contExt capabilities of MLLMs. More information can be found on the 🌐 website, 📃 paper, 🤗 HuggingFace and the GitHub.
March 2024: 🎉🎉 Two papers CMB and AceGPT were accepted to NAACL’24 main conference!
Before 2024
Nov 2023: HuatuoGPT2 released! Try it out on the 🌐 demo! HuatuoGPT2 employs an innovative domain adaptation method to significantly boost its medical knowledge and dialogue proficiency and showcases SOTA performance in several medical benchmarks, especially surpassing GPT-4 in expert evaluations and the fresh medical licensing exams. More info can be found in 📃 paper and 🤗 HuggingFace.
Sep 2023: We publish AceGPT that achieved top performance among open-source Arabic language models in benchmark tests. More info can be found in 📃 paper and 🤗 HuggingFace.
Aug 2023: Checkout our 📃 new paper that focuses on benchmarking prevalent Medical LLMs for their medical knowledge and clinical diagnostic capabilities. More information can be found on the 🌐 website and the 🤗 HuggingFace.
Jul 2023: Start the journey in CUHK-sz as a research assistant under the guidance of Benyou Wang.
Jun 2023: I defended my master's degree and got my master's degree in software engineering. Thanks to all those who have supported me.
Aug 2022 - Apr 2023: Finished my internship with Jiaxing Zhang on LLM SFT.
Papers
Preprints
Aligning Multimodal LLM with Human Preference: A Survey
Tao Yu, Yi-Fan Zhang, Chaoyou Fu, Junkang Wu, Jinda Lu, Kun Wang, Xingyu Lu, Yunhang Shen, Guibin Zhang, Dingjie Song, Yibo Yan, Tianlong Xu, Qingsong Wen, Zhang Zhang, Yan Huang, Liang Wang, Tieniu Tan
arXiv preprint arXiv:2503.14504, 2025/3/18, project pageA Survey on Post-training of Large Language Models
Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, Zhenhan Dai, Yifeng Xie, Yihan Cao, Lichao Sun, Pan Zhou, Lifang He, Hechang Chen, Yu Zhang, Qingsong Wen, Tianming Liu, Neil Zhenqiang Gong, Jiliang Tang, Caiming Xiong, Heng Ji, Philip S. Yu, Jianfeng Gao
PreprintFrom Correctness to Comprehension: AI Agents for Personalized Error Diagnosis in Education
Yi-Fan Zhang, Hang Li, Dingjie Song, Lichao Sun, Tianlong Xu, Qingsong Wen
PreprintOn the Compositional Generalization of Multimodal LLMs for Medical Imaging
Zhenyang Cai, Junying Chen, Rongsheng Wang, Weihong Wang, Yonglin Deng, Dingjie Song, Yize Chen, Zixu Zhang, Benyou Wang
arxiv, Under Review, project pageBlenderLLM: Training Large Language Models for Computer-Aided Design with Self-improvement
Yuhao Du, Shunian Chen, Wenbo Zan, Peizhao Li, Mingxuan Wang, Dingjie Song, Bo Li, Yan Hu, Benyou Wang
arxiv, Under Review, project pageBoth Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination
Dingjie Song*, Sicheng Lai*, Shunian Chen, Lichao Sun, Benyou Wang
arxiv, Under Review, project pageLongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via Hybrid Architecture
Xidong Wang*, Dingjie Song*, Shunian Chen, Chen Zhang, Benyou Wang
arxiv, Under Review, project page, code and data
2025
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
Wentao Ge*, Shunian Chen*, Guiming Hardy Chen*, Junying Chen, Zhihong Chen, Nuo Chen, Wenya Xie, Shuo Yan, Chenghao Zhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
NAACL 2025, project pageLess is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs
Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, Benyou Wang
COLING 2025, code and model
2024
MileBench: Benchmarking MLLMs in Long Context
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, Benyou Wang
COLM 2024, project page, code and dataHuatuoGPT-II, One-stage Training for Medical Adaption of LLMs
Junying Chen, Xidong Wang, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xiang Wan, Haizhou Li, Benyou Wang
COLM 2024, project page, code and dataAceGPT, Localizing Large Language Models in Arabic
Huang Huang*, Fei Yu*, Jianqing Zhu*, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, Jinchao Xu
NAACL 2024, code and dataCMB: A Comprehensive Medical Benchmark in Chinese
Xidong Wang*, Guiming Hardy Chen*, Dingjie Song*, Zhiyi Zhang*, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, Haizhou Li
NAACL 2024, project page, code and data
2023
- Episode-based Prompt Learning for Any-shot Intent Detection
Pengfei Sun*, Dingjie Song*, Yawen Ouyang, Zhen Wu, Xinyu Dai
NLPCC 2023 Oral
2022
- Self-Supervised Task Augmentation for Few-Shot Intent Detection
Pengfei Sun, Yawen Ouyang, Dingjie Song, Xinyu Dai
JCST 2022, code and data
Awards
- Outstanding Graduate Student, Nanjing University, 2022
- Yingcai Scholarship, Nanjing University, 2022
- Renmin Scholarship (People’s Scholarship), Nanjing University, 2018-2021
- Third Runner’s Up in 15th Citi Cup Financial Innovation Application Competition, Citigroup, 2019
- Second Runner’s Up in 2019 “Chain to Future” University Blockchain Technology Application Competition, CCF, 2019
- Outstanding Student Leader of the Communist Youth League, Nanjing University, 2018-2019
Services
- Conference reviewer: EMNLP, ACL Rolling Review
