Research in agents, scientific discovery, and language models.
* denotes equal contribution.
arXiv, Under Review
TL;DR Matching feature and codebook distributions provides a unified approach to reducing vector-quantization instability and codebook collapse, with Wasserstein and maximum mean discrepancy objectives supported by theoretical analysis and visual-tokenization experiments.
arXiv, Under Review
TL;DR This survey organizes AI research automation by workflow stages and human oversight, proposing evaluation dimensions that expose limitations in scientific validity, reproducibility, and evidence provenance across domains.
arXiv, Under Review
TL;DR Medical AI Scientist uses clinician–engineer co-reasoning to turn medical literature into evidence-grounded research ideas, executable experiments, and manuscripts, supporting clinical research workflows with different levels of autonomy.
@misc{2603.285892026,
title = {Towards a Medical AI Scientist},
author = {Hongtao Wu and Boyun Zheng and Dingjie Song and Yu Jiang and Jianfeng Gao and Lei Xing and Lichao Sun and Yixuan Yuan},
year = {2026},
eprint = {2603.28589},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2603.28589}
}arXiv, Under Review
TL;DR CML-Bench evaluates generated movie scripts through dialogue coherence, character consistency, and plot plausibility, and pairs this evaluation with targeted prompting instructions that improve script quality in human-aligned assessments.
arXiv, Under Review
TL;DR Agentic Robot coordinates a reasoning planner, vision-language-action executor, and temporal verifier through a standardized workflow, enabling closed-loop verification and error recovery for more reliable long-horizon robotic manipulation.
arXiv, Under Review
TL;DR This survey reviews methods for aligning multimodal LLMs with human preferences across image, video, and audio tasks, organizing alignment algorithms, preference dataset construction, evaluation benchmarks, and future research directions.
arXiv, Under Review
TL;DR This survey organizes LLM post-training methods and datasets around fine-tuning, alignment, reasoning, efficiency, and integration, mapping how these approaches extend pretrained models and identifying open challenges for further research.
arXiv, Under Review
TL;DR MathCCS benchmarks student error diagnosis and personalized feedback, while a collaborative agent framework combines learning history with multimodal analysis to improve error classification and generate more useful suggestions.
arXiv, Under Review
TL;DR BlenderLLM improves CAD script generation through instruction tuning and iterative self-improvement, supported by the BlendNet training dataset and CADBench evaluation suite for assessing script functionality and accuracy.
EMNLP 2026
TL;DR OpenSkill turns documentation, repositories, and web resources into reusable skills and self-built practice tasks, enabling agents to improve without target-task supervision and transfer learned skills across models.
@misc{2606.067412026,
title = {OpenSkill: Open-World Self-Evolution for LLM Agents},
author = {Zhiling Yan and Dingjie Song and Hanrong Zhang and Wei Liang and Yuxuan Zhang and Yutong Dai and Lifang He and Philip S. Yu and Ran Xu and Xiang Li and Lichao Sun},
year = {2026},
eprint = {2606.06741},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2606.06741}
}EMNLP 2026, System Demonstrations Track
TL;DR Dr. Claw coordinates coding agents through persistent project state and reusable skills, connecting research planning, execution, and writing in a human-guided workflow that preserves decisions and supports recovery.
@misc{drclaw2026,
title = {Dr. Claw: An AI Scientist Workspace for Vibe Research},
author = {Dingjie Song and Hanrong Zhang and Dawei Liu and Yixin Liu and Zongxia Li and Zhengqing Yuan and Siqi Zhang and Henry Peng Zou and Zhiling Yan and Yuxuan Zhang and Yanfang Ye and Philip S. Yu and Lichao Sun},
year = {2026},
eprint = {2609.00365},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2609.00365}
}EMNLP Findings 2026
TL;DR ClawBench evaluates agents on everyday workflows across live websites, blocking final submissions to prevent real-world side effects and revealing substantial gaps in completing practical online tasks.
AIED 2026 Oral
TL;DR ScratchMath evaluates how well multimodal LLMs explain and classify errors in authentic handwritten mathematics, revealing substantial gaps from human experts in recognizing student work and reasoning about mistakes.
KDD 2026, Datasets and Benchmarks Track
TL;DR LiveMedBench combines newly collected clinical cases with automated, case-specific evaluation rubrics to reduce contamination risks, revealing that LLMs struggle to adapt medical knowledge to individual patient contexts.
EMNLP Findings 2025
TL;DR LongLLaVA combines Mamba and Transformer blocks with progressive multimodal training to handle long image sequences efficiently, reducing memory demands while retaining competitive performance on multimodal benchmarks.
@inproceedings{wang-etal-2025-longllava,
title = "{L}ong{LL}a{VA}: Scaling Multi-modal {LLM}s to 1000 Images Efficiently via a Hybrid Architecture",
author = "Wang, Xidong and
Song, Dingjie and
Chen, Shunian and
Chen, Junying and
Cai, Zhenyang and
Zhang, Chen and
Sun, Lichao and
Wang, Benyou",
editor = "Christodoulopoulos, Christos and
Chakraborty, Tanmoy and
Rose, Carolyn and
Peng, Violet",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-emnlp.1168/",
doi = "10.18653/v1/2025.findings-emnlp.1168",
pages = "21419--21436",
ISBN = "979-8-89176-335-7"
}EMNLP Findings 2025, ICML 2025 DIG-BUG Workshop Oral
TL;DR MM-Detect distinguishes unimodal and cross-modal benchmark contamination, showing that inflated multimodal evaluation scores can stem from unimodal pretraining as well as later multimodal training stages.
@inproceedings{song-etal-2025-text,
title = "Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal {LLM}",
author = "Song, Dingjie and
Lai, Sicheng and
Wang, Mingxuan and
Chen, Shunian and
Sun, Lichao and
Wang, Benyou",
editor = "Christodoulopoulos, Christos and
Chakraborty, Tanmoy and
Rose, Carolyn and
Peng, Violet",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-emnlp.556/",
doi = "10.18653/v1/2025.findings-emnlp.556",
pages = "10527--10542",
ISBN = "979-8-89176-335-7"
}MICCAI 2025
TL;DR SAMed-2 adapts SAM-2 for medical segmentation with a temporal adapter and confidence-based memory, improving learning across imaging tasks while reducing the effects of noisy annotations and catastrophic forgetting.
ACL 2025
TL;DR Med-MAT organizes medical imaging datasets by modality, anatomy, and task, showing that multimodal LLMs can generalize to unseen combinations and that this ability helps explain the benefits of multi-task training.
NAACL 2025
TL;DR MLLM-Bench evaluates open-ended multimodal responses through pairwise comparisons using sample-specific criteria and a multimodal judge, with evaluations showing strong agreement with human judgments across diverse cognitive tasks.
COLING 2025
TL;DR TRIM uses a CLIP-based metric to select and reduce image tokens, lowering the computational cost of multimodal language models while largely preserving performance across visual question-answering benchmarks.
@inproceedings{song-etal-2025-less,
title = "Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal {LLM}s",
author = "Song, Dingjie and
Wang, Wenjun and
Chen, Shunian and
Wang, Xidong and
Guan, Michael X. and
Wang, Benyou",
editor = "Rambow, Owen and
Wanner, Leo and
Apidianaki, Marianna and
Al-Khalifa, Hend and
Eugenio, Barbara Di and
Schockaert, Steven",
booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
month = jan,
year = "2025",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.coling-main.508/",
pages = "7614--7623"
}COLM 2024
TL;DR MileBench combines diagnostic and realistic tasks to test long-context, multi-image understanding, revealing that most evaluated open models struggle increasingly as the number of images grows.
@inproceedings{song2024milebench,
title = {MileBench: Benchmarking MLLMs in Long Context},
author = {Dingjie Song and Shunian Chen and Guiming Hardy Chen and Fei Yu and Xiang Wan and Benyou Wang},
booktitle = {First Conference on Language Modeling},
year = {2024},
url = {https://openreview.net/forum?id=Uhwze2LEwq}
}COLM 2024
TL;DR HuatuoGPT-II unifies medical pretraining and instruction data into input-output pairs for one-stage adaptation, improving Chinese medical question answering without a separate pretraining and fine-tuning pipeline.
NAACL 2024
TL;DR AceGPT adapts language models to Arabic through continued pretraining, instruction tuning, and reinforcement learning with culturally informed AI feedback, addressing Arabic language use and local values.
NAACL 2024
TL;DR CMB evaluates medical language models using Chinese professional examinations and clinical case questions, covering local medical practice, including traditional Chinese medicine, rather than relying on translated English benchmarks.
NLPCC 2023 Oral
TL;DR Episode-based Prompt Learning recasts intent detection as prompted sentence-pair classification and simulates varying label availability during training, supporting both unseen intents and intents with only a few labeled examples.
JCST 2022
TL;DR STAM combines task augmentation with self-supervised auxiliary objectives to reduce meta-learning overfitting when training tasks are scarce, improving generalization to new intents with few labeled examples.
