Plenary Speeches

Plenary Speech II

On Reusing Model Parameters

Professor Dacheng Tao
Nanyang Technological University

The rapid rise of large language models and foundation models has fundamentally reshaped the paradigm of deep learning. Beyond conventional end-to-end training, an increasingly important question is how to effectively reuse existing model parameters to improve performance, scalability, and data efficiency. Parameter reuse, through model fusion, knowledge transfer, model editing, and adaptation with limited data, offers a principled alternative to training ever-larger models from scratch.

In this talk, we revisit recent advances in deep model fusion from the broader perspective of parameter reuse. We present a systematic taxonomy of existing approaches and analyze their underlying mechanisms, scalability, and theoretical implications. In particular, we introduce our recent developments, including (1) weight learning based model fusion and data-adaptive MoE upscaling, (2) subspace learning approaches that exploit structured parameter geometry, and (3) enhanced multi-task fusion strategies that integrate pre- and post-finetuning to reduce representation bias between merged and task-specific models.

By framing model fusion as a principled study of parameter reuse, we highlight both its practical advantages, improved efficiency, robustness, and reduced reliance on annotated data, and the open challenges that remain for large-scale foundation models.

Biosketch

Dacheng Tao is currently a Distinguished University Professor and the Inaugural Director of the Generative AI Lab in the College of Computing and Data Science at Nanyang Technological University. He was an Australian Laureate Fellow and the founding director of the Sydney AI Centre at the University of Sydney, the inaugural director of JD Explore Academy and senior vice president at JD.com, and the chief AI scientist at UBTECH Robotics. He mainly applies statistics and mathematics to artificial intelligence, and his research is detailed in one monograph and over 300 publications. His publications have been cited over 140K times and he has an h-index 180+ in Google Scholar. He received the 2015 and 2020 Australian Eureka Prize, the 2018 IEEE ICDM Research Contributions Award, 2020 research super star by The Australian, the 2019 Diploma of The Polish Neural Network Society, and the 2021 IEEE Computer Society McCluskey Technical Achievement Award. He is a Fellow of the Australian Academy of Science, ACM and IEEE.