大语言模型理论研究进展与趋势
刘勇, 胡啸林, 唐鹏威, 龚子瑄

• 三大理论维度：聚焦于表达能力、优化理论、泛化理论三大理论维度，其共同决定模型的性能与稳定性。
• 理论指导实践：扩展法则、数据配比、参数高效微调和提示工程等，为预训练、微调与部署提供理论依据。
• 涌现能力机理：情境学习（ICL）和思维链（CoT）等能力可从三大理论维度进行解释，为未来模型设计提供理论支撑。

Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization
Zixuan Gong*, Xiaolin Hu*, Huayi Tang, Yong Liu (* Equal contribution)

• We explore the emergence of in-context learning (ICL) capabilities in auto-regressive next-token prediction models.
• To bridge the pre-training and ICL phases, we introduce a two-level expectation over data and topic distributions, providing PAC-Bayes generalization bounds to support our analysis.
• Additionally, we model the training process using Stochastic Differential Equations (SDEs), demonstrating that ICL arises from the exceptional generalization across sequences and topics.

ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
Pengwei Tang, Xiaolin Hu, Yong Liu

• We propose Adaptive Decomposed Prompt Tuning (ADePT), which can produce unique token embedding offset for each token.
• ADePT addresses the limitations of DePT, enabling better optimization and generalization without increasing inference time or parameters.
• Experiments on 23 NLP tasks and 4 PLMs show ADePT outperforms leading PEFT methods and even full fine-tuning in some cases.

Enhancing In-Context Learning with just SVD-Based Pruning: A Theoretical Perspective
Xinhao Yao, Xiaolin Hu, Shenzhi Yang, Yong Liu

• We show an exciting phenomenon that SVD-based weight pruning can enhance In-Context Learning (ICL) performance.
• we conduct theoretical analysis by presenting the implicit gradient descent (GD) of ICL and giving generalization bounds of ICL.
• We further propose a simple, derivative-free algorithm to enhance ICL. Experiments demonstrate its effectiveness.

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang

• We propose PMSS, enabling high-rank updates at low costs by selecting skeletons from pre-trained weights.
• PMSS overcomes LoRA’s low-rank limitations and optimizes initialization to utilize semantic and linguistic information.
• Experiments show PMSS outperforms LoRA and excels in tasks like DROP and math reasoning with fewer trainable parameters.

Neural Retrievers are Biased Towards LLM-Generated Content
Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, Gang Wang, Jun Xu

• We explore how LLM-generated texts influence IR systems, revealing a source bias where neural models favor LLM-generated documents.
• We use information theory to explain this bias, showing it arises from the focused semantics of LLM-generated content.

Stability and Generalization of Zeroth-Order Decentralized Stochastic Gradient Descent with Changing Topology
Xiaolin Hu, Zixuan Gong, Gengze Xu, Wei Liu, Jian Luan, Bin Wang, Yong Liu (Oral)

• This paper provides the first generalization analysis of ZO-DSGD with changing topology.
• The obtained generalization bounds align with SGD in (strongly) convex cases and with DSGD in non-convex cases.
• The results reflect the impact of client count, sample size, and topology on generalization performance.

Generalization Bounds for Federated Learning: Fast Rates, Unparticipating Clients and Unbounded Losses
Xiaolin Hu, Shaojie Li, Yong Liu

Video

• We present a theoretical analysis of the generalization error for non-participating clients in federated learning.
• The obtained generalization bounds in high probability form capture the performance of a single trial, rather than the average over multiple trials.
• We derive generalization bounds for heavy-tail losses, applicable to federated learning with unbounded losses, such as cross-entropy.

A Deep Learning Framework for Solving Rectangular Waveguide Problems
Xiaolin Hu, Nicholas E. Buri, APMC 2020 (Oral) |

Project

• We employ Physics Informed Neural Networks (PINNs) to solve rectangular waveguide problems.
• We successfully apply PINNs to the task of solving electric and magnetic fields, which can be described by partial differential equations (PDEs).
• We also show the applicability of the framework for predicting the unknown parameters such as wavenumber.