Description

This topics course intends to explore scaling limits of neural networks. In particular, we will study the limits as the number of parameters in a neural network approaches infinity in different regimes, where the limiting object is intended to be used as a model for finite size neural networks in practice. Beyond convergence, we will also study the properties and the theoretical properties of the limiting objects, and their implications on practice.

Topics potentially covered in this course include: Neural network Gaussian processes, neural tangent kernel, limiting spectrum, double descent, mean-field neural network PDE, feature learning, dynamical mean field theory, single index models, neural ODE, infinite-depth ResNets, neural covariance SDE, and other extensions.

Background in probability, differential equations, and stochastic analysis are desirable but not required.

Tentative List of Topics

• Overview of deep learning theory and open problems
• Overview of scaling limits in probability theory
• Neural network Gaussian processes
• Neural tangent kernel
• Generalization in the kernel regime and double descent
• Feature learning scaling
• Mean field PDE
• Dynamical mean field theory (DMFT)
• Application to single index models
• Differential equations and infinite depth limits, Neural ODEs
• ODE limits for ResNets
• SDE limits for shaped MLPs
• Recent progress on training dynamics and limiting spectrum

Scribed Lecture Notes

• Lecture 1
• Lecture 2
• Lecture 3
• Lecture 4