Article Source

• Scaling Limits of Neural Networks

Scaling Limits of Neural Networks

• Big Data Conference 2024, 9/7/2024
• Speaker: Boris Hanin, Princeton University
• Title: Scaling Limits of Neural Networks

ABSTRACT

Neural networks are often studied analytically through scaling limits: regimes in which taking some structural network parameters (e.g. depth, width, number of training datapoints, and so on) to infinity results in simplified models of learning. I will motivative and discuss recent results using several such approaches. I will emphasize both new theoretical insights into how model, training data, and optimizer impact learning and their practical implications for hyperparameter transfer.