Authors: Jamie Simon , Daniel Kunin , Alexander Atanasov , Enric Boix-Adserà , Blake Bordelon , Jeremy Cohen , Nikhil Ghosh , Florentin Guth , Arthur Jacot , Mason Kamb , Dhruva Karkada , Eric J. Michaud , Berkan Ottlik , Joseph Turnbull View PDF HTML (experimental) Abstract: In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the…