Leiden University Student Repository

Chalkias, Christos (2025)

Statistical mechanics of Deep Learning: A Perturbative solution of a fully connected network and the role of non-linear activation functions

Master thesis | Physics (MSc)

Physics has proved to be an inspiration for understanding the inner workings of neural networks, undeniably the most indispensable tool driving the ongoing AI revolution. Ideas from statistical physics and quantum field theory provide excellent candidates for the building blocks of a theory of deep learning. This master thesis tries and shed light on the role of non-linear activation functions by using perturbation theory to analyze the partition function of neural networks. For brevity a shallow, 2-layer constant width architecture is studied. Easily interpretable results for restrictive cases of single arithmetic examples show an increase in temperature resistant expressivity of the network. This enables the learning algorithm to train such networks more efficiently.

Author

Chalkias, Christos

Faculty

Faculty of Science

Specialisation

Classical/Quantum Information

Supervisors

Schalm, Koenraad

ECTS Credits

24

Language

en