Show Menu

Events

Nasim Rahaman (MPI-IS Tübingen, and Mila, Montréal): On the Spectral Bias of Neural Networks

Ort: MPI für Mathematik in den Naturwissenschaften Leipzig,  , Videobroadcast

Video broadcast: Math Machine Learning seminar MPI MIS + UCLA In this talk, I will introduce a phenomenon called "the spectral bias”, which shows that even though neural networks are quite capable of fitting random target functions with perfect performance, they are biased towards learning the “simpler” components of the target function earlier in the training. Precisely, we inspect the learning process through the lens of Fourier analysis to find that the lower frequencies of the target function are learned first, even when they are underrepresented in the spectrum of the latter. This observation leads to a clearer picture of the kinds of label noise neural networks are particularly vulnerable against, and to why early stopping is as effective as it is. Finally, we also look at the curious interplay between spectral bias and the shape of the data-manifold, which might explain the unreasonable effectiveness of positional embeddings (also known as Fourier features) in the implicit representation learning literature (i.e. Neural Radiance Fields (NeRF) and friends). <br /><br /><p><b>One day before the seminar, an announcement with the Zoom link will be sent to those who registered with Valeria Hünniger. For this please contact her at <a href="mailto:vah@mis.mpg.de">Valeria Hünniger</a>.</b></p>

No Attachment

Beginn: July 10, 2020, 5 p.m.

Ende: July 10, 2020, 6:30 p.m.