ICML 2024: Paper Review #2

Shikai Qiu, Andres Potapczynski, Marc Finzi, Micah Goldblum, Andrew Gordon Wilson

In this paper, the authors seek to find more compute-efficient alternatives to replace dense linear layers, investigating structured alternatives such as low-rank matrices, monarch matrices and Kronecker products.

The authors claim that these approaches have largely failed in the past due to choosing hyperparameters poorly when these alternatives are used in place of dense linear layers.

To address this, they adapt the initialisation scheme derived from the maximal update parametrization work to support with these structured matrices, and use it to optimise some simple hyperparameters (like learning rate). They show that by doing this, they are able to achieve better test performance per flop on a number of tasks.

Jan E. Gerken and Pan Kessel

In this work, the authors use the theory of neural tangent kernels to prove that ensembles of infinitely wide deep neural networks are equivariant at all stages of training if trained with full data augmentation.

The authors then demonstrate this property empirically for ensembles of wide and deep neural networks applied to several image classification tasks. In these cases, the neural network ensemble becomes equivariant to relevant symmetries in the data even when the underlying members of the ensemble do not display this property.