Skip to main content
We're enhancing our site and your experience, so please keep checking back as we evolve.
Back to News
NeurIPs Paper Reviews 2023 #2

NeurIPs Paper Reviews 2023 #2

23 January 2024
  • Quantitative Research

Our team of quantitative researchers have shared the most interesting research presented during workshops and seminars at NeurIPs 2023.

Discover the perspectives of quantitative researcher Paul, as he discusses his most compelling findings from the conference.

NeurIPs Booth 2022

Sharpness-Aware Minimization Leads to Low-Rank Features

Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, Nicolas Flammarion

In overparametrised neural networks, sharpness of minima has been observed to correlate negatively with the generalisation error of the model. Sharpness-aware minimisation (SAM) is a recent algorithm that introduces an explicit sharpness penalty to the optimisation objective which has been shown to improve model performance.

In this paper, the authors investigate the effect that SAM has on the features of the model. They demonstrate that SAM reduces the feature rank at different layers, as measured by the number of principal components that are needed to capture 99% of the variance, compared to networks that are trained using standard minimisation algorithms. This can for instance be used to reduce the dimensionality of the feature space, improving the performance of downstream tasks. In contrast, the authors found that directly imposing a lower feature rank on the model itself did not lead to improved generalisation. This suggests that the low rank is a useful side effect but not a full explanation of the benefits of SAM.

To further understand the mechanism behind this effect, the authors study a two-layer ReLU network. They show, both experimentally and theoretically, that SAM decreases pre-activation values within the network. This, in turn, reduces the number of non-zero activations and results in the observed low rank of the features.

Sharpness-Aware Minimization Leads to Low-Rank Features
NeurIPS 2022 Paper Reviews

Read paper reviews from NeurIPS 2022 from a number of our quantitative researchers and machine learning practitioners.

Read now

When Do Neural Nets Outperform Boosted Trees on Tabular Data?

Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, Colin White

This paper presents a comprehensive study comparing the performance of neural network (NN), gradient boosted decision trees (GBDT), and baseline algorithms like linear or k-nearest neighbour models on a large number of tabular datasets. It also introduces a benchmark suite of challenging tabular datasets to accelerate research in this area.

The study shows that no single algorithm dominates on all datasets, nearly all algorithms examined ranked first on at least one dataset. When aggregating the algorithms by their respective family, GBDTs are high-performing on slightly more datasets than NNs and baseline methods while also being faster than NNs. However, in many cases the difference in performance between NNs and GBDTs is either negligible or, at fixed budget, tuning the hyperparameters of GBDT is more useful than trying out different methods.

Additionally, the authors present a metafeature analysis to identify dataset properties that correlate with superior performance of certain techniques, which is helpful for practitioners selecting suitable methods for their respective datasets. For example, the authors demonstrate that GBDTs tend to outperform NNs on datasets with heavy-tailed, skewed, or high-variance features.

Quantitative Research and Machine Learning

Want to learn more about life as a researcher at G-Research?

Learn more

Read more of our quantitative researchers thoughts

NeurIPs Paper Reviews 2023 #1

Discover the perspectives of Danny, one of our machine learning engineers, on the following papers:

  • A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning
  • Normalization Layers Are All That Sharpness-Aware Minimization Needs
Paper Review #1
NeurIPS Paper Review 2023 #3

Discover the perspectives of Szymon, one of our quantitative researchers, on the following papers:

  • Convolutional State Space Models for Long-Range Spatiotemporal Modeling
  • How to Scale Your EMA
Paper Review #3
NeurIPS Paper Review 2023 #4

Discover the perspectives of Dustin, our scientific director, on the following papers:

  • Abide by the law and follow the flow: conservation laws for gradient flows
  • The Tunnel Effect: Building Data Representations in Deep Neural Networks
Paper Review #4
NeurIPS Paper Review 2023 #5

Discover the perspectives of Laurynas, one of our machine learning engineers, on the following papers:

  • Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
  • QLoRA: Efficient Finetuning of Quantized LLMs
Paper Review #5
NeurIPS Paper Review 2023 #6

Discover the perspectives of Rui, one of our quantitative analyst, on the following papers:

  • Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting
  • Conformal Prediction for Time Series with Modern Hopfield Networks
Paper Review #6

Latest News

Insights from SXSW 2024: Reflections of a G-Research Software Engineer
  • 08 Apr 2024

Here, Michael W, an engineer in our Dallas engineering and infrastructure hub, provides an overview of what he took away from his time at SXSW 2024.

Read article
Insights from SXSW 2024: Reflections of a G-Research Platform Engineer
  • 22 Mar 2024

Here, Robin L, a Big Data Platform engineer in our Dallas engineering and infrastructure hub, provides an overview of what he took away from his time at SXSW 2024.

Read article
An interview with James Ashton
  • 05 Mar 2024

We spoke to James Ashton (financial journalist and author) on Arm’s success, the widespread use of its technology and some thoughts on what’s next for the business.

Read article

Latest Events

  • Quantitative Engineering
  • Quantitative Research

TUM Quant Challenge at the TUM Makeathon

27 Apr 2024 Technical University of Munich
  • Quantitative Engineering
  • Quantitative Research

Tubingen Quant Challenge

25 Apr 2024 Hotel Krone, Uhlandstrasse 1, 72072 Tübingen, Germany
  • Machine Learning
  • Quantitative Research

Spring into Quant Finance 2024

15 Apr 2024 - 19 Apr 2024 Royal-Riviera Hotel, Av. Jean Monnet, 06230 Saint-Jean-Cap-Ferrat, France

Stay up to date with
G-Research