Relating Regularization and Generalization through the Intrinsic Dimension of Activations

Given a pair of models with similar training set performance, it is natural to assume that the model that possesses simpler internal representations would exhibit better generalization. In this work, we provide empirical evidence for this intuition through an analysis of the intrinsic dimension (ID) of model activations, which can be thought of as the minimal number of factors of variation in the model's representation of the data. First, we show that common regularization techniques uniformly decrease the last-layer ID (LLID) of validation set activations for image classification models and show how this strongly affects generalization performance. We also investigate how excessive regularization decreases a model's ability to extract features from data in earlier layers, leading to a negative effect on validation accuracy even while LLID continues to decrease and training accuracy remains near-perfect. Finally, we examine the LLID over the course of training of models that exhibit grokking. We observe that well after training accuracy saturates, when models ``grok'' and validation accuracy suddenly improves from random to perfect, there is a co-occurent sudden drop in LLID, thus providing more insight into the dynamics of sudden generalization.Comment: NeurIPS 2022 OPT and HITY workshop

Going Green: Cost and GHG Emissions in the Canadian Electricity Sector

<p>Electricity generation in Canada contributes significantly to national carbon emissions. In order to meet global greenhouse gas (GHG) emissions targets, Canada must pivot away from generating electricity from fossil fuels such as coal and natural gas and move towards electricity sources with smaller carbon footprints, such as wind, solar, and hydroelectricity. However, because renewable energy technology grows progressively more cost-effective with time, there is an economic incentive to withhold investment in the short term.</p><br><p>This study determines the optimal time and type of energy to invest in to keep costs at a minimum while still meeting worldwide emissions targets. Our group established an emissions ceiling (the maximum amount of carbon dioxide one can release into the atmosphere while adhering to emissions targets) for the Canadian electricity sector by proportionally scaling down the worldwide GHG cap. Using data from OpenEI, our group used feedforward neural networks to predict the levelized cost of energy (LCOE) for Canada's three largest renewable sources of electricity (wind, solar, and hydroelectricity) and its two biggest fossil fuel sources (coal and natural gas) over time. Canada's future demand for electricity was predicted using the same technique, and with data provided by StatsCan. Our group developed an ideal investment plan by meeting each year's increasing electricity demand with the least expensive generation method of that year. When the emissions ceiling was reached, electricity from fossil fuels would be completely phased out, making the Canadian electricity sector completely carbon-free. </p><br><p>Results demonstrated that wind energy is currently the most cost-effective method of generating electricity (figure 2), and should be invested in immediately to meet growing demand. We predicted that the electricity sector will reach its emissions ceiling by 2033 if one followed the ideal investment plan outlined above. When the ceiling is reached, solar, hydroelectric and wind-based electricity generation will all be more cost-efficient than either coal or natural gas, providing multiple viable replacements for fossil fuels. </p