On the Role of Optimization in Double Descent: A Least Squares Study

Empirically it has been observed that the performance of deep neural networks steadily improves as we increase model size, contradicting the classical view on overfitting and generalization. Recently, the double descent phenomena has been proposed to reconcile this observation with theory, suggesting that the test error has a second descent when the model becomes sufficiently overparameterized, as the model size itself acts as an implicit regularizer. In this paper we add to the growing body of work in this space, providing a careful study of learning dynamics as a function of model size for the least squares scenario. We show an excess risk bound for the gradient descent solution of the least squares objective. The bound depends on the smallest non-zero eigenvalue of the covariance matrix of the input features, via a functional form that has the double descent behavior. This gives a new perspective on the double descent curves reported in the literature. Our analysis of the excess risk allows to decouple the effect of optimization and generalization error. In particular, we find that in case of noiseless regression, double descent is explained solely by optimization-related quantities, which was missed in studies focusing on the Moore-Penrose pseudoinverse solution. We believe that our derivation provides an alternative view compared to existing work, shedding some light on a possible cause of this phenomena, at least in the considered least squares setting. We empirically explore if our predictions hold for neural networks, in particular whether the covariance of intermediary hidden activations has a similar behavior as the one predicted by our derivations

Hierarchically Clustered Adaptive Quantization CMAC and Its Learning Convergence

No abstract availabl

Superior Information, Income Shocks and the Permanent Income Hypothesis

According to the permanent income hypothesis with quadratic preferences, savings should react only to transitory income shocks, but not to permanent shocks. The problem is that income shock components are not separately observable. I show how the combination of income realizations with subjective expectations can help to identify separately the transitory and the permanent shock to income, thus providing a powerful test of the theory. The empirical analysis is performed on a sample of Italian households drawn from the 1989-1991 Survey of Household Income and Wealth.Subjective expectations, income shocks, permanent income hypothesis

Dynamic Spatial Autoregressive Models with Autoregressive and Heteroskedastic Disturbances

We propose a new class of models specifically tailored for spatio-temporal data analysis. To this end, we generalize the spatial autoregressive model with autoregressive and heteroskedastic disturbances, i.e. SARAR(1,1), by exploiting the recent advancements in Score Driven (SD) models typically used in time series econometrics. In particular, we allow for time-varying spatial autoregressive coefficients as well as time-varying regressor coefficients and cross-sectional standard deviations. We report an extensive Monte Carlo simulation study in order to investigate the finite sample properties of the Maximum Likelihood estimator for the new class of models as well as its flexibility in explaining several dynamic spatial dependence processes. The new proposed class of models are found to be economically preferred by rational investors through an application in portfolio optimization.Comment: 33 pages, 5 figure

Currency risk and imperfect knowledge: Cointegrated VAR analyses with survey data

Much progress has been made in understanding excess returns in the foreign exchange market through the use of survey data on traders\u27 exchange rate forecasts. On the whole, this literature, which is reviewed in chapter 1, has found that excess returns derive from both violations of the rational expectations hypothesis (non white-noise forecast errors) as well as a time-varying risk premium. What this literature has not done however is to determine whether any of the existing models of the risk premium can account for the time-varying risk premium found in survey data. The second and third chapters use the Cointegrated VAR model to test the Capital Asset Pricing Model (CAPM), the Consumption CAPM, and the Keynes-Imperfect Knowledge Economics (IKE) gap model, which relate the risk premium to the exchange rate\u27s variance, covariance with consumption, and deviation from Purchasing Power Parity respectively. The strongest support is found for the Keynes-IKE gap model. The analysis of this model is then extended in chapter 4 to the I(2) CVAR framework, which is a unique empirical approach designed to account for data which undergoes persistent changes over time without the need for data transformations which cause a loss of information. The I(2) model also allows for more rigorous testing of the theory and a better examination of the dynamics between the exchange rate, expectations, prices, and interest rates. The Keynes-IKE gap model still performs quite well. Further, persistent changes are found for the real exchange rate in several instances, which is problematic for standard REH theory but fully compatible with the IKE theory

Information Theory for Complex Systems Scientists

In the 21st century, many of the crucial scientific and technical issues facing humanity can be understood as problems associated with understanding, modelling, and ultimately controlling complex systems: systems comprised of a large number of non-trivially interacting components whose collective behaviour can be difficult to predict. Information theory, a branch of mathematics historically associated with questions about encoding and decoding messages, has emerged as something of a lingua franca for those studying complex systems, far exceeding its original narrow domain of communication systems engineering. In the context of complexity science, information theory provides a set of tools which allow researchers to uncover the statistical and effective dependencies between interacting components; relationships between systems and their environment; mereological whole-part relationships; and is sensitive to non-linearities missed by commonly parametric statistical models. In this review, we aim to provide an accessible introduction to the core of modern information theory, aimed specifically at aspiring (and established) complex systems scientists. This includes standard measures, such as Shannon entropy, relative entropy, and mutual information, before building to more advanced topics, including: information dynamics, measures of statistical complexity, information decomposition, and effective network inference. In addition to detailing the formal definitions, in this review we make an effort to discuss how information theory can be interpreted and develop the intuition behind abstract concepts like "entropy," in the hope that this will enable interested readers to understand what information is, and how it is used, at a more fundamental level

Municipal wastewater treatment with pond technology : historical review and future outlook

Facing an unprecedented population growth, it is difficult to overstress the assets for wastewater treatment of waste stabilization ponds (WSPs), i.e. high removal efficiency, simplicity, and low cost, which have been recognized by numerous scientists and operators. However, stricter discharge standards, changes in wastewater compounds, high emissions of greenhouse gases, and elevated land prices have led to their replacements in many places. This review aims at delivering a comprehensive overview of the historical development and current state of WSPs, and providing further insights to deal with their limitations in the future. The 21st century is witnessing changes in the way of approaching conventional problems in pond technology, in which WSPs should no longer be considered as a low treatment technology. Advanced models and technologies have been integrated for better design, control, and management. The roles of algae, which have been crucial as solar-powered aeration, will continue being a key solution. Yet, the separation of suspended algae to avoid deterioration of the effluent remains a major challenge in WSPs while in the case of high algal rate pond, further research is needed to maximize algal growth yield, select proper strains, and optimize harvesting methods to put algal biomass production in practice. Significant gaps need to be filled in understanding mechanisms of greenhouse gas emission, climate change mitigation, pond ecosystem services, and the fate and toxicity of emerging contaminants. From these insights, adaptation strategies are developed to deal with new opportunities and future challenges

Computational mechanisms underpinning greater exploratory behaviour in excess weight relative to healthy weight adolescents

Obesity in adolescence is associated with cognitive changes that lead to difficulties in shifting unhealthy habits in favour of alternative healthy behaviours, similar to addictive behaviours. An outstanding question is whether this shift in goal-directed behaviour is driven by over-exploitation or over-exploration of rewarding outcomes. Here, we addressed this question by comparing explore/exploit behaviour on the Iowa Gambling Task in 43 adolescents with excess weight against 38 adolescents with healthy weight. We computationally modelled both exploitation behaviour (e.g., reinforcement sensitivity and inverse decay parameters), and explorative behaviour (e.g., maximum directed exploration value). We found that overall, adolescents with excess weight displayed more behavioural exploration than their healthy-weight counterparts – specifically, demonstrating greater overall switching behaviour. Computational models revealed that this behaviour was driven by a higher maximum directed exploration value in the excess-weight group (U = 520.00, p = .005, BF10 = 5.11). Importantly, however, we found substantial evidence that groups did not differ in reinforcement sensitivity (U = 867.00, p = .641, BF10 = 0.30). Overall, our study demonstrates a preference for exploratory behaviour in adolescents with excess weight, independent of sensitivity to reward. This pattern could potentially underpin an intrinsic desire to explore energy-dense unhealthy foods – an as-yet untapped mechanism that could be targeted in future treatments of obesity in adolescents.Junta de AndaluciaNational Health and Medical Research Council (NHMRC) of Australia GNT200946