Mathematical optimization in deep learning

Mathematical Optimization plays a pillar role in Machine Learning (ML) and Neural Networks (NN) are amongst the most popular and effective ML architectures and are the subject of a very intense investigation. They have also been proven immensely powerful at solving prediction tasks in areas such as speech recognition, image classification, robotics and quantum physics. In this work we present the problem of training a Deep Neural Network (DNN), specifically the continuous optimization problem arising in Feed-Forward Networks with Rectified Linear Unit (ReLU) activation. Then we will discuss the inverse problem, presenting a model for a trained DNN as a 0-1 Mixed Integer Linear Program (MILP). Some applications, such as feature visualization and the construction of adversarial examples will be outlined. Computational experiments are reported for both direct and inverse problem. The remainder of the text contains the AMPL codes used for solving the posed problems.La optimización matemática juega un papel fundamental en el aprendizaje automático (AA), y las redes neuronales (NN) se encuentran entre las estructuras más populares y efectivas dentro de este campo. Por ello, son objecto de una intensa investigación. Además, han demostrado ser inmensamente potentes resolviendo tareas de predicción en áreas como reconocimiento automático del habla, clasificación de imágenes, robótica y física cuántica. En este trabajo, se presenta el problema de entrenar una red neuronal profunda (DNN), específicamente el problema de optimización continua que surge en las redes neuronales prealimentadas (FNN) con rectificador (ReLU) como función de activación. Posteriormente, se discutirá el problema inverso, presentaremos un modelo para una DNN que ya ha sido entrenada como un problema de programación lineal en enteros mixta. Describiremos algunas aplicaciones, como visualización de características y la construcción de ejemplos maliciosos. Se realizarán los experimentos computacionales para ambos problemas, el directo y el inverso. Los códigos de AMPL para los problemas planteados se encuentran al final del documento.Universidad de Sevilla. Doble Grado en Física y Matemática

Is it Worth Refining Linear Approximations to Non-Linear Rational Expectations Models?

We characterize the balanced growth path of the basic neoclassical growth economy using standard, almost linear numerical solution methods, as well as the parameterized expectations approach, which preserves the nonlinearity in the model. We also apply the same methods after adding indivisible labor to the basic model, and to a monetary version of that economy, subject to a cash-in-advance constraint. In a unified framework we tackle the question of how much of the nonlinear structure of the original problem is useful to maintain when using an “almost” linear method. We show that it is possible to find an almost linear method to solve these models as accurately as by parameterizing expectations. Our results show the importance of performing log-linear approximations, as well as the convenience of refining a linear solution method by mixing some structure of the original non-linear problem with structure of the approximated system.Linear-quadratic approximation, numerical accuracy, simulation,numerical methods.

Discretionary policy in a monetary union with sovereign debt

This paper examines the interactions between multiple national fiscal policymakers and a single monetary policy maker in response to shocks to government debt in some or all of the countries of a monetary union. We assume that national governments respond to excess debt in an optimal manner, but that they do not have access to a commitment technology. This implies that national fiscal policy gradually reduces debt: the lack of a commitment technology precludes a random walk in steady-state debt, but the need to maintain national competitiveness avoids excessively rapid debt reduction. If the central bank can commit, it adjusts its policies only slightly in response to higher debt, allowing national fiscal policy to undertake most of the adjustment. However, if it cannot commit, then optimal monetary policy involves using interest rates to rapidly reduce debt, with significant welfare costs. We show that in these circumstances the central bank would do better to ignore national fiscal policies in formulating its policy

Min-Max Predictive Control of a Pilot Plant using a QP Approach

47th IEEE Conference on Decision and Control 9-11 Dec. 2008The practical implementation of min-max MPC (MMMPC) controllers is limited by the computational burden required to compute the control law. This problem can be circumvented by using approximate solutions or upper bounds of the worst possible case of the performance index. In a previous work, the authors presented a computationally efficient MMMPC control strategy in which a close approximation of the solution of the min-max problem is computed using a quadratic programming problem. In this paper, this approach is validated through its application to a pilot plant in which the temperature of a reactor is controlled. The behavior of the system and the controller are illustrated by means of experimental results