Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning

In the predict-then-optimize framework, the objective is to train a predictive model, mapping from environment features to parameters of an optimization problem, which maximizes decision quality when the optimization is subsequently solved. Recent work on decision-focused learning shows that embedding the optimization problem in the training pipeline can improve decision quality and help generalize better to unseen tasks compared to relying on an intermediate loss function for evaluating prediction quality. We study the predict-then-optimize framework in the context of sequential decision problems (formulated as MDPs) that are solved via reinforcement learning. In particular, we are given environment features and a set of trajectories from training MDPs, which we use to train a predictive model that generalizes to unseen test MDPs without trajectories. Two significant computational challenges arise in applying decision-focused learning to MDPs: (i) large state and action spaces make it infeasible for existing techniques to differentiate through MDP problems, and (ii) the high-dimensional policy space, as parameterized by a neural network, makes differentiating through a policy expensive. We resolve the first challenge by sampling provably unbiased derivatives to approximate and differentiate through optimality conditions, and the second challenge by using a low-rank approximation to the high-dimensional sample-based derivatives. We implement both Bellman--based and policy gradient--based decision-focused learning on three different MDP problems with missing parameters, and show that decision-focused learning performs better in generalization to unseen tasks

Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunities

Decision-focused learning (DFL) is an emerging paradigm in machine learning which trains a model to optimize decisions, integrating prediction and optimization in an end-to-end system. This paradigm holds the promise to revolutionize decision-making in many real-world applications which operate under uncertainty, where the estimation of unknown parameters within these decision models often becomes a substantial roadblock. This paper presents a comprehensive review of DFL. It provides an in-depth analysis of the various techniques devised to integrate machine learning and optimization models, introduces a taxonomy of DFL methods distinguished by their unique characteristics, and conducts an extensive empirical evaluation of these methods proposing suitable benchmark dataset and tasks for DFL. Finally, the study provides valuable insights into current and potential future avenues in DFL research.Comment: Experimental Survey and Benchmarkin

Fairguard: Harness Logic-based Fairness Rules in Smart Cities

Smart cities operate on computational predictive frameworks that collect, aggregate, and utilize data from large-scale sensor networks. However, these frameworks are prone to multiple sources of data and algorithmic bias, which often lead to unfair prediction results. In this work, we first demonstrate that bias persists at a micro-level both temporally and spatially by studying real city data from Chattanooga, TN. To alleviate the issue of such bias, we introduce Fairguard, a micro-level temporal logic-based approach for fair smart city policy adjustment and generation in complex temporal-spatial domains. The Fairguard framework consists of two phases: first, we develop a static generator that is able to reduce data bias based on temporal logic conditions by minimizing correlations between selected attributes. Then, to ensure fairness in predictive algorithms, we design a dynamic component to regulate prediction results and generate future fair predictions by harnessing logic rules. Evaluations show that logic-enabled static Fairguard can effectively reduce the biased correlations while dynamic Fairguard can guarantee fairness on protected groups at run-time with minimal impact on overall performance.Comment: This paper was accepted by the 8th ACM/IEEE Conference on Internet of Things Design and Implementatio

A bilevel framework for decision-making under uncertainty with contextual information

In this paper, we propose a novel approach for data-driven decision-making under uncertainty in the presence of contextual information. Given a finite collection of observations of the uncertain parameters and potential explanatory variables (i.e., the contextual information), our approach fits a parametric model to those data that is specifically tailored to maximizing the decision value, while accounting for possible feasibility constraints. From a mathematical point of view, our framework translates into a bilevel program, for which we provide both a fast regularization procedure and a big-M-based reformulation that can be solved using off-the-shelf optimization solvers. We showcase the benefits of moving from the traditional scheme for model estimation (based on statistical quality metrics) to decision-guided prediction using three different practical problems. We also compare our approach with existing ones in a realistic case study that considers a strategic power producer that participates in the Iberian electricity market. Finally, we use these numerical simulations to analyze the conditions (in terms of the firm’s cost structure and production capacity) under which our approach proves to be more advantageous to the producer.This work was supported in part by the European Research Council (ERC) under the EU Horizon 2020 research and innovation program (grant agreement No. 755705), in part by the Spanish Ministry of Science and Innovation (AEI/10.13039/501100011033) through project PID2020-115460GB-I00, and in part by the Junta de Andalucía (JA), the Universidad de Málaga and the European Regional Development Fund (FEDER) through the research projects P20_00153 and UMA2018‐FEDERJA‐001. M. Á. Muñoz is also funded by the Spanish Ministry of Science, Innovation and Universities through the State Training Subprogram 2018 of the State Program for the Promotion of Talent and its Employability in R&D&I, within the framework of the State Plan for Scientific and Technical Research and Innovation 2017-2020 and by the European Social Fund. Finally, the authors thankfully acknowledge the computer resources, technical expertise, and assistance provided by the SCBI (Supercomputing and Bioinformatics) center of the University of Malaga

Feature-driven improvement of renewable energy forecasting and trading

M. A. Muñoz, J. M. Morales, and S. Pineda, Feature-driven Improvement of Renewable Energy Forecasting and Trading, IEEE Transactions on Power Systems, 2020.Inspired from recent insights into the common ground of machine learning, optimization and decision-making, this paper proposes an easy-to-implement, but effective procedure to enhance both the quality of renewable energy forecasts and the competitive edge of renewable energy producers in electricity markets with a dual-price settlement of imbalances. The quality and economic gains brought by the proposed procedure essentially stem from the utilization of valuable predictors (also known as features) in a data-driven newsvendor model that renders a computationally inexpensive linear program. We illustrate the proposed procedure and numerically assess its benefits on a realistic case study that considers the aggregate wind power production in the Danish DK1 bidding zone as the variable to be predicted and traded. Within this context, our procedure leverages, among others, spatial information in the form of wind power forecasts issued by transmission system operators (TSO) in surrounding bidding zones and publicly available in online platforms. We show that our method is able to improve the quality of the wind power forecast issued by the Danish TSO by several percentage points (when measured in terms of the mean absolute or the root mean square error) and to significantly reduce the balancing costs incurred by the wind power producer.European Research Council (ERC) under the EU Horizon 2020 research and innovation programme (grant agreement No. 755705) Spanish Ministry of Economy, Industry, and Competitiveness through project ENE2017-83775-P

The intersection of machine learning with forecasting and optimisation: theory and applications

Forecasting and optimisation are two major fields of operations research that are widely used in practice. These methods have contributed to each other growth in several ways. However, the nature of the relationship between these two fields and integrating them have not been explored or understood enough. We advocate the integration of these two fields and explore several problems that require both forecasting and optimisation to deal with the uncertainties. We further investigate some of the methodologies that lie at the intersection of machine learning with prediction and optimisation to address real-world problems. Finally, we provide several research directions for those interested to work in this domain