Combining Local and Global Direct Derivative-free Optimization for Reinforcement Learning

We consider the problem of optimization in policy space for reinforcement learning. While a plethora of methods have been applied to this problem, only a narrow category of them proved feasible in robotics. We consider the peculiar characteristics of reinforcement learning in robotics, and devise a combination of two algorithms from the literature of derivative-free optimization. The proposed combination is well suited for robotics, as it involves both off-line learning in simulation and on-line learning in the real environment. We demonstrate our approach on a real-world task, where an Autonomous Underwater Vehicle has to survey a target area under potentially unknown environment conditions. We start from a given controller, which can perform the task under foreseeable conditions, and make it adaptive to the actual environment

Risk of hospitalization for heart failure in patients with type 2 diabetes newly treated with DPP-4 inhibitors or other oral glucose-lowering medications: A retrospective registry study on 127,555 patients from the Nationwide OsMed Health-DB Database

Aims Oral glucose-lowering medications are associated with excess risk of heart failure (HF). Given the absence of comparative data among drug classes, we performed a retrospective study in 32 Health Services of 16 Italian regions accounting for a population of 18 million individuals, to assess the association between HF risk and use of sulphonylureas, DPP-4i, and glitazones. Methods and results We extracted data on patients with type 2 diabetes who initiated treatment with DPP-4i, thiazolidinediones, or sulphonylureas alone or in combination with metformin during an accrual time of 2 years. The endpoint was hospitalization for HF (HHF) occurring after the first 6 months of therapy, and the observation was extended for up to 4 years. A total of 127 555 patients were included, of whom 14.3% were on DPP-4i, 72.5% on sulphonylurea, 13.2% on thiazolidinediones, with average 70.7% being on metformin as combination therapy. Patients in the three groups differed significantly for baseline characteristics: age, sex, Charlson index, concurrent medications, and previous cardiovascular events. During an average 2.6-year follow-up, after adjusting for measured confounders, use of DPP-4i was associated with a reduced risk of HHF compared with sulphonylureas [hazard ratio (HR) 0.78; 95% confidence interval (CI) 0.62-0.97; P = 0.026]. After propensity matching, the analysis was restricted to 39 465 patients, and the use of DPP-4i was still associated with a lower risk of HHF (HR 0.70; 95% CI 0.52-0.94; P = 0.018). Conclusion In a very large observational study, the use of DPP-4i was associated with a reduced risk of HHF when compared with sulphonylureas

On generalized Nash equilibrium problems with linear coupling constraints and mixed-integer variables

We define and discuss different enumerative methods to compute solutions of generalized Nash equilibrium problems with linear coupling constraints and mixed-integer variables. We propose both branch-and-bound methods based on merit functions for the mixed-integer game, and branch-and-prune methods that exploit the concept of dominance to make effective cuts. We show that under mild assumptions the equilibrium set of the game is finite and we define an enumerative method to compute the whole of it. We show that our branch-and-prune method can be suitably modified in order to make a general equilibrium selection over the solution set of the mixed-integer game. We define an application in economics that can be modelled as a Nash game with linear coupling constraints and mixed-integer variables, and we adapt the branch-and-prune method to efficiently solve it

Numerically tractable optimistic bilevel problems

We consider a class of optimistic bilevel problems. Specifically, we address bilevel problems in which at the lower level the objective function is fully convex and the feasible set does not depend on the upper level variables.We show that this nontrivial class of mathematical programs is sufficiently broad to encompass significant real-world applications and proves to be numerically tractable. From this respect, we establish that the stationary points for a relaxation of the original problem can be obtained addressing a suitable generalized Nash equilibrium problem. The latter game is proven to be convex and with a nonempty solution set. Leveraging this correspondence, we provide a provably convergent, easily implementable scheme to calculate stationary points of the relaxed bilevel program. As witnessed by some numerical experiments on an application in economics, this algorithm turns out to be numerically viable also for big dimensional problems

Effectively managing diagnostic tests to monitor the COVID-19 outbreak in Italy

Urged by the outbreak of the COVID-19 in Italy, this study aims at helping to tackle the spread of the disease by resorting to operations research techniques. In particular, we propose a mathematical program to model the problem of establishing how many diagnostic tests the Italian regions must perform in order to maximize the overall disease detection capability. An important feature of our approach is its simplicity: data we resort to are easy to obtain and one can employ standard optimization tools to address the problem. The results we obtain when applying our method to the Italian case seem promising

Nonsingularity and Stationarity Results for Quasi-Variational Inequalities

The optimality system of a quasi-variational inequality can be reformulated as a non-smooth equation or a constrained equation with a smooth function. Both reformulations can be exploited by algorithms, and their convergence to solutions usually relies on the nonsingularity of the Jacobian, or the fact that the merit function has no nonoptimal stationary points. We prove new sufficient conditions for the absence of nonoptimal constrained or unconstrained stationary points that are weaker than some known ones. All these conditions exploit some properties of a certain matrix, but do not require the nonsingularity of the Jacobian. Further, we present new necessary and sufficient conditions for the nonsingularity of the Jacobian that are based on the signs of certain determinants. Additionally, we consider generalized Nash equilibrium problems that are a special class of quasi-variational inequalities. Exploiting their structure, we also prove some new sufficient conditions for stationarity and nonsingularity results

Combining approximation and exact penalty in hierarchical programming

We address the minimization of an objective function over the solution set of a (non-parametric) lower-level variational inequality. This problem is a special instance of semi-infinite programs and encompasses, as particular cases, simple (smooth) bilevel and equilibrium selection problems. We resort to a suitable approximated version of the hierarchical problem. We show that this, on the one hand, does not perturb the original (exact) program ‘too much’, on the other hand, allows one to rely on some suitable exact penalty approaches whose convergence properties are established

On Nested Affine Variational Inequalities: The Case of Multi-Portfolio Selection

We deal with nested affine variational inequalities, i.e., hierarchical problems involving an affine (upper-level) variational inequality whose feasible set is the solution set of another affine (lower-level) variational inequality. We apply this modeling tool to the multi-portfolio selection problem, where the lower-level variational inequality models the Nash equilibrium problem made up by the different accounts, while the upper-level variational inequality is instrumental to perform a selection over this equilibrium set. We propose a projected averaging Tikhonov-like algorithm for the solution of this problem, which only requires the monotonicity of the variational inequalities for both the upper- and the lower-level in order to converge. Finally, we provide complexity properties

Interactions Between Bilevel Optimization and Nash Games

We aim at building a bridge between bilevel programming and generalized Nash equilibrium problems. First, we present two Nash games that turn out to be linked to the (approximated) optimistic version of the bilevel problem. Specifically, on the one hand we establish relations between the equilibrium set of a Nash game and global optima of the (approximated) optimistic bilevel problem. On the other hand, correspondences between equilibria of another Nash game and stationary points of the (approximated) optimistic bilevel problem are obtained. Then, building on these ideas, we also propose different Nash-like models that are related to the (approximated) pessimistic version of the bilevel problem. This analysis, being of independent theoretical interest, leads also to algorithmic developments. Finally, we discuss the intrinsic complexity characterizing both the optimistic bilevel and the Nash game models