Generating Adversarial Examples with Task Oriented Multi-Objective Optimization

Deep learning models, even the-state-of-the-art ones, are highly vulnerable to adversarial examples. Adversarial training is one of the most efficient methods to improve the model's robustness. The key factor for the success of adversarial training is the capability to generate qualified and divergent adversarial examples which satisfy some objectives/goals (e.g., finding adversarial examples that maximize the model losses for simultaneously attacking multiple models). Therefore, multi-objective optimization (MOO) is a natural tool for adversarial example generation to achieve multiple objectives/goals simultaneously. However, we observe that a naive application of MOO tends to maximize all objectives/goals equally, without caring if an objective/goal has been achieved yet. This leads to useless effort to further improve the goal-achieved tasks, while putting less focus on the goal-unachieved tasks. In this paper, we propose \emph{Task Oriented MOO} to address this issue, in the context where we can explicitly define the goal achievement for a task. Our principle is to only maintain the goal-achieved tasks, while letting the optimizer spend more effort on improving the goal-unachieved tasks. We conduct comprehensive experiments for our Task Oriented MOO on various adversarial example generation schemes. The experimental results firmly demonstrate the merit of our proposed approach. Our code is available at \url{https://github.com/tuananhbui89/TAMOO}

Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectives

Machine learning (ML) methods offer a wide range of configurable hyperparameters that have a significant influence on their performance. While accuracy is a commonly used performance objective, in many settings, it is not sufficient. Optimizing the ML models with respect to multiple objectives such as accuracy, confidence, fairness, calibration, privacy, latency, and memory consumption is becoming crucial. To that end, hyperparameter optimization, the approach to systematically optimize the hyperparameters, which is already challenging for a single objective, is even more challenging for multiple objectives. In addition, the differences in objective scales, the failures, and the presence of outlier values in objectives make the problem even harder. We propose a multi-objective Bayesian optimization (MoBO) algorithm that addresses these problems through uniform objective normalization and randomized weights in scalarization. We increase the efficiency of our approach by imposing constraints on the objective to avoid exploring unnecessary configurations (e.g., insufficient accuracy). Finally, we leverage an approach to parallelize the MoBO which results in a 5x speed-up when using 16x more workers.Comment: Preprint with appendice

Evolutionary multi-objective worst-case robust optimisation

Many real-world problems are subject to uncertainty, and often solutions should not only be good, but also robust against environmental disturbances or deviations from the decision variables. While most papers dealing with robustness aim at finding solutions with a high expected performance given a distribution of the uncertainty, we examine the trade-off between the allowed deviations from the decision variables (tolerance level), and the worst case performance given the allowed deviations. In this research work, we suggest two multi-objective evolutionary algorithms to compute the available trade-offs between allowed tolerance level and worst-case quality of the solutions, and the tolerance level is defined as robustness which could also be the variations from parameters. Both algorithms are 2-level nested algorithms. While the first algorithm is point-based in the sense that the lower level computes a point of worst case for each upper level solution, the second algorithm is envelope-based, in the sense that the lower level computes a whole trade-off curve between worst-case fitness and tolerance level for each upper level solution. Our problem can be considered as a special case of bi-level optimisation, which is computationally expensive, because each upper level solution is evaluated by calling a lower level optimiser. We propose and compare several strategies to improve the efficiency of both algorithms. Later, we also suggest surrogate-assisted algorithms to accelerate both algorithms

Bi-objective facility location in the presence of uncertainty

Multiple and usually conflicting objectives subject to data uncertainty are main features in many real-world problems. Consequently, in practice, decision-makers need to understand the trade-off between the objectives, considering different levels of uncertainty in order to choose a suitable solution. In this paper, we consider a two-stage bi-objective single source capacitated model as a base formulation for designing a last-mile network in disaster relief where one of the objectives is subject to demand uncertainty. We analyze scenario-based two-stage risk-neutral stochastic programming, adaptive (two-stage) robust optimization, and a two-stage risk-averse stochastic approach using conditional value-at-risk (CVaR). To cope with the bi-objective nature of the problem, we embed these concepts into two criterion space search frameworks, the $\epsilon$-constraint method and the balanced box method, to determine the Pareto frontier. Additionally, a matheuristic technique is developed to obtain high-quality approximations of the Pareto frontier for large-size instances. In an extensive computational experiment, we evaluate and compare the performance of the applied approaches based on real-world data from a Thies drought case, Senegal

Parametrization of stochastic inputs using generative adversarial networks with application in geology

We investigate artificial neural networks as a parametrization tool for stochastic inputs in numerical simulations. We address parametrization from the point of view of emulating the data generating process, instead of explicitly constructing a parametric form to preserve predefined statistics of the data. This is done by training a neural network to generate samples from the data distribution using a recent deep learning technique called generative adversarial networks. By emulating the data generating process, the relevant statistics of the data are replicated. The method is assessed in subsurface flow problems, where effective parametrization of underground properties such as permeability is important due to the high dimensionality and presence of high spatial correlations. We experiment with realizations of binary channelized subsurface permeability and perform uncertainty quantification and parameter estimation. Results show that the parametrization using generative adversarial networks is very effective in preserving visual realism as well as high order statistics of the flow responses, while achieving a dimensionality reduction of two orders of magnitude

Space Systems Resilience Engineering and Global System Reliability Optimisation Under Imprecision and Epistemic Uncertainty

The paper introduces the concept of design for resilience in the context of space systems engineering and proposes a method to account for imprecision and epistemic uncertainty. Resilience can be seen as the ability of a system to adjust its functioning prior to, during, or following changes and disturbances, so that it can sustain required operations under both expected and unexpected conditions. Mathematically speaking this translates into the attribute of a dynamical system (or time dependent system) to be simultaneously robust and reliable. However, the quantification of robustness and reliability in the early stage of the design of a space systems is generally affected by uncertainty that is epistemic in nature. As the design evolves from Phase A down to phase E, the level of epistemic uncertainty is expected to decrease but still a level of variability can exist in the expected operational conditions and system requirements. The paper proposes a representation of a complex space system using the so called Evidence Network Models (ENM): a non-directed (unlike Bayesian network models) network of interconnected nodes where each node represents a subsystem with associated epistemic uncertainty on system performance and failure probability. Once the reliability and uncertainty on the performance of the spacecraft are quantified, a design optimisation process is applied to improve resilience and performance. The method is finally applied to an example of preliminary design of a small satellite in Low Earth Orbit (LEO). The spacecraft is divided in 5 subsystems, AOCS, TTC, OBDH, Power and Payload. The payload is a simple camera acquiring images at scheduled times. The assumption is that each component has multiple functionalities and both the performance of the component and the reliability associated to each functionality are affected by a level of imprecision. The overall performance indicator is the sum of the performance indicators of all the components

Metaheuristics for black-box robust optimisation problems

Our interest is in the development of algorithms capable of tackling robust black-box optimisation problems, where the number of model runs is limited. When a desired solution cannot be implemented exactly (implementation uncertainty) the aim is to find a robust one. Here that is to find a point in the decision variable space such that the worst solution from within an uncertainty region around that point still performs well. This thesis comprises three research papers. One has been published, one accepted for publication, and one submitted for publication. We initially develop a single-solution based approach, largest empty hypersphere (LEH), which identifies poor performing points in the decision variable space and repeatedly moves to the centre of the region devoid of all such points. Building on this we develop population based approaches using a particle swarm optimisation (PSO) framework. This combines elements of the LEH approach, a local descent directions (d.d.) approach for robust problems, and a series of novel features. Finally we employ an automatic generation of algorithms technique, genetic programming (GP), to evolve a population of PSO based heuristics for robust problems. We generate algorithmic sub-components, the design rules by which they are combined to form complete heuristics, and an evolutionary GP framework. The best performing heuristics are identified. With the development of each heuristic we perform experimental testing against comparator approaches on a suite of robust test problems of dimension between 2D and 100D. Performance is shown to improve with each new heuristic. Furthermore the generation of large numbers of heuristics in the GP process enables an assessment of the best performing sub-components. This can be used to indicate the desirable features of an effective heuristic for tackling the problem under consideration. Good performance is observed for the following characteristics: inner maximisation by random sampling, a small number of inner points, particle level stopping conditions, a small swarm size, a Global topology, and particle movement using a baseline inertia formulation augmented by LEH and d.d. capabilities

AI/Machine learning approach to identifying potential statistical arbitrage opportunities with FX and Bitcoin Markets

In this study, a methodology is presented where a hybrid system combining an evolutionary algorithm with artificial neural networks (ANNs) is designed to make weekly directional change forecasts on the USD by inferring a prediction using closing spot rates of three currency pairs: EUR/USD, GBP/USD and CHF/USD. The forecasts made by the genetically trained ANN are compared to those made by a new variation of the simple moving average (MA) trading strategy, tailored to the methodology, as well as a random model. The same process is then repeated for the three major cryptocurrencies namely: BTC/USD, ETH/USD and XRP/USD. The overall prediction accuracy, uptrend and downtrend prediction accuracy is analyzed for all three methods within the fiat currency as well as the cryptocurrency contexts. The best models are then evaluated in terms of their ability to convert predictive accuracy to a profitable investment given an initial investment. The best model was found to be the hybrid model on the basis of overall prediction accuracy and accrued returns

Design optimisation of complex space systems under epistemic uncertainty

This thesis presents an innovative methodology for System Design Optimisation (SDO) through the framework of Model-Based System Engineering (MBSE) that bridges system modelling, Constrained Global Optimisation (CGO), Uncertainty Quantification (UQ), System Dynamics (SD) and other mathematical tools for the design of Complex Engineered and Engineering Systems (CEdgSs) under epistemic uncertainty. The problem under analysis has analogies with what is nowadays studied as Generative Design under Uncertainty. The method is finally applied to the design of Space Systems which are Complex Engineered Systems (CEdSs) composed of multiple interconnected sub-systems. A critical aspect in the design of Space Systems is the uncertainty involved. Much of the uncertainty is epistemic and is here modelled with Dempster Shafer Theory (DST). Designing space systems is a complex task that involves the coordination of different disciplines and problems. The thesis then proposes a set of building blocks, that is a toolbox of methodologies for the solution of problems which are of interest also if considered independently. It proposes then a holistic framework that couples these building blocks to form a SDO procedure. With regard to the building blocks, the thesis includes a network-based modelling procedure for CEdSs and a generalisation for CEdgSs where the system and the whole design process are both taken into account. Then, it presents a constraint min-max solver as an algorithmic procedures for the solution of the general Optimisation Under Uncertainty (OUU) problem. An extension of the method for the Multi-Objective Problems (MOP) is also proposed in Appendix as a minor result. A side contribution for the optimisation part refers to the extension of the global optimiser Multi Population Adaptive Inflationary Differential Evolution Algorithm (MP-AIDEA) with the introduction of constraint handling and multiple objective functions. The Constraint Multi-Objective Problem (CMOP) solver is however a preliminary result and it is reported in Appendix. Furthermore, the thesis proposes a decomposition methodology for the computational reduction of UQ with DST. As a partial contribution, a second approach based on a Binary Tree decomposition is also reported in Appendix. With regard to the holistic approach, instead, the thesis gives a new dentition and proposes a framework for system network robustness and for system network resilience. It finally presents the framework for the optimisation of the whole design process through the use of a multi-layer network model.This thesis presents an innovative methodology for System Design Optimisation (SDO) through the framework of Model-Based System Engineering (MBSE) that bridges system modelling, Constrained Global Optimisation (CGO), Uncertainty Quantification (UQ), System Dynamics (SD) and other mathematical tools for the design of Complex Engineered and Engineering Systems (CEdgSs) under epistemic uncertainty. The problem under analysis has analogies with what is nowadays studied as Generative Design under Uncertainty. The method is finally applied to the design of Space Systems which are Complex Engineered Systems (CEdSs) composed of multiple interconnected sub-systems. A critical aspect in the design of Space Systems is the uncertainty involved. Much of the uncertainty is epistemic and is here modelled with Dempster Shafer Theory (DST). Designing space systems is a complex task that involves the coordination of different disciplines and problems. The thesis then proposes a set of building blocks, that is a toolbox of methodologies for the solution of problems which are of interest also if considered independently. It proposes then a holistic framework that couples these building blocks to form a SDO procedure. With regard to the building blocks, the thesis includes a network-based modelling procedure for CEdSs and a generalisation for CEdgSs where the system and the whole design process are both taken into account. Then, it presents a constraint min-max solver as an algorithmic procedures for the solution of the general Optimisation Under Uncertainty (OUU) problem. An extension of the method for the Multi-Objective Problems (MOP) is also proposed in Appendix as a minor result. A side contribution for the optimisation part refers to the extension of the global optimiser Multi Population Adaptive Inflationary Differential Evolution Algorithm (MP-AIDEA) with the introduction of constraint handling and multiple objective functions. The Constraint Multi-Objective Problem (CMOP) solver is however a preliminary result and it is reported in Appendix. Furthermore, the thesis proposes a decomposition methodology for the computational reduction of UQ with DST. As a partial contribution, a second approach based on a Binary Tree decomposition is also reported in Appendix. With regard to the holistic approach, instead, the thesis gives a new dentition and proposes a framework for system network robustness and for system network resilience. It finally presents the framework for the optimisation of the whole design process through the use of a multi-layer network model