Financial Applications of Random Matrix Theory: a short review

We discuss the applications of Random Matrix Theory in the context of financial markets and econometric models, a topic about which a considerable number of papers have been devoted to in the last decade. This mini-review is intended to guide the reader through various theoretical results (the Marcenko-Pastur spectrum and its various generalisations, random SVD, free matrices, largest eigenvalue statistics, etc.) as well as some concrete applications to portfolio optimisation and out-of-sample risk estimation.Comment: To appear in the "Handbook on Random Matrix Theory", Oxford University Pres

Data-driven modelling, forecasting and uncertainty analysis of disaggregated demands and wind farm power outputs

Correct analysis of modern power supply systems requires to evaluate much wider ranges of uncertainties introduced by the implementation of new technologies on both supply and demand sides. On the supply side, these uncertainties are due to the increased contributions of renewable generation sources (e.g., wind and PV), whose stochastic output variations are difficult to predict and control, as well as due to the significant changes in system operating conditions, coming from the implementation of various control and balancing actions, increased automation and switching functionalities, and frequent network reconfiguration. On the demand side, these uncertainties are due to the installation of new types of loads, featuring strong spatio-temporal variations of demands (e.g., EV charging), as well as due to the deployment of different demand-side management schemes. Modern power supply systems are also characterised by much higher availability of measurements and recordings, coming from a number of recently deployed advanced monitoring, data acquisition and control systems, and providing valuable information on system operating and loading conditions, state and status of network components and details on various system events, transients and disturbances. Although the processing of large amounts of measured data brings its own challenges (e.g., data quality, performance, and incorporation of domain knowledge), these data open new opportunities for a more accurate and comprehensive evaluation of the overall system performance, which, however, require new data-driven analytical approaches and modelling tools. This PhD research is aimed at developing and evaluating novel and improved data-driven methodologies for modelling renewable generation and demand, in general, and for assessing the corresponding uncertainties and forecasting, in particular. The research and methods developed in this thesis use actual field measurements of several onshore and offshore wind farms, as well as measured active and reactive power demands at several low voltage (LV) individual household levels, up to the demands at medium voltage (MV) substation level. The models are specifically built to be implemented for power system analysis and are actually used by a number of researchers and PhD students in Edinburgh and elsewhere (e.g., collaborations with colleagues from Italy and Croatia), which is discussed and illustrated in the thesis through the selected study cases taken from this joint research efforts. After literature review and discussion of basic concepts and definitions, the first part of the thesis presents data-driven analysis, modelling, uncertainty evaluation and forecasting of (predominantly residential) demands and load profiles at LV and MV levels. The analysis includes both aggregation and disaggregation of measured demands, where the latter is considered in the context of identifying demand-manageable loads (e.g., heating). For that purpose, periodical changes in demands, e.g., half-daily, daily, weekly, seasonal and annual, are represented with Fourier/frequency components and correlated with the corresponding exploratory meteorological variables (e.g., temperature, solar irradiance), allowing to select the combination of components maximising the positive or negative correlations as an additional predictor variable. Convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) are then used to represent dependencies among multiple dimensions and to output the estimated disaggregated time series of specific load types (with Bayesian optimisation applied to select appropriate CNN-BiLSTM hyperparameters). In terms of load forecasting, both tree-based and neural network-based models are analysed and compared for the day-ahead and week-ahead forecasting of demands at MV substation level, which are also correlated with meteorological data. Importantly, the presented load forecasting methodologies allow, for the first time, to forecast both total/aggregate demands and corresponding disaggregated demands of specific load types. In terms of the supply side analysis, the thesis presents data-driven evaluation, modelling, uncertainty evaluation and forecasting of wind-based electricity generation systems. The available measurements from both the individual wind turbines (WTs) and the whole wind farms (WFs) are used to formulate simple yet accurate operational models of WTs and WFs. First, available measurements are preprocessed, to remove outliers, as otherwise obtained WT/WF models may be biased, or even inaccurate. A novel simulation-based approach that builds on a procedure recommended in a standard is presented for processing all outliers due to applied averaging window (typically 10 minutes) and WT hysteresis effects (around the cut-in and cut-out wind speeds). Afterwards, the importance of distinguishing between WT-level and WF-level analysis is discussed and a new six-parameter power curve model is introduced for accurate modelling of both cut-in and cut-out regions and for taking into account operating regimes of a WF (WTs in normal/curtailed operation, or outage/fault). The modelling framework in the thesis starts with deterministic models (e.g., CNN-BiLSTM and power curve models) and is then extended to include probabilistic models, building on the Bayesian inference and Copula theory. In that context, the thesis presents a set of innovative data-driven WT and WF probabilistic models, which can accurately model cross-correlations between the WT/WF power output (Pout), wind speed (WS), air density (AD) and wind direction (WD). Vine Copula and Gaussian mixture Copula model (GMCM) are combined, for the first time, to evaluate the uncertainty of Pout values, conditioning on other explanatory variables (which may be either deterministic, or also uncertain). In terms of probabilistic wind energy forecasting, Bayesian CNN-BiLSTM model is used to analyse and efficiently handle high dimensionality of both input meteorological variables (WS, AD and WD) and additional uncertainties due to WF operating regimes. The presented results demonstrate that the developed Vine-GMCM and operational WF model can accurately integrate and effectively correlate all propagated uncertainties, ultimately resulting in much higher confidence levels of the forecasted WF power outputs than in the existing literature

Quantitative Modelling of Climate Change Impact on Hydro-climatic Extremes

In recent decades, climate change has caused a more volatile climate leading to more extreme events such as severe rainstorms, heatwaves and floods which are likely to become more frequent. Aiming to reveal climate change impact on the hydroclimatic extremes in a quantitative sense, this thesis presents a comprehensive analysis from three main strands. The first strand focuses on developing a quantitative modelling framework to quantify the spatiotemporal variation of hydroclimatic extremes for the areas of concern. A spatial random sampling toolbox (SRS-GDA) is designed for randomizing the regions of interest (ROIs) with different geographic locations, sizes, shapes and orientations where the hydroclimatic extremes are parameterised by a nonstationary distribution model whose parameters are assumed to be time-varying. The parameters whose variation with respect to different spatial features of ROIs and climate change are finally quantified by various statistical models such as the generalised linear model. The framework is applied to quantify the spatiotemporal variation of rainfall extremes in Great Britain (GB) and Australia and is further used in a comparison study to quantify the bias between observed and climate projected extremes. Then the framework is extended to a multivariate framework to estimate the time-varying joint probability of more than one hydroclimatic variable in the perspective of non-stationarity. A case study for evaluating compound floods in Ho Chi Minh City, Vietnam is applied for demonstrating the application of the framework. The second strand aims to recognise, classify and track the development of hydroclimatic extremes (e.g., severe rainstorms) by developing a stable computer algorithm (i.e., the SPER toolbox). The SPER toolbox can detect the boundary of the event area, extract the spatial and physical features of the event, which can be used not only for pattern recognition but also to support AI-based training for labelling/cataloguing the pattern from the large-sized, grid-based, multi-scaled environmental datasets. Three illustrative cases are provided; and as the front-end of AI study, an example for training a convolution neural network is given for classifying the rainfall extremes in the last century of GB. The third strand turns to support decision making by building both theory-driven and data-driven decision-making models to simulate the decisions in the context of flood forecasting and early warning, using the data collected via laboratory-style experiments based on various information of probabilistic flood forecasts and consequences. The research work demonstrated in this thesis has been able to bridge the knowledge gaps in the related field and it also provides a precritical insight in managing future risks arising from hydroclimatic extremes, which makes perfect sense given the urgent situation of climate change and the related challenges our societies are facing

Graph-based Facial Affect Analysis: A Review of Methods, Applications and Challenges

Facial affect analysis (FAA) using visual signals is important in human-computer interaction. Early methods focus on extracting appearance and geometry features associated with human affects, while ignoring the latent semantic information among individual facial changes, leading to limited performance and generalization. Recent work attempts to establish a graph-based representation to model these semantic relationships and develop frameworks to leverage them for various FAA tasks. In this paper, we provide a comprehensive review of graph-based FAA, including the evolution of algorithms and their applications. First, the FAA background knowledge is introduced, especially on the role of the graph. We then discuss approaches that are widely used for graph-based affective representation in literature and show a trend towards graph construction. For the relational reasoning in graph-based FAA, existing studies are categorized according to their usage of traditional methods or deep models, with a special emphasis on the latest graph neural networks. Performance comparisons of the state-of-the-art graph-based FAA methods are also summarized. Finally, we discuss the challenges and potential directions. As far as we know, this is the first survey of graph-based FAA methods. Our findings can serve as a reference for future research in this field.Comment: 20 pages, 12 figures, 5 table

Dynamic corrosion risk-based integrity assessment of marine and offshore systems

Corrosion poses a serious integrity threat to marine and offshore systems. This critical issue leads to high rate of offshore systems degradation, failure, and associated risks. The microbiologically influenced corrosion (microbial corrosion), which is a type of corrosion mechanism, presents inherent complexity due to interactions among influential factors and the bacteria. The stochastic nature of the vital operating parameters and the unstable microbial metabolism affect the prediction of microbial corrosion induced failure and the systems’ integrity management strategy. The unstable and dynamic characteristics of the corrosion induced risk factors need to be captured for a robust integrity management strategy for corroding marine and offshore systems. This thesis proposes dynamic methodology for risk-based integrity assessment of microbially influenced corroding marine and offshore systems. Firstly, a novel probabilistic network based structure is presented to capture the non-linear interactions among the monitoring operating parameters and the bacteria (e.g., sulfate-reducing bacteria) for the microbial corrosion rate predictions. A Markovian stochastic formulation is developed for the corroding offshore system failure probability prediction using the degradation rate as the transition intensity. The analysis results show that the non-linear interactions among the microbial corrosion influential parameters increase the corrosion rate and decrease the corroding system's failure time. Secondly, a dynamic model is introduced to evaluate the offshore system's operational safety under microbial corrosion induced multiple defect interactions. An effective Bayesian network - Markovian mixture structure is integrated with the Monte Carlo algorithm to forecast the effects of defects interactions and the corrosion response parameters’ variability on offshore system survivability under multispecies biofilm architecture. The results reveal the impact of defects interaction on the system's survivability profile under different operational scenarios and suggest the critical intervention time based on the corrosivity index to prevent total failure of the offshore system. Finally, a probabilistic investigation is carried out to determine the parametric interdependencies' effects on the corroding system reliability using a Copula-based Monte Carlo algorithm. The model simultaneously captures the failure modes and the non-linear correlation effects on the offshore system reliability under multispecies biofilm structure. The research outputs suggest a realistic reliability-based integrity management strategy that is consistent with industry best practices. Furthermore, a dynamic risk-based assessment framework is developed considering the evolving characteristics of the influential microbial corrosion factors. A novel dynamic Bayesian network structure is developed to capture the corrosion's evolving stochastic process and the importance of input parameters based on their temporal interrelationship. The associated loss scenarios due to microbial corrosion induced failures are modeled using a loss aggregation technique. A subsea pipeline is used to demonstrate the model performance. The proposed integrated model provides a risk-based prognostic tool to aid engineers and integrity managers for making effective safety and risk strategies. This work explores the microbial corrosion induced failure mechanisms and develops dynamic risk-based tools under different operational scenarios for systems’ integrity management in the marine and offshore oil and gas industries

Decision-Making with Heterogeneous Sensors - A Copula Based Approach

Statistical decision making has wide ranging applications, from communications and signal processing to econometrics and finance. In contrast to the classical one source-one receiver paradigm, several applications have been identified in the recent past that require acquiring data from multiple sources or sensors. Information from the multiple sensors are transmitted to a remotely located receiver known as the fusion center which makes a global decision. Past work has largely focused on fusion of information from homogeneous sensors. This dissertation extends the formulation to the case when the local sensors may possess disparate sensing modalities. Both the theoretical and practical aspects of multimodal signal processing are considered. The first and foremost challenge is to \u27adequately\u27 model the joint statistics of such heterogeneous sensors. We propose the use of copula theory for this purpose. Copula models are general descriptors of dependence. They provide a way to characterize the nonlinear functional relationships between the multiple modalities, which are otherwise difficult to formalize. The important problem of selecting the `best\u27 copula function from a given set of valid copula densities is addressed, especially in the context of binary hypothesis testing problems. Both, the training-testing paradigm, where a training set is assumed to be available for learning the copula models prior to system deployment, as well as generalized likelihood ratio test (GLRT) based fusion rule for the online selection and estimation of copula parameters are considered. The developed theory is corroborated with extensive computer simulations as well as results on real-world data. Sensor observations (or features extracted thereof) are most often quantized before their transmission to the fusion center for bandwidth and power conservation. A detection scheme is proposed for this problem assuming unifom scalar quantizers at each sensor. The designed rule is applicable for both binary and multibit local sensor decisions. An alternative suboptimal but computationally efficient fusion rule is also designed which involves injecting a deliberate disturbance to the local sensor decisions before fusion. The rule is based on Widrow\u27s statistical theory of quantization. Addition of controlled noise helps to \u27linearize\u27 the higly nonlinear quantization process thus resulting in computational savings. It is shown that although the introduction of external noise does cause a reduction in the received signal to noise ratio, the proposed approach can be highly accurate when the input signals have bandlimited characteristic functions, and the number of quantization levels is large. The problem of quantifying neural synchrony using copula functions is also investigated. It has been widely accepted that multiple simultaneously recorded electroencephalographic signals exhibit nonlinear and non-Gaussian statistics. While the existing and popular measures such as correlation coefficient, corr-entropy coefficient, coh-entropy and mutual information are limited to being bivariate and hence applicable only to pairs of channels, measures such as Granger causality, even though multivariate, fail to account for any nonlinear inter-channel dependence. The application of copula theory helps alleviate both these limitations. The problem of distinguishing patients with mild cognitive impairment from the age-matched control subjects is also considered. Results show that the copula derived synchrony measures when used in conjunction with other synchrony measures improve the detection of Alzheimer\u27s disease onset

Resilience in Transportation Networks

The functionality of transportation networks is greatly challenged by risk factors such as increasing climate-related hazards, rising population exposure, and greater city vulnerability. Inevitably, the transportation network cannot withstand the impact of an overwhelming disaster, which results in rapid declines in the performance of road net-work. As a next step, the authorities need to restore the performance of the road net-work to an acceptable state as soon as possible and rebalance the conflict between the capacity of the road network and travel demand. Resilience is defined as the process of system performance degradation followed by recovery. To improve the transportation network resilience and maintain regular traffic, it is crucial to identify which factors are related to the resilience and investigate how these factors impact resilience. In this thesis, four factors, i.e., road networks, evacuees, disruption types and au-thorities, are identified to analyze resilience mechanisms. Firstly, the change in vehicle speed during a disaster is used as a measure of resilience, and we analyze the quantita-tive relationship between resilience and the structural characteristics and properties of the road network in multiple disruptions in multiple cities. The results show that the connectivity of the road network, the predictability of disruption, and the population density affect the resilience of the road network in different ways. Secondly, as the road connectivity plays a crucial role during the evacuation pe-riod and considering more frequent and extensive bushfires, we explore a practical and challenging problem: are bushfire fatalities related to road network characteristics? Con-nectivity index (CI), a composite metric that takes into account redundancy, connectivi-ty, and population exposure is designed. The statistical analysis of real-world data sug-gests that CI is significantly negatively correlated with historical bushfire fatalities. This parsimonious and simple graph-theoretic measure can provide planners a useful metric to reduce vulnerability and increase resilience among areas that are prone to bushfires. Finally, a modelling framework for optimizing road network pre-disaster invest-ment strategy under different disaster damage levels is proposed. A bi-level multi-objective optimization model is formulated, in which the upper-level aims to maximize the capacity-based functionality and robustness of the road network, and the lower-level is the user equilibrium problem. To efficiently solve the model, the Shapley value is used to select candidate edges and obtain a near-optimal project order. For more reality, the heterogeneity of road segments to hazards and the correlation of road segments in dif-ferent hazard phases are considered. Realistic speed data is used to explore the depend-ency between different disaster states with copula functions. The numerical results illus-trate that the investment strategy is significantly influenced by the road edge character-istics and the level of disaster damage. Critical sections that can significantly improve the overall functionality of the network are identified. Overall, the core contribution of this thesis is to provide insights into the evalua-tion and analysis of resilience in transportation networks, as well as develop modelling frameworks to promote resilience. The results of this work can provide a theoretical ba-sis for road network design, pre-disaster investment and post-disaster emergency rescue