Forecasting Cryptocurrency Value by Sentiment Analysis: An HPC-Oriented Survey of the State-of-the-Art in the Cloud Era

This chapter surveys the state-of-the-art in forecasting cryptocurrency value by Sentiment Analysis. Key compounding perspectives of current challenges are addressed, including blockchains, data collection, annotation, and filtering, and sentiment analysis metrics using data streams and cloud platforms. We have explored the domain based on this problem-solving metric perspective, i.e., as technical analysis, forecasting, and estimation using a standardized ledger-based technology. The envisioned tools based on forecasting are then suggested, i.e., ranking Initial Coin Offering (ICO) values for incoming cryptocurrencies, trading strategies employing the new Sentiment Analysis metrics, and risk aversion in cryptocurrencies trading through a multi-objective portfolio selection. Our perspective is rationalized on the perspective on elastic demand of computational resources for cloud infrastructures

A survey on financial applications of metaheuristics

Modern heuristics or metaheuristics are optimization algorithms that have been increasingly used during the last decades to support complex decision-making in a number of fields, such as logistics and transportation, telecommunication networks, bioinformatics, finance, and the like. The continuous increase in computing power, together with advancements in metaheuristics frameworks and parallelization strategies, are empowering these types of algorithms as one of the best alternatives to solve rich and real-life combinatorial optimization problems that arise in a number of financial and banking activities. This article reviews some of the works related to the use of metaheuristics in solving both classical and emergent problems in the finance arena. A non-exhaustive list of examples includes rich portfolio optimization, index tracking, enhanced indexation, credit risk, stock investments, financial project scheduling, option pricing, feature selection, bankruptcy and financial distress prediction, and credit risk assessment. This article also discusses some open opportunities for researchers in the field, and forecast the evolution of metaheuristics to include real-life uncertainty conditions into the optimization problems being considered.This work has been partially supported by the Spanish Ministry of Economy and Competitiveness (TRA2013-48180-C3-P, TRA2015-71883-REDT), FEDER, and the Universitat Jaume I mobility program (E-2015-36)

전이 엔트로피와 기계학습에 기반한 금융투자 실증 연구

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 산업공학과, 2020. 8. 장우진.Stock markets have been studied extensively as one of the crucial fields of economy. In particular, research has been actively conducted to analyze and predict the stock market based on relationships among the dynamics of stock prices and returns. In this context, transfer entropy is a non-parametric indicator in analyzing relationships between components of a system, and has a more flexible analytical ability than correlation or Granger-causality. The study of stock price prediction is also being studied from traditional linear models to the latest machine learning models, and research on the optimal asset allocation strategy based on these studies are conducted. The purpose of this dissertation is to derive ETE based network indicator with a market explanatory power for the US stock market by using effective transfer entropy, which is mainly used in econophysics and information theory. The improvement of the performance of the stock price direction prediction through various machine learning algorithms by ETE based network indicator is also analyzed. Furthermore, we apply the prediction result of the stock price through the machine learning algorithm with ETE based network indicator to optimal portfolio strategy through the Black-Litterman model to study the practical use of the investment strategy. At first, we explore that the ETE based on 3 and 6 months moving windows can be regarded as the market explanatory variable by analyzing the association between the financial crises and statistical explanatory power among the stocks. We found that 3 and 6 months moving windows ETEs increase in major financial crises, and that the sectors related to the financial crises have a statistical explanatory power to other sectors through the time-varying analysis of the ETE network indicators. Then, we discover that the prediction performance on the stock price direction can be improved when the ETE driven variable is integrated as a new feature in the logistic regression, multilayer perceptron, random forest, XGBoost, and long short-term memory network. Meanwhile, we suggest utilizing the adjusted accuracy derived from the risk-adjusted return in finance as a prediction performance measure. Notably, we confirm that the multilayer perceptron and long short-term memory network are more suitable for stock price prediction. Lastly, we examined the possibility for investors to develop an investment strategy that maximizes profits through the Black-Litterman model using ETE and machine learning. The characteristics of the inflow and outflow ETE network indicators with market explanatory power and the stock price direction prediction results using machine learning algorithms are applied to the investor's view of the Black-Literman model. The Black-Litterman portfolio, which applies the results of the stock price direction prediction using machine learning algorithms to the investor's view, provides a better return on risk than the market portfolio and market index, and the Black-Litterman portfolio with the ETE network indicator has the highest yield. The use of ETE and stock price prediction leads to improved return on investment, and improving predictive performance increases the return on investment. This dissertation is the first study on the optimal portfolio establishment strategy through the Black-Litterman model and stock price direction prediction using machine learning algorithm to apply ETE of information theory to the financial investment field.주식 시장은 경제 분야의 중요한 부분으로 광범위하게 연구되고 있다. 특히, 주식 시장의 구성 요소들인 주식 가격과 그 수익률의 관계를 예측하고 분석하는 연구는 투자자들이 최적 투자 전략을 세우기 위해 중요한 과업 중 하나이다. 이러한 맥락에서, 어떠한 시스템의 구성 요소들 간의 관계를 분석하는 데 있어 전이 엔트로피(Transfer entropy)는 비모수 지표로써 상관 관계나 그레인저-인과관계에 비해 요소 간 통계적 설명력을 확인하기에 용이하다. 주식 가격의 예측과 이를 통한 최적 자산 배분 전략에 대한 연구 또한 전통적인 선형 모델부터 최신의 머신 러닝 모델의 적용까지 다양하게 연구되고 있다. 본 학위논문의 목적은 경제물리학과 정보이론 분야에서 사용되는 효율적 전이 엔트로피(Effective transfer entropy, ETE)를 이용하여 미국 주식 시장에서 시장 구성 요소 간 발생하는 정보 흐름의 특징을 파악하여 시장의 특성을 나타낼 수 있는 시장 설명력 있는 ETE 기반의 네트워크 지표를 도출하고, 이 네트워크 지표의 사용이 다양한 머신 러닝 알고리즘을 통한 주가 방향 예측에서 성능 향상을 가져다 주는 지에 대해 연구한다. 나아가, 시장 설명력 있는 ETE 네트워크 지표의 구조적 특징과 머신 러닝 알고리즘을 통한 주가 방향 예측 결과를 투자자 관점을 고려한 최적 포트폴리오 구성 전략인 블랙-리터만 모형(Black-Litterman model)에 적용하여 결과적으로 정보 이론과 머신 러닝 기법을 이용한 실제 투자 전략 활용성에 대해 연구한다. 먼저, 미국 주식 시장의 주요 금융 위기들과 주식들 간의 통계적 설명력을 ETE를 통해 분석함으로써 3개월과 6개월 이동창을 기반으로 하는 ETE가 미국 주식 시장에 대해 설명력 있는 지표임을 확인했다. 해당 지표가 주요 금융 위기에서 그 값이 커지고, ETE 네트워크 지표의 시계열 분석을 통해 각 금융위기에서 해당 금융 위기와 관련된 섹터들이 다른 섹터들에 통계적 설명력이 있는 것을 확인했다. 다음으로, 로지스틱 회귀(Logistic regression, LR), 다층 퍼셉트론(Multilayer perceptron, MLP), 랜덤 포레스트(Random forest, RF), XGBoost(XGB) 및 Long short-term memory network(LSTM)의 5개 머신 러닝 알고리즘에 대해 ETE 네트워크 지표가 새로운 변수로 추가되었을 때 주가 방향 예측에 대한 예측 성능이 향상되는 것을 확인했다. 한편, 예측 모델의 예측 성능 평가에 대한 지표로 금융 분야에서 쓰이는 위험 조정 수익률로부터 도출한 수정 정확도 활용을 제안했고, 이 평가 지표를 이용한 분석을 통해 해당 5개 모델 중 MLP와 LSTM이 미국 주식 시장에 대한 주가 방향 예측에서 더 적합한 모델임을 확인했다. 마지막으로, 시장 설명력 있는 유입 및 유출 ETE 네트워크 지표의 특징과 머신 러닝 알고리즘을 이용한 주가 방향 예측 결과를 블랙-리터만 모형의 투자자 관점에 적용하여, 머신 러닝 알고리즘을 이용한 주가 방향 예측 결과를 투자자 관점에 적용한 블랙-리터만 포트폴리오는 시장 포트폴리오와 시장 인덱스보다 나은 위험 대비 수익률을 보이고, ETE 네트워크 지표를 적용한 블랙-리터만 포트폴리오는 가장 높은 수익률을 보임을 확인했다. ETE와 주가 방향 예측의 사용이 투자 수익률 향상으로 이어지고, 예측 성능을 향상시키면 투자 수익률도 함께 증가하는 결과를 활용하여 투자자들이 ETE와 머신 러닝을 활용한 블랙-리터만 모형을 통해 수익을 극대화 할 수 있는 투자 전략을 수립할 수 있는 가능성에 대해 확인했다. 본 학위논문은 정보 이론의 ETE를 금융 투자 분야에 적용할 수 있도록, 머신 러닝 알고리즘을 이용한 주가 방향 예측과 블랙-리터만 모형을 통한 최적 포트폴리오 구성 전략에 대한 첫 번째 연구이다.Chapter 1 Introduction 1 1.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . 7 Chapter 2 Literature Review 9 2.1 Analysis of transfer entropy . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Stock price prediction based on machine learning . . . . . . . . . . . 12 2.3 The Black-Litterman model . . . . . . . . . . . . . . . . . . . . . . . 17 Chapter 3 Effective transfer entropy analysis for the US market 21 3.1 Effective transfer entropy . . . . . . . . . . . . . . . . . . . . . . . . 21 3.2 Data and experiment set-ups . . . . . . . . . . . . . . . . . . . . . . 24 3.2.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2.2 Experiment set-ups . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.1 Overall analysis of Effective transfer entropy . . . . . . . . . 31 3.3.2 Sector analysis of Effective transfer entropy . . . . . . . . . . 33 3.4 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 40 Chapter 4 Predicting the direction of US stock prices using ETE and machine learning techniques 45 4.1 Machine learning algorithms . . . . . . . . . . . . . . . . . . . . . . . 45 4.1.1 Logistic regression . . . . . . . . . . . . . . . . . . . . . . . . 45 4.1.2 Multi-layer perceptron . . . . . . . . . . . . . . . . . . . . . . 45 4.1.3 Random forest . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.1.4 Extreme gradient boosting . . . . . . . . . . . . . . . . . . . 47 4.1.5 Long short-term memory network . . . . . . . . . . . . . . . 48 4.1.6 Adjusted accuracy . . . . . . . . . . . . . . . . . . . . . . . . 49 4.2 Data and experiment set-ups . . . . . . . . . . . . . . . . . . . . . . 51 4.2.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2.2 Experiment set-ups . . . . . . . . . . . . . . . . . . . . . . . . 51 4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.3.1 Prediction performance in different models . . . . . . . . . . 57 4.3.2 Prediction performance in different sectors . . . . . . . . . . . 66 4.4 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 78 Chapter 5 The Black-Litterman model for ETE and machine learning 81 5.1 The Black-Litterman model . . . . . . . . . . . . . . . . . . . . . . . 81 5.2 Data and experiment set-ups . . . . . . . . . . . . . . . . . . . . . . 84 5.2.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.2.2 Experiment set-ups . . . . . . . . . . . . . . . . . . . . . . . . 85 5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.3.1 Prediction performance in different models and sectors . . . . 91 5.3.2 Portfolio performances for cumulative return . . . . . . . . . 91 5.4 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 97 Chapter 6 Conclusion 103 6.1 Contributions and Limitations . . . . . . . . . . . . . . . . . . . . . 103 6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Bibliography 107 국문초록 125Docto

Applications of Genetic Programming to Finance and Economics: Past, Present, Future

While the origins of Genetic Programming (GP) stretch back over fifty years, the field of GP was invigorated by John Koza’s popularisation of the methodology in the 1990s. A particular feature of the GP literature since then has been a strong interest in the application of GP to real-world problem domains. One application domain which has attracted significant attention is that of finance and economics, with several hundred papers from this subfield being listed in the Genetic Programming Bibliography. In this article we outline why finance and economics has been a popular application area for GP and briefly indicate the wide span of this work. However, despite this research effort there is relatively scant evidence of the usage of GP by the mainstream finance community in academia or industry. We speculate why this may be the case, describe what is needed to make this research more relevant from a finance perspective, and suggest some future directions for the application of GP in finance and economics

AI Applications to Power Systems

Today, the flow of electricity is bidirectional, and not all electricity is centrally produced in large power plants. With the growing emergence of prosumers and microgrids, the amount of electricity produced by sources other than large, traditional power plants is ever-increasing. These alternative sources include photovoltaic (PV), wind turbine (WT), geothermal, and biomass renewable generation plants. Some renewable energy resources (solar PV and wind turbine generation) are highly dependent on natural processes and parameters (wind speed, wind direction, temperature, solar irradiation, humidity, etc.). Thus, the outputs are so stochastic in nature. New data-science-inspired real-time solutions are needed in order to co-develop digital twins of large intermittent renewable plants whose services can be globally delivered

On Explainable Deep Learning for Macroeconomic Forecasting and Finance

Deep Learning (DL) has gained momentum in recent years due to its incredible generalisation performance achieved across many learning tasks. Nevertheless, practitioners and academics have sometime been reluctant to apply these models because perceived as black boxes. This is particularly problematic in Economics and Finance. The objective of this thesis is to develop interpretable DL models and explainable DL tools with a focus on macroeconomic and financial applications. In doing so we highlight connections between such models and the standard economic ones. The first part of this work introduces a new class of interpretable models called Deep Dynamic Factor Models. The study merges the DL literature on autoencoders with that of the Econometrics on Dynamic Factor Models. Empirical validations of the approach are carried out both on synthetic and on real-time macroeconomic data. Part two of the work analyses feature attribution methods and Shapley values among explainability tools that are used to additively decompose model predictions. One of their limitations is highlighted, given that it is necessary to define a baseline that represents the missingness of a feature. A solution to the problem is proposed and compared against the ones currently in use both on simulated data and in the financial context of credit card default. We show that the proposed baseline is the only one that accounts for the specific use of the model. The final part of the work discusses the use of DL techniques for dynamic asset allocation. Using US market data, a comparison in recursive out-of-sample among different machine learning, economic-financial and hybrid models, including the one introduced in the first part of the work, is performed. Finally, a nonlinear factor-based portfolio performance attribution via the use of Shapley values and the baseline proposed in part two of the work is presented

Risk Management using Model Predictive Control

Forward planning and risk management are crucial for the success of any system or business dealing with the uncertainties of the real world. Previous approaches have largely assumed that the future will be similar to the past, or used simple forecasting techniques based on ad-hoc models. Improving solutions requires better projection of future events, and necessitates robust forward planning techniques that consider forecasting inaccuracies. This work advocates risk management through optimal control theory, and proposes several techniques to combine it with time-series forecasting. Focusing on applications in foreign exchange (FX) and battery energy storage systems (BESS), the contributions of this thesis are three-fold. First, a short-term risk management system for FX dealers is formulated as a stochastic model predictive control (SMPC) problem in which the optimal risk-cost profiles are obtained through dynamic control of the dealers’ positions on the spot market. Second, grammatical evolution (GE) is used to automate non-linear time-series model selection, validation, and forecasting. Third, a novel measure for evaluating forecasting models, as a part of the predictive model in finite horizon optimal control applications, is proposed. Using both synthetic and historical data, the proposed techniques were validated and benchmarked. It was shown that the stochastic FX risk management system exhibits better risk management on a risk-cost Pareto frontier compared to rule-based hedging strategies, with up to 44.7% lower cost for the same level of risk. Similarly, for a real-world BESS application, it was demonstrated that the GE optimised forecasting models outperformed other prediction models by at least 9%, improving the overall peak shaving capacity of the system to 57.6%

Risk Management using Model Predictive Control

Forward planning and risk management are crucial for the success of any system or business dealing with the uncertainties of the real world. Previous approaches have largely assumed that the future will be similar to the past, or used simple forecasting techniques based on ad-hoc models. Improving solutions requires better projection of future events, and necessitates robust forward planning techniques that consider forecasting inaccuracies. This work advocates risk management through optimal control theory, and proposes several techniques to combine it with time-series forecasting. Focusing on applications in foreign exchange (FX) and battery energy storage systems (BESS), the contributions of this thesis are three-fold. First, a short-term risk management system for FX dealers is formulated as a stochastic model predictive control (SMPC) problem in which the optimal risk-cost profiles are obtained through dynamic control of the dealers’ positions on the spot market. Second, grammatical evolution (GE) is used to automate non-linear time-series model selection, validation, and forecasting. Third, a novel measure for evaluating forecasting models, as a part of the predictive model in finite horizon optimal control applications, is proposed. Using both synthetic and historical data, the proposed techniques were validated and benchmarked. It was shown that the stochastic FX risk management system exhibits better risk management on a risk-cost Pareto frontier compared to rule-based hedging strategies, with up to 44.7% lower cost for the same level of risk. Similarly, for a real-world BESS application, it was demonstrated that the GE optimised forecasting models outperformed other prediction models by at least 9%, improving the overall peak shaving capacity of the system to 57.6%

High-Performance Modelling and Simulation for Big Data Applications

This open access book was prepared as a Final Publication of the COST Action IC1406 “High-Performance Modelling and Simulation for Big Data Applications (cHiPSet)“ project. Long considered important pillars of the scientific method, Modelling and Simulation have evolved from traditional discrete numerical methods to complex data-intensive continuous analytical optimisations. Resolution, scale, and accuracy have become essential to predict and analyse natural and complex systems in science and engineering. When their level of abstraction raises to have a better discernment of the domain at hand, their representation gets increasingly demanding for computational and data resources. On the other hand, High Performance Computing typically entails the effective use of parallel and distributed processing units coupled with efficient storage, communication and visualisation systems to underpin complex data-intensive applications in distinct scientific and technical domains. It is then arguably required to have a seamless interaction of High Performance Computing with Modelling and Simulation in order to store, compute, analyse, and visualise large data sets in science and engineering. Funded by the European Commission, cHiPSet has provided a dynamic trans-European forum for their members and distinguished guests to openly discuss novel perspectives and topics of interests for these two communities. This cHiPSet compendium presents a set of selected case studies related to healthcare, biological data, computational advertising, multimedia, finance, bioinformatics, and telecommunications