Reinforcement Learning Applied to Trading Systems: A Survey

Financial domain tasks, such as trading in market exchanges, are challenging and have long attracted researchers. The recent achievements and the consequent notoriety of Reinforcement Learning (RL) have also increased its adoption in trading tasks. RL uses a framework with well-established formal concepts, which raises its attractiveness in learning profitable trading strategies. However, RL use without due attention in the financial area can prevent new researchers from following standards or failing to adopt relevant conceptual guidelines. In this work, we embrace the seminal RL technical fundamentals, concepts, and recommendations to perform a unified, theoretically-grounded examination and comparison of previous research that could serve as a structuring guide for the field of study. A selection of twenty-nine articles was reviewed under our classification that considers RL's most common formulations and design patterns from a large volume of available studies. This classification allowed for precise inspection of the most relevant aspects regarding data input, preprocessing, state and action composition, adopted RL techniques, evaluation setups, and overall results. Our analysis approach organized around fundamental RL concepts allowed for a clear identification of current system design best practices, gaps that require further investigation, and promising research opportunities. Finally, this review attempts to promote the development of this field of study by facilitating researchers' commitment to standards adherence and helping them to avoid straying away from the RL constructs' firm ground.Comment: 38 page

Developing A Multi-Agent and Self-Adaptive Framework with Deep Reinforcement Learning for Dynamic Portfolio Risk Management

Deep or reinforcement learning (RL) approaches have been adapted as reactive agents to quickly learn and respond with new investment strategies for portfolio management under the highly turbulent financial market environments in recent years. In many cases, due to the very complex correlations among various financial sectors, and the fluctuating trends in different financial markets, a deep or reinforcement learning based agent can be biased in maximising the total returns of the newly formulated investment portfolio while neglecting its potential risks under the turmoil of various market conditions in the global or regional sectors. Accordingly, a multi-agent and self-adaptive framework namely the MASA is proposed in which a sophisticated multi-agent reinforcement learning (RL) approach is adopted through two cooperating and reactive agents to carefully and dynamically balance the trade-off between the overall portfolio returns and their potential risks. Besides, a very flexible and proactive agent as the market observer is integrated into the MASA framework to provide some additional information on the estimated market trends as valuable feedbacks for multi-agent RL approach to quickly adapt to the ever-changing market conditions. The obtained empirical results clearly reveal the potential strengths of our proposed MASA framework based on the multi-agent RL approach against many well-known RL-based approaches on the challenging data sets of the CSI 300, Dow Jones Industrial Average and S&P 500 indexes over the past 10 years. More importantly, our proposed MASA framework shed lights on many possible directions for future investigation.Comment: Accepted by The 23rd International Conference on Autonomous Agents and Multi-Agent System

Artificial Intelligence and Machine Learning Approaches to Energy Demand-Side Response: A Systematic Review

Recent years have seen an increasing interest in Demand Response (DR) as a means to provide flexibility, and hence improve the reliability of energy systems in a cost-effective way. Yet, the high complexity of the tasks associated with DR, combined with their use of large-scale data and the frequent need for near real-time de-cisions, means that Artificial Intelligence (AI) and Machine Learning (ML) — a branch of AI — have recently emerged as key technologies for enabling demand-side response. AI methods can be used to tackle various challenges, ranging from selecting the optimal set of consumers to respond, learning their attributes and pref-erences, dynamic pricing, scheduling and control of devices, learning how to incentivise participants in the DR schemes and how to reward them in a fair and economically efficient way. This work provides an overview of AI methods utilised for DR applications, based on a systematic review of over 160 papers, 40 companies and commercial initiatives, and 21 large-scale projects. The papers are classified with regards to both the AI/ML algorithm(s) used and the application area in energy DR. Next, commercial initiatives are presented (including both start-ups and established companies) and large-scale innovation projects, where AI methods have been used for energy DR. The paper concludes with a discussion of advantages and potential limitations of reviewed AI techniques for different DR tasks, and outlines directions for future research in this fast-growing area

Enhancing portfolio management using artificial intelligence: literature review

Building an investment portfolio is a problem that numerous researchers have addressed for many years. The key goal has always been to balance risk and reward by optimally allocating assets such as stocks, bonds, and cash. In general, the portfolio management process is based on three steps: planning, execution, and feedback, each of which has its objectives and methods to be employed. Starting from Markowitz's mean-variance portfolio theory, different frameworks have been widely accepted, which considerably renewed how asset allocation is being solved. Recent advances in artificial intelligence provide methodological and technological capabilities to solve highly complex problems, and investment portfolio is no exception. For this reason, the paper reviews the current state-of-the-art approaches by answering the core question of how artificial intelligence is transforming portfolio management steps. Moreover, as the use of artificial intelligence in finance is challenged by transparency, fairness and explainability requirements, the case study of post-hoc explanations for asset allocation is demonstrated. Finally, we discuss recent regulatory developments in the European investment business and highlight specific aspects of this business where explainable artificial intelligence could advance transparency of the investment process

Systematic Trading: Calibration Advances through Machine Learning

Systematic trading in finance uses computer models to define trade goals, risk controls and rules that can execute trade orders in a methodical way. This thesis investigates how performance in systematic trading can be crucially enhanced by both i) persistently reducing the bid-offer spread quoted by the trader through optimized and realistically backtested strategies and ii) improving the out-of-sample robustness of the strategy selected through the injection of theory into the typically data-driven calibration processes. While doing so it brings to the foreground sound scientific reasons that, for the first time to my knowledge, technically underpin popular academic observations about the recent nature of the financial markets. The thesis conducts consecutive experiments across strategies within the three important building blocks of systematic trading: a) execution, b) quoting and c) risk-reward allowing me to progressively generate more complex and accurate backtested scenarios as recently demanded in the literature (Cahan et al. (2010)). The three experiments conducted are: 1. Execution: an execution model based on support vector machines. The first experiment is deployed to improve the realism of the other two. It analyses a popular model of execution: the volume weighted average price (VWAP). The VWAP algorithm targets to split the size of an order along the trading session according to the expected intraday volume's profile since the activity in the markets typically resembles convex seasonality – with more activity around the open and the closing auctions than along the rest of the day. In doing so, the main challenge is to provide the model with a reasonable expected profile. After proving in my data sample that two simple static approaches to the profile overcome the PCA-ARMA from Bialkowski et al. (2008) (a popular two-fold model composed by a dynamic component around an unsupervised learning structure) a further combination of both through an index based on supervised learning is proposed. The Sample Sensitivity Index hence successfully allows estimating the expected volume's profile more accurately by selecting those ranges of time where the model shall be less sensitive to past data through the identification of patterns via support vector machines. Only once the intraday execution risk has been defined can the quoting policy of a mid-frequency (in general, up to a week) hedging strategy be accurately analysed. 2. Quoting: a quoting model built upon particle swarm optimization. The second experiment analyses for the first time to my knowledge how to achieve the disruptive 50% bid-offer spread discount observed in Menkveld (2013) without increasing the risk profile of a trading agent. The experiment depends crucially on a series of variables of which market impact and slippage are typically the most difficult to estimate. By adapting the market impact model in Almgren et al. (2005) to the VWAP developed in the previous experiment and by estimating its slippage through its errors' distribution a framework within which the bid-offer spread can be assessed is generated. First, a full-replication spread, (that set out following the strict definition of a product in order to hedge it completely) is calculated and fixed as a benchmark. Then, by allowing benefiting from a lower market impact at the cost of assuming deviation risk (tracking error and tail risk) a non-full-replication spread is calibrated through particle swarm optimization (PSO) as in Diez et al. (2012) and compared with the benchmark. Finally, it is shown that the latter can reach a discount of a 50% with respect to the benchmark if a certain number of trades is granted. This typically occurs on the most liquid securities. This result not only underpins Menkveld's observations but also points out that there is room for further reductions. When seeking additional performance, once the quoting policy has been defined, a further layer with a calibrated risk-reward policy shall be deployed. 3. Risk-Reward: a calibration model defined within a Q-learning framework. The third experiment analyses how the calibration process of a risk-reward policy can be enhanced to achieve a more robust out-of-sample performance – a cornerstone in quantitative trading. It successfully gives a response to the literature that recently focusses on the detrimental role of overfitting (Bailey et al. (2013a)). The experiment was motivated by the assumption that the techniques underpinned by financial theory shall show a better behaviour (a lower deviation between in-sample and out-of-sample performance) than the classical data-driven only processes. As such, both approaches are compared within a framework of active trading upon a novel indicator. The indicator, called the Expectations' Shift, is rooted on the expectations of the markets' evolution embedded in the dynamics of the prices. The crucial challenge of the experiment is the injection of theory within the calibration process. This is achieved through the usage of reinforcement learning (RL). RL is an area of ML inspired by behaviourist psychology concerned with how software agents take decisions in an specific environment incentivised by a policy of rewards. By analysing the Q-learning matrix that collects the set of state/actions learnt by the agent within the environment, defined by each combination of parameters considered within the calibration universe, the rationale that an autonomous agent would have learnt in terms of risk management can be generated. Finally, by then selecting the combination of parameters whose attached rationale is closest to that of the portfolio manager a data-driven solution that converges to the theory-driven solution can be found and this is shown to successfully outperform out-of-sample the classical approaches followed in Finance. The thesis contributes to science by addressing what techniques could underpin recent academic findings about the nature of the trading industry for which a scientific explanation was not yet given: • A novel agent-based approach that allows for a robust out-of-sampkle performance by crucially providing the trader with a way to inject financial insights into the generally data-driven only calibration processes. It this way benefits from surpassing the generic model limitations present in the literature (Bailey et al. (2013b), Schorfheid and Wolpin (2012), Van Belle and Kerr (2012) or Weiss and Kulikowski (1991)) by finding a point where theory-driven patterns (the trader's priors tend to enhance out-of-sample robustness) merge with data-driven ones (those that allow to exploit latent information). • The provision of a technique that, to the best of my knowledge, explains for the first time how to reduce the bid-offer spread quoted by a traditional trader without modifying her risk appetite. A reduction not previously addressed in the literature in spite of the fact that the increasing regulation against the assumption of risk by market makers (e.g. Dodd–Frank Wall Street Reform and Consumer Protection Act) does yet coincide with the aggressive discounts observed by Menkveld (2013). As a result, this thesis could further contribute to science by serving as a framework to conduct future analyses in the context of systematic trading. • The completion of a mid-frequency trading experiment with high frequency execution information. It is shown how the latter can have a significant effect on the former not only through the erosion of its performance but, more subtly, by changing its entire strategic design (both, optimal composition and parameterization). This tends to be highly disregarded by the financial literature. More importantly, the methodologies disclosed herein have been crucial to underpin the setup of a new unit in the industry, BBVA's Global Strategies & Data Science. This disruptive, global and cross-asset team gives an enhanced role to science by successfully becoming the main responsible for the risk management of the Bank's strategies both in electronic trading and electronic commerce. Other contributions include: the provision of a novel risk measure (flowVaR); the proposal of a novel trading indicator (Expectations’ Shift); and the definition of a novel index that allows to improve the estimation of the intraday volume’s profile (Sample Sensitivity Index)

Dynamic portfolio rebalancing through reinforcement learning

Portfolio managements in financial markets involve risk management strategies and opportunistic responses to individual trading behaviours. Optimal portfolios constructed aim to have a minimal risk with highest accompanying investment returns, regardless of market conditions. This paper focuses on providing an alternative view in maximising portfolio returns using Reinforcement Learning (RL) by considering dynamic risks appropriate to market conditions through dynamic portfolio rebalancing. The proposed algorithm is able to improve portfolio management by introducing the dynamic rebalancing of portfolios with vigorous risk through an RL agent. This is done while accounting for market conditions, asset diversifications, risk and returns in the global financial market. Studies have been performed in this paper to explore four types of methods with variations in fully portfolio rebalancing and gradual portfolio rebalancing, which combine with and without the use of the Long Short-Term Memory (LSTM) model to predict stock prices for adjusting the technical indicator centring. Performances of the four methods have been evaluated and compared using three constructed financial portfolios, including one portfolio with global market index assets with different risk levels, and two portfolios with uncorrelated stock assets from different sectors and risk levels. Observed from the experiment results, the proposed RL agent for gradual portfolio rebalancing with the LSTM model on price prediction outperforms the other three methods, as well as returns of individual assets in these three portfolios. The improvements of the returns using the RL agent for gradual rebalancing with prediction model are achieved at about 27.9–93.4% over those of the full rebalancing without prediction model. It has demonstrated the ability to dynamically adjust portfolio compositions according to the market trends, risks and returns of the global indices and stock assets

Machine Learning and Finance: A Review using Latent Dirichlet Allocation Technique (LDA)

The aim of this paper is provide a first comprehensive structuring of the literature applying machine learning to finance. We use a probabilistic topic modelling approach to make sense of this diverse body of research spanning across the disciplines of finance, economics, computer sciences, and decision sciences. Through the topic modelling approach, a Latent Dirichlet Allocation Technique (LDA), we can extract the 14 coherent research topics that are the focus of the 6,148 academic articles during the years 1990-2019 analysed. We first describe and structure these topics, and then further show how the topic focus has evolved over the last two decades. Our study thus provides a structured topography for finance researchers seeking to integrate machine learning research approaches in their exploration of finance phenomena. We also showcase the benefits to finance researchers of the method of probabilistic modelling of topics for deep comprehension of a body of literature, especially when that literature has diverse multi-disciplinary actors

A superior active portfolio optimization model for stock exchange

Due to the vast number of stocks and the multiple appearances of developing investment portfolios, investors in the financial market face multiple investment opportunities. In this regard, the investor task becomes extremely difficult as investors define their preferences for expected return and the amount to which they want to avoid potential investment risks. This research attempts to design active portfolios that outperform the performance of the appropriate market index. To achieve this aim, technical analysis and optimization procedures were used based on a hybrid model. It combines the strong features of the Markowitz model with the General Reduced Gradient (GRG) algorithm to maintain a good compromise between diversification and exploitation. The proposed model is used to construct an active portfolio optimization model for the Iraq Stock Exchange (ISX) for the period from January 2010 to February 2020. This is applied to all 132 companies registered on the exchange. In addition to the market portfolio, two methods, namely, Equal Weight (EW) and Markowitz were used to generate active portfolios to compare the research findings. After a thorough review based on the Sharpe ratio criterion, the suggested model demonstrated its robustness, resulting in maximizing earnings with low risks