2,438 research outputs found
Regime-switching recurrent reinforcement learning for investment decision making
This paper presents the regime-switching recurrent reinforcement learning (RSRRL) model and describes its application to investment problems. The RSRRL is a regime-switching extension of the recurrent reinforcement learning (RRL) algorithm. The basic RRL model was proposed by Moody and Wu (Proceedings of the IEEE/IAFE 1997 on Computational Intelligence for Financial Engineering (CIFEr). IEEE, New York, pp 300-307 1997) and presented as a methodology to solve stochastic control problems in finance. We argue that the RRL is unable to capture all the intricacies of financial time series, and propose the RSRRL as a more suitable algorithm for such type of data. This paper gives a description of two variants of the RSRRL, namely a threshold version and a smooth transition version, and compares their performance to the basic RRL model in automated trading and portfolio management applications. We use volatility as an indicator/transition variable for switching between regimes. The out-of-sample results are generally in favour of the RSRRL models, thereby supporting the regime-switching approach, but some doubts exist regarding the robustness of the proposed models, especially in the presence of transaction cost
Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning
Pair trading is one of the most effective statistical arbitrage strategies
which seeks a neutral profit by hedging a pair of selected assets. Existing
methods generally decompose the task into two separate steps: pair selection
and trading. However, the decoupling of two closely related subtasks can block
information propagation and lead to limited overall performance. For pair
selection, ignoring the trading performance results in the wrong assets being
selected with irrelevant price movements, while the agent trained for trading
can overfit to the selected assets without any historical information of other
assets. To address it, in this paper, we propose a paradigm for automatic pair
trading as a unified task rather than a two-step pipeline. We design a
hierarchical reinforcement learning framework to jointly learn and optimize two
subtasks. A high-level policy would select two assets from all possible
combinations and a low-level policy would then perform a series of trading
actions. Experimental results on real-world stock data demonstrate the
effectiveness of our method on pair trading compared with both existing pair
selection and trading methods.Comment: 10 pages, 6 figure
Predictive Crypto-Asset Automated Market Making Architecture for Decentralized Finance using Deep Reinforcement Learning
The study proposes a quote-driven predictive automated market maker (AMM)
platform with on-chain custody and settlement functions, alongside off-chain
predictive reinforcement learning capabilities to improve liquidity provision
of real-world AMMs. The proposed AMM architecture is an augmentation to the
Uniswap V3, a cryptocurrency AMM protocol, by utilizing a novel market
equilibrium pricing for reduced divergence and slippage loss. Further, the
proposed architecture involves a predictive AMM capability, utilizing a deep
hybrid Long Short-Term Memory (LSTM) and Q-learning reinforcement learning
framework that looks to improve market efficiency through better forecasts of
liquidity concentration ranges, so liquidity starts moving to expected
concentration ranges, prior to asset price movement, so that liquidity
utilization is improved. The augmented protocol framework is expected have
practical real-world implications, by (i) reducing divergence loss for
liquidity providers, (ii) reducing slippage for crypto-asset traders, while
(iii) improving capital efficiency for liquidity provision for the AMM
protocol. To our best knowledge, there are no known protocol or literature that
are proposing similar deep learning-augmented AMM that achieves similar capital
efficiency and loss minimization objectives for practical real-world
applications.Comment: 20 pages, 6 figures, 1 algorith
Automating Vehicles by Deep Reinforcement Learning using Task Separation with Hill Climbing
Within the context of autonomous driving a model-based reinforcement learning
algorithm is proposed for the design of neural network-parameterized
controllers. Classical model-based control methods, which include sampling- and
lattice-based algorithms and model predictive control, suffer from the
trade-off between model complexity and computational burden required for the
online solution of expensive optimization or search problems at every short
sampling time. To circumvent this trade-off, a 2-step procedure is motivated:
first learning of a controller during offline training based on an arbitrarily
complicated mathematical system model, before online fast feedforward
evaluation of the trained controller. The contribution of this paper is the
proposition of a simple gradient-free and model-based algorithm for deep
reinforcement learning using task separation with hill climbing (TSHC). In
particular, (i) simultaneous training on separate deterministic tasks with the
purpose of encoding many motion primitives in a neural network, and (ii) the
employment of maximally sparse rewards in combination with virtual velocity
constraints (VVCs) in setpoint proximity are advocated.Comment: 10 pages, 6 figures, 1 tabl
Defy the Game: Automated Market Making using Deep Reinforcement Learning
Automated market makers have gained popularity in the financial market for their ability to provide
liquidity without needing a centralized intermediary (market maker). However, they suffer from the
problems of slippage and impermanent loss, which can lead to losses for both liquidity providers and takers.
This work implements a pseudo-arbitrage rule to solve the impermanent loss issues related to arbitrage
opportunities. The mechanism implements a trusted external oracle to get the market conditions, put them
on the automated market maker, and match the bonding curve to them. Next, the application of a Double
Deep Q-Learning reinforcement learning algorithm is proposed to reduce these issues in automated market
makers. The algorithm adjusts the curvature of the bonding curve function to adapt to market conditions
quickly. This work describes the model, the simulation environment used to learn and test the proposed
approach, and the metrics used to evaluate its performance. Finally, it explains the results of the experiments
and analysis of their implications. The approach shows promise in reducing slippage and impermanent loss
and recommending improvements and future works
- …