D'ERAMO, CARLO

NUARA, ALESSANDRO

RESTELLI, MARCELLO

Archivio istituzionale della ricerca - Politecnico di Milano

Estimating Maximum Expected Value through Gaussian ApproximationA. ProofsTheorem 1. The proof of Theorem 1 follows directly fromobserving that µˆWE⇤ is always smaller than µˆME⇤ . In fact,the ME estimator can be seen as a weighted estimatorthat gives probability one to the variable associated to thelargest sample mean µˆi, so that any other weighting cannotproduce a larger value.Theorem 2. If we compare the expected value of DE re-ported in Equation (4) with the value of the estimatorWE inEquation (3), we can notice strong similarities. The maindifference is that in DE the sample mean of variable Xiand its probability of being the maximum are computedw.r.t. two independent set of samples, while in WE thesetwo quantities are positively correlated. It follows thatWE has a positive bias w.r.t. DE.Theorem 3. Starting from the definition of WE (3), we canderive the bound to the variance as followsVar µˆWE⇤ = Var MXi=1µˆi(S)wSi! Var MXi=1µˆi(S)!=MXi=1Var (µˆi(S)) ,where the inequality is a consequence of the maximizationof each weight wSi with one and the last equality comesfrom the independence of the sample means.Theorem 4. Since the weights wi computed by DWE arenot random variables, it followsVar µˆDWE⇤ = Var MXi=1µˆi(S)wi!=MXi=1w2iVar (µˆi(S)) maxi21,...,M 2i|Si| ,where the inequality is motivated by w2i  1, 8i.B. ForexThe indicators chosen to define the states are: Moving Av-erage Convergence/Divergence indicator, Relative StrengthIndex, Momentum, Channel Commodity Index, StochasticOscillator, Bollinger Bands, Moving Average Cross-OverThe actions suggested by the indicators are computed bysetting the parameters and the entry-exit conditions follow-ing the most used and common rules for these indicators.In particular all the signals are defined using implemented0 200 400 600 800 1,000 1,200 1,400-10%-5%0%5%10%15%20%25%DaysCumulativeProfitQ-LearningDouble Q-LearningW Q-LearningW Q-Learning with W-policyFigure 9. Cumulative profit in test set after 30 training episodes.Results are averaged over 100 experiments.Matlab Financial Toolbox functions and setting parametersand conditions like below.- Moving Average Covergence/Divergence indicator: theMACD is calculated by subtracting the 26-period exponen-tial moving average from the 12-period moving average. A9 period exponential moving average is used as signal line.When the MACD falls below the signal line a long positionsignal is produced. Otherwise if the MACD surpasses thesignal line a short position signal is produced.- Relative Strength Index: the period chosen for the RSI is20 days. When the RSI is under the value of 30 an oversoldmarket condition occurs, so a long position signal is pro-duced. Similarly, when the RSI is over 70, a short positionsignal is generated. When the value is between these twobounds a close position signal is produced.- Momentum: the momentum uses a period of 14 days. Thestrategy is to be always in the market. In particular if thevalue of the momentum is less than zero a long positionsignal is generated. Otherwise a short position signal isgenerated.- Channel Commodity Index: the CCI is used with a 20days period. The long position signal is produced whenthe CCI cross and surpasses the lower bound of -100. Theshort position is taken when the CCI falls behind the valueof 100. Otherwise the close position signal is generated.- Stochastic Oscillator: the Stochastic oscillator uses thehigh prices, low prices, and close prices with a 14 daysperiod for the %K line and 3 for the %D line. If both linesare above the value of 80 and the %K line falls behind %Dline, then a short position signal is generated. If both linesEstimating Maximum Expected Value through Gaussian Approximationare below 20 and the %K line surpasses the %D line, a longposition signal is generated. In other conditions a closeposition signal is generated.- Bollinger Bands: the Bollinger Bands signal is producedwith a window size of 20. Long position signal is producedwhen the closing price surpasses the upper Bollinger Band.When the closing price falls down the lower band, then ashort position signal is taken. Otherwise a close positionsignal is generated.- Moving Average Cross-Over: we used two moving aver-age of 20 and 200 periods. A long position signal is pro-duced when the 20 period moving average falls behind the200 periods one. Otherwise a short position signal is gen-erated.In Figure 9, we can compare the cumulative rewards of thealgorithms on the test set after 30 training episodes. Ob-serving the ranges of the values, we can see that the DoubleQ-learning agent gains and loses less than the other agents,so we can deduce that it entries in the market less often.

Estimating Maximum Expected Value through Gaussian Approximation

https://re.public.polimi.it/retrieve/handle/11311/1005717/159014/deramo16-supp.pdf

Estimating Maximum Expected Value through Gaussian Approximation

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Politecnico di Milano