Heinz recently completed a comprehensive experiment in self-play using the FRITZ chess engine to establish the ‘decreasing returns’ hypothesis with specific levels of statistical confidence. This note revisits the results and recalculates the confidence levels of this and other hypotheses. These appear to be better than Heinz’ initial analysis suggests

Haworth, Guy McCrossan

English

Central Archive at the University of Reading

Self-play: Statistical Significance SELF-PLAY: STATISTICAL SIGNIFICANCE   G.M cC. Haworth1  UK   ABSTRACT  Heinz recently completed a comprehensive experiment in self-play using the FRITZ chess engine to establish the ‘decreasing returns’ hypothesis with specific levels of statistical confidence. This note revisits the results and recalculates the confidence levels of this and other hypotheses. These appear to be better than Heinz’ initial analysis suggests.   1. INTRODUCTION  Heinz (2000a/b, 2001a/b) reported an extensive self-play experiment with FRITZ to investigate the widely-held belief that an increment of search-depth gives decreasing benefit as depth increases. After a ‘White v Black’ calibration match between identical players, some seven ‘alternating colours’ matches were staged in which one player had one ply advantage in fixed iteration depth p, although both sides had search extension capabil-ity. The experiments included 3,000 games per match, taking the depth p beyond 9 plies to a record 12 plies.   This note offers an analysis of the statistical significance of the results which extends Heinz’ own analysis, dif-fers somewhat from it, and provides confidence levels for: • Heinz’ null hypothesis, given the approximating binary win/loss model he adopted, • Heinz’ null hypothesis based on a ternary win/draw/loss model, and • two subsidiary null hypotheses, on win- and draw-rates, suggested by the data.  Section 2 tables Heinz’ results, section 3 establishes the relevant mathematics, and section 4 analyses the ex-perimental results. The summary mentions some open questions that remain.   2. THE SELF-PLAY EXPERIMENT  Let Pp be a player whose search strategy has a fixed iteration depth parameter of p plies. Let Mp be a match be-tween Pp and Pp-1. The one match Mp of n games in Heinz’ experiment results in an observed score Spo for Pp and an observed number - Wpo, Dpo and Lpo – of, respectively, Pp wins, draws and losses: n = 3,000. Table 1 gives this data, and the derived observed Pp score, win, draw and loss frequencies, namely spo,wpo, dpo and lpo.   p M p W po D po L po S po n w po d po l po s po ELO pCal Mcal: P8W-P8B 924 1,288 788 1,568.0 3,000 0.3080 0.4293 0.2627 0.5227 Cal6 M6: P6-P5 1,686 915 399 2,143.5 3,000 0.5620 0.3050 0.1330 0.7145 +159 67 M7: P7-P6 1,643 1,066 291 2,176.0 3,000 0.5477 0.3553 0.0970 0.7253 +169 78 M8: P8-P7 1,457 1,212 331 2,063.0 3,000 0.4857 0.4040 0.1103 0.6877 +137 89 M9: P9-P8 1,433 1,235 332 2,050.5 3,000 0.4777 0.4117 0.1107 0.6835 +134 910 M10: P10-P9 1,252 1,451 297 1,977.5 3,000 0.4173 0.4837 0.0990 0.6592 +115 1011 M11: P11-P10 1,124 1,525 351 1,886.5 3,000 0.3747 0.5083 0.1170 0.6288 +92 1112 M12: P12-P11 1,059 1,592 349 1,855.0 3,000 0.3530 0.5307 0.1163 0.6183 +84 12Table 1. Results of the Heinz self-play experiment.                                                            1 33, Alexandra Rd., Reading, Berkshire, RG1 5PG, UK. email: guy.haworth@laposte.net  7th Computer Olympiad Workshop July 2002                                                            3. UNDERLYING MATHEMATICS  Let the corresponding underlying, intrinsic Pp score, win, draw and loss frequencies in Mp be sp, wp, dp and lp respectively. These are unknown, and treated as if they were random variables Sp, Wp, Dp and Lp whose distri-butions are to be estimated as well as possible from the observed data. wp, dp and lp are binary variables, while sp is a ternary variable which can be approximated by a binary variable. The experiment was carefully designed to ensure that the matches Mp were a sequence of independent Bernoulli trials of these random variables.   Let B be a sequence of n Bernoulli trials of a binary, random variable X with two values 0/1: X = 1 with prob-ability x, X = 0 with probability 1-x. The following notation and results are standard (Feller, 1968):  E(X) ≡ mean µ ≡ ΣP(v).v = P(1).1 = x.1 = x; E(X2) ≡ ΣP(v).v2 = P(1).12 = x.12 = x  Var(X) ≡ E((X – µ)2) = E(X2) - 2 µ .E(X) + µ2 = E(X2) - µ2 = x – x2 = x(1 – x)    Var(nX) ≡ n.Var(X) = n.x(1 – x) for n ≥ 0  Var(-X) = Var(X)  If Y is a second, independent random variable in the trial, Var(X ± Y) = Var(X) + Var(Y)  σ(X) ≡ standard deviation(X) ≡ sqrt Var(X) = sqrt[x(1 – x)]  σ(nX) ≡ standard deviation(nX) ≡ sqrt Var(nX) = sqrt[n.x(1 – x)]  As n → ∞, the distribution of nX → a normal distribution φ, mean n.x, standard dev. σ = sqrt[n.x(1 – x)]  Let xo = x ± ε.σ(x) ≈ x: then Prob[rate xo is observed | mean x] ≈ φ(ε) and → φ(ε) as n → ∞  But x = xo ± ε.σ(x) ≈ xo ± ε.σ(xo) as σ(xo) ≈ σ(x). Therefore, Prob[rate x is observed | mean xo] ≈ φ(ε)   3.1 Principles of Statistical Inference  If the initial, neutral, ‘know nothing’ assumption is made, namely, that any value in the interval (0, 1) is equally likely, then the following Bayesian relation holds between the underlying mean x and the observed rate xo:  Prob[mean of X is ~x | rate xo is observed] ∝ Prob[rate xo is observed | mean of X is ~x]  But, if xo ≈ x, Prob[rate xo is observed | mean of X is x] ≈ Prob[rate x is observed | mean is xo]. Therefore,  Prob[mean of X is ~x | rate xo is observed] → Prob[rate x observed | mean is ~xo] as n → ∞.  Therefore, in the limit, the distribution of the random variable E(X), the mean of the random variable X, is the same as the distribution of X, assuming the observed rate xo is the mean.   Note that, as n = 3,000, generously large, n.x(1-x) > 290 for all frequencies that occurred in the experiment. This is certainly large enough to ensure that: • observed frequency rates are close to the intrinsic success rates for the random variables, • the normal distribution φ((X-µ)/σ) is a suitable approximation to the actual probability density functions, • the theory above may be applied to the experiment, knowing that approximation errors are ignorable.  3.2 Comparison of Binary and Ternary Models  Rather than analyse match M results with a ternary win/draw/loss model with random variables W and D, Heinz used a binary model with a two-valued random variable S: Pr(win, S = 1) = s. Thus, as above:  E(nS) = n.s and Var(nS) = n.s(1 – s) However, if S is a 3-valued random variable with Pr(S = 1) = w and Pr(draw) = d, then:  E(S)  = s = w + d/2; E(S2) = w + d/4  Var(nS) = n.Var(S) = n.[E(S2) – (w + d/2)2] = n.[(w + d/4) – (w + d/2)2] =    = n.[(s – d/4) – s2] = n.[s(1-s) – d/4] ≤ n.s(1-s), the variance of the equivalent binary model. Thus, the variance in the ternary model is always less than in the binary model2. Heinz’ substitution of the bi-nary model was therefore indeed a conservative one, increasing the apparent variances in his experiment. 2 In the extreme, for example, it is 0 rather than 0.25 for d = 1 and one trial. Self-play: Statistical Significance 4. ANALYSIS OF THE SELF-PLAY RESULTS  Heinz’ null hypothesis, S, is effectively that “sp and therefore Sp reduce as p increases”. The confidence level with which this statement may be made is the estimated probability of this being demonstrated again by a fur-ther match Mp of the same length.  Crucially, Heinz did not use the independence of Si and Sj: Var(Si - Sj) = Var(Si) + Var(Sj) = σ2 and Si - Sj = kijσ. The value of k determines the confidence level with which we might say that Si > Sj. i  \  j 6 7 8 9 10 11 12 6 7 8 9 10 11 126 -0.934 2.271 2.619 4.629 7.095 7.941 17.5% 98.8% 99.6% 100.0% 100.0% 100.0%7 ——— 3.206 3.554 5.566 8.036 8.884 ——— 99.9% 100.0% 100.0% 100.0% 100.0%8 ——— ——— 0.348 2.355 4.813 5.656 ——— ——— 63.6% 99.1% 100.0% 100.0%9 ——— ——— ——— 2.007 4.473 5.316 ——— ——— ——— 97.8% 100.0% 100.0%10 ——— ——— ——— ——— 2.455 3.295 ——— ——— ——— ——— 99.3% 100.0%11 ——— ——— ——— ——— ——— 0.839 ——— ——— ——— ——— ——— 79.9%(Si – Sj)/σ(Si – Sj): binary model Confidence Levels for the null hypothesis S  on scores Table 2a. (Si – Sj) in units of σ(Si – Sj) and confidence levels for the null hypothesis: binary model.  The distribution of the random variable Sp in the ternary model also converges on a normal distribution by the Central Limit Theorem. As proved above, Var(Si) and Var(Si – Sj) = Var(Si) + Var(Sj) are lower in this model and Heinz’ null hypothesis S may be stated with somewhat more confidence.   i  \  j 6 7 8 9 10 11 12 6 7 8 9 10 11 126 -1.216 2.991 3.458 6.299 9.699 10.958 11.2% 99.9% 100.0% 100.0% 100.0% 100.0%7 ——— 4.357 4.843 7.832 11.354 12.677 ——— 100.0% 100.0% 100.0% 100.0% 100.0%8 ——— ——— 0.479 3.347 6.869 8.150 ——— ——— 68.4% 100.0% 100.0% 100.0%9 ——— ——— ——— 2.860 6.388 7.667 ——— ——— ——— 99.8% 100.0% 100.0%10 ——— ——— ——— ——— 3.625 4.915 ——— ——— ——— ——— 100.0% 100.0%11 ——— ——— ——— ——— ——— 1.256 ——— ——— ——— ——— ——— 89.5%(Si – Sj)/σ(Si – Sj): ternary model Confidence Levels for the null hypothesis S  on scores       Table 2b. (Si – Sj) in units of σ(Si – Sj) and confidence levels for the null hypothesis: ternary model.  Clearly, we can state with 99.9%+ confidence that the winner’s score decreases in the two match sequences {M6, M8, M10, M12} and {M7, M9, M11}.  The null hypotheses W, “Wp reduces as p increases”, and D, “Dp increases as p increases”, are suggested by the data: confidence levels have also been calculated for these statements. Both Wp and Dp are binary variables. Curiously, there is no obvious trend in the corresponding binary variable Lp.  i  \  j 6 7 8 9 10 11 12 6 7 8 9 10 11 126 1.117 5.937 6.561 11.327 14.803 16.618 86.8% 100.0% 100.0% 100.0% 100.0% 100.0%7 ——— 4.814 5.437 10.189 13.648 15.452 ——— 100.0% 100.0% 100.0% 100.0% 100.0%8 ——— ——— 0.620 5.331 8.738 10.508 ——— ——— 73.2% 100.0% 100.0% 100.0%9 ——— ——— ——— 4.708 8.111 9.877 ——— ——— ——— 100.0% 100.0% 100.0%10 ——— ——— ——— ——— 3.382 5.131 ——— ——— ——— ——— 100.0% 100.0%11 ——— ——— ——— ——— ——— 1.745 ——— ——— ——— ——— ——— 95.9%(Wi – Wj)/σ(Wi – Wj) Confidence Levels for the null hypothesis W on wins Table 3. (Wi – Wj) in units of σ(Wi – Wj) and confidence levels for the null hypothesis.  Again, the two match sequences {M6, M8, M10, M12} and {M7, M9, M11} demonstrate the null hypothesis with high levels of confidence, as do {M7, M8} and {M9, M10, M11, M12}.    7th Computer Olympiad Workshop July 2002 i  \  j 6 7 8 9 10 11 12 6 7 8 9 10 11 126 -4.151 -8.059 -8.669 -14.402 -16.387 -18.204 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%7 ——— -3 .889 -4.495 -10.158 -12.108 -13.888 ——— 100.0% 100.0% 100.0% 100.0% 100.0%8 ——— ——— -0.604 -6.230 -8.158 -9.913 ——— ——— 72.7% 100.0% 100.0% 100.0%9 ——— ——— ——— -5.623 -7.547 -9.299 ——— ——— ——— 100.0% 100.0% 100.0%10 ——— ——— ——— ——— -1.911 -3.645 ——— ——— ——— ——— 97.2% 100.0%11 ——— ——— ——— ——— ——— -1.732 ——— ——— ——— ——— ——— 95.8%(Di – Dj)/σ(Di – Dj) Confidence Levels for the null hypothesis D on draws Table 4. (Di – Dj) in units of σ(Di – Dj) and confidence levels for the null hypothesis.  For a third time, the two match sequences {M6, M8, M10, M12} and {M7, M9, M11} demonstrate with high levels of confidence, the null hypothesis that the rate of draws increases with p. The sequences {M7, M8} and {M9, M10, M11, M12} are equally encouraging.   5. SUMMARY  This note has revisited Heinz’ comprehensive self-play experiment in chess and re-assessed the significance of his results for the null hypothesis “an increment of search-depth yields decreasing returns”. It appears that the confidence levels with which this may be stated are better than as suggested in his original paper.  Ancillary null hypotheses, on the decreasing frequency of wins and the increasing frequency of draws, were also suggested by the data and hold with good levels of confidence.  Some open questions that arose during this investigation: • to what extent are the characteristics of the chosen engine, FRITZ, affecting the experimental results? • why is there no comparable trend in lp, the underlying mean of Lp? • given a possible stm/sntm position evaluation bias, should Ms be compared with Ms-2 rather than Ms-1?  • does the one ply search-depth advantage initially deliver increasing benefits with depth, c.f. M6 and M7? • are there other perhaps increasing-return trends in shallow-search matches, e.g., M2-M7? • as p → ∞, does sp decrease to a limit greater, equal to or less than 0.5?  The apparently anomalous M6/M7 results suggest further investigation at the conveniently shallow end of the experiment.  Were Heinz’ experimental results here a ‘one off’ or does the one-ply advantage increasingly overcome random play at small depths of search?  Thanks goes to Ernst Heinz for his comprehensive self-play experiment and results.   6. REFERENCES  Feller, W. (1968). An Introduction to Probability Theory and Its Applications. 3rd Edition. J. Wiley and Sons: Series in Probability and Mathematical Statistics. ISBN 0-4712-5708-7. Heinz, E.A. (2000a). Scalable Search in Computer Chess. Vieweg Verlag. ISBN 3-5280-5732-7, esp. p. 178. Heinz, E.A. (2000b). A New Self-Play Experiment in Computer Chess. Technical Memo No. 608 (MIT-LCS-TM-608). Laboratory or Computer Science, Massachussetts Institute of Technology. Heinz, E.A. (2001a). New Self-play Results in Computer Chess. Proceedings of the 2nd International Confer-ence on Computers and Games (eds. I. Frank and T.A. Marsland), pp. 267-280. Lecture Notes in Computer Science (eds. G. Goos, J. Harmanis and J. van Leeuwen), Springer Verlag. ISBN 3-540-43080-6. Heinz, E.A. (2001b). Self-play, Deep Search and Diminishing Returns. ICGA Journal, Vol. 24, No. 2, pp. 75-79. ISSN 1389-6911.  

A New Self-Play Experiment in Computer Chess.

An Introduction to Probability Theory and Its Applications. 3rd Edition.

Deep Search and Diminishing Returns.

New Self-play Results in Computer Chess.

Scalable Search in Computer Chess. Vieweg Verlag.

Self-play: statistical significance

Self-Play: Statistical Significance 115NOTE  SELF-PLAY: STATISTICAL SIGNIFICANCE   G.M cC. Haworth1 Reading, UK   ABSTRACT  Heinz recently completed a comprehensive experiment in self-play using the FRITZ chess engine to establish the ‘decreasing returns’ hypothesis with specific levels of statistical confidence. This note revisits the results and recalculates the confidence levels of this and other hypotheses. These appear to be better than Heinz’ initial analysis suggests.   1. INTRODUCTION  Heinz (2000a/b, 2001a/b) reported an extensive self-play experiment with FRITZ to investigate the widely-held belief that an increment of search-depth gives decreasing benefit as depth increases. After a ‘White v Black’ calibration match between identical players, some seven ‘alternating colours’ matches were staged in which one player had one ply advantage in fixed iteration depth p, although both sides had search extension capability. The experiments included 3,000 games per match, taking the depth p beyond 9 plies to a record 12 plies.   This note offers an analysis of the statistical significance of the results which extends Heinz’ own analysis, differs somewhat from it, and provides confidence levels for: • Heinz’ null hypothesis, given the approximating binary win/loss model he adopted, • Heinz’ null hypothesis based on a ternary win/draw/loss model, and • two subsidiary null hypotheses, on win- and draw-rates, suggested by the data.  Section 2 tables Heinz’ results, section 3 establishes the relevant mathematics, and section 4 analyses the experimental results. The summary mentions some open questions that remain.   2. THE SELF-PLAY EXPERIMENT  Let Pp be a player whose search strategy has a fixed iteration depth parameter of p plies. Let Mp be a match between Pp and Pp-1. The one match Mp of n games in Heinz’ experiment results in an observed score Spo for Pp and an observed number - Wpo, Dpo and Lpo – of, respectively, Pp wins, draws and losses: n = 3,000. Table 1 gives this data, and the derived observed Pp score, win, draw and loss frequencies, namely spo,wpo, dpo and lpo. p M p W po D po L po S po n w po d po l po s po ELO pCal Mcal: P8W-P8B 924 1,288 788 1,568.0 3,000 0.3080 0.4293 0.2627 0.5227 Cal6 M6: P6-P5 1,686 915 399 2,143.5 3,000 0.5620 0.3050 0.1330 0.7145 +159 67 M7: P7-P6 1,643 1,066 291 2,176.0 3,000 0.5477 0.3553 0.0970 0.7253 +169 78 M8: P8-P7 1,457 1,212 331 2,063.0 3,000 0.4857 0.4040 0.1103 0.6877 +137 89 M9: P9-P8 1,433 1,235 332 2,050.5 3,000 0.4777 0.4117 0.1107 0.6835 +134 910 M10: P10-P9 1,252 1,451 297 1,977.5 3,000 0.4173 0.4837 0.0990 0.6592 +115 1011 M11: P11-P10 1,124 1,525 351 1,886.5 3,000 0.3747 0.5083 0.1170 0.6288 +92 1112 M12: P12-P11 1,059 1,592 349 1,855.0 3,000 0.3530 0.5307 0.1163 0.6183 +84 12 Table 1: Results of the Heinz self-play experiment.                                                           1 33, Alexandra Rd., Reading, Berkshire, RG1 5PG, UK. email: guy.haworth@laposte.net  ICGA Journal June 2003 116 3. UNDERLYING MATHEMATICS  Let the corresponding underlying, intrinsic Pp score, win, draw and loss frequencies in Mp be sp, wp, dp and lp respectively. These are unknown, and are treated as if they were random variables Sp, Wp, Dp and Lp whose distributions are to be estimated as well as possible from the observed data. wp, dp and lp are binary variables, while sp is a ternary variable which can be approximated by a binary variable. The experiment was carefully designed to ensure that the matches Mp were a sequence of independent Bernoulli trials of these random variables.   Let B be a sequence of n Bernoulli trials of a binary, random variable X with two values 0/1: X = 1 with probability x, X = 0 with probability 1-x. The following notation and results are standard (Feller, 1968):  E(X) ≡ mean µ ≡ ΣP(v).v = P(1).1 = x.1 = x;  E(X2) ≡ ΣP(v).v2 = P(1).12 = x.12 = x  Var(X) ≡ E((X – µ)2) = E(X2) - 2 µ .E(X) + µ2 = E(X2) - µ2 = x – x2 = x(1 – x)    Var(nX) ≡ n.Var(X) = n.x(1 – x) for n ≥ 0  Var(-X) = Var(X)  If Y is a second, independent random variable in the trial, Var(X ± Y) = Var(X) + Var(Y)  σ(X) ≡ standard deviation(X) ≡ sqrt Var(X) = sqrt[x(1 – x)]  σ(nX) ≡ standard deviation(nX) ≡ sqrt Var(nX) = sqrt[n.x(1 – x)]  As n → ∞, the distribution of nX → a normal distribution φ, mean n.x, standard dev. σ = sqrt[n.x(1 – x)]  Let xo = x ± ε.σ(x) ≈ x: then Prob[rate xo is observed | mean x] ≈ φ(ε) and → φ(ε) as n → ∞  But x = xo ± ε.σ(x) ≈ xo ± ε.σ(xo) as σ(xo) ≈ σ(x). Therefore, Prob[rate x is observed | mean xo] ≈ φ(ε)   3.1 Principles of Statistical Inference  If the initial, neutral, ‘know nothing’ assumption is made, namely, that any value in the interval (0, 1) is equally likely, then the following Bayesian relation holds between the underlying mean x and the observed rate xo:  Prob[mean of X is ~x | rate xo is observed] ∝ Prob[rate xo is observed | mean of X is ~x]  But, if xo ≈ x, Prob[rate xo is observed | mean of X is x] ≈ Prob[rate x is observed | mean is xo]. Therefore,  Prob[mean of X is ~x | rate xo is observed] → Prob[rate x observed | mean is ~xo] as n → ∞.  Therefore, in the limit, the distribution of the random variable E(X), the mean of the random variable X, is the same as the distribution of X, assuming the observed rate xo is the mean.   Note that, as n = 3,000, generously large, n.x(1-x) > 290 for all frequencies that occurred in the experiment. This is certainly large enough to ensure that: • observed frequency rates are close to the intrinsic success rates for the random variables, • the normal distribution φ((X-µ)/σ) is a suitable approximation to the actual probability density functions, • the theory above may be applied to the experiment, knowing that approximation errors are ignorable.  3.2 Comparison of Binary and Ternary Models  Rather than analyse match M results with a ternary win/draw/loss model with random variables W and D, Heinz used a binary model with a two-valued random variable S: Pr(win, S = 1) = s. Thus, as above:  E(nS) = n.s and Var(nS) = n.s(1 – s)  However, if S is a 3-valued random variable with Pr(S = 1) = w and Pr(draw) = d, then:  E(S)  = s = w + d/2; E(S2) = w + d/4  Var(nS) = n.Var(S) = n.[E(S2) – (w + d/2)2] = n.[(w + d/4) – (w + d/2)2] =    = n.[(s – d/4) – s2] = n.[s(1-s) – d/4] ≤ n.s(1-s), the variance of the equivalent binary model.  Thus, the variance in the ternary model is always less than in the binary model2. Heinz’ substitution of the binary model was therefore indeed a conservative one, increasing the apparent variances in his experiment.                                                             2 In the extreme, for example, it is 0 rather than 0.25 for d = 1 and one trial. Self-Play: Statistical Significance 1174. ANALYSIS OF THE SELF-PLAY RESULTS  Heinz’ null hypothesis, S, is effectively that “sp and therefore Sp reduce as p increases”. The confidence level with which this statement may be made is the estimated probability of this being demonstrated again by a further match Mp of the same length. (see Table 2a for the binary model).  Crucially, Heinz did not use the independence of Si and Sj: Var(Si - Sj) = Var(Si) + Var(Sj) = σ2 and Si - Sj = kijσ. The value of k determines the confidence level with which we might say that Si > Sj. i  \  j 6 7 8 9 10 11 12 6 7 8 9 10 11 126 -0.934 2.271 2.619 4.629 7.095 7.941 17.5% 98.8% 99.6% 100.0% 100.0% 100.0%7 ——— 3.206 3.554 5.566 8.036 8.884 ——— 99.9% 100.0% 100.0% 100.0% 100.0%8 ——— ——— 0.348 2.355 4.813 5.656 ——— ——— 63.6% 99.1% 100.0% 100.0%9 ——— ——— ——— 2.007 4.473 5.316 ——— ——— ——— 97.8% 100.0% 100.0%10 ——— ——— ——— ——— 2.455 3.295 ——— ——— ——— ——— 99.3% 100.0%11 ——— ——— ——— ——— ——— 0.839 ——— ——— ——— ——— ——— 79.9%(Si – Sj)/σ(Si – Sj): binary model Confidence Levels for the null hypothesis S  on scores Table 2a: (Si – Sj) in units of σ(Si – Sj) and confidence levels for the null hypothesis: binary model.  The distribution of the random variable Sp in the ternary model also converges on a normal distribution by the Central Limit Theorem. As proved above, Var(Si) and Var(Si – Sj) = Var(Si) + Var(Sj) are lower in this model and Heinz’ null hypothesis S may be stated with somewhat more confidence. (see Table 2b).  i  \  j 6 7 8 9 10 11 12 6 7 8 9 10 11 126 -1.216 2.991 3.458 6.299 9.699 10.958 11.2% 99.9% 100.0% 100.0% 100.0% 100.0%7 ——— 4.357 4.843 7.832 11.354 12.677 ——— 100.0% 100.0% 100.0% 100.0% 100.0%8 ——— ——— 0.479 3.347 6.869 8.150 ——— ——— 68.4% 100.0% 100.0% 100.0%9 ——— ——— ——— 2.860 6.388 7.667 ——— ——— ——— 99.8% 100.0% 100.0%10 ——— ——— ——— ——— 3.625 4.915 ——— ——— ——— ——— 100.0% 100.0%11 ——— ——— ——— ——— ——— 1.256 ——— ——— ——— ——— ——— 89.5%(Si – Sj)/σ(Si – Sj): ternary model Confidence Levels for the null hypothesis S  on scores Table 2b: (Si – Sj) in units of σ(Si – Sj) and confidence levels for the null hypothesis: ternary model.  Clearly, we can state with 99.9%+ confidence that the winner’s score decreases in the two match sequences {M6, M8, M10, M12} and {M7, M9, M11}.  The null hypotheses W, “Wp reduces as p increases”, and D, “Dp increases as p increases”, are suggested by the data: confidence levels have also been calculated for these statements (see Tables 3 and 4). Both Wp and Dp are binary variables. Curiously, there is no obvious trend in the corresponding binary variable Lp.   i  \  j 6 7 8 9 10 11 12 6 7 8 9 10 11 126 1.117 5.937 6.561 11.327 14.803 16.618 86.8% 100.0% 100.0% 100.0% 100.0% 100.0%7 ——— 4.814 5.437 10.189 13.648 15.452 ——— 100.0% 100.0% 100.0% 100.0% 100.0%8 ——— ——— 0.620 5.331 8.738 10.508 ——— ——— 73.2% 100.0% 100.0% 100.0%9 ——— ——— ——— 4.708 8.111 9.877 ——— ——— ——— 100.0% 100.0% 100.0%10 ——— ——— ——— ——— 3.382 5.131 ——— ——— ——— ——— 100.0% 100.0%11 ——— ——— ——— ——— ——— 1.745 ——— ——— ——— ——— ——— 95.9%(Wi – Wj)/σ(Wi – Wj) Confidence Levels for the null hypothesis W on wins Table 3: (Wi – Wj) in units of σ(Wi – Wj) and confidence levels for the null hypothesis.  Again, the two match sequences {M6, M8, M10, M12} and {M7, M9, M11} demonstrate the null hypothesis with high levels of confidence, as do {M7, M8} and {M9, M10, M11, M12}.   ICGA Journal June 2003 118 i  \  j 6 7 8 9 10 11 12 6 7 8 9 10 11 126 -4.151 -8.059 -8.669 -14.402 -16.387 -18.204 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%7 ——— -3 .889 -4.495 -10.158 -12.108 -13.888 ——— 100.0% 100.0% 100.0% 100.0% 100.0%8 ——— ——— -0.604 -6.230 -8.158 -9.913 ——— ——— 72.7% 100.0% 100.0% 100.0%9 ——— ——— ——— -5.623 -7.547 -9.299 ——— ——— ——— 100.0% 100.0% 100.0%10 ——— ——— ——— ——— -1.911 -3.645 ——— ——— ——— ——— 97.2% 100.0%11 ——— ——— ——— ——— ——— -1.732 ——— ——— ——— ——— ——— 95.8%(Di – Dj)/σ(Di – Dj) Confidence Levels for the null hypothesis D on draws Table 4: (Di – Dj) in units of σ(Di – Dj) and confidence levels for the null hypothesis.  For a third time, the two match sequences {M6, M8, M10, M12} and {M7, M9, M11} demonstrate with high levels of confidence, the null hypothesis that the rate of draws increases with p. The sequences {M7, M8} and {M9, M10, M11, M12} are equally encouraging.   5. SUMMARY  This note has revisited Heinz’ comprehensive self-play experiment in chess and re-assessed the significance of his results for the null hypothesis “an increment of search depth yields decreasing returns”. It appears that the confidence levels with which this may be stated are better than as suggested in his original paper.  Ancillary null hypotheses, on the decreasing frequency of wins and the increasing frequency of draws, were also suggested by the data and hold with good levels of confidence.  Some open questions that arose during this investigation: • to what extent are the characteristics of the chosen engine, FRITZ, affecting the experimental results? • why is there no comparable trend in lp, the underlying mean of Lp? • given a possible stm/sntm position evaluation bias, should Ms be compared with Ms-2 rather than Ms-1?  • does the one ply search-depth advantage initially deliver increasing benefits with depth, c.f. M6 and M7? • are there other perhaps increasing-return trends in shallow-search matches, e.g., M2-M7? • as p → ∞, does sp decrease to a limit greater, equal to or less than 0.5?  The apparently anomalous M6/M7 results suggest further investigation at the conveniently shallow end of the experiment.  Were Heinz’ experimental results here a ‘one off’ or does the one-ply advantage increasingly overcome random play at small depths of search?  Thanks goes to Ernst Heinz for his comprehensive self-play experiment and results.   6. REFERENCES  Feller, W. (1968). An Introduction to Probability Theory and Its Applications. 3rd Edition. J. Wiley and Sons: Series in Probability and Mathematical Statistics. ISBN 0-4712-5708-7. Heinz, E.A. (2000a). Scalable Search in Computer Chess. Vieweg Verlag. Wiesbaden, esp. p. 178. ISBN 3-5280-5732-7.  Heinz, E.A. (2000b). A New Self-Play Experiment in Computer Chess. Technical Memo No. 608 (MIT-LCS-TM-608). Laboratory or Computer Science, Massachussetts Institute of Technology. Heinz, E.A. (2001a). New Self-play Results in Computer Chess. Proceedings of the 2nd International Conference on Computers and Games (eds. I. Frank and T.A. Marsland), pp. 267-280. Lecture Notes in Computer Science (eds. G. Goos, J. Harmanis and J. van Leeuwen), Springer-Verlag, Berlin. ISBN 3-540-43080-6. Heinz, E.A. (2001b). Self-play, Deep Search and Diminishing Returns. ICGA Journal, Vol. 24, No. 2, pp. 75-79. ISSN 1389-6911. 

Self-play: statistical significance

Abstract

Similar works

Full text

Available Versions

Central Archive at the University of Reading

Central Archive at the University of Reading