Barbero, F.

Browne, C.

English

This paper proposes Heuristic Sampling (HS) for generating self-play trials for games with a defined state evaluation function, with speeds comparable to random playouts but game length estimates comparable to those produced by intelligent AI agents. HS produces plausible results up to thousands of times faster than more rigorous methods

Maastricht University Research Portal

   Heuristic Sampling for Fast Plausible PlayoutsCitation for published version (APA):Browne, C., & Barbero, F. (2021). Heuristic Sampling for Fast Plausible Playouts. In 2021 IEEECONFERENCE ON GAMES (COG) (pp. 1042-1045). IEEE.https://doi.org/10.1109/COG52621.2021.9619147Document status and date:Published: 01/01/2021DOI:10.1109/COG52621.2021.9619147Document Version:Publisher's PDF, also known as Version of recordDocument license:TavernePlease check the document version of this publication:• A submitted manuscript is the version of the article upon submission and before peer-review. There canbe important differences between the submitted version and the official published version of record.People interested in the research are advised to contact the author for the final version of the publication,or visit the DOI to the publisher's website.• The final author version and the galley proof are versions of the publication after peer review.• The final published version features the final layout of the paper including the volume, issue and pagenumbers.Link to publicationGeneral rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyrightowners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with theserights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.• You may not further distribute the material or use it for any profit-making activity or commercial gain• You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above,please follow below link for the End User Agreement:www.umlib.nl/taverne-licenseTake down policyIf you believe that this document breaches copyright please contact us at:repository@maastrichtuniversity.nlproviding details and we will investigate your claim.Download date: 02 Oct. 2023Heuristic Sampling for Fast Plausible PlayoutsCameron Browne and Fabio BarberoDepartment of Data Science and Knowledge EngineeringMaastricht UniversityMaastricht, The Netherlandscameron.browne@maastrichtuniversity.nl, fabio.barbero@student.maastrichtuniversity.nlAbstract—This paper proposes Heuristic Sampling (HS) forgenerating self-play trials for games with a defined state evalu-ation function, with speeds comparable to random playouts butgame length estimates comparable to those produced by intelli-gent AI agents. HS produces plausible results up to thousandsof times faster than more rigorous methods.Index Terms—AI, Board Games, Game Length, LudiiI. INTRODUCTIONThis short note addresses a simple and practical question:what is the fastest way to reliably estimate the average lengthof games that will occur between humans playing a given ruleset? Game length – along with balance and decisiveness – hasemerged as a key indicator of abstract board game quality inautomated game evaluation [1], [2]. Overly short game lengthsindicate that the rule set may be trivial, flawed or subjectto easily obtained winning strategies, while excessive gamelengths indicate that the rule set may be poorly tuned, rewardstalling play, or simply not converge to a result.Both extremes are usually warning signs of serious prob-lems in a rule set (with some exceptions), and while gamelength alone is not sufficient to estimate the quality of a game,it provides a useful filter for quickly identifying flawed cases.A. ContextThis question is important for the Digital Ludeme Project,which aims to reconstruct missing rules of ancient boardgames based on historical evidence and automated evaluationsof potential rule sets that fit the known details [3]. The Ludiigeneral game system [4] implements over 1,000 ludemes(game-related concepts) that can be combined in differentways to define new rule sets, providing many thousands – ormillions – of plausible rule sets to test for each reconstructiontask. It is imperative to have fast and reliable game evaluationmetrics, which ideally operate at the millisecond level.Potential rule sets are evaluated through automated self-playtrials between agents [2]. Random playouts provide the speedrequired but produce unrealistic games involving poor movechoices that no intelligent player would make, hence can givemisleading results. Intelligent playouts by AI agents give moreThis research is funded by the European Research Council as part ofthe Digital Ludeme Project (ERC Consolidator Grant #771292) led byCameron Browne at Maastricht University’s Department of Data Science andKnowledge Engineering.realistic results but are orders of magnitude slower to generateand too slow for practical purposes.Specifically, we want a method that produces an observedmean game length Lobs that approximates the expected meangame length Lexp of a given rule set within a certain tolerance.For the purposes of this exercise, a value of Lobs betweenLexp/2 and 2× Lexp is deemed to be acceptable.In this paper, we present a simple method called HeuristicSampling that produces playouts with plausible game lengthsnot far off those produced by intelligent play but with speedscomparable to random play.II. HEURISTIC SAMPLINGHeuristic Sampling (HS) applies when a heuristic state eval-uation function is available. It involves successively choosingfor each state Si a high-reward action a from the set ofavailable actions Ai and applying it without lookahead search.This may be described as a form of tournament selectionin which the maximal element is chosen from a randomlyselected subset of the available actions for each state.We denote a HS search as HS1/n where 1/n is the fractionof available actions to sample at each state. Specifically:1. for each state Si2. generate the set of available actions Ai3. if |Ai| == 1 return Ai04. randomly select max(2, d |Ai|n e) actions from Ai5. for each selected action a6. apply a to Si to give Sia7. if Sia is a winning state then return a8. else if Sia is a losing state then ignore a9. else evaluate Sia with heuristic function10. return action a with highest seen heuristic estimateHS implicitly handles winning “mate-in-one” [5] or “deci-sive” [6] moves, if such moves are among the randomly chosensubset for their state. It also encourages greater explorationof the game state space than traditional search methods, bynot following the same line of optimal play every time. Thiscan prove beneficial when the aim is to exercise rule sets forevaluation purposes rather than outright playing strength.A. Same-Turn ContinuationIn Ludii, a turn is a consecutive sequence of actions by thesame player [4]. For example, making a line of three in Nine978-1-6654-3886-5/21/$31.00 ©2021 IEEEFig. 1. White makes a line of three in Nine Men’s Morris, then captures a black piece to complete their turn.Men’s Morris allows the mover to then capture an enemy piece(Fig. 1); both actions are part of that player’s turn.Same-Turn Continuation (STC) involves repeatedly apply-ing HS to the current state while it is the same player’s turnto move, so that the turn’s ultimate heuristic estimate is takenat the end of the turn. That is, while ever it remains the sameplayer’s turn to move after an action has been applied, thenanother action from the resulting state should be chosen andapplied. The resulting heuristic estimate at the end of the turnshould then be used as the value for the turn’s first action.Without STC, only the first action in such chained sequenceswould be considered, giving poor results for games in whichstrong moves are contingent upon a (weaker) dependent movebeing made first. STC is similar in principle to quiesencesearch, in which the search continues beyond its nominal depthfor unstable states until a “quiet” state is reached [7].III. EXPERIMENTSThe experiments were conducted using the Ludii generalgame system.1 The heuristic state estimates for each game areprovided by a set of 15 general heuristics implemented forLudii, with weight vectors learnt per game through gradientdescent. The result is that Ludii provides competent but notoverly strong heuristic state evaluations for most games.Experiments were run on the following games:1) Tic-Tac-Toe (Lexp = 9 plies): Standard game on 3×3board, game tree complexity ≈ 105.2) Connect4 (Lexp = 36 plies): Standard game on 6×7board, game tree complexity ≈ 1021.3) English Draughts (Lexp = 70 plies): Standard game,8×8 board, game tree complexity ≈ 1031.4) Nine Men’s Morris (Lexp = 50 plies): Standard game,nine pieces per player, game tree complexity ≈ 1050.5) Halma (plies unknown): Two-player game on a reduced6×6 board, game tree complexity unknown.6) Lines of Action (Lexp = 44 plies): Standard game on8×8 board, game tree complexity ≈ 1064.7) Gomoku (Lexp = 30 plies): Standard game on 15×15board, game tree complexity ≈ 1070.8) Chess (Lexp = 70 plies): Standard game on 8×8 boardusing FIDE rules, game tree complexity ≈ 10123.1Ludii is available at: http://ludii.games9) Shogi (Lexp = 115 plies): Standard game on 9×9 board,game tree complexity ≈ 10226.These games were selected to test the methods over a rangeof game types and complexities. Expected game lengths Lexpwere taken from the “Game Complexity” Wikipedia page.2Note that these expected game lengths are specified in termsof plies whereas Ludii measures game length in terms of turns,i.e. consecutive sequences of moves by the same player; thesewill typically be identical but this is not guaranteed.A. Playout MethodsThe following playout methods were compared:1) Random: Uniformly random playouts.2) HS1/n: Heuristic Sampling with Same-Turn Continua-tion, for sampling ratios 1/2, 1/4 and 1/8.3) ABd: Standard Alpha Beta to depths of d = 1, 2 and 3.AB1 is essentially equivalent to HS1/1 without STC.4) UCT : Standard UCT search with a budget of 1,000iterations per move using uniformly random playoutsand a default exploration constant of C =√2.Each search method was applied to 100 trials of each gamewhere possible (Shogi involved fewer trials due to excessiverun times). A cap of 1,000 turns was applied for all trialsexcept for Random search; trials that exceeded this turn limitwere abandoned as draws and excluded from the sample.IV. RESULTSTable I shows the expected length of each game Lexp (inplies) and the observed length Lobs (in turns) obtained foreach playout method, in addition to the number of completedtrials N , minimum and maximum observed lengths, standarddeviation (SD), standard error (SE) and average time per trialin seconds.Timings were taken on a standard consumer machine withsix 2.9 GHz i9 cores, similar to machines that Ludii usersperforming reconstruction tasks will typically use. Timings aretherefore indicative only and can be multiplied by a factor ofaround ×10 to estimate their serial equivalents.Fig. 2 summarizes these results and highlights the perfor-mance of each playout method against expected length Lexpper game (dotted line). Additional comparisons between HS2https://en.wikipedia.org/wiki/Game_complexityTic-Tac-ToeRandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT0246810Connect4RandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT0816243240English DraughtsRandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT04080120160200Nine Men’s MorrisRandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT04080120160200HalmaRandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT080160240320400Lines of ActionRandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT04080120160200GomokuRandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT0255075100125ChessRandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT0100200300400500ShogiRandomHS    1/8HS    1/4HS    1/2AB  1 AB  2 AB  3 UCT0150300450600750Fig. 2. Mean number of turns per playout for each method applied to each game (dotted lines show expected game length).with and without STC reveal that HS without STC performsalmost no better than Random for games such as Nine Men’sMorris with chained action sequences per turn.V. DISCUSSIONAs expected, the stronger searches (AB3 and UCT ) givebest results but are thousands of times slower than Randomand HS. HS performs almost as well as AB for most cases– sometimes better! – and produces results in the acceptableLexp/2 to 2× Lexp range for all games tested except Chess.Chess is unusual in that many if not most games betweenhuman players end in a resignation before the game is fullyplayed out and recorded game lengths reflect this truncation.HS1/2 generally outperforms HS1/4 and HS1/8 but not inall cases. Smaller sampling ratios may be preferable in caseswhere heuristic state evaluations are expensive to compute.Note that Random playouts can be much longer than HSplayouts due to the undirected nature of purely random play.HS can actually be much faster than Random in such cases!Another surprise is that ABd can produce worse gamelength estimates than ABd−1 in several cases. This may bedue to the “Odd-Even Effect” in AB search, in which oddsearch depths (terminating on the owner’s ply) can give overlyoptimistic results, while even search depths (terminating on theopponent’s ply) can give overly pessimistic results.3VI. CONCLUSIONThe results of this experiment suggest that Heuristic Sam-pling with Same-Turn Continuation can produce playoutswith acceptably plausible game lengths with the speed of3https://www.chessprogramming.org/Odd-Even_Effectrandom playouts – and often faster – thus offering a potentialapproach for testing large numbers of reconstructed rule setsfor obvious flaws within a reasonable time. HS producedresults generally comparable to more rigorous search methodsbut up to thousands of times more quickly.Future work will include evaluating HS over a wider rangeof games and looking more closely at exactly how reliableits game length estimates are for identifying flawed rule sets.Further, the question of what constitutes the range of desirablegame lengths for a given rule set remains open.We will also investigate the use of HS as the playout policyin a Monte Carlo Tree Search framework, to see whether theimproved playout quality (over uniformly random playouts)can produce superior playing strength with fewer iterations.ACKNOWLEDGMENTThanks to the Ludii team (Éric Piette, Matthew Stephensonand especially Dennis Soemers) for help developing the Ludiiframework in which these experiments were run. The Ludiisource code is provided as open source under a CC BY-NC-ND 4.0 licence (https://github.com/Ludeme/Ludii).REFERENCES[1] I. Althöfer, “Computer-Aided Game Inventing,” Friedrich-Schiller Univ.,Faculty Math. Comp. Sci., Jena, Tech. Rep., 2003.[2] C. Browne, “Automatic Generation and Evaluation of RecombinationGames,” Ph.D. dissertation, Queensland University of Technology, 2009.[3] C. Browne, “Modern Techniques for Ancient Games,” in IEEE Conf.Comput. Intell. Games. Maastricht: IEEE Press, 2018, pp. 490–497.[4] É. Piette, D. J. N. J. Soemers, M. Stephenson, C. F. Sironi, M. H. M.Winands, and C. Browne, “Ludii – The Ludemic General Game System,”in Proc. ECAI 2020. IOS Press, 2020, pp. 411–418.[5] R. J. Lorentz, “Improving Monte-Carlo Tree Search in Havannah,” inProc. Comput. and Games, LNCS 6515, Kanazawa, 2010, pp. 105–115.[6] F. Teytaud and O. Teytaud, “On the Huge Benefit of Decisive Movesin Monte-Carlo Tree Search Algorithms,” in Proc. IEEE Symp. Comput.Intell. Games, Dublin, 2010, pp. 359–364.[7] D. Beal, “A Generalised Quiescence Search Algorithm,” Artificial Intel-ligence, vol. 43, pp. 85–98, 1990.TABLE IMEAN NUMBER OF TURNS PER GAME, PER PLAYOUT METHOD.Game Method Lobs N Min Max SD SE Time per TrialRandom 7.62 100 5 9 0.142 0.278 0.000148sHS1/8 6.88 100 5 9 0.132 0.259 0.000621sTic-Tac-Toe HS1/4 7.05 100 5 9 0.139 0.272 0.000640sHS1/2 6.88 100 5 9 0.155 0.303 0.000769sLexp = 9 AB1 7.05 100 5 9 0.185 0.363 0.000583sAB2 9 100 9 9 0.0 0.0 0.000929sAB3 8.22 100 7 9 0.098 0.192 0.00312sUCT 9 100 9 9 0.0 0.0 0.0294sRandom 20.46 100 7 37 0.729 1.428 0.000226sHS1/8 20.02 100 7 42 0.719 1.41 0.000736sConnect4 HS1/4 19.29 100 7 42 0.697 1.37 0.000709sHS1/2 19.81 100 7 42 0.97 1.90 0.000996sLexp = 36 AB1 16.88 100 11 29 0.564 1.11 0.00109sAB2 13 100 13 13 0.0 0.0 0.00250sAB3 22.45 100 7 42 1.204 2.36 0.190sUCT 32.75 100 17 42 0.601 1.178 0.403sRandom 71.3 100 37 161 3.12 6.12 0.00192sHS1/8 70.5 100 35 145 2.54 4.99 0.00572sEnglish Draughts HS1/4 65.2 100 33 197 2.41 4.71 0.00606sHS1/2 70.5 100 37 139 2.25 4.4 0.00943sLexp = 70 AB1 67.2 100 43 105 1.76 3.454 0.00948sAB2 67.0 93 47 87 1.64 3.22 0.00784sAB3 94.0 63 74 158 2.042 4.002 0.418sUCT 194.0 92 53 879 16.424 32.19 13.9sRandom 183.2 100 56 790 11.801 23.13 0.00432sHS1/8 101.0 100 36 431 6.22 12.2 0.00176sNine Men’s Morris HS1/4 95.9 100 36 273 4.57 8.96 0.00215sHS1/2 58.9 100 30 173 2.44 4.78 0.00349sLexp = 50 AB1 170.6 100 43 532 10.2 19.9 0.00491sAB2 41.9 100 20 65 0.909 1.78 0.00481sAB3 175.2 84 44 963 19.9 39.1 0.377sUCT 54.8 100 31 149 2.22 4.34 6.86sRandom 333.0 100 77 923 15.22 29.83 0.00435sHS1/8 70.84 100 36 140 2.09 4.10 0.00163sHalma 6×6 HS1/4 46.1 100 26 97 1.19 2.33 0.00135sHS1/2 37.8 100 20 71 1.011 1.98 0.00133sLexp not known AB1 20.9 100 14 26 0.272 0.532 0.00116sAB2 23.2 45 17 36 0.653 1.28 0.0463sAB3 31.4 23 20 70 2.69 5.27 0.470sUCT 134.83 88 29 922 16.9 33.1 45.5sRandom 208.3 100 49 417 7.734 15.158 0.00896sHS1/8 66.6 100 32 150 2.518 4.94 0.00424sLines of Action HS1/4 38.5 100 23 77 1.02 2.00 0.00353sHS1/2 30.6 100 23 43 0.418 0.82 0.00440sLexp = 44 AB1 28.1 100 22 38 0.307 0.602 0.00762sAB2 43.8 97 22 163 2.13 4.181 0.200sAB3 38.8 98 23 73 0.778 1.53 1.24sUCT 161.0 100 30 518 10.6 20.7 184.4sRandom 112.2 100 52 171 2.64 5.17 0.000353sHS1/8 28.1 100 10 57 0.862 1.69 0.00360sGomoku HS1/4 24.6 100 10 43 0.708 1.39 0.00736sHS1/2 20.8 100 10 43 0.608 1.19 0.0105sLexp = 30 AB1 20.9 100 13 45 0.653 1.28 0.0125sAB2 26.5 100 13 81 1.35 2.65 0.141sAB3 56.9 100 11 113 2.73 5.36 2.28sUCT 41.47 100 20 70 1.12 2.20 0.904sRandom 429.5 100 54 739 13.2 25.9 0.0243sHS1/8 309.1 100 5 516 10.7 21.0 0.0317sChess HS1/4 238.0 100 16 471 11.1 21.7 0.0376sHS1/2 248.6 100 16 435 10.8 21.2 0.0800sLexp = 70 AB1 216.8 100 12 459 10.2 20.1 0.020sAB2 142.3 100 22 501 9.58 18.8 0.564sAB3 144.8 100 24 402 7.92 15.5 6.93sUCT 149.8 100 9 532 11.4 22.3 25.7sRandom 756.4 100 100 2501 68.5 134.3 0.172sHS1/8 187.9 100 75 587 9.5 18.6 0.593sShogi HS1/4 171.4 100 31 459 8.80 17.3 1.18sHS1/2 155.0 100 22 533 9.14 17.9 2.50sLexp = 115 AB1 161.1 100 38 871 10.8 21.2 5.17sAB2 172.3 96 30 728 10.9 21.3 60.1sAB3 94.0 10 38 114 8.26 16.194 751.5sUCT 140.1 10 77 218 14.2 27.7 4,790.9s

Heuristic Sampling for Fast Plausible Playouts

https://cris.maastrichtuniversity.nl/files/142709356/CoG_2021_Heuristic_Sampling.pdf

Heuristic Sampling for Fast Plausible Playouts

Abstract

Similar works

Full text

Available Versions

Maastricht University Research Portal