We propose a long-term memory design for artificial general intelligence based on Solomonoff's incremental machine learning methods. We introduce four synergistic update algorithms that use a Stochastic Context-Free Grammar as a guiding probability distribution of programs. The update algorithms accomplish adjusting production probabilities, re-using previous solutions, learning programming idioms and discovery of frequent subprograms. A controlled experiment with a long training sequence shows that our incremental learning approach is effective. © 2011 Springer-Verlag Berlin Heidelberg

B. Merialdo

J. Schmidhuber

J.E. Hopcroft

L. Levin

M.J. Zaki

R.J. Solomonoff

Özkural, E.

Bilkent University Institutional Repository

Towards Heuristic Algorithmic MemoryEray ÖzkuralBilkent University Computer Engineering DepartmentAnkara, TurkeyAbstract. We propose a long-term memory design for artificial gen-eral intelligence based on Solomonoff’s incremental machine learningmethods. We introduce four synergistic update algorithms that use aStochastic Context-Free Grammar as a guiding probability distributionof programs. The update algorithms accomplish adjusting productionprobabilities, re-using previous solutions, learning programming idiomsand discovery of frequent subprograms. A controlled experiment with along training sequence shows that our incremental learning approach iseffective.1 IntroductionTeramachine is a universal induction system that features integrated long-termmemory, as a candidate for Solomonoff’s “Phase 1 machine” that he proposedto use as the basis of a powerful AGI system called Alpha [1]. We propose anautomatic memory which is recalled appropriately during induction. After eachinduction problem, the solution is stored in the memory, which is a realization ofSolomonoff’s idea of guiding probability density function (pdf) of programs. Thepresent system may be viewed as an advanced version of OOPS [2]. We updatethe guiding pdf after each induction problem so that the heuristic solutions thatwe invent are stored as algorithmic information in our memory system. Hence,our memory design is called Heuristic Algorithmic Memory (HAM).If an induction system’s probability distribution of programs is fixed, thenthe system does not have any real long-term learning ability. We can solve thisproblem by changing the probability distribution so that we extrapolate from thealready invented solution programs, allowing more difficult problems to be solved[3]. Modifying the probability distribution essentially defines an implicit programcode. Thus, after each solution we are implicitly modifying the reference machine.Relative to the implicit universal code, Levin search [4] still has an optimal orderof complexity and is effective for approximating Solomonoff induction [5]. Theextraction of algorithmic information from solutions affords an effective kind oftime-space tradeoff, which works extremely favorably in terms of additional spacerequirement. The successful extraction of each single bit of mutual algorithmicinformation among two problems may potentially result in a speed-up of twofor the latter problem. However, re-using algorithmic information from previoussolutions entails a coding cost which manifests itself as a time penalty duringprogram search (Levin search in our work).J. Schmidhuber, K.R. Thórisson, and M. Looks (Eds.): AGI 2011, LNAI 6830, pp. 382–387, 2011.c© Springer-Verlag Berlin Heidelberg 2011Towards Heuristic Algorithmic Memory 383The reader is referred to [2,1,6] for a background on incremental machinelearning. A longer version of this paper is available on the aRxiV [7], and aprevious version explains the R5RS Scheme grammar which we use [8].2 Stochastic Context-Free Grammar UpdatesA Stochastic Context-Free Grammar (SCFG) is a Context-Free Grammar aug-mented by a probability value on each production. For each head non-terminal,the probabilities of its productions must sum to one. We can extend Levin Searchprocedure to work with a SCFG that assigns probabilities to each sentence inthe language. For this, we need two things, first a generation logic for individualsentences, and second a search strategy to enumerate the sentences that meetthe termination condition of LSearch [2]. In the present system, we use left-most derivation to generate a sentence, intermediate steps are thus left-sententialforms [9, Chapter 5]. The calculation of the a priori probability of a sentencedepends on the fact that in a derivation S ⇒ α1 ⇒ α2 ⇒ ... ⇒ αn whereproductions p1, p2, ..., pn have been applied in order to start symbol S, the prob-ability of the sentence αn is P (αn) =∏1≤i≤n pi. Note that the productions ina derivation are conditionally independent. While this makes it much easier forus to calculate probabilities of sentential forms, it limits the expressive power ofpdf. Note that search algorithm details are beyond the scope of this paper.The most critical part of our design is updating the SCFG so that the dis-covered solutions in a training sequence will be more probable in subsequentsearches. We propose four synergistic update algorithms for HAM. Our SCFGstructure extends the usual productions with production procedures, which dy-namically generate productions.2.1 Modifying Production ProbabilitiesThe simplest kind of update is modifying the probabilities as new solutionsare added to the solution corpus. For this, however, the search algorithm mustsupply the derivation that led to the solution (which we do), or the solution mustbe parsed using the same grammar. Then, the probability for each productionA → β in the solution corpus can be easily calculated by the ratio of frequencyof productions A → β in the solution corpus to the frequency of productions inthe corpus with a head of A. The production procedures are excluded from thisupdate as they can be variant. However, we cannot simply write the probabilitiescalculated this way over the initial probabilities, as initially there will be fewsolutions, and most probabilities will be zero. We use exponential smoothing tosolve this problem:s0 = p0st = αpt + (1 − α)st−1where p0 is the initial probability, pt is the probability in the corpus for problemt, st the smoothed probability for problem t, and α is the smoothing factor. Weused a smoothing factor of 0.125. See [10] for the application of smoothing in asimilar problem. Other methods like Laplace’s rule may be used instead [1].384 E. Özkural2.2 Re-using Previous SolutionsIn the course of a training sequence, the solutions can be incorporated in fullby adding the solutions to the grammar. In the case of Scheme, there could bemany possible implementations. The simplest design is to add all the solutionsto the library of the Scheme interpreter, add a hook non-terminal previous-solutionto the grammar, and then extend the previous-solution with the syntax to call thenew solution. We assume that this syntax is provided in the problem definition.The new solution among other previous solutions is given a probability of γ inthe hope that this solution will be re-used soon, and then the probabilities ofthe old productions of previous-solution are normalized so that they sum to 1− γ.We currently use a γ of 0.5. If it is difficult to add the solutions to the Schemeinterpreter as in our case, then all the solutions can be added as define blocksin the beginning of the program produced, which requires avoiding redundantdefinitions [7].2.3 Learning Programming IdiomsProgrammers do not only learn of concrete solutions to problems, but they alsolearn abstract programs, or program schemas. One way to formalize this is thatthey learn sentential forms. If we can extract appropriate sentential forms, wecan add these to the grammar, as well. We construct the derivation tree from theleftmost derivation, with an obvious algorithm that we will omit. The currentabstraction algorithm starts with the derivation sub-trees rooted at each expres-sion in the current solution. For each derivation sub-tree, we prune the leavesfrom the bottom-up. At each pruning step, an abstract expression is output.The pruning is iterated until a few symbols remain. Every abstract expressionthus found is added to a new non-terminal that contains the abstract expressionsof the current solution with equal probability. The new non-terminal is addedto the top-level non-terminal abstract-expression with 0.5 probability, which isitself one of the productions for expression. These productions may later be mod-ified and used by update algorithms one and two. Note that the orthogonalityof the language helps us in integrating programming idioms into HAM. Thus,several sentential forms are learnt from a single solution in this fashion corre-sponding to different syntactic abstractions. We anticipate that the system willeventually learn complex programming idioms like recursion patterns and dataconstructors.2.4 Frequent Sub-program MiningMining the solution corpus further enhances the guiding probability distribution.Frequent sub-programs in the solution corpus, i.e., sub-programs that occur witha frequency above a given support threshold, can be added again as alternativeproductions to the commonly occurring non-terminal expression in the Schemegrammar. For instance, if the solution corpus contains several (lambda (x y) (*x y) ) subprograms, the frequent sub-program mining would discover that andwe can add it as an alternative expression to the Scheme grammar.Towards Heuristic Algorithmic Memory 385We would like to find all frequent subprograms that occur twice or more sothat we can increase the probability of such sub-programs accordingly. We firstinterpret the problem of finding frequent sub-programs as a syntactic problem,disregarding semantic equivalences between sub-programs. Once formulated inour program representations of derivation trees as labelled rooted frequent sub-tree mining, the frequent sub-program mining algorithm is a reasonable extensionof traditional frequent pattern mining algorithms. We have implemented a BFSpatterned fast mining algorithm by exploiting the property that every sub-treeof a frequent tree is frequent (see [11] for an advanced algorithm). We findfrequent sub-trees (with a support threshold of 2 currently) of all sub-trees ofderivation trees rooted at expression in the solution corpus. At each update, anon-terminal hook frequent-expression in the grammar is rewritten by assigningprobabilities according to the frequency of each frequent sub-program. Note thatmost frequent expressions are abstract (i.e., sentential forms).3 ExperimentsOur experimental tests were carried out at TUBITAK ULAKBIM High Perfor-mance Computing Center on 144 AMD Opteron cores. We know of no previousdemonstration of realistic experiments over a long training sequence for generalpurpose machine learning. Solomonoff had stated: “It cannot be emphasized toostrongly, that the goal of early training sequence design, is not to solve hard prob-lems, but to get problem solving information into the machine. Since Lsearch iseasily adapted to parallel search, there is a tendency to try to solve fairly difficultproblems on inadequately trained machines. The success of such efforts is more atribute to progress in hardware design then to our understanding and exploitingmachine learning.” [12, Section 6]. We can show the effectiveness of our mem-ory system leaving no place for doubt through controlled experiments. We runthe entire training sequence with updates turned off and on. If the update algo-rithms cause a significant speed-up over search with no update, we can concludethat the update algorithms are effective. We use Conceptual Jump Size (CJS)to calculate the difficulty of a problem. CJS = ti/pi where ti is the running timeof solution program and pi is its a priori probability. The upper bound of LevinSearch’s running time is 2.CJS [12, Appendix A]. Our experiments are preferredto calculating CJS’s by hand, as in these experiments we are using Scheme R5RSin its full glory. Note that we are interested in only detecting whether any in-formation transfer occurs across problems rather than trying to solve difficultproblems with a machine that has no long-term memory. The running time ofa trial program is measured in Scheme execution cycles, which is the number ofprimitive Scheme operations (e.g., CAR) that are evaluated.We have developed a training sequence composed of operator induction prob-lems. For each problem, we have a set of input and output pairs, and we ap-proximate operator induction [1,13]. Training sequence 1 contains, in order, thesquare function sqr, the addition of two variables add, a function to test if theargument is zero is0, all of which have 3 example pairs, fourth power of a numberpow4 with just 2 example pairs, boolean nand, and xor functions with 4 example386 E. ÖzkuralTable 1. Performance of training sequence 1 with no update, |HAM | = 17145Problem Time Trials Errors Cycles Max Cyc. pi ti CJS H(si)sqr 16.28 5.34 × 105 1.57 × 105 5.46 × 106 2.05 × 108 2.19 × 10−7 37 1.68 × 108 22.12add 19.9759 1.03 × 106 3.13 × 105 1.13 × 107 4.1 × 108 9.77 × 10−8 40 4.09 × 108 23.28is0 7.57 41210 9531 430336 1.10 × 107 3.95 × 10−6 34 8.59 × 106 17.94pow4 1759.45 3.34 × 108 1.38 × 108 3.24 × 109 2.55 × 1011 1.67 × 10−10 26 1.55 × 1011 32.47nand 3497.17 6.48 × 108 2.71 × 108 6.69 × 109 5.13 × 1011 2.01 × 10−10 56 2.78 × 1011 32.21xor 1848.8 3.38 × 108 1.3 × 108 3.54 × 109 2.53 × 1011 2.01 × 10−10 52 2.58 × 1011 32.21all 7150.06Table 2. Performance of training sequence 1 with updateProblem Time Trials Errors Cycles Max Cyc. pi ti CJS H(si) |HAM|sqr 11.4 6.34 × 105 1.81 × 105 6.64 × 106 2.35 × 108 2.19 × 10−7 37 1.68 × 108 22.12 17318add 7.63 2.46 × 105 8.52 × 104 3.39 × 106 8.19 × 107 0.33 × 10−6 40 1.19 × 108 21.5 17515is0 2.72 10202 2969 136363 2.14 × 106 0.13 × 10−4 34 2.60 × 106 16.22 17566pow4 6.45 2.62 × 105 8.92 × 104 3.6 × 106 9.86 × 107 0.72 × 10−6 54 7.39 × 107 20.38 17617nand 209.53 2.55 × 107 1.12 × 107 3.72 × 108 1.51 × 1010 0.50 × 10−8 56 1.11 × 1010 27.57 17962xor 4.22 43749 14216 667625 1.18 × 107 0.47 × 10−5 57 1.19 × 107 17.68 18438all 245.1pairs each. Tables 1 and 2 convey the performance of our system on trainingsequence 1 without update and with update, respectively.For each problem, we give the time in seconds, number of trials, number ofScheme errors, number of Scheme execution cycles spent, number of maximumScheme cycles allocated to search, a priori probability of solution (pi), runningtime of solution in Scheme cycles (ti), Conceptual Jump Size, the length of theimplicit program code of the solution (H(si) = −lg(pi)) and the size of HAM inbytes after the update, respectively. Total time for the training sequence is alsogiven. The initial time limit is 106 cycles.The overall speed-up of training sequence 1 with updates is 29.17 comparedto the tests with no HAM update. This result indicates a consistent success oftransfer learning in a long training sequence. The search time for the solutionsin Table 2 tend to decrease compared to Table 1. The memory size has increasedonly 1293 bytes, for storing information for 6 operator induction problems, whichcorresponds to %7.5 increase in memory for 29.17 speed-up, which is a veryfavorable time-space trade-off. The solution of logical functions took longer thanprevious problems in Table 1, but we saw significant time savings in Table 2.Previous solutions are re-used aggressively. In Table 2, pow4 solution (define(pow4 x ) (define (sqr x ) (* x x)) (sqr (sqr x ) )) re-uses the sqr solutionand takes only 2.62 × 106 trials, its CJS speeds up 2097.4 times over the casewith no update, and the search achieves 272 speed-up in running time.4 Conclusion and Future WorkWe have proposed four update algorithms for incremental machine learning. Theeffectiveness of our update logic has been demonstrated with experiments in onelong training sequence, a feat that has not been accomplished before to the bestof our knowledge. In the future, we plan to implement Q/A induction and thePhase 2 of Solomonoff’s Alpha system [1].Towards Heuristic Algorithmic Memory 387References1. Solomonoff, R.J.: Progress in incremental machine learning. NIPS Workshop onUniversal Learning Algorithms and Optimal Search (2002)2. Schmidhuber, J.: Optimal ordered problem solver. Machine Learning 54, 211–256(2004)3. Solomonoff, R.J.: A system for incremental learning based on algorithmic proba-bility. In: Proceedings of the Sixth Israeli Conference on Artificial Intelligence, TelAviv, Israel, pp. 515–527 (1989)4. Levin, L.: Universal problems of full search. Problems of Information Transmis-sion 9(3), 256–266 (1973)5. Solomonoff, R.J.: Optimum sequential search. Technical report, Oxbridge Research(1984)6. Solomonoff, R.J.: Algorithmic probability: Theory and applications. In: Dehmer,M., Emmert-Streib, F. (eds.) Information Theory and Statistical Learning, pp.1–23. Springer Science+Business Media, N.Y (2009)7. Özkural, E.: Teraflop-scale incremental machine learning. CoRR abs/1103.1003(2011), http://arxiv.org/abs/1103.10038. Özkural, E.: Gigamachine: incremental machine learning on desktop computers.Draft (2009), http://examachine.net/papers/gigamachine-draft.pdf9. Hopcroft, J.E., Rajeev Motwani, J.U.: Introduction to Automata Theory, Lan-guages, and Computation, 2nd edn. Addison Wesley, Reading (2001)10. Merialdo, B.: Tagging english text with a probabilistic model. Computational Lin-guistics 20, 155–171 (1993)11. Zaki, M.J.: Efficiently mining frequent trees in a forest. In: Proceedings of theeighth ACM SIGKDD international conference on Knowledge discovery and datamining. KDD 2002, pp. 71–80. ACM Press, New York (2002)12. Solomonoff, R.J.: Algorithmic probability, heuristic programming and agi. In:Third Conference on Artificial General Intelligence, pp. 251–157 (2010)13. Solomonoff, R.J.: Three kinds of probabilistic induction: Universal distributionsand convergence theorems. The Computer Journal 51(5), 566–570 (2008)

Towards heuristic algorithmic memory

Eray Özkural

Crossref

Towards Heuristic Algorithmic Memory

http://repository.bilkent.edu.tr/bitstream/11693/28377/1/Towards%20heuristic%20algorithmic%20memory.pdf

Towards heuristic algorithmic memory

Abstract

Similar works

Full text

Available Versions

Bilkent University Institutional Repository

Crossref