Mourao, Kira

Petrick, Ron

Steedman, Mark

English

We investigate the problem of learning action effects in partially observable STRIPS planning domains. Our approach is based on a voted kernel perceptron learning model, where action and state information is encoded in a compact vector representation as input to the learning mechanism, and resulting state changes are produced as output. Our approach relies on deictic features that embody a notion of attention and reduce the size of the representation. We evaluate our approach on a number of partially observable planning domains, adapted from domains used in the International Planning Competition, and show that it can quickly learn the dynamics of such domains, with low average error rates. Furthermore, we show that our approach handles noisy domains, and scales independently of the number of objects in a domain, making it suitable for large planning scenarios

Edinburgh Research Explorer

A training algorithm for optimal margin classiﬁers.

Acquiring linguistic argument structure from multimodal input using attentive focus.

ARMAR-III: A humanoid platform for perception-action integration.

Blocks world revisited.

Deictic codes for the embodiment of cognition (with commentary).

Early integration of vision and manipulation.

Extending the knowledge-basedapproachtoplanningwithincompleteinformation and sensing.

From margin to sparsity.

From primitive behaviors to goal directed behavior using affordances.

Large margin classiﬁ-cation using the perceptron algorithm.

Learning Action Models for Reactive Autonomous Agents.

Learning action models from plan examples using weighted MAX-SAT.

Learning by experimentation: Incremental reﬁnement of incomplete planning domains.

Learning by observation and practice: An incremental approach for planning operator acquisition.

Learning partially observable deterministic action models.

Learning symbolic models of stochastic world domains.

On convergence proofs for perceptrons.

Pengi: an implementation of a theory of activity.

Robustinformation extraction with perceptrons.

Schema learning: Experience-based construction of predictive action models.

STRIPS: A new approach to the application of theorem proving to problem solving.

The initial development of object knowledge by a learning robot. Robotics and Autonomous Systems 56(11):879–890.

The perceptron: a probabilistic model for information storage and organization in the brain.

the planning domain deﬁnition language.

Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control 25:821–837.

Using kernel perceptrons to learn action effects for planning.

Learning action effects in partially observable domains

We investigate the problem of learning action effects in partially observable STRIPS planning domains. Our approach is based on a voted kernel perceptron learning model, where action and state information is encoded in a compact vector representation as input to the learning mechanism, and resulting state changes are produced as output. Our approach relies on deictic features that assume an attentional mechanism that reduces the size of the representation. We evaluate our approach on a number of partially observable planning domains, and show that it can quickly learn the dynamics of such domains, with low average error rates. We show that our approach handles noisy domains, conditional effects, and that it scales independently of the number of objects in a domain

     Edinburgh Research Explorer                                      Learning action effects in partially observable domainsCitation for published version:Mourao, K, Petrick, R & Steedman, M 2010, Learning action effects in partially observable domains. in HCoelho, R Studer & M Wooldridge (eds), ECAI 2010 - 19th European Conference on Artificial Intelligence,Lisbon, Portugal, August 16-20, 2010, Proceedings.. IOS Press, pp. 973-974. DOI: 10.3233/978-1-60750-606-5-973Digital Object Identifier (DOI):10.3233/978-1-60750-606-5-973Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublished In:ECAI 2010 - 19th European Conference on Artificial Intelligence, Lisbon, Portugal, August 16-20, 2010,Proceedings.General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 2019Learning action effects in partially observable domainsKira Moura˜o and Ronald P. A. Petrick and Mark Steedman 1Abstract. We investigate the problem of learning action effectsin partially observable STRIPS planning domains. Our approach isbased on a voted kernel perceptron learning model, where action andstate information is encoded in a compact vector representation asinput to the learning mechanism, and resulting state changes are pro-duced as output. Our approach relies on deictic features that assumean attentional mechanism that reduces the size of the representation.We evaluate our approach on a number of partially observable plan-ning domains, and show that it can quickly learn the dynamics ofsuch domains, with low average error rates. We show that our ap-proach handles noisy domains, conditional effects, and that it scalesindependently of the number of objects in a domain.1 INTRODUCTION AND MOTIVATIONAcquiring a domain model automatically through learning and ex-perience gives an agent greater ﬂexibility to handle unexpected sit-uations, and avoids the need for a predeﬁned world model. Existingapproaches either work within the space of transition rules to ﬁnd a“good” set, or all consistent sets, of rules [2, 3, 12, 15], or they op-erate at the sensor level by constructing transition rules from actionsand robot sensor data coded as sets of objects or raw sensor readings,and predicates derived from this data [7, 10]. The former, high-level,methods have been applied to partially observable [2, 15] or non-deterministic [12] domains, but are not applicable to domains whichare both noisy and partially observable; few are also able to learnconditional effects. The latter, low-level, methods can learn in noisy,partially observable domains, but the domains are much simpler,without relations between objects, and sometimes without objects atall. Here, we extend our previous work on learning action models innoiseless, fully observable domains [11]. Our method learns the ef-fects of STRIPS actions [4], extended to admit conditional effects,in deterministic, noisy and partially observable versions of the morecomplex domains typical of the high-level approaches.2 REPRESENTATIONWe learn action models from sequences of interleaved actions andstate observations. Each observation initially encompasses as muchof the world state as the agent is able to detect, with some parts ofthe state potentially unobserved or corrupted by noise. We reducethe size of each observation by only considering objects which canbe identiﬁed by a deictic reference [1], and then transform each ob-servation into a vector to use as input to the learning model.A deictic representation maintains pointers to objects of interest inthe world, with objects coded relative to the agent or current action.1 University of Edinburgh, UK, email: kira.mourao@ed.ac.uk,{rpetrick,steedman}@inf.ed.ac.ukFigure 1. Computing deictic references: an example from the BlocksWorlddomain, in which an agent can manipulate a set of blocks on a table. Giventhe action stack (A, B), i.e., stack block A on top of block B, the initial setof objects of interest is {A, B}. The only object related to A or B is C,since B is on C. Therefore the full set of objects of interest is {A, B, C}.We take a similar approach to previous work applying deictic repre-sentations to learn domain dynamics [3, 12]. For a given action in-stance we construct the set of objects of interest, consisting of the setof objects which are parameters of the action, and the objects which,in the current state, are related to any object in the action parameters(see Figure 1). This single step computation is in contrast to previousapproaches, where the set of objects under consideration is the fulltransitive closure under all relations among objects. Also, whereasprevious approaches ignored objects if they were not uniquely de-ﬁned by deictic reference, we allow deictic references to any set ofobjects that are indistinguishable relative to the action parameters.An input vector representing the reduced state space is then con-structed by assigning a bit for each action, 0-ary ﬂuent, and for eachpossible relation involving only the objects in the reduced state space.The value of a bit is 1 (−1) if the corresponding ﬂuent is true (false),or if the corresponding action is (not) the current action. Bits for un-observed or unused ﬂuents are set to an arbitrary value N, which isignored during learning.Vectors representing an action’s effects on a state are identical inform to the input vectors, except that actions are excluded from thevector, and bits are set to 1 (−1) if the corresponding ﬂuent changes(does not change). Bits corresponding to unobserved or unused ﬂu-ents are set to N.3 LEARNING MODELThe task of the learning mechanism is to learn the deterministic as-sociations between action-state pairs and their effects. It is assumedthat the number and type of parameters of each action, predicate andfunction are known. Action preconditions and effects are not known,and effects may be conditional. Disjunctive effects are not allowed.Instead, all effects are conjunctions of predicates, meaning it is sufﬁ-cient to learn the rule for each predicate separately. Using the vectorrepresentation deﬁned above, state transitions can be learnt using abank of classiﬁers, one for each bit of the output vector.To address our learning problem we construct a variant of theperceptron algorithm [13], using the voted perceptron [5], which isECAI 2010H. Coelho et al. (Eds.)IOS Press, 2010© 2010 The authors and IOS Press. All rights reserved.doi:10.3233/978-1-60750-606-5-9739730 5000 10000 15000 2000000.10.20.30.40.50.60.70.80.91Number of training examplesF scoreZenoTravel5% obs, 0% noise10% obs, 0% noise15% obs, 0% noise5% obs, 10% noise10% obs, 10% noise15% obs, 10% noise0 5000 10000 15000 2000000.10.20.30.40.50.60.70.80.91Number of training examplesF scoreDepots5% obs, 0% noise10% obs, 0% noise15% obs, 0% noise5% obs, 10% noise10% obs, 10% noise15% obs, 10% noise0 5000 10000 15000 2000000.10.20.30.40.50.60.70.80.91Number of training examplesF scoreBriefcase5% obs, 0% noise10% obs, 0% noise15% obs, 0% noise5% obs, 10% noise10% obs, 10% noise15% obs, 10% noiseFigure 2. Results of learning action models in standard planning domains. Error bars are 95% conﬁdence intervals. In noiseless, fully observable domains,models fully predict all test cases after less than 200 examples (results not shown). While observing only a small fraction of the state, without noise, the learningmodel completely predicts the test set after 20000 examples, in many of the test cases: with 15% of the state observable, the F-score is not signiﬁcantly differentfrom 1 (t-tests, p > 0.05) in any of our domains. With noise, the learnt models are clearly poorer, but some aspects of each domain are still learnt: with 15% ofthe state observable, the F-score is signiﬁcantly different from 1 for ZenoTravel (p = 0.004) and Briefcase (p = 0.046), but not for Depots (p > 0.05).noise-tolerant [8] and computationally efﬁcient, producing perfor-mance close to the best performing maximal-margin classiﬁers (e.g.SVMs) on similar problems. We use the DNF kernel [14], which al-lows the perceptron to run over the feature space of all possible con-junctions of bits in the input space, i.e., the space of possible rules.4 EXPERIMENTSWe tested the learning model on standard planning domains from the3rd International Planning Competition (IPC): Depots, ZenoTraveland DriverLog; a standard BlocksWorld domain; and Briefcase, adomain with conditional effects. Sequences of random actions andresulting states were generated from PDDL domain descriptions [9]and used as training and testing data.2 Speciﬁc problems from theIPC were used to set the sizes of the initial states for each sequence.The actual initial states were generated at random using the IPC3problem generator and a Briefcase state generator [6].To determine error bounds on our results, we used 10 differentrandomly generated training and testing sets. Each training set con-sisted of 1000-20000 actions and matching state observations. Par-tial observability was simulated by randomly selecting a fraction (5-20%) of bits to retain in each state vector, and setting the remainingbits to N. Sensor noise at 10% was simulated by ﬂipping each bitin the state vector with probability 0.1. Each test set was a fully ob-servable, noiseless sequence of 2000 actions and observations. Wemeasured the performance on our test sets by considering the ﬂuentswhich our model predicted would change versus the ﬂuents whichdid change, and calculating the balanced F-measure, the harmonicmean of precision and recall (true positives/predicted changes andtrue positives/actual changes, respectively). Selected results of theexperiments are shown in Figure 2.5 CONCLUSIONS AND FUTURE WORKWe have presented a method for learning deterministic action modelswhich is fast, scalable and handles noise and partial observability ofthe world state. Furthermore, the error rate of the predictions madeby the model is low. The speed, scalability and accuracy make theapproach highly suitable for use in planning applications.Additionally, our approach can learn conditional effects. Note thatthe success or failure of an action, which depends on its precondi-2 All data was generated using the Random Action Generator 0.5 available athttp://magma.cs.uiuc.edu/filter/.tions, is a form of conditional effect (the action has a null effect un-less the preconditions are satisﬁed). Therefore action preconditionscan also be learnt, if examples of action failures as well as actionsuccesses are provided.A key step in future work will be to extract STRIPS-style rulesfrom the sets of ordered pairings of entire states presently learnt bythe model, so that the learning model can be integrated with standardplanning software. We also plan to apply our method to intrinsicallynoisy and partially observable real-world robot environments.ACKNOWLEDGEMENTSThis work was funded by the EU PACO-PLUS project (FP6-2004-IST-4-27657) and Edinburgh University Neuroinformatics DTC.REFERENCES[1] P. E. Agre and D. Chapman, ‘Pengi: an implementation of a theory ofactivity’, in Proc. of AAAI, pp. 268–272, (1987).[2] E. Amir and A. Chang, ‘Learning partially observable deterministic ac-tion models’, JAIR, 33, 349–402, (2008).[3] S. S. Benson, Learning Action Models for Reactive AutonomousAgents, Ph.D. dissertation, Stanford University, 1996.[4] R. E. Fikes and N. J. Nilsson, ‘STRIPS: A new approach to the applica-tion of theorem proving to problem solving’, AIJ, 2, 189–208, (1971).[5] Y. Freund and R. Schapire, ‘Large margin classiﬁcation using the per-ceptron algorithm’, Machine Learning, 37, 277–296, (1999).[6] J. Hoffmann and B. Nebel. FF domain collection.http://www.loria.fr/˜hoffmanj/ff-domains.html.[7] M. Holmes and C. Isbell, ‘Schema learning: Experience-based con-struction of predictive action models’, in NIPS 17, pp. 585–562, (2005).[8] R. Khardon and G. M. Wachman, ‘Noise tolerant variants of the per-ceptron algorithm’, JMLR, 8, 227–248, (2007).[9] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram,M. Veloso, D. Weld, and D. Wilkins, ‘PDDL - the planning domaindeﬁnition language’, Technical report, CVC TR-98-003, Yale, (1998).[10] J. Modayil and B. Kuipers, ‘The initial development of object knowl-edge by a learning robot’, Robot. Auton. Syst., 56(11), 879–890, (2008).[11] K. Moura˜o, R. Petrick, and M. Steedman, ‘Using kernel perceptrons tolearn action effects for planning’, in Proc. of CogSys, pp. 45–50, (2008).[12] H. M. Pasula, L. S. Zettlemoyer, and L. P. Kaelbling, ‘Learning sym-bolic models of stochastic domains’, JAIR, 29, 309–352, (2007).[13] F. Rosenblatt, ‘The perceptron: a probabilistic model for informationstorage and organization in the brain’, Psych. Rev., 65(6), 386–408,(1958).[14] K. Sadohara, ‘Learning of boolean functions using support vector ma-chines’, in Proc. of ALT, pp. 106–118, (2001).[15] Q. Yang, K. Wu, and Y. Jiang, ‘Learning action models from plan ex-amples using weighted MAX-SAT’, AIJ, 171(2-3), 107–143, (2007).K. Mourão et al. / Learning Action Effects in Partially Observable Domains974

https://www.pure.ed.ac.uk/ws/files/13027019/FAIA215_0973.pdf

Learning action effects in partially observable domains

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer

Edinburgh Research Explorer