Learning to play using low-complexity rule-based policies: Illustrations through Ms. Pac-Man

In this article we propose a method that can deal with certain combinatorial reinforcement learning tasks. We demonstrate the approach in the popular Ms. Pac-Man game. We define a set of high-level observation and action modules, from which rule-based policies are constructed automatically. In these policies, actions are temporally extended, and may work concurrently. The policy of the agent is encoded by a compact decision list. The components of the list are selected from a large pool of rules, which can be either hand-crafted or generated automatically. A suitable selection of rules is learnt by the cross-entropy method, a recent global optimization algorithm that fits our framework smoothly. Cross-entropy-optimized policies perform better than our hand-crafted policy, and reach the score of average human players. We argue that learning is successful mainly because (i) policies may apply concurrent actions and thus the policy space is sufficiently rich, (ii) the search is biased towards low-complexity policies and therefore, solutions with a compact description can be found quickly if they exist

Knowledge Representation Concepts for Automated SLA Management

Outsourcing of complex IT infrastructure to IT service providers has increased substantially during the past years. IT service providers must be able to fulfil their service-quality commitments based upon predefined Service Level Agreements (SLAs) with the service customer. They need to manage, execute and maintain thousands of SLAs for different customers and different types of services, which needs new levels of flexibility and automation not available with the current technology. The complexity of contractual logic in SLAs requires new forms of knowledge representation to automatically draw inferences and execute contractual agreements. A logic-based approach provides several advantages including automated rule chaining allowing for compact knowledge representation as well as flexibility to adapt to rapidly changing business requirements. We suggest adequate logical formalisms for representation and enforcement of SLA rules and describe a proof-of-concept implementation. The article describes selected formalisms of the ContractLog KR and their adequacy for automated SLA management and presents results of experiments to demonstrate flexibility and scalability of the approach.Comment: Paschke, A. and Bichler, M.: Knowledge Representation Concepts for Automated SLA Management, Int. Journal of Decision Support Systems (DSS), submitted 19th March 200

Interaction Expressions - A Powerful Formalism for Describing Inter-Workflow Dependencies

Current workflow management technology does not provide adequate means for describing and implementing workflow ensembles, i. e., dynamically evolving collections of more or less independent workflows which have to synchronize only now and then. Interaction expressions are proposed as a simple yet powerful formalism to remedy this shortcoming. Besides operators for sequential composition, iteration, and selection, which are well-known from regular expressions, they provide parallel composition and iteration, conjunction, and several advanced features like parametric expressions, multipliers, and quantifiers. The paper introduces interaction expressions semi-formally, gives examples of their typical use, and describes their implementation and integration with state-of-the-art workflow technology. Major design principles, such as orthogonality and implicit and predictive choice, are discussed and compared with several related approaches