On Experimentation in Software-Intensive Systems

Context: Delivering software that has value to customers is a primary concern of every software company. Prevalent in web-facing companies, controlled experiments are used to validate and deliver value in incremental deployments. At the same that web-facing companies are aiming to automate and reduce the cost of each experiment iteration, embedded systems companies are starting to adopt experimentation practices and leverage their activities on the automation developments made in the online domain. Objective: This thesis has two main objectives. The first objective is to analyze how software companies can run and optimize their systems through automated experiments. This objective is investigated from the perspectives of the software architecture, the algorithms for the experiment execution and the experimentation process. The second objective is to analyze how non web-facing companies can adopt experimentation as part of their development process to validate and deliver value to their customers continuously. This objective is investigated from the perspectives of the software development process and focuses on the experimentation aspects that are distinct from web-facing companies. Method: To achieve these objectives, we conducted research in close collaboration with industry and used a combination of different empirical research methods: case studies, literature reviews, simulations, and empirical evaluations. Results: This thesis provides six main results. First, it proposes an architecture framework for automated experimentation that can be used with different types of experimental designs in both embedded systems and web-facing systems. Second, it proposes a new experimentation process to capture the details of a trustworthy experimentation process that can be used as the basis for an automated experimentation process. Third, it identifies the restrictions and pitfalls of different multi-armed bandit algorithms for automating experiments in industry. This thesis also proposes a set of guidelines to help practitioners select a technique that minimizes the occurrence of these pitfalls. Fourth, it proposes statistical models to analyze optimization algorithms that can be used in automated experimentation. Fifth, it identifies the key challenges faced by embedded systems companies when adopting controlled experimentation, and we propose a set of strategies to address these challenges. Sixth, it identifies experimentation techniques and proposes a new continuous experimentation model for mission-critical and business-to-business. Conclusion: The results presented in this thesis indicate that the trustworthiness in the experimentation process and the selection of algorithms still need to be addressed before automated experimentation can be used at scale in industry. The embedded systems industry faces challenges in adopting experimentation as part of its development process. In part, this is due to the low number of users and devices that can be used in experiments and the diversity of the required experimental designs for each new situation. This limitation increases both the complexity of the experimentation process and the number of techniques used to address this constraint

A society of mind approach to cognition and metacognition in a cognitive architecture

This thesis investigates the concept of mind as a control system using the "Society of Agents" metaphor. "Society of Agents" describes collective behaviours of simple and intelligent agents. "Society of Mind" is more than a collection of task-oriented and deliberative agents; it is a powerful concept for mind research and can benefit from the use of metacognition. The aim is to develop a self configurable computational model using the concept of metacognition. A six tiered SMCA (Society of Mind Cognitive Architecture) control model is designed that relies on a society of agents operating using metrics associated with the principles of artificial economics in animal cognition. This research investigates the concept of metacognition as a powerful catalyst for control, unify and self-reflection. Metacognition is used on BDI models with respect to planning, reasoning, decision making, self reflection, problem solving, learning and the general process of cognition to improve performance.One perspective on how to develop metacognition in a SMCA model is based on the differentiation between metacognitive strategies and metacomponents or metacognitive aids. Metacognitive strategies denote activities such as metacomphrension (remedial action) and metamanagement (self management) and schema training (meaning full learning over cognitive structures). Metacomponents are aids for the representation of thoughts. To develop an efficient, intelligent and optimal agent through the use of metacognition requires the design of a multiple layered control model which includes simple to complex levels of agent action and behaviours. This SMCA model has designed and implemented for six layers which includes reflexive, reactive, deliberative (BDI), learning (Q-Ieamer), metacontrol and metacognition layers

Implementation of experimental test beds for Real Time diagnosis

Experimental test beds to study Real Time agents are common topic in AI. A principal contribution of this paper is to develop a framework for such experimental test beds. Design and implementation of the test bed and the extension of the framework for different equipment and agents are discussed. Results of an experiment with a simple digital circuit and heuristic agents to test the working of the test bed are reported

Design and implementation aspects of open source next generation networks (NGN) test-bed software toolkits

Informations- und Kommunikationstechnologien bilden seit langem das immer wichtiger werdende Rückgrat der weltweiten Wirtschaft und Telekommunikation, in der speziell Telekommunikationsnetze und -dienste einen elementaren Anteil tragen. Durch die Konvergenz von Telekommunikations- und Internettechnologien hat sich die Telekommunikationslandschaft in der letzten Dekade drastisch verändert. Bislang geschlossene Telekommunikationsumgebungen haben sich imWandel zum sogenannten Next Generation Network (NGN) hinsichtlich unterstützter Zugangsnetztechnologien und angebotener multimedialer Anwendungen sowie der eingesetzten Protokolle und Dienste zu komplexen, hochdynamischen, Multi-Service Infrastrukturen gewandelt. Die Kontrollschicht solcher NGNs ist dabei von übergeordneter Bedeutung, da diese zwischen den Zugangsnetzen und den Anwendungen sitzt. Der Einsatz und die Optimierung des IP-Multimedia Subsystem (IMS) wurde in diesem Kontext Jahrelang erforscht und diskutiert und es repräsentiert heute die weltweit anerkannte Kontrollplattform für feste und mobile Telekommunikationsnetze. Die Forschung an Protokollen und Diensten in diesen NGN Umgebungen ist aufgrund der Konvergenz von Technologien, Anwendungen und Business Modellen sowie der hohen Dynamik aber kurzen Innovationszyklen hochkomplex. Der frühzeitigen Zugang zu herstellerunabhängigen – aber dicht an der Produktwelt angelehnten - Validierungsinfrastrukturen, sogenannten offenen Technologietest-beds, kurz Test-beds, ist daher für Forschungs- und Entwicklungsabteilungen unerlässlich Die vorliegende Dissertation beschreibt die umfangreiche Forschungsarbeit des Autors auf dem Gebiet der offenen NGN Test-beds über die letzten neun Jahre und konzentriert sich dabei auf Entwurf, Entwicklung und Bereitstellung des Open Source IMS Core Projekt, das seit Jahren die Grundlage für eine Vielzahl von NGN Test-beds und zahllose NGN Forschungs- und Entwicklungsprojekte im akademischen als auch Industrienahen Umfeld rund um den Globus darstellt. Dabei wird ein großer Schwerpunkt auf die Anforderungen hinsichtlich Flexibilität, Leistung, Funktionalitätsumfang und Interoperabilität, sowie elementare Designprinzipien von Test-bedwerkzeugen gelegt. Die Arbeit beschreibt und bewertet darüberhinaus den Einsatz von Open Source Prinzipien und veranschaulicht die Vorteile dieses Ansatzes hinsichtlich Einfluss und Nachhaltigkeit der Forschung anhand des Aufbaus einer globalen Open Source IMS Core (OpenIMSCore) Forschungs-Community. Außerdem veranschaulicht die Arbeit zum Ende die Wiederverwendbarkeit der wesentlichen angewendeten Designprinzipien an anderen maßgeblich durch den Autor entwickelten Test-bed Werkzeugen, insbesondere dem Open Evolved Packet Core (OpenEPC) für die nahtlose Integration verschiedener Breitbandnetztechnologien.Information and Communication Technologies provide for a long time already the backbone of telecommunication networks, such that communication services represent an elementary foundation of today’s globally connected economy. The telecommunication landscape has experienced dramatic transformations through the convergence of the Telecom and the Internet worlds. The previously closed telecommunication domain is currently transforming itself through the so-called NGN evolution into a highly dynamic multiservice infrastructure, supporting rich multimedia applications, as well as providing comprehensive support for various access technologies. The control layer of such NGNs is then of paramount importance, as representing the convergent mediator between access and services. The use and the optimization of the IP-Multimedia Subsystem (IMS) was researched and considered in this domain for many years now, such that today it represents the world-wide recognized control platform for fixed and mobile NGNs. Research on protocols and services for such NGN architectures, due to the convergence of technologies, applications and business models, as well as for enabling highly dynamic and short innovation cycles, is highly complex and requires early access to vendor independent - yet close to real life systems - validation environments, the so-called open technology test-beds. The present thesis describes the extensive research of the author over the last nine years in the field of open NGN test-beds. It focuses on the design, development and deployment of the Open Source IMS Core project, which represents since years the foundation of numerous NGN test-beds and countless NGN Research & Development projects in the academia as well as the industry domain around the globe. A major emphasis is given for ensuring flexibility, performance, reference functionality and inter-operability, as well as satisfying elementary design principles of such test-bed toolkits. The study also describes and evaluates the use of Open Source principles, highlighting the advantages of using it in regard to the creation, impact and sustainability of a global OpenIMSCore research community. Moreover, the work documents that the essential design principles and methodology employed can be reused in a generic way to create test-bed toolkits in other technology domains. This is shown by introducing the OpenEPC project, which provides for seamless integration of different mobile broadband technologies