HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions

Creating automation scripts for tasks involving Graphical User Interface (GUI) interactions is hard. It is challenging because not all software applications allow access to a program’s internal state, nor do they all have accessibility APIs. Although much of the internal state is exposed to the user through the GUI, it is hard to programmatically operate the GUI’s widgets. To that end, we developed a system prototype that learns by demonstration, called HILC (Help, It Looks Confusing). Users, both programmers and non-programmers, train HILC to synthesize a task script by demonstrating the task. A demonstration produces the needed screenshots and their corresponding mouse-keyboard signals. After the demonstration, the user answers follow-up questions. We propose a user-in-the-loop framework that learns to generate scripts of actions performed on visible elements of graphical applications. Although pure programming by demonstration is still unrealistic due to a computer’s limited understanding of user intentions, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system, to help with confusing parts of the demonstrations. Our models of events and appearances are surprisingly simple but are combined effectively to cope with varying amounts of supervision. The best available baseline, Sikuli Slides, struggled to assist users in the majority of the tests in our user study experiments. The prototype with our proposed approach successfully helped users accomplish simple linear tasks, complicated tasks (monitoring, looping, and mixed), and tasks that span across multiple applications. Even when both systems could ultimately perform a task, ours was trained and refined by the user in less time

PhoneLab: Cloud-Backed Development Environment for Smartphones

We will develop a Scala-based language and a development environment to simplify the construction of cloud-backed smartphone applications, both by professionals and by end users. We will develop programming assistance tools that use cloud analysis (running on http://ecocloud.ch infrastructure) to suggest code fragments, and enable development and customization of applications both from the desktop and directly from smartphones

Computational Intelligence for Cooperative Swarm Control

Over the last few decades, swarm intelligence (SI) has shown significant benefits in many practical applications. Real-world applications of swarm intelligence include disaster response and wildlife conservation. Swarm robots can collaborate to search for survivors, locate victims, and assess damage in hazardous environments during an earthquake or natural disaster. They can coordinate their movements and share data in real-time to increase their efficiency and effectiveness while guiding the survivors. In addition to tracking animal movements and behaviour, robots can guide animals to or away from specific areas. Sheep herding is a significant source of income in Australia that could be significantly enhanced if the human shepherd could be supported by single or multiple robots. Although the shepherding framework has become a popular SI mechanism, where a leading agent (sheepdog) controls a swarm of agents (sheep) to complete a task, controlling a swarm of agents is still not a trivial task, especially in the presence of some practical constraints. For example, most of the existing shepherding literature assumes that each swarm member has an unlimited sensing range to recognise all other members’ locations. However, this is not practical for physical systems. In addition, current approaches do not consider shepherding as a distributed system where an agent, namely a central unit, may observe the environment and commu- nicate with the shepherd to guide the swarm. However, this brings another hurdle when noisy communication channels between the central unit and the shepherd af- fect the success of the mission. Also, the literature lacks shepherding models that can cope with dynamic communication systems. Therefore, this thesis aims to design a multi-agent learning system for effective shepherding control systems in a partially observable environment under communication constraints. To achieve this goal, the thesis first introduces a new methodology to guide agents whose sensing range is limited. In this thesis, the sheep are modelled as an induced network to represent the sheep’s sensing range and propose a geometric method for finding a shepherd-impacted subset of sheep. The proposed swarm optimal herding point uses a particle swarm optimiser and a clustering mechanism to find the sheepdog’s near-optimal herding location while considering flock cohesion. Then, an improved version of the algorithm (named swarm optimal modified centroid push) is proposed to estimate the sheepdog’s intermediate waypoints to the herding point considering the sheep cohesion. The approaches outperform existing shepherding methods in reducing task time and increasing the success rate for herding. Next, to improve shepherding in noisy communication channels, this thesis pro- poses a collaborative learning-based method to enhance communication between the central unit and the herding agent. The proposed independent pre-training collab- orative learning technique decreases the transmission mean square error by half in 10% of the training time compared to existing approaches. The algorithm is then ex- tended so that the sheepdog can read the modulated herding points from the central unit. The results demonstrate the efficiency of the new technique in time-varying noisy channels. Finally, the central unit is modelled as a mobile agent to lower the time-varying noise caused by the sheepdog’s motion during the task. So, I propose a Q-learning- based incremental search to increase transmission success between the shepherd and the central unit. In addition, two unique reward functions are presented to ensure swarm guidance success with minimal energy consumption. The results demonstrate an increase in the success rate for shepherding

Apprenticeship Bootstrapping for Autonomous Aerial Shepherding of Ground Swarm

Aerial shepherding of ground vehicles (ASGV) musters a group of uncrewed ground vehicles (UGVs) from the air using uncrewed aerial vehicles (UAVs). This inspiration enables robust uncrewed ground-air coordination where one or multiple UAVs effectively drive a group of UGVs towards a goal. Developing artificial intelligence (AI) agents for ASGV is a non-trivial task due to the sub-tasks, multiple skills, and their non-linear interaction required to synthesise a solution. One approach to developing AI agents is Imitation learning (IL), where humans demonstrate the task to the machine. However, gathering human data from complex tasks in human-swarm interaction (HSI) requires the human to perform the entire job, which could lead to unexpected errors caused by a lack of control skills and human workload due to the length and complexity of ASGV. We hypothesise that we can bootstrap the overall task by collecting human data from simpler sub-tasks to limit errors and workload for humans. Therefore, this thesis attempts to answer the primary research question of how to design IL algorithms for multiple agents. We propose a new learning scheme called Apprenticeship Bootstrapping (AB). In AB, the low-level behaviours of the shepherding agents are trained from human data using our proposed hierarchical IL algorithms. The high-level behaviours are then formed using a proposed gesture demonstration framework to collect human data from synthesising more complex controllers. The transferring mechanism is performed by aggregating the proposed IL algorithms. Experiments are designed using a mixed environment, where the UAV flies in a simulated robotic Gazebo environment, while the UGVs are physical vehicles in a natural environment. A system is designed to allow switching between humans controlling the UAVs using low-level actions and humans controlling the UAVs using high-level actions. The former enables data collection for developing autonomous agents for sub-tasks. At the same time, in the latter, humans control the UAV by issuing commands that call the autonomous agents for the sub-tasks. We baseline the learnt agents against Str\"{o}mbom scripted behaviours and show that the system can successfully generate autonomous behaviours for ASGV

Learn to automate GUI tasks from demonstration

This thesis explores and extends Computer Vision applications in the context of Graphical User Interface (GUI) environments to address the challenges of Programming by Demonstration (PbD). The challenges are explored in PbD which could be addressed through innovations in Computer Vision, when GUIs are treated as an application domain, analogous to automotive or factory settings. Existing PbD systems were restricted by domain applications or special application interfaces. Although they use the term Demonstration, the systems did not actually see what the user performs. Rather they listen to the demonstrations through internal communications via operating system. Machine Vision and Human in the Loop Machine Learning are used to circumvent many restrictions, allowing the PbD system to watch the demonstration like another human observer would. This thesis will demonstrate that our prototype PbD systems allow non-programmer users to easily create their own automation scripts for their repetitive and looping tasks. Our PbD systems take their input from sequences of screenshots, and sometimes from easily available keyboard and mouse sniffer software. It will also be shown that the problem of inconsistent human demonstration can be remedied with our proposed Human in the Loop Computer Vision techniques. Lastly, the problem is extended to learn from demonstration videos. Due to the sheer complexity of computer desktop GUI manipulation videos, attention is focused on the domain of video game environments. The initial studies illustrate that it is possible to teach a computer to watch gameplay videos and to estimate what buttons the user pressed

Camp No and soldier-writers: disidentification and ethical remapping in post-9/11 narratives of dissent

America’s post-9/11 soldier-writers challenge pivotal contemporary assumptions about allegiance, solidarity, national identity, and the political-emotional maps of responsibility and belonging that artists, activists, and citizens at large draw up mentally as they picture their affiliations with and duties to their community, territory, country, or state institution. Organized around a triad of concepts (parrhesia, cosmopolitanism, and dissensus), this project argues that this generation of writers represents a significant literary movement. Specifically, I read their work as the loci of a “dissenting” overhauling of the official narratives and rhetorical maps that chart the United States’ Global War on Terror

The pedagogy of computer programming using cognitive development through an e-learning object

Motivated by the needs of a pedagogy focusing on minimizing the learning difficulties in program semantics knowledge and logical reasoning, this research project develops a cognitive development-based pedagogy for introductory programming to support students in organizing and constructing knowledge to learn computer programming. A pedagogy is described as a practice and learning theory that defines the teaching and learning. Regarding the practice of this pedagogy, it uses a cognitive learning tool, called e-learning object, to support the scaffolding. With regard to the theory, this pedagogy is developed based on Vygotsky’s Zone of Proximal Development and Piaget’s theory for cognitive development. In particular the scaffolding of this pedagogy includes three major learning processes. The first two learning processes focus on supporting students constructing knowledge on program semantics and conceptually map this knowledge to the coding process. The last learning process extends the learning to self-practice by demanding students to complete a set of exercises independently. All of these learning processes are supported by using the e-learning object, which is the major cognitive learning tool used in this pedagogy to support cognitive development. It is called e-learning object as it is designed by organizing a group of learning objects, in which each of them is to deliver the concepts of a specific unit topic of program control. Together with the course materials, these learning objects are accessed through the college’s ‘Blackboard System’. In addition to the major objective of improving students’ learning performance, this cognitive development-based pedagogy also extends from this objective to find out whether the positive learning outcome connects to cognitive development, and also whether this pedagogy can be embraced by teachers for use in their teaching processes. With these objectives, six research questions are defined in two stages of study. Research questions Q1 and Q2 are used to study students’ learning outcomes in year 1 and 2, and research questions Q3 to Q5 are used to find out whether students’ learning outcomes are connected to cognitive development. Research question Q6 focuses on whether this pedagogy matches teachers’ knowledge of using it, based on their knowledge of applying technology-based pedagogy. The research methodology of this project is the triangulation design where quantitative data are enriched by the collection of qualitative data. This mixture of quantitative and qualitative data collection in different research questions enables this study to interpret the values of this cognitive development-based pedagogy with different views from students and teachers. The research methods mainly include the quasi-experimental method, survey method and the rating scale anchoring method. With these methods, data are collected by using pre-test and post-test papers, questionnaires, and a checklist of rating scale anchoring mental specifications. They are analysed by two-tailed t-test, descriptive method with mean analysis and the one- way repeated measure ANOVA. These research and data analysis methods have been proven effective and used widely, in educational research projects. This research project makes four major contributions: (i) the e-learning object used in this pedagogy can be used to improve students’ learning performance in computer programming; (ii) evidence that a pedagogy focusing on cognitive development can be used to improve students’ learning performance without being limited by programming languages; (iii) development of a cognitive development- based pedagogy for wide use in introductory programming without being limited by teachers’ knowledge and programming languages; and (iv) learning with this cognitive development-based pedagogy builds up students’ problem-solving skills and applies them to different subject areas. With these achieved goals, this project therefore provides a conceptual and operational model for a pedagogical approach to Computer Science teachers design and use in their teaching process

Human-machine communication for educational systems design

This book contains the papers presented at the NATO Advanced Study Institute (ASI) on the Basics of man-machine communication for the design of educational systems, held August 16-26, 1993, in Eindhoven, The Netherland