Jeeves - A visual programming environment for mobile experience sampling

The Experience Sampling Method (ESM) captures participants’ thoughts and feelings in their everyday environments. Mobile and wearable technologies afford us opportunities to reach people using ESM in varying contexts. However, a lack of programming knowledge often hinders researchers in creating ESM applications. In practice, they rely on specialised tools for app creation. Our initial review of these tools indicates that most are expensive commercial services, and none utilise the full potential of sensors for creating context-aware applications. We present “Jeeves”, a visual language to facilitate ESM application creation. Inspired by successful visual languages in literature, our block-based notation enables researchers to visually construct ESM study specifications. We demonstrate its applicability by replicating existing ESM studies found in medical and psychology literature. Our preliminary study with 20 participants demonstrates that both non-programmers and programmers are able to successfully utilise Jeeves. We discuss future work in extending Jeeves with alternative mobile technologies.PostprintPeer reviewe

Jeeves - an Experience Sampling study creation tool

Ubiquitous mobile technology affords clinicians new opportunities to enhance personalised, patient-centric care remotely, easing the burden on both patient and clinician. The Experience Sampling Method (ESM) involves the repeated assessment of patients on their symptoms or behaviours, and their external contexts, as they go about their everyday lives, enhancing ecological validity and minimising recall bias. While previously conducted with paper diaries, ESM smartphone applications are now being employed, that have a range of benefits over paper-based methods including the ability to scale to many more patients. However, development of such applications is time-consuming and requires considerable programming knowledge. This has prompted the development of ESM creation tools that alleviate a researcher from the burden of programming an ESM application from scratch. This paper presents our work on Jeeves, a visual environment for creating secure ESM Android applications, and a usability evaluation we conducted with health psychology students.Postprin

Overcoming mental blocks:A blocks-based approach to experience sampling studies

Experience Sampling Method (ESM) studies repeatedly survey participants on their behaviours and experiences as they go about their everyday lives. Smartphones afford an ideal platform for ESM study applications as devices seldom leave their users, and can automatically sense surrounding context to augment subjective survey responses. ESM studies are employed in fields such as psychology and social science where researchers are not necessarily programmers and require tools for application creation. Previous tools using web forms, text files, or flowchart paradigms are either insufficient to model the potential complexity of study protocols, or fail to provide a low threshold to entry. We demonstrate that blocks programming simultaneously lowers the barriers to creating simple study protocols, while enabling the creation of increasingly sophisticated protocols. We discuss the design of Jeeves, our blocks-based environment for ESM studies, and explain advantages that blocks afford in ESM study design.Postprin

End-User Development of Experience Sampling Smartphone Apps - Recommendations and Requirements

Professional programmers are significantly outnumbered by end-users of software, making it problematic to predict the diverse, dynamic needs of these users in advance. An end-user development (EUD) approach, supporting the creation and modification of software independent of professional developers, is one potential solution. EUD activities are applicable to the work practices of psychology researchers and clinicians, who increasingly rely on software for assessment of participants and patients, but must also depend on developers to realise their requirements. In practice, however, the adoption of EUD technology by these two end-user groups is contingent on various contextual factors that are not well understood. In this paper, we therefore establish recommendations for the design of EUD tools allowing non-programmers to develop apps to collect data from participants in their everyday lives, known as "experience sampling" apps. We first present interviews conducted with psychology researchers and practising clinicians on their current working practices and motivation to adopt EUD tools. We then describe our observation of a chronic disease management clinic. Finally, we describe three case studies of psychology researchers using our EUD tool Jeeves to undertake experience sampling studies, and synthesise recommendations and requirements for tools allowing the EUD of experience sampling apps

TAPping into mental models with blocks

Trigger-Action Programming (TAP) has been shown to support end-users' rule-based mental models of context-aware applications. However, when desired behaviours increase in complexity, this can lead to ambiguity that confuses events, states, and how they can be combined in meaningful ways. Blocks programming could provide a solution, through constrained editing of visual triggers, conditions and actions. We observed slips and mistakes by users performing TAP with Jeeves, our domain-specific blocks environment, and propose solutions.Postprin

Characterizing Visual Programming Approaches for End-User Developers: A Systematic Review

Recently many researches have explored the potential of visual programming in robotics, the Internet of Things (IoT), and education. However, there is a lack of studies that analyze the recent evidence-based visual programming approaches that are applied in several domains. This study presents a systematic review to understand, compare, and reflect on recent visual programming approaches using twelve dimensions: visual programming classification, interaction style, target users, domain, platform, empirical evaluation type, test participants’ type, number of test participants, test participants’ programming skills, evaluation methods, evaluation measures, and accessibility of visual programming tools. The results show that most of the selected articles discussed tools that target IoT and education, while other fields such as data science, robotics are emerging. Further, most tools use abstractions to hide implementation details and use similar interaction styles. The predominant platforms for the tools are web and mobile, while desktop-based tools are on the decline. Only a few tools were evaluated with a formal experiment, whilst the remaining ones were evaluated with evaluation studies or informal feedback. Most tools were evaluated with students with little to no programming skills. There is a lack of emphasis on usability principles in the design stage of the tools. Additionally, only one of the tools was evaluated for expressiveness. Other areas for exploration include supporting end users throughout the life cycle of applications created with the tools, studying the impact of tutorials on improving learnability, and exploring the potential of machine learning to improve debugging solutions developed with visual programming

Visual Attention in Dynamic Environments and its Application to Playing Online Games

Abstract In this thesis we present a prototype of Cognitive Programs (CPs) - an executive controller built on top of Selective Tuning (ST) model of attention. CPs enable top-down control of visual system and interaction between the low-level vision and higher-level task demands. Abstract We implement a subset of CPs for playing online video games in real time using only visual input. Two commercial closed-source games - Canabalt and Robot Unicorn Attack - are used for evaluation. Their simple gameplay and minimal controls put the emphasis on reaction speed and attention over planning. Abstract Our implementation of Cognitive Programs plays both games at human expert level, which experimentally proves the validity of the concept. Additionally we resolved multiple theoretical and engineering issues, e.g. extending the CPs to dynamic environments, finding suitable data structures for describing the task and information flow within the network and determining the correct timing for each process