Investigating the Perceptibility of Smartphone Notifications and Methods for Context-Aware Data Assessment in Experience Sampling Studies

Eine zentrale Aufgabe in der Mensch-Maschine-Interaktion ist die Durchführung von Nutzerstudien. Diese ermöglichen einen tieferen Einblick in das Verhalten von Nutzern, dienen aber auch dazu, Labels zum Annotieren von Daten zu sammeln. Die traditionelle Methode zum Erfassen von subjektivem Feedback ist die Experience Sampling Method (ESM). Durch das Beantworten von Fragebögen stellen Probanden nicht nur Informationen über sich selbst, sondern auch über ihre Umgebung zur Verfügung. Außerdem können ihre Antworten als Label für Daten, welche zeitgleich erhoben wurden, dienen. Inzwischen sind Smartphones zur Hauptplattform zum Durchführen von ESM Studien geworden. Sie werden genutzt, um ESM-Abfragen in Form von Benachrichtigungen auszusenden, um die gesammelten Labels zu speichern und um sie den Sensordaten zuzuweisen, welche im Hintergrund gesammelt wurden. In ESM-Studien wird angestrebt, möglichst viele und qualitativ hochwertige Daten zu sammeln. Um dieses Ziel zu erreichen, bedarf es einer großen Menge sorgfältig beantworteter ESM-Abfragen. Die Probanden wiederum wollen in der Regel so wenig Abfragen wie möglich erhalten. Es ist notwendig, einen Kompromiss zwischen Abfragehäufigkeit und Probandenzufriedenheit zu finden. Beim Erstellen von ESM-Studien ergeben sich verschiedene Herausforderungen. Einerseits sind diese mit der ESM-App und deren Funktionalität verbunden. Andererseits stehen sie aber auch mit dem Ausliefern von ESM-Abfragen und deren Wahrnehmung durch den Nutzer im Zusammenhang. ESM-Abfragen müssen in Situationen ausgesandt werden, welche für den Studiendesigner von Interesse sind. Dies bedarf eines akkuraten Erkennungssystems, welches in die ESM-App eingebunden werden muss. Sowohl die Anzahl und Häufigkeit der Abfragen als auch die Länge des Feedback-Fragebogens sollten auf ein Minimum reduziert werden. Beides sind Herausforderungen, welche die ESM-App, welche zur Durchführung der Studie genutzt wird, adressieren muss. Um das Erstellen von ESM-Anwendungen zu erleichtern, ist es empfehlenswert, auf ein primäres Entwicklungswerkzeug zurückzugreifen. Im besten Fall ist solch ein Werkzeug einfach zu nutzen und bietet Zugriff auf eine weitreichende Menge an Sensoren, aus denen kontextuelle Informationen abgeleitet werden können - beispielsweise, um ereignisbasiert Abfragen auszusenden. Im Rahmen dieser Dissertation stellen wir ESMAC vor, den ESM App Configurator. ESMAC stellt verschiedene Abfragetypen zur Verfügen, ebenso wie verschiedene Einstellungen, um die Anzahl an Abfragen pro Tag zu begrenzen (inquiry limit) oder um ein abfragefreies Zeitfenster zwischen zwei aufeinanderfolgenden Abfragen zu definieren (inter-notification time). Zudem bietet es Zugriff auf eine Vielzahl an Sensormesswerten und -Informationen.Diese Werte werden automatisch erfasst und benötigen keine Abfrage vom Nutzer, was zu einer reduzierten Fragebogenlänge führen kann. Um Informationen in Situationen zu sammeln, welche für den Studiendesigner von Interesse sind, bietet ESMAC eine Auswahl an ereignisbasierten Abfragen. Ereignisbasierte Abfragen fanden bereits in diversen ESM-Studien Anwendung. Dennoch wurde ihre Nützlichkeit bisher nicht explizit untersucht. Zwei Faktoren, welche für verschiedene Forschungsbereiche relevant sind, sind Ortswechsel und Aktivitätsänderungen des Nutzers. Diese können beispielsweise für die Erkennung der Unterbrechbarkeit eines Nutzers genutzt werden oder zum Überwachen von Zustandsänderungen bei Patienten, welche unter affektiven Störungen leiden. Am Beispiel einer Studie, welche auf die Erfassung dieser beiden Faktoren ausgerichtet ist, zeigen wir, dass ereignisbasierte Abfragen nützlich sind, vor allem wenn die ausgewählten ereignisbasierten Abfragen (hier: Ortswechsel) im Zusammenhang mit den zu erfassenden Daten stehen (hier: Feedback über die Mobilität und Aktivität des Nutzers). Die Erfassung von Datenlabels bedarf nicht nur ereignisbasierter Abfragen, sondern auch zeitnaher Antworten von den Probanden, um die Labels möglichst akkurat den gesammelten Daten zuweisen zu können. Hierzu ist es notwendig, dass die Probanden die eingehenden Abfragen rechtzeitig bemerken. Abfragen werden unter Umständen nicht wahrgenommen, weil eine zu unauffällige Benachrichtigungsmodalität gewählt wurde oder weil die ESM-Abfragen in einem überfüllten Notification Drawer des Smartphones untergehen. Die Wahrnehmbarkeit von Benachrichtigungen wird durch verschiedene kontextuelle Faktoren beeinflusst, z.B. die Position des Smartphones, den aktuellen Ort oder die (soziale) Aktivität des Nutzers. Aber auch inhaltliche Eigenschaften wie die empfundene Wichtigkeit einer Benachrichtigung können einen Einfluss haben. Als Grundlage für spätere Forschung untersuchen wir Methoden, um diese Einflussfaktoren zu erfassen. Zuerst stellen wir eine Methode zur Position-Transition-Korrektur vor, welche die Erkennung der aktuellen Smartphone-Position verbessert. Diese Methode basiert auf der Annahme, dass jeder Wechsel von einer Position zur nächsten über das Halten des Geräts in der Hand erfolgt. Als nächstes untersuchen wir verschiedene Methoden zur Ortserfassung, unter Achtung der Privatsphäre des Benutzers. Wir stellen vor, wie WLAN-Informationen und Ortstypen genutzt werden können, um den Aufenthaltsort eines Nutzers zu beschreiben und Ortswechsel zu erkennen, ohne den exakten Standort abzuspeichern. Basierend auf dem Ortstypen präsentieren wir eine Methode, um abzuschätzen, ob ein Smartphone-Nutzer in Begleitung ist. Abschließend untersuchen wir noch Smartphone-Features, welche mit der empfundenen Wichtigkeit einer Benachrichtigung in Zusammenhang stehen könnten. Nachdem wir Methoden zum Erfassen von Einflussfaktoren untersucht haben, betrachten wir Zusammenhänge zwischen der Wahrnehmung von eingehenden Benachrichtigungen und verschiedenen Benachrichtigungsmodalitäten. Diese Betrachtung erfolgt unter Berücksichtigung (a) der aktuellen Position des Smartphones und (b) des aktuellen Ortes des Smartphone-Nutzers und möglicher ortsbasierter Aktivitäten. Wir stellen eine Studie vor, welche Aufschluss darüber gibt, wie angenehm und wahrnehmbar verschiedene Benachrichtigungsmodalitäten sind - abhängig davon, wo das Smartphone vom Nutzer aufbewahrt wird. Für den aktuellen Ort und ortsbezogene Aktivitäten stellen wir passende Benachrichtigungsmodalitäten vor, über welche wir im Rahmen einer Onlineumfrage und einer Laborstudie Rückmeldung erhalten haben. Abschließend erstellen und evaluieren wir verschiedene Designs, um wichtige Benachrichtigungen - welche ESM-Abfragen einschließen - hervorzuheben, indem ihre Sichtbarkeit im Notification Drawer erhöht wird. Diese Designs basieren auf Feedback von Interviewprobanden als auch auf Erkenntnissen aus der Literatur. Wir stellen Eigenschaften von Benachrichtigungsdesigns vor, welche von Probanden einer Onlineumfrage als angenehm und nützlich empfunden wurden. Zudem empfehlen wir auch Kombinationen verschiedener Designeigenschaften. Die Beiträge dieser Dissertation können wie folgt zusammengefasst werden: - Vorstellung eines Tools, um kontextsensitive ESM-Apps zu erstellen - Bestätigung der Relevanz von ereignisbasierten Abfragen am Beispiel einer ESM-Studie mit Fokus auf Ortswechsel und Aktivitätsänderungen - Vorstellung eines Position-Transition-Korrekturmechanismus zum Verbessern der Erkennung der Smartphone-Position - Vorstellung zweier Methoden zur Ortserfassung ohne konkrete Offenlegung und Speicherung des konkreten Aufenthaltsortes - Vorstellung einer ortsbasierten Methode zum Abschätzen, ob sich ein Smartphone-Nutzer in Begleitung befindet oder nicht - Vorstellen von vier Typen von Wichtigkeit und von Smartphone-Features, welche mit der empfundenen Wichtigkeit von Benachrichtigungen in Zusammenhang stehen - Empfehlungen für die Auswahl von Benachrichtigungsmodalitäten abhängig von der (a) Smartphone-Position als auch (b) des aktuellen Ortes und möglicher ortsbasierter Aktivitäten - Empfehlungen für Designanpassungen von Smartphone-Benachrichtigungen, um solche von höherer Wichtigkeit hervorzuhebe

Mapping Sensorimotor Function and Controlling Upper Limb Neuroprosthetics with Electrocorticography

Electrocorticography (ECoG) occupies a unique intermediate niche between microelectrode recordings of single neurons and recordings of whole brain activity via functional magnetic resonance imaging (fMRI). ECoG’s combination of high temporal resolution and wide area coverage make it an ideal modality for both functional brain mapping and brain-machine interface (BMI) for control of prosthetic devices. This thesis demonstrates the utility of ECoG, particularly in high gamma frequencies (70-120 Hz), for passive online mapping of language and motor behaviors, online control of reaching and grasping of an advanced robotic upper limb, and mapping somatosensory digit representations in the postcentral gyrus. The dissertation begins with a brief discussion of the framework for neuroprosthetic control developed by the collaboration between Johns Hopkins and JHU Applied Physics Laboratory (JHU/APL). Second, the methodology behind an online spatial-temporal functional mapping (STFM) system is described. Trial-averaged spatiotemporal maps of high gamma activity were computed during a visual naming and a word reading task. The system output is subsequently shown and compared to stimulation mapping. Third, simultaneous and independent ECoG-based control of reaching and grasping is demonstrated with the Modular Prosthetic Limb (MPL). The STFM system was used to identify channels whose high gamma power significantly and selectively increases during either reaching or grasping. Using this technique, two patients were able to rapidly achieve naturalistic control over simple movements by the MPL. Next, high-density ECoG (hdECoG) was used to map the cortical responses to mechanical vibration of the fingertips. High gamma responses exhibited a strong yet overlapping somatotopy that was not well replicated in other frequency bands. These responses are strong enough to be detected in single trials and used to classify the finger being stimulated with over 98% accuracy. Finally, the role of ECoG is discussed for functional mapping and BMI applications. ECoG occupies a unique role among neural recording modalities as a tool for functional mapping, but must prove its value relative to stimulation mapping. For BMI, ECoG lags microelectrode arrays but hdECoG may provide a more robust long-term interface with optimal spacing for sampling relevant cortical representations

Scalable Teaching and Learning via Intelligent User Interfaces

The increasing demand for higher education and the educational budget cuts lead to large class sizes. Learning at scale is also the norm in Massive Open Online Courses (MOOCs). While it seems cost-effective, the massive scale of class challenges the adoption of proven pedagogical approaches and practices that work well in small classes, especially those that emphasize interactivity, active learning, and personalized learning. As a result, the standard teaching approach in today’s large classes is still lectured-based and teacher-centric, with limited active learning activities, and with relatively low teaching and learning effectiveness. This dissertation explores the usage of Intelligent User Interfaces (IUIs) to facilitate the efficient and effective adoption of the tried-and-true pedagogies at scale. The first system is MindMiner, an instructor-side data exploration and visualization system for peer review understanding. MindMiner helps instructors externalize and quantify their subjective domain knowledge, interactively make sense of student peer review data, and improve data exploration efficiency via distance metric learning. MindMiner also helps instructors generate customized feedback to students at scale. We then present BayesHeart, a probabilistic approach for implicit heart rate monitoring on smartphones. When integrated with MOOC mobile clients, BayesHeart can capture learners’ heart rates implicitly when they watch videos. Such information is the foundation of learner attention/affect modeling, which enables a ‘sensorless’ and scalable feedback channel from students to instructors. We then present CourseMIRROR, an intelligent mobile system integrated with Natural Language Processing (NLP) techniques that enables scalable reflection prompts in large classrooms. CourseMIRROR 1) automatically reminds and collects students’ in-situ written reflections after each lecture; 2) continuously monitors the quality of a student’s reflection at composition time and generates helpful feedback to scaffold reflection writing; 3) summarizes the reflections and presents the most significant ones to both instructors and students. Last, we present ToneWars, an educational game connecting Chinese as a Second Language (CSL) learners with native speakers via collaborative mobile gameplay. We present a scalable approach to enable authentic competition and skill comparison with native speakers by modeling their interaction patterns and language skills asynchronously. We also prove the effectiveness of such modeling in a longitudinal study

Eating Behavior In-The-Wild and Its Relationship to Mental Well-Being

The motivation for eating is beyond survival. Eating serves as means for socializing, exploring cultures, etc. Computing researchers have developed various eating detection technologies that can leverage passive sensors available on smart devices to automatically infer when and, to some extent, what an individual is eating. However, despite their significance in eating literature, crucial contextual information such as meal company, type of food, location of meals, the motivation of eating episodes, the timing of meals, etc., are difficult to detect through passive means. More importantly, the applications of currently developed automated eating detection systems are limited. My dissertation addresses several of these challenges by combining the strengths of passive sensing technologies and EMAs (Ecological Momentary Assessment). EMAs are a widely adopted tool used across a variety of disciplines that can gather in-situ information about individual experiences. In my dissertation, I demonstrate the relationship between various eating contexts and the mental well-being of college students and information workers through naturalistic studies. The contributions of my dissertation are four-fold. First, I develop a real-time meal detection system that can detect meal-level episodes and trigger EMAs to gather contextual data about one’s eating episode. Second, I deploy this system in a college student population to understand their eating behavior during day-to-day life and investigate the relationship of these eating behaviors with various mental well-being outcomes. Third, based on the limitations of passive sensing systems to detect short and sporadic chewing episodes present in snacking, I develop a snacking detection system and operationalize the definition of snacking in this thesis. Finally, I investigate the causal relationship between stress levels experienced by remote information workers during their workdays and its effect on lunchtime. This dissertation situates the findings in an interdisciplinary context, including ubiquitous computing, psychology, and nutrition.Ph.D

Understanding receptivity to interruptions in mobile human-computer interaction

Interruptions have a profound impact on our attentional orientation in everyday life. Recent advances in mobile information technology increase the number of potentially disruptive notifications on mobile devices by an increasing availability of services. Understanding the contextual intricacies that make us receptive to these interruptions is paramount to devising technology that supports interruption management. This thesis makes a number of contributions to the methodology of studying mobile experiences in situ, understanding receptivity to interruptions, and designing context-sensitive systems. This thesis presents a series of real-world studies that investigate opportune moments for interruptions in mobile settings. In order to facilitate the study of the multi-faceted ways opportune moments surface from participants' involvement in the world this thesis develops: - a model of the contextual factors that interact to guide receptivity to interruptions, and - an adaptation of the Experience-Sampling Method (ESM) to capture behavioural response to interruptions in situ. In two naturalistic experiments, participants' experiences of being interrupted on a mobile phone are sampled as they go about their everyday lives. In a field study, participants' experiences are observed and recorded as they use a notification-driven mobile application to create photo-stories in a theme park. Experiment 1 explores the effects of content and time of delivery of the interruption. The results show that receptivity to text messages is significantly affected by message content, while scheduling one's own interruption times in advance does not improve receptivity over randomly timed interruptions. Experiment 2 investigates the hypothesis that opportune moments to deliver notifications are located at the endings of episodes of mobile interaction such as texting and calling. This notification strategy is supported by significant effects in behavioural measures of receptivity, while self-reports and interviews reveal complexities in the subjective experience of the interruption. By employing a mixed methods approach of interviews, observations and an analysis of system logs in the field study, it is shown that participants appreciated location-based notifications as prompts to foreground the application during relative 'downtimes' from other activities. However, an unexpected quantity of redundant notifications meant that visitors soon habituated to and eventually ignored them, which suggests careful, sparing use of notifications in interactive experiences. Overall, the studies showed that contextual mediation of the timing of interruptions (e.g. by phone activity in Experiment 2 and opportune places in the field study) is more likely to lead to interruptions at opportune moments than when participants schedule their own interruptions. However, momentary receptivity and responsiveness to an interruption is determined by the complex and situated interactions of local and relational contextual factors. These contextual factors are captured in a model of receptivity that underlies the interruption process. The studies highlight implications for the design of systems that seek to manage interruptions by adapting the timing of interruptions to the user's situation. In particular, applications to manage interruptions in personal communication and pervasive experiences are considered

The potential of emerging wearable physiological sensing in the space of human-subject studies

PhD ThesisIn recent years, novel sensing means in the form of smartwatches and fitness trackers with integrated sophisticated sensing emerged on the consumer market. While their primary purpose is to provide consumers with an overview of rough-grained health-related metrics, these signals offer to pick up fine-grained changes within the human body. This thesis considers the suitability of these novel wearable sensing devices to be used in affective research. Firstly, and based on the work with concrete state-of-the-art wearables, issues around the access of research-suitable data are discussed. The findings are put in context by examining common wearable device architectures and data access means provided. The discussion concludes with aspects researchers need to consider when seeking data access from state-of-the-art or future wearables. Secondly, two research probes explore the application of four exemplary devices to detect stress and affect in the wild and in the lab. Issues around the data reliability and participant comfort arose. The experiences are reflected upon to provide researchers with a summary of aspects to consider when applying wearable sensing devices in affective research. Lastly, this thesis contributes a Design Space for Physiological Measurement Tools. This design space was evaluated with a qualitative study enquiring research experts experiences. The resulting Design Space presents seven distinct dimensions of factors to consider when choosing a wearable sensing device for research. This design space has been applied to a novel sensing device which was used for a study on interpersonal synchrony. The insights and the ‘Design Space for Physiological Measurement Tools’ provide researchers with a tool to apply when they consider to use wearable physiological sensing devices in research

Understanding receptivity to interruptions in mobile human-computer interaction

Interruptions have a profound impact on our attentional orientation in everyday life. Recent advances in mobile information technology increase the number of potentially disruptive notifications on mobile devices by an increasing availability of services. Understanding the contextual intricacies that make us receptive to these interruptions is paramount to devising technology that supports interruption management. This thesis makes a number of contributions to the methodology of studying mobile experiences in situ, understanding receptivity to interruptions, and designing context-sensitive systems. This thesis presents a series of real-world studies that investigate opportune moments for interruptions in mobile settings. In order to facilitate the study of the multi-faceted ways opportune moments surface from participants' involvement in the world this thesis develops: - a model of the contextual factors that interact to guide receptivity to interruptions, and - an adaptation of the Experience-Sampling Method (ESM) to capture behavioural response to interruptions in situ. In two naturalistic experiments, participants' experiences of being interrupted on a mobile phone are sampled as they go about their everyday lives. In a field study, participants' experiences are observed and recorded as they use a notification-driven mobile application to create photo-stories in a theme park. Experiment 1 explores the effects of content and time of delivery of the interruption. The results show that receptivity to text messages is significantly affected by message content, while scheduling one's own interruption times in advance does not improve receptivity over randomly timed interruptions. Experiment 2 investigates the hypothesis that opportune moments to deliver notifications are located at the endings of episodes of mobile interaction such as texting and calling. This notification strategy is supported by significant effects in behavioural measures of receptivity, while self-reports and interviews reveal complexities in the subjective experience of the interruption. By employing a mixed methods approach of interviews, observations and an analysis of system logs in the field study, it is shown that participants appreciated location-based notifications as prompts to foreground the application during relative 'downtimes' from other activities. However, an unexpected quantity of redundant notifications meant that visitors soon habituated to and eventually ignored them, which suggests careful, sparing use of notifications in interactive experiences. Overall, the studies showed that contextual mediation of the timing of interruptions (e.g. by phone activity in Experiment 2 and opportune places in the field study) is more likely to lead to interruptions at opportune moments than when participants schedule their own interruptions. However, momentary receptivity and responsiveness to an interruption is determined by the complex and situated interactions of local and relational contextual factors. These contextual factors are captured in a model of receptivity that underlies the interruption process. The studies highlight implications for the design of systems that seek to manage interruptions by adapting the timing of interruptions to the user's situation. In particular, applications to manage interruptions in personal communication and pervasive experiences are considered

Decoding user behaviour from Smartphone interaction event streams

The smartphone has become an everyday device for many people around the world and has led to an evolution in the way we use these devices. This has led to increased research interest in the effects of smartphone use on psychological traits, which could have a positive impact in clinical or self-help settings by identifying positively influencing variables. In this thesis, a new model to extract behaviour information from a stream of usage is presented. The model aligns with previous methods in the research area but focuses on establishing a generalisable three-step process of processing user interaction to extract new user behaviour knowledge. This introduces a structured approach to smartphone usage evaluation and enables the implementation of customisable applications. It also creates a baseline to compare previously defined metrics which describe smartphone usage. Usage derived from metrics which could be considered high-level such as screen-on time is self-evident and therefore are common measure to distinguish usage between users. However, within usage sessions, they suffer from limitations such as a strong skew towards short bursts of usage because of how smartphones are often used. By utilising direct interactions with the user interface (such as taps and scrolls), usage at a lower level can be considered which can carry more elemental characteristics of behaviour. Thus, they can be used to model behaviour more accurately, which can be aligned with the user’s mental state to identify habits which are caused by problematic use patterns. This enables the isolation of user trait classes reflecting smartphone addiction and impulsivity