Anticipatory Mobile Computing: A Survey of the State of the Art and Research Challenges

Today's mobile phones are far from mere communication devices they were ten years ago. Equipped with sophisticated sensors and advanced computing hardware, phones can be used to infer users' location, activity, social setting and more. As devices become increasingly intelligent, their capabilities evolve beyond inferring context to predicting it, and then reasoning and acting upon the predicted context. This article provides an overview of the current state of the art in mobile sensing and context prediction paving the way for full-fledged anticipatory mobile computing. We present a survey of phenomena that mobile phones can infer and predict, and offer a description of machine learning techniques used for such predictions. We then discuss proactive decision making and decision delivery via the user-device feedback loop. Finally, we discuss the challenges and opportunities of anticipatory mobile computing.Comment: 29 pages, 5 figure

Personalization in cultural heritage: the road travelled and the one ahead

Over the last 20 years, cultural heritage has been a favored domain for personalization research. For years, researchers have experimented with the cutting edge technology of the day; now, with the convergence of internet and wireless technology, and the increasing adoption of the Web as a platform for the publication of information, the visitor is able to exploit cultural heritage material before, during and after the visit, having different goals and requirements in each phase. However, cultural heritage sites have a huge amount of information to present, which must be filtered and personalized in order to enable the individual user to easily access it. Personalization of cultural heritage information requires a system that is able to model the user (e.g., interest, knowledge and other personal characteristics), as well as contextual aspects, select the most appropriate content, and deliver it in the most suitable way. It should be noted that achieving this result is extremely challenging in the case of first-time users, such as tourists who visit a cultural heritage site for the first time (and maybe the only time in their life). In addition, as tourism is a social activity, adapting to the individual is not enough because groups and communities have to be modeled and supported as well, taking into account their mutual interests, previous mutual experience, and requirements. How to model and represent the user(s) and the context of the visit and how to reason with regard to the information that is available are the challenges faced by researchers in personalization of cultural heritage. Notwithstanding the effort invested so far, a definite solution is far from being reached, mainly because new technology and new aspects of personalization are constantly being introduced. This article surveys the research in this area. Starting from the earlier systems, which presented cultural heritage information in kiosks, it summarizes the evolution of personalization techniques in museum web sites, virtual collections and mobile guides, until recent extension of cultural heritage toward the semantic and social web. The paper concludes with current challenges and points out areas where future research is needed

Low-resource Multi-task Audio Sensing for Mobile and Embedded Devices via Shared Deep Neural Network Representations

Continuous audio analysis from embedded and mobile devices is an increasingly important application domain. More and more, appliances like the Amazon Echo, along with smartphones and watches, and even research prototypes seek to perform multiple discriminative tasks simultaneously from ambient audio; for example, monitoring background sound classes (e.g., music or conversation), recognizing certain keywords (‘Hey Siri’ or ‘Alexa’), or identifying the user and her emotion from speech. The use of deep learning algorithms typically provides state-of-the-art model performances for such general audio tasks. However, the large computational demands of deep learning models are at odds with the limited processing, energy and memory resources of mobile, embedded and IoT devices. In this paper, we propose and evaluate a novel deep learning modeling and optimization framework that speci cally targets this category of embedded audio sensing tasks. Although the supported tasks are simpler than the task of speech recognition, this framework aims at maintaining accuracies in predictions while minimizing the overall processor resource footprint. The proposed model is grounded in multi-task learning principles to train shared deep layers and exploits, as input layer, only statistical summaries of audio lter banks to further lower computations. We nd that for embedded audio sensing tasks our framework is able to maintain similar accuracies, which are observed in comparable deep architectures that use single-task learning and typically more complex input layers. Most importantly, on an average, this approach provides almost a 2.1⇥ reduction in runtime, energy, and memory for four separate audio sensing tasks, assuming a variety of task combinations.Microsoft Researc

Employing Environmental Data and Machine Learning to Improve Mobile Health Receptivity

Behavioral intervention strategies can be enhanced by recognizing human activities using eHealth technologies. As we find after a thorough literature review, activity spotting and added insights may be used to detect daily routines inferring receptivity for mobile notifications similar to just-in-time support. Towards this end, this work develops a model, using machine learning, to analyze the motivation of digital mental health users that answer self-assessment questions in their everyday lives through an intelligent mobile application. A uniform and extensible sequence prediction model combining environmental data with everyday activities has been created and validated for proof of concept through an experiment. We find that the reported receptivity is not sequentially predictable on its own, the mean error and standard deviation are only slightly below by-chance comparison. Nevertheless, predicting the upcoming activity shows to cover about 39% of the day (up to 58% in the best case) and can be linked to user individual intervention preferences to indirectly find an opportune moment of receptivity. Therefore, we introduce an application comprising the influences of sensor data on activities and intervention thresholds, as well as allowing for preferred events on a weekly basis. As a result of combining those multiple approaches, promising avenues for innovative behavioral assessments are possible. Identifying and segmenting the appropriate set of activities is key. Consequently, deliberate and thoughtful design lays the foundation for further development within research projects by extending the activity weighting process or introducing a model reinforcement.BMBF, 13GW0157A, Verbundprojekt: Self-administered Psycho-TherApy-SystemS (SELFPASS) - Teilvorhaben: Data Analytics and Prescription for SELFPASSTU Berlin, Open-Access-Mittel - 201

Exploring Digital Literacy Practices in Middle School English Language Arts Classrooms

The intent of this mixed methods study was to explore what middle school English Language Arts (ELA) teachers believe are the necessary digital literacies required for students and to determine what teacher practices are currently being conducted to support these necessary literacies. To investigate digital literacy practices, three data collection instruments were developed for collecting data – an interview schedule designed for six secondary instructional coaches, a questionnaire that was administered to 122 middle school ELA teachers, and six classroom observations. The data were analyzed through inductive and descriptive approaches. The findings suggest that middle school teachers understand their role in the development of digital literacies and some teachers are working to navigate digital reading and writing practices. However, inconsistent use of digital tools and other barriers prevent the research site from fully supporting digital literacy development. A professional development proposal was developed that outlines suggested topics and learning outcomes

Exploring Digital Literacy Practices in Middle School English Language Arts Classrooms

The intent of this mixed methods study was to explore what middle school English Language Arts (ELA) teachers believe are the necessary digital literacies required for students and to determine what teacher practices are currently being conducted to support these necessary literacies. To investigate digital literacy practices, three data collection instruments were developed for collecting data – an interview schedule designed for six secondary instructional coaches, a questionnaire that was administered to 122 middle school ELA teachers, and six classroom observations. The data were analyzed through inductive and descriptive approaches. The findings suggest that middle school teachers understand their role in the development of digital literacies and some teachers are working to navigate digital reading and writing practices. However, inconsistent use of digital tools and other barriers prevent the research site from fully supporting digital literacy development. A professional development proposal was developed that outlines suggested topics and learning outcomes

Detecting Stressful Social Interactions Using Wearable Physiological and Inertial Sensors

Stress is unavoidable in everyday life which can result in several health related short and long-term adverse consequences. Previous research found that most of the stress events occur due to interpersonal tension followed by work related stress. Enabling automated detection of stressful social interactions using wearable technology will help trigger just-in-time interventions which can help the user cope with the stressful situation. In this dissertation, we show the feasibility of differentiating stressful social interactions from other stressors i.e., work and commute.However, collecting reliable ground truth stressor data in the natural environment is challenging. This dissertation addresses this challenge by designing a Day Reconstruction Method (DRM) based contextual stress visualization that highlights the continuous stress inferences from a wearable sensor with surrounding activities such as conversation, physical activity, and location on a timeline diagram. This dissertation proposes a Conditional Random Field, Context-Free Grammar (CRF-CFG) model to detect conversation from breathing patterns to support the visualization. The advantage of breathing signal is that it does not capture the content of the conversation and hence, is more privacy preserving compared to audio. It proposes a framework to systematically analyze the breathing data collected in the natural environment. However, it requires wearing of chest worn sensor. This dissertation aims to determine stressful social interaction without wearing chest worn sensor or without requiring any conversation model which is privacy sensitive. Therefore, it focuses on detecting stressful social interactions directly from stress time-series only which can be captured using increasingly available wrist worn sensor.This dissertation proposes a framework to systematically analyze the respiration data collected in the natural environment. The analysis includes screening the low-quality data, segmenting the respiration time-series by cycles, and develop time-domain features. It proposes a Conditional Random Field, Context-Free Grammar (CRF-CFG) model to detect conversation episodes from breathing patterns. This system is validated against audio ground-truth in the field with an accuracy of 71.7\%.This dissertation introduces the stress cycle concept to capture the cyclical patterns and identifies novel features from stress time-series data. Furthermore, wrist-worn accelerometry data shows that hand gestures have a distinct pattern during stressful social interactions. The model presented in this dissertation augments accelerometry patterns with the stress cycle patterns for more accurate detection. Finally, the model is trained and validated using data collected from 38 participants in free-living conditions. The model can detect the stressful interactions with an F1-score of 0.83 using stress cycle features and enable the delivery of stress intervention within 3.9 minutes since the onset of a stressful social interaction

Probabilistic modelling and inference of human behaviour from mobile phone time series

With an estimated 4.1 billion subscribers around the world, the mobile phone offers a unique opportunity to sense and understand human behaviour from location, co-presence and communication data. While the benefit of modelling this unprecedented amount of data is widely recognised, a number of challenges impede the development of accurate behaviour models. In this thesis, we identify and address two modelling problems and show that their consideration improves the accuracy of behaviour inference. We first examine the modelling of long-range dependencies in human behaviour. Human behaviour models only take into account short-range dependencies in mobile phone time series. Using information theory, we quantify long-range dependencies in mobile phone time series for the first time, demonstrate that they exhibit periodic oscillations and introduce novel tools to analyse them. We further show that considering what the user did 24 hours earlier improves accuracy when predicting user behaviour five hours or longer in advance. The second problem that we address is the modelling of temporal variations in human behaviour. The time spent by a user on an activity varies from one day to the next. In order to recognise behaviour patterns despite temporal variations, we establish a methodological connection between human behaviour modelling and biological sequence alignment. This connection allows us to compare, cluster and model behaviour sequences and introduce novel features for behaviour recognition which improve its accuracy. The experiments presented in this thesis have been conducted on the largest publicly available mobile phone dataset labelled in an unsupervised fashion and are entirely repeatable. Furthermore, our techniques only require cellular data which can easily be recorded by today's mobile phones and could benefit a wide range of applications including life logging, health monitoring, customer profiling and large-scale surveillance

Entity Recognition via Multimodal Sensor Fusion with Smart Phones

This thesis serves as an exploration that takes the sensors within a cell phone beyond the current state of recognition activities. Current state of the art sensor recognition processes tend to focus on recognizing user activity. Utilizing the same sensors available for user activity classification, this thesis validates the ability to gather data about entities separate from the user carrying the smart phone. With the ability to sense entities, the ability to recognize and classify a multitude of items, situations, and phenomena opens a new realm of possibilities for how devices perceive and react to their environment

Inferring Mood-While-Eating with Smartphone Sensing and Community-Based Model Personalization

The interplay between mood and eating has been the subject of extensive research within the fields of nutrition and behavioral science, indicating a strong connection between the two. Further, phone sensor data have been used to characterize both eating behavior and mood, independently, in the context of mobile food diaries and mobile health applications. However, limitations within the current body of literature include: i) the lack of investigation around the generalization of mood inference models trained with passive sensor data from a range of everyday life situations, to specific contexts such as eating, ii) no prior studies that use sensor data to study the intersection of mood and eating, and iii) the inadequate examination of model personalization techniques within limited label settings, as we commonly experience in mood inference. In this study, we sought to examine everyday eating behavior and mood using two datasets of college students in Mexico (N_mex = 84, 1843 mood-while-eating reports) and eight countries (N_mul = 678, 329K mood reports incl. 24K mood-while-eating reports), containing both passive smartphone sensing and self-report data. Our results indicate that generic mood inference models decline in performance in certain contexts, such as when eating. Additionally, we found that population-level (non-personalized) and hybrid (partially personalized) modeling techniques were inadequate for the commonly used three-class mood inference task (positive, neutral, negative). Furthermore, we found that user-level modeling was challenging for the majority of participants due to a lack of sufficient labels and data from the negative class. To address these limitations, we employed a novel community-based approach for personalization by building models with data from a set of similar users to a target user