Anticipatory Mobile Computing: A Survey of the State of the Art and Research Challenges

Today's mobile phones are far from mere communication devices they were ten years ago. Equipped with sophisticated sensors and advanced computing hardware, phones can be used to infer users' location, activity, social setting and more. As devices become increasingly intelligent, their capabilities evolve beyond inferring context to predicting it, and then reasoning and acting upon the predicted context. This article provides an overview of the current state of the art in mobile sensing and context prediction paving the way for full-fledged anticipatory mobile computing. We present a survey of phenomena that mobile phones can infer and predict, and offer a description of machine learning techniques used for such predictions. We then discuss proactive decision making and decision delivery via the user-device feedback loop. Finally, we discuss the challenges and opportunities of anticipatory mobile computing.Comment: 29 pages, 5 figure

SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party ) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals' personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.Comment: 14 pages, 11 figure

Energy-efficient Continuous Context Sensing on Mobile Phones

With the ever increasing adoption of smartphones worldwide, researchers have found the perfect sensor platform to perform context-based research and to prepare for context-based services to be also deployed for the end-users. However, continuous context sensing imposes a considerable challenge in balancing the energy consumption of the sensors, the accuracy of the recognized context and its latency. After outlining the common characteristics of continuous sensing systems, we present a detailed overview of the state of the art, from sensors sub-systems to context inference algorithms. Then, we present the three main contribution of this thesis. The first approach we present is based on the use of local communications to exchange sensing information with neighboring devices. As proximity, location and environmental information can be obtained from nearby smartphones, we design a protocol for synchronizing the exchanges and fairly distribute the sensing tasks. We show both theoretically and experimentally the reduction in energy needed when the devices can collaborate. The second approach focuses on the way to schedule mobile sensors, optimizing for both the accuracy and energy needs. We formulate the optimal sensing problem as a decision problem and propose a two-tier framework for approximating its solution. The first tier is responsible for segmenting the sensor measurement time series, by fitting various models. The second tier takes care of estimating the optimal sampling, selecting the measurements that contributes the most to the model accuracy. We provide near-optimal heuristics for both tiers and evaluate their performances using environmental sensor data. In the third approach we propose an online algorithm that identifies repeated patterns in time series and produces a compressed symbolic stream. The first symbolic transformation is based on clustering with the raw sensor data. Whereas the next iterations encode repetitive sequences of symbols into new symbols. We define also a metric to evaluate the symbolization methods with regard to their capacity at preserving the systems' states. We also show that the output of symbols can be used directly for various data mining tasks, such as classification or forecasting, without impacting much the accuracy, but greatly reducing the complexity and running time. In addition, we also present an example of application, assessing the user's exposure to air pollutants, which demonstrates the many opportunities to enhance contextual information when fusing sensor data from different sources. On one side we gather fine grained air quality information from mobile sensor deployments and aggregate them with an interpolation model. And, on the other side, we continuously capture the user's context, including location, activity and surrounding air quality. We also present the various models used for fusing all these information in order to produce the exposure estimation

AndroMedia : Towards a Context-aware Mobile Music Recommender

Portable music players have made it possible to listen to a personal collection of music in almost every situation, and they are often used during some activity to provide a stimulating audio environment. Studies have demonstrated the effects of music on the human body and mind, indicating that selecting music according to situation can, besides making the situation more enjoyable, also make humans perform better. For example, music can boost performance during physical exercises, alleviate stress and positively affect learning. We believe that people intuitively select different types of music for different situations. Based on this hypothesis, we propose a portable music player, AndroMedia, designed to provide personalised music recommendations using the user's current context and listening habits together with other user's situational listening patterns. We have developed a prototype that consists of a central server and a PDA client. The client uses Bluetooth sensors to acquire context information and logs user interaction to infer implicit user feedback. The user interface also allows the user to give explicit feedback. Large user interface elements facilitate touch-based usage in busy environments. The prototype provides the necessary framework for using the collected information together with other user's listening history in a context- enhanced collaborative filtering algorithm to generate context-sensitive recommendations. The current implementation is limited to using traditional collaborative filtering algorithms. We outline the techniques required to create context-aware recommendations and present a survey on mobile context-aware music recommenders found in literature. As opposed to the explored systems, AndroMedia utilises other users' listening habits when suggesting tunes, and does not require any laborious set up processes

Sensing and Visualizing Social Context from Spatial Proximity

The concept of pervasive computing, as introduced by Marc Weiser under the name ubiquitous computing in the early 90s, spurred research into various kinds of context-aware systems and applications. There is a wide range of contextual parameters, including location, time, temperature, devices and people in proximity, which have been part of the initial ideas about context-aware computing. While locational context is already a well understood concept, social context---based on the people around us---proves to be harder to grasp and to operationalize. This work continues the line of research into social context, which is based on the proximity and meeting patterns of people in the physical space. It takes this research out of the lab and out of well controlled situations into our urban environments, which are full of ambiguity and opportunities. The key to this research is the tool that caused dramatic change in individual and collective behavior during the last 20 years and which is a manifestation of many of the ideas of the pervasive computing paradigm: the mobile phone. In this work, the mobile is regarded as a proxy for people. Through it, the social environment becomes accessible to digital measurement and processing. To understand the large amount of data that now becomes available to automatic measurement, we will turn to the discipline of social network analysis. It provides powerful methods, that are able to condense data and extract relevant meaning. Visualization helps to understand and interpret the results. This thesis contains a number of experiments, that demonstrate how the automatic measurement of social proximity data through Bluetooth can be used to measure variables of personal behavior, group behavior and the behavior of groups in relation to places. The principal contributions are: * A methodology to visualize personal social context by using an ego proximity network. Specific episodes can be localized and compared. * method to compare different days in terms of social context, e.g. to support automatic diary applications. * A method to compose social geographic maps. Locations of similar social context are detected and combined. * Functions to measure short-term changes in social activity, based on the distinction between strange and familiar devices. * The characterization of Bluetooth inquiries for social proximity sensing. * A dataset of Bluetooth sightings from an ego perspective in seven different settings. Additionally, some settings feature multiple stationary scanners and Cell-ID measurements. * Soft- and hardware to capture, collect, store and analyze Bluetooth proximity data

Sensing and Signal Processing in Smart Healthcare

In the last decade, we have witnessed the rapid development of electronic technologies that are transforming our daily lives. Such technologies are often integrated with various sensors that facilitate the collection of human motion and physiological data and are equipped with wireless communication modules such as Bluetooth, radio frequency identification, and near-field communication. In smart healthcare applications, designing ergonomic and intuitive human–computer interfaces is crucial because a system that is not easy to use will create a huge obstacle to adoption and may significantly reduce the efficacy of the solution. Signal and data processing is another important consideration in smart healthcare applications because it must ensure high accuracy with a high level of confidence in order for the applications to be useful for clinicians in making diagnosis and treatment decisions. This Special Issue is a collection of 10 articles selected from a total of 26 contributions. These contributions span the areas of signal processing and smart healthcare systems mostly contributed by authors from Europe, including Italy, Spain, France, Portugal, Romania, Sweden, and Netherlands. Authors from China, Korea, Taiwan, Indonesia, and Ecuador are also included

From Signal to Social : Steps Towards Pervasive Social Context

The widespread adoption of smartphones with advanced sensing, computing and data transfer capabilities has made scientific studies of human social behavior possible at a previously unprecedented scale. It has also allowed context-awareness to become a natural feature in many applications using features such as activity recognition and location information. However, one of the most important aspects of context remains largely untapped at scale, i.e. social interactions and social context. Social interaction sensing has been explored using smartphones and specialized hardware for research purposes within computational social science and ubiquitous computing, but several obstacles remain to make it usable in practice by applications at industrial scale. In this thesis, I explore methods of physical proximity sensing and extraction of social context information from user-generated data for the purpose of context-aware applications. Furthermore, I explore the application space made possible through these methods, especially in the class of use cases that are characterized by embodied social agency, through field studies and a case study.A major concern when collecting context information is the impact on user privacy. I have performed a user study in which I have surveyed the user attitudes towards the privacy implications of proximity sensing. Finally, I present results from quantitatively estimating the sensitivity of a simple type of context information, i.e. application usage, in terms of risk of user re-identification