DSP.Ear: Leveraging co-processor support for continuous audio sensing on smartphones

The rapidly growing adoption of sensor-enabled smartphones has greatly fueled the proliferation of applications that use phone sensors to monitor user behavior. A central sensor among these is the microphone which enables, for instance, the detection of valence in speech, or the identification of speakers. Deploying multiple of these applications on a mobile device to continuously monitor the audio environment allows for the acquisition of a diverse range of sound-related contextual inferences. However, the cumulative processing burden critically impacts the phone battery. To address this problem, we propose DSP.Ear - an integrated sensing system that takes advantage of the latest low-power DSP co-processor technology in commodity mobile devices to enable the continuous and simultaneous operation of multiple established algorithms that perform complex audio inferences. The system extracts emotions from voice, estimates the number of people in a room, identifies the speakers, and detects commonly found ambient sounds, while critically incurring little overhead to the device battery. This is achieved through a series of pipeline optimizations that allow the computation to remain largely on the DSP. Through detailed evaluation of our prototype implementation we show that, by exploiting a smartphone's co-processor, DSP.Ear achieves a 3 to 7 times increase in the battery lifetime compared to a solution that uses only the phone's main processor. In addition, DSP.Ear is 2 to 3 times more power efficient than a naive DSP solution without optimizations. We further analyze a large-scale dataset from 1320 Android users to show that in about 80-90% of the daily usage instances DSP.Ear is able to sustain a full day of operation (even in the presence of other smartphone workloads) with a single battery charge.This work was supported by Microsoft Research through its PhD Scholarship Program.This is the author's accepted manuscript. The final version is available from ACM in the proceedings of the ACM Conference on Embedded Networked Sensor Systems: http://dl.acm.org/citation.cfm?id=2668349

LEO: Scheduling sensor inference algorithms across heterogeneous mobile processors and network resources

Mobile apps that use sensors to monitor user behavior often employ resource heavy inference algorithms that make computational offloading a common practice. However, existing schedulers/off loaders typically emphasize one primary offloading aspect without fully exploring complementary goals (e.g., heterogeneous resource management with only partial visibility into underlying algorithms, or concurrent sensor app execution on a single resource) and as a result, may overlook performance benefits pertinent to sensor processing. We bring together key ideas scattered in existing offloading solutions to build LEO - a scheduler designed to maximize the performance for the unique workload of continuous and intermittent mobile sensor apps without changing their inference accuracy. LEO makes use of domain specific signal processing knowledge to smartly distribute the sensor processing tasks across the broader range of heterogeneous computational resources of high-end phones (CPU, co-processor, GPU and the cloud). To exploit short-lived, but substantial optimization opportunities, and remain responsive to the needs of near real-time apps such as voice-based natural user interfaces, LEO runs as a service on a low-power co-processor unit (LPU) to perform both frequent and joint schedule optimization for concurrent pipelines. Depending on the workload and network conditions, LEO is between 1:6 and 3 times more energy efficient than conventional cloud offloading with CPU-bound sensor sampling. In addition, even if a general-purpose scheduler is optimized directly to leverage an LPU, we find LEO still uses only a fraction (< 1=7) of the energy overhead for scheduling and is up to 19% more energy efficient for medium to heavy workloads

Smart phone based systems for social psychological research: Challenges and design guidelines

Social psychology research deals with understanding many aspects of human behavior, and this helps not only to gain insights into this complex phenomenon but also to provide useful feedback to the participants. Social psychological research is mainly conducted through self-reports and surveys, however, this methodology is laborious and requires considerable offline analysis. Moreover, self-reports are also found to be biased towards pleasant experiences. Mobile phones represent a perfect platform for conducting social psychological research as they are ubiquitous, unobtrusive, and sensor-rich devices. However, limited battery and computing power, and expensive data plans make it difficult to support various demanding sensing and computation requirements of the social psychological research. In this paper, we describe the specific challenges in building systems based on off-the-shelf mobile phones for conducting social experiments, and provide design guidelines based on our recent works for implementing such systems. © 2011 ACM

Beyond location check-ins: Exploring physical and soft sensing to augment social check-in apps

Smartphone sensing research has been advancing at a brisk pace. Yet, current social networking services often only take advantage of location sensing: applications like Foursquare use the phone's GPS and Wi-Fi radios to infer the user's location to simplify checking-in to a place. However, smartphone sensing could be exploited to considerably expand the spectrum of information a user can share with a few clicks with friends: not only the location of an event but activities such as 'cooking dinner' or 'waiting for a bus' can be predicted and suggested to the user to ease the check-in process. In this paper we show how mobile phone sensing can be used in this sense. For this prediction process to be accurate however, sensors need to be sampled often, with a considerable impact on the phone battery. To alleviate this issue, we explore streams of phone usage data (soft sensors), such as application usage, messages, and phone calls for predicting the user's activity in a more efficient fashion for augmenting mobile social check-in apps. We have deployed our application and collected a dataset of over 2700 check-ins to 48 activities from 20 users. Our analysis shows a prediction accuracy of 75% when offering 5 check-in suggestions to users. Furthermore, we show that when using only soft sensors we can achieve very similar performance to that obtained with real sensors, thereby significantly reducing the impact on the phone battery. This finding might have a potentially high impact on smartphone based activity check-in apps

LEO: Scheduling sensor inference algorithms across heterogeneous mobile processors and network resources

Mobile apps that use sensors to monitor user behavior often employ resource heavy inference algorithms that make computational offloading a common practice. However, existing schedulers/off loaders typically emphasize one primary offloading aspect without fully exploring complementary goals (e.g., heterogeneous resource management with only partial visibility into underlying algorithms, or concurrent sensor app execution on a single resource) and as a result, may overlook performance benefits pertinent to sensor processing. We bring together key ideas scattered in existing offloading solutions to build LEO - a scheduler designed to maximize the performance for the unique workload of continuous and intermittent mobile sensor apps without changing their inference accuracy. LEO makes use of domain specific signal processing knowledge to smartly distribute the sensor processing tasks across the broader range of heterogeneous computational resources of high-end phones (CPU, co-processor, GPU and the cloud). To exploit short-lived, but substantial optimization opportunities, and remain responsive to the needs of near real-time apps such as voice-based natural user interfaces, LEO runs as a service on a low-power co-processor unit (LPU) to perform both frequent and joint schedule optimization for concurrent pipelines. Depending on the workload and network conditions, LEO is between 1:6 and 3 times more energy efficient than conventional cloud offloading with CPU-bound sensor sampling. In addition, even if a general-purpose scheduler is optimized directly to leverage an LPU, we find LEO still uses only a fraction (< 1=7) of the energy overhead for scheduling and is up to 19% more energy efficient for medium to heavy workloads

Open source smartphone libraries for computational social science

The ubiquity of sensor-rich and computationally powerful smartphones makes them an ideal platform for conducting social and behavioural research. However, building sensor data collection tools remains arduous and challenging: it requires an understanding of the varying sensor programming interfaces as well as the research issues related to building sensor-sampling systems. To alleviate this problem and facilitate the development of social sensing and data collection applications, we are developing a set of open-source smartphone libraries to collect, store and transfer, and query sensor data. Furthermore, we have also developed a library that can trigger notifications based on time or sensor events to assist experience sampling methods. This paper presents these libraries' architecture, initial feedback from developers using it, and a sensing application that we built using them to study daily affect. Copyright © 2013 ACM

Sociablesense: Exploring the Trade-offs of Adaptive Sampling and Computation Offloading for Social Sensing

The interactions and social relations among users in work-places have been studied by many generations of social psy-chologists. There is evidence that groups of users that in-teract more in workplaces are more productive. However, it is still hard for social scientists to capture fine-grained data about phenomena of this kind and to find the right means to facilitate interaction. It is also difficult for users to keep track of their level of sociability with colleagues. While mo-bile phones offer a fantastic platform for harvesting long term and fine grained data, they also pose challenges: bat-tery power is limited and needs to be traded-off for sensor reading accuracy and data transmission, while energy costs in processing computationally intensive tasks are high. In this paper, we propose SociableSense, a smart phone

SociableSense: Exploring the trade-offs of adaptive sampling and computation offloading for social sensing

The interactions and social relations among users in workplaces have been studied by many generations of social psychologists. There is evidence that groups of users that interact more in workplaces are more productive. However, it is still hard for social scientists to capture fine-grained data about phenomena of this kind and to find the right means to facilitate interaction. It is also difficult for users to keep track of their level of sociability with colleagues. While mobile phones offer a fantastic platform for harvesting long term and fine grained data, they also pose challenges: battery power is limited and needs to be traded-off for sensor reading accuracy and data transmission, while energy costs in processing computationally intensive tasks are high. In this paper, we propose SociableSense, a smart phones based platform that captures user behavior in office environments, while providing the users with a quantitative measure of their sociability and that of colleagues. We tackle the technical challenges of building such a tool: the system provides an adaptive sampling mechanism as well as models to decide whether to perform computation of tasks, such as the execution of classification and inference algorithms, locally or remotely. We perform several micro-benchmark tests to fine-tune and evaluate the performance of these mechanisms and we show that the adaptive sampling and computation distribution schemes balance trade-offs among accuracy, energy, latency, and data traffic. Finally, by means of a social psychological study with ten participants for two working weeks, we demonstrate that SociableSense fosters interactions among the participants and helps in enhancing their sociability. © 2011 ACM

Smartphone sensing offloading for efficiently supporting social sensing applications

Mobile phones play a pivotal role in supporting ubiquitous and unobtrusive sensing of human activities. However, maintaining a highly accurate record of a user's behavior throughout the day imposes significant energy demands on the phone's battery. In this work, we investigate a new approach that can lead to significant energy savings for mobile applications that require continuous sensing of social activities. This is achieved by opportunistically offloading sensing to sensors embedded in the environment, leveraging sensing that may be available in typical modern buildings (e.g., room occupancy sensors, RFID access control systems). In this article, we present the design, implementation, and evaluation of METIS: an adaptive mobile sensing platform that efficiently supports social sensing applications. The platform implements a novel sensor task distribution scheme that dynamically decides whether to perform sensing on the phone or in the infrastructure, considering the energy consumption, accuracy, and mobility patterns of the user. By comparing the sensing distribution scheme with sensing performed solely on the phone or exclusively on the fixed remote sensors, we show, through benchmarks using real traces, that the opportunistic sensing distribution achieves over 60% and 40% energy savings, respectively. This is confirmed through a real world deployment in an office environment for over a month: we developed a social application over our frameworks, that is able to infer the collaborations and meetings of the users. In this setting the system preserves over 35% more battery life over pure phone sensing. © 2013 Elsevier B.V. All rights reserved