A Modular Approach for Synchronized Wireless Multimodal Multisensor Data Acquisition in Highly Dynamic Social Settings

Existing data acquisition literature for human behavior research provides wired solutions, mainly for controlled laboratory setups. In uncontrolled free-standing conversation settings, where participants are free to walk around, these solutions are unsuitable. While wireless solutions are employed in the broadcasting industry, they can be prohibitively expensive. In this work, we propose a modular and cost-effective wireless approach for synchronized multisensor data acquisition of social human behavior. Our core idea involves a cost-accuracy trade-off by using Network Time Protocol (NTP) as a source reference for all sensors. While commonly used as a reference in ubiquitous computing, NTP is widely considered to be insufficiently accurate as a reference for video applications, where Precision Time Protocol (PTP) or Global Positioning System (GPS) based references are preferred. We argue and show, however, that the latency introduced by using NTP as a source reference is adequate for human behavior research, and the subsequent cost and modularity benefits are a desirable trade-off for applications in this domain. We also describe one instantiation of the approach deployed in a real-world experiment to demonstrate the practicality of our setup in-the-wild.Comment: 9 pages, 8 figures, Proceedings of the 28th ACM International Conference on Multimedia (MM '20), October 12--16, 2020, Seattle, WA, USA. First two authors contributed equall

Can smartwatches replace smartphones for posture tracking?

This paper introduces a human posture tracking platform to identify the human postures of sitting, standing or lying down, based on a smartwatch. This work develops such a system as a proof-of-concept study to investigate a smartwatch's ability to be used in future remote health monitoring systems and applications. This work validates the smartwatches' ability to track the posture of users accurately in a laboratory setting while reducing the sampling rate to potentially improve battery life, the first steps in verifying that such a system would work in future clinical settings. The algorithm developed classifies the transitions between three posture states of sitting, standing and lying down, by identifying these transition movements, as well as other movements that might be mistaken for these transitions. The system is trained and developed on a Samsung Galaxy Gear smartwatch, and the algorithm was validated through a leave-one-subject-out cross-validation of 20 subjects. The system can identify the appropriate transitions at only 10 Hz with an F-score of 0.930, indicating its ability to effectively replace smart phones, if needed

Experience: Design, Development and Evaluation of a Wearable Device for mHealth Applications

Wrist-worn devices hold great potential as a platform for mobile health (mHealth) applications because they comprise a familiar, convenient form factor and can embed sensors in proximity to the human body. Despite this potential, however, they are severely limited in battery life, storage, bandwidth, computing power, and screen size. In this paper, we describe the experience of the research and development team designing, implementing and evaluating Amulet – an open-hardware, open-software wrist-worn computing device – and its experience using Amulet to deploy mHealth apps in the field. In the past five years the team conducted 11 studies in the lab and in the field, involving 204 participants and collecting over 77,780 hours of sensor data. We describe the technical issues the team encountered and the lessons they learned, and conclude with a set of recommendations. We anticipate the experience described herein will be useful for the development of other research-oriented computing platforms. It should also be useful for researchers interested in developing and deploying mHealth applications, whether with the Amulet system or with other wearable platforms

Continuous Estimation of Smoking Lapse Risk from Noisy Wrist Sensor Data Using Sparse and Positive-Only Labels

Estimating the imminent risk of adverse health behaviors provides opportunities for developing effective behavioral intervention mechanisms to prevent the occurrence of the target behavior. One of the key goals is to find opportune moments for intervention by passively detecting the rising risk of an imminent adverse behavior. Significant progress in mobile health research and the ability to continuously sense internal and external states of individual health and behavior has paved the way for detecting diverse risk factors from mobile sensor data. The next frontier in this research is to account for the combined effects of these risk factors to produce a composite risk score of adverse behaviors using wearable sensors convenient for daily use. Developing a machine learning-based model for assessing the risk of smoking lapse in the natural environment faces significant outstanding challenges requiring the development of novel and unique methodologies for each of them. The first challenge is coming up with an accurate representation of noisy and incomplete sensor data to encode the present and historical influence of behavioral cues, mental states, and the interactions of individuals with their ever-changing environment. The next noteworthy challenge is the absence of confirmed negative labels of low-risk states and adequate precise annotations of high-risk states. Finally, the model should work on convenient wearable devices to facilitate widespread adoption in research and practice. In this dissertation, we develop methods that account for the multi-faceted nature of smoking lapse behavior to train and evaluate a machine learning model capable of estimating composite risk scores in the natural environment. We first develop mRisk, which combines the effects of various mHealth biomarkers such as stress, physical activity, and location history in producing the risk of smoking lapse using sequential deep neural networks. We propose an event-based encoding of sensor data to reduce the effect of noises and then present an approach to efficiently model the historical influence of recent and past sensor-derived contexts on the likelihood of smoking lapse. To circumvent the lack of confirmed negative labels (i.e., annotated low-risk moments) and only a few positive labels (i.e., sensor-based detection of smoking lapse corroborated by self-reports), we propose a new loss function to accurately optimize the models. We build the mRisk models using biomarker (stress, physical activity) streams derived from chest-worn sensors. Adapting the models to work with less invasive and more convenient wrist-based sensors requires adapting the biomarker detection models to work with wrist-worn sensor data. To that end, we develop robust stress and activity inference methodologies from noisy wrist-sensor data. We first propose CQP, which quantifies wrist-sensor collected PPG data quality. Next, we show that integrating CQP within the inference pipeline improves accuracy-yield trade-offs associated with stress detection from wrist-worn PPG sensors in the natural environment. mRisk also requires sensor-based precise detection of smoking events and confirmation through self-reports to extract positive labels. Hence, we develop rSmoke, an orientation-invariant smoking detection model that is robust to the variations in sensor data resulting from orientation switches in the field. We train the proposed mRisk risk estimation models using the wrist-based inferences of lapse risk factors. To evaluate the utility of the risk models, we simulate the delivery of intelligent smoking interventions to at-risk participants as informed by the composite risk scores. Our results demonstrate the envisaged impact of machine learning-based models operating on wrist-worn wearable sensor data to output continuous smoking lapse risk scores. The novel methodologies we propose throughout this dissertation help instigate a new frontier in smoking research that can potentially improve the smoking abstinence rate in participants willing to quit

Bio-inspired soft robotic systems: Exploiting environmental interactions using embodied mechanics and sensory coordination

Despite the widespread development of highly intelligent robotic systems exhibiting great precision, reliability, and dexterity, robots remain incapable of performing basic manipulation tasks that humans take for granted. Manipulation in unstructured environments continues to be acknowledged as a significant challenge. Soft robotics, the use of less rigid materials in robots, has been proposed as one means of addressing these limitations. The technique enables more compliant interactions with the environment, allowing for increasingly adaptive behaviours better suited to more human-centric applications. Embodied intelligence is a biologically inspired concept in which intelligence is a function of the entire system, not only the controller or `brain'. This thesis focuses on the use of embodied intelligence for the development of soft robots, with a particular focus on how it can aid both perception and adaptability. Two main hypotheses are raised: first, that the mechanical design and fabrication of soft-rigid hybrid robots can enable increasingly environmentally adaptive behaviours, and second, that sensing materials and morphology can provide intelligence that assists perception through embodiment. A number of approaches and frameworks for the design and development of embodied systems are presented that address these hypotheses. It is shown how embodiment in soft sensor morphology can be used to perform localised processing and thereby distribute the intelligence over the body of a system. Specifically in soft robots, sensor morphology utilises the directional deformations created by interactions with the environment to aid in perception. Building on and formalising these ideas, a number of morphology-based frameworks are proposed for detecting different stimuli. The multifaceted role of materials in soft robots is demonstrated through the development of materials capable of both sensing and changes in material property. Such materials provide additional functionality beyond their integral scaffolding and static mechanical characteristics. In particular, an integrated material has been created exhibiting both sensing capabilities and also variable stiffness and `tack’ force, thereby enabling complex single-point grasping. To maximise the intelligence that can be gained through embodiment, a design approach to soft robots, `soft-rigid hybrid' design is introduced. This approach exploits passive behaviours and body dynamics to provide environmentally adaptive behaviours and sensing. It is leveraged by multi-material 3D printing techniques and novel approaches and frameworks for designing mechanical structures. The findings in this thesis demonstrate that an embodied approach to soft robotics provides capabilities and behaviours that are not currently otherwise achievable. Utilising the concept of `embodiment' results in softer robots with an embodied intelligence that aids perception and adaptive behaviours, and has the potential to bring the physical abilities of robots one step closer to those of animals and humans.EPSR