Situation inference and context recognition for intelligent mobile sensing applications

The usage of smart devices is an integral element in our daily life. With the richness of data streaming from sensors embedded in these smart devices, the applications of ubiquitous computing are limitless for future intelligent systems. Situation inference is a non-trivial issue in the domain of ubiquitous computing research due to the challenges of mobile sensing in unrestricted environments. There are various advantages to having robust and intelligent situation inference from data streamed by mobile sensors. For instance, we would be able to gain a deeper understanding of human behaviours in certain situations via a mobile sensing paradigm. It can then be used to recommend resources or actions for enhanced cognitive augmentation, such as improved productivity and better human decision making. Sensor data can be streamed continuously from heterogeneous sources with different frequencies in a pervasive sensing environment (e.g., smart home). It is difficult and time-consuming to build a model that is capable of recognising multiple activities. These activities can be performed simultaneously with different granularities. We investigate the separability aspect of multiple activities in time-series data and develop OPTWIN as a technique to determine the optimal time window size to be used in a segmentation process. As a result, this novel technique reduces need for sensitivity analysis, which is an inherently time consuming task. To achieve an effective outcome, OPTWIN leverages multi-objective optimisation by minimising the impurity (the number of overlapped windows of human activity labels on one label space over time series data) while maximising class separability. The next issue is to effectively model and recognise multiple activities based on the user's contexts. Hence, an intelligent system should address the problem of multi-activity and context recognition prior to the situation inference process in mobile sensing applications. The performance of simultaneous recognition of human activities and contexts can be easily affected by the choices of modelling approaches to build an intelligent model. We investigate the associations of these activities and contexts at multiple levels of mobile sensing perspectives to reveal the dependency property in multi-context recognition problem. We design a Mobile Context Recognition System, which incorporates a Context-based Activity Recognition (CBAR) modelling approach to produce effective outcome from both multi-stage and multi-target inference processes to recognise human activities and their contexts simultaneously. Upon our empirical evaluation on real-world datasets, the CBAR modelling approach has significantly improved the overall accuracy of simultaneous inference on transportation mode and human activity of mobile users. The accuracy of activity and context recognition can also be influenced progressively by how reliable user annotations are. Essentially, reliable user annotation is required for activity and context recognition. These annotations are usually acquired during data capture in the world. We research the needs of reducing user burden effectively during mobile sensor data collection, through experience sampling of these annotations in-the-wild. To this end, we design CoAct-nnotate --- a technique that aims to improve the sampling of human activities and contexts by providing accurate annotation prediction and facilitates interactive user feedback acquisition for ubiquitous sensing. CoAct-nnotate incorporates a novel multi-view multi-instance learning mechanism to perform more accurate annotation prediction. It also includes a progressive learning process (i.e., model retraining based on co-training and active learning) to improve its predictive performance over time. Moving beyond context recognition of mobile users, human activities can be related to essential tasks that the users perform in daily life. Conversely, the boundaries between the types of tasks are inherently difficult to establish, as they can be defined differently from the individuals' perspectives. Consequently, we investigate the implication of contextual signals for user tasks in mobile sensing applications. To define the boundary of tasks and hence recognise them, we incorporate such situation inference process (i.e., task recognition) into the proposed Intelligent Task Recognition (ITR) framework to learn users' Cyber-Physical-Social activities from their mobile sensing data. By recognising the engaged tasks accurately at a given time via mobile sensing, an intelligent system can then offer proactive supports to its user to progress and complete their tasks. Finally, for robust and effective learning of mobile sensing data from heterogeneous sources (e.g., Internet-of-Things in a mobile crowdsensing scenario), we investigate the utility of sensor data in provisioning their storage and design QDaS --- an application agnostic framework for quality-driven data summarisation. This allows an effective data summarisation by performing density-based clustering on multivariate time series data from a selected source (i.e., data provider). Thus, the source selection process is determined by the measure of data quality. Nevertheless, this framework allows intelligent systems to retain comparable predictive results by its effective learning on the compact representations of mobile sensing data, while having a higher space saving ratio. This thesis contains novel contributions in terms of the techniques that can be employed for mobile situation inference and context recognition, especially in the domain of ubiquitous computing and intelligent assistive technologies. This research implements and extends the capabilities of machine learning techniques to solve real-world problems on multi-context recognition, mobile data summarisation and situation inference from mobile sensing. We firmly believe that the contributions in this research will help the future study to move forward in building more intelligent systems and applications

Addressing Variability in Speech when Recognizing Emotion and Mood In-the-Wild

Bipolar disorder is a chronic mental illness, affecting 4% of Americans, that is characterized by periodic mood changes ranging from severe depression to extreme compulsive highs. Both mania and depression profoundly impact the behavior of affected individuals, resulting in potentially devastating personal and social consequences. Bipolar disorder is managed clinically with regular interactions with care providers, who assess mood, energy levels, and the form and content of speech. Recent work has proposed smartphones for automatically monitoring mood using speech. Much of the early work in speech-centered mood detection has been done in the laboratory or clinic and is not reflective of the variability found in real-world conversations and conditions. Outside of these settings, automatic mood detection is hard, as the recordings include environmental noise, differences in recording devices, and variations in subject speaking patterns. Without addressing these issues, it is difficult to move towards a passive mobile health system. My research works to address this variability present in speech so that such a system can be created, allowing for interventions to mitigate the life-changing effects of mood transitions. However detecting mood directly from speech is difficult, as mood varies over the course of days or weeks, while speech fluctuates rapidly. To address this, my thesis explores how an intermediate step can be used to aid in this prediction. For example, one of the major symptoms of bipolar disorder is emotion dysregulation - changes in the way emotions are perceived and a lack of inhibition in their expression. My work has supported the relationship between automatically extracted emotion estimates and mood. Because of this, my thesis explores how to mitigate the variability found when detecting emotion from speech. The remainder of my thesis is focused on employing these emotion-based features, as well as features based on language content, to real-world applications. This dissertation is divided into the following parts: Part I: I address the direct classification of mood from speech. This is accomplished by addressing variability due to recording device using preprocessing and multi-task learning. I then show how both subject-specific and population-general information can be combined to significantly improve mood detection. Part II: I explore the automatic detection of emotion from speech and how to control for the other factors of variability present in the speech signal. I use progressive networks as a method to augment emotion with other paralinguistic data including gender and speaker, as well as other datasets. Additionally, I introduce a novel domain generalization method for cross-corpus detection. Part III: I demonstrate real-world applications of speech mood monitoring using everyday conversations. I show how the previously introduced generalized model can predict emotion from the speech of individuals with suicidal ideation, demonstrating its effectiveness across domains. Furthermore, I use these predictions to distinguish individuals with suicidal thoughts from healthy controls. Lastly, I introduce a novel framework for intervention detection in individuals with bipolar disorder. I then create a natural speech mood monitoring system based on features derived from measures of emotion and automatic speech recognition (ASR) transcripts and show effective intervention detection. I conclude this dissertation with the following future directions: (1) Extending my emotion generalization system to include multiple modalities and factors of variability; (2) Expanding natural speech mood monitoring by including more devices, exploring other data besides speech, and investigating mood rating causality.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/153461/1/gideonjn_1.pd

Gaussian processes for modeling of facial expressions

Automated analysis of facial expressions has been gaining significant attention over the past years. This stems from the fact that it constitutes the primal step toward developing some of the next-generation computer technologies that can make an impact in many domains, ranging from medical imaging and health assessment to marketing and education. No matter the target application, the need to deploy systems under demanding, real-world conditions that can generalize well across the population is urgent. Hence, careful consideration of numerous factors has to be taken prior to designing such a system. The work presented in this thesis focuses on tackling two important problems in automated analysis of facial expressions: (i) view-invariant facial expression analysis; (ii) modeling of the structural patterns in the face, in terms of well coordinated facial muscle movements. Driven by the necessity for efficient and accurate inference mechanisms we explore machine learning techniques based on the probabilistic framework of Gaussian processes (GPs). Our ultimate goal is to design powerful models that can efficiently handle imagery with spontaneously displayed facial expressions, and explain in detail the complex configurations behind the human face in real-world situations. To effectively decouple the head pose and expression in the presence of large out-of-plane head rotations we introduce a manifold learning approach based on multi-view learning strategies. Contrary to the majority of existing methods that typically treat the numerous poses as individual problems, in this model we first learn a discriminative manifold shared by multiple views of a facial expression. Subsequently, we perform facial expression classification in the expression manifold. Hence, the pose normalization problem is solved by aligning the facial expressions from different poses in a common latent space. We demonstrate that the recovered manifold can efficiently generalize to various poses and expressions even from a small amount of training data, while also being largely robust to corrupted image features due to illumination variations. State-of-the-art performance is achieved in the task of facial expression classification of basic emotions. The methods that we propose for learning the structure in the configuration of the muscle movements represent some of the first attempts in the field of analysis and intensity estimation of facial expressions. In these models, we extend our multi-view approach to exploit relationships not only in the input features but also in the multi-output labels. The structure of the outputs is imposed into the recovered manifold either from heuristically defined hard constraints, or in an auto-encoded manner, where the structure is learned automatically from the input data. The resulting models are proven to be robust to data with imbalanced expression categories, due to our proposed Bayesian learning of the target manifold. We also propose a novel regression approach based on product of GP experts where we take into account people's individual expressiveness in order to adapt the learned models on each subject. We demonstrate the superior performance of our proposed models on the task of facial expression recognition and intensity estimation.Open Acces

Evaluating professional development programmes aimed at promoting Classroom Action Research: Perspectives from teachers in Indonesia

This is an exploratory study which focuses on exploring teacher perceptions on the impact of professional development (PD) programmes aimed at promoting Classroom Action Research (CAR) that teachers followed in Jakarta, Indonesia. The Indonesian government by means of its education reform policy has been taking measures in promoting CAR to teachers as a way of improving its teacher quality, having one of its measures organising PD programmes about CAR for teachers and providing guidelines for PD programme providers. However, difficulties arise in the implementation as my research found that the quality of PD programmes available to teachers has been poor and without any evaluation process. While it is acknowledged that PD programmes need to be evaluated to ascertain the quality as well as to assess its impact on teachers as the participants, research on the impact of PD programmes that is specifically about CAR is fragmented and an integrated view on such evaluation is still missing. My literature review of this study reveals a gap in the existing frameworks for the evaluation of PD programmes specifically about CAR, resulting in the development of the conceptual framework. This framework suggests four research questions in this study, focusing on four levels of impact on PD programmes: teacher experience, teacher learning, teachers’ change in practice and influencing factors. This study tries to answer the research questions by applying a multiple-case study design in three different groups of teachers as participants of three PD programmes about CAR in Jakarta, Indonesia. The data were generated over a period of three months through an observation of all three programmes; a survey of one hundred teachers; interviews with the three programme providers, eleven heads of schools and sixteen teachers; and document analyses. This study demonstrates how the conceptual framework was deductively discussed through the application of case studies and inductively extended through the findings of teachers’ perceptions on the impact of the PD programmes about CAR. The result of the study generates an extended evaluation framework having all four levels of impact in the conceptual framework thoroughly elaborated and expanded in detail. First, there are eight main features of the programme which are highlighted to be effective in the discussion of teacher experience on the programme. For example, the features ‘sustained duration’ and ‘quality of mentor’ are revealed to be effective in helping teachers during the implementation of CAR. Second, the discussion highlights nine specific knowledge and skills in conducting CAR and three specific attitudes towards CAR as the impact of the PD programmes on teacher learning. For example, the findings reveal that several teachers were able to reflect on their previous CAR cycle and to trust the importance of CAR after following the programme. Third, there are three kinds of change in classroom practice, two kinds of change in personal level and two kinds of change in interpersonal level which are highlighted as the impact of the PD programmes on teachers’ change in practice. For instance, the findings reveal that a teacher had strong will in using CAR and, consequently, sought more effective teaching method and shared the result with the others. At last, the findings highlight the importance of teacher characteristics as well as organisational support in hindering or helping the impact accounted for as influencing factors. For instance, the characteristic of open-minded and progressive heads of schools provides support for teachers in conducting CAR. On the other hand, the characteristic of a new teacher lacking professional experience and motivation in following the programme limited his expectations of the programme, and, consequently, the implementation of CA

Multiscale Modelling of Neuronal Dynamics and Their Dysfunction in the Developing Brain

Over the last few decades, an increasing number of neurodevelopmental disorders has been associated with molecular causes – such as genetic mutations, or autoantibodies affecting synaptic transmission. Yet understanding the pathophysiology that leads from particular molecular disruptions at the synapse to patients’ signs and symptoms remains challenging, even today. The work presented in this thesis illustrates how computational models can help bridge the explanatory gap between disruptions at the molecular scale and brain dysfunction at the level of integrated circuits. I utilise computational models at different scales of neuronal function, ranging from the neuronal membrane, to integrated cortical microcircuits and whole-brain sensory processing networks. These computational models are informed with, and further constrained by both empirical data derived from a number of model systems of neurodevelopmental disorders, and clinical patient data. The worked examples in this thesis include the biophysical characterisation of an epilepsy-causing mutation in the voltage-gated sodium channel gene SCN1A, calcium imaging in a larval zebrafish model of epileptic seizures in the immature brain, electrophysiological recordings from patients with NMDA receptor antibody encephalitis as well as from a mouse model of the disorder, and pharmacologically induced NMDA receptor blockade in young adults that captures features of acute psychosis and schizophrenia. The combination of this diverse range of empirical data and different computational models offers a mechanistic, multi-scale account of how specific phenotypic features in neurodevelopmental disorders emerge. This provides novel insights both in regard to the specific conditions included here, but also concerning the link between molecular determinants and their neurodevelopmental phenotypes more broadly