Meeting decision detection: multimodal information fusion for multi-party dialogue understanding

Modern advances in multimedia and storage technologies have led to huge archives of human conversations in widely ranging areas. These archives offer a wealth of information in the organization contexts. However, retrieving and managing information in these archives is a time-consuming and labor-intensive task. Previous research applied keyword and computer vision-based methods to do this. However, spontaneous conversations, complex in the use of multimodal cues and intricate in the interactions between multiple speakers, have posed new challenges to these methods. We need new techniques that can leverage the information hidden in multiple communication modalities – including not just “what” the speakers say but also “how” they express themselves and interact with others. In responding to this need, the thesis inquires into the multimodal nature of meeting dialogues and computational means to retrieve and manage the recorded meeting information. In particular, this thesis develops the Meeting Decision Detector (MDD) to detect and track decisions, one of the most important outcomes of the meetings. The MDD involves not only the generation of extractive summaries pertaining to the decisions (“decision detection”), but also the organization of a continuous stream of meeting speech into locally coherent segments (“discourse segmentation”). This inquiry starts with a corpus analysis which constitutes a comprehensive empirical study of the decision-indicative and segment-signalling cues in the meeting corpora. These cues are uncovered from a variety of communication modalities, including the words spoken, gesture and head movements, pitch and energy level, rate of speech, pauses, and use of subjective terms. While some of the cues match the previous findings of speech segmentation, some others have not been studied before. The analysis also provides empirical grounding for computing features and integrating them into a computational model. To handle the high-dimensional multimodal feature space in the meeting domain, this thesis compares empirically feature discriminability and feature pattern finding criteria. As the different knowledge sources are expected to capture different types of features, the thesis also experiments with methods that can harness synergy between the multiple knowledge sources. The problem formalization and the modeling algorithm so far correspond to an optimal setting: an off-line, post-meeting analysis scenario. However, ultimately the MDD is expected to be operated online – right after a meeting, or when a meeting is still in progress. Thus this thesis also explores techniques that help relax the optimal setting, especially those using only features that can be generated with a higher degree of automation. Empirically motivated experiments are designed to handle the corresponding performance degradation. Finally, with the users in mind, this thesis evaluates the use of query-focused summaries in a decision debriefing task, which is common in the organization context. The decision-focused extracts (which represent compressions of 1%) is compared against the general-purpose extractive summaries (which represent compressions of 10-40%). To examine the effect of model automation on the debriefing task, this evaluation experiments with three versions of decision-focused extracts, each relaxing one manual annotation constraint. Task performance is measured in actual task effectiveness, usergenerated report quality, and user-perceived success. The users’ clicking behaviors are also recorded and analyzed to understand how the users leverage the different versions of extractive summaries to produce abstractive summaries. The analysis framework and computational means developed in this work is expected to be useful for the creation of other dialogue understanding applications, especially those that require to uncover the implicit semantics of meeting dialogues

A privacy-aware and secure system for human memory augmentation

The ubiquity of digital sensors embedded in today's mobile and wearable devices (e.g., smartphones, wearable cameras, wristbands) has made technology more intertwined with our life. Among many other things, this allows us to seamlessly log our daily experiences in increasing numbers and quality, a process known as ``lifelogging''. This practice produces a great amount of pictures and videos that can potentially improve human memory. Consider how a single photograph can bring back distant childhood memories, or how a song can help us reminisce about our last vacation. Such a vision of a ``memory augmentation system'' can offer considerable benefits, but it also raises new security and privacy challenges. Maybe obviously, a system that captures everywhere we go, and everything we say, see, and do, is greatly increasing the danger to our privacy. Any data breach of such a memory repository, whether accidental or malicious, could negatively impact both our professional and private reputation. In addition, the threat of memory manipulation might be the most worrisome aspect of a memory augmentation system: if an attacker is able to remove, add, or change our captured information, the resulting data may implant memories in our heads that never took place, or, in turn, accelerate the loss of other memories. Starting from such key challenges, this thesis investigates how to design secure memory augmentation systems. In the course of this research, we develop tools and prototypes that can be applied by researchers and system engineers to develop pervasive applications that help users capture and later recall episodic memories in a secure fashion. We build trusted sensors and protocols to securely capture and store experience data, and secure software for the secure and privacy-aware exchange of experience data with others. We explore the suitability of various access control models to put users in control of the plethora of data that the system captures on their behalf. We also explore the possibility of using in situ physical gestures to control different aspects regarding the capturing and sharing of experience data. Ultimately, this thesis contributes to the design and development of secure systems for memory augmentation

Experiences of Colorado Parents as They Recognized Their Child\u27s Mental Illness

Mental illness is not only the leading cause of disability among adults, but there is also an emerging public health crisis in childhood mental illness. A majority of parents do not recognize symptoms of psychological disorder in their children, and current policies and programs for mental health service delivery are not sufficiently responsive to the early help-seeking dynamics of families. Using a concurrent mixed methods design, this study explored how parents in the Pikes Peak region of Colorado learned to recognize their child\u27s mental illness. Phenomenological interviews, augmented by poetic inquiry and quantitative measurements, were used to discover factors that inhibited or enhanced five mothers\u27 recognitions. These factors were then evaluated using a frequency distribution analysis and a rank-order correlation. The phenomenon of recognition was, for these mothers, a process of waiting to hear that \u27normal\u27 had stopped, wherein they miscategorized symptoms as normal behaviors in a passing developmental phase. Prior experience with mental illness appeared to significantly decrease both the length of time and the level of distress necessary for recognition. Ultimately, recognition did not occur until someone in their social network validated their concerns and provided explicit confirmation, which galvanized them to seek treatment. Governance network collaborations can facilitate positive social change by standardizing guidance on how to differentiate symptoms of a disorder from normal childhood development. Public policies and programs such as universal mental health screening, mental health literacy, and more supportive and responsive school policies can foster dialogue for parental recognition in Colorado and throughout the country

The Perception of Emotion from Acoustic Cues in Natural Speech

Knowledge of human perception of emotional speech is imperative for the development of emotion in speech recognition systems and emotional speech synthesis. Owing to the fact that there is a growing trend towards research on spontaneous, real-life data, the aim of the present thesis is to examine human perception of emotion in naturalistic speech. Although there are many available emotional speech corpora, most contain simulated expressions. Therefore, there remains a compelling need to obtain naturalistic speech corpora that are appropriate and freely available for research. In that regard, our initial aim was to acquire suitable naturalistic material and examine its emotional content based on listener perceptions. A web-based listening tool was developed to accumulate ratings based on large-scale listening groups. The emotional content present in the speech material was demonstrated by performing perception tests on conveyed levels of Activation and Evaluation. As a result, labels were determined that signified the emotional content, and thus contribute to the construction of a naturalistic emotional speech corpus. In line with the literature, the ratings obtained from the perception tests suggested that Evaluation (or hedonic valence) is not identified as reliably as Activation is. Emotional valence can be conveyed through both semantic and prosodic information, for which the meaning of one may serve to facilitate, modify, or conflict with the meaning of the other—particularly with naturalistic speech. The subsequent experiments aimed to investigate this concept by comparing ratings from perception tests of non-verbal speech with verbal speech. The method used to render non-verbal speech was low-pass filtering, and for this, suitable filtering conditions were determined by carrying out preliminary perception tests. The results suggested that nonverbal naturalistic speech provides sufficiently discernible levels of Activation and Evaluation. It appears that the perception of Activation and Evaluation is affected by low-pass filtering, but that the effect is relatively small. Moreover, the results suggest that there is a similar trend in agreement levels between verbal and non-verbal speech. To date it still remains difficult to determine unique acoustical patterns for hedonic valence of emotion, which may be due to inadequate labels or the incorrect selection of acoustic parameters. This study has implications for the labelling of emotional speech data and the determination of salient acoustic correlates of emotion

Poetry therapy in practice: identifying the mechanisms of poetry therapy and other percieved effects on participants

Background - Poetry therapy is a promising but heterogeneous and under-evidenced form of creative art therapy. Theories of change have been proffered but are modelspecific and poorly evidenced in the empirical literature. Aims – To systematically retrieve, review, and synthesise empirical literature exploring mechanisms of pan-theoretical poetry therapy, providing a united understanding of how poetry therapy operates to guide future research and practice. Methods – A systematic search of six databases yielded 161 papers. Fourteen met the inclusion criteria, spanning individual and group approaches. Mechanisms and effects were extracted and synthesised into a governing framework and logic model. Stakeholder consultation was used to validate results. Results – 25 primary mechanisms and 54 associated effects were identified. These were synthesised into logic model characterised by five core processes: Engaging, Feeling, Exploring, Connecting, and Transferring (“EFECT”). These processes were associated with multifarious benefits, impacting cognitive, emotional and behavioural domains. Conclusions – The mechanisms and effects of poetry therapy can be understood via the EFECT model. This should now be tested empirically. The model can then be used to guide a united, rigorous research programme, helping to bring poetry therapy into evidence-based policy and practice

Service User Involvement in Recovery-oriented Care Planning: A Realist Synthesis

Background: Service user involvement (SUI) is one of the main guiding principles in mental health care and is widely accepted as integral to recovery-oriented practice. Yet whilst there is a wealth of literature that advocates its benefits, service users are marginalised from recovery-oriented care-planning processes. A lack of effective implementation support; little guidance about how to facilitate SUI; and a number of barriers to genuine SUI in recovery-oriented care planning in acute inpatient units all indicate the complex nature of this issue. It remains an area that is currently under-researched and inadequately understood and warrants more sophisticated theorisation and explanation in order to support practice improvement. Aim: This study investigated which changes to practice work best, in what circumstances, and to what extent, to embed an active role for service users' involvement in recovery-oriented care planning during acute inpatient care. Design: A realist synthesis, combined with qualitative methods, was conducted to theoretically explore the causal mechanisms that underlie SUI in care planning and how contextual factors influence the link between these causal mechanisms and outcomes. The study was conducted in three stages: theory gleaning; theory refinement and theory consolidation. Initial programme theories related to SUI in recovery-oriented care planning were developed in the theory gleaning stage. These theories were refined iteratively, using evidence from a realist review (secondary data) and interview data (primary data). With stakeholder involvement, refined programme theories were finely tuned using 'if-then' statements in the consolidation stage. Setting: This study was conducted in NHS mental health hospitals and community mental health centres in the south-east of England, which provide secondary mental health care for service users with mental health problems. Results: Five programme theories relating to the acute care pathway were identified following the realist synthesis: 1) 'Provider-controlled care transition' (admission to acute inpatient units), referring to the limitations to service users' active involvement at this stage; 2) 'Care plan as a recovery tool?' - addressing the infrastructural and organisational limitations to active SUI in recovery-oriented care-plan formulation; 3) 'Ward rounds as a non-inclusive arena for shared decision making', highlighting their unfulfilled potential for shared decision making about treatment; 4) 'Peer support worker intervention as a key factor in service users' recovery', concerning their positive impact; and 5) 'Provider-controlled care transition from acute inpatient units' (discharge practice), highlighting limitations of current practice in preparing service users for transition into the community. Conclusions: The study identified practices required to embed an active role for service users to be involved in recovery-oriented care planning, namely multi-contextual interventions at various levels (macro, meso and micro) of the mental health system. The study uncovered blockages and contentions that restrain SUI in recovery-oriented care planning throughout the mental health system, impacting upon desirable outcomes. Implications for practice: Five key practice improvement areas were identified: 1) The focus of care and access to acute inpatients units should be on a needs-led, rather than resource-led or demand-driven, basis. 2) The use of multidisciplinary meetings as a forum for care-plan formulation can create a cohesive approach and provide equal opportunities to contribute to the care plan, facilitating a shared ownership. 3) Limiting the number of professionals, particularly those who have not been involved with the service users' care, may enhance an environment conducive for shared decision making. Service users should feel their views are validated by professionals. Professionals should focus on preparing service users for the ward-round process and meetings. Opportunities and access for service users to build therapeutic relationships with the treating doctors is a vital component. 4) Adding peer support workers as part of a multidisciplinary team has the potential to promote SUI in care planning. Their presence in ward rounds and care-planning meetings might create a more user-friendly atmosphere for service users. 5) Practice in acute inpatient units should have an increasing focus on preparing service users for transition into the community, and constraints on resources should not dictate or anticipate decisions on discharging service users, instead, wherever possible, it should be collaborative in nature

Mediators and moderators of self-injurious behaviours and borderline personality disorder.

Self-harm is a diagnostic criteria of Borderline personality disorder (BPD), and is a significant mental health problem in its own right. Three studies (A/s=46 - 340) are reported that investigated the role of executive functions, attachment orientation, and social cognition as potential mediators and moderators of the relationship between BPD features and self-harm in a non-clinical sample. Findings from Study 1 indicated that personality and attentional control factors interact to determine self-harm likelihood whereby high attentional focusing and shifting abilities are protective when BPD features are low but high focusing may be a possible maintaining factor when BPD features are high. Findings from Study 2 indicated that the individuals who have previously self-harmed exhibited EF deficits compared to controls, particularly deficits primarily related to problem solving, difficulty disengaging attention, and cognitive switching. However, these differences did not influence the relationship between BPD features and self-harm. Instead, self-esteem mediated the relationship between BPD and intent to self-harm after exposure to the vignettes. In Study 3, findings revealed that global self-esteem and attachment anxiety completely mediated the relationship between BPD features and intent to self-harm after exposure to the vignettes. The findings of this study have a number of important implications for definitions, theoretical conceptualisations, and therapeutic interventions