The Verbal and Non Verbal Signals of Depression -- Combining Acoustics, Text and Visuals for Estimating Depression Level

Depression is a serious medical condition that is suffered by a large number of people around the world. It significantly affects the way one feels, causing a persistent lowering of mood. In this paper, we propose a novel attention-based deep neural network which facilitates the fusion of various modalities. We use this network to regress the depression level. Acoustic, text and visual modalities have been used to train our proposed network. Various experiments have been carried out on the benchmark dataset, namely, Distress Analysis Interview Corpus - a Wizard of Oz (DAIC-WOZ). From the results, we empirically justify that the fusion of all three modalities helps in giving the most accurate estimation of depression level. Our proposed approach outperforms the state-of-the-art by 7.17% on root mean squared error (RMSE) and 8.08% on mean absolute error (MAE).Comment: 10 pages including references, 2 figure

Аналитический обзор систем автоматического определения депрессии по речи

В последние годы в медицинской и научно-технической среде возрос интерес к задаче автоматического определения наличия депрессивного состояния у людей. Депрессия является одним из самых распространенных психических заболеваний, непосредственно влияющих на жизнь человека. В данном обзоре представлены и проанализированы работы за последние два года на тему определения депрессивного состояния у людей. Приведены основные понятия, относящиеся к определению депрессии, описаны как одномодальные, так и многомодальные корпусы, содержащие записи информантов с установленным диагнозом депрессии, а также записи контрольных групп, людей без депрессии. Рассмотрены как теоретические исследования, так и работы, в которых описаны автоматические системы для определения депрессивного состояния — от одномодальных до многомодальных. Часть рассмотренных систем решает задачу регрессивной классификации, предсказывая степень тяжести депрессии (отсутствие, слабая, умеренная, тяжелая), а другая часть – задачу бинарной классификации, предсказывая наличие заболевания у человека или его отсутствие. Представлена оригинальная классификация методов вычисления информативных признаков по трем коммуникативным модальностям (аудио, видео и текстовая информация). Описаны современные методы, используемые для определения депрессии в каждой из модальностей и в совокупности. Наиболее популярными методами моделирования и распознавания депрессии в рассмотренных работах являются нейронные сети. В ходе аналитического обзора выявлено, что основными признаками депрессии считаются психомоторная заторможенность, которая влияет на все коммуникативные модальности, и сильная корреляция с аффективными величинами валентности, активации и доминации, при этом наблюдается обратная корреляция между депрессией и агрессией. Выявленные корреляции подтверждают взаимосвязь аффективных расстройств с эмоциональными состояниями человека. В множестве рассмотренных работ наблюдается тенденция объединения модальностей для улучшения качества определения депрессии

Аналитический обзор систем автоматического определения депрессии по речи

In recent years the interest in automatic depression detection has grown within medical and scientific-technical communities. Depression is one of the most widespread mental illnesses that affects human life. In this review we present and analyze the latest researches devoted to depression detection. Basic notions related to the definition of depression were specified, the review includes both unimodal and multimodal corpora containing records of informants diagnosed with depression and control groups of non-depressed people. Theoretical and practical researches which present automated systems for depression detection were reviewed. The last ones include unimodal as well as multimodal systems. A part of reviewed systems addresses the challenge of regressive classification predicting the degree of depression severity (non-depressed, mild, moderate and severe), and another part solves a problem of binary classification predicting the presence of depression (if a person is depressed or not). An original classification of methods for computing of informative features for three communicative modalities (audio, video, text information) is presented. New methods for depression detection in every modality and all modalities in total are defined. The most popular methods for depression detection in reviewed studies are neural networks. The survey has shown that the main features of depression are psychomotor retardation that affects all communicative modalities and strong correlation with affective values of valency, activation and domination, also there has been observed an inverse correlation between depression and aggression. Discovered correlations confirm interrelation of affective disorders and human emotional states. The trend observed in many reviewed papers is that combining modalities improves the results of depression detection systems.В последние годы в медицинской и научно-технической среде возрос интерес к задаче автоматического определения наличия депрессивного состояния у людей. Депрессия является одним из самых распространенных психических заболеваний, непосредственно влияющих на жизнь человека. В данном обзоре представлены и проанализированы работы за последние два года на тему определения депрессивного состояния у людей. Приведены основные понятия, относящиеся к определению депрессии, описаны как одномодальные, так и многомодальные корпусы, содержащие записи информантов с установленным диагнозом депрессии, а также записи контрольных групп,людей без депрессии.Рассмотрены как теоретические исследования, так и работы, в которых описаны автоматические системы для определения депрессивного состояния — от одномодальных до многомодальных. Часть рассмотренных систем решает задачу регрессивной классификации, предсказывая степень тяжести депрессии (отсутствие, слабая, умеренная, тяжелая), а другая часть – задачу бинарной классификации, предсказывая наличие заболевания у человека или его отсутствие. Представлена оригинальная классификация методов вычисления информативных признаков по трем коммуникативным модальностям (аудио, видео и текстовая информация). Описаны современные методы, используемыедля определения депрессии в каждой из модальностей и в совокупности. Наиболее популярными методами моделирования и распознавания депрессии в рассмотренных работах являются нейронные сети. В ходе аналитического обзора выявлено, что основными признаками депрессии считаются психомоторная заторможенность, которая влияет на все коммуникативные модальности, и сильная корреляция с аффективными величинами валентности, активации и доминации, при этом наблюдается обратная корреляция между депрессией и агрессией. Выявленные корреляции подтверждают взаимосвязь аффективных расстройств с эмоциональными состояниями человека. В множестве рассмотренных работ наблюдается тенденция объединения модальностей для улучшения качества определения депрессии

What does not happen: quantifying embodied engagement using NIMI and self-adaptors

Previous research into the quantification of embodied intellectual and emotional engagement using non-verbal movement parameters has not yielded consistent results across different studies. Our research introduces NIMI (Non-Instrumental Movement Inhibition) as an alternative parameter. We propose that the absence of certain types of possible movements can be a more holistic proxy for cognitive engagement with media (in seated persons) than searching for the presence of other movements. Rather than analyzing total movement as an indicator of engagement, our research team distinguishes between instrumental movements (i.e. physical movement serving a direct purpose in the given situation) and non-instrumental movements, and investigates them in the context of the narrative rhythm of the stimulus. We demonstrate that NIMI occurs by showing viewers’ movement levels entrained (i.e. synchronised) to the repeating narrative rhythm of a timed computer-presented quiz. Finally, we discuss the role of objective metrics of engagement in future context-aware analysis of human behaviour in audience research, interactive media and responsive system and interface design

When a few words are not enough: improving text classification through contextual information

Traditional text classification approaches may be ineffective when applied to texts with insufficient or limited number of words due to brevity of text and sparsity of feature space. The lack of contextual information can make texts ambiguous; hence, text classification approaches relying solely on words may not properly capture the critical features of a real-world problem. One of the popular approaches to overcoming this problem is to enrich texts with additional domain-specific features. Thus, this thesis shows how it can be done in two realworld problems in which text information alone is insufficient for classification. While one problem is depression detection based on the automatic analysis of clinical interviews, another problem is detecting fake online news. Depression profoundly affects how people behave, perceive, and interact. Language reveals our ideas, moods, feelings, beliefs, behaviours and personalities. However, because of inherent variations in the speech system, no single cue is sufficiently discriminative as a sign of depression on its own. This means that language alone may not be adequate for understanding a person’s mental characteristics and states. Therefore, adding contextual information can properly represent the critical features of texts. Speech includes both linguistic content (what people say) and acoustic aspects (how words are said), which provide important clues about the speaker’s emotional, physiological and mental characteristics. Therefore, we study the possibility of effectively detecting depression using unobtrusive and inexpensive technologies based on the automatic analysis of language (what you say) and speech (how you say it). For fake news detection, people seem to use their cognitive abilities to hide information, which induces behavioural change, thereby changing their writing style and word choices. Therefore, the spread of false claims has polluted the web. However, the claims are relatively short and include limited content. Thus, capturing only text features of the claims will not provide sufficient information to detect deceptive claims. Evidence articles can help support the factual claim by representing the central content of the claim more authentically. Therefore, we propose an automated credibility assessment approach based on linguistic analysis of the claim and its evidence articles

Behavior quantification as the missing link between fields: Tools for digital psychiatry and their role in the future of neurobiology

The great behavioral heterogeneity observed between individuals with the same psychiatric disorder and even within one individual over time complicates both clinical practice and biomedical research. However, modern technologies are an exciting opportunity to improve behavioral characterization. Existing psychiatry methods that are qualitative or unscalable, such as patient surveys or clinical interviews, can now be collected at a greater capacity and analyzed to produce new quantitative measures. Furthermore, recent capabilities for continuous collection of passive sensor streams, such as phone GPS or smartwatch accelerometer, open avenues of novel questioning that were previously entirely unrealistic. Their temporally dense nature enables a cohesive study of real-time neural and behavioral signals. To develop comprehensive neurobiological models of psychiatric disease, it will be critical to first develop strong methods for behavioral quantification. There is huge potential in what can theoretically be captured by current technologies, but this in itself presents a large computational challenge -- one that will necessitate new data processing tools, new machine learning techniques, and ultimately a shift in how interdisciplinary work is conducted. In my thesis, I detail research projects that take different perspectives on digital psychiatry, subsequently tying ideas together with a concluding discussion on the future of the field. I also provide software infrastructure where relevant, with extensive documentation. Major contributions include scientific arguments and proof of concept results for daily free-form audio journals as an underappreciated psychiatry research datatype, as well as novel stability theorems and pilot empirical success for a proposed multi-area recurrent neural network architecture.Comment: PhD thesis cop

Multimodal Data Analysis of Dyadic Interactions for an Automated Feedback System Supporting Parent Implementation of Pivotal Response Treatment

abstract: Parents fulfill a pivotal role in early childhood development of social and communication skills. In children with autism, the development of these skills can be delayed. Applied behavioral analysis (ABA) techniques have been created to aid in skill acquisition. Among these, pivotal response treatment (PRT) has been empirically shown to foster improvements. Research into PRT implementation has also shown that parents can be trained to be effective interventionists for their children. The current difficulty in PRT training is how to disseminate training to parents who need it, and how to support and motivate practitioners after training. Evaluation of the parents’ fidelity to implementation is often undertaken using video probes that depict the dyadic interaction occurring between the parent and the child during PRT sessions. These videos are time consuming for clinicians to process, and often result in only minimal feedback for the parents. Current trends in technology could be utilized to alleviate the manual cost of extracting data from the videos, affording greater opportunities for providing clinician created feedback as well as automated assessments. The naturalistic context of the video probes along with the dependence on ubiquitous recording devices creates a difficult scenario for classification tasks. The domain of the PRT video probes can be expected to have high levels of both aleatory and epistemic uncertainty. Addressing these challenges requires examination of the multimodal data along with implementation and evaluation of classification algorithms. This is explored through the use of a new dataset of PRT videos. The relationship between the parent and the clinician is important. The clinician can provide support and help build self-efficacy in addition to providing knowledge and modeling of treatment procedures. Facilitating this relationship along with automated feedback not only provides the opportunity to present expert feedback to the parent, but also allows the clinician to aid in personalizing the classification models. By utilizing a human-in-the-loop framework, clinicians can aid in addressing the uncertainty in the classification models by providing additional labeled samples. This will allow the system to improve classification and provides a person-centered approach to extracting multimodal data from PRT video probes.Dissertation/ThesisDoctoral Dissertation Computer Science 201

2012-2013 Course Catalog

2012-2013 Course Catalo