CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Emotion-aware voice interfaces based on speech signal processing

Voice interfaces (VIs) will become increasingly widespread in current daily lives as AI techniques progress. VIs can be incorporated into smart devices like smartphones, as well as integrated into autos, home automation systems, computer operating systems, and home appliances, among other things. Current speech interfaces, however, are unaware of users’ emotional states and hence cannot support real communication. To overcome these limitations, it is necessary to implement emotional awareness in future VIs. This thesis focuses on how speech signal processing (SSP) and speech emotion recognition (SER) can enable VIs to gain emotional awareness. Following an explanation of what emotion is and how neural networks are implemented, this thesis presents the results of several user studies and surveys. Emotions are complicated, and they are typically characterized using category and dimensional models. They can be expressed verbally or nonverbally. Although existing voice interfaces are unaware of users’ emotional states and cannot support natural conversations, it is possible to perceive users’ emotions by speech based on SSP in future VIs. One section of this thesis, based on SSP, investigates mental restorative effects on humans and their measures from speech signals. SSP is less intrusive and more accessible than traditional measures such as attention scales or response tests, and it can provide a reliable assessment for attention and mental restoration. SSP can be implemented into future VIs and utilized in future HCI user research. The thesis then moves on to present a novel attention neural network based on sparse correlation features. The detection accuracy of emotions in the continuous speech was demonstrated in a user study utilizing recordings from a real classroom. In this section, a promising result will be shown. In SER research, it is unknown if existing emotion detection methods detect acted emotions or the genuine emotion of the speaker. Another section of this thesis is concerned with humans’ ability to act on their emotions. In a user study, participants were instructed to imitate five fundamental emotions. The results revealed that they struggled with this task; nevertheless, certain emotions were easier to replicate than others. A further study concern is how VIs should respond to users’ emotions if SER techniques are implemented in VIs and can recognize users’ emotions. The thesis includes research on ways for dealing with the emotions of users. In a user study, users were instructed to make sad, angry, and terrified VI avatars happy and were asked if they would like to be treated the same way if the situation were reversed. According to the results, the majority of participants tended to respond to these unpleasant emotions with neutral emotion, but there is a difference among genders in emotion selection. For a human-centered design approach, it is important to understand what the users’ preferences for future VIs are. In three distinct cultures, a questionnaire-based survey on users’ attitudes and preferences for emotion-aware VIs was conducted. It was discovered that there are almost no gender differences. Cluster analysis found that there are three fundamental user types that exist in all cultures: Enthusiasts, Pragmatists, and Sceptics. As a result, future VI development should consider diverse sorts of consumers. In conclusion, future VIs systems should be designed for various sorts of users as well as be able to detect the users’ disguised or actual emotions using SER and SSP technologies. Furthermore, many other applications, such as restorative effects assessments, can be included in the VIs system

CHORUS Deliverable 2.2: Second report - identification of multi-disciplinary key issues for gap analysis toward EU multimedia search engines roadmap

After addressing the state-of-the-art during the first year of Chorus and establishing the existing landscape in multimedia search engines, we have identified and analyzed gaps within European research effort during our second year. In this period we focused on three directions, notably technological issues, user-centred issues and use-cases and socio- economic and legal aspects. These were assessed by two central studies: firstly, a concerted vision of functional breakdown of generic multimedia search engine, and secondly, a representative use-cases descriptions with the related discussion on requirement for technological challenges. Both studies have been carried out in cooperation and consultation with the community at large through EC concertation meetings (multimedia search engines cluster), several meetings with our Think-Tank, presentations in international conferences, and surveys addressed to EU projects coordinators as well as National initiatives coordinators. Based on the obtained feedback we identified two types of gaps, namely core technological gaps that involve research challenges, and “enablers”, which are not necessarily technical research challenges, but have impact on innovation progress. New socio-economic trends are presented as well as emerging legal challenges

비유언어와 문맥 표지를 이용한 반어법 자동 분류 연구

학위논문 (석사)-- 서울대학교 대학원 : 언어학과, 2014. 8. 신효필.본 논문은 고빈도 비유언어(figurative language)를 이용한 반어법 자동 인식 방법을 제안한다. 반어법과 비유언어들(직유법, 은유법, 의인법, 과장법)을 인식하는 문제는 컴퓨터 언어학에서 매우 중요한 분야이다. 이런 비유 언어들은 표면적인 의미와 다른 의미를 내포하기 때문에 그 문장의 의미를 파악하는데 필요한 연구이다. 과장법이나 과소 법 같은 비유언어와 달리 특별히 반어법은 그 표현적 의미와 정 반대 또는 부합하지 않는 의미를 내포하기 때문에 더욱 문제가 된다. 구어에서 반어법이 사용될 때는 운율이라는 요소가 인식에 중요한 역할을 하는 반면, 문어에서 반어법은 운율 정보가 없기 때문에 더 인식이 어렵다. 또한, 반어법은 대부분의 경우 표면적으로 나타나는 명확한 단서를 포함하지 않고, 단지 준언어적, 문맥적 화용적인 단서만을 갖기 때문에 인식에 더 어려움이 크다. 반어법의 단서가 되는 예로는 청자에게 문자 그대로 이해되기를 바라지 않음을 암시하는 과장법, 과소법, 수사적 질문법, 부가 의문문 같은 것들이 존재한다. 본고는 동시에 나타나는 비유언어들을 각각 인식하여 그 결과를 반어법 검출기에 제공하는 방식의 분할-정복법을 소개한다. 짧은 길이의 트위터와 상대적으로 긴 아마존 상품평에 대해 실행한 실험은 이러한 비유언어들을 개별적으로 인식하여 반어법의 자동 인식에 사용하는 것이 비유언어들을 한번에 인식하는 방법 보다 반어법 인식에 효과적이라는 사실을 밝혔다. 또한, 지금까지 개별적으로 제한된 문맥만을 고려한 과장법, 과소법 연구와 달리 본 연구는 반어법 인식에 사용되는 기존의 연구 방법을 과장법과 과소법 인식에도 적용할 수 있는 가능성을 제시하였다는 의의가 있다.This thesis proposes a linguistic-based irony detection method which uses these frequently co-occurring figurative languages to identify areas where irony is likely to occur. The detection and proper interpretation of irony and other figurative languages represents an important area of research for Computational Linguistics. Since figurative languages typically convey meanings which differ from their literal interpretations, interpreting such utterances at face value is likely to give incorrect results. Irony in particular represents a special challenge as, unlike some figurative languages like hyperbole or understatement which express sentiments which are more-or-less in line with their literal interpretation, differing only in intensity, ironic utterances convey intended meanings incongruent with – or even the exact opposite of – their literal interpretation. Compounding the need for effective irony detection is ironys near ubiquitous use in online writings and computer-mediated communications, both of which are commonly used in Computational Linguistics experiments. While irony in spoken contexts tends to be denoted using prosody, irony in written contexts is much harder to detect. One of the major difficulties is that irony typically does not present with any explicit clues such as punctuation marks or verbal inflections. Instead, irony tends to be denoted using paralinguistic, contextual, or pragmatic cues. Among these are the co-occurrence of figurative languages such as hyperbole, understatement, rhetorical questions, tag questions, or other ironic utterances which alert the listener that the speaker does not expect to be interpreted literally. This thesis introduces a divide-and-conquer approach to irony detection where co-occurring figurative languages are identified independently and then fed into an overall irony detector. Experiments on both short-form Twitter tweets and longer-form Amazon product reviews show not only that co-textual figurative languages are useful in the automatic classification of irony but that identifying these co-occurring figurative languages separately yields better overall irony detection by resolving conflicts between conflicting features, such as those for hyperbole and understatement. This thesis also introduces detection methods for hyperbole and understatement in general contexts by adapting existing approaches to irony detection. Before this point hyperbole detection was focused only on specialized contexts while understatement detection had been largely ignored. Experiments show that these proposed automated hyperbole and understatement detection methods outperformed methods which rely on fixed vocabularies.1 Introduction 1 1.1 What is Irony? 2 1.2 Irony and Co-textual Markers 4 1.2.1 Hyperbole 6 1.2.2 Understatement 7 1.2.3 Rhetorical Questions 8 1.2.4 Tag Questions 9 2 Previous Works 10 2.1 Irony Detection 10 2.2 Detection of Co-textual Markers 12 3 Data Collection 15 3.1 Twitter Data 15 3.1.1 Twitter Irony Corpus 18 3.1.2 Twitter Hyperbole Corpus 18 3.1.3 Twitter Understatement Corpus 18 3.2 Amazon Data 19 4 Experimental Set-up 21 4.1 Hyperbole Detection 22 4.2 Understatement Detection 23 4.3 Rhetorical Question Detection 25 4.4 Tag Question Detection 27 4.5 Irony Detection 28 4.5.1 Twitter Data 30 4.5.2 Amazon Product Review Data 30 5 Results and Discussion 33 5.1 Hyperbole 33 5.2 Understatement 39 5.3 Irony 44 5.3.1 Twitter 44 5.3.2 Amazon Product Reviews 50 6 Conclusions and Future Work 57 7 References 60 Appendix 1 Hyperbole Word List 66 Appendix 2 Hedge Word List 69Maste