Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings

In this paper we present a novel interactive multimodal learning system, which facilitates search and exploration in large networks of social multimedia users. It allows the analyst to identify and select users of interest, and to find similar users in an interactive learning setting. Our approach is based on novel multimodal representations of users, words and concepts, which we simultaneously learn by deploying a general-purpose neural embedding model. We show these representations to be useful not only for categorizing users, but also for automatically generating user and community profiles. Inspired by traditional summarization approaches, we create the profiles by selecting diverse and representative content from all available modalities, i.e. the text, image and user modality. The usefulness of the approach is evaluated using artificial actors, which simulate user behavior in a relevance feedback scenario. Multiple experiments were conducted in order to evaluate the quality of our multimodal representations, to compare different embedding strategies, and to determine the importance of different modalities. We demonstrate the capabilities of the proposed approach on two different multimedia collections originating from the violent online extremism forum Stormfront and the microblogging platform Twitter, which are particularly interesting due to the high semantic level of the discussions they feature

Multimodal music information processing and retrieval: survey and future challenges

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years

Multimodal Grounding for Language Processing

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.Comment: The paper has been published in the Proceedings of the 27 Conference of Computational Linguistics. Please refer to this version for citations: https://www.aclweb.org/anthology/papers/C/C18/C18-1197

Media literacy at all levels: making the humanities more inclusive

The decline of the humanities, combined with the arrival of students focused on science, technology, engineering, and mathematics (STEM), represent an opportunity for the development of innovative approaches to teaching languages and literatures. Expanding the instructional focus from traditional humanities students, who are naturally more text-focused, to address the needs of more application-oriented STEM learners ensures that language instructors prepare all students to become analytical and critical consumers and producers of digital media. Training students to question motives both in their own and authentic media messages and to justify their own interpretations results in more sophisticated second language (L2) communication. Even where institutional structures impede comprehensive curriculum reform, individual instructors can integrate media literacy training into their own classes. Tis article demonstrates ways of reaching and retaining larger numbers of students at all levels—if necessary, one course at a time.Published versio

Crisis translation: considering language needs in multilingual disaster settings

Purpose: The purpose of this conceptual paper is to highlight the role that language translation can play in disaster prevention and management and to make the case for increased attention to language translation in crisis communication. Approach: The article draws on literature relating to disaster management to suggest that translation is a perennial issue in crisis communication. Findings: Although communication with multicultural and multilinguistic communities is seen as being in urgent need of attention, we find that the role of translation in enabling this is underestimated, if not unrecognised. Value: This article raises awareness of the need for urgent attention to be given by scholars and practitioners to the role of translation in crisis communication

Contours of Inclusion: Inclusive Arts Teaching and Learning

The purpose of this publication is to share models and case examples of the process of inclusive arts curriculum design and evaluation. The first section explains the conceptual and curriculum frameworks that were used in the analysis and generation of the featured case studies (i.e. Understanding by Design, Differentiated Instruction, and Universal Design for Learning). Data for the cases studies was collected from three urban sites (i.e. Los Angeles, San Francisco, and Boston) and included participant observations, student and teacher interviews, curriculum documentation, digital documentation of student learning, and transcripts from discussion forum and teleconference discussions from a professional learning community.The initial case studies by Glass and Barnum use the curricular frameworks to analyze and understand what inclusive practices look like in two case studies of arts-in-education programs that included students with disabilities. The second set of precedent case studies by Kronenberg and Blair, and Jenkins and Agois Hurel uses the frameworks to explain their process of including students by providing flexible arts learning options to support student learning of content standards. Both sets of case studies illuminate curricular design decisions and instructional strategies that supported the active engagement and learning of students with disabilities in educational settings shared with their peers. The second set of cases also illustrate the reflective process of using frameworks like Universal Design for Learning (UDL) to guide curricular design, responsive instructional differentiation, and the use of the arts as a rich, meaningful, and engaging option to support learning. Appended are curriculum design and evaluation tools. (Individual chapters contain references.

Report on the Information Retrieval Festival (IRFest2017)

The Information Retrieval Festival took place in April 2017 in Glasgow. The focus of the workshop was to bring together IR researchers from the various Scottish universities and beyond in order to facilitate more awareness, increased interaction and reflection on the status of the field and its future. The program included an industry session, research talks, demos and posters as well as two keynotes. The first keynote was delivered by Prof. Jaana Kekalenien, who provided a historical, critical reflection of realism in Interactive Information Retrieval Experimentation, while the second keynote was delivered by Prof. Maarten de Rijke, who argued for more Artificial Intelligence usage in IR solutions and deployments. The workshop was followed by a "Tour de Scotland" where delegates were taken from Glasgow to Aberdeen for the European Conference in Information Retrieval (ECIR 2017