Multimodal research: Addressing the complexity of multimodal environments and the challenges for CALL

Multimodality, the study of the interaction of language with other semiotic resources such as images and sound resources, has significant implications for computer assisted language learning (CALL) with regards to understanding the impact of digital environments on language teaching and learning. In this paper, we explore recent manifestations of CALL in 3-D virtual worlds, illustrated by the example of Second Life. The multimodal analyses of a conventional face-to-face lesson and three language learning activities in Second Life highlight some of the affordances and challenges presented by 3-D virtual environments. The results suggest that while multimodal resources integrate naturally to facilitate language teaching and learning in an orderly, structured and goal-orientated manner in classroom lessons, the often uncoordinated use (or absence) of avatars' gaze, facial expression, body posture, gesture, as well as the unclear proxemics and use of space pose problems for effective communication in a 3-D virtual world. In addition, a technology-oriented register, alongside traditional instructional and regulative genres and registers, is introduced to help students cope with the demands of learning a language in a 3-D virtual environment. The study raises the issue of the relative effectiveness of 3-D virtual worlds for language teaching and learning. In doing so, a digital approach to multimodal research is proposed in order to address the complexity of multimodal learning environments and the various challenges for CALL

A Mixed Methods Empirical Examination of Changes in Emphasis and Style in the Extremist Magazines Dabiq and Rumiyah

The change in name of ISIS’s flagship English language magazine from Dabiq to Rumiyah prompted media speculation about its significance. This article uses a mixed methods approach that integrates a qualitative social semiotic discourse analysis approach with quantitative methods of information visualisation to examine empirically changes in emphasis and approach in both magazines over time to determine whether the changes are ones of style or substance. The paper argues that, while ISIS has changed its strategic focus over time in response to its changing fortunes, the organisation’s underlying world view, values and ultimate aims remain consistent and unchanged

Developing misinformation immunity:How to reason-check fallacious news in a human–computer interaction environment

To counter the fake news phenomenon, the scholarly community has attempted to debunk and prebunk disinformation. However, misinformation still constitutes a major challenge due to the variety of misleading techniques and their continuous updates which call for the exercise of critical thinking to build resilience. In this study we present two open access chatbots, the Fake News Immunity Chatbot and the Vaccinating News Chatbot, which combine Fallacy Theory and Human–Computer Interaction to inoculate citizens and communication gatekeepers against misinformation. These chatbots differ from existing tools both in function and form. First, they target misinformation and enhance the identification of fallacious arguments; and second, they are multiagent and leverage discourse theories of persuasion in their conversational design. After having described both their backend and their frontend design, we report on the evaluation of the user interface and impact on users’ critical thinking skills through a questionnaire, a crowdsourced survey, and a pilot qualitative experiment. The results shed light on the best practices to design user-friendly active inoculation tools and reveal that the two chatbots are perceived as increasing critical thinking skills in the current misinformation ecosystem

Interpreting text and image relations in violent extremist discourse: A mixed methods approach for big data analytics

This article presents a mixed methods approach for analysing text and image relations in violent extremist discourse. The approach involves integrating multimodal discourse analysis with data mining and information visualisation, resulting in theoretically informed empirical techniques for automated analysis of text and image relations in large datasets. The approach is illustrated by a study which aims to analyse how violent extremist groups use language and images to legitimise their views, incite violence, and influence recruits in online propaganda materials, and how the images from these materials are re-used in different media platforms in ways that support and resist violent extremism. The approach developed in this article contributes to what promises to be one of the key areas of research in the coming decades: namely the interdisciplinary study of big (digital) datasets of human discourse, and the implications of this for terrorism analysis and research