A Study on Effects of Multimodal Science Text Design on Meaning-Making of Science Content: Science Teachers’ Perspectives

Numerous studies demonstrated that the meaning-making of scientific knowledge is affected by the design of multimodal science texts. Various modes are co-operated together in certain inter-semiotic mechanisms to produce meaning in multimodal texts. Based on this perspective, this research seeks to investigate the effect of mode level in science texts and compositional arrangement on the meaning-making of science concepts and processes. In this context, four science texts with the same content (transformation of energy) at different mode densities and two science texts with the same content (covalent bonding) one of which is arranged in accordance with variation theory of learning are designed. By using the case study method, this research explored six experienced science teachers’ views about the effects of mode level and multimodal text composition on meaning-making. The data were collected with semi-structured interviews. The thematic analysis was employed for data analysis. The findings demonstrated that mode density may affect meaning-making and so learning since different modes have affordance to represent different meaning and meaning relationship types. Besides, multimodal text composition may foreground the critical aspects of content, and help to design a coherent multimodal science text

Online leadership discourse in higher education: a digital multimodal discourse perspective

As leadership discourses in higher education are increasingly being mediated online, texts previously reserved for staff are now being made available in the public domain. As such, these texts become accessible for study, critique and evaluation. Additionally, discourses previously confined to the written domain are now increasingly multimodal. Thus, an approach is required that is capable of relating detailed, complex multimodal discourse analyses to broader sociocultural perspectives to account for the complex meaning-making practices that operate in online leadership discourses. For this purpose, a digital multimodal discourse approach is proposed and illustrated via a small-scale case study of the online leadership discourse of an Australian university. The analysis of two short video texts demonstrates how a digital multimodal discourse perspective facilitates the identification of key multimodal systems used for meaning-making in online communication, how meaning arises through combinations of semiotic choices (not individual choices), and how the results of multimodal discourse analysis using digital technology can reveal larger sociocultural patterns – in this case, divergent leadership styles and approaches as reflected in online discourse, at a time of immense change within the higher education sector

Trajectories of learning across museums and classrooms

This paper explores the use of social and mobile technologies on school field trips as means of enhancing the visitor experience. It employs the notion of a ‘trajectory’ (Ludvigsen et al. 2010; Pierroux et al., 2010; Littleton & Kerawalla, 2012) as appropriate means of connecting learners temporal experiences with informal and formal learning contexts. The focus of the analysis is on a group’s trajectory with an aim to examine the meanings made and represented in multimodal ‘ensembles’ and further, to explore whether artefacts and tools encountered or used inform students’ ensembles and assist them in making connections across the settings. This paper aims to contribute to contemporary discourse on technology-enhanced museum learning by exploring aspects of the visitor experience, such as meaning making across and between contexts

Detecting Sarcasm in Multimodal Social Platforms

Sarcasm is a peculiar form of sentiment expression, where the surface sentiment differs from the implied sentiment. The detection of sarcasm in social media platforms has been applied in the past mainly to textual utterances where lexical indicators (such as interjections and intensifiers), linguistic markers, and contextual information (such as user profiles, or past conversations) were used to detect the sarcastic tone. However, modern social media platforms allow to create multimodal messages where audiovisual content is integrated with the text, making the analysis of a mode in isolation partial. In our work, we first study the relationship between the textual and visual aspects in multimodal posts from three major social media platforms, i.e., Instagram, Tumblr and Twitter, and we run a crowdsourcing task to quantify the extent to which images are perceived as necessary by human annotators. Moreover, we propose two different computational frameworks to detect sarcasm that integrate the textual and visual modalities. The first approach exploits visual semantics trained on an external dataset, and concatenates the semantics features with state-of-the-art textual features. The second method adapts a visual neural network initialized with parameters trained on ImageNet to multimodal sarcastic posts. Results show the positive effect of combining modalities for the detection of sarcasm across platforms and methods.Comment: 10 pages, 3 figures, final version published in the Proceedings of ACM Multimedia 201

The Process of Creation: A Novel Methodology for Analyzing Multimodal Data

In the 21st century, meaning making is a multimodal act; we communicate what we know and how we know it using much more than printed text on a blank page. As a result, qualitative researchers need new methodologies, methods, and tools for working with the complex artifacts that our research subjects produce. In this article we describe the co-development of an analytic methodology and a tool for working with youth produced films as multimodal artifacts of youth engagement with identity. Specifically, we describe how to employ this multimodal framework in data analysis, with an emphasis on how different modes interact with one another, and how new meanings are made possible through multimodal interactions

A multimodal literary analysis of a television commercial

Literature and advertisements are analogous in the way they mimic the surrounding culture, art and practices and in their ability to elicit various emotional responses in people. Over the years, literature has had a great influence in advertisements and much of advertisement content has used poetic verses, prose forms and dramatic elements. Television commercials largely draw inspiration from drama in the construction of the advertisers’ messages. As a multimodal text, a television commercial (TVC) utilises multiple modes of meaning to employ literary elements in its content. Based on the view of TVCs as literary texts, this paper examines the literary elements of setting and character in a TVC through multimodal analysis. The multimodal elements that construct setting and character in a corporate TVC are identified and analysed. In line with multimodal analysis, the meanings of the modes are discussed by taking into account the socio-cultural context in which the TVC is produced and viewed. In doing so, the significant role of setting and character in meaning making of a TVC is revealed. This paper demonstrates the application of the multimodal approach to analysis of literary elements in a TVC and subsequently contributes to developing and promoting applications of multimodal analytical approaches in literary studies

The suitcase project: a journey in multimodal reading of graphic novels with emergent bilingual fourth grade students

Thesis (M.A.) University of Alaska Fairbanks, 2019This teacher action research focuses on how three fourth grade students interact and make meaning as they read the graphic novel, Amulet. While reading from the graphic novel, students engaged in the reading as design process to make meaning. These three students are Yup'ik students enrolled in a dual language school. Students interacted with peers and different modalities of meaning as they engaged in the meaning-making process. Data sources include a teacher research journal, audio recordings of readings and discussions, and students' reader response journals. Data analysis followed constructivist grounded theory. As there were various types of data collected and a multimodal text was used, multimodal data analysis was used to interpret the relationship across the various modes used in the study. Three main findings emerged from the data: 1. Vocabulary can be learned through multiple modes. 2. Students used words to mediate meaning socially and in a private manner. 3. Combined visuals and text support meaning making. These findings led to the conclusion that meaning making and research are both multimodal. The findings also reveal how emergent bilingual students were active meaning-makers and could read and respond to a graphic novel successfully. At times, writing prompts were used. While students designed meaning with multimodal texts, the writing prompts constricted their responses to certain topics, such as setting and characters, and did not allow for students to continue designing meaning in their own ways. Students were able to continue designing their own meaning when responding to the text in a natural, multimodal way without prompts constricting thoughts relating to the text

Multimodal Sentiment Analysis: A Survey

Multimodal sentiment analysis has become an important research area in the field of artificial intelligence. With the latest advances in deep learning, this technology has reached new heights. It has great potential for both application and research, making it a popular research topic. This review provides an overview of the definition, background, and development of multimodal sentiment analysis. It also covers recent datasets and advanced models, emphasizing the challenges and future prospects of this technology. Finally, it looks ahead to future research directions. It should be noted that this review provides constructive suggestions for promising research directions and building better performing multimodal sentiment analysis models, which can help researchers in this field.Comment: It needs to be returned for major modification

The Playful Writing Project: Exploring the synergy between play and writing with Reception class teachers

This study highlights the potential of play to support young children's mark-making and writing and, in turn, shows how writing can become a driver of play. By detailing elements of this symbiotic relationship, the intention was for teachers as practitioner–researchers to deepen their professional knowledge of how best to support young children in meaningful writing activity, thereby countering policy-led reductive understandings of both play and literacy. During a year-long research project, six English reception class teachers gathered photographs, video and multimodal observations of children and used these to analyse how multimodal mark-making and writing occurred as part of play in their classrooms. From this analysis, the concept of ‘playful writing’ was developed. This identified three interconnecting characteristics that mark-making and writing has as part of play activity: social function, multimodal movement and material possibilities. The findings of the study indicate that young children's classroom play stimulates the multimodal, social and material dimensions needed for the creation of meaningful and transformative writing. The study highlights the importance of harnessing teachers' expertise in paying close attention to children's playful literacy, in order to create a strong knowledge base from which to make the best decisions about literacy practices with young children