Introduction: The nexus of multimodality, multimodal literacy, and English language teaching in research and practice in higher education settings

The aims of this Special Issue are to (1) advance the current state of research-based knowledge about how multimodal and multimedia resources can be leveraged to enhance multimodal communication practices in English language teaching in higher education, and (2) to provide a platform for original research-based practical applications that incorporate innovative multisemiotic resources and techniques, thereby offering new perspectives on the benefits of the multimodal approach when teaching English for both general and specific purposes at the university level. In the following section, we discuss the role of multimodal literacy in the context of enhancing language proficiency as the underlying objective of English language practitioners

Exploring cognitive issues in visual information retrieval

A study was conducted that compared user performance across a range of search tasks supported by both a textual and a visual information retrieval interface (VIRI). Test scores representing seven distinct cognitive abilities were examined in relation to user performance. Results indicate that, when using VIRIs, visual-perceptual abilities account for significant amounts of within-subjects variance, particularly when the relevance criteria were highly specific. Visualisation ability also seemed to be a critical factor when users were required to change topical perspective within the visualisation. Suggestions are made for navigational cues that may help to reduce the effects of these individual differences

Towards a generic framework for situated collaborative storytelling

How we assimilate stories into our common experiences and shape culture is the field of study known as narrative intelligence. By following these assumptions and investigating theories of conversation and rhetoric, this paper outlines a generic framework for a visual collaborative storytelling system that emphasises participatory narration and shared understanding in a situated context. © 2009 ACM

English language learning kindergartners\u27 dynamic responses to picturebook reading

This study explores the nature of English language learning by kindergartners as they engage in multimodal literacy practices in response to picturebook readings in a mainstream classroom. Three focal Spanish-speaking kindergartners and their classroom teacher took part in this study. Data were collected daily for four months in a half-day morning kindergarten program. The participants\u27 verbal and nonverbal classroom interactions during picturebook readings were coded and analyzed to characterize the nature of ELL kindergartners\u27 multimodal responses to picturebook readings. Findings indicate that the classroom instruction did not fully address the differences in the focal children\u27s levels of English language proficiency. Further, the use of various modes of expression by individual children for meaning-making received limited support in terms of language development and literacy learning

Embedding Intelligence. Designerly reflections on AI-infused products

Artificial intelligence is more-or-less covertly entering our lives and houses, embedded into products and services that are acquiring novel roles and agency on users. Products such as virtual assistants represent the first wave of materializa- tion of artificial intelligence in the domestic realm and beyond. They are new interlocutors in an emerging redefined relationship between humans and computers. They are agents, with miscommunicated or unclear proper- ties, performing actions to reach human-set goals. They embed capabilities that industrial products never had. They can learn users’ preferences and accordingly adapt their responses, but they are also powerful means to shape people’s behavior and build new practices and habits. Nevertheless, the way these products are used is not fully exploiting their potential, and frequently they entail poor user experiences, relegating their role to gadgets or toys. Furthermore, AI-infused products need vast amounts of personal data to work accurately, and the gathering and processing of this data are often obscure to end-users. As well, how, whether, and when it is preferable to implement AI in products and services is still an open debate. This condition raises critical ethical issues about their usage and may dramatically impact users’ trust and, ultimately, the quality of user experience. The design discipline and the Human-Computer Interaction (HCI) field are just beginning to explore the wicked relationship between Design and AI, looking for a definition of its borders, still blurred and ever-changing. The book approaches this issue from a human-centered standpoint, proposing designerly reflections on AI-infused products. It addresses one main guiding question: what are the design implications of embedding intelligence into everyday objects

Multiliteracies for academic purposes : a metafunctional exploration of intersemiosis and multimodality in university textbook and computer-based learning resources in science

This thesis is situated in the research field of systemic functional linguistics (SFL) in education and within a professional context of multiliteracies for academic purposes. The overall aim of the research is to provide a metafunctional account of multimodal and multisemiotic meaning-making in print and electronic learning materials in first year science at university. The educational motivation for the study is to provide insights for teachers and educational designers to assist them in the development of students’ multiliteracies, particularly in the context of online learning environments. The corpus comprises online and CD-ROM learning resources in biology, physics and chemistry and textbooks in physics and biology, which are typical of those used in undergraduate science courses in Australia. Two underlying themes of the research are to compare the different affordances of textbook and screen formats and the disciplinary variation found in these formats. The two stage research design consisted of a multimodal content analysis, followed by a SF-based multimodal discourse analysis of a selection of the texts. In the page and screen formats of these pedagogical texts, the analyses show that through the mechanisms of intersemiosis, ideationally, language and image are reconstrued as disciplinary knowledge. This knowledge is characterised by a high level of technicality in image and verbiage, by taxonomic relations across semiotic resources and by interdependence among elements in the image, caption, label and main text. Interpersonally, pedagogical roles of reader/learner/viewer/ and writer/teacher/designer are enacted differently to some extent across formats through the different types of activities on the page and screen but the source of authority and truth remains with the teacher/designer, regardless of format. Roles are thus minimally negotiable, despite the claims of interactivity in the screen texts. Textually, the organisation of meaning across text and image in both formats is reflected in the layout, which is determined by the underlying design grid and in the use of graphic design resources of colour, font, salience and juxtaposition. Finally, through the resources of grammatical metaphor and the reconstrual of images as abstract, both forms of semiosis work together to shift meanings from congruence to abstraction, into the specialised realm of science

Science and EAL teachers’ perspectives and practices in building word knowledge in implementing the new Victorian EAL curriculum

The recent implementation of The Victorian Curriculum F-10: EAL requires content teachers who teach EAL students to be familiar with the revised EAL curriculum for the purposes of planning and developing approaches to assist learners’ development in English. In the literature and in curriculum frameworks, word knowledge is considered an important aspect of EAL students’ learning. However, little is known about what pedagogical practices teachers across the curriculum perceive as being important, and use, in developing EAL students’ vocabulary. In this study, we investigated linguistically responsive vocabulary teaching in a Year 7 science class. Our aim was to elucidate teachers’ perceptions and practices in teaching vocabulary in science. The qualitative case study drew on principles of linguistically responsive instruction (LRI), which refers to practices for meeting the needs of students in culturally and linguistically diverse classrooms. Analysis of interview and classroom data from an EAL teacher and a science teacher revealed a range of LRI practices for developing word knowledge based on understanding the distinction between conversational and academic language, language learning principles, responsive teacher talk, plurilingual awareness, and the importance of social interaction for learners. We offer recommendations for a whole school approach to LRI, adaptation to online LRI, and curriculum development

17 ways to say yes:Toward nuanced tone of voice in AAC and speech technology

People with complex communication needs who use speech-generating devices have very little expressive control over their tone of voice. Despite its importance in human interaction, the issue of tone of voice remains all but absent from AAC research and development however. In this paper, we describe three interdisciplinary projects, past, present and future: The critical design collection Six Speaking Chairs has provoked deeper discussion and inspired a social model of tone of voice; the speculative concept Speech Hedge illustrates challenges and opportunities in designing more expressive user interfaces; the pilot project Tonetable could enable participatory research and seed a research network around tone of voice. We speculate that more radical interactions might expand frontiers of AAC and disrupt speech technology as a whole

The Colour concept generator: a computer tool to propose colour concepts for products.

This thesis documents research undertaken into the design and evaluation of a computer tool (Colour Concept Generator) to produce colour schemes for products from verbal descriptors depicting a required aesthetic image or style. The system was designed to translate between descriptive words and colour combinations and aims to provide a form of ideas stimulus for a product designer at the initial stages of the design process. The computer system uses elements of artificial intelligence (AI) to `learn' colour and descriptor semiotic relations from a product designer based upon a proposed objective criteria or to reflect a designers personal style. Colour concepts for products can then be generated from descriptors based upon these semiotic relations. The philosophy of the research is based upon the idea of computing colour aesthetics at the front end of the design process and the design of an Al software mechanism to facilitate this. The problem was analysed with respect to the available literature on colour and a set of detail requirements for the system were presented. The system was then designed and code based upon the requirements and evaluated in terms of the overall philosophy, system methodology and application of computer media. The research is a contribution to the field of computer aided design regarding colour aesthetics and demonstrates the possibility of using an artificial intelligent machine to inspire and stimulate creative human thought. The Al software mechanism of the Colour Concept Generator is presented as an application of Al to aesthetic design. 1