Multimodaalinen rekontekstualisaatio European University Associationin trendiraporteissa

This thesis studies multimodal recontextualization in European University Assocation Trend Reports. Recontextualization refers to the changes that are required for parts of discourse to move between contexts. Regular reports are published by the European University Association on the topic of the Bologna Process and its implementation across Europe. Two research questions guide the study: 1. What evidence of recontextualization can be found between textual elements and diagrams in higher education policy documents? 2. To what extent can multimodal analysis improve our understanding of recontextualization? The primary multimodal framework rests on Bateman’s (2011) abstract model for semiotic modes. These modes are realised through application of semiotic resources on a material substrate while being interpreted contextually based on an understanding of the relevant discourse semantics. Of particular note in this study are the semiotic modes of text-flow, page-flow, layout and the diagrammatic mode. The typology used for categorising and operationalizing diagrams is based on that of Engelhardt and Richards (2018). The most common types of diagrams in the dataset are various bar charts, table charts and choropleth maps. The total number of diagrams in the corpus is 212, of which these three categories form 88%. The way information is visually encoded is considered in terms of the principles of arranging, varying and linking performed by the diagrams. The recontextualization analysis proceeds by considering the transformations that take place when information within text-based contexts is transferred or translated over to a diagrammatic visualization. The methodology for this follows Van Leeuwen (2008), and primarily concerns the transformations of substitution, deletion and legitimation. The analysis shows that the diagrams in the dataset are used for three primary purposes: The summarization of information presented elsewhere, reinstatement of in-text claims for additional legitimation, and the presentation of temporal information in order to allow for comparison between trend reports or participating countries. The recontextualizing transformations of substitution and deletion are present in nearly every diagram in the dataset, while legitimation is used more conservatively to provide support for claims made in the text. These recontextualizations take advantage of tresources offered by the diagrammatic mode that allows them to present information by using otherwise unavailable dimensions such as temporal or the spatial. The results support the notion that analysis of recontextualization can be successfully combined with a multimodal approach, and this has been found to potentially support both approaches. This is in line with the interest multimodal researchers have shown in the demarcation of and transition of meaning between semiotic modes

6th International Conference on Multimodality - Conference guide with abstracts

The aim of the conference was to contribute to moving forward the field of multimodal research and to help connect the diverse community of scholars working within it. 6ICOM is a place where we can explore the full range of different ways in which multimodality has been taken up and where we can recognize their points of connection. The conference was organised with support from the UK’s Economic and Social Research Council, who fund MODE. MODE is a node of the National Centre for Research Methods based in the Institute of Education and aims to develop and promote multimodal methods for researching digital data and environments. 6ICOM’s programme includes an impressive set of paper presentations (125) and invited keynotes (5). The presenters engage with a wide range of disciplines, ideas and methods, reflecting the diverse character of multimodality and latest developments in the field. They speak to a range of contexts, theoretical and methodological approaches, technologies and types of data

The (re)invention of feedback

This thesis argues that, far from being obstacles to teacher response, the new media and avant-garde practices of appropriation, the readymade, and nonlinearity can actually work to orient feedback toward textual pratices that both conceptually and technologically embrace networks. The aim is to construct pedagogy (generally) and teacher response (specifically) as creative acts composed in response to the creative act that is student work

Oh no, it’s raining! A study of how information in online weather reports is interpreted, integrated, and used in everyday decision-making by laypeople

Different people in different occupations depend on weather forecasts to plan their work and recreational schedules. People with no expertise in meteorology frequently interpret weather forecasts and uncertainty information. These non-experts apply their prior knowledge and experiences in a variety of fields and their abilities to synthesize different types of information to interpret forecasts. Initial studies of communication and the interpretation of forecasts and uncertainty information focused on separate pieces of information rather than the situations of ordinary users. In this study, situations of typical users are simulated to increase the ecological validity when examining how different user groups interpret, integrate, and use information from online weather reports in their everyday decision-making. First, qualitative interviews of twenty-one Norwegians from five different user groups (farmers, exterior painters, tour guides, and upper secondary school teachers and students) were conducted. Second, sixteen upper secondary school students participated in an eye-tracking study. Immediately after this study, the participants were asked to verbalize their thoughts (think out loud) when viewing the gaze data. Semi-structured interviews were conducted to ensure the collection of rich data. In this study, the participants were given weather forecasts from one selected online weather report (www.Yr.no), which served as a basis for both data collections. The verbal data were analyzed by assigning codes and categories to the transcribed statements. The main findings of the study are as follows: a) For each representation, such as tables, diagrams, numbers and symbols, a set of strengths and functions (affordances) was ascribed and exploited by the participants. b) Only part of the representations that provided forecast and uncertainty information at the website was used by each participant. c) Nuances such as color and the number of drops were important in the interpretations of the weather symbols and forecast uncertainty, which were sometimes interpreted differently than intended by the forecast provider. d) Prior knowledge affected the participants’ interpretations and even superseded the given information in apparent conflicts. e) The interpretations were also affected by the integration of information from different representations, which was performed to create a dynamic picture of the weather and to control and compare information. f) The decision-making process influenced the construction of different reading paths and the selections of representations in different situations. g) The participants used a varying amount of information in their decision-making; their selection was dependent on the importance of the envisaged activity and the weather conditions for the day. h) Additionally, in situations in which the participants had a lack of experiences, this lack provides a possible explanation for why part of the information was occasionally not understood and used. i) Evaluations of weather dynamics and the degree of certainty in the forecast were disregarded when quick decisions were made. Some implications of the findings for communication and future research are as follows: a) Providers of online weather reports should take care in the details of the information they present because such nuances may be interpreted as substantial information. b) Uncertainty information should be easy to understand and use, and the benefits of this information should be clear to enable users to interpret the degree of certainty as intended. c) Information communicated in online weather reports should enable the use of different decision-processes. d) A comprehensive use of multimodal information in communication appears to be an advantage when information is used by different users in different situations. e) However, some users should be guided and supported to facilitate the interpretation, integration, and use of information from multiple representations in situations where they lack experiences and/or aim for an elaborate decision process. f) One possibility to support persons that lack experiences and have low situation awareness might be to provide consequences and impacts of forecast weather. g) Notably, forecast providers should take into account the needs of the forecast users. h) To achieve this goal, users’ needs should be addressed in a coproduction process. i) Future studies should investigate the situations of typical users and different decision-making processes

Beyond Media Borders, Volume 1

This open access book promotes the idea that all media types are multimodal and that comparing media types, through an intermedial lens, necessarily involves analysing these multimodal traits. The collection includes a series of interconnected articles that illustrate and clarify how the concepts developed in Elleström’s influential article The Modalities of Media: A Model for Understanding Intermedial Relations (Palgrave Macmillan, 2010) can be used for methodical investigation and interpretation of media traits and media interrelations. The authors work with a wide range of old and new media types that are traditionally investigated through limited, media-specific concepts. The publication is a significant contribution to interdisciplinary research, advancing the frontiers of conceptual as well as practical understanding of media interrelations. This is the first of two volumes. It contains Elleström’s revised article and six other contributions focusing especially on media integration: how media products and media types are combined and merged in various ways

Multimodal interaction with mobile devices : fusing a broad spectrum of modality combinations

This dissertation presents a multimodal architecture for use in mobile scenarios such as shopping and navigation. It also analyses a wide range of feasible modality input combinations for these contexts. For this purpose, two interlinked demonstrators were designed for stand-alone use on mobile devices. Of particular importance was the design and implementation of a modality fusion module capable of combining input from a range of communication modes like speech, handwriting, and gesture. The implementation is able to account for confidence value biases arising within and between modalities and also provides a method for resolving semantically overlapped input. Tangible interaction with real-world objects and symmetric multimodality are two further themes addressed in this work. The work concludes with the results from two usability field studies that provide insight on user preference and modality intuition for different modality combinations, as well as user acceptance for anthropomorphized objects.Diese Dissertation präsentiert eine multimodale Architektur zum Gebrauch in mobilen Umständen wie z. B. Einkaufen und Navigation. Außerdem wird ein großes Gebiet von möglichen modalen Eingabekombinationen zu diesen Umständen analysiert. Um das in praktischer Weise zu demonstrieren, wurden zwei teilweise gekoppelte Vorführungsprogramme zum \u27stand-alone\u27; Gebrauch auf mobilen Geräten entworfen. Von spezieller Wichtigkeit war der Entwurf und die Ausführung eines Modalitäts-fusion Modul, das die Kombination einer Reihe von Kommunikationsarten wie Sprache, Handschrift und Gesten ermöglicht. Die Ausführung erlaubt die Veränderung von Zuverlässigkeitswerten innerhalb einzelner Modalitäten und außerdem ermöglicht eine Methode um die semantisch überlappten Eingaben auszuwerten. Wirklichkeitsnaher Dialog mit aktuellen Objekten und symmetrische Multimodalität sind zwei weitere Themen die in dieser Arbeit behandelt werden. Die Arbeit schließt mit Resultaten von zwei Feldstudien, die weitere Einsicht erlauben über die bevorzugte Art verschiedener Modalitätskombinationen, sowie auch über die Akzeptanz von anthropomorphisierten Objekten

Multimodal interaction with mobile devices : fusing a broad spectrum of modality combinations

This dissertation presents a multimodal architecture for use in mobile scenarios such as shopping and navigation. It also analyses a wide range of feasible modality input combinations for these contexts. For this purpose, two interlinked demonstrators were designed for stand-alone use on mobile devices. Of particular importance was the design and implementation of a modality fusion module capable of combining input from a range of communication modes like speech, handwriting, and gesture. The implementation is able to account for confidence value biases arising within and between modalities and also provides a method for resolving semantically overlapped input. Tangible interaction with real-world objects and symmetric multimodality are two further themes addressed in this work. The work concludes with the results from two usability field studies that provide insight on user preference and modality intuition for different modality combinations, as well as user acceptance for anthropomorphized objects.Diese Dissertation präsentiert eine multimodale Architektur zum Gebrauch in mobilen Umständen wie z. B. Einkaufen und Navigation. Außerdem wird ein großes Gebiet von möglichen modalen Eingabekombinationen zu diesen Umständen analysiert. Um das in praktischer Weise zu demonstrieren, wurden zwei teilweise gekoppelte Vorführungsprogramme zum 'stand-alone'; Gebrauch auf mobilen Geräten entworfen. Von spezieller Wichtigkeit war der Entwurf und die Ausführung eines Modalitäts-fusion Modul, das die Kombination einer Reihe von Kommunikationsarten wie Sprache, Handschrift und Gesten ermöglicht. Die Ausführung erlaubt die Veränderung von Zuverlässigkeitswerten innerhalb einzelner Modalitäten und außerdem ermöglicht eine Methode um die semantisch überlappten Eingaben auszuwerten. Wirklichkeitsnaher Dialog mit aktuellen Objekten und symmetrische Multimodalität sind zwei weitere Themen die in dieser Arbeit behandelt werden. Die Arbeit schließt mit Resultaten von zwei Feldstudien, die weitere Einsicht erlauben über die bevorzugte Art verschiedener Modalitätskombinationen, sowie auch über die Akzeptanz von anthropomorphisierten Objekten

Beyond Media Borders, Volume 1

This open access book promotes the idea that all media types are multimodal and that comparing media types, through an intermedial lens, necessarily involves analysing these multimodal traits. The collection includes a series of interconnected articles that illustrate and clarify how the concepts developed in Elleström’s influential article The Modalities of Media: A Model for Understanding Intermedial Relations (Palgrave Macmillan, 2010) can be used for methodical investigation and interpretation of media traits and media interrelations. The authors work with a wide range of old and new media types that are traditionally investigated through limited, media-specific concepts. The publication is a significant contribution to interdisciplinary research, advancing the frontiers of conceptual as well as practical understanding of media interrelations. This is the first of two volumes. It contains Elleström’s revised article and six other contributions focusing especially on media integration: how media products and media types are combined and merged in various ways

Beyond Media Borders, Volume 2

This open access book promotes the idea that all media types are multimodal and that comparing media types, through an intermedial lens, necessarily involves analysing these multimodal traits. The collection includes a series of interconnected articles that illustrate and clarify how the concepts developed in Elleström’s influential article The Modalities of Media: A Model for Understanding Intermedial Relations (Palgrave Macmillan, 2010) can be used for methodical investigation and interpretation of media traits and media interrelations. The authors work with a wide range of old and new media types that are traditionally investigated through limited, media-specific concepts. The publication is a significant contribution to interdisciplinary research, advancing the frontiers of conceptual as well as practical understanding of media interrelations. This is the second of two volumes. It contains a concluding article by Elleström and seven contributions concentrated on the issue of media transformations: how media characteristics are transferred and transfigured among various media products and media types

Self-directed multimodal learning in higher education

This book aims to provide an overview of theoretical and practical considerations in terms of self-directed multimodal learning within the university context. Multimodal learning is approached in terms of the levels of multimodality and specifically blended learning and the mixing of modes of delivery (contact and distance education). As such, this publication will provide a unique snapshot of multimodal practices within higher education through a self-directed learning epistemological lens. The book covers issues such as what self-directed multimodal learning entails, mapping of specific publications regarding blended learning, blended learning in mathematics, geography, natural science and computer literacy, comparative experiences in distance education as well as situated and culturally appropriate learning in multimodal contexts. This book provides a unique focus on multimodality in terms of learning and delivery within the context of self-directed learning. Therefore, the publication would not only advance the scholarship of blended and open distance learning in South Africa, but also the contribute to enriching the discourse regarding self-direction. From this book readers will get an impression of the latest trends in literature in terms of multimodal self-directed learning in South Africa as well as unique empirical work being done in this regard