The interface between rhetoric and layout in multimodal artefacts

Peer reviewe

Reading paths and visual perception in multimodal research, psychology and brain sciences

Peer reviewe

Culture in the design of mHealth UI:An effort to increase acceptance among culturally specific groups

Purpose: Designers of mobile applications have long understood the importance of users’ preferences in making the user experience easier, convenient and therefore valuable. The cultural aspects of groups of users are among the key features of users’ design preferences, because each group’s preferences depend on various features that are culturally compatible. The process of integrating culture into the design of a system has always been an important ingredient for effective and interactive human computer interface. This study aims to investigate the design of a mobile health (mHealth) application user interface (UI) based on Arabic culture. It was argued that integrating certain cultural values of specific groups of users into the design of UI would increase their acceptance of the technology. Design/methodology/approach: A total of 135 users responded to an online survey about their acceptance of a culturally designed mHealth. Findings: The findings showed that culturally based language, colours, layout and images had a significant relationship with users’ behavioural intention to use the culturally based mHealth UI. Research limitations/implications: First, the sample and the data collected of this study were restricted to Arab users and Arab culture; therefore, the results cannot be generalized to other cultures and users. Second, the adapted unified theory of acceptance and use of technology model was used in this study instead of the new version, which may expose new perceptions. Third, the cultural aspects of UI design in this study were limited to the images, colours, language and layout. Practical implications: It encourages UI designers to implement the relevant cultural aspects while developing mobile applications. Originality/value: Embedding Arab cultural aspects in designing UI for mobile applications to satisfy Arab users and enhance their acceptance toward using mobile applications, which will reflect positively on their lives.</p

Encoding the UX: User Interface as a Site of Encounter between Data Journalists and Their Constructed Audiences

This paper examines how data journalism producers encode their audiences’ user experience – or how they construct meaning within the parameters of a user interface. We argue that interfaces enact data journalists’ perceptual image of their audience, and are therefore a meaningful object of study to explore: (a) the relationship between data reporters and their publics; (b) the tensions that emerge from that relationship; and (c) how those tensions are negotiated. Our combined evaluation of producer and artefact perspectives is based on interviews with 12 data journalism producers in legacy UK newsrooms, and interaction design, and multimodal discourse analysis of 12 stories produced by our informants. Findings suggest that data journalism producers’ efforts to create content that resonates with their audiences have resulted in the diversification of data journalism formats, and a more mature view of interactivity, with producers striving to offer experiences in whose meaning negotiation the reader takes centre stage. The reader is perceived as a co-creator of meaning through the expectation of either physical interaction or cognitive immersion. Yet, the paternalistic element, even in the most discovery-/reader-driven stories, remains stronger, manifesting in a carefully crafted experience and pointing to journalism producers’ traditional self-perception as gatekeepers of information

Multimedia analysis and design: a conceptual framework

The paper proposes a conceptual model inspired to semiotic theories, to be applied to the analysis and design of multimedia. We introduce a meta-model with four levels of semantic aggregation. There results a framework of concepts, relations and processes accounting for the multiple meanings that arise from a multimedia text. We explore the effectiveness of the model by considering four commercial clips by the brand Lanc\uf4me. Our analysis confirms that the framework is well suited to the analysis, indexing, design of narrative multimedia

Sociotechnical structures, materialist semiotics, and online language learning

Based on a study of the digital literacy practices of immigrant Filipino students in Vancouver, this paper focuses on how learners with unequal access to resources engage with different tools to locate information and find opportunities for language learning online. Data was collected through interviews and observations of participants as they used YouTube, Google Search, and Google Translate to decode unfamiliar words and find resources for learning. Framed through a materialist semiotic lens, this study examined how the students negotiated their resources on these platforms to achieve different intentions. Findings show that the way learners navigate these spaces can vary based on the devices they use (laptop vs. mobile phone), the user interface (browser vs. app), and the orientation they choose (landscape vs. portrait). The material dimensions of the screen determine the arrangement of semiotic forms, and varying configurations of devices, interfaces, and orientations shape the information made available to the learner and the digital literacy practices of scrolling, clicking, and shifting tabs. Recognizing how the online environment of a platform can shift across these layers of mediation, this paper conceptualizes the linguistic and semiotic forms that constitute design as sociotechnical structures which provide various learning affordances and constraints

AI2D-RST : A multimodal corpus of 1000 primary school science diagrams

This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural sciences, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowdsourced descriptions, which was originally developed to support research on automatic diagram understanding and visual question answering. Building on the segmentation of diagram layouts in AI2D, the AI2D-RST corpus presents a new multi-layer annotation schema that provides a rich description of their multimodal structure. Annotated by trained experts, the layers describe (1) the grouping of diagram elements into perceptual units, (2) the connections set up by diagrammatic elements such as arrows and lines, and (3) the discourse relations between diagram elements, which are described using Rhetorical Structure Theory (RST). Each annotation layer in AI2D-RST is represented using a graph. The corpus is freely available for research and teaching.Peer reviewe

Free speech platforms and the impact of the U.S. insurrection: Misinformation in memes

The circulation of false information online, whether intentional or unintentional, has become one of the most pressing threats to social cohesion and security for governments around the world. The U.S. insurrection on January 6th, 2021 was a striking example of how polarising and inflammatory rhetoric posted on social media platforms can influence offline movements and connect to real-world violence. It also highlighted the negative effects of meme culture, post-truth and free speech ideology within right-wing groups in the U.S. and beyond. Looking at this event as a main case study and considering the ramifications more than two years on, this paper broadly aims to unpack the forces, influences and affordances of so-called free speech platforms that contributed to the insurrection through a thorough literature review. More specifically, I identify how and why the rise of meme sharing on the platform Gab has helped generate right-wing identity, shape attitudes towards the mainstream media and increase the proliferation of false narratives and culture wars. Through a social semiotic multimodal analysis, this thesis contributes to existing scholarship by assessing the ideological role of memes posted on Gab in the two years since the Capitol attack. The findings indicate that blatant falsehoods are reinforced and masked by appeals to humour, emotion, values and beliefs within right-wing communities. It also reveals that the concept of free speech is being instrumentalised to attack public institutions.Master's Thesis in Digital CultureDIKULT350MAHF-DIKU

Research genres and multiliteracies: channelling the audience's gaze in powerpoint presentations

Tese (doutorado) - Universidade Federal de Santa Catarina, Centro de Comunicação e Expressão. Programa de Pós-Graduação em Letras/Inglês e Literatura Correspondente.PowerPoint-supported presentations have become an important event for creating and sharing scientific knowledge within and across disciplines (LaPorte et al., 2002; Kunkel, 2004; Tardy, 2005; Adams, 2006). Yet little is known about the ways semiotic resources enabled by PowerPoint technology of slide editing and management (e.g. slide dimensions, layout, colour) are combined with conventional resources of "research talks" (Swales, 2005[2004]) and contribute to building presentations that are valued in specific contexts. In order to inform our understanding of how research meanings are multimodally made under the influence of the software, in this thesis I investigate a set of fourteen PowerPoint Research Presentations (PPRPs) from Applied Linguistics. Two planes of cohesion are explored: (1) along the slideshows; and (2) between the slideshows and the performance. Regarding the first plane, the analysis of "periodicity" (Martin and Rose, 2007[2003]) revealed that applied linguists foreground the software's 'modularised logic', construing 'serial expansion' (Martin and Rose (2007[2003]). Others however customise slideshows so as to build 'Design Hierarchies', in which particular slides are assigned higher discursive statuses. These presenters construed a path for their audiences gaze by a configuration of semiotic resources of the display mode - e.g. slide position, background, layout, typography. As for the second plane of cohesion, I propose that slides and performance relate by 'synchronicity'. The tool recontextualizes the system of taxis (Halliday, 2009c; Halliday and Matthiessen, 2004) to account for the semantic interdependency between the displayed discourse and the performative discourse at a given point in PPRPs. In each of the cohesive planes, I set out to identify the software resources that play a role in construing cohesive ties, and evaluate both their "functional specialization" (cf. Halliday, 2009e[1975]; Kress, 2008[2003]; Jewitt and Kress, 2008[2003]) and the demands they impose on presenters and on audiences in terms of genre, discipline, software and multimodal literacies. By indicating some of the ways in which the software influences the "process of semiotic production" (Kress and van Leeuwen, 2001) of such practice, I intend to move beyond prescriptive (e. g. Costa, 2001; Cyphert, 2004; DuFrene and Lehman, 2004; Grant, 2010) as well as technically-focused (e.g. Downing and Garmon, 2002; Jones, 2003) accounts of PowerPoint. As a conclusion, I suggest that descriptions of the meaning potential in PPRPs and its conditions of access should be incorporated in pedagogies of academic multiliteracies (New London Group, 1996; Kope and Kalantizs, 2000).Apresentações de pesquisa com uso de PowerPoint desempenham um papel importante na criação e negociação de conhecimento científico em diferentes disciplinas (LaPorte et al., 2002; Kunkel, 2004; Tardy, 2005; Adams, 2006). Entretanto, pouco sabemos sobre os modos como os recursos semióticos potencializados pela tecnologia PowerPoint para edição e gerenciamento de slides (e.g. dimensões do slide, arranjo, cor) são combinados com recursos convencionais dos "relatos de pesquisa" (Swales, 2005[2004]) e contribuem para construir apresentações valorizadas em contextos específicos. No intuito de informar nosso entendimento sobre como significados de pesquisa são multimodalmente construídos sob a influência do software, nesta tese, investigo um conjunto de quatorze apresentações de pesquisa em PowerPoint (APPP) em Linguística Aplicada. Dois planos coesivos são explorados: (1) ao longo do texto em slides; e (2) entre os slides e a performance. No tocante ao primeiro plano, a análise da "periodicidade" (Martin e Rose, 2007[2003]) da informação revelou que os linguistas aplicados tendem a aderir à 'lógica modularizada' do software, realizando "expansão em série" (Martin e Rose (2007[2003]) do discurso. Outros, porém, 'personalizam' o texto em slides de modo a construir 'Hieraquias de Desenho', as quais atribuem valor de informação superordinada à determinados slides. Esses apresentadores direcionam o olhar de sua audiência por meio de uma configuração de recursos semióticos particulares do modo de exibição (e.g. sequência, fundo, arranjo, tipografia). Quanto ao segundo plano coesivo, proponho que slides e performance se relacionam por 'sincronicidade'. Essa erramenta recontextualiza o sistema de taxe (Halliday, 2009c; Halliday e Matthiessen, 2004) para explicar a interdependência semântica entre o discurso exibido e o discurso performado em um determinado ponto da APPP. Em cada um dos planos coesivos, busco identificar os recursos do software que desempenham função coesiva e avaliar tanto a sua "especialização funcional" (cf. Halliday, 2009e[1975]; Kress, 2008[2003]; Jewitt e Kress, 2008[2003]) quanto as demandas de letramento que impõem nos apresentadores e na audiência no que tange a gênero, disciplina, software e multimodalidade. Ao apontar alguns dos modos pelos quais o software influencia o "processo de produção semiótica" (Kress e van Leeuwen, 2001) dessa prática, pretendo ir além de orientações prescritivas (e. g. Costa, 2001; Cyphert, 2004; DuFrene e Lehman, 2004; Grant, 2010) e focadas em aspectos técnicos (e.g. Downing and Garmon, 2002; Jones, 2003). Sugiro, por fim, que a descrição dos significados potenciais em APPP e suas condições de acesso sejam incorporadas em pedagogias de multiletramentos acadêmicos (New London Group, 1996; Kope e Kalantizs, 2000)

Modelling the structure of a multimodal artefact

This dissertation studied the structure of multimodal artefacts, or how language, image and other semiotic modes combine and interact in documents. This places the study within the emerging field of multimodal research, which uses linguistic methods to study the interaction of multiple semiotic modes. Despite the growing amount of multimodal research, the structure of multimodal artefacts has not received the attention it warrants. Previous studies have been either very detailed or exceedingly abstract, leaving a significant gap between data and theory, which this dissertation attempted to bridge. To do so, the dissertation adopted a data-driven approach to multimodal analysis, addressing the structure of multimodal artefacts, the factors that shape the artefact structure, and the role of structure in the recognition and interpretation of the artefacts. The data consisted of tourist brochures produced by the city of Helsinki between 1967 and 2008, which allowed a longitudinal perspective to their multimodal structure. A total of 58 double-pages were annotated for their content, visual appearance, layout and rhetorical organisation, and compiled into an XML-based multimodal corpus. To study the corpus, the dissertation developed visualisation methods that combined information from multiple analytical layers of the corpus to represent the multimodal structures in the data. The study revealed the functional motivation behind the structure of the tourist brochures, identifying patterns in their hierarchical and rhetorical organisation, which were used to fulfil specific communicative tasks. The configuration of these patterns, in turn, signalled how the brochure was to be interpreted. The results also showed that after the year 1985, which marked the introduction of desktop publishing software, the organising principles of the tourist brochures have shifted towards a more fragmented and non-linear structure.Väitöskirja osoittaa, että kielitieteen menetelmiä voidaan soveltaa menestyksekkäästi myös visuaalisen viestinnän tutkimukseen. Vaikka kieltä ja kuvaa yhdistelevät tekstit ovat jo pitkään olleet osa arkipäiväistä elämää, kielen ja kuvan vuorovaikutuksesta tiedetään edelleen melko vähän, joka asettaa haasteen esimerkiksi keskustelulle medialukutaidon vaatimuksista. Kielitiede auttaa vastaamaan haasteeseen, sillä tieteenalalla on pitkä perinne viestinnän rakenteiden järjestelmällisessä tutkimuksessa. Tärkeintä on löytää sopiva teoria: tekstin rakennetta kuvaavat teoriat soveltuvat paremmin visuaalisen viestinnän tutkimukseen kuin esimerkiksi kieliopin teoria. Väitöskirjatutkimus avaa näin uusia mahdollisuuksia visuaalisen viestinnän empiiriseen tutkimukseen, jonka tuottamaa tietoa voidaan soveltaa tulevaisuudessa esimerkiksi opetusmateriaalien ja julkisen viestinnän kehittämisessä. Tutkimuksen aineisto koostuu Helsingin kaupungin vuosina 1967-2008 julkaisemista englanninkielisistä matkailuesitteistä. Laajasta aineistosta valittiin yhteensä 58 aukeamaa, jotka analysoitiin sisältöä, ulkoasua, sommittelua sekä sisällön välisiä suhteita kuvaavan mallin avulla. Analyysin pohjalta luotiin tietokoneen avulla useita kielen ja kuvan vuorovaikutusta sekä esitteen rakennetta kuvaavia malleja, sekä kehitettiin uusia työkaluja tutkittujen rakenteiden visualisoimiseksi. Havaintojen perusteella matkailuesitteiden rakenne muuttuu 1980-luvulla pirstaleiseksi, samaan aikaan kun tietokoneet otetaan käyttöön graafisessa suunnittelussa. Siinä missä vanhat esitteet käyttävät ilmaisuun valtaosin kirjoitettua kieltä, uudet pilkkovat sisällön pieniin yksiköihin, jotka yhdistelevät kieltä ja kuvaa. Muutos havaittiin kehitettyjen mallien avulla, sillä sitä on vaikea todentaa silmämääräisesti. Havainto on merkittävä, koska visuaalisen viestinnän kasvava merkitys liitetään usein visuaalisen sisällön määrään, ei sen yhteistoimintaan ja vuorovaikutukseen kielen kanssa. Visuaalisen sisällön määrä pysyy kuitenkin ennallaan tutkitulla aikavälillä - keskimäärin 50 % esitteiden pinta-alasta on valokuvia, kuvitusta ja muita graafisia elementtejä - vaikka sisällön esittämiseen käytetyt rakenteet muuttuvat samaan aikaan perusteellisesti