The Impact of an Integrated iPad Daily Multiliteracies Pedagogy on Elementary Students' Reading Achievement, Skills, Engagement, Collaboration and Learning, and Self Perception

This study investigated the impact of the absence and presence of digital technology as part of a daily literacy program on second grade elementary students reading achievement, skill development and engagement, collaboration and learning, and student and parent perceptions of reading ability and engagement. A mixed methods approach including PM Benchmark Assessments, video footage, questionnaires, and interviews, was incorporated. Students in the Integrated iPad Literacy Program (IiLP) demonstrated greater gains than students in the Critical Reflective Literacy Program which used an approach that did not include technology. Both student and parent perceptions paired greater reading engagement with greater reading ability. With findings suggesting that infusing technology, multiliteracies, and multimodalities into a literacy program positively impacts student literacy and thinking development, recommendations for altering current literacy programs are discussed

Shared cues, different violence organisations: comparing visual recruitment strategies of extremists, gangs, PMCs/mercenaries, and militaries

Extremists, gangs, militaries, and private military contractors (PMC)/mercenaries share the need to recruit, and all employ video for this purpose. How they use these videos to attract new members remains unclear, however. The study is a response to calls for a visual turn in violent extremism and builds upon an emerging shift within the literature, which examines the persuasive power of videos produced by some violence organisations (VOs), exploring the role that narratives, multimodalities, and symbols play in recruitment. It takes an empirical approach to answer the question, ‘what visual strategies are used by violence organisations to recruit members to their group or call them to action?’ Theoretically, it combines Hogg's theory on group membership and identity, Ellul's work on myths and the technique of propaganda, and Mirzoeff’s approach to intervisuality. Through primary analysis of 117 videos, produced and/or circulated by VOs, it shows that VOs encode their videos with similar cues designed to enhance feelings of belonging, identity, and shared beliefs in order to persuade and influence viewers. It therefore makes a pertinent contribution to the literature regarding comparative analysis, which tends to treat these VOs as distinct groups, failing to adequately consider the significance of shared approaches. The findings indicate that VOs’ recruitment and call to action videos should not be viewed independently, but as part of the broader ecosystem of online video content, designed not merely to entertain, but also to persuade; thus, they have implications for responding to extremist content

Brain Computer Interfaces and Emotional Involvement: Theory, Research, and Applications

This reprint is dedicated to the study of brain activity related to emotional and attentional involvement as measured by Brain–computer interface (BCI) systems designed for different purposes. A BCI system can translate brain signals (e.g., electric or hemodynamic brain activity indicators) into a command to execute an action in the BCI application (e.g., a wheelchair, the cursor on the screen, a spelling device or a game). These tools have the advantage of having real-time access to the ongoing brain activity of the individual, which can provide insight into the user’s emotional and attentional states by training a classification algorithm to recognize mental states. The success of BCI systems in contemporary neuroscientific research relies on the fact that they allow one to “think outside the lab”. The integration of technological solutions, artificial intelligence and cognitive science allowed and will allow researchers to envision more and more applications for the future. The clinical and everyday uses are described with the aim to invite readers to open their minds to imagine potential further developments

The Impact of Sound on Virtual Landscape Perception: An Empirical Evaluation of Aural-Visual Interaction for 3D Visualization

An understanding of quantitative and qualitative landscape characteristics is necessary to successfully articulate intervention or change in the landscape. In landscape planning and design 3D visualizations have been used to successfully communicate various aspects of landscape to a diverse population, though they have been shown to lag behind real-world experience in perceptual experiments. There is evidence that engaging other senses can alter the perception of 3D visualizations, which this thesis used as a departure point for the research project. Three research questions guide the investigation. The first research question is: How do fundamental elements in visualizations (i.e. terrain, vegetation and built form) interact with fundamental sound types (i.e. anthropogenic, mechanical and natural) to affect perceived realism of, and preference for, 3D landscape visualization? The research used empirical methods of a controlled experiment and statistical analysis of quantitative survey responses to examine the perceptual responses to the interaction aural and visual stimuli in St. James’s Park, London, UK. The visualizations were sourced from Google Earth, and the sounds recorded in situ, with Google Earth chosen as it is being used more frequently in landscape planning and design processes, though has received very little perceptual research focus. The second research question is: Do different user characteristics interact with combined aural-visual stimuli to alter perceived realism and preferences for 3D visualization? The final research question emerged out of the experiment design concentrating on research methodology: How effective is the Internet for aural-visual data collection compared to the laboratory setting? The results of the quantitative analysis can be summarized as follows: For research question 1 the results show that sound alters 3D visualization perception both positively and negatively, which varies by landscape element. For all visual conditions mechanical sound significantly lowers preference. For visualizations showing terrain only perceived realism and preference are significantly lowered by anthropogenic sound and significantly raised by natural sound for both realism and preference. For visualizations showing a combination of terrain with built form anthropogenic and mechanical sound significantly raises perceived realism. For visualizations showing a combination of terrain, vegetation and some built form a more complicated interaction occurs for realism, which is moderated by the amount of built form in the scene, e.g. with no buildings in the scene traffic and speech significantly lower realism ratings in similar ways while a small amount of built form visible resulted in speech significantly raising realism ratings. Preference was significantly lowered by anthropogenic and mechanical sound the most out of all three visual conditions. For research question 2 the results confirm that perception can vary for realism by gender and first language differences, and preference by age, first language, cultural and professional background and 3D familiarity. Finally for research question 3 and implications for Internet-based multisensory experiments there is strong evidence that audio hardware and experimental condition (laboratory vs. online) do not significantly alter realism and preference ratings, though larger display sizes can have a significant but very small effect on preference ratings (+/- 0.08 on a 5-point scale). The results indicate that sound significantly alters the perception of realism and preference for landscape simulated via 3D visualizations, with the congruence of aural and visual stimuli having a strong impact on both perceptual responses. The results provide important empirical evidence for future research to build upon, and raise important questions relating to authenticity of landscape experience, particularly when relying solely on visual material as visuals alone do not accurately simulate landscape experience. In addition the research confirms the cross-sensory nature of perception in virtual environments. As a result the inclusion of sound for landscape visualization and aesthetic research is concluded to be of critical importance. The research results suggest that when using sound with 3D visualizations the sound content match the visualized material, and to avoid using sounds that contain human speech unless there is a very strong reason to do so (e.g. there are humans in the visualization). The final chapter discusses opportunities for integrating sound with 3D visualizations in order to increase the perception of realism and preference in landscape planning and design processes, and concludes with areas for future research

Cognitive Activity Support Tools: Design of the Visual Interface

This dissertation is broadly concerned with interactive computational tools that support the performance of complex cognitive activities, examples of which are analytical reasoning, decision making, problem solving, sense making, forecasting, and learning. Examples of tools that support such activities are visualization-based tools in the areas of: education, information visualization, personal information management, statistics, and health informatics. Such tools enable access to information and data and, through interaction, enable a human-information discourse. In a more specific sense, this dissertation is concerned with the design of the visual interface of these tools. This dissertation presents a large and comprehensive theoretical framework to support research and design. Issues treated herein include interaction design and patterns of interaction for cognitive and epistemic support; analysis of the essential properties of interactive visual representations and their influences on cognitive and perceptual processes; an analysis of the structural components of interaction and how different operational forms of interaction components affect the performance of cognitive activities; an examination of how the information-processing load should be distributed between humans and tools during the performance of complex cognitive activities; and a categorization of common visualizations according to their structure and function, and a discussion of the cognitive utility of each category. This dissertation also includes a chapter that describes the design of a cognitive activity support tool, as guided by the theoretical contributions that comprise the rest of the dissertation. Those that may find this dissertation useful include researchers and practitioners in the areas of data and information visualization, visual analytics, medical and health informatics, data science, journalism, educational technology, and digital games

Interactive visualisation for low literacy users

Sixteen percent (5.2 million) of the UK population possess low levels of literacy. The Government and other non-profit organisations, due to funding reforms, are forced to reduce the provision of face-to-face advice, and therefore, are pushing advice services via telephone or internet. As a consequence, low literacy users are experiencing difficulties finding the information they need to solve their day to day problems online. This thesis evaluates how walk in clients of a local Citizens Advice Bureau (CAB) who come to get social service information, obtain information online using the Adviceguide website. The thesis presents two challenges: (i) knowing the users in a way that can help consider design solutions that are probably not in a typical designer’s standard repertoire of design patterns, and (ii) knowing what is the problem that needs to be addressed. It is not simply an issue of usability or the need for simpler language, but understanding that these low literacy users are very different from the high literacy users. These low literacy users need this information to solve their day-to-day problems and are likely to be less successful in doing so. By providing an information architecture that permits them of a reasoning space and context, while supporting less abstract skills by visualized information in an unconventional way. The above challenges leave us with these research questions to address: what is the basis of such a design, how can these designs be incorporated into existing non-traditional interface proof of concept and finally how can these designs be evaluated

Digital Dialogism: Space, Time, and Queerness in Video Games

Video games are multimodal pieces of media; they communicate meaning through many layers of signification including aural, visual, narrative, mechanical, and more. To understand the ways that games communicate meaning and influence interpretation, it is crucial to not just examine the various layers of game modalities, but the ways that those layers communicate with each other as multimodal objects. By adapting Mikhail Bakhtin’s literary and language theory of dialogism (1981), this dissertation argues that because games are multimodal, they have layers of different “voices” that communicate ideas about the game to its players. These dialogic multimodalities “speak” different meanings to players, who then transform their interaction with these multimodalities into a narrative whole. Joining queer theory, narrative theory, and game studies, this dissertation examines one of the most successful video game titles to date, The Elder Scrolls V: Skyrim (2011), which in addition to its widespread popularity, has also been identified by white supremacist groups as a game that supports white nationalist causes. Through a dialogic analysis of the multimodalities of temporal and spatial representation within the game, this dissertation identifies narrative, genre, gameplay, and representational elements of Skyrim that support white nationalist play while also silencing potential anti-racist perspectives within the game. Chapter 1, Chapter 2, and Chapter 3 work together towards a functional version of dialogism for the study of games, proving its relevance, formalizing the changes I make to the original theory, and indicating how important dialogic readings can be. Chapter 4 argues that the construction of timespace of Skyrim follows a chronotope of domination, where the player’s use of and engagement with the game are devoted to the control of time and space. Chapter 5 examines player self-narration and embodiment in queered space, looking at how spaces communicate to players, and Chapter 6 makes the case that player use and manipulation of queered time in the game encourages players to understand and interact with Skyrim in particular ways. Together, these chapters suggest that the ways players are oriented to play Skyrim, based on its spaces and temporalities, points players towards interpretations of the game that normalize and uphold instances of white supremacy based on narrative, interactive, and mechanical means

Language and image in the film Sex and the city: a multimodal investigation of the representation of women

Tese (doutorado) - Universidade Federal de Santa Catarina, Centro de Comunicação e Expressão. Programa de Pós-Graduação em Letras/Inglês e Literatura Correspondente.Over the past decade, an increasing number of studies have explored the structure and role of multimodal texts in contemporary society (Böhlke, 2008; Ferreira, 2011; Heberle & Meurer, 2007; Iedema, 2001; Kress & van Leeuwen, 1996; 2006; Thibault, 2000). Following the early focus on still images, more recent research has addressed the dynamic text (O'Halloran, 2004). In this context, the present research investigates the identities of women (Butler, 1990, 1993, 2004; Benwell & Stokoe, 2006) construed in the first film Sex and the City (2008) in terms of both verbal language (Halliday, 1994; Halliday & Matthiessen, 2004; Martin, Matthiessen & Painter, 2010) and the dynamic image (Bateman, 2007, 2009; Bednarek, 2010; Böhlke, 2008, Iedema, 2001; O'Halloran, 2004; Thibault, 2000; Tseng, 2009; van Leeuwen, 1991, 1999) as well as their intermodal complementarity (Painter & Martin, in press), focusing on coupling and commitment (Martin, 2008a, 2008b, 2010). Verbal language is addressed in terms of ideational meanings by means of transitivity analysis (Halliday & Matthiessen, 2004) and the analysis of the dynam image is carried out by following the multi-level procedures proposed by Baldry and Thibault (2005). Regarding the film text, overall results show that it affords more meanings than the still images in systemic functional terms, which has contributed to more effective intermodal complementarity. Considering the identities of women construed, data analysis has demonstrated that the coupling of meanings committed suggests that (young) women's main pursuit in life is fashion labels and heteronormative love. Additionally, overall results also reveal that women are mostly involved in processes of 'action' as dynamic participants, which highlights the space in the filmic text for the 'doings', 'happenings'' and 'behaviors' in which women take on the active role. However, as in Bezerra (2008), the discourse analysis (Fairclough, 2003; van Leeuwen, 2008) of women's social actions has shown that they are considerably constrained to the domestic, nonspecialized field (Martin, 1992). These results seem to confirm the role of the media in maintaining dominant and ideologically invested representations of women (Bhabha, 1992), which need to be continuously challenged, since identities should always be seen as unstable and impermanent (Bauman, 2004).Durante a última década, um número crescente de estudos têm explorado a estrutura e o papel de textos multimodais na sociedade contemporânea (Böhlke, 2008; Ferreira, 2011; Heberle & Meurer, 2007; Iedema, 2001; Kress & van Leeuwen, 1996; 2006; Thibault, 2000). Depois do foco inicial em imagens estáticas, pesquisas mais recentes têm abordado o texto dinâmico (O'Halloran, 2004). Neste contexto, a presente pesquisa investiga as identidades das mulheres (Butler, 1990, 1993, 2004; Benwell & Stokoe, 2006) no primeiro filme Sex and the City (2008) em termos da linguagem verbal (Halliday, 1994; Halliday & Matthiessen, 2004; Martin, Matthiessen & Painter, 2010) e da imagem dinâmica (Bateman, 2007, 2009; Bednarek, 2010; Böhlke, 2008, Iedema, 2001; O'Halloran, 2004; Thibault, 2000; Tseng , 2009; van Leeuwen, 1991, 1999), bem como da sua complementaridade intermodal (Painter & Martin, no prelo), concentrando-se no acoplamento e na calibragem de significados (Martin, 2008a, 2008b, 2010). A linguagem verbal é investigada quanto aos significados ideacionais por meio de análise de transitividade (Halliday e Matthiessen, 2004), ao passo que a análise da imagem dinâmica segue os procedimentos propostos por Baldry e Thibault (2005). Em relação ao texto fílmico, resultados gerais mostram que ele constrói mais tipos de significados do que as imagens estáticas, o que contribuiu para uma complementaridade intermodal mais eficaz. Considerando-se as identidades das mulheres, o acoplamento de significados cometidos sugere que mulheres (jovens) focam suas buscas na moda (grifes) e no amor heteronormativo. Além disso, resultados gerais revelam que as mulheres estão principalmente envolvidas em processos de 'ação' como participantes dinâmicos, o que evidencia o espaço no texto fílmico para os 'fazeres', 'acontecimentos' e 'comportamentos' nos quais as mulheres assumem papel ativo. No entanto, como em Bezerra (2008), a análise do discurso (Fairclough, 2003; van Leeuwen, 2008) das ações sociais das mulheres mostrou que elas estão consideravelmente restritas à esfera doméstica, não especializada (Martin, 1992). Estes resultados parecem confirmar o papel da mídia na manutenção de representações dominantes e ideologicamente investidas das mulheres (Bhabha, 1992), que precisam ser continuamente desafiadas, já que as identidades devem ser sempre vistas como instáveis e impermanentes (Bauman, 2004)