Making modality: transmodal composing in a digital media studio.

The multiple media that exist for communication have historically been theorized as possessing different available means for persuasion and meaning-making. The exigence of these means has been the object of theoretical debate that ranges from cultural studies, language studies, semiology, and philosophies of the mind. This dissertation contributes to such debates by sharing the results of an ethnographically informed study of multimedia composing in a digital media studio. Drawing from Cultural Historical Activity Theory and theories of enactive perception, I analyze the organizational and infrastructural design of a media studio as well as the activity of composer/designers working in said studio. Throughout this analysis I find that implicit in the organization and infrastructure of the media studio is an ethos of conceptualizing communication technology as a legitimizing force. Such an ethos is troubled by my analysis of composer/designers working in the studio, whose activities do not seek outside legitimization but instead contribute to the media milieu. Following these analyses, I conclude that media’s means for persuasion and meaning-making emerge from local practices of communication and design. Finally, I provide a framework for studying the emergence of such means

Translinguality, transmodality, and difference : exploring dispositions and change in language and learning.

This collaborative piece explores the potential synergy arising from the confluence of two growing areas of research, teaching, and practice in composition (broadly defined): multi- (or trans-)modality, and trans- (or multi-) linguality

Advancing performability in playable media : a simulation-based interface as a dynamic score

When designing playable media with non-game orientation, alternative play scenarios to gameplay scenarios must be accompanied by alternative mechanics to game mechanics. Problems of designing playable media with non-game orientation are stated as the problems of designing a platform for creative explorations and creative expressions. For such design problems, two requirements are articulated: 1) play state transitions must be dynamic in non-trivial ways in order to achieve a significant level of engagement, and 2) pathways for players’ experience from exploration to expression must be provided. The transformative pathway from creative exploration to creative expression is analogous to pathways for game players’ skill acquisition in gameplay. The paper first describes a concept of simulation-based interface, and then binds that concept with the concept of dynamic score. The former partially accounts for the first requirement, the latter the second requirement. The paper describes the prototype and realization of the two concepts’ binding. “Score” is here defined as a representation of cue organization through a transmodal abstraction. A simulation based interface is presented with swarm mechanics and its function as a dynamic score is demonstrated with an interactive musical composition and performance

Translating novel findings of perceptual-motor codes into the neuro-rehabilitation of movement disorders

The bidirectional flow of perceptual and motor information has recently proven useful as rehabilitative tool for re-building motor memories. We analyzed how the visual-motor approach has been successfully applied in neurorehabilitation, leading to surprisingly rapid and effective improvements in action execution. We proposed that the contribution of multiple sensory channels during treatment enables individuals to predict and optimize motor behavior, having a greater effect than visual input alone. We explored how the state-of-the-art neuroscience techniques show direct evidence that employment of visual-motor approach leads to increased motor cortex excitability and synaptic and cortical map plasticity. This super-additive response to multimodal stimulation may maximize neural plasticity, potentiating the effect of conventional treatment, and will be a valuable approach when it comes to advances in innovative methodologies

Investigating the use of multimodal screencasts to teach disciplinary concepts in higher education.

This research study explores the use of multimodal lecture screencasts to teach disciplinary concepts in an Irish higher education (HE) context. It builds on an Inquiry Graphics (IG) framework, extending it into a multimodal inquiry framework (MMI) to examine screencasts crafted by lecturers to teach key concepts within their discipline. Multimodality is a widely recognised and applied approach that observes communication as including language but also encompassing other modes of communication, such as sound, image, touch, gesture, feeling, etc. However, studies that provide an in-depth examination of multimodality in teaching and learning in higher education are still scarce. The proposed MMI framework provides a lens to explore graphic-pictorial, linguistic, aural, and spatial- design modes and analyse the semiotic organisation of lecturers’ screencasts, to understand how multimodality relates to teaching and reveals lecturers’ semiotic choices. Qualitative IG elicitation interviews were conducted with 16 HE lecturers from a range of disciplines, where the IG framework provided an analytical opportunity to co-examine the underlying assumptions about how content is presented multimodally. An awareness of the semiotic dimensions of each mode was uncovered, along with structures within the lecturers’ sociocultural context which influenced their decision-making. The use of the MMI framework revealed the semiotic purpose of the graphic-pictorial elements primarily as unprobed representations of the chosen concept. Linguistic choices helped explain the concept within the discipline, while prosodic features of the voice, along with music, were often used intentionally by the lecturer to highlight the relative importance of the elements on screen. The enactment of software features in the screencast design indicated lecturers’ embodied cognition through multimedia, along with digital fluency. The MMI framework may be a helpful teaching tool to support HE lecturers in video and multimedia analysis to unpack the plurality of conceptual representations within multimodal digital artefacts

Multi-form Visualisation: An approach to acousmatic composition

This practice-based doctoral research addresses a critical issue in acousmatic composition: the journey from the immaterial world of sonic imagination to the realisation of musical sound. This was an exploratory journey, where my personal sensibility for visual arts practice met my curiosity and profound interest in acousmatic music. Methodologically, the project approached acousmatic composition as an organic process, intertwining visual sensibilities and musical domains by offering a critical approach to the listening experience and to my compositional practice. A key metaphor used is that of the blank page as a space for multi-form visualisation, where gestures derived from sketching and other visual stimuli are used as guides and catalysts for the realisation of sound. In this approach, a process of deliberately blurring boundaries between real and imaginary realms affords a space to daydream to be moved by sounds, the flow of mental images, virtual sensations, and memory-images that one can associate with traces, dots, shapes or textures. This parallel allows me to find my way within the sonic realm, shaping sound materials and sequences that progressively define a musical structure. This space, which has no proper physical existence, invites sonic and visual perception and imagination to confront, destroy and renew each another, directing the music’s emergence through a feedback loop between the visual and the aural. A key conceptual tool in this practice is the notion of sensory qualia and a blend of phenomenological and ecological views of sound and bodily centered, internally registered responses. By focusing on qualitative sensations derived from drawing, painting and sensations of motion in the natural world, parallels with the sonic imagination are stimulated. The graphical expression of gestures deployed in space and time becomes a space of boundless, imaginative reflection of the composer’s sonic conceptions and expectations

Redesigning landscape architecture in higher education: a multimodal social semiotic approach

This investigation is a case study of landscape architectural design education in South Africa. Current forms of landscape architectural education are influenced by Global North perspectives and often, if not consciously, privilege particular ways of meaningmaking, and exclude or marginalise experiences or ways of knowing that are different. The aim of this research is to develop a landscape architectural pedagogy for diversity that fosters multiple perspectives and valorises resources that students bring to their learning environment, in order that students may both access and challenge the dominant landscape educational discourse. In grappling with these concerns, this research finds resonance with a multimodal social semiotic approach. Instead of labelling students as (in) competent or (under)prepared, a multimodal social semiotic approach emphasises the interest, agency and resourcefulness of the student as meaning-maker. The research thus reframes landscape architectural design processes through a multimodal social semiotic lens, providing new insights and clarity to these processes. The approach foregrounds interpersonal and social meanings of space and, to some extent, challenges traditional landscape architectural design practices that tend to value compositional and conceptual meanings. The methodology centers around a spatial model project in the second half of a first-year landscape architectural design studio subject. The data includes students' texts and their presentations. The research develops a methodological framework that outlines a range of ideational, interpersonal and textual meaningpotentials of landscape spatial and visual texts and applies this framework to the analysis of students' 2D and 3D texts. Through careful analysis of students' design trajectories, this research uncovers the types of resources students draw on, including semiotic, experiential, social, interactive and pedagogical resources. The analysis shows that students' transformation of resources results in innovative spatial designs, and expands on what and how landscape spaces can mean. Through the investigation, tenets for a multimodal pedagogy for diversity are developed: recognition of the rich and diverse resources students bring to their learning environment; acknowledgment that these resources are apt ‘precedent' for landscape architectural design processes; and explicit attention to multimodal moments and activities that may prompt re-(inner) conceptualisation in design trajectories. This pedagogical approach begins to address past educational imbalances and inequalities, and ensures that diverse, Global South perspectives contribute to the production of knowledge

Language, literacy and technology: embodied peer-interaction and collaborative writing in an ESOL classroom

This study focuses on digital literacies, and real-time multimodal design, within the context of migrant adult learners in the UK. It seeks to understand the frameworks of peer-interaction when second-language learners are paired at a computer and how they negotiate second-language writing. In this research, pairs of students sharing the same language were tasked with an environmental project which included the digital design of an image, designing a four-page booklet using Publisher, a website and to produce all of these using English as a second language. The process was videoed across a threehour classroom session with four pairs of learners: Kurdish, Polish, French and Arabic. New literacies, embodied peer-interaction and second-language writing are the primary fields informing this research. The outcomes of the research are: (1) a methodology is developed for the collection and analysis of multimodal data when learners collaborate at a computer; (2) the field of new literacies is extended through an analysis of the design-process, as opposed to product-analysis; (3) a peer-interaction framework is presented which broadens our understanding of classroom interaction, including linguistic, paralinguistic and mediating resources when learners share technology; (4) the field of second-language writing is extended through an analysis of peer-writing with technology. The research concludes with a peer-interaction framework comprised of learner alignment and misalignment across language, literacy and technology. Spoken and written language goes through an iterative cycle of transformation. The central finding from the research is the naming and defining of transmodal talk within a peer-interaction framework. The sequentiality of this process has common features across all the pairs of learners. Transmodal talk is presented to identify the fluid process of transposing off-screen dialogue to on-screen text. They both shape and mediate each other through temporal mapping and polyvocality