From Narrative to Visual Narrative to Audiovisual Narrative: the Multimodal Discourse Theory Connection (Invited Talk)

Models of narrative have been proposed from many perspectives and most of these nowadays promote further the notion that narrative is a transmedial phenomenon: i.e., stories can be told making use of distinct and multiple forms of expressions. This raises a range of theoretical and practical questions, as well as rendering the task of providing computational models of narrative both more interesting and more challenging. Central to this endeavour are issues concerned with the potential mutual conditioning of narrative forms and the media employed. Methods are required for isolating narrative properties and mechanisms that may be generalised across media, while at the same time appropriately respecting differences in medial affordances. In this discussion paper I set out a corresponding approach to characterising narrative that draws on a fine-grained formal characterisation of multimodal discourse developed on the basis of both functional and formal linguistic models of discourse, generalised to the multimodal case. After briefly setting out the theoretical principles on which the account builds, I position narrative with respect to the framework and give an example of how audiovisual narratives such as film are accounted for. It will be suggested that a common anchoring in a well specified notion of discourse as an intrinsically multimodal phenomenon offers beneficial new angles on how narratives can be modelled, as well as establishing bridges between humanistic understandings of narrative and complementary computational accounts of narratives involving communicative goal-based planning

Introducing the diagrammatic semiotic mode

Peer reviewe

Semiotically-grounded distant viewing of diagrams : insights from two multimodal corpora

In this article, we argue for the benefits of combining large-scale analyses of visual materials currently pursued within digital humanities with insights from multimodality research, which is an emerging discipline that studies how human communication relies on appropriate combinations of expressive resources. We show that concepts developed within the field of multimodality research provide appropriate metadata schemes for various modes of expression in large corpora and datasets. We illustrate the proposed approach using a common mode of expression, diagrams, and analyse two recent multimodal diagram corpora using statistical and computational methods. Our results suggest that multimodally-motivated metadata schemes can provide a robust foundation for computational analyses of large corpora and datasets. Even if a corpus or dataset is not designed to support full-blown analyses of multimodal communication, our results imply that multimodality theory can still be used to impose tighter analytical control over a variety of visual materials.Peer reviewe

Introducing the diagrammatic semiotic mode

Peer reviewe

From data to patterns : on the role of models in empirical multimodality research

Peer reviewe