The Ideational Meanings of Multimodal Texts Presented in an EFL Textbook

This study explores the ideational meanings of multimodal texts in an EFL textbook entitled “Bahasa Inggris: When English Rings a Bell”. This study aims to describe the meanings of visual and verbal modes from ideational aspects. The visual modes were analyzed by using Visual Grammar Design framework (Kress van Leeuwen, 2006). Meanwhile, the verbal modes were analyzed using Systemic Functional Grammar theory focusing on the Transitivity system (Halliday, 1994). The ideational aspects of this study focus on components like participants, processes and circumstances. This study applied Systemic Functional Multimodal Discourse Analysis (henceforth SF-MDA) methodology. The findings of this study found that the dominant process types used in realizing the ideational meaning of the verbal text is relational process. Meanwhile, the ideational meaning of visual text is realized mostly through speech process. This study showed that the presentation of multimodal texts in the EFL textbook is relevant to the teaching and learning activity for the Junior high school students. Keywords: multimodality, ideational meaning, systemic functional grammar, multimodal analysi

The University of Sussex-Huawei locomotion and transportation dataset for multimodal analytics with mobile devices

Scientific advances build on reproducible research which need publicly available benchmark datasets. The computer vision and speech recognition communities have led the way in establishing benchmark datasets. There are much less datasets available in mobile computing, especially for rich locomotion and transportation analytics. This paper presents a highly versatile and precisely annotated large-scale dataset of smartphone sensor data for multimodal locomotion and transportation analytics of mobile users. The dataset comprises 7 months of measurements, collected from all sensors of 4 smartphones carried at typical body locations, including the images of a body-worn camera, while 3 participants used 8 different modes of transportation in the southeast of the United Kingdom, including in London. In total 28 context labels were annotated, including transportation mode, participant’s posture, inside/outside location, road conditions, traffic conditions, presence in tunnels, social interactions, and having meals. The total amount of collected data exceed 950 GB of sensor data, which corresponds to 2812 hours of labelled data and 17562 km of traveled distance. We present how we set up the data collection, including the equipment used and the experimental protocol. We discuss the dataset, including the data curation process, the analysis of the annotations and of the sensor data. We discuss the challenges encountered and present the lessons learned and some of the best practices we developed to ensure high quality data collection and annotation. We discuss the potential applications which can be developed using this large-scale dataset. In particular, we present how a machine-learning system can use this dataset to automatically recognize modes of transportations. Many other research questions related to transportation analytics, activity recognition, radio signal propagation and mobility modelling can be adressed through this dataset. The full dataset is being made available to the community, and a thorough preview is already publishe

Wearable Computing for Health and Fitness: Exploring the Relationship between Data and Human Behaviour

Health and fitness wearable technology has recently advanced, making it easier for an individual to monitor their behaviours. Previously self generated data interacts with the user to motivate positive behaviour change, but issues arise when relating this to long term mention of wearable devices. Previous studies within this area are discussed. We also consider a new approach where data is used to support instead of motivate, through monitoring and logging to encourage reflection. Based on issues highlighted, we then make recommendations on the direction in which future work could be most beneficial

Language and image in the film Sex and the city: a multimodal investigation of the representation of women

Tese (doutorado) - Universidade Federal de Santa Catarina, Centro de Comunicação e Expressão. Programa de Pós-Graduação em Letras/Inglês e Literatura Correspondente.Over the past decade, an increasing number of studies have explored the structure and role of multimodal texts in contemporary society (Böhlke, 2008; Ferreira, 2011; Heberle & Meurer, 2007; Iedema, 2001; Kress & van Leeuwen, 1996; 2006; Thibault, 2000). Following the early focus on still images, more recent research has addressed the dynamic text (O'Halloran, 2004). In this context, the present research investigates the identities of women (Butler, 1990, 1993, 2004; Benwell & Stokoe, 2006) construed in the first film Sex and the City (2008) in terms of both verbal language (Halliday, 1994; Halliday & Matthiessen, 2004; Martin, Matthiessen & Painter, 2010) and the dynamic image (Bateman, 2007, 2009; Bednarek, 2010; Böhlke, 2008, Iedema, 2001; O'Halloran, 2004; Thibault, 2000; Tseng, 2009; van Leeuwen, 1991, 1999) as well as their intermodal complementarity (Painter & Martin, in press), focusing on coupling and commitment (Martin, 2008a, 2008b, 2010). Verbal language is addressed in terms of ideational meanings by means of transitivity analysis (Halliday & Matthiessen, 2004) and the analysis of the dynam image is carried out by following the multi-level procedures proposed by Baldry and Thibault (2005). Regarding the film text, overall results show that it affords more meanings than the still images in systemic functional terms, which has contributed to more effective intermodal complementarity. Considering the identities of women construed, data analysis has demonstrated that the coupling of meanings committed suggests that (young) women's main pursuit in life is fashion labels and heteronormative love. Additionally, overall results also reveal that women are mostly involved in processes of 'action' as dynamic participants, which highlights the space in the filmic text for the 'doings', 'happenings'' and 'behaviors' in which women take on the active role. However, as in Bezerra (2008), the discourse analysis (Fairclough, 2003; van Leeuwen, 2008) of women's social actions has shown that they are considerably constrained to the domestic, nonspecialized field (Martin, 1992). These results seem to confirm the role of the media in maintaining dominant and ideologically invested representations of women (Bhabha, 1992), which need to be continuously challenged, since identities should always be seen as unstable and impermanent (Bauman, 2004).Durante a última década, um número crescente de estudos têm explorado a estrutura e o papel de textos multimodais na sociedade contemporânea (Böhlke, 2008; Ferreira, 2011; Heberle & Meurer, 2007; Iedema, 2001; Kress & van Leeuwen, 1996; 2006; Thibault, 2000). Depois do foco inicial em imagens estáticas, pesquisas mais recentes têm abordado o texto dinâmico (O'Halloran, 2004). Neste contexto, a presente pesquisa investiga as identidades das mulheres (Butler, 1990, 1993, 2004; Benwell & Stokoe, 2006) no primeiro filme Sex and the City (2008) em termos da linguagem verbal (Halliday, 1994; Halliday & Matthiessen, 2004; Martin, Matthiessen & Painter, 2010) e da imagem dinâmica (Bateman, 2007, 2009; Bednarek, 2010; Böhlke, 2008, Iedema, 2001; O'Halloran, 2004; Thibault, 2000; Tseng , 2009; van Leeuwen, 1991, 1999), bem como da sua complementaridade intermodal (Painter & Martin, no prelo), concentrando-se no acoplamento e na calibragem de significados (Martin, 2008a, 2008b, 2010). A linguagem verbal é investigada quanto aos significados ideacionais por meio de análise de transitividade (Halliday e Matthiessen, 2004), ao passo que a análise da imagem dinâmica segue os procedimentos propostos por Baldry e Thibault (2005). Em relação ao texto fílmico, resultados gerais mostram que ele constrói mais tipos de significados do que as imagens estáticas, o que contribuiu para uma complementaridade intermodal mais eficaz. Considerando-se as identidades das mulheres, o acoplamento de significados cometidos sugere que mulheres (jovens) focam suas buscas na moda (grifes) e no amor heteronormativo. Além disso, resultados gerais revelam que as mulheres estão principalmente envolvidas em processos de 'ação' como participantes dinâmicos, o que evidencia o espaço no texto fílmico para os 'fazeres', 'acontecimentos' e 'comportamentos' nos quais as mulheres assumem papel ativo. No entanto, como em Bezerra (2008), a análise do discurso (Fairclough, 2003; van Leeuwen, 2008) das ações sociais das mulheres mostrou que elas estão consideravelmente restritas à esfera doméstica, não especializada (Martin, 1992). Estes resultados parecem confirmar o papel da mídia na manutenção de representações dominantes e ideologicamente investidas das mulheres (Bhabha, 1992), que precisam ser continuamente desafiadas, já que as identidades devem ser sempre vistas como instáveis e impermanentes (Bauman, 2004)

Detecting Eating Episodes with an Ear-mounted Sensor

In this paper, we propose Auracle, a wearable earpiece that can automatically recognize eating behavior. More specifically, in free-living conditions, we can recognize when and for how long a person is eating. Using an off-the-shelf contact microphone placed behind the ear, Auracle captures the sound of a person chewing as it passes through the bone and tissue of the head. This audio data is then processed by a custom analog/digital circuit board. To ensure reliable (yet comfortable) contact between microphone and skin, all hardware components are incorporated into a 3D-printed behind-the-head framework. We collected field data with 14 participants for 32 hours in free-living conditions and additional eating data with 10 participants for 2 hours in a laboratory setting. We achieved accuracy exceeding 92.8% and F1 score exceeding 77.5% for eating detection. Moreover, Auracle successfully detected 20-24 eating episodes (depending on the metrics) out of 26 in free-living conditions. We demonstrate that our custom device could sense, process, and classify audio data in real time. Additionally, we estimateAuracle can last 28.1 hours with a 110 mAh battery while communicating its observations of eating behavior to a smartphone over Bluetooth

Multimodality in VR: A survey

Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive, to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR, and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research, and demonstrate that multimodality plays a fundamental role in VR; enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer