Recuperação multimodal e interativa de informação orientada por diversidade

Orientador: Ricardo da Silva TorresTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Os métodos de Recuperação da Informação, especialmente considerando-se dados multimídia, evoluíram para a integração de múltiplas fontes de evidência na análise de relevância de itens em uma tarefa de busca. Neste contexto, para atenuar a distância semântica entre as propriedades de baixo nível extraídas do conteúdo dos objetos digitais e os conceitos semânticos de alto nível (objetos, categorias, etc.) e tornar estes sistemas adaptativos às diferentes necessidades dos usuários, modelos interativos que consideram o usuário mais próximo do processo de recuperação têm sido propostos, permitindo a sua interação com o sistema, principalmente por meio da realimentação de relevância implícita ou explícita. Analogamente, a promoção de diversidade surgiu como uma alternativa para lidar com consultas ambíguas ou incompletas. Adicionalmente, muitos trabalhos têm tratado a ideia de minimização do esforço requerido do usuário em fornecer julgamentos de relevância, à medida que mantém níveis aceitáveis de eficácia. Esta tese aborda, propõe e analisa experimentalmente métodos de recuperação da informação interativos e multimodais orientados por diversidade. Este trabalho aborda de forma abrangente a literatura acerca da recuperação interativa da informação e discute sobre os avanços recentes, os grandes desafios de pesquisa e oportunidades promissoras de trabalho. Nós propusemos e avaliamos dois métodos de aprimoramento do balanço entre relevância e diversidade, os quais integram múltiplas informações de imagens, tais como: propriedades visuais, metadados textuais, informação geográfica e descritores de credibilidade dos usuários. Por sua vez, como integração de técnicas de recuperação interativa e de promoção de diversidade, visando maximizar a cobertura de múltiplas interpretações/aspectos de busca e acelerar a transferência de informação entre o usuário e o sistema, nós propusemos e avaliamos um método multimodal de aprendizado para ranqueamento utilizando realimentação de relevância sobre resultados diversificados. Nossa análise experimental mostra que o uso conjunto de múltiplas fontes de informação teve impacto positivo nos algoritmos de balanceamento entre relevância e diversidade. Estes resultados sugerem que a integração de filtragem e re-ranqueamento multimodais é eficaz para o aumento da relevância dos resultados e também como mecanismo de potencialização dos métodos de diversificação. Além disso, com uma análise experimental minuciosa, nós investigamos várias questões de pesquisa relacionadas à possibilidade de aumento da diversidade dos resultados e a manutenção ou até mesmo melhoria da sua relevância em sessões interativas. Adicionalmente, nós analisamos como o esforço em diversificar afeta os resultados gerais de uma sessão de busca e como diferentes abordagens de diversificação se comportam para diferentes modalidades de dados. Analisando a eficácia geral e também em cada iteração de realimentação de relevância, nós mostramos que introduzir diversidade nos resultados pode prejudicar resultados iniciais, enquanto que aumenta significativamente a eficácia geral em uma sessão de busca, considerando-se não apenas a relevância e diversidade geral, mas também o quão cedo o usuário é exposto ao mesmo montante de itens relevantes e nível de diversidadeAbstract: Information retrieval methods, especially considering multimedia data, have evolved towards the integration of multiple sources of evidence in the analysis of the relevance of items considering a given user search task. In this context, for attenuating the semantic gap between low-level features extracted from the content of the digital objects and high-level semantic concepts (objects, categories, etc.) and making the systems adaptive to different user needs, interactive models have brought the user closer to the retrieval loop allowing user-system interaction mainly through implicit or explicit relevance feedback. Analogously, diversity promotion has emerged as an alternative for tackling ambiguous or underspecified queries. Additionally, several works have addressed the issue of minimizing the required user effort on providing relevance assessments while keeping an acceptable overall effectiveness. This thesis discusses, proposes, and experimentally analyzes multimodal and interactive diversity-oriented information retrieval methods. This work, comprehensively covers the interactive information retrieval literature and also discusses about recent advances, the great research challenges, and promising research opportunities. We have proposed and evaluated two relevance-diversity trade-off enhancement work-flows, which integrate multiple information from images, such as: visual features, textual metadata, geographic information, and user credibility descriptors. In turn, as an integration of interactive retrieval and diversity promotion techniques, for maximizing the coverage of multiple query interpretations/aspects and speeding up the information transfer between the user and the system, we have proposed and evaluated a multimodal learning-to-rank method trained with relevance feedback over diversified results. Our experimental analysis shows that the joint usage of multiple information sources positively impacted the relevance-diversity balancing algorithms. Our results also suggest that the integration of multimodal-relevance-based filtering and reranking was effective on improving result relevance and also boosted diversity promotion methods. Beyond it, with a thorough experimental analysis we have investigated several research questions related to the possibility of improving result diversity and keeping or even improving relevance in interactive search sessions. Moreover, we analyze how much the diversification effort affects overall search session results and how different diversification approaches behave for the different data modalities. By analyzing the overall and per feedback iteration effectiveness, we show that introducing diversity may harm initial results whereas it significantly enhances the overall session effectiveness not only considering the relevance and diversity, but also how early the user is exposed to the same amount of relevant items and diversityDoutoradoCiência da ComputaçãoDoutor em Ciência da ComputaçãoP-4388/2010140977/2012-0CAPESCNP

A Review of the "Digital Turn" in the New Literacy Studies

Digital communication has transformed literacy practices and assumed great importance in the functioning of workplace, recreational, and community contexts. This article reviews a decade of empirical work of the New Literacy Studies, identifying the shift toward research of digital literacy applications. The article engages with the central theoretical, methodological, and pragmatic challenges in the tradition of New Literacy Studies, while highlighting the distinctive trends in the digital strand. It identifies common patterns across new literacy practices through cross-comparisons of ethnographic research in digital media environments. It examines ways in which this research is taking into account power and pedagogy in normative contexts of literacy learning using the new media. Recommendations are given to strengthen the links between New Literacy Studies research and literacy curriculum, assessment, and accountability in the 21st century

Enfrentando os desafios do design de textos multimodais para promover a pedagogia das multiliteracias

This article aims to add to the understanding of the challenges involved in designing digital texts to promote multiliteracies pedagogy. A multiliteracies approach calls for multimodal meaning-making and cultural diversity to be integrated into new school curriculum content, and accordingly, we analyse an interactive children’s story app, named Mobeybou in Brazil. The re-search question addressed was: what can we learn about the design of multimodal texts aimed at promoting intercultural learning from the design of this story app? The app incorporates tangible and digital storytelling materials to promote intercultural skills among young children, focusing specifically on Brazil. Mobeybou in Brazil was studied to characterize the design of its multimodal representations of meaning, using categories from the grammar of storytelling and multimodal meaning-making, particularly those concerned with representing the experiential diversity and personal positioning of the app users. The findings provide evidence of the complexity involved in designing multimodal texts to meet the challenges of promoting multiliteracies pedagogy, highlighting the urgent need to narrow the interface between research undertaken in education, semiotics, and digital media design. The article concludes by identifying the study’s limitations and some future developments.Este artigo pretende contribuir para a compreensão dos desafios colocados pelo design de textos digitais especificamente destinados a promover a pedagogia das multiliteracias. Esta abordagem pedagógica defende que a compreensão e produção de significados multimodais e a diversidade cultural são conteúdos incontornáveis no novo currículo escolar. Apresentamos o estudo de uma story app multimodal interativa, intitulada Mobeybou no Brasil, desenvolvida para dar resposta à seguinte questão de investigação: o que se pode aprender sobre o design de textos multimodais destinados a promover a aprendizagem intercultural a partir do design desta story app? Na nossa análise, utilizamos categorias da gramática da narrativa e da representação multi-modal incidentes na representação da diversidade experiencial e do posicionamento pessoal dos utilizadores. Os resultados sugerem a complexidade envolvida no design de textos multimodais que respondem aos desafios da pedagogia das multiliteracias, salientando a necessidade urgen-te do estreitamento da colaboração entre a investigação realizada nas áreas da educação, da se-miótica e do design de média digitais. Por fim, identificamos as limitações do estudo, apontando alguns desenvolvimentos futuros.This research is funded by the FCT within the scope of the project PTDC/CEDEDG/ 0736/2021, by CIEd – Research Centre on Education, Institute of Education, University of Minho, projects UIDB/01661/2020 and UIDP/01661/2020, through national funds of FCT/MCTES-PT, and by the LARSyS - FCT Plurianual funding 2020-202

A Critical Approach to Multiliteracy: Automates Intelligents

In this paper we present and analyse a website with a complex rhyzomatic structure in connection with the results of a cybertask in which students were asked to read various information sources by navigating a range of websites. The results and discussion include issues such as: a) culture of learning and students’ task representation; b) possible relationships between learning styles and ways of navigating and managing information to solve a task; c) criteria students use to evaluate their navigation practice

Using Film as a Multimodal Text in the Language Classroom

Film is a powerful medium for language acquisition; Not only does viewing films allow learners to experience language used in various real-life contexts, but the medium of film itself is a form of communication. Through the active viewing of films, students can explore how people with diverse backgrounds, accents, and cultures communicate using multiple modes such as gestures, images, and sounds. When learners are familiar with the techniques used by filmmakers to convey meaning, they can use the medium of film themselves to tell their stories and share their perspectives through filmmaking projects. Such projects can be collaborative and engaging and build on the four skills as well 21st century skills. This thesis establishes a theoretical foundation by introducing key theories related to the use of film for language learning and how they can be beneficial if implemented effectively. The author then examines teacher preparedness for implementing film viewing and filmmaking in the classroom, first by reviewing the literature of studies that have been done, then goes on to discuss the results and implications of original research conducted by the author of teachers and their experiences and perceptions of using film in their classrooms. The paper concludes with the lessons learned from the design and delivery of multiple iterations of a professional development workshop for educators interested in using film

Interpersonal Meaning in Textbooks for Teaching English as a Foreign Language in China: A Multimodal Approach

There is increasing awareness among linguists that discourse analysis inevitably involves analyses of meanings arising from the combination of multiple modes of communication. The evolving multimodal pedagogic environment for teaching English as a foreign language (henceforth EFL), among other communicative contexts, calls for a social, semiotic, and linguistic explanation. Situated within the theoretical landscape of social semiotics and in the pedagogic context of EFL education, the present study aims to elucidate how linguistic and visual semiotic resources are co-deployed to construe interpersonal meaning in multimodal textbooks. The data drawn upon are eighteen EFL textbooks for primary and secondary schooling, published by People’s Education Press between 2002 and 2006. The research design consists of three complementary sub-studies. First, it investigates the ways in which the semantic regions of ENGAGEMENT and GRADUATION can be modelled in multimodal texts, with special reference to the interplay of voices in textbook discourse. The second sub-study analyzes how verbal and visual semiotic resources are co-deployed to construe the ‘emotion and attitude’ goal highlighted in curriculum standards, with a particular focus on verbiage-image relations. Third, it extends the linguistic concept ‘modality’ to multimodal discourse, exploring coding orientation in texts for different educational contexts and between different constituent genres. The main findings of this thesis are as follows: (1) A range of multimodal resources (i.e. labelling, dialogue balloon, jointly-constructed text, illustration and highlighting) are identified as enabling editor voice to negotiate meanings with reader voice and character voice. It is found that the way in which an ENGAGEMENT value can be scaled is strongly associated with the intrinsic property of the given multimodal resource. The interaction between multiple voices is closely related to contact, social distance, and point of view. (2) It is shown that images play an essential role in realizing attitudinal meanings. Together with verbal APPRAISAL resources, visual semiotic features work to position the readers in ways that align them to set pedagogic goals, guiding them in completing jointly-constructed texts. Moreover, an attitudinal shift from an emotional release to a more institutionalized type of evaluation can be identified as students advance through the school years. (3) It is argued that what counts as real in multimodal texts is socially defined and specific to a given communicative context. The nature of pedagogic discourse should be taken into account when visual displays are produced for pedagogic materials. The implications of this study include both theoretical and pedagogic aspects。Theoretically it adapts and extends APPRAISAL analysis to multimodal discourse, exploring the intersemiotic complementarity and co-instantiation in construing global evaluative stance. This semiotic exploration, in return, suggests ways in which discourse analysis may help textbook users better understand and interpret the multimodal features. With the affordances as well as limitations of semiotic resources made explicit, we may have one step further towards a comprehensive and critical understanding of multimodal construal of interpersonal meaning in pedagogic materials