Multimodal Data Value Chain (M-DVC): A Conceptual Tool to Support the Development of Multimodal Learning Analytics Solutions

Producción CientíficaMultimodal Learning Analytics (MMLA) systems, understood as those that exploit multimodal evidence of learning to better model a learning situation, have not yet spread widely in educational practice. Their inherent technical complexity, and the lack of educational stakeholder involvement in their design, are among the hypothesized reasons for the slow uptake of this emergent field. To aid in the process of stakeholder communication and systematization leading to the specification of MMLA systems, this paper proposes a Multimodal Data Value Chain (M-DVC). This conceptual tool, derived from both the field of Big Data and the needs of MMLA scenarios, has been evaluated in terms of its usefulness for stakeholders, in three authentic case studies of MMLA systems currently under development. The results of our mixed-methods evaluation highlight the usefulness of the M-DVC to elicit unspoken assumptions or unclear data processing steps in the initial stages of development. The evaluation also revealed limitations of the M-DVC in terms of the technical terminology employed, and the need for more detailed contextual information to be included. These limitations also prompt potential improvements for the M-DVC, on the path towards clearer specification and communication within the multi-disciplinary teams needed to build educationally-meaningful MMLA solutions.Junta de Castilla y León (Project VA257P18)Ministerio de Ciencia, Innovación y Universidades (Project TIN2017-85179-C3-2-R)CEITER (grant agreements no. 669074

Context-aware multimodal learning analytics taxonomy

Analysis of learning interactions can happen for different purposes. As educational practices increasingly take place in hybrid settings, data from both spaces are needed. At the same time, to analyse and make sense of machine aggregated data afforded by Technology-Enhanced Learning (TEL) environments, contextual information is needed. We posit that human labelled (classroom observations) and automated observations (multimodal learning data) can enrich each other. Researchers have suggested learning design (LD) for contextualisation, the availability of which is often limited in authentic settings. This paper proposes a Context-aware MMLA Taxonomy, where we categorize systematic documentation and data collection within different research designs and scenarios, paying special attention to authentic classroom contexts. Finally, we discuss further research directions and challenges.Analysis of learning interactions can happen for different purposes. As educational practices increasingly take place in hybrid settings, data from both spaces are needed. At the same time, to analyse and make sense of machine aggregated data afforded by Technology-Enhanced Learning (TEL) environments, contextual information is needed. We posit that human labelled (classroom observations) and automated observations (multimodal learning data) can enrich each other. Researchers have suggested learning design (LD) for contextualisation, the availability of which is often limited in authentic settings. This paper proposes a Context-aware MMLA Taxonomy, where we categorize systematic documentation and data collection within different research designs and scenarios, paying special attention to authentic classroom contexts. Finally, we discuss further research directions and challenges

Collocated Collaboration Analytics: Principles and Dilemmas for Mining Multimodal Interaction Data

© 2019, Copyright © 2017 Taylor & Francis Group, LLC. Learning to collaborate effectively requires practice, awareness of group dynamics, and reflection; often it benefits from coaching by an expert facilitator. However, in physical spaces it is not always easy to provide teams with evidence to support collaboration. Emerging technology provides a promising opportunity to make collocated collaboration visible by harnessing data about interactions and then mining and visualizing it. These collocated collaboration analytics can help researchers, designers, and users to understand the complexity of collaboration and to find ways they can support collaboration. This article introduces and motivates a set of principles for mining collocated collaboration data and draws attention to trade-offs that may need to be negotiated en route. We integrate Data Science principles and techniques with the advances in interactive surface devices and sensing technologies. We draw on a 7-year research program that has involved the analysis of six group situations in collocated settings with more than 500 users and a variety of surface technologies, tasks, grouping structures, and domains. The contribution of the article includes the key insights and themes that we have identified and summarized in a set of principles and dilemmas that can inform design of future collocated collaboration analytics innovations

CIMLA: A Modular and Modifiable Data Preparation, Organization, and Fusion Infrastructure to Partially Support the Development of Context-aware MMLA Solutions

Multimodal Learning Analytics (MMLA) solutions aim to provide a more holistic picture of a learning situation by processing multimodal educational data. Considering contextual information of a learning situation is known to help in providing more relevant outputs to educational stakeholders. However, most of the MMLA solutions are still in prototyping phase and dealing with different dimensions of an authentic MMLA situation that involve multiple cross-disciplinary stakeholders like teachers, researchers, and developers. One of the reasons behind still being in prototyping phase of the development lifecycle is related to the challenges that software developers face at different levels in developing context-aware MMLA solutions. In this paper, we identify the requirements and propose a data infrastructure called CIMLA. It includes different data processing components following a standard data processing pipeline and considers contextual information following a data structure. It has been evaluated in three authentic MMLA scenarios involving different cross-disciplinary stakeholders following the Software Architecture Analysis Method. Its fitness was analyzed in each of the three scenarios and developers were interviewed to assess whether it meets functional and non-functional requirements. Results showed that CIMLA supports modularity in developing context-aware MMLA solutions and each of its modules can be reused with required modifications in the development of other solutions. In the future, the current involvement of a developer in customizing the configuration file to consider contextual information can be investigated

The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences

This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human annotated samples. The main contributions of this thesis are: a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks

Pathway to Future Symbiotic Creativity

This report presents a comprehensive view of our vision on the development path of the human-machine symbiotic art creation. We propose a classification of the creative system with a hierarchy of 5 classes, showing the pathway of creativity evolving from a mimic-human artist (Turing Artists) to a Machine artist in its own right. We begin with an overview of the limitations of the Turing Artists then focus on the top two-level systems, Machine Artists, emphasizing machine-human communication in art creation. In art creation, it is necessary for machines to understand humans' mental states, including desires, appreciation, and emotions, humans also need to understand machines' creative capabilities and limitations. The rapid development of immersive environment and further evolution into the new concept of metaverse enable symbiotic art creation through unprecedented flexibility of bi-directional communication between artists and art manifestation environments. By examining the latest sensor and XR technologies, we illustrate the novel way for art data collection to constitute the base of a new form of human-machine bidirectional communication and understanding in art creation. Based on such communication and understanding mechanisms, we propose a novel framework for building future Machine artists, which comes with the philosophy that a human-compatible AI system should be based on the "human-in-the-loop" principle rather than the traditional "end-to-end" dogma. By proposing a new form of inverse reinforcement learning model, we outline the platform design of machine artists, demonstrate its functions and showcase some examples of technologies we have developed. We also provide a systematic exposition of the ecosystem for AI-based symbiotic art form and community with an economic model built on NFT technology. Ethical issues for the development of machine artists are also discussed

TENSOR: retrieval and analysis of heterogeneous online content for terrorist activity recognition

The proliferation of terrorist generated content online is a cause for concern as it goes together with the rise of radicalisation and violent extremism. Law enforcement agencies (LEAs) need powerful platforms to help stem the influence of such content. This article showcases the TENSOR project which focusses on the early detection of online terrorist activities, radicalisation and recruitment. Operating under the H2020 Secure Societies Challenge, TENSOR aims to develop a terrorism intelligence platform for increasing the ability of LEAs to identify, gather and analyse terrorism-related online content. The mechanisms to tackle this challenge by bringing together LEAs, industry, research, and legal experts are presented

Detecting fake news and disinformation using artificial intelligence and machine learning to avoid supply chain disruptions

Peer reviewedPostprin

EFAR-MMLA: An evaluation framework to assess and report generalizability of machine learning models in MMLA

Producción CientíficaMultimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.Fondo Europeo de Desarrollo Regional - Agencia Nacional de Investigación (grants TIN2017-85179-C3-2-R and TIN2014-53199-C3-2-R)Fondo Europeo de Desarrollo Regional - Junta de Castilla y León (grant VA257P18)Comisión Europea (grant 588438-EPP-1- 2017-1-EL-EPPKA2-KA

BC4LLM: Trusted Artificial Intelligence When Blockchain Meets Large Language Models

In recent years, artificial intelligence (AI) and machine learning (ML) are reshaping society's production methods and productivity, and also changing the paradigm of scientific research. Among them, the AI language model represented by ChatGPT has made great progress. Such large language models (LLMs) serve people in the form of AI-generated content (AIGC) and are widely used in consulting, healthcare, and education. However, it is difficult to guarantee the authenticity and reliability of AIGC learning data. In addition, there are also hidden dangers of privacy disclosure in distributed AI training. Moreover, the content generated by LLMs is difficult to identify and trace, and it is difficult to cross-platform mutual recognition. The above information security issues in the coming era of AI powered by LLMs will be infinitely amplified and affect everyone's life. Therefore, we consider empowering LLMs using blockchain technology with superior security features to propose a vision for trusted AI. This paper mainly introduces the motivation and technical route of blockchain for LLM (BC4LLM), including reliable learning corpus, secure training process, and identifiable generated content. Meanwhile, this paper also reviews the potential applications and future challenges, especially in the frontier communication networks field, including network resource allocation, dynamic spectrum sharing, and semantic communication. Based on the above work combined and the prospect of blockchain and LLMs, it is expected to help the early realization of trusted AI and provide guidance for the academic community