Situating multimodal learning analytics

The digital age has introduced a host of new challenges and opportunities for the learning sciences community. These challenges and opportunities are particularly abundant in multimodal learning analytics (MMLA), a research methodology that aims to extend work from Educational Data Mining (EDM) and Learning Analytics (LA) to multimodal learning environments by treating multimodal data. Recognizing the short-term opportunities and longterm challenges will help develop proof cases and identify grand challenges that will help propel the field forward. To support the field's growth, we use this paper to describe several ways that MMLA can potentially advance learning sciences research and touch upon key challenges that researchers who utilize MMLA have encountered over the past few years

Proceedings of the 1st joint workshop on Smart Connected and Wearable Things 2016

These are the Proceedings of the 1st joint workshop on Smart Connected and Wearable Things (SCWT'2016, Co-located with IUI 2016). The SCWT workshop integrates the SmartObjects and IoWT workshops. It focusses on the advanced interactions with smart objects in the context of the Internet-of-Things (IoT), and on the increasing popularity of wearables as advanced means to facilitate such interactions

Interaction patterns in exploratory learning environments for mathematics: a sequential analysis of feedback and external representations in Chinese schools

Feedback in exploratory learning systems has been depicted as an important contributor to encourage exploration. However, few studies have explored learners’ interaction patterns associated with feedback and the use of external representations in exploratory learning environments. This study used Fractions Lab, an exploratory learning environment for mathematics, to facilitate children’s conceptual understanding of fractions in three Chinese schools. Students (n = 189) from six different classes were invited to use Fractions Lab, and 260,000 event logs were collected. Beyond demonstrating the overall efficacy of the approach, lag sequential analysis supported us in approaching a deeper understanding of patterns of interaction. The findings highlight that the design of three-levels of feedback (Socratic, guidance, and didactic-procedural feedback) played different roles in supporting students to use external representations to perform mathematical tasks in an exploratory learning environment. This study sheds light on how these interaction patterns might be applied to the Fractions Lab system in order to provide increasingly tailored support, based on cultural differences, to enhance students’ technology-mediated learning experiences

Error handling in multimodal voice-enabled interfaces of tour-guide robots using graphical models

Mobile service robots are going to play an increasing role in the society of humans. Voice-enabled interaction with service robots becomes very important, if such robots are to be deployed in real-world environments and accepted by the vast majority of potential human users. The research presented in this thesis addresses the problem of speech recognition integration in an interactive voice-enabled interface of a service robot, in particular a tour-guide robot. The task of a tour-guide robot is to engage visitors to mass exhibitions (users) in dialogue providing the services it is designed for (e.g. exhibit presentations) within a limited time. In managing tour-guide dialogues, extracting the user goal (intention) for requesting a particular service at each dialogue state is the key issue. In mass exhibition conditions speech recognition errors are inevitable because of noisy speech and uncooperative users of robots with no prior experience in robotics. They can jeopardize the user goal identification. Wrongly identified user goals can lead to communication failures. Therefore, to reduce the risk of such failures, methods for detecting and compensating for communication failures in human-robot dialogue are needed. During the short-term interaction with visitors, the interpretation of the user goal at each dialogue state can be improved by combining speech recognition in the speech modality with information from other available robot modalities. The methods presented in this thesis exploit probabilistic models for fusing information from speech and auxiliary modalities of the robot for user goal identification and communication failure detection. To compensate for the detected communication failures we investigate multimodal methods for recovery from communication failures. To model the process of modality fusion, taking into account the uncertainties in the information extracted from each input modality during human-robot interaction, we use the probabilistic framework of Bayesian networks. Bayesian networks are graphical models that represent a joint probability function over a set of random variables. They are used to model the dependencies among variables associated with the user goals, modality related events (e.g. the event of user presence that is inferred from the laser scanner modality of the robot), and observed modality features providing evidence in favor of these modality events. Bayesian networks are used to calculate posterior probabilities over the possible user goals at each dialogue state. These probabilities serve as a base in deciding if the user goal is valid, i.e. if it can be mapped into a tour-guide service (e.g. exhibit presentation) or is undefined – signaling a possible communication failure. The Bayesian network can be also used to elicit probabilities over the modality events revealing information about the possible cause for a communication failure. Introducing new user goal aspects (e.g. new modality events and related features) that provide auxiliary information for detecting communication failures makes the design process cumbersome, calling for a systematic approach in the Bayesian network modelling. Generally, introducing new variables for user goal identification in the Bayesian networks can lead to complex and computationally expensive models. In order to make the design process more systematic and modular, we adapt principles from the theory of grounding in human communication. When people communicate, they resolve understanding problems in a collaborative joint effort of providing evidence of common shared knowledge (grounding). We use Bayesian network topologies, tailored to limited computational resources, to model a state-based grounding model fusing information from three different input modalities (laser, video and speech) to infer possible grounding states. These grounding states are associated with modality events showing if the user is present in range for communication, if the user is attending to the interaction, whether the speech modality is reliable, and if the user goal is valid. The state-based grounding model is used to compute probabilities that intermediary grounding states have been reached. This serves as a base for detecting if the the user has reached the final grounding state, or wether a repair dialogue sequence is needed. In the case of a repair dialogue sequence, the tour-guide robot can exploit the multiple available modalities along with speech. For example, if the user has failed to reach the grounding state related to her/his presence in range for communication, the robot can use its move modality to search and attract the attention of the visitors. In the case when speech recognition is detected to be unreliable, the robot can offer the alternative use of the buttons modality in the repair sequence. Given the probability of each grounding state, and the dialogue sequence that can be executed in the next dialogue state, a tour-guide robot has different preferences on the possible dialogue continuation. If the possible dialogue sequences at each dialogue state are defined as actions, the introduced principle of maximum expected utility (MEU) provides an explicit way of action selection, based on the action utility, given the evidence about the user goal at each dialogue state. Decision networks, constructed as graphical models based on Bayesian networks are proposed to perform MEU-based decisions, incorporating the utility of the actions to be chosen at each dialogue state by the tour-guide robot. These action utilities are defined taking into account the tour-guide task requirements. The proposed graphical models for user goal identification and dialogue error handling in human-robot dialogue are evaluated in experiments with multimodal data. These data were collected during the operation of the tour-guide robot RoboX at the Autonomous System Lab of EPFL and at the Swiss National Exhibition in 2002 (Expo.02). The evaluation experiments use component and system level metrics for technical (objective) and user-based (subjective) evaluation. On the component level, the technical evaluation is done by calculating accuracies, as objective measures of the performance of the grounding model, and the resulting performance of the user goal identification in dialogue. The benefit of the proposed error handling framework is demonstrated comparing the accuracy of a baseline interactive system, employing only speech recognition for user goal identification, and a system equipped with multimodal grounding models for error handling

Context-Aware Recommendation Systems in Mobile Environments

Nowadays, the huge amount of information available may easily overwhelm users when they need to take a decision that involves choosing among several options. As a solution to this problem, Recommendation Systems (RS) have emerged to offer relevant items to users. The main goal of these systems is to recommend certain items based on user preferences. Unfortunately, traditional recommendation systems do not consider the user’s context as an important dimension to ensure high-quality recommendations. Motivated by the need to incorporate contextual information during the recommendation process, Context-Aware Recommendation Systems (CARS) have emerged. However, these recent recommendation systems are not designed with mobile users in mind, where the context and the movements of the users and items may be important factors to consider when deciding which items should be recommended. Therefore, context-aware recommendation models should be able to effectively and efficiently exploit the dynamic context of the mobile user in order to offer her/him suitable recommendations and keep them up-to-date.The research area of this thesis belongs to the fields of context-aware recommendation systems and mobile computing. We focus on the following scientific problem: how could we facilitate the development of context-aware recommendation systems in mobile environments to provide users with relevant recommendations? This work is motivated by the lack of generic and flexible context-aware recommendation frameworks that consider aspects related to mobile users and mobile computing. In order to solve the identified problem, we pursue the following general goal: the design and implementation of a context-aware recommendation framework for mobile computing environments that facilitates the development of context-aware recommendation applications for mobile users. In the thesis, we contribute to bridge the gap not only between recommendation systems and context-aware computing, but also between CARS and mobile computing.<br /

Demography and the Cultural Evolution of Pictorial Styles

Image-making is a nearly-universal human behaviour. Cultures around the world have made images to convey information about living kinds, objects and ideas for at least 75,000 years. However, from a stylistic point of view, the visual strategies and conventions to represent things in pictures can vary greatly over time and space; in particular, pictorial styles can differ in figurativeness, varying from inter-subjectively recognisable representations of things to stylised and abstract forms. Are there any patterns to this variability, and what might its ecological causes be? In recent Cultural Evolution research, factors such as demography and the structure of interaction between groups of individuals have been shown to affect the evolution of languages and technology. Capitalising on these studies, I investigate the style evolution in relation with the socio-demographic variable of population contact, and in particular the influence of inter-group contact on the figurativeness of pictorial representations. For this purpose, I firstly conducted an experimental study, simulating isolated and contact social groups with laboratory micro-societies performing a drawing task; secondly, I quantitatively analysed a real-world dataset of Aboriginal Australian rock art from contact and isolated communities; then I qualitatively explored the evolution of a contemporary pictorial communication item: emoji. Results show that pictorial representations from isolated groups tend to become abstract and opaque to outsiders, whereas in contact groups they retain figurativeness and external understandability. This supports the idea that intergroup contact is an important factor in the cultural evolution of pictorial styles, because the need to communicate with outsiders and be accessible to the widest possible audience encourages figurativeness. I discuss the implications of these findings for the archaeology and anthropology of art, and the parallels with language evolution. Finally, addressing the need for research outreach in Cultural Evolution, I designed the blueprint of an exhibition aimed at disseminating my research findings while offering lay audiences an engaging and transformative experience

Electronic Imaging & the Visual Arts. EVA 2013 Florence

Important Information Technology topics are presented: multimedia systems, data-bases, protection of data, access to the content. Particular reference is reserved to digital images (2D, 3D) regarding Cultural Institutions (Museums, Libraries, Palace – Monuments, Archaeological Sites). The main parts of the Conference Proceedings regard: Strategic Issues, EC Projects and Related Networks & Initiatives, International Forum on “Culture & Technology”, 2D – 3D Technologies & Applications, Virtual Galleries – Museums and Related Initiatives, Access to the Culture Information. Three Workshops are related to: International Cooperation, Innovation and Enterprise, Creative Industries and Cultural Tourism

Enabling Privacy and Trust in Edge AI Systems

Recent advances in mobile computing and the Internet of Things (IoT) enable the global integration of heterogeneous smart devices via wireless networks. A common characteristic across these modern day systems is their ability to collect and communicate streaming data, making machine learning (ML) appealing for processing, reasoning, and predicting about the environment. More recently, low network latency requirements have made offloading intelligence to the cloud undesirable. These novel requirements have led to the emergence of edge computing, an approach that brings computation closer to the device with low latency, high throughput, and enhanced reliability. Together, they enable ML-powered information processing and control pipelines spanning end devices, edge computing, and cloud environments. However, continuous collaboration between cloud, edge and device is susceptible to information leakage and loss, leading to insecure and unreliable operation. This raises an important question: how can we design, develop, and evaluate high-performing ML systems that are trustworthy and privacy-preserving in resource-constrained edge environments? In this thesis, I address this question by designing and implementing privacy-preserving and trustworthy ML systems for distributed applications. I first introduce a system that establishes trust in the explanations generated from a popular visualization technique, saliency maps, using counterfactual reasoning. Through the proposed evaluation system, I assess the degree to which hypothesized explanations correspond to the semantics of edge-based reinforcement learning environments. Second, I examine the privacy implications of personalized models in distributed mobile services by proposing time-series based model inversion attacks. To thwart such attacks, I present a distributed framework, Pelican, that learns and deploys transfer learning-based personalized ML models in a privacy preserving manner on resource-constrained mobile devices. Third, I investigate ML models that are deployed on local devices for inference and highlight the ease with which proprietary information embedded in these models can be exposed. For mitigating such attacks, I present a secure on-device application framework, SODA, which is supported by real-time adversarial detection. Finally, I present an end-to-end privacy-aware system for a real-world application to model group interaction behavior via mobility sensing. The proposed system, W4-Groups, distributes computation across device, edge, and cloud resources to strengthen its privacy and trustworthiness guarantees