The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences

This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human annotated samples. The main contributions of this thesis are: a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks

The internet of ontological things: On symmetries between ubiquitous problems and their computational solutions in the age of smart objects

This dissertation is about an abstract form of computer network that has recently earned a new physical incarnation called “the Internet of Things.” It surveys the ontological transformations that have occurred over recent decades to the computational components of this network, objects—initially designed as abstract algorithmic agents in a source code of computer programming but now transplanted into real-world objects. Embodying the ideal of modularity, objects have provided computer programmers with more intuitive means to construct a software application with lots of simple and reusable functional building blocks. Their capability of being reassembled into many different networks for a variety of applications has also embodied another ideal of computing machines, namely general-purposiveness. In the algorithmic cultures of the past century, these objects existed as mere abstractions to help humans to understand electromagnetic signals that had infiltrated every corner of automatized spaces from private to public. As an instrumental means to domesticate these elusive signals into programmable architectures according to the goals imposed by professional programmers and amateur end-users, objects promised a universal language for any computable human activities. This utopian vision for the object-oriented domestication of the digital has had enough traction for the growth of the software industry as it has provided an alibi to hide another process of colonization occurring on the flipside of their interfacing between humans and machines: making programmable the highest number of online and offline human activities possible. A more recent media age, which this dissertation calls the age of the Internet of Things, refers to the second phase of this colonization of human cultures by the algorithmic objects, no longer trapped in the hard-wired circuit boards of personal computer, but now residing in real-life objects with new wireless communicability. Chapters of this dissertation examine each different computer application—a navigation system in a smart car, smart home, open-world video games, and neuro-prosthetics—as each particular case of this object-oriented redefinition of human cultures

Understanding and Mitigating Multi-sided Exposure Bias in Recommender Systems

Fairness is a critical system-level objective in recommender systems that has been the subject of extensive recent research. It is especially important in multi-sided recommendation platforms where it may be crucial to optimize utilities not just for the end user, but also for other actors such as item sellers or producers who desire a fair representation of their items. Existing solutions do not properly address various aspects of multi-sided fairness in recommendations as they may either solely have one-sided view (i.e. improving the fairness only for one side), or do not appropriately measure the fairness for each actor involved in the system. In this thesis, I aim at first investigating the impact of unfair recommendations on the system and how these unfair recommendations can negatively affect major actors in the system. Then, I seek to propose solutions to tackle the unfairness of recommendations. I propose a rating transformation technique that works as a pre-processing step before building the recommendation model to alleviate the inherent popularity bias in the input data and consequently to mitigate the exposure unfairness for items and suppliers in the recommendation lists. Also, as another solution, I propose a general graph-based solution that works as a post-processing approach after recommendation generation for mitigating the multi-sided exposure bias in the recommendation results. For evaluation, I introduce several metrics for measuring the exposure fairness for items and suppliers, and show that these metrics better capture the fairness properties in the recommendation results. I perform extensive experiments to evaluate the effectiveness of the proposed solutions. The experiments on different publicly-available datasets and comparison with various baselines confirm the superiority of the proposed solutions in improving the exposure fairness for items and suppliers.Comment: Doctoral thesi