Grounding semantic cognition using computational modelling and network analysis

The overarching objective of this thesis is to further the field of grounded semantics using a range of computational and empirical studies. Over the past thirty years, there have been many algorithmic advances in the modelling of semantic cognition. A commonality across these cognitive models is a reliance on hand-engineering “toy-models”. Despite incorporating newer techniques (e.g. Long short-term memory), the model inputs remain unchanged. We argue that the inputs to these traditional semantic models have little resemblance with real human experiences. In this dissertation, we ground our neural network models by training them with real-world visual scenes using naturalistic photographs. Our approach is an alternative to both hand-coded features and embodied raw sensorimotor signals. We conceptually replicate the mutually reinforcing nature of hybrid (feature-based and grounded) representations using silhouettes of concrete concepts as model inputs. We next gradually develop a novel grounded cognitive semantic representation which we call scene2vec, starting with object co-occurrences and then adding emotions and language-based tags. Limitations of our scene-based representation are identified for more abstract concepts (e.g. freedom). We further present a large-scale human semantics study, which reveals small-world semantic network topologies are context-dependent and that scenes are the most dominant cognitive dimension. This finding leads us to conclude that there is no meaning without context. Lastly, scene2vec shows promising human-like context-sensitive stereotypes (e.g. gender role bias), and we explore how such stereotypes are reduced by targeted debiasing. In conclusion, this thesis provides support for a novel computational viewpoint on investigating meaning - scene-based grounded semantics. Future research scaling scene-based semantic models to human-levels through virtual grounding has the potential to unearth new insights into the human mind and concurrently lead to advancements in artificial general intelligence by enabling robots, embodied or otherwise, to acquire and represent meaning directly from the environment

The Constructivistly-Organised Dimensional-Appraisal (CODA) Model and Evidence for the Role of Goal-directed Processes in Emotional Episodes Induced by Music

The study of affective responses to music is a flourishing field. Advancements in the study of this phenomena have been complemented by the introduction of several music-specific models of emotion, with two of the most well-cited ones being the BRECVEMA and the Multifactorial Process Model. These two models have undoubtedly contributed to the field. However, contemporary developments in the wider affective sciences (broadly described as the ‘rise of affectivism’) have yet to be incorporated into the music emotion literature. These developments in the affective sciences may aid in addressing remaining gaps in the music literature, in particular for acknowledging individual and contextual differences. The first aim of this thesis was to outline contemporary theories from the wider affective sciences and subsequently critique current popular models of musical emotions through the lens of these advancements. The second aim was to propose a new model based on this critique: the Constructivistly-Organised Dimensional-Appraisal (CODA) model. This CODA model draws together multiple competing models into a single framework centralised around goal-directed appraisal mechanisms which are key to the wider affective sciences but are a less commonly acknowledged component of musical affect. The third aim was to empirically test some of the core hypotheses of the CODA model. In particular, examining goal-directed mechanisms, their validity in a musical context, and their ability to address individual and contextual differences in musically induced affect. Across four experiments which include exploratory and lab-based designs through to real- world applications, the results are supportive of the role of goal-directed mechanisms in musically induced emotional episodes. Experiment one presents a first test battery of multiple appraisal dimensions developed for music. The results show that several of the hypothesised appraisal dimensions are valid dimensions is a musical context. Moreover, these mechanisms cluster into goal-directed latent variables. Experiment two develops a new set of stimuli annotations relating to musical goals, showing that music can be more or less appropriate for different musical goals (functions). Experiment three, using the new stimuli set from experiment two, tests the effects of different goals with more or less appropriate music on musically induced affect. These results show that goal-directed mechanisms can change induced core-affect (valence and arousal) and intensity, even for the same piece of music. Experiment four extends the study of goal-directed mechanisms into a real-world context through an interdisciplinary and cross-cultural design. The final experiment demonstrates how goal-directed mechanisms can be manipulated through different algorithms to induce negative affect in a Colombian population. The main conclusions of this thesis are that the CODA model, more specifically goal-directed mechanisms, provide a valuable, non-reductive, and more efficient approach to addressing individual and contextual differences for musically induced emotional episodes in the new era of affectivism

Designing Embodied Interactive Software Agents for E-Learning: Principles, Components, and Roles

Embodied interactive software agents are complex autonomous, adaptive, and social software systems with a digital embodiment that enables them to act on and react to other entities (users, objects, and other agents) in their environment through bodily actions, which include the use of verbal and non-verbal communicative behaviors in face-to-face interactions with the user. These agents have been developed for various roles in different application domains, in which they perform tasks that have been assigned to them by their developers or delegated to them by their users or by other agents. In computer-assisted learning, embodied interactive pedagogical software agents have the general task to promote human learning by working with students (and other agents) in computer-based learning environments, among them e-learning platforms based on Internet technologies, such as the Virtual Linguistics Campus (www.linguistics-online.com). In these environments, pedagogical agents provide contextualized, qualified, personalized, and timely assistance, cooperation, instruction, motivation, and services for both individual learners and groups of learners. This thesis develops a comprehensive, multidisciplinary, and user-oriented view of the design of embodied interactive pedagogical software agents, which integrates theoretical and practical insights from various academic and other fields. The research intends to contribute to the scientific understanding of issues, methods, theories, and technologies that are involved in the design, implementation, and evaluation of embodied interactive software agents for different roles in e-learning and other areas. For developers, the thesis provides sixteen basic principles (Added Value, Perceptible Qualities, Balanced Design, Coherence, Consistency, Completeness, Comprehensibility, Individuality, Variability, Communicative Ability, Modularity, Teamwork, Participatory Design, Role Awareness, Cultural Awareness, and Relationship Building) plus a large number of specific guidelines for the design of embodied interactive software agents and their components. Furthermore, it offers critical reviews of theories, concepts, approaches, and technologies from different areas and disciplines that are relevant to agent design. Finally, it discusses three pedagogical agent roles (virtual native speaker, coach, and peer) in the scenario of the linguistic fieldwork classes on the Virtual Linguistics Campus and presents detailed considerations for the design of an agent for one of these roles (the virtual native speaker)

Cognitive Reasoning and Inferences through Psychologically based Personalised Modelling of Emotions Using Associative Classifiers

Presentation available from: http://www.slideshare.net/AladdinAyesh/v1-cognitive-informaticspresentation201

Deconstructing the right to privacy considering the impact of fashion recommender systems on an individual’s autonomy and identity

Computing ‘fashion’ into a system of algorithms that personalise an individual’s shopping journey is not without risks to the way we express, assess, and develop aspects of our identity. This study uses an interdisciplinary research approach to examine how an individual’s interaction with algorithms in the fashion domain shapes our understanding of an individual’s privacy, autonomy, and identity. Using fashion theory and psychology, I make two contributions to the meaning of privacy to protect notions of identity and autonomy, and develop a more nuanced perspective on this concept using ‘fashion identity’. One, a more varied outlook on privacy allows us to examine how algorithmic constructions impose inherent reductions on individual sense-making in developing and reinventing personal fashion choices. A “right to not be reduced” allows us to focus on the individual’s practice of identity and choice with regard to the algorithmic entities incorporating imperfect semblances on the personal and social aspects of fashion. Second, I submit that we need a new perspective on the right to privacy to address the risks of algorithmic personalisation systems in fashion. There are gaps in the law regarding capturing the impact of algorithmic personalisation systems on an individual’s inference of knowledge about fashion, as well as the associations of fashion applied to individual circumstances. Focusing on the case law of the European Court of Human Rights (ECtHR) and the General Data Protection Regulation (GDPR), as well as aspects of EU non-discrimination and consumer law, I underline that we need to develop a proactive approach to the right to privacy entailing the incorporation of new values. I define these values to include an individual’s perception and self-relationality, describing the impact of algorithmic personalisation systems on an individual’s inference of knowledge about fashion, as well as the associations of fashion applied to individual circumstances. The study concludes with recommendations regarding the use of AI techniques in fashion using an international human rights approach. I argue that the “right to not be reduced” requires new interpretative guidance informing international human rights standards, including Article 17 of the International Covenant on Civil and Political Rights (ICCPR). Moreover, I consider that the “right to not be reduced” requires us to consider novel choices that inform the design and deployment of algorithmic personalisation systems in fashion, considering the UN Guiding Principles on Business and Human Rights and the EU Commission’s Proposal for an AI Act

Towards Entity Status

Discourse entities are an important construct in computational linguistics. They introduce an additional level of representation between referring expressions and that which they refer to: the level of mental representation. In this thesis, I first explore some semiotic and communication theoretic aspects of discourse entities. Then, I develop the concept of "entity status". Entity status is a meta-variable that collects two dimensions formations about the role that an entity plays a discourse, and management informations about how the entity is created, accessed, and updated. Finally, the concept is applied to two case studies: the first one focusses on the choice of referring expressions in radio news, while the second looks at the conditions under which a discourse entity can be mentioned as a pronoun.Diskursentitäten sind ein wichtiger Konstrukt in der Computerlinguistik. Sie führen eine zusätzliche Repräsentationsebene ein zwischen referierenden Ausdrücken, und dem, auf das diese Ausdrücke referieren: die Ebene der mentalen Repräsentation. In dieser Dissertation erkunde ich zunächst einige semiotische und kommunikationstheoretische Aspekte von Diskursentitäten. Danach führe ich den Begriff des "Entitätenstatus" ein. Entitätenstatus ist eine Meta-Variable, die zwei Dimensionen von Information über eine Diskursentität vereinigt: Struktur-Informationen über die Rolle, die eine Entität im Diskurs spielt, und Verwaltungs-Informationen über Erstellung, Zugriff und Update. Dieser Begriff wird schlussendlich auf zwei Fallstudien angewendet: die erste Studie konzentriert sich auf die Wahl referierender Ausdrücke in Radionachrichten, während die zweite Studie die Bedingungen untersucht, in denen eine Diskursentität als Pronomen erwähnt werden kann