Combining neural networks with symbolic approaches to perform complex event processing on non-symbolic data

This thesis presents three approaches to detecting situations of interest from non-symbolic data inputs such as images, audio and video through the use of Complex Event Processing (CEP) in an agile, reliable and efficient manner. These approaches must be agile, meaning that they must allow the implementation of solutions for a range of situations. We want them to be reliable, meaning that they must correctly detect the situations we are interested in. Finally, we want them to be efficient in terms of time and training data requirements. First, we present ProbCEP, an approach that combines proxy models with symbolic programming to perform CEP. We consider proxy models consisting of pre-trained neural networks, which allow the system to use non-symbolic inputs. However, the data used to train such proxy models is not necessarily related to the situations of interest (called complex events) we want to detect. Logic rules are used to define under which conditions these complex events occur, based on the output of the proxy models. We also show how the speed of the system can be significantly increased using specific optimization techniques. Then, we show two neuro-symbolic approaches, DeepProbCEP and Neuroplex. These approaches are designed to train a system to identify complex events from an input stream using small amounts of training data. Thanks to the injection of human knowledge into the system, these approaches require significantly less data than neural-only approaches. Following that, we explore the reliability of DeepProbCEP against several adversarial attacks that poison the training data. We also demonstrate that DeepProbCEP is an agile approach, allowing users to change the behaviour of the system to adapt to many situations. Finally, we discuss the potential of the research advances presented in this thesis for real-world applications, as well as possible areas for future research

A GEOSPATIAL STRATEGY TO DETECT AND IDENTIFY POTENTIAL CLANDESTINE MASS GRAVES IN UKRAINE

This study seeks to improve the understanding of how social media analysis and hyperspectral remote sensing can be applied in a multi-modal target-centric approach to detect and identify clandestine mass graves in Ukraine. Fifteen mass graves have been discovered in central, southern, and eastern Ukraine since the Russian invasion of Ukraine on 24 February 2022. This geospatial strategy recommends conducting retroactive searches for additional potential clandestine mass graves and prospective searches for suspicious activities that may be indicative of future or ongoing potential clandestine mass grave locations in areas currently under Russian control. This social media collection and analysis will direct targeted hyperspectral imagery collection on areas of interest. The resulting hyperspectral data will be further analyzed using spectral signature libraries, statistical methods, and vegetation indices to identify any vegetation anomalies with unusually high or low Nitrogen concentrations - a proxy signature for clandestine graves. This research will first present background information on the historical use of clandestine mass graves and the targeting of civilians in conflict. Second, it will examine the methodologies presented in this proof-of-concept. Third, it will evaluate the limitations of this geospatial strategy. This thesis provides new insights for detecting and identifying clandestine mass graves in Ukraine and provides potential audiences with another tool to locate and document these atrocities. Primary Reader and Advisor: Jack O’Connor Secondary Reader: Anonymou

Sensing, interpreting, and anticipating human social behaviour in the real world

Low-level nonverbal social signals like glances, utterances, facial expressions and body language are central to human communicative situations and have been shown to be connected to important high-level constructs, such as emotions, turn-taking, rapport, or leadership. A prerequisite for the creation of social machines that are able to support humans in e.g. education, psychotherapy, or human resources is the ability to automatically sense, interpret, and anticipate human nonverbal behaviour. While promising results have been shown in controlled settings, automatically analysing unconstrained situations, e.g. in daily-life settings, remains challenging. Furthermore, anticipation of nonverbal behaviour in social situations is still largely unexplored. The goal of this thesis is to move closer to the vision of social machines in the real world. It makes fundamental contributions along the three dimensions of sensing, interpreting and anticipating nonverbal behaviour in social interactions. First, robust recognition of low-level nonverbal behaviour lays the groundwork for all further analysis steps. Advancing human visual behaviour sensing is especially relevant as the current state of the art is still not satisfactory in many daily-life situations. While many social interactions take place in groups, current methods for unsupervised eye contact detection can only handle dyadic interactions. We propose a novel unsupervised method for multi-person eye contact detection by exploiting the connection between gaze and speaking turns. Furthermore, we make use of mobile device engagement to address the problem of calibration drift that occurs in daily-life usage of mobile eye trackers. Second, we improve the interpretation of social signals in terms of higher level social behaviours. In particular, we propose the first dataset and method for emotion recognition from bodily expressions of freely moving, unaugmented dyads. Furthermore, we are the first to study low rapport detection in group interactions, as well as investigating a cross-dataset evaluation setting for the emergent leadership detection task. Third, human visual behaviour is special because it functions as a social signal and also determines what a person is seeing at a given moment in time. Being able to anticipate human gaze opens up the possibility for machines to more seamlessly share attention with humans, or to intervene in a timely manner if humans are about to overlook important aspects of the environment. We are the first to propose methods for the anticipation of eye contact in dyadic conversations, as well as in the context of mobile device interactions during daily life, thereby paving the way for interfaces that are able to proactively intervene and support interacting humans.Blick, Gesichtsausdrücke, Körpersprache, oder Prosodie spielen als nonverbale Signale eine zentrale Rolle in menschlicher Kommunikation. Sie wurden durch vielzählige Studien mit wichtigen Konzepten wie Emotionen, Sprecherwechsel, Führung, oder der Qualität des Verhältnisses zwischen zwei Personen in Verbindung gebracht. Damit Menschen effektiv während ihres täglichen sozialen Lebens von Maschinen unterstützt werden können, sind automatische Methoden zur Erkennung, Interpretation, und Antizipation von nonverbalem Verhalten notwendig. Obwohl die bisherige Forschung in kontrollierten Studien zu ermutigenden Ergebnissen gekommen ist, bleibt die automatische Analyse nonverbalen Verhaltens in weniger kontrollierten Situationen eine Herausforderung. Darüber hinaus existieren kaum Untersuchungen zur Antizipation von nonverbalem Verhalten in sozialen Situationen. Das Ziel dieser Arbeit ist, die Vision vom automatischen Verstehen sozialer Situationen ein Stück weit mehr Realität werden zu lassen. Diese Arbeit liefert wichtige Beiträge zur autmatischen Erkennung menschlichen Blickverhaltens in alltäglichen Situationen. Obwohl viele soziale Interaktionen in Gruppen stattfinden, existieren unüberwachte Methoden zur Augenkontakterkennung bisher lediglich für dyadische Interaktionen. Wir stellen einen neuen Ansatz zur Augenkontakterkennung in Gruppen vor, welcher ohne manuelle Annotationen auskommt, indem er sich den statistischen Zusammenhang zwischen Blick- und Sprechverhalten zu Nutze macht. Tägliche Aktivitäten sind eine Herausforderung für Geräte zur mobile Augenbewegungsmessung, da Verschiebungen dieser Geräte zur Verschlechterung ihrer Kalibrierung führen können. In dieser Arbeit verwenden wir Nutzerverhalten an mobilen Endgeräten, um den Effekt solcher Verschiebungen zu korrigieren. Neben der Erkennung verbessert diese Arbeit auch die Interpretation sozialer Signale. Wir veröffentlichen den ersten Datensatz sowie die erste Methode zur Emotionserkennung in dyadischen Interaktionen ohne den Einsatz spezialisierter Ausrüstung. Außerdem stellen wir die erste Studie zur automatischen Erkennung mangelnder Verbundenheit in Gruppeninteraktionen vor, und führen die erste datensatzübergreifende Evaluierung zur Detektion von sich entwickelndem Führungsverhalten durch. Zum Abschluss der Arbeit präsentieren wir die ersten Ansätze zur Antizipation von Blickverhalten in sozialen Interaktionen. Blickverhalten hat die besondere Eigenschaft, dass es sowohl als soziales Signal als auch der Ausrichtung der visuellen Wahrnehmung dient. Somit eröffnet die Fähigkeit zur Antizipation von Blickverhalten Maschinen die Möglichkeit, sich sowohl nahtloser in soziale Interaktionen einzufügen, als auch Menschen zu warnen, wenn diese Gefahr laufen wichtige Aspekte der Umgebung zu übersehen. Wir präsentieren Methoden zur Antizipation von Blickverhalten im Kontext der Interaktion mit mobilen Endgeräten während täglicher Aktivitäten, als auch während dyadischer Interaktionen mittels Videotelefonie

Media’s influence on the 21st century society: A global criminological systematic review

This investigation assumes that the media can reduce or spread criminal activities and tendencies based on how the concerned parties apply the policies and community standards that guide these platforms’ use. In total, 254 materials were gathered across several search systems between October 2021 and September 2022. Qualitative data were used from the selected materials to synthesise and summarise the content on the examined 21st-century events and media’s influence on crime. It is not possible to reject the premise that the media influences opinions on crime and the legal system. Nevertheless, the data reveals that no causal media effect can be directly established. However, the same data uncovers how media portrays an activity affects how people perceive it. Advances in technology, media, and criminology may have affected the analysis of records, including the time and quality of resources. More accurate and fair media coverage of crime would lead to a more informed and aware population. On the other hand, media houses that promote and reward good behaviour should be applauded. These two steps ensure the media cannot be ignored when assessing crime and how the public perceives it, as it can encourage crime and shift perceptions. Therefore, further research, stricter laws and policies, and community education on crime prevention and media screening are needed. The fact that unfavourable media coverage of crime can ruin a business, either directly or indirectly (consumer behaviour changes due to crime), makes this paper of utmost importance for businessmen, politicians, and local agencies.Esta dissertação presume que os media podem ser utilizados para reduzir ou difundir atividades ou tendências criminosas, dependendo da aplicação de políticas e padrões comunitários que influenciam tais plataformas. Foram utilizados 254 materiais reunidos em diversos sistemas de pesquisa entre outubro de 2021 e setembro de 2022. Estes compreendem publicações do século XXI que examinam a influência dos media nas práticas criminais e suas perceções. Apesar deste estudo não possibilitar estabelecer uma relação causal, não é, ainda assim, possível rejeitar a premissa de que os media influenciam as perceções face ao crime. Determina, contudo, que o modo como os media divulgam uma atividade afeta a perceção social face à mesma. Uma população mais informada e consciente depende de uma cobertura mediática mais fatual. Os media que promovem e recompensam o bom comportamento devem ser louvados. Os media não podem ser ignorados na avaliação do crime e da sua perceção, tendo o poder de incentivar a criminalidade e potenciar alterações nas perceções sociais. Consequentemente, é necessário investigar mais, aplicar leis e políticas mais rigorosas, e investir em programas de educação comunitária de prevenção à criminalidade e interpretação dos media. Esta dissertação é de elevada importância a empresários, políticos e outros órgãos locais, pelo fato de a cobertura desfavorável do crime pelos media poder arruinar um indivíduo, organização ou até um negócio, seja de forma direta (críticas ao estabelecimento) ou indireta (mudanças no comportamento do consumidor devido à ocorrência de crimes numa região)