Deep Feature Learning and Adaptation for Computer Vision

We are living in times when a revolution of deep learning is taking place. In general, deep learning models have a backbone that extracts features from the input data followed by task-specific layers, e.g. for classification. This dissertation proposes various deep feature extraction and adaptation methods to improve task-specific learning, such as visual re-identification, tracking, and domain adaptation. The vehicle re-identification (VRID) task requires identifying a given vehicle among a set of vehicles under variations in viewpoint, illumination, partial occlusion, and background clutter. We propose a novel local graph aggregation module for feature extraction to improve VRID performance. We also utilize a class-balanced loss to compensate for the unbalanced class distribution in the training dataset. Overall, our framework achieves state-of-the-art (SOTA) performance in multiple VRID benchmarks. We further extend our VRID method for visual object tracking under occlusion conditions. We motivate visual object tracking from aerial platforms by conducting a benchmarking of tracking methods on aerial datasets. Our study reveals that the current techniques have limited capabilities to re-identify objects when fully occluded or out of view. The Siamese network based trackers perform well compared to others in overall tracking performance. We utilize our VRID work in visual object tracking and propose Siam-ReID, a novel tracking method using a Siamese network and VRID technique. In another approach, we propose SiamGauss, a novel Siamese network with a Gaussian Head for improved confuser suppression and real time performance. Our approach achieves SOTA performance on aerial visual object tracking datasets. A related area of research is developing deep learning based domain adaptation techniques. We propose continual unsupervised domain adaptation, a novel paradigm for domain adaptation in data constrained environments. We show that existing works fail to generalize when the target domain data are acquired in small batches. We propose to use a buffer to store samples that are previously seen by the network and a novel loss function to improve the performance of continual domain adaptation. We further extend our continual unsupervised domain adaptation research for gradually varying domains. Our method outperforms several SOTA methods even though they have the entire domain data available during adaptation

Mental and motor representation for music performance

This research proposes a theory of nonconscious motor representation which precedes mental representation of the outcome of motor actions in music performance. The music performer faces the problem of how to escape sedimented musical paradigms to produce novel configurations of dynamics, timing and tone colour. If the sound were mentally represented as an action goal prior to being produced, it would tend to be assimilated to a known action goal. The proposed theory is intended to account for creativity in music performance, but has implications in other areas for both creativity and motor actions. The investigation began with an ethnographic study of two 'posthardcore' rock bands in London and Bristol. Posthardcore musicians work with minimal explicit knowledge of music theory and cognitive involvement in performance is actively eschewed. Serendipitous musical felicities in performance are valued. Such felicities depend on adjustment and fine control of dynamics, timing and tone colour within the parameters of the given. A selective survey of music aesthetics shows that the defining qualities of music are the production of immanent rather than representational meaning; polysemy; and processuality. Taking an analytic philosophy and cognitive science approach, I argue that apprehensions of immanent meaning depend on relationships between proximal percepts within the specious present. A general argument for nonconceptual perceptual content as perception of relations between magnitudes within the specious present is extended to music and argued to account for both the polysemic richness of music and its processuality. Nonconceptual relational perception can account for novel apprehensions by music listeners, but not for the production of novel configurations by the performer. I argue that motor creativity in music performance is achieved through the nonconscious parameterization of inverse models without conscious representation of the goal of the action. Conscious representation for the performer occurs when they hear their own performance

Image and Video Forensics

Nowadays, images and videos have become the main modalities of information being exchanged in everyday life, and their pervasiveness has led the image forensics community to question their reliability, integrity, confidentiality, and security. Multimedia contents are generated in many different ways through the use of consumer electronics and high-quality digital imaging devices, such as smartphones, digital cameras, tablets, and wearable and IoT devices. The ever-increasing convenience of image acquisition has facilitated instant distribution and sharing of digital images on digital social platforms, determining a great amount of exchange data. Moreover, the pervasiveness of powerful image editing tools has allowed the manipulation of digital images for malicious or criminal ends, up to the creation of synthesized images and videos with the use of deep learning techniques. In response to these threats, the multimedia forensics community has produced major research efforts regarding the identification of the source and the detection of manipulation. In all cases (e.g., forensic investigations, fake news debunking, information warfare, and cyberattacks) where images and videos serve as critical evidence, forensic technologies that help to determine the origin, authenticity, and integrity of multimedia content can become essential tools. This book aims to collect a diverse and complementary set of articles that demonstrate new developments and applications in image and video forensics to tackle new and serious challenges to ensure media authenticity

Knowledge Transfer in Object Recognition.

PhD Thesis.Abstract Object recognition is a fundamental and long-standing problem in computer vision. Since the latest resurgence of deep learning, thousands of techniques have been proposed and brought to commercial products to facilitate people’s daily life. Although remarkable achievements in object recognition have been witnessed, existing machine learning approaches remain far away from human vision system, especially in learning new concepts and Knowledge Transfer (KT) across scenarios. One main reason is that current learning approaches address isolated tasks by independently training predefined models, without considering any knowledge learned from previous tasks or models. In contrast, humans have an inherent ability to transfer the knowledge acquired from earlier tasks or people to new scenarios. Therefore, to scaling object recognition in realistic deployment, effective KT schemes are required. This thesis studies several aspects of KT for scaling object recognition systems. Specifically, to facilitate the KT process, several mechanisms on fine-grained and coarse-grained object recognition tasks are analyzed and studied, including 1) cross-class KT on person re-identification (reid); 2) cross-domain KT on person re-identification; 3) cross-model KT on image classification; 4) cross-task KT on image classification. In summary, four types of knowledge transfer schemes are discussed as follows: Chapter 3 Cross-class KT in person re-identification, one of representative fine-grained object recognition tasks, is firstly investigated. The nature of person identity classes for person re-id are totally disjoint between training and testing (a zero-shot learning problem), resulting in the highly demand of cross-class KT. To solve that, existing person re-id approaches aim to derive a feature representation for pairwise similarity based matching and ranking, which is able to generalise to test. However, current person re-id methods assume the provision of accurately cropped person bounding boxes and each of them is in the same resolution, ignoring the impact of the background noise and variant scale of images to cross-class KT. This is more severed in practice when person bounding boxes must be detected automatically given a very large number of images and/or videos (un-constrained scene images) processed. To address these challenges, this chapter provides two novel approaches, aiming to promote cross-class KT and boost re-id performance. 1) This chapter alleviates inaccurate person bounding box by developing a joint learning deep model that optimises person re-id attention selection within any auto-detected person bounding boxes by reinforcement learning of background clutter minimisation. Specifically, this chapter formulates a novel unified re-id architecture called Identity DiscriminativE Attention reinforcement Learning (IDEAL) to accurately select re-id attention in auto-detected bounding boxes for optimising re-id performance. 2) This chapter addresses multi-scale problem by proposing a Cross-Level Semantic Alignment (CLSA) deep learning approach capable of learning more discriminative identity feature representations in a unified end-to-end model. This 4 is realised by exploiting the in-network feature pyramid structure of a deep neural network enhanced by a novel cross pyramid-level semantic alignment loss function. Extensive experiments show the modelling advantages and performance superiority of both IDEAL and CLSA over the state-of-the-art re-id methods on widely used benchmarking datasets. Chapter 4 In this chapter, we address the problem of cross-domain KT in unsupervised domain adaptation for person re-id. Specifically, this chapter considers cross-domain KT as follows: 1) Unsupervised domain adaptation: “train once, run once” pattern, transferring knowledge from source domain to specific target domain and the model is restricted to be applied on target domain only; 2) Universal re-id: “train once, run everywhere” pattern, transferring knowledge from source domain to any target domains, and therefore is capable of deploying any domains of re-id task. This chapter firstly develops a novel Hierarchical Unsupervised Domain Adaptation (HUDA) method for unsupervised domain adaptation for re-id. It can automatically transfer labelled information of an existing dataset (a source domain) to an unlabelled target domain for unsupervised person re-id. Specifically, HUDA is designed to model jointly global distribution alignment and local instance alignment in a two-level hierarchy for discovering transferable source knowledge in unsupervised domain adaptation. Crucially, this approach aims to overcome the under-constrained learning problem of existing unsupervised domain adaptation methods, lacking of the local instance alignment constraint. The consequence is more effective and cross-domain KT from the labelled source domain to the unlabelled target domain. This chapter further addresses the limitation of “train once, run once ” for existing domain adaptation person re-id approaches by presenting a novel “train once, run everywhere” pattern. This conventional “train once, run once” pattern is unscalable to a large number of target domains typically encountered in real-world deployments, due to the requirement of training a separate model for each target domain as supervised learning methods. To mitigate this weakness, a novel “Universal Model Learning” (UML) approach is formulated to enable domain-generic person re-id using only limited training data of a “single” seed domain. Specifically, UML trains a universal re-id model to discriminate between a set of transformed person identity classes. Each of such classes is formed by applying a variety of random appearance transformations to the images of that class, where the transformations simulate camera viewing conditions of any domains for making the model domain generic. Chapter 5 The third problem considered in this thesis is cross-model KT in coarse-grained object recognition. This chapter discusses knowledge distillation in image classification. Knowledge distillation is an effective approach to transfer knowledge from a large teacher neural network to a small student (target) network for satisfying the low-memory and fast running requirements. Whilst being able to create stronger target networks compared to the vanilla non-teacher based learning strategy, this scheme needs to train additionally a large teacher model with expensive computational cost and requires complex multi-stages training. This chapter firstly presents a Self-Referenced Deep Learning (SRDL) strategy to accelerate the training process. Unlike both vanilla optimisation and knowledge distillation, SRDL distils the knowledge discovered by the in-training target model back to itself for regularising the subsequent learning procedure therefore eliminating the need for training a large teacher model. Secondly, an On-the-fly Native Ensemble (ONE) learning strategy for one-stage knowledge distillation is proposed to solve the weakness of complex multi-stages training. Specifically, ONE only trains a single multi-branch network while simultaneously establishing a strong teacher on-the-fly to enhance the learning of target network. Chapter 6 Forth, this thesis studies the cross-task KT in coarse-grained object recognition. This chapter focuses on the few-shot classification problem, which aims to train models capable of recognising new, previously unseen categories from the novel task by using only limited training samples. Existing metric learning approaches constitute a highly popular strategy, learning discriminative representations such that images, containing different classes, are well separated in an embedding space. The commonly held assumption that each class is summarised by a sin5 gle, global representation (referred to as a prototype) that is then used as a reference to infer class labels brings significant drawbacks. This formulation fails to capture the complex multi-modal latent distributions that often exist in real-world problems, and yields models that are highly sensitive to the prototype quality. To address these limitations, this chapter proposes a novel Mixture of Prototypes (MP) approach that learns multi-modal class representations, and can be integrated into existing metric based methods. MP models class prototypes as a group of feature representations carefully designed to be highly diverse and maximise ensembling performance. Furthermore, this thesis investigates the benefit of incorporating unlabelled data in cross-task KT, and focuses on the problem of Semi-Supervised Few-shot Learning (SS-FSL). Recent SSFSL work has relied on popular Semi-Supervised Learning (SSL) concepts, involving iterative pseudo-labelling, yet often yields models that are susceptible to error propagation and sensitive to initialisation. To address this limitation, this chapter introduces a novel prototype-based approach (Fewmatch) for SS-FSL that exploits model Consistency Regularization (CR) in a robust manner and promotes cross-task unlabelled data knowledge transfer. Fewmatch exploits unlabelled data via Dynamic Prototype Refinement (DPR) approach, where novel class prototypes are alternatively refined 1) explicitly, using unlabelled data with high confidence class predictions and 2) implicitly, by model fine-tuning using a data selective model CR loss. DPR affords CR convergence, with the explicit refinement providing an increasingly stronger initialisation and alleviates the issue of error propagation, due to the application of CR. Chapter 7 draws conclusions and suggests future works that extend the ideas and methods developed in this thesi

Biometric Systems

Because of the accelerating progress in biometrics research and the latest nation-state threats to security, this book's publication is not only timely but also much needed. This volume contains seventeen peer-reviewed chapters reporting the state of the art in biometrics research: security issues, signature verification, fingerprint identification, wrist vascular biometrics, ear detection, face detection and identification (including a new survey of face recognition), person re-identification, electrocardiogram (ECT) recognition, and several multi-modal systems. This book will be a valuable resource for graduate students, engineers, and researchers interested in understanding and investigating this important field of study

Process Mining Workshops

This open access book constitutes revised selected papers from the International Workshops held at the Third International Conference on Process Mining, ICPM 2021, which took place in Eindhoven, The Netherlands, during October 31–November 4, 2021. The conference focuses on the area of process mining research and practice, including theory, algorithmic challenges, and applications. The co-located workshops provided a forum for novel research ideas. The 28 papers included in this volume were carefully reviewed and selected from 65 submissions. They stem from the following workshops: 2nd International Workshop on Event Data and Behavioral Analytics (EDBA) 2nd International Workshop on Leveraging Machine Learning in Process Mining (ML4PM) 2nd International Workshop on Streaming Analytics for Process Mining (SA4PM) 6th International Workshop on Process Querying, Manipulation, and Intelligence (PQMI) 4th International Workshop on Process-Oriented Data Science for Healthcare (PODS4H) 2nd International Workshop on Trust, Privacy, and Security in Process Analytics (TPSA) One survey paper on the results of the XES 2.0 Workshop is included

Process Mining Workshops

This open access book constitutes revised selected papers from the International Workshops held at the Third International Conference on Process Mining, ICPM 2021, which took place in Eindhoven, The Netherlands, during October 31–November 4, 2021. The conference focuses on the area of process mining research and practice, including theory, algorithmic challenges, and applications. The co-located workshops provided a forum for novel research ideas. The 28 papers included in this volume were carefully reviewed and selected from 65 submissions. They stem from the following workshops: 2nd International Workshop on Event Data and Behavioral Analytics (EDBA) 2nd International Workshop on Leveraging Machine Learning in Process Mining (ML4PM) 2nd International Workshop on Streaming Analytics for Process Mining (SA4PM) 6th International Workshop on Process Querying, Manipulation, and Intelligence (PQMI) 4th International Workshop on Process-Oriented Data Science for Healthcare (PODS4H) 2nd International Workshop on Trust, Privacy, and Security in Process Analytics (TPSA) One survey paper on the results of the XES 2.0 Workshop is included

Lost in technology: Towards a critique of repugnant rights

Modern law is founded on an idea of justice that is made felt through rights and entitlements legal subjects enjoy. As such, for law and its idea of justice, rights are inherently good and therefore abundant. On encounter with injustice, it has become commonplace to inquire what laws and rights have been flouted, as if injustice would disappear in encounter with rights that encode justice. But what if no number of laws and rights – even with faultless execution – is up for the task of upholding what we deem just? In this dissertation, I look at the heart of this question, and find the law’s answer not simply wanting but repugnant. The research is animated by interaction of three topoi: personhood, technology, and international law. The first part concerns how these concepts are perceived in law and by those working with laws. As part of the unearthing of the conceptual ground rules, a trilemma between effectiveness, responsiveness, and coherence familiar from regulatory research and international law rears its head. I show how setting the priority on effective and responsive solutions has amounted to derogation of justice and diminishment of law’s foundational entity, a natural person. I explore whether these outcomes could be avoided within liberal international law and answer my own question on the negative. I title this systematic outcome a theory of repugnant rights. The latter part of the dissertation concerns technology, its regulation, and tendency to produce repugnant outcomes in international law. I focus on bio- and information technologies and their legal coding as tools to dismantle legal protection provided by our quality of being human. I will show how intricate legal norms break and remake us in ways that blur the boundaries between persons and things. Once something falls beyond or below the category of a person, its legal status can be warped, twisted, and turned – all while remaining at arm’s length from the person it was once legally part of. Technological intervention to such things allows for effective circumvention of legal shelter provided by human rights, as I show through example of regulation of surrogacy and data storage. To come to terms with the repugnancy, I seek shelter from anger as a transitory category that would enable us to move across the present impasse with rights. I suggest that at the very least international lawyers ought to be angry at quotidian horrors international law upholds. And through such anger overcome the misery and repugnancy of international law.--- Moderni oikeus pohjaa ajatukseen oikeudenmukaisuudesta, joka ilmenee oikeussubjektien nauttimien ja käyttämien oikeuksien välityksellä. Näin ymmärrettynä oikeuden ja sen omaaman oikeudenmukaisuuden käsityksen kannalta oikeudet ovat itseisarvoisesti hyviä, mikä selittää niiden suuren määrän. Kun kohtaamme epäoikeudenmukaisuutta tapaamme kysyä, mitä lakeja ja oikeuksia on loukattu, ikään kuin epäoikeudenmukaisuus kaikkoaisi sen kohdatessa oikeuden sisältämän oikeudenmukaisuuden idean. Mutta entä jos mikään määrä lakeja ja oikeuksia – edes täydellisesti täytäntöönpantuna – ei riitä puolustamaan oikeudenmukaisena pitämäämme? Väitöskirjassani kurkistan tämän kysymyksen ytimeen ja löydän vastauksen, joka ei ole ainoastaan riittämätön vaan myös vastenmielinen. Väitöksessäni operoin oikeushenkilön, teknologian ja kansainvälisen oikeuden rajapinnoilla. Väitökseni ensimmäinen osa koskee sitä, kuinka oikeuden ja lakien parissa työskentelevät mieltävät nämä käsitteet. Näiden käsitteiden tarkastelun yhteydessä havaitsen sääntelytutkimuksesta ja kansainvälisestä oikeudesta tutun tehokkuuden, responsiivisuuden ja johdonmukaisuuden välisen trilemman. Osoitan, miten tehokkaiden ja responsiivisten ratkaisujen asettaminen etusijalle on merkinnyt lipeämistä oikeudenmukaisuudesta ja samalla oikeuden keskeisen subjektin, luonnollisen henkilön, merkityksen pienentymistä. Tutkin, voitaisiinko tämä trilemma välttää liberaalin kansainvälisen oikeuden puitteissa, ja vastaan omaan kysymykseeni kielteisesti. Nimeän tämän tuloksen vastenmielisten oikeuksien teoriaksi. Väitöskirjan jälkimmäinen osa käsittelee teknologiaa, sen säätelyä ja sen taipumusta tuottaa vastenmielisiä lopputuloksia kansainvälisessä oikeudessa. Tarkastelen lähemmin bio- ja informaatioteknologioita ja niiden oikeudellista sääntelyä, sekä sitä millaisia välineitä ne tarjoavat ihmisyyden tarjoaman oikeudellisen suojan purkamiseen. Osoitan kuinka monimutkaiset oikeudelliset normit rikkovat ja muokkaavat meitä tavoilla, jotka hämärtävät ihmisten ja asioiden välisiä rajoja. Kun jokin ei ole enää henkilö, sen oikeudellista asemaa voidaan vääristää, vääntää ja kääntää. Teknologinen puuttuminen tällaisiin esineisiin ja asioihin mahdollistaa ihmisoikeuksien tarjoaman laillisen suojan tehokkaan kiertämisen, kuten osoitan sijaissynnytyksen ja datan tallennuksen sääntelyn kautta. Vastauksena oikeuden vastenmielisyydelle haen suojaa vihasta. Viha tarjoaa sellaisen tilapäisen kategorian, jonka avulla voimme välttää havaitsemani oikeuksien umpikujan. Katson, että kansainvälisen oikeuden harjoittajien olisi vähintäänkin oltava vihaisia kohdatessaan kansainvälisen oikeuden synnyttämiä ja mahdollistamia jokapäiväisiä kauhuja. Turvautumalla vihaan, jonka voimme myöhemmin asettaa sivuun, voisimme selättää kansainvälisen oikeuden surkeuden ja sen vastenmielisyyden

Security and privacy of resource constrained devices

The thesis aims to present a comprehensive and holistic overview on cybersecurity and privacy & data protection aspects related to IoT resource-constrained devices. Chapter 1 introduces the current technical landscape by providing a working definition and architecture taxonomy of ‘Internet of Things’ and ‘resource-constrained devices’, coupled with a threat landscape where each specific attack is linked to a layer of the taxonomy. Chapter 2 lays down the theoretical foundations for an interdisciplinary approach and a unified, holistic vision of cybersecurity, safety and privacy justified by the ‘IoT revolution’ through the so-called infraethical perspective. Chapter 3 investigates whether and to what extent the fast-evolving European cybersecurity regulatory framework addresses the security challenges brought about by the IoT by allocating legal responsibilities to the right parties. Chapters 4 and 5 focus, on the other hand, on ‘privacy’ understood by proxy as to include EU data protection. In particular, Chapter 4 addresses three legal challenges brought about by the ubiquitous IoT data and metadata processing to EU privacy and data protection legal frameworks i.e., the ePrivacy Directive and the GDPR. Chapter 5 casts light on the risk management tool enshrined in EU data protection law, that is, Data Protection Impact Assessment (DPIA) and proposes an original DPIA methodology for connected devices, building on the CNIL (French data protection authority) model

Security and Privacy of Resource Constrained Devices

The thesis aims to present a comprehensive and holistic overview on cybersecurity and privacy & data protection aspects related to IoT resource-constrained devices. Chapter 1 introduces the current technical landscape by providing a working definition and architecture taxonomy of ‘Internet of Things’ and ‘resource-constrained devices’, coupled with a threat landscape where each specific attack is linked to a layer of the taxonomy. Chapter 2 lays down the theoretical foundations for an interdisciplinary approach and a unified, holistic vision of cybersecurity, safety and privacy justified by the ‘IoT revolution’ through the so-called infraethical perspective. Chapter 3 investigates whether and to what extent the fast-evolving European cybersecurity regulatory framework addresses the security challenges brought about by the IoT by allocating legal responsibilities to the right parties. Chapters 4 and 5 focus, on the other hand, on ‘privacy’ understood by proxy as to include EU data protection. In particular, Chapter 4 addresses three legal challenges brought about by the ubiquitous IoT data and metadata processing to EU privacy and data protection legal frameworks i.e., the ePrivacy Directive and the GDPR. Chapter 5 casts light on the risk management tool enshrined in EU data protection law, that is, Data Protection Impact Assessment (DPIA) and proposes an original DPIA methodology for connected devices, building on the CNIL (French data protection authority) model