Motion Learning for Dynamic Scene Understanding

An important goal of computer vision is to automatically understand the visual world. With the introduction of deep networks, we see huge progress in static image understanding. However, we live in a dynamic world, so it is far from enough to merely understand static images. Motion plays a key role in analyzing dynamic scenes and has been one of the fundamental research topics in computer vision. It has wide applications in many fields, including video analysis, socially-aware robotics, autonomous driving, etc. In this dissertation, we study motion from two perspectives: geometric and semantic. From the geometric perspective, we aim to accurately estimate the 3D motion (or scene flow) and 3D structure of the scene. Since manually annotating motion is difficult, we propose self-supervised models for scene flow estimation from image and point cloud sequences. From the semantic perspective, we aim to understand the meanings of different motion patterns and first show that motion benefits detecting and tracking objects from videos. Then we propose a framework to understand the intentions and predict the future locations of agents in a scene. Finally, we study the role of motion information in action recognition

End-to-end anomaly detection in stream data

Nowadays, huge volumes of data are generated with increasing velocity through various systems, applications, and activities. This increases the demand for stream and time series analysis to react to changing conditions in real-time for enhanced efficiency and quality of service delivery as well as upgraded safety and security in private and public sectors. Despite its very rich history, time series anomaly detection is still one of the vital topics in machine learning research and is receiving increasing attention. Identifying hidden patterns and selecting an appropriate model that fits the observed data well and also carries over to unobserved data is not a trivial task. Due to the increasing diversity of data sources and associated stochastic processes, this pivotal data analysis topic is loaded with various challenges like complex latent patterns, concept drift, and overfitting that may mislead the model and cause a high false alarm rate. Handling these challenges leads the advanced anomaly detection methods to develop sophisticated decision logic, which turns them into mysterious and inexplicable black-boxes. Contrary to this trend, end-users expect transparency and verifiability to trust a model and the outcomes it produces. Also, pointing the users to the most anomalous/malicious areas of time series and causal features could save them time, energy, and money. For the mentioned reasons, this thesis is addressing the crucial challenges in an end-to-end pipeline of stream-based anomaly detection through the three essential phases of behavior prediction, inference, and interpretation. The first step is focused on devising a time series model that leads to high average accuracy as well as small error deviation. On this basis, we propose higher-quality anomaly detection and scoring techniques that utilize the related contexts to reclassify the observations and post-pruning the unjustified events. Last but not least, we make the predictive process transparent and verifiable by providing meaningful reasoning behind its generated results based on the understandable concepts by a human. The provided insight can pinpoint the anomalous regions of time series and explain why the current status of a system has been flagged as anomalous. Stream-based anomaly detection research is a principal area of innovation to support our economy, security, and even the safety and health of societies worldwide. We believe our proposed analysis techniques can contribute to building a situational awareness platform and open new perspectives in a variety of domains like cybersecurity, and health

Correcting inter-sectional accuracy differences in drowsiness detection systems using generative adversarial networks (GANs)

Doctoral Degrees. University of KwaZulu-Natal, Durban.oad accidents contribute to many injuries and deaths among the human population. There is substantial evidence that proves drowsiness is one of the most prominent causes of road accidents all over the world. This results in fatalities and severe injuries for drivers, passengers, and pedestrians. These alarming facts are raising the interest in equipping vehicles with robust driver drowsiness detection systems to minimise accident rates. One of the primary concerns of motor industries is the safety of passengers and as a consequence they have invested significantly in research and development to equip vehicles with systems that can help minimise to road accidents. A number research endeavours have attempted to use Artificial intelligence, and particularly Deep Neural Networks (DNN), to build intelligent systems that can detect drowsiness automatically. However, datasets are crucial when training a DNN. When datasets are unrepresentative, trained models are prone to bias because they are unable to generalise. This is particularly problematic for models trained in specific cultural contexts, which may not represent a wide range of races, and thus fail to generalise. This is a specific challenge for driver drowsiness detection task, where most publicly available datasets are unrepresentative as they cover only certain ethnicity groups. This thesis investigates the problem of an unrepresentative dataset in the training phase of Convolutional Neural Networks (CNNs) models. Firstly, CNNs are compared with several machine learning techniques to establish their superior suitability for the driver drowsiness detection task. An investigation into the implementation of CNNs was performed and highlighted that publicly available datasets such as NTHU, DROZY and CEW do not represent a wide spectrum of ethnicity groups and lead to biased systems. A population bias visualisation technique was proposed to help identify the regions, or individuals where a model is failing to generalise on a picture grid. Furthermore, the use of Generative Adversarial Networks (GANs) with lightweight convolutions called Depthwise Separable Convolutions (DSC) for image translation to multi-domain outputs was investigated in an attempt to generate synthetic datasets. This thesis further showed that GANs can be used to generate more realistic images with varied facial attributes for predicting drowsiness across multiple ethnicity groups. Lastly, a novel framework was developed to detect bias and correct it using synthetic generated images which are produced by GANs. Training models using this framework results in a substantial performance boost

Targeting cholesterol esterification as a novel immune checkpoint in viral infections and cancer

Identifying metabolic targets that constrain tumours and viruses while boosting exhausted, dysfunctional T cells can provide novel therapeutic checkpoints. Modulating cholesterol esterification by inhibiting the enzyme acyl-CoA:cholesterol acyltransferase (ACAT) has a direct antitumour and antiviral effect and enhances murine anti-tumour CD8+ T cells. In this thesis, I showed that reduced formation of cholesterol-rich microdomains within the cell membrane (lipid rafts) was a feature of PD-1hi exhausted CD8+ T cells. I therefore investigated the potential for rescuing exhausted human T cells by modulating cholesterol esterification and lipid raft formation. Inhibiting ACAT enhanced the expansion of functional virus- and tumour-specific T cells from donors with chronic hepatitis B virus (HBV) infection, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, and hepatocellular carcinoma. The immune-boosting effect was not limited to circulating T cells but could also enhance the function of T cells directly ex vivo from the immunosuppressive liver and tumour microenvironment in the majority of donors. ACAT inhibition led to a redistribution of intracellular cholesterol with reduced neutral lipid droplets and increased lipid raft formation, resulting in enhanced T cell receptor (TCR) signalling and T cell effector function. Additionally, ACAT inhibition induced TCR-independent bioenergetic rewiring with a skewing towards utilization of oxidative phosphorylation. ACAT inhibition had a complementary effect with other immunotherapies, with increased responsiveness to PD-1 blockade and enhanced functional avidity of TCR-engineered T cells recognizing HBV and tumour cells. Taken together, reduced lipid rafts are a feature of exhausted T cells and modulating cholesterol esterification by ACAT inhibition is a promising novel immunotherapeutic approach to boost exhausted antiviral and antitumour T cells in acute and chronic infection and in cancer

Genome-wide discovery of translational control mechanisms.

Genome-wide discovery of translation control mechanisms The control of mRNA translation into proteins is critical for the adaptation of eukaryotic cells to environmental changes and stress conditions. Glucose starvation in yeast is one of the prototypical eukaryotic stresses. The early translation-based responses to glucose starvation are critically important to trigger the subsequent events leading to transcriptional reprogramming, but the mRNAs involved and the mechanisms of their selective regulation remain obscure. I analysed glucose-specific translational control primarily using Translation Complex Profile sequencing (TCP-seq). TCP-seq measures the distribution of ribosomal complexes along mRNA using a combination of in vivo fixation, complex purification and limited nuclease digestion, and high-throughput sequencing of the protected mRNA fragments (footprints). Here, to dissect different aspects of translation regulation, I separately recorded footprints of single ribosomes and disomes. I furthermore sequenced footprints of three distinct complexes involving small ribosomal subunits (SSU), those derived from polysomes, 'singular' subunits in solution, as well as those that selectively contained the scanning translation initiation factor eIF4A. During starvation, total RNA-seq revealed transcripts involved in metabolic processes are downregulated whereas transcripts involved in alcoholic catabolic processes are upregulated, as a coping mechanism to economize their energy and shift to alcohol as a source of nutrition after longer duration of starvation. Promoter analysis of these clusters revealed the presence of motifs that are indicative of modulating expression of ribosomal protein genes and ribosome biogenesis genes that are consistent with links to cell growth, stress and nutrient conditions. Investigating different cellular pools with RNA-seq hints towards rapid sequestration into granules as well. Under starvation and globally suppressed protein biosynthesis, I observed an increased level of translation of many glucosynthetic mRNAs, mRNAs encoding heat shock proteins, hexokinase, 3-phosphoglycerate kinase. We also demonstrate that stochastically co-localised ribosomes are linked with translation imitation rate and provide a robust variable to model and quantify specific protein output from mRNA. I further propose a new measure of translation efficiency (TE) which may be more robust to the regularly-encountered biases of the classical TE measure and can contain more accurate information, enabling ranking mRNAs by the absolute protein output during rapidly changing transcriptome background, as occurs in glucose starvation. Overall, these data uncover a complex picture of rapid translational changes and present a collection of transcripts involved in the primary, acute response to the stress. The results suggest that already early in the response several major pathways are involved in the translational control simultaneously, as mRNAs are specifically degraded or preserved and their translation is selectively shut down, unperturbed or upregulated

A highly condensed genome without heterochromatin : orchestration of gene expression and epigenomics in Paramecium tetraurelia

Epigenetic regulation in unicellular ciliates can be as complex as in metazoans and is well described regarding small RNA (sRNA) mediated effects. The ciliate Paramecium harbors several copies of sRNA-biogenesis related proteins involved in genome rearrangements resulting in chromatin alterations. The global chromatin organization thereby is poorly understood, and unusual characteristics of the somatic nucleus, like high polyploidy, high genome coding density, and absence of heterochromatin, ought to call for complex regulation to orchestrate gene expression. The present study characterized the nucleosomal organization required for gene regulation and proper Polymerase II activity. Histone marks reveal broad domains in gene bodies, whereas intergenic regions are nucleosome free. Low occupancy in silent genes suggests that gene inactivation does not involve nucleosome recruitment. Thus, Paramecium gene regulation counteracts the current understanding of chromatin biology. Apart from global nucleosome studies, two sRNA binding proteins (Ptiwis) classically associated with transposon silencing were investigated in the background of transgene-induced silencing. Surprisingly, both Ptiwis also load sRNAs from endogenous loci in vegetative growth, revealing a broad diversity of Ptiwi functions. Together, the studies enlighten epigenetic mechanisms that regulate gene expression in a condensed genome, with Ptiwis contributing to transcriptome and chromatin dynamics.Epigenetische Regulation kann in einzelligen Ciliaten so komplex sein wie in Vielzellern und wurde umfassend angesichts kleiner RNA (sRNA)-vermittelter Effekte untersucht. Der Ciliat Paramecium besitzt mehrere Kopien sRNA-Biogenese assoziierter Proteine, die an Genomprozessierungen und resultierenden Chromatinänderungen beteiligt sind. Die globale Organisation des Chromatins ist dabei kaum verstanden und obskure Eigenschaften des somatischen Kerns, wie hohe Polyploidie, Kodierungsdichte und Fehlen von Heterochromatin, sollten eine komplexe Regulation zur Steuerung der Genexpression erfordern. Die vorliegende Studie charakterisiert die Chromatinorganisation, die für die Genregulation und Polymerase II Aktivität notwendig ist. Histonmodifikationen zeigen breite Verteilungen in Genen, während intergenische Regionen Nukleosomen-frei sind. Ein Stilllegen von Genen scheint ohne die Rekrutierung von Nukleosomen zu erfolgen, womit die Genregulation in Paramecium dem aktuellen Verständnis der Chromatinbiologie widerspricht. Neben Nukleosomenstudien wurden zwei sRNA-bindende Proteine (Ptiwis), die klassisch mit Transposon-Silencing assoziiert sind, im Hintergrund des Transgeninduzierten Silencings untersucht. Überraschenderweise laden Ptiwis sRNAs von endogenen Loci im vegetativen Wachstum, was vielfältige Ptiwi-Funktionen offenbart. Die Studien zeigen epigenetische Mechanismen zur Genregulation in einem kompakten Genom, wobei Ptiwis zur Transkriptom- und Chromatindynamik beitragen

xxAI - Beyond Explainable AI

This is an open access book. Statistical machine learning (ML) has triggered a renaissance of artificial intelligence (AI). While the most successful ML models, including Deep Neural Networks (DNN), have developed better predictivity, they have become increasingly complex, at the expense of human interpretability (correlation vs. causality). The field of explainable AI (xAI) has emerged with the goal of creating tools and models that are both predictive and interpretable and understandable for humans. Explainable AI is receiving huge interest in the machine learning and AI research communities, across academia, industry, and government, and there is now an excellent opportunity to push towards successful explainable AI applications. This volume will help the research community to accelerate this process, to promote a more systematic use of explainable AI to improve models in diverse applications, and ultimately to better understand how current explainable AI methods need to be improved and what kind of theory of explainable AI is needed. After overviews of current methods and challenges, the editors include chapters that describe new developments in explainable AI. The contributions are from leading researchers in the field, drawn from both academia and industry, and many of the chapters take a clear interdisciplinary approach to problem-solving. The concepts discussed include explainability, causability, and AI interfaces with humans, and the applications include image processing, natural language, law, fairness, and climate science.https://digitalcommons.unomaha.edu/isqafacbooks/1000/thumbnail.jp

xxAI - Beyond Explainable AI

This is an open access book. Statistical machine learning (ML) has triggered a renaissance of artificial intelligence (AI). While the most successful ML models, including Deep Neural Networks (DNN), have developed better predictivity, they have become increasingly complex, at the expense of human interpretability (correlation vs. causality). The field of explainable AI (xAI) has emerged with the goal of creating tools and models that are both predictive and interpretable and understandable for humans. Explainable AI is receiving huge interest in the machine learning and AI research communities, across academia, industry, and government, and there is now an excellent opportunity to push towards successful explainable AI applications. This volume will help the research community to accelerate this process, to promote a more systematic use of explainable AI to improve models in diverse applications, and ultimately to better understand how current explainable AI methods need to be improved and what kind of theory of explainable AI is needed. After overviews of current methods and challenges, the editors include chapters that describe new developments in explainable AI. The contributions are from leading researchers in the field, drawn from both academia and industry, and many of the chapters take a clear interdisciplinary approach to problem-solving. The concepts discussed include explainability, causability, and AI interfaces with humans, and the applications include image processing, natural language, law, fairness, and climate science