110 research outputs found

    Evaluating Adversarial Robustness of Detection-based Defenses against Adversarial Examples

    Get PDF
    Machine Learning algorithms provide astonishing performance in a wide range of tasks, including sensitive and critical applications. On the other hand, it has been shown that they are vulnerable to adversarial attacks, a set of techniques that violate the integrity, confidentiality, or availability of such systems. In particular, one of the most studied phenomena concerns adversarial examples, i.e., input samples that are carefully manipulated to alter the model output. In the last decade, the research community put a strong effort into this field, proposing new evasion attacks and methods to defend against them. With this thesis, we propose different approaches that can be applied to Deep Neural Networks to detect and reject adversarial examples that present an anomalous distribution with respect to training data. The first leverages the domain knowledge of the relationships among the considered classes integrated through a framework in which first-order logic knowledge is converted into constraints and injected into a semi-supervised learning problem. Within this setting, the classifier is able to reject samples that violate the domain knowledge constraints. This approach can be applied in both single and multi-label classification settings. The second one is based on a Deep Neural Rejection (DNR) mechanism to detect adversarial examples, based on the idea of rejecting samples that exhibit anomalous feature representations at different network layers. To this end, we exploit RBF SVM classifiers, which provide decreasing confidence values as samples move away from the training data distribution. Despite technical differences, this approach shares a common backbone structure with other proposed methods that we formalize in a unifying framework. As all of them require comparing input samples against an oversized number of reference prototypes, possibly at different representation layers, they suffer from the same drawback, i.e., high computational overhead and memory usage, that makes these approaches unusable in real applications. To overcome this limitation, we introduce FADER (Fast Adversarial Example Rejection), a technique for speeding up detection-based methods by employing RBF networks as detectors: by fixing the number of required prototypes, their runtime complexity can be controlled. All proposed methods are evaluated in both black-box and white-box settings, i.e., against an attacker unaware of the defense mechanism, and against an attacker who knows the defense and adapts the attack algorithm to bypass it, respectively. Our experimental evaluation shows that the proposed methods increase the robustness of the defended models and help detect adversarial examples effectively, especially when the attacker does not know the underlying detection system

    Heart Diseases Diagnosis Using Artificial Neural Networks

    Get PDF
    Information technology has virtually altered every aspect of human life in the present era. The application of informatics in the health sector is rapidly gaining prominence and the benefits of this innovative paradigm are being realized across the globe. This evolution produced large number of patientsโ€™ data that can be employed by computer technologies and machine learning techniques, and turned into useful information and knowledge. This data can be used to develop expert systems to help in diagnosing some life-threating diseases such as heart diseases, with less cost, processing time and improved diagnosis accuracy. Even though, modern medicine is generating huge amount of data every day, little has been done to use this available data to solve challenges faced in the successful diagnosis of heart diseases. Highlighting the need for more research into the usage of robust data mining techniques to help health care professionals in the diagnosis of heart diseases and other debilitating disease conditions. Based on the foregoing, this thesis aims to develop a health informatics system for the classification of heart diseases using data mining techniques focusing on Radial Basis functions and emerging Neural Networks approach. The presented research involves three development stages; firstly, the development of a preliminary classification system for Coronary Artery Disease (CAD) using Radial Basis Function (RBF) neural networks. The research then deploys the deep learning approach to detect three different types of heart diseases i.e. Sleep Apnea, Arrhythmias and CAD by designing two novel classification systems; the first adopt a novel deep neural network method (with Rectified Linear unit activation) design as the second approach in this thesis and the other implements a novel multilayer kernel machine to mimic the behaviour of deep learning as the third approach. Additionally, this thesis uses a dataset obtained from patients, and employs normalization and feature extraction means to explore it in a unique way that facilitates its usage for training and validating different classification methods. This unique dataset is useful to researchers and practitioners working in heart disease treatment and diagnosis. The findings from the study reveal that the proposed models have high classification performance that is comparable, or perhaps exceed in some cases, the existing automated and manual methods of heart disease diagnosis. Besides, the proposed deep-learning models provide better performance when applied on large data sets (e.g., in the case of Sleep Apnea), with reasonable performance with smaller data sets. The proposed system for clinical diagnoses of heart diseases, contributes to the accurate detection of such disease, and could serve as an important tool in the area of clinic support system. The outcome of this study in form of implementation tool can be used by cardiologists to help them make more consistent diagnosis of heart diseases

    Who is the director of this movie? Automatic style recognition based on shot features

    Get PDF
    We show how low-level formal features, such as shot duration, meant as length of camera takes, and shot scale, i.e. the distance between the camera and the subject, are distinctive of a director's style in art movies. So far such features were thought of not having enough varieties to become distinctive of an author. However our investigation on the full filmographies of six different authors (Scorsese, Godard, Tarr, Fellini, Antonioni, and Bergman) for a total number of 120 movies analysed second by second, confirms that these shot-related features do not appear as random patterns in movies from the same director. For feature extraction we adopt methods based on both conventional and deep learning techniques. Our findings suggest that feature sequential patterns, i.e. how features evolve in time, are at least as important as the related feature distributions. To the best of our knowledge this is the first study dealing with automatic attribution of movie authorship, which opens up interesting lines of cross-disciplinary research on the impact of style on the aesthetic and emotional effects on the viewers

    Biologically Interpretable, Integrative Deep Learning for Cancer Survival Analysis

    Get PDF
    Identifying complex biological processes associated to patients\u27 survival time at the cellular and molecular level is critical not only for developing new treatments for patients but also for accurate survival prediction. However, highly nonlinear and high-dimension, low-sample size (HDLSS) data cause computational challenges in survival analysis. We developed a novel family of pathway-based, sparse deep neural networks (PASNet) for cancer survival analysis. PASNet family is a biologically interpretable neural network model where nodes in the network correspond to specific genes and pathways, while capturing nonlinear and hierarchical effects of biological pathways associated with certain clinical outcomes. Furthermore, integration of heterogeneous types of biological data from biospecimen holds promise of improving survival prediction and personalized therapies in cancer. Specifically, the integration of genomic data and histopathological images enhances survival predictions and personalized treatments in cancer study, while providing an in-depth understanding of genetic mechanisms and phenotypic patterns of cancer. Two proposed models will be introduced for integrating multi-omics data and pathological images, respectively. Each model in PASNet family was evaluated by comparing the performance of current cutting-edge models with The Cancer Genome Atlas (TCGA) cancer data. In the extensive experiments, PASNet family outperformed the benchmarking methods, and the outstanding performance was statistically assessed. More importantly, PASNet family showed the capability to interpret a multi-layered biological system. A number of biological literature in GBM supported the biological interpretation of the proposed models. The open-source software of PASNet family in PyTorch is publicly available at https://github.com/DataX-JieHao

    Expanding the theoretical framework of reservoir computing

    Get PDF

    ๊นŠ์€ ์‹ ๊ฒฝ๋ง์„ ์ด์šฉํ•œ ๊ฐ•์ธํ•œ ํŠน์ง• ํ•™์Šต

    Get PDF
    ํ•™์œ„๋…ผ๋ฌธ (๋ฐ•์‚ฌ)-- ์„œ์šธ๋Œ€ํ•™๊ต ๋Œ€ํ•™์› : ์ „๊ธฐยท์ปดํ“จํ„ฐ๊ณตํ•™๋ถ€, 2016. 8. ์œค์„ฑ๋กœ.์ตœ๊ทผ ๊ธฐ๊ณ„ ํ•™์Šต์˜ ๋ฐœ์ „์œผ๋กœ ์ธ๊ณต ์ง€๋Šฅ์€ ์šฐ๋ฆฌ์—๊ฒŒ ํ•œ ๊ฑธ์Œ ๋” ๊ฐ€๊นŒ์ด ๋‹ค๊ฐ€์˜ค๊ฒŒ ๋˜์—ˆ๋‹ค. ํŠนํžˆ ์ž์œจ ์ฃผํ–‰์ด๋‚˜ ๊ฒŒ์ž„ ํ”Œ๋ ˆ์ด ๋“ฑ ์ตœ์‹  ์ธ๊ณต ์ง€๋Šฅ ํ”„๋ ˆ์ž„์›Œํฌ๋“ค์— ์žˆ์–ด์„œ, ๋”ฅ ๋Ÿฌ๋‹์ด ์ค‘์š”ํ•œ ์—ญํ• ์„ ํ•˜๊ณ  ์žˆ๋Š” ์ƒํ™ฉ์ด๋‹ค. ๋”ฅ ๋Ÿฌ๋‹์ด๋ž€ multi-layered neural networks ๊ณผ ๊ด€๋ จ๋œ ๊ธฐ์ˆ ๋“ค์„ ์ด์นญํ•˜๋Š” ์šฉ์–ด๋กœ์„œ, ๋ฐ์ดํ„ฐ์˜ ์–‘์ด ๊ธ‰์†ํ•˜๊ฒŒ ์ฆ๊ฐ€ํ•˜๋ฉฐ, ์‚ฌ์ „ ์ง€์‹๋“ค์ด ์ถ•์ ๋˜๊ณ , ํšจ์œจ์ ์ธ ํ•™์Šต ์•Œ๊ณ ๋ฆฌ์ฆ˜๋“ค์ด ๊ฐœ๋ฐœ๋˜๋ฉฐ, ๊ณ ๊ธ‰ ํ•˜๋“œ์›จ์–ด๋“ค์ด ๋งŒ๋“ค์–ด์ง์— ๋”ฐ๋ผ ๋น ๋ฅด๊ฒŒ ๋ณ€ํ™”ํ•˜๊ณ  ์žˆ๋‹ค. ํ˜„์žฌ ๋”ฅ ๋Ÿฌ๋‹์€ ๋Œ€๋ถ€๋ถ„์˜ ์ธ์‹ ๋ฌธ์ œ์—์„œ ์ตœ์ฒจ๋‹จ ๊ธฐ์ˆ ๋กœ ํ™œ์šฉ๋˜๊ณ  ์žˆ๋‹ค. ์—ฌ๋Ÿฌ ๋ ˆ์ด์–ด๋กœ ๊ตฌ์„ฑ๋œ ๊นŠ์€ ์‹ ๊ฒฝ๋ง์€ ๋งŽ์€ ์–‘์˜ ํŒŒ๋ผ๋ฏธํ„ฐ๋ฅผ ํ•™์Šตํ•˜๊ธฐ ๋•Œ๋ฌธ์—, ๋ฐฉ๋Œ€ํ•œ ํŒŒ๋ผ๋ฏธํ„ฐ ์ง‘ํ•ฉ ์†์—์„œ ์ข‹์€ ํ•ด๋ฅผ ํšจ์œจ์ ์œผ๋กœ ์ฐพ์•„๋‚ด๋Š” ๊ฒƒ์ด ์ค‘์š”ํ•˜๋‹ค. ๋ณธ ๋…ผ๋ฌธ์—์„œ๋Š” ๊นŠ์€ ์‹ ๊ฒฝ๋ง์˜ ์„ธ ๊ฐ€์ง€ ์ด์Šˆ์— ๋Œ€ํ•ด ์ ‘๊ทผํ•˜๋ฉฐ, ๊ทธ๊ฒƒ๋“ค์„ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•œ regularization ๊ธฐ๋ฒ•๋“ค์„ ์ œ์•ˆํ•œ๋‹ค. ์ฒซ์งธ๋กœ, ์‹ ๊ฒฝ๋ง ๊ตฌ์กฐ๋Š” adversarial perturbations ์ด๋ผ๋Š” ๋‚ด์žฌ์ ์ธ blind spots ๋“ค์— ๋งŽ์ด ๋…ธ์ถœ๋˜์–ด ์žˆ๋‹ค. ์ด๋Ÿฌํ•œ adversarial perturbations ์— ๊ฐ•์ธํ•œ ์‹ ๊ฒฝ๋ง์„ ๋งŒ๋“ค๊ธฐ ์œ„ํ•˜์—ฌ, ํ•™์Šต ์ƒ˜ํ”Œ๊ณผ ๊ทธ๊ฒƒ์˜ adversarial perturbations ์™€์˜ ์ฐจ์ด๋ฅผ ์ตœ์†Œํ™”ํ•˜๋Š” manifold loss term์„ ๋ชฉ์  ํ•จ์ˆ˜์— ์ถ”๊ฐ€ํ•˜์˜€๋‹ค. ๋‘˜์งธ๋กœ, restricted Boltzmann machines ์˜ ํ•™์Šต์— ์žˆ์–ด์„œ, ์ƒ๋Œ€์ ์œผ๋กœ ์ž‘์€ ํฌ๊ธฐ๋ฅผ ๊ฐ€์ง€๋Š” ํด๋ž˜์Šค๋ฅผ ํ•™์Šตํ•˜๋Š” ๋ฐ์— ๊ธฐ์กด์˜ contrastive divergence ์•Œ๊ณ ๋ฆฌ์ฆ˜์€ ํ•œ๊ณ„์ ์„ ๊ฐ€์ง€๊ณ  ์žˆ์—ˆ๋‹ค. ๋ณธ ๋…ผ๋ฌธ์—์„œ๋Š” ์ž‘์€ ํด๋ž˜์Šค์— ๋” ๋†’์€ ํ•™์Šต ๊ฐ€์ค‘์น˜๋ฅผ ๋ถ€์—ฌํ•˜๋Š” boosting ๊ฐœ๋…๊ณผ categorical features๋ฅผ ๊ฐ€์ง„ ๋ฐ์ดํ„ฐ์— ์ ํ•ฉํ•œ ์ƒˆ๋กœ์šด regularization ๊ธฐ๋ฒ•์„ ์กฐํ•ฉํ•˜์—ฌ ๊ธฐ์กด์˜ ํ•œ๊ณ„์ ์— ์ ‘๊ทผํ•˜์˜€๋‹ค. ๋งˆ์ง€๋ง‰์œผ๋กœ, ์‹ ๊ฒฝ๋ง์˜ ํŒŒ๋ผ๋ฏธํ„ฐ๋ฅผ ํ•™์Šตํ•˜๊ธฐ์— ์ถฉ๋ถ„ํ•˜์ง€ ์•Š์€ ๋ฐ์ดํ„ฐ๊ฐ€ ์ฃผ์–ด์ง„ ๊ฒฝ์šฐ, ๋” ์ •๊ตํ•œ data augmentation ๊ธฐ๋ฒ•์„ ๋‹ค๋ฃฌ๋‹ค. ์ƒ˜ํ”Œ์˜ ์ฐจ์›์ด ๋งŽ์„์ˆ˜๋ก, ๋ฐ์ดํ„ฐ ์ƒ์„ฑ์˜ ๊ธฐ์ €์— ๊น”๋ ค์žˆ๋Š” ์‚ฌ์ „ ์ง€์‹์„ ํ™œ์šฉํ•˜์—ฌ augmentation์„ ํ•˜๋Š” ๊ฒƒ์ด ๋”์šฑ ๋” ํ•„์š”ํ•˜๋‹ค. ๋‚˜์•„๊ฐ€, ๋ณธ ๋…ผ๋ฌธ์€ junction splicing signals ํ•™์Šต์„ ์œ„ํ•œ ์ฒซ ๋ฒˆ์งธ ๊นŠ์€ ์‹ ๊ฒฝ๋ง ๋ชจ๋ธ๋ง ๊ฒฐ๊ณผ๋ฅผ ์ œ์‹œํ•˜๊ณ  ์žˆ๋‹ค. Junction prediction ๋ฌธ์ œ๋Š” positive ์ƒ˜ํ”Œ ์ˆ˜๊ฐ€ ๋งค์šฐ ์ ์–ด ํŒจํ„ด ๋ชจ๋ธ๋ง์ด ํž˜๋“ค๋ฉฐ, ์ด๋Š” ์ƒ๋ช…์ •๋ณดํ•™ ๋ถ„์•ผ์—์„œ ๊ฐ€์žฅ ์ค‘์š”ํ•œ ๋ฌธ์ œ ์ค‘ ํ•˜๋‚˜๋กœ์„œ, ์ „์ฒด gene expression process ๋ฅผ ์ดํ•ดํ•˜๋Š” ์ฒซ ๊ฑธ์Œ์ด๋ผ๊ณ  ํ•  ์ˆ˜ ์žˆ๋‹ค. ์š”์•ฝํ•˜๋ฉด, ๋ณธ ๋…ผ๋ฌธ์€ ๋”ฅ ๋Ÿฌ๋‹์œผ๋กœ ์ด๋ฏธ์ง€์™€ ๋Œ€์šฉ๋Ÿ‰ ์œ ์ „์ฒด ๋ฐ์ดํ„ฐ๋ฅผ ์œ„ํ•œ ํšจ๊ณผ์ ์ธ ํ‘œํ˜„๋ฒ•์„ ํ•™์Šตํ•  ์ˆ˜ ์žˆ๋Š” regularization ๊ธฐ๋ฒ•๋“ค์„ ์ œ์•ˆํ•˜์˜€์œผ๋ฉฐ, ์œ ๋ช…ํ•œ ๋ฒค์น˜๋งˆํฌ ๋ฐ์ดํ„ฐ์™€ biomedical imaging ๋ฐ์ดํ„ฐ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๊ทธ ์‹คํšจ์„ฑ์„ ๊ฒ€์ฆํ•˜์˜€๋‹ค.Recent advances in machine learning continue to bring us closer to artificial intelligence. In particular, deep learning plays a key role in cutting-edge frameworks such as autonomous driving and game playing. Deep learning refers to a class of multi-layered neural networks, which is rapidly evolving as the amount of data increases, prior knowledge builds up, efficient training schemes are being developed, and high-end hardwares are being build. Currently, deep learning is a state-of-the-art technique for most recognition tasks. As deep neural networks learn many parameters, there has been a variety of attempts to obtain reasonable solutions over a wide search space. In this dissertation, three issues in deep learning are discussed and approaches to solve them with regularization techniques are suggested. First, deep neural networks expose the problem of intrinsic blind spots called adversarial perturbations. Thus, we must construct neural networks that resist the directions of adversarial perturbations by introducing an explicit loss term to minimize the differences between the original and adversarial samples. Second, training restricted Boltzmann machines show limited performance when handling minority samples in class-imbalanced datasets. Our approach addresses this limitation and is combined with a new regularization concept for datasets that have categorical features. Lastly, insufficient data handling is required to be more sophisticated when deep networks learn numerous parameters. Given high-dimensional samples, we must augment datasets with adequate prior knowledge to estimate a high-dimensional distribution. Furthermore, this dissertation shows the first application of deep belief networks to identifying junction splicing signals. Junction prediction is one of the major problems in the field of bioinformatics, and is a starting point to understanding the entire gene expression process. In summary, this dissertation proposes a set of deep learning regularization schemes that can learn the meaningful representation underlying large-scale genomic datasets and image datasets. The effectiveness of these methods was confirmed with a number of experimental studies.Chapter 1 Introduction 1 1.1 Deep neural networks 1 1.2 Issue 1: adversarial examples handling 3 1.3 Issue 2: class-imbalance handling 5 1.4 Issue 3: insufficient data handling 5 1.5 Organization 6 Chapter 2 Background 10 2.1 Basic operations for deep networks 10 2.2 History of deep networks 12 2.3 Modern deep networks 14 2.3.1 Contrastive divergence 16 2.3.2 Deep manifold learning 18 Chapter 3 Adversarial examples handling 20 3.1 Introduction 20 3.2 Methods 21 3.2.1 Manifold regularized networks 21 3.2.2 Generation of adversarial examples 25 3.3 Results and discussion 26 3.3.1 Improved classification performance 28 3.3.2 Disentanglement and generalization 30 3.4 Summary 33 Chapter 4 Class-imbalance handling 35 4.1 Introduction 35 4.1.1 Numerical interpretation of DNA sequences 37 4.1.2 Review of junction prediction problem 41 4.2 Methods 44 4.2.1 Boosted contrastive divergence with categorical gradients 44 4.2.2 Stacking and fine-tuning 46 4.2.3 Initialization and parameter setting 47 4.3 Results and discussion 47 4.3.1 Experiment preparation 47 4.3.2 Improved prediction performance and runtime 49 4.3.3 More robust prediction by proposed approach 51 4.3.4 Effects of regularization on performance 53 4.3.5 Efficient RBM training by boosted CD 54 4.3.6 Identification of non-canonical splice sites 57 4.4 Summary 58 Chapter 5 Insufficient data handling 60 5.1 Introduction 60 5.2 Backgrounds 62 5.2.1 Understanding comets 62 5.2.2 Assessing DNA damage from tail shape 65 5.2.3 Related image processing techniques 66 5.3 Methods 68 5.3.1 Preprocessing 70 5.3.2 Binarization 70 5.3.3 Filtering and overlap correction 72 5.3.4 Characterization and classification 75 5.4 Results and discussion 76 5.4.1 Test data preparation 76 5.4.2 Binarization 77 5.4.3 Robust identification of comets 79 5.4.4 Classification 81 5.4.5 More accurate characterization by DeepComet 82 5.5 Summary 85 Chapter 6 Conclusion 87 6.1 Dissertation summary 87 6.2 Future work 89 Bibliography 91Docto

    The Shallow and the Deep:A biased introduction to neural networks and old school machine learning

    Get PDF
    The Shallow and the Deep is a collection of lecture notes that offers an accessible introduction to neural networks and machine learning in general. However, it was clear from the beginning that these notes would not be able to cover this rapidly changing and growing field in its entirety. The focus lies on classical machine learning techniques, with a bias towards classification and regression. Other learning paradigms and many recent developments in, for instance, Deep Learning are not addressed or only briefly touched upon.Biehl argues that having a solid knowledge of the foundations of the field is essential, especially for anyone who wants to explore the world of machine learning with an ambition that goes beyond the application of some software package to some data set. Therefore, The Shallow and the Deep places emphasis on fundamental concepts and theoretical background. This also involves delving into the history and pre-history of neural networks, where the foundations for most of the recent developments were laid. These notes aim to demystify machine learning and neural networks without losing the appreciation for their impressive power and versatility
    • โ€ฆ
    corecore