Resiliency in Deep Convolutional Neural Networks

The enormous success and popularity of deep convolutional neural networks for object detection has prompted their deployment in various real-world applications. However, their performance in the presence of hardware faults or damage that could occur in the field has not been studied. This thesis explores the resiliency of six popular network architectures for image classification, AlexNet, VGG16, ResNet, GoogleNet, SqueezeNet and YOLO9000, when subjected to various degrees of failures. We introduce failures in a deep network by dropping a percentage of weights at each layer. We then assess the effects of these failures on classification performance. We find the fitness of the weights and then dropped from least fit to most fit weights. Finally, we determine the ability of the network to self-heal and recover its performance by retraining its healthy portions after partial damage. We try different methods to re-train the healthy portion by varying the optimizer. We also try to find the time and resources required for re-training. We also reduce the number of parameters in GoogleNet, VGG16 to the size of SqueezeNet and re-trained with varying percentage of dataset. This can be used as a network pruning method

Learning representations for binary-classification without backpropagation

The family of feedback alignment (FA) algorithms aims to provide a more biologically motivated alternative to backpropagation (BP), by substituting the computations that are unrealistic to be implemented in physical brains. While FA algorithms have been shown to work well in practice, there is a lack of rigorous theory proofing their learning capabilities. Here we introduce the first feedback alignment algorithm with provable learning guarantees. In contrast to existing work, we do not require any assumption about the size or depth of the network except that it has a single output neuron, i.e., such as for binary classification tasks. We show that our FA algorithm can deliver its theoretical promises in practice, surpassing the learning performance of existing FA methods and matching backpropagation in binary classification tasks. Finally, we demonstrate the limits of our FA variant when the number of output neurons grows beyond a certain quantity

Comparison of effects on subjective intelligibility and quality of speech in babble for two algorithms: A deep recurrent neural network and spectral subtraction.

The effects on speech intelligibility and sound quality of two noise-reduction algorithms were compared: a deep recurrent neural network (RNN) and spectral subtraction (SS). The RNN was trained using sentences spoken by a large number of talkers with a variety of accents, presented in babble. Different talkers were used for testing. Participants with mild-to-moderate hearing loss were tested. Stimuli were given frequency-dependent linear amplification to compensate for the individual hearing losses. A paired-comparison procedure was used to compare all possible combinations of three conditions. The conditions were: speech in babble with no processing (NP) or processed using the RNN or SS. In each trial, the same sentence was played twice using two different conditions. The participants indicated which one was better and by how much in terms of speech intelligibility and (in separate blocks) sound quality. Processing using the RNN was significantly preferred over NP and over SS processing for both subjective intelligibility and sound quality, although the magnitude of the preferences was small. SS processing was not significantly preferred over NP for either subjective intelligibility or sound quality. Objective computational measures of speech intelligibility predicted better intelligibility for RNN than for SS or NP

Neural Network Architectures and Ensembles for Packet Classification: Addressing Visibility, Security and Quality of Service Challenges in Communication Networks

Increasingly researchers are turning to machine learning techniques such as artificial neural networks (ANN) to address communication network research challenges in the areas of enhanced security, quality of service, visibility and control. Central to each is the need to classify packets. Determining an effective architecture for the artificial neural network is more difficult because traditional techniques such as principal component analysis (PCA) show reduced effectiveness. Presented are the techniques for preprocessing datasets and selecting input traffic features for the multi-layer perceptron (MLP) architecture. This methodology achieves classification accuracy above 99%. An investigation into neural network architectures revealed the optimal structure and parameters for communication packet classification. This work also studies optimization algorithms with completely balanced datasets and provides performance criteria for training time and accuracy. The application of MLPs to security challenges is also investigated. Port scans are a persistent problem on contemporary communication networks. Sequential MLPs are investigated to classify packets and determine TCP packet type. Following classification, analysis is performed in order to discover scan attempts. Neural networks can be used to successfully classify general packet traffic and more complex TCP classes at rates that are above 99\%. The proposed methodology achieves accurate scan detection without having to utilize an intrusion detection system. In order to harness the power of Convolutional Neural Networks (CNNs), the conversion of packets to images is investigated. Additionally, a sequence of packets are combined into larger images to gain insight into conversations, exchanges, losses and threats. The use of this technique to identify potential latency problems is demonstrated. This approach of using contemporary network traffic and convolutional neural networks has success rate for individual packets exceeding 99%. Larger images achieve the same high level of accuracy. Finally, neural network ensembles are researched that reach 100% accuracy for packet classification. Ensembles are also studied to accurately predict Mean Opinion Score for voice traffic and explored for their use in combating adversarial attacks against the source data

Otimização de redes convolucionais para classificação de imagens

As redes convolucionais têm demonstrado eficácia na resolução de diversos tipos de problemas, nomeadamente classificação de imagens, reconhecimento ou localização de objetos, reconhecimento de som, entre outros. A elevada robustez destas redes permite a sua difusão em áreas de extrema importância para a sociedade, tal como a área biomédica, que inclui um vasto conjunto de tarefas de análise de imagens e diagnóstico clínico. Contudo, o sucesso destes modelos está dependente da identificação da estrutura e dos valores de outros hiperparâmetros, que melhor se ajustam à resolução de um problema específico, sendo que esta tarefa requer elevado esforço computacional e conhecimento pericial. De modo a reduzir estas dificuldades, este trabalho propõe uma metodologia de otimização autónoma dos hiperparâmetros de diferentes arquiteturas convolucionais. A metodologia baseia-se na utilização do algoritmo de inteligência coletiva, Particle Swarm Optimization (PSO), aplicado na otimização de quatro conhecidas arquiteturas convolucionais: AlexNet, VGGNet, ResNet e DenseNet e para resolução de três problemas de diagnóstico médico. Procurando-se identificar a solução que proporciona o melhor desempenho com a menor complexidade possível. Os resultados obtidos demonstraram a eficácia e a rapidez do PSO na identificação de soluções efetivas, sendo alcançados resultados superiores comparativamente a outras abordagens, em duas das três benchmarks em estudo. A técnica ensemble proposta permitiu a obtenção de um F1Score macroAVG de 91.1% e de 96.6%, respetivamente, para as benchmarks Breast Histopathology e Colorectal Histopathology. Os modelos estão disponíveis numa plataforma web que possibilita a realização de diagnósticos de imagens biomédicas, através do uso e da partilha de modelos convolucionais na plataforma. A plataforma demonstrou ser extremamente eficiente e célere na resposta ao diagnóstico de imagens, sendo obtidas as previsões num intervalo inferior a 10 segundos. O seu acesso é público e está disponível no repositório https://github.com/bundasmanu/ProjetoMestrado

Compensação digital de distorções da fibra em sistemas de comunicação óticos de longa distância

The continuous increase of traffic demand in long-haul communications motivated the network operators to look for receiver side techniques to mitigate the nonlinear effects, resulting from signal-signal and signal-noise interaction, thus pushing the current Capacity boundaries. Machine learning techniques are a very hot-topic with given proofs in the most diverse applications. This dissertation aims to study nonlinear impairments in long-haul coherent optical links and the current state of the art in DSP techniques for impairment mitigation as well as the integration of machine learning strategies in optical networks. Starting with a simplified fiber model only impaired by ASE noise, we studied how to integrate an ANN-based symbol estimator into the signal pipeline, enabling to validate the implementation by matching the theoretical performance. We then moved to nonlinear proof of concept with the incorporation of NLPN in the fiber link. Finally, we evaluated the performance of the estimator under realistic simulations of Single and Multi- Channel links in both SSFM and NZDSF fibers. The obtained results indicate that even though it may be hard to find the best architecture, Nonlinear Symbol Estimator networks have the potential to surpass more conventional DSP strategies.O aumento contínuo de tráfego nas comunicações de longo-alcance motivou os operadores de rede a procurar técnicas do lado do receptor para atenuar os efeitos não lineares resultantes da interacção sinal-sinal e sinal-ruído, alargando assim os limites da capacidade do sistema. As técnicas de aprendizagem-máquina são um tópico em ascenção com provas dadas nas mais diversas aplicações e setores. Esta dissertação visa estudar as principais deficiências nas ligações de longo curso e o actual estado da arte em técnicas de DSP para mitigação das mesmas, bem como a integração de estratégias de aprendizagem-máquina em redes ópticas. Começando com um modelo simplificado de fibra apenas perturbado pelo ruído ASE, estudámos como integrar um estimador de símbolos baseado em ANN na cadeia do prodessamento de sinal, conseguindo igualar o desempenho teórico. Procedemos com uma prova de conceito perante não linearidades com a incorporação do ruído de fase não linear na propagação. Finalmente, avaliamos o desempenho do estimador com simulações realistas de links Single e Multi canal tanto em fibras SSFM como NZDSF. Os resultados obtidos indicam que apesar da dificuldade de encontrar a melhor arquitectura, a estimação não linear baseada em redes neuronais têm o potencial para ultrapassar estratégias DSP mais convencionais.Mestrado em Engenharia Eletrónica e Telecomunicaçõe

An efficient and effective convolutional neural network for visual pattern recognition

Convolutional neural networks (CNNs) are a variant of deep neural networks (DNNs) optimized for visual pattern recognition, which are typically trained using first order learning algorithms, particularly stochastic gradient descent (SGD). Training deeper CNNs (deep learning) using large data sets (big data) has led to the concept of distributed machine learning (ML), contributing to state-of-the-art performances in solving computer vision problems. However, there are still several outstanding issues to be resolved with currently defined models and learning algorithms. Propagations through a convolutional layer require flipping of kernel weights, thus increasing the computation time of a CNN. Sigmoidal activation functions suffer from gradient diffusion problem that degrades training efficiency, while others cause numerical instability due to unbounded outputs. Common learning algorithms converge slowly and are prone to hyperparameter overfitting problem. To date, most distributed learning algorithms are still based on first order methods that are susceptible to various learning issues. This thesis presents an efficient CNN model, proposes an effective learning algorithm to train CNNs, and map it into parallel and distributed computing platforms for improved training speedup. The proposed CNN consists of convolutional layers with correlation filtering, and uses novel bounded activation functions for faster performance (up to 1.36x), improved learning performance (up to 74.99% better), and better training stability (up to 100% improvement). The bounded stochastic diagonal Levenberg-Marquardt (B-SDLM) learning algorithm is proposed to encourage fast convergence (up to 5.30% faster and 35.83% better than first order methods) while having only a single hyperparameter. B-SDLM also supports mini-batch learning mode for high parallelism. Based on known previous works, this is among the first successful attempts of mapping a stochastic second order learning algorithm to be deployed in distributed ML platforms. Running the distributed B-SDLM on a 16- core cluster achieves up to 12.08x and 8.72x faster to reach a certain convergence state and accuracy on the Mixed National Institute of Standards and Technology (MNIST) data set. All three complex case studies tested with the proposed algorithms give comparable or better classification accuracies compared to those provided in previous works, but with better efficiency. As an example, the proposed solutions achieved 99.14% classification accuracy for the MNIST case study, and 100% for face recognition using AR Purdue data set, which proves the feasibility of proposed algorithms in visual pattern recognition tasks

Detec??o de patologias lar?ngeas por meio da an?lise de sinais de voz utilizando Deep Neural Networks

A fala ? o principal mecanismo natural de comunica??o entre seres humanos.O sistema de forma??o e transmiss?o natural da voz, principal elemento da fala, ? comprometido pelo surgimento de patologias lar?ngeas. Esta pesquisa trata da aplica??o de classificadores baseados em redes neurais profundas (Deep Neural Networks - DNNs) na discrimina??o entre sinais de vozes saud?veis e de vozes afetadas pelas patologias lar?ngeas organofuncionais edema de Reinke, carcinoma, leocoplasia, p?lipos e a paralisia das pregas vocais, de origem neurol?gica. A metodologia proposta ? baseada na an?lise do comportamento din?mico do sinal de voz avaliado, dispensando medidas ou aplica??es de t?cnicas comumente usadas na extra??o de caracter?sticas. Foi investigado o uso de DNNs com 04,05 e 06 camadas com 200 neur?nios ocultos ativados pela fun??o unidade linear retificada (Rectified LinearUnit - ReLU),um neur?nio na camada de sa?da,ativado pela fun??o sigmoide e uma camada de entrada que recebe os 400 dados que comp?e cada segmento extra?do do sinal de voz avaliado. No total, 07 algoritmos de aprendizagem, utilizando como fun??o custo a entropia cruzada bin?ria (Binary Cross-entropy), foram avaliados individualmente para o treinamento de cada DNN. Os sinais de voz utilizados nesta pesquisa foram extra?dos da base de dados Saarbruecken Voice Database (SVD), desenvolvida na Alemanha. Da base, foram selecionados 640 sinais de voz da vogal sustentada /a/, sendo 320 sinais de vozes saud?veis e 320 afetados por patologias lar?ngeas. A discrimina??o foi realizada por classes,sendo: a classe saud?vel; a classe patologias, composta por todos os sinais patol?gicos selecionados da base SVD; a classe das vozes afetadas apenas por patologias lar?ngeas organofuncionais; e, por fim,a classe de sinais de voz afetados apenas por paralisia das pregas vocais, compondo a categoria de patologia lar?ngea de origem neurol?gica. Foram considerados 04 casos de classifica??o entre os sinais de voz selecionados, sendo eles: saud?vel x patologias, saud?vel x patologias organofuncionais, saud?vel x paralisia das pregas vocais e patologias organofuncionais x paralisia das pregas vocais. Para cada caso discriminativo, 28 classificadores foram implementados e avaliados por meio do F1 score e pelo coeficiente de correla??o de Mathews (CCM) (aplicado apenas na discrimina??o entre as classes patol?gicas), e pelas m?tricas acur?cia, sensibilidade e especificidade. Al?m disso, foram investigados os efeitos da inclus?o de taxas de sobreposi??o (0%,25%,50% e 75%) aplicadas durante a extra??o dos segmentos. A t?cnica de valida??o cruzada k- fold, com k = 10, foi implementada nesta pesquisa para sele??o dos conjuntos de dados de treino e teste. Os resultados indicam que o m?todo proposto possui o seu melhor desempenho na discrimina??o entre vozes saud?veis e afetadas por paralisia das pregas vocais, com base na detec??o de segmentos do sinal de voz sem taxa de sobreposi??o,utilizando o classificador com 4 camadas ocultas,treinado pelo algoritmo de aprendizagem Adadelta,no qual foram obtidos ap?s a valida??o cruzada 88,68 ?3,04% para acur?cia, 92,04 ? 5,82% para sensibilidade, 85,33 ? 6,53% para especificidade e F1 score igual 0,89. Conclui-se que ? poss?vel discriminar vozes saud?veis e afetadas por patologias lar?ngeas, com base na an?lise do comportamento din?mico de segmentos do sinal de voz utilizando DNNs.Instituto Federal da Para?b