5 research outputs found

    Training Modular Neural Networks with Marquardt-Levenberg Algorithm

    Get PDF

    Sensitivity of neural networks to random change with perturbed weights and biases

    Get PDF
    Computer Scienc

    Regularization and Compression of Deep Neural Networks

    Get PDF
    Deep neural networks (DNN) are the state-of-the-art machine learning models outperforming traditional machine learning methods in a number of domains from vision and speech to natural language understanding and autonomous control. With large amounts of data becoming available, the task performance of DNNs in these domains predictably scales with the size of the DNNs. However, in data-scarce scenarios, large DNNs overfit to the training dataset resulting in inferior performance. Additionally, in scenarios where enormous amounts of data is available, large DNNs incur large inference latencies and memory costs. Thus, while imperative for achieving state-of-the-art performances, large DNNs require large amounts of data for training and large computational resources during inference. These two problems could be mitigated by sparsely training large DNNs. Imposing sparsity constraints during training limits the capacity of the model to overfit to the training set while still being able to obtain good generalization. Sparse DNNs have most of their weights close to zero after training. Therefore, most of the weights could be removed resulting in smaller inference costs. To effectively train sparse DNNs, this thesis proposes two new sparse stochastic regularization techniques called Bridgeout and Sparseout. Furthermore, Bridgeout is used to prune convolutional neural networks for low-cost inference. Bridgeout randomly perturbs the weights of a parametric model such as a DNN. It is theoretically shown that Bridgeout constrains the weights of linear models to a sparse subspace. Empirically, Bridgeout has been shown to perform better, on image classification tasks, than state-of-the-art DNNs when the data is limited. Sparseout is an activations counter-part of Bridgeout, operating on the outputs of the neurons instead of the weights of the neurons. Theoretically, Sparseout has been shown to be a general case of the commonly used Dropout regularization method. Empirical evidence suggests that Sparseout is capable of controlling the level of activations sparsity in neural networks. This flexibility allows Sparseout to perform better than Dropout on image classification and language modelling tasks. Furthermore, using Sparseout, it is found that activation sparsity is beneficial to recurrent neural networks for language modeling but densification of activations favors convolutional neural networks for image classification. To address the problem of high computational cost during inference, this thesis evaluates Bridgeout for pruning convolutional neural networks (CNN). It is shown that recent CNN architectures such as VGG, ResNet and Wide-ResNet trained with Bridgeout are more robust to one-shot filter pruning compared to non-sparse stochastic regularization

    A Biologically-Inspired Neural Network Architecture for Image Processing

    Get PDF
    This thesis project included a literature survey of biological and artificial neural network research followed by development and testing of high- order and image recognition hierarchical neural network algorithms. Following training, performance testing of second-order and third-order networks yielded maximum accuracies comparable to those achieved by multilayer perceptron classifiers operating on test data sets. Several versions of an image classification algorithm were tested for learning performance using pixel data from forward-looking infrared (FLIR) images of tanks, trucks, target boards, and clutter. Employing the biologically-motivated Lambertization and contrast normalization of pixel windows, correlations with multiple Gabor function wavelets, and a 'phase synchronizing' local averaging routine, the image classification network extracted data features. Different network versions fed the extracted features to varying output classification schemes. To improve separation of problem classes, recommendations were made for varying the parameters of the Gabor function wavelets and modifying the phase synchronization scheme to extract more suitable features from image pixel data.http://archive.org/details/abiologicallyins1094530625NAApproved for public release; distribution is unlimited.Approved for public release; distribution is unlimited
    corecore