Optimization of convolutional neural networks for image classification using genetic algorithms and bayesian optimization

Notwithstanding the recent successes of deep convolutional neural networks for classification tasks, they are sensitive to the selection of their hyperparameters, which impose an exponentially large search space on modern convolutional models. Traditional hyperparameter selection methods include manual, grid, or random search, but these require expert knowledge or are computationally burdensome. Divergently, Bayesian optimization and evolutionary inspired techniques have surfaced as viable alternatives to the hyperparameter problem. Thus, an alternative hybrid approach that combines the advantages of these techniques is proposed. Specifically, the search space is partitioned into discrete-architectural, and continuous and categorical hyperparameter subspaces, which are respectively traversed by a stochastic genetic search, followed by a genetic-Bayesian search. Simulations on a prominent image classification task reveal that the proposed method results in an overall classification accuracy improvement of 0.87% over unoptimized baselines, and a greater than 97% reduction in computational costs compared to a commonly employed brute force approach.Electrical and Mining EngineeringM. Tech. (Electrical Engineering

Comparision Of Adversarial And Non-Adversarial LSTM Music Generative Models

Algorithmic music composition is a way of composing musical pieces with minimal to no human intervention. While recurrent neural networks are traditionally applied to many sequence-to-sequence prediction tasks, including successful implementations of music composition, their standard supervised learning approach based on input-to-output mapping leads to a lack of note variety. These models can therefore be seen as potentially unsuitable for tasks such as music generation. Generative adversarial networks learn the generative distribution of data and lead to varied samples. This work implements and compares adversarial and non-adversarial training of recurrent neural network music composers on MIDI data. The resulting music samples are evaluated by human listeners, their preferences recorded. The evaluation indicates that adversarial training produces more aesthetically pleasing music.Comment: Submitted to a 2023 conference, 20 pages, 13 figure

Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model

Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variations to result in networks learning unique feature sets from the data. Using public model libraries comprising thousands of models trained on canonical datasets like ImageNet, we observe that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other -- independent of overall performance. Given any arbitrary pairing of pretrained models and no external rankings (such as separate test sets, e.g. due to data privacy), we investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation -- a task made particularly difficult as additional knowledge can be contained in stronger, equiperformant or weaker models. Yet facilitating robust transfer in scenarios agnostic to pretrained model pairings would unlock auxiliary gains and knowledge fusion from any model repository without restrictions on model and problem specifics - including from weaker, lower-performance models. This work therefore provides an initial, in-depth exploration on the viability of such general-purpose knowledge transfer. Across large-scale experiments, we first reveal the shortcomings of standard knowledge distillation techniques, and then propose a much more general extension through data partitioning for successful transfer between nearly all pretrained models, which we show can also be done unsupervised. Finally, we assess both the scalability and impact of fundamental model properties on successful model-agnostic knowledge transfer

BLOX : micro neural architecture search benchmark and algorithms

Neural architecture search (NAS) has been successfully used to design numerous high-performance neural networks. However, NAS is typically compute-intensive, so most existing approaches restrict the search to decide the operations and topological structure of a single block only, then the same block is stacked repeatedly to form an end-to-end model. Although such an approach reduces the size of search space, recent studies show that a macro search space, which allows blocks in a model to be different, can lead to better performance. To provide a systematic study of the performance of NAS algorithms on a macro search space, we release Blox – a benchmark that consists of 91k unique models trained on the CIFAR-100 dataset. The dataset also includes runtime measurements of all the models on a diverse set of hardware platforms. We perform extensive experiments to compare existing algorithms that are well studied on cell-based search spaces, with the emerging blockwise approaches that aim to make NAS scalable to much larger macro search spaces. The Blox benchmark and code are available at https://github.com/SamsungLabs/blox

Neural Networks for CollaborativeFiltering

Recommender systems are an integral part of almost all modern e-commerce companies. They contribute significantly to the overall customer satisfaction by helping the user discover new and relevant items, which consequently leads to higher sales and stronger customer retention. It is, therefore, not surprising that large e-commerce shops like Amazon or streaming platforms like Netflix and Spotify even use multiple recommender systems to further increase user engagement. Finding the most relevant items for each user is a difficult task that is critically dependent on the available user feedback information. However, most users typically interact with products only through noisy implicit feedback, such as clicks or purchases, rather than providing explicit information about their preferences, such as product ratings. This usually makes large amounts of behavioural user data necessary to infer accurate user preferences. One popular approach to make the most use of both forms of feedback is called collaborative filtering. Here, the main idea is to compare individual user behaviour with the behaviour of all known users. Although there are many different collaborative filtering techniques, matrix factorization models are among the most successful ones. In contrast, while neural networks are nowadays the state-of-the-art method for tasks such as image recognition or natural language processing, they are still not very popular for collaborative filtering tasks. Therefore, the main focus of this thesis is the derivation of multiple wide neural network architectures to mimic and extend matrix factorization models for various collaborative filtering problems and to gain insights into the connection between these models. The basics of the proposed architecture are wide and shallow feedforward neural networks, which will be established for rating prediction tasks on explicit feedback datasets. These networks consist of large input and output layers, which allow them to capture user and item representation similar to matrix factorization models. By deriving all weight updates and comparing the structure of both models, it is proven that a simplified version of the proposed network can mimic common matrix factorization models: a result that has not been shown, as far as we know, in this form before. Additionally, various extensions are thoroughly evaluated. The new findings of this evaluation can also easily be transferred to other matrix factorization models. This neural network architecture can be extended to be used for personalized ranking tasks on implicit feedback datasets. For these problems, it is necessary to rank products according to individual preferences using only the provided implicit feedback. One of the most successful and influential approaches for personalized ranking tasks is Bayesian Personalized Ranking, which attempts to learn pairwise item rankings and can also be used in combination with matrix factorization models. It is shown, how the introduction of an additional ranking layer forces the network to learn pairwise item rankings. In addition, similarities between this novel neural network architecture and a matrix factorization model trained with Bayesian Personalized Ranking are proven. To the best of our knowledge, this is the first time that these connections have been shown. The state-of-the-art performance of this network is demonstrated in a detailed evaluation. The most comprehensive feedback datasets consist of a mixture of explicit as well as implicit feedback information. Here, the goal is to predict if a user will like an item, similar to rating prediction tasks, even if this user has never given any explicit feedback at all: a problem, that has not been covered by the collaborative filtering literature yet. The network to solve this task is composed out of two networks: one for the explicit and one for the implicit feedback. Additional item features are learned using the implicit feedback, which capture all information necessary to rank items. Afterwards, these features are used to improve the explicit feedback prediction. Both parts of this combined network have different optimization goals, are trained simultaneously and, therefore, influence each other. A detailed evaluation shows that this approach is helpful to improve the network's overall predictive performance especially for ranking metrics

Evaluation Of Automated Eye Blink Artefact Removal Using Stacked Dense Autoencoder

The presence of artefacts in Electroencephalograph (EEG) signals can have a considerable impact on the information they portray. In this comparative study, the automated removal of eye blink artefacts using the constrained latent representation of a stacked dense autoencoders (SDAE) and comparing its ability to that of the manual independent component analysis (ICA) approach was evaluated. A comparative evaluation of 5 stacked dense autoencoder architectures lead to a chosen architecture for which the ability to automatically detect and remove eye blink artefacts were both statistically and humanistically evaluated. The ability of the stacked dense autoencoder was statistically evaluated with the manual approach of ICA using the correlation coefficient, a comparative affect on the SNR using both approaches and a humanistic evaluation using visual inspections of the components of the stacked dense autoencoder reconstruction to that of the post ICA reconstruction where an inverse RMSE allowed for a further statistical evaluation of this comparison. It was found that the stacked dense autoencoder was unable to reconstruct random signal segments in any meaningful capacity when compared to that of ICA

DEEP-AD: The deep learning model for diagnostic classification and prognostic prediction of alzheimer's disease

In terms of context, the aim of this dissertation is to aid neuroradiologists in their clinical judgment regarding the early detection of AD by using DL. To that aim, the system design research methodology is suggested in this dissertation for achieving three goals. The first goal is to investigate the DL models that have performed well at identifying patterns associated with AD, as well as the accuracy so far attained, limitations, and gaps. A systematic review of the literature (SLR) revealed a shortage of empirical studies on the early identification of AD through DL. In this regard, thirteen empirical studies were identified and examined. We concluded that three-dimensional (3D) DL models have been generated far less often and that their performance is also inadequate to qualify them for clinical trials. The second goal is to provide the neuroradiologist with the computer-interpretable information they need to analyze neuroimaging biomarkers. Given this context, the next step in this dissertation is to find the optimum DL model to analyze neuroimaging biomarkers. It has been achieved in two steps. In the first step, eight state-of-the-art DL models have been implemented by training from scratch using end-to-end learning (E2EL) for two binary classification tasks (AD vs. CN and AD vs. stable MCI) and compared by utilizing MRI scans from the publicly accessible datasets of neuroimaging biomarkers. Comparative analysis is carried out by utilizing efficiency-effects graphs, comprehensive indicators, and ranking mechanisms. For the training of the AD vs. sMCI task, the EfficientNet-B0 model gets the highest value for the comprehensive indicator and has the fewest parameters. DenseNet264 performed better than the others in terms of evaluation matrices, but since it has the most parameters, it costs more to train. For the AD vs. CN task by DenseNet264, we achieved 100% accuracy for training and 99.56% accuracy for testing. However, the classification accuracy was still only 82.5% for the AD vs. sMCI task. In the second step, fusion of transfer learning (TL) with E2EL is applied to train the EfficientNet-B0 for the AD vs. sMCI task, which achieved 95.29% accuracy for training and 93.10% accuracy for testing. Additionally, we have also implemented EfficientNet-B0 for the multiclass AD vs. CN vs. sMCI classification task with E2EL to be used in ensemble of models and achieved 85.66% training accuracy and 87.38% testing accuracy. To evaluate the model’s robustness, neuroradiologists must validate the implemented model. As a result, the third goal of this dissertation is to create a tool that neuroradiologists may use at their convenience. To achieve this objective, this dissertation proposes a web-based application (DEEP-AD) that has been created by making an ensemble of Efficient-Net B0 and DenseNet 264 (based on the contribution of goal 2). The accuracy of a DEEP-AD prototype has undergone repeated evaluation and improvement. First, we validated 41 subjects of Spanish MRI datasets (acquired from HT Medica, Madrid, Spain), achieving an accuracy of 82.90%, which was later verified by neuroradiologists. The results of these evaluation studies showed the accomplishment of such goals and relevant directions for future research in applied DL for the early detection of AD in clinical settings.En términos de contexto, el objetivo de esta tesis es ayudar a los neurorradiólogos en su juicio clínico sobre la detección precoz de la AD mediante el uso de DL. Para ello, en esta tesis se propone la metodología de investigación de diseño de sistemas para lograr tres objetivos. El segundo objetivo es proporcionar al neurorradiólogo la información interpretable por ordenador que necesita para analizar los biomarcadores de neuroimagen. Dado este contexto, el siguiente paso en esta tesis es encontrar el modelo DL óptimo para analizar biomarcadores de neuroimagen. Esto se ha logrado en dos pasos. En el primer paso, se han implementado ocho modelos DL de última generación mediante entrenamiento desde cero utilizando aprendizaje de extremo a extremo (E2EL) para dos tareas de clasificación binarias (AD vs. CN y AD vs. MCI estable) y se han comparado utilizando escaneos MRI de los conjuntos de datos de biomarcadores de neuroimagen de acceso público. El análisis comparativo se lleva a cabo utilizando gráficos de efecto-eficacia, indicadores exhaustivos y mecanismos de clasificación. Para el entrenamiento de la tarea AD vs. sMCI, el modelo EfficientNet-B0 obtiene el valor más alto para el indicador exhaustivo y tiene el menor número de parámetros. DenseNet264 obtuvo mejores resultados que los demás en términos de matrices de evaluación, pero al ser el que tiene más parámetros, su entrenamiento es más costoso. Para la tarea AD vs. CN de DenseNet264, conseguimos una accuracy del 100% en el entrenamiento y del 99,56% en las pruebas. Sin embargo, la accuracy de la clasificación fue sólo del 82,5% para la tarea AD vs. sMCI. En el segundo paso, se aplica la fusión del aprendizaje por transferencia (TL) con E2EL para entrenar la EfficientNet-B0 para la tarea AD vs. sMCI, que alcanzó una accuracy del 95,29% en el entrenamiento y del 93,10% en las pruebas. Además, también hemos implementado EfficientNet-B0 para la tarea de clasificación multiclase AD vs. CN vs. sMCI con E2EL para su uso en conjuntos de modelos y hemos obtenido una accuracy de entrenamiento del 85,66% y una precisión de prueba del 87,38%. Para evaluar la solidez del modelo, los neurorradiólogos deben validar el modelo implementado. Como resultado, el tercer objetivo de esta disertación es crear una herramienta que los neurorradiólogos puedan utilizar a su conveniencia. Para lograr este objetivo, esta disertación propone una aplicación basada en web (DEEP-AD) que ha sido creada haciendo un ensemble de Efficient-Net B0 y DenseNet 264 (basado en la contribución del objetivo 2). La accuracy del prototipo DEEP-AD ha sido sometida a repetidas evaluaciones y mejoras. En primer lugar, validamos 41 sujetos de conjuntos de datos de MRI españoles (adquiridos de HT Medica, Madrid, España), logrando una accuracy del 82,90%, que posteriormente fue verificada por neurorradiólogos. Los resultados de estos estudios de evaluación mostraron el cumplimiento de dichos objetivos y las direcciones relevantes para futuras investigaciones en DL, aplicada en la detección precoz de la AD en entornos clínicos.Escuela de DoctoradoDoctorado en Tecnologías de la Información y las Telecomunicacione

Differentiable world programs

L'intelligence artificielle (IA) moderne a ouvert de nouvelles perspectives prometteuses pour la création de robots intelligents. En particulier, les architectures d'apprentissage basées sur le gradient (réseaux neuronaux profonds) ont considérablement amélioré la compréhension des scènes 3D en termes de perception, de raisonnement et d'action. Cependant, ces progrès ont affaibli l'attrait de nombreuses techniques ``classiques'' développées au cours des dernières décennies. Nous postulons qu'un mélange de méthodes ``classiques'' et ``apprises'' est la voie la plus prometteuse pour développer des modèles du monde flexibles, interprétables et exploitables : une nécessité pour les agents intelligents incorporés. La question centrale de cette thèse est : ``Quelle est la manière idéale de combiner les techniques classiques avec des architectures d'apprentissage basées sur le gradient pour une compréhension riche du monde 3D ?''. Cette vision ouvre la voie à une multitude d'applications qui ont un impact fondamental sur la façon dont les agents physiques perçoivent et interagissent avec leur environnement. Cette thèse, appelée ``programmes différentiables pour modèler l'environnement'', unifie les efforts de plusieurs domaines étroitement liés mais actuellement disjoints, notamment la robotique, la vision par ordinateur, l'infographie et l'IA. Ma première contribution---gradSLAM--- est un système de localisation et de cartographie simultanées (SLAM) dense et entièrement différentiable. En permettant le calcul du gradient à travers des composants autrement non différentiables tels que l'optimisation non linéaire par moindres carrés, le raycasting, l'odométrie visuelle et la cartographie dense, gradSLAM ouvre de nouvelles voies pour intégrer la reconstruction 3D classique et l'apprentissage profond. Ma deuxième contribution - taskography - propose une sparsification conditionnée par la tâche de grandes scènes 3D encodées sous forme de graphes de scènes 3D. Cela permet aux planificateurs classiques d'égaler (et de surpasser) les planificateurs de pointe basés sur l'apprentissage en concentrant le calcul sur les attributs de la scène pertinents pour la tâche. Ma troisième et dernière contribution---gradSim--- est un simulateur entièrement différentiable qui combine des moteurs physiques et graphiques différentiables pour permettre l'estimation des paramètres physiques et le contrôle visuomoteur, uniquement à partir de vidéos ou d'une image fixe.Modern artificial intelligence (AI) has created exciting new opportunities for building intelligent robots. In particular, gradient-based learning architectures (deep neural networks) have tremendously improved 3D scene understanding in terms of perception, reasoning, and action. However, these advancements have undermined many ``classical'' techniques developed over the last few decades. We postulate that a blend of ``classical'' and ``learned'' methods is the most promising path to developing flexible, interpretable, and actionable models of the world: a necessity for intelligent embodied agents. ``What is the ideal way to combine classical techniques with gradient-based learning architectures for a rich understanding of the 3D world?'' is the central question in this dissertation. This understanding enables a multitude of applications that fundamentally impact how embodied agents perceive and interact with their environment. This dissertation, dubbed ``differentiable world programs'', unifies efforts from multiple closely-related but currently-disjoint fields including robotics, computer vision, computer graphics, and AI. Our first contribution---gradSLAM---is a fully differentiable dense simultaneous localization and mapping (SLAM) system. By enabling gradient computation through otherwise non-differentiable components such as nonlinear least squares optimization, ray casting, visual odometry, and dense mapping, gradSLAM opens up new avenues for integrating classical 3D reconstruction and deep learning. Our second contribution---taskography---proposes a task-conditioned sparsification of large 3D scenes encoded as 3D scene graphs. This enables classical planners to match (and surpass) state-of-the-art learning-based planners by focusing computation on task-relevant scene attributes. Our third and final contribution---gradSim---is a fully differentiable simulator that composes differentiable physics and graphics engines to enable physical parameter estimation and visuomotor control, solely from videos or a still image