Practical recommendations for gradient-based training of deep architectures

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures

Neural networks optimization through genetic algorithm searches: A review

Neural networks and genetic algorithms are the two sophisticated machine learning techniques presently attracting attention from scientists, engineers, and statisticians, among others. They have gained popularity in recent years. This paper presents a state of the art review of the research conducted on the optimization of neural networks through genetic algorithm searches. Optimization is aimed toward deviating from the limitations attributed to neural networks in order to solve complex and challenging problems. We provide an analysis and synthesis of the research published in this area according to the application domain, neural network design issues using genetic algorithms, types of neural networks and optimal values of genetic algorithm operators (population size, crossover rate and mutation rate). This study may provide a proper guide for novice as well as expert researchers in the design of evolutionary neural networks helping them choose suitable values of genetic algorithm operators for applications in a specific problem domain. Further research direction, which has not received much attention from scholars, is unveiled

Learning to estimate a surrogate respiratory signal from cardiac motion by signal-to-signal translation

In this work, we develop a neural network-based method to convert a noisy motion signal generated from segmenting rebinned list-mode cardiac SPECT images, to that of a high-quality surrogate signal, such as those seen from external motion tracking systems (EMTs). This synthetic surrogate will be used as input to our pre-existing motion correction technique developed for EMT surrogate signals. In our method, we test two families of neural networks to translate noisy internal motion to external surrogate: 1) fully connected networks and 2) convolutional neural networks. Our dataset consists of cardiac perfusion SPECT acquisitions for which cardiac motion was estimated (input: center-of-count-mass - COM signals) in conjunction with a respiratory surrogate motion signal acquired using a commercial Vicon Motion Tracking System (GT: EMT signals). We obtained an average R-score of 0.76 between the predicted surrogate and the EMT signal. Our goal is to lay a foundation to guide the optimization of neural networks for respiratory motion correction from SPECT without the need for an EMT.Comment: Medical Imaging Meets NeurIP

Visual Steering for One-Shot Deep Neural Network Synthesis

Recent advancements in the area of deep learning have shown the effectiveness of very large neural networks in several applications. However, as these deep neural networks continue to grow in size, it becomes more and more difficult to configure their many parameters to obtain good results. Presently, analysts must experiment with many different configurations and parameter settings, which is labor-intensive and time-consuming. On the other hand, the capacity of fully automated techniques for neural network architecture search is limited without the domain knowledge of human experts. To deal with the problem, we formulate the task of neural network architecture optimization as a graph space exploration, based on the one-shot architecture search technique. In this approach, a super-graph of all candidate architectures is trained in one-shot and the optimal neural network is identified as a sub-graph. In this paper, we present a framework that allows analysts to effectively build the solution sub-graph space and guide the network search by injecting their domain knowledge. Starting with the network architecture space composed of basic neural network components, analysts are empowered to effectively select the most promising components via our one-shot search scheme. Applying this technique in an iterative manner allows analysts to converge to the best performing neural network architecture for a given application. During the exploration, analysts can use their domain knowledge aided by cues provided from a scatterplot visualization of the search space to edit different components and guide the search for faster convergence. We designed our interface in collaboration with several deep learning researchers and its final effectiveness is evaluated with a user study and two case studies.Comment: 9 pages, submitted to IEEE Transactions on Visualization and Computer Graphics, 202

An Expert System to Improve the Energy Efficiency of the Reaction Zone of a Petrochemical Plant

Energy is the most important cost factor in the petrochemical industry. Thus, energy efficiency improvement is an important way to reduce these costs and to increase predictable earnings, especially in times of high energy price volatility. This work describes the development of an expert system for the improvement of this efficiency of the reaction zone of a petrochemical plant. This system has been developed after a data mining process of the variables registered in the plant. Besides, a kernel of neural networks has been embedded in the expert system. A graphical environment integrating the proposed system was developed in order to test the system. With the application of the expert system, the energy saving on the applied zone would have been about 20%.Junta de Andalucía TIC-570