Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks

The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, sometimes even better than, the original dense networks. Sparsity promises to reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field

Applications of Intelligent Vision in Low-Cost Mobile Robots

With the development of intelligent information technology, we have entered an era of 5G and AI. Mobile robots embody both of these technologies, and as such play an important role in future developments. However, the development of perception vision in consumer-grade low-cost mobile robots is still in its infancies. With the popularity of edge computing technology in the future, high-performance vision perception algorithms are expected to be deployed on low-power edge computing chips. Within the context of low-cost mobile robotic solutions, a robot intelligent vision system is studied and developed in this thesis. The thesis proposes and designs the overall framework of the higher-level intelligent vision system. The core system includes automatic robot navigation and obstacle object detection. The core algorithm deployments are implemented through a low-power embedded platform. The thesis analyzes and investigates deep learning neural network algorithms for obstacle object detection in intelligent vision systems. By comparing a variety of open source object detection neural networks on high performance hardware platforms, combining the constraints of hardware platform, a suitable neural network algorithm is selected. The thesis combines the characteristics and constraints of the low-power hardware platform to further optimize the selected neural network. It introduces the minimize mean square error (MMSE) and the moving average minmax algorithms in the quantization process to reduce the accuracy loss of the quantized model. The results show that the optimized neural network achieves a 20-fold improvement in inference performance on the RK3399PRO hardware platform compared to the original network. The thesis concludes with the application of the above modules and systems to a higher-level intelligent vision system for a low-cost disinfection robot, and further optimization is done for the hardware platform. The test results show that while achieving the basic service functions, the robot can accurately identify the obstacles ahead and locate and navigate in real time, which greatly enhances the perception function of the low-cost mobile robot

Automatic design of deep neural network architectures with evolutionary computation

Deep Neural Networks (DNNs) are algorithms with widespread use in the extraction of knowledge from raw data. DNNs are used to solve problems in the fields of computer vision, natural language understanding, signal processing, and others. DNNs are state-of-the-art machine learning models capable of achieving better results than humans in many tasks. However, their application in fields outside computer science and engineering has been hindered due to the tedious process of trial and error multiple computationally intensive models. Thus, the development of algorithms that could allow for the automatic development of DNNs would further advance the field. Two central problems need to be addressed to allow the automatic design of DNN models: generation and pruning. The automatic generation of DNN architectures would allow for the creation of state-of-the-art models without relying on knowledge from human experts. In contrast, the automatic pruning of DNN architectures would reduce the computational complexity of such models for use in less powerful hardware. The generation and pruning of DNN models can be seen as a combinatorial optimization problem, which can be solved with the tools from the Evolutionary Computation (EC) field. This Ph.D. work proposes the use of Particle Swarm Optimization (PSO) for DNN architecture searching with competitive results and fast convergence, called psoCNN. Another algorithm based on Evolution Strategy (ES) is used for the pruning of DNN architectures, called DeepPruningES. The proposed psoCNN algorithm is capable of finding CNN architectures, a particular type of DNN, for image classification tasks with comparable results to human-crafted DNN models. Likewise, the DeepPruningES algorithm is capable of reducing the number of floating operations of a given DNN model up to 80 percent, and it uses the principles of Multi-Criteria Decision Making (MCDM) to output three pruned model with different trade-offs between computational complexity and classification accuracy. These ideas are then applied to the creation of a unified framework for searching highly accurate, and compact DNN applied for Medical Imaging Diagnostics, and the pruning of Generative Adversarial Networks (GANs) for Medical Imaging Synthesis with competitive results