11 research outputs found
Synthesis and Pruning as a Dynamic Compression Strategy for Efficient Deep Neural Networks
The brain is a highly reconfigurable machine capable of task-specific
adaptations. The brain continually rewires itself for a more optimal
configuration to solve problems. We propose a novel strategic synthesis
algorithm for feedforward networks that draws directly from the brain's
behaviours when learning. The proposed approach analyses the network and ranks
weights based on their magnitude. Unlike existing approaches that advocate
random selection, we select highly performing nodes as starting points for new
edges and exploit the Gaussian distribution over the weights to select
corresponding endpoints. The strategy aims only to produce useful connections
and result in a smaller residual network structure. The approach is
complemented with pruning to further the compression. We demonstrate the
techniques to deep feedforward networks. The residual sub-networks that are
formed from the synthesis approaches in this work form common sub-networks with
similarities up to ~90%. Using pruning as a complement to the strategic
synthesis approach, we observe improvements in compression.Comment: 29TH ACM INTERNATIONAL CONFERENCE ON INFORMATION AND KNOWLEDGE
MANAGEMENT, 9th International Symposium DATAMOD 2020 FROM DATA TO MODELS AND
BACK, 16 Pages, 7 Figures, 3 Tables, 2 Equation
A comparative study of preprocessing and model compression techniques in deep learning for forest sound classification
Deep-learning models play a significant role in modern software solutions, with the capabilities of handling complex tasks, improving accuracy, automating processes, and adapting to diverse domains, eventually contributing to advancements in various industries. This study provides a comparative study on deep-learning techniques that can also be deployed on resource-constrained edge devices. As a novel contribution, we analyze the performance of seven Convolutional Neural Network models in the context of data augmentation, feature extraction, and model compression using acoustic data. The results show that the best performers can achieve an optimal trade-off between model accuracy and size when compressed with weight and filter pruning followed by 8-bit quantization. In adherence to the study workflow utilizing the forest sound dataset, MobileNet-v3-small and ACDNet achieved accuracies of 87.95% and 85.64%, respectively, while maintaining compact sizes of 243 KB and 484 KB, respectively. Henceforth, this study concludes that CNNs can be optimized and compressed to be deployed in resource-constrained edge devices for classifying forest environment sounds
Mathematical Optimization Algorithms for Model Compression and Adversarial Learning in Deep Neural Networks
Large-scale deep neural networks (DNNs) have made breakthroughs in a variety of tasks, such as image recognition, speech recognition and self-driving cars. However, their large model size and computational requirements add a significant burden to state-of-the-art computing systems. Weight pruning is an effective approach to reduce the model size and computational requirements of DNNs. However, prior works in this area are mainly heuristic methods. As a result, the performance of a DNN cannot maintain for a high weight pruning ratio. To mitigate this limitation, we propose a systematic weight pruning framework for DNNs based on mathematical optimization. We first formulate the weight pruning for DNNs as a non-convex optimization problem, and then systematically solve it using alternating direction method of multipliers (ADMM). Our work achieves a higher weight pruning ratio on DNNs without accuracy loss and a higher acceleration on the inference of DNNs on CPU and GPU platforms compared with prior works.
Besides the issue of model size, DNNs are also sensitive to adversarial attacks, a small invisible noise on the input data can fully mislead a DNN. Research on the robustness of DNNs follows two directions in general. The first is to enhance the robustness of DNNs, which increases the degree of difficulty for adversarial attacks to fool DNNs. The second is to design adversarial attack methods to test the robustness of DNNs. These two aspects reciprocally benefit each other towards hardening DNNs. In our work, we propose to generate adversarial attacks with low distortion via convex optimization, which achieves 100% attack success rate with lower distortion compared with prior works. We also propose a unified min-max optimization framework for the adversarial attack and defense on DNNs over multiple domains. Our proposed method performs better compared with the prior works, which use average-based strategies to solve the problems over multiple domains
Frivolous Units: Wider Networks Are Not Really That Wide
A remarkable characteristic of overparameterized deep neural networks (DNNs)
is that their accuracy does not degrade when the network's width is increased.
Recent evidence suggests that developing compressible representations is key
for adjusting the complexity of large networks to the learning task at hand.
However, these compressible representations are poorly understood. A promising
strand of research inspired from biology is understanding representations at
the unit level as it offers a more granular and intuitive interpretation of the
neural mechanisms. In order to better understand what facilitates increases in
width without decreases in accuracy, we ask: Are there mechanisms at the unit
level by which networks control their effective complexity as their width is
increased? If so, how do these depend on the architecture, dataset, and
training parameters? We identify two distinct types of "frivolous" units that
proliferate when the network's width is increased: prunable units which can be
dropped out of the network without significant change to the output and
redundant units whose activities can be expressed as a linear combination of
others. These units imply complexity constraints as the function the network
represents could be expressed by a network without them. We also identify how
the development of these units can be influenced by architecture and a number
of training factors. Together, these results help to explain why the accuracy
of DNNs does not degrade when width is increased and highlight the importance
of frivolous units toward understanding implicit regularization in DNNs
Safety and Reliability - Safe Societies in a Changing World
The contributions cover a wide range of methodologies and application areas for safety and reliability that contribute to safe societies in a changing world. These methodologies and applications include: - foundations of risk and reliability assessment and management
- mathematical methods in reliability and safety
- risk assessment
- risk management
- system reliability
- uncertainty analysis
- digitalization and big data
- prognostics and system health management
- occupational safety
- accident and incident modeling
- maintenance modeling and applications
- simulation for safety and reliability analysis
- dynamic risk and barrier management
- organizational factors and safety culture
- human factors and human reliability
- resilience engineering
- structural reliability
- natural hazards
- security
- economic analysis in risk managemen
Review of Particle Physics
The Review summarizes much of particle physics and cosmology. Using data from previous editions, plus 2,143 new measurements from 709 papers, we list, evaluate, and average measured properties of gauge bosons and the recently discovered Higgs boson, leptons, quarks, mesons, and baryons. We summarize searches for hypothetical particles such as supersymmetric particles, heavy bosons, axions, dark photons, etc. Particle properties and search limits are listed in Summary Tables. We give numerous tables, figures, formulae, and reviews of topics such as Higgs Boson Physics, Supersymmetry, Grand Unified Theories, Neutrino Mixing, Dark Energy, Dark Matter, Cosmology, Particle Detectors, Colliders, Probability and Statistics. Among the 120 reviews are many that are new or heavily revised, including a new review on Machine Learning, and one on Spectroscopy of Light Meson Resonances.
The Review is divided into two volumes. Volume 1 includes the Summary Tables and 97 review articles. Volume 2 consists of the Particle Listings and contains also 23 reviews that address specific aspects of the data presented in the Listings
Review of Particle Physics
The Review summarizes much of particle physics and cosmology. Using data from previous editions, plus 2,143 new measurements from 709 papers, we list, evaluate, and average measured properties of gauge bosons and the recently discovered Higgs boson, leptons, quarks, mesons, and baryons. We summarize searches for hypothetical particles such as supersymmetric particles, heavy bosons, axions, dark photons, etc. Particle properties and search limits are listed in Summary Tables. We give numerous tables, figures, formulae, and reviews of topics such as Higgs Boson Physics, Supersymmetry, Grand Unified Theories, Neutrino Mixing, Dark Energy, Dark Matter, Cosmology, Particle Detectors, Colliders, Probability and Statistics. Among the 120 reviews are many that are new or heavily revised, including a new review on Machine Learning, and one on Spectroscopy of Light Meson Resonances.
The Review is divided into two volumes. Volume 1 includes the Summary Tables and 97 review articles. Volume 2 consists of the Particle Listings and contains also 23 reviews that address specific aspects of the data presented in the Listings.
The complete Review (both volumes) is published online on the website of the Particle Data Group (pdg.lbl.gov) and in a journal. Volume 1 is available in print as the PDG Book. A Particle Physics Booklet with the Summary Tables and essential tables, figures, and equations from selected review articles is available in print, as a web version optimized for use on phones, and as an Android app