10 research outputs found
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
In this paper, we introduce a new perspective on training deep neural
networks capable of state-of-the-art performance without the need for the
expensive over-parameterization by proposing the concept of In-Time
Over-Parameterization (ITOP) in sparse training. By starting from a random
sparse network and continuously exploring sparse connectivities during
training, we can perform an Over-Parameterization in the space-time manifold,
closing the gap in the expressibility between sparse training and dense
training. We further use ITOP to understand the underlying mechanism of Dynamic
Sparse Training (DST) and indicate that the benefits of DST come from its
ability to consider across time all possible parameters when searching for the
optimal sparse connectivity. As long as there are sufficient parameters that
have been reliably explored during training, DST can outperform the dense
neural network by a large margin. We present a series of experiments to support
our conjecture and achieve the state-of-the-art sparse training performance
with ResNet-50 on ImageNet. More impressively, our method achieves dominant
performance over the overparameterization-based sparse methods at extreme
sparsity levels. When trained on CIFAR-100, our method can match the
performance of the dense model even at an extreme sparsity (98%). Code can be
found https://github.com/Shiweiliuiiiiiii/In-Time-Over-Parameterization.Comment: 16 pages; 10 figures; Published in Proceedings of the 38th
International Conference on Machine Learning. Code can be found
https://github.com/Shiweiliuiiiiiii/In-Time-Over-Parameterizatio
Review of graph-based hazardous event detection methods for autonomous driving systems
Automated and autonomous vehicles are often required to operate in complex road environments with potential hazards that may lead to hazardous events causing injury or even death. Therefore, a reliable autonomous hazardous event detection system is a key enabler for highly autonomous vehicles (e.g., Level 4 and 5 autonomous vehicles) to operate without human supervision for significant periods of time. One promising solution to the problem is the use of graph-based methods that are powerful tools for relational reasoning. Using graphs to organise heterogeneous knowledge about the operational environment, link scene entities (e.g., road users, static objects, traffic rules) and describe how they affect each other. Due to a growing interest and opportunity presented by graph-based methods for autonomous hazardous event detection, this paper provides a comprehensive review of the state-of-the-art graph-based methods that we categorise as rule-based, probabilistic, and machine learning-driven. Additionally, we present an in-depth overview of the available datasets to facilitate hazardous event training and evaluation metrics to assess model performance. In doing so, we aim to provide a thorough overview and insight into the key research opportunities and open challenges