5 research outputs found
FreeREA: Training-Free Evolution-based Architecture Search
In the last decade, most research in Machine Learning contributed to the
improvement of existing models, with the aim of increasing the performance of
neural networks for the solution of a variety of different tasks. However, such
advancements often come at the cost of an increase of model memory and
computational requirements. This represents a significant limitation for the
deployability of research output in realistic settings, where the cost, the
energy consumption, and the complexity of the framework play a crucial role. To
solve this issue, the designer should search for models that maximise the
performance while limiting its footprint. Typical approaches to reach this goal
rely either on manual procedures, which cannot guarantee the optimality of the
final design, or upon Neural Architecture Search algorithms to automatise the
process, at the expenses of extremely high computational time. This paper
provides a solution for the fast identification of a neural network that
maximises the model accuracy while preserving size and computational
constraints typical of tiny devices. Our approach, named FreeREA, is a custom
cell-based evolution NAS algorithm that exploits an optimised combination of
training-free metrics to rank architectures during the search, thus without
need of model training. Our experiments, carried out on the common benchmarks
NAS-Bench-101 and NATS-Bench, demonstrate that i) FreeREA is the first method
able to provide very accurate models in minutes of search time; ii) it
outperforms State of the Art training-based and training-free techniques in all
the datasets and benchmarks considered, and iii) it can easily generalise to
constrained scenarios, representing a competitive solution for fast Neural
Architecture Search in generic constrained applications.Comment: 16 pages, 4 figurre
Entropic Score metric: Decoupling Topology and Size in Training-free NAS
Neural Networks design is a complex and often daunting task, particularly for
resource-constrained scenarios typical of mobile-sized models. Neural
Architecture Search is a promising approach to automate this process, but
existing competitive methods require large training time and computational
resources to generate accurate models. To overcome these limits, this paper
contributes with: i) a novel training-free metric, named Entropic Score, to
estimate model expressivity through the aggregated element-wise entropy of its
activations; ii) a cyclic search algorithm to separately yet synergistically
search model size and topology. Entropic Score shows remarkable ability in
searching for the topology of the network, and a proper combination with
LogSynflow, to search for model size, yields superior capability to completely
design high-performance Hybrid Transformers for edge applications in less than
1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet
classification.Comment: 10 pages, 3 figure
Fault-Aware Design and Training to Enhance DNNs Reliability with Zero-Overhead
7 pages, 6 figuresDeep Neural Networks (DNNs) enable a wide series of technological advancements, ranging from clinical imaging, to predictive industrial maintenance and autonomous driving. However, recent findings indicate that transient hardware faults may corrupt the models prediction dramatically. For instance, the radiation-induced misprediction probability can be so high to impede a safe deployment of DNNs models at scale, urging the need for efficient and effective hardening solutions. In this work, we propose to tackle the reliability issue both at training and model design time. First, we show that vanilla models are highly affected by transient faults, that can induce a performances drop up to 37%. Hence, we provide three zero-overhead solutions, based on DNN re-design and re-train, that can improve DNNs reliability to transient faults up to one order of magnitude. We complement our work with extensive ablation studies to quantify the gain in performances of each hardening component
Fault-Aware Design and Training to Enhance DNNs Reliability with Zero-Overhead
7 pages, 6 figuresDeep Neural Networks (DNNs) enable a wide series of technological advancements, ranging from clinical imaging, to predictive industrial maintenance and autonomous driving. However, recent findings indicate that transient hardware faults may corrupt the models prediction dramatically. For instance, the radiation-induced misprediction probability can be so high to impede a safe deployment of DNNs models at scale, urging the need for efficient and effective hardening solutions. In this work, we propose to tackle the reliability issue both at training and model design time. First, we show that vanilla models are highly affected by transient faults, that can induce a performances drop up to 37%. Hence, we provide three zero-overhead solutions, based on DNN re-design and re-train, that can improve DNNs reliability to transient faults up to one order of magnitude. We complement our work with extensive ablation studies to quantify the gain in performances of each hardening component
Fault-Aware Design and Training to Enhance DNNs Reliability with Zero-Overhead
7 pages, 6 figuresDeep Neural Networks (DNNs) enable a wide series of technological advancements, ranging from clinical imaging, to predictive industrial maintenance and autonomous driving. However, recent findings indicate that transient hardware faults may corrupt the models prediction dramatically. For instance, the radiation-induced misprediction probability can be so high to impede a safe deployment of DNNs models at scale, urging the need for efficient and effective hardening solutions. In this work, we propose to tackle the reliability issue both at training and model design time. First, we show that vanilla models are highly affected by transient faults, that can induce a performances drop up to 37%. Hence, we provide three zero-overhead solutions, based on DNN re-design and re-train, that can improve DNNs reliability to transient faults up to one order of magnitude. We complement our work with extensive ablation studies to quantify the gain in performances of each hardening component