8,926 research outputs found
Basic Enhancement Strategies When Using Bayesian Optimization for Hyperparameter Tuning of Deep Neural Networks
Compared to the traditional machine learning models, deep neural networks (DNN) are known to be highly sensitive to the choice of hyperparameters. While the required time and effort for manual tuning has been rapidly decreasing for the well developed and commonly used DNN architectures, undoubtedly DNN hyperparameter optimization will continue to be a major burden whenever a new DNN architecture needs to be designed, a new task needs to be solved, a new dataset needs to be addressed, or an existing DNN needs to be improved further. For hyperparameter optimization of general machine learning problems, numerous automated solutions have been developed where some of the most popular solutions are based on Bayesian Optimization (BO). In this work, we analyze four fundamental strategies for enhancing BO when it is used for DNN hyperparameter optimization. Specifically, diversification, early termination, parallelization, and cost function transformation are investigated. Based on the analysis, we provide a simple yet robust algorithm for DNN hyperparameter optimization - DEEP-BO (Diversified, Early-termination-Enabled, and Parallel Bayesian Optimization). When evaluated over six DNN benchmarks, DEEP-BO mostly outperformed well-known solutions including GP-Hedge, BOHB, and the speed-up variants that use Median Stopping Rule or Learning Curve Extrapolation. In fact, DEEP-BO consistently provided the top, or at least close to the top, performance over all the benchmark types that we have tested. This indicates that DEEP-BO is a robust solution compared to the existing solutions. The DEEP-BO code is publicly available at <uri>https://github.com/snu-adsl/DEEP-BO</uri>
Comprehensive Security Framework for Global Threats Analysis
Cyber criminality activities are changing and becoming more and more professional. With the growth of financial flows through the Internet and the Information System (IS), new kinds of thread arise involving complex scenarios spread within multiple IS components. The IS information modeling and Behavioral Analysis are becoming new solutions to normalize the IS information and counter these new threads. This paper presents a framework which details the principal and necessary steps for monitoring an IS. We present the architecture of the framework, i.e. an ontology of activities carried out within an IS to model security information and User Behavioral analysis. The results of the performed experiments on real data show that the modeling is effective to reduce the amount of events by 91%. The User Behavioral Analysis on uniform modeled data is also effective, detecting more than 80% of legitimate actions of attack scenarios
Improving Performance of Iterative Methods by Lossy Checkponting
Iterative methods are commonly used approaches to solve large, sparse linear
systems, which are fundamental operations for many modern scientific
simulations. When the large-scale iterative methods are running with a large
number of ranks in parallel, they have to checkpoint the dynamic variables
periodically in case of unavoidable fail-stop errors, requiring fast I/O
systems and large storage space. To this end, significantly reducing the
checkpointing overhead is critical to improving the overall performance of
iterative methods. Our contribution is fourfold. (1) We propose a novel lossy
checkpointing scheme that can significantly improve the checkpointing
performance of iterative methods by leveraging lossy compressors. (2) We
formulate a lossy checkpointing performance model and derive theoretically an
upper bound for the extra number of iterations caused by the distortion of data
in lossy checkpoints, in order to guarantee the performance improvement under
the lossy checkpointing scheme. (3) We analyze the impact of lossy
checkpointing (i.e., extra number of iterations caused by lossy checkpointing
files) for multiple types of iterative methods. (4)We evaluate the lossy
checkpointing scheme with optimal checkpointing intervals on a high-performance
computing environment with 2,048 cores, using a well-known scientific
computation package PETSc and a state-of-the-art checkpoint/restart toolkit.
Experiments show that our optimized lossy checkpointing scheme can
significantly reduce the fault tolerance overhead for iterative methods by
23%~70% compared with traditional checkpointing and 20%~58% compared with
lossless-compressed checkpointing, in the presence of system failures.Comment: 14 pages, 10 figures, HPDC'1
Using Rollback Avoidance to Mitigate Failures in Next-Generation Extreme-Scale Systems
High-performance computing (HPC) systems enable scientists to numerically model complex phenomena in many important physical systems. The next major milestone in the development of HPC systems is the construction of the first supercomputer capable executing more than an exaflop, 10^18 floating point operations per second. On systems of this scale, failures will occur much more frequently than on current systems. As a result, resilience is a key obstacle to building next-generation extreme-scale systems. Coordinated checkpointing is currently the most widely-used mechanism for handling failures on HPC systems. Although coordinated checkpointing remains effective on current systems, increasing the scale of today\u27s systems to build next-generation systems will increase the cost of fault tolerance as more and more time is taken away from the application to protect against or recover from failure. Rollback avoidance techniques seek to mitigate the cost of checkpoint/restart by allowing an application to continue its execution rather than rolling back to an earlier checkpoint when failures occur. These techniques include failure prediction and preventive migration, replicated computation, fault-tolerant algorithms, and software-based memory fault correction. In this thesis, I examine how rollback avoidance techniques can be used to address failures on extreme-scale systems. Using a combination of analytic modeling and simulation, I evaluate the potential impact of rollback avoidance on these systems. I then present a novel rollback avoidance technique that exploits similarities in application memory. Finally, I examine the feasibility of using this technique to protect against memory faults in kernel memory
DynaQuant: Compressing Deep Learning Training Checkpoints via Dynamic Quantization
With the increase in the scale of Deep Learning (DL) training workloads in
terms of compute resources and time consumption, the likelihood of encountering
in-training failures rises substantially, leading to lost work and resource
wastage. Such failures are typically offset by a checkpointing mechanism, which
comes at the cost of storage and network bandwidth overhead. State-of-the-art
approaches involve lossy model compression mechanisms, which induce a tradeoff
between the resulting model quality (accuracy) and compression ratio. Delta
compression is then used to further reduce the overhead by only storing the
difference between consecutive checkpoints. We make a key enabling observation
that the sensitivity of model weights to compression varies during training,
and different weights benefit from different quantization levels (ranging from
retaining full precision to pruning). We propose (1) a non-uniform quantization
scheme that leverages this variation, (2) an efficient search mechanism that
dynamically finds the best quantization configurations, and (3) a
quantization-aware delta compression mechanism that rearranges weights to
minimize checkpoint differences, thereby maximizing compression. We instantiate
these contributions in DynaQuant - a framework for DL workload checkpoint
compression. Our experiments show that DynaQuant consistently achieves a better
tradeoff between accuracy and compression ratios compared to prior works,
enabling a compression ratio up to 39x and withstanding up to 10 restores with
negligible accuracy impact for fault-tolerant training. DynaQuant achieves at
least an order of magnitude reduction in checkpoint storage overhead for
training failure recovery as well as transfer learning use cases without any
loss of accuracy
Risks from low dose/dose rate radiation: what an understanding of DNA damage response mechanisms can tell us
The DNA damage response (DDR) mechanisms represent a vital line of defense against exogenous and endogenous DNA damage to enhance two distinct outcomes, survival and the maintenance of genomic stability. The latter is critical for cancer avoidance. DDR processes encompass repair pathways and signal transduction mechanisms that activate cell cycle checkpoint arrest and apoptosis. DNA double strand breaks (DSBs) represent important radiation-induced lesions. The major DSB repair pathways are DNA non-homologous end-joining (NHEJ) and homologous recombination (HR) and ataxia telangiectasia mutated (ATM) activates the DSB signaling response. To evaluate the ability of these pathways to protect against low doses or dose rate radiation exposure, it is important to consider the fidelity of DSB repair and the sensitivity of checkpoint arrest and apoptosis. Radiation-induced DSBs are more complex than endogenously-induced DSBs, with the potential for multiple lesions to arise in close proximity. NHEJ, the major DSB repair pathway, cannot accurately reconstitute sequence information lost at DSBs. Both pathways have the potential to cause translocations by rejoining erroneous DNA ends. Thus, complete accuracy of repair cannot be guaranteed and the formation of translocations, which have the potential to initiate carcinogenesis, can arise. Additionally, the G2/M checkpoint has a defined sensitivity, allowing some chromosome breakage to occur. Thus, genomic rearrangements can potentially arise even if the G1/S checkpoint is efficient. The sensitivity of apoptosis is currently unclear but will likely differ between tissues. In summary, it is unlikely that the DDR mechanisms can fully protect cells from genomic rearrangements following exposure to low doses or dose rate radiation
A runtime heuristic to selectively replicate tasks for application-specific reliability targets
In this paper we propose a runtime-based selective task replication technique for task-parallel high performance computing applications. Our selective task replication technique is automatic and does not require modification/recompilation of OS, compiler or application code. Our heuristic, we call App_FIT, selects tasks to replicate such that the specified reliability target for an application is achieved. In our experimental evaluation, we show that App FIT selective replication heuristic is low-overhead and highly scalable. In addition, results indicate that complete task replication is overkill for achieving reliability targets. We show that with App FIT, we can tolerate pessimistic exascale error rates with only 53% of the tasks being replicated.This work was supported by FI-DGR 2013 scholarship and the European Community’s
Seventh Framework Programme [FP7/2007-2013] under the Mont-blanc 2
Project (www.montblanc-project.eu), grant agreement no. 610402 and in part by the
European Union (FEDER funds) under contract TIN2015-65316-P.Peer ReviewedPostprint (author's final draft
Effective Node Clustering and Data Dissemination In Large-Scale Wireless Sensor Networks
The denseness and random distribution of large-scale WSNs makes it quite difficult to replace or recharge nodes. Energy efficiency and management is a major design goal in these networks. In addition, reliability and scalability are two other major goals that have been identified by researchers as necessary in order to further expand the deployment of such networks for their use in various applications. This thesis aims to provide an energy efficient and effective node clustering and data dissemination algorithm in large-scale wireless sensor networks. In the area of clustering, the proposed research prolongs the lifetime of the network by saving energy through the use of node ranking to elect cluster heads, contrary to other existing cluster-based work that selects a random node or the node with the highest energy at a particular time instance as the new cluster head. Moreover, a global knowledge strategy is used to maintain a level of universal awareness of existing nodes in the subject area and to avoid the problem of disconnected or forgotten nodes. In the area of data dissemination, the aim of this research is to effectively manage the data collection by developing an efficient data collection scheme using a ferry node and applying a selective duty cycle strategy to the sensor nodes. Depending on the application, mobile ferries can be used for collecting data in a WSN, especially those that are large in scale, with delay tolerant applications. Unlike data collection via multi-hop forwarding among the sensing nodes, ferries travel across the sensing field to collect data. A ferry-based approach thus eliminates, or minimizes, the need for the multi-hop forwarding of data, and as a result, energy consumption at the nodes will be significantly reduced. This is especially true for nodes that are near the base station as they are used by other nodes to forward data to the base station. MATLAB is used to design, simulate and evaluate the proposed work against the work that has already been done by others by using various performance criteria
Tolerating Correlated Failures in Massively Parallel Stream Processing Engines
Fault-tolerance techniques for stream processing engines can be categorized
into passive and active approaches. A typical passive approach periodically
checkpoints a processing task's runtime states and can recover a failed task by
restoring its runtime state using its latest checkpoint. On the other hand, an
active approach usually employs backup nodes to run replicated tasks. Upon
failure, the active replica can take over the processing of the failed task
with minimal latency. However, both approaches have their own inadequacies in
Massively Parallel Stream Processing Engines (MPSPE). The passive approach
incurs a long recovery latency especially when a number of correlated nodes
fail simultaneously, while the active approach requires extra replication
resources. In this paper, we propose a new fault-tolerance framework, which is
Passive and Partially Active (PPA). In a PPA scheme, the passive approach is
applied to all tasks while only a selected set of tasks will be actively
replicated. The number of actively replicated tasks depends on the available
resources. If tasks without active replicas fail, tentative outputs will be
generated before the completion of the recovery process. We also propose
effective and efficient algorithms to optimize a partially active replication
plan to maximize the quality of tentative outputs. We implemented PPA on top of
Storm, an open-source MPSPE and conducted extensive experiments using both real
and synthetic datasets to verify the effectiveness of our approach
- …