99,333 research outputs found
Cost-benefit analysis for software process improvement
Justification of investments to improve software development processes and technol- ogy continues to be a significant challenge for software management. Managers interested in improving quality, cost, and cycle-time of their products have a large set of methods, tools, and techniques from which to choose. The implementation of one or more of these potential improvements can require considerable time and cost. Decision makers must be able to understand the benefits from each proposed improvement and decide which improvements to implement. While a variety of approaches exist for evaluating the costs and benefits of a few specific improvements such as inspections or systematic reuse, there is no general model for evaluating software process improvements.
The result of this research is a practical, useful framework to assist practitioners in evaluating potential process improvements. This general framework can accommodate a variety of methods for estimating the cost-benefit effects of a process change. To illustrate this framework a set of cost-benefit templates for Emerald and Cleanroom technologies were developed and validated. Methods for evaluating effects range from constants and simple equations to bayesian decision models and dynamic process simulations. A prototype tool was developed to assist in performing cost-benefit analysis of software process improvements
Is One Hyperparameter Optimizer Enough?
Hyperparameter tuning is the black art of automatically finding a good
combination of control parameters for a data miner. While widely applied in
empirical Software Engineering, there has not been much discussion on which
hyperparameter tuner is best for software analytics. To address this gap in the
literature, this paper applied a range of hyperparameter optimizers (grid
search, random search, differential evolution, and Bayesian optimization) to
defect prediction problem. Surprisingly, no hyperparameter optimizer was
observed to be `best' and, for one of the two evaluation measures studied here
(F-measure), hyperparameter optimization, in 50\% cases, was no better than
using default configurations.
We conclude that hyperparameter optimization is more nuanced than previously
believed. While such optimization can certainly lead to large improvements in
the performance of classifiers used in software analytics, it remains to be
seen which specific optimizers should be applied to a new dataset.Comment: 7 pages, 2 columns, accepted for SWAN1
Expert Elicitation for Reliable System Design
This paper reviews the role of expert judgement to support reliability
assessments within the systems engineering design process. Generic design
processes are described to give the context and a discussion is given about the
nature of the reliability assessments required in the different systems
engineering phases. It is argued that, as far as meeting reliability
requirements is concerned, the whole design process is more akin to a
statistical control process than to a straightforward statistical problem of
assessing an unknown distribution. This leads to features of the expert
judgement problem in the design context which are substantially different from
those seen, for example, in risk assessment. In particular, the role of experts
in problem structuring and in developing failure mitigation options is much
more prominent, and there is a need to take into account the reliability
potential for future mitigation measures downstream in the system life cycle.
An overview is given of the stakeholders typically involved in large scale
systems engineering design projects, and this is used to argue the need for
methods that expose potential judgemental biases in order to generate analyses
that can be said to provide rational consensus about uncertainties. Finally, a
number of key points are developed with the aim of moving toward a framework
that provides a holistic method for tracking reliability assessment through the
design process.Comment: This paper commented in: [arXiv:0708.0285], [arXiv:0708.0287],
[arXiv:0708.0288]. Rejoinder in [arXiv:0708.0293]. Published at
http://dx.doi.org/10.1214/088342306000000510 in the Statistical Science
(http://www.imstat.org/sts/) by the Institute of Mathematical Statistics
(http://www.imstat.org
- âŠ