Development of An Empirical Approach to Building Domain-Specific Knowledge Applied to High-End Computing

This dissertation presents an empirical approach for building and storing knowledge about software engineering through human-subject research. It is based on running empirical studies in stages, where previously held hypotheses are supported or refuted in different contexts, and new hypotheses are generated. The approach is both mixed-methods based and opportunistic, and focuses on identifying a diverse set of potential sources for running studies. The output produced is an experience base which contains a set of these hypotheses, the empirical evidence which generated them, and the implications for practitioners and researchers. This experience base is contained in a software system which can be navigated by stakeholders to trace the "chain of evidence" of hypotheses as they evolve over time and across studies. This approach has been applied to the domain of high-end computing, to build knowledge related to programmer productivity. The methods include controlled experiments and quasi-experiments, case studies, observational studies, interviews, surveys, and focus groups. The results of these studies have been stored in a proof-of-concept system that implements the experience base

An Empirical Study of Process Discipline and Software Quality

There is a widespread, but not universal, belief in the software community that software organizations and projects can systematically improve their ability to meet commitments and build high-quality products using principles of software quality management. Quality affects cost and schedule, therefore the engineering practices that affect quality are also a management concern. Understanding the factors that influence software quality is crucial to the continuing maturation of the software industry; an improved understanding of software quality drivers will help software engineers and managers make more informed decisions in controlling and improving the software process.My research is motivated by a desire to understand the effect of disciplined processes and effective teams on improving performance and lessening variability with respect to software quality. Classroom data provides insight into interpersonal differences between competent professionals as increasingly disciplined processes are adopted. Project data using similar processes enables an exploration of the impact of effective teams on software quality.My results show that:* Program size, programmer ability, and disciplined processes significantly affect software quality.* Factors frequently used as surrogates for programmer ability, e.g., years of experience, and technology, e.g., programming language, do not significantly impact software quality.* Recommended practices are not necessarily followed even when processes are consistently performed, e.g., peer reviews may be consistently performed, but the review rates may exceed recommended practice for effective reviews.* When moving from ad hoc processes to disciplined processes, top-quartile performers improve more than 2X; bottom-quartile performers improve more than 4X. * Rigorous statistical techniques that allow for individual differences confirm the importance of process discipline and following recommended practice for improving software quality