Analysis of limitations and metrology weaknesses of enterprise architecture (EA) measurement solutions & proposal of a COSMIC-based approach to EA measurement

The literature on enterprise architecture (EA) posits that EA is of considerable value for organizations. However, while the EA literature documents a number of proposals for EA measurement solutions, there is little evidence-based research supporting their achievements and limitations. This thesis aims at helping the EA community to understand the existing trends in EA measurement research and to recognize the existing gaps, limitations, and weaknesses in EA measurement solutions. Furthermore, this thesis aims to assist the EA community to design EA measurement solutions based on measurement and metrology best practices. The research goal of this thesis is to contribute to the EA body of knowledge by shaping new perspectives for future research avenues in EA measurement research. To achieve the research goal, the following research objectives are defined: 1. To classify the EA measurement solutions into specific categories in order to identify research themes and explain the structure of the research area. 2. To evaluate the EA measurement solutions from a measurement and metrology perspective. 3. To identify the measurement and metrology issues in EA measurement solutions. 4. To propose a novel EA measurement approach based on measurement and metrology guidelines and best practices. To achieve the first objective, this thesis conducts a systematic mapping study (SMS to help understand the state-of-the-art of EA measurement research and classify the research area in order to acquire a general understanding about the existing research trends. To achieve the second and third objectives, this thesis conducts a systematic literature review (SLR) to evaluate the EA measurement solutions from a measurement and metrology perspective, and hence, to reveal the weaknesses of EA measurement solutions and propose relevant solutions to these weaknesses. To perform this evaluation, we develop an evaluation process based on combining both the components of the evolution theory and the concepts of measurement and metrology best practices, such as ISO 15939. To achieve the fourth objective, we propose a mapping between two international standards: • COSMIC - ISO/IEC 19761: a method for measuring the functional size of software. • ArchiMate: a modelling language for EA. This mapping results in proposing a novel EA measurement approach that overcomes the weaknesses and limitations found in the existing EA measurement solutions. The research results demonstrate that: 1. The current publications on EA measurement are trending toward an increased focus on the “enterprise IT architecting” school of thought, lacks the rigorous terminology found in science and engineering and shows limited adoption of knowledge from other disciplines in the proposals of EA measurement solutions. 2. There is a lack of attention to attaining appropriate metrology properties in EA measurement proposals: all EA measurement proposals are characterized with insufficient metrology coverage scoring, theoretical and empirical definitions. 3. The proposed novel EA measurement approach demonstrates that it is handy for EA practitioners, and easy to adopt by organizations

Planning the Future of U.S. Particle Physics (Snowmass 2013): Chapter 1: Summary

These reports present the results of the 2013 Community Summer Study of the APS Division of Particles and Fields ("Snowmass 2013") on the future program of particle physics in the U.S. Chapter 1 contains the Executive Summary and the summaries of the reports of the nine working groups.Comment: 51 page

Performance management in Thai R&D organizations : exploring the interplay between R&D institutions and R&D contexts

Measuring research and development (R&D) performance has become a fundamental concern for R&D organisations. However, the complexity of measurement problems in R&D organisations has resulted in a situation where there is an excess of literature around the areas of R&D measurement, and yet a scarcity of generally accepted measurement approaches (Brown & Gobeli, 1992). This might be because the design of performance measurement (PM) for an R&D organisation combines several interrelated contexts that make each R&D measurement unique. This thesis, therefore, reviews several major R&D distinctions which should be taken into account when the R&D PM design is considered. These considerations are R&D measurement levels and perspectives, R&D key measures, R&D key drivers, and types of R&D institutions.;Taking a quality-based approach, the thesis combines several techniques, i.e. in-depth interviews, cognitive mapping interviews, document analysis, multiple case studies, and cross-case analysis. The interviews involve 30 respondents who are all experienced in R&D management in four different Thai R&D institutes, under the Ministry of Science and Technology.;The results indicate issues in three areas of investigation. First, the four cases studied measure R&D performance at different levels, for different purposes, and applying different measures and techniques. At a corporate level, instead of emphasising financial areas, the output measurement seems to be significant, as well as deliberative to quantitative methods. Meanwhile, at a team level, the measures highlight both quantitative and qualitative measures, for the purpose of monitoring the process and progress of research.;Second, the output mixes, stage of R&D, and sources of research questions could lead to the identification of three major types of R&D organisations: discipline-based, profession-based, and domain-based. The R&D measures that a firm applies seem to be interrelated with the type of R&D institution that firm represents.;Finally, the main key driver in this study is R&D collaboration. However, collaboration functions differ according to the different types of R&D organisations. A discipline-based organisation tends to use collaboration as a tool to explore new knowledge and to strengthen the firm's competency, whereas a profession-based organisation tends to use collaboration to gain market information and increase its ability to utilise R&D.;The study developed implications of both theoretical and managerial importance, identifying patterns of interrelationship between R&D institutions and key performance measures, and between R&D institutions and their collaboration mechanisms. Additionally, the main managerial implication could benefit R&D management practitioners, R&D managers, and R&D policymakers. Overall, the study's results demonstrate the importance of understanding the constraints of each R&D measurement context, i.e. levels of measurement, areas of measurement, and stages of R&D, for the performance measurement system.;Also, this study shows that each type of R&D institute may significantly be interrelated with other features, i.e. with key measures, and with key driver's mechanisms. Conducive to measuring and managing R&D performance efficiently, managers may realise the unique role of each type of R&D organisation (as well as its key measures and performance drivers) and design their performance measurement accordingly. Therefore, the benefits of this study may be seen as practical knowledge which could be employed to design R&D PM and, ultimately, to complement a strategic formulation to improve a firm's R&D performance.Measuring research and development (R&D) performance has become a fundamental concern for R&D organisations. However, the complexity of measurement problems in R&D organisations has resulted in a situation where there is an excess of literature around the areas of R&D measurement, and yet a scarcity of generally accepted measurement approaches (Brown & Gobeli, 1992). This might be because the design of performance measurement (PM) for an R&D organisation combines several interrelated contexts that make each R&D measurement unique. This thesis, therefore, reviews several major R&D distinctions which should be taken into account when the R&D PM design is considered. These considerations are R&D measurement levels and perspectives, R&D key measures, R&D key drivers, and types of R&D institutions.;Taking a quality-based approach, the thesis combines several techniques, i.e. in-depth interviews, cognitive mapping interviews, document analysis, multiple case studies, and cross-case analysis. The interviews involve 30 respondents who are all experienced in R&D management in four different Thai R&D institutes, under the Ministry of Science and Technology.;The results indicate issues in three areas of investigation. First, the four cases studied measure R&D performance at different levels, for different purposes, and applying different measures and techniques. At a corporate level, instead of emphasising financial areas, the output measurement seems to be significant, as well as deliberative to quantitative methods. Meanwhile, at a team level, the measures highlight both quantitative and qualitative measures, for the purpose of monitoring the process and progress of research.;Second, the output mixes, stage of R&D, and sources of research questions could lead to the identification of three major types of R&D organisations: discipline-based, profession-based, and domain-based. The R&D measures that a firm applies seem to be interrelated with the type of R&D institution that firm represents.;Finally, the main key driver in this study is R&D collaboration. However, collaboration functions differ according to the different types of R&D organisations. A discipline-based organisation tends to use collaboration as a tool to explore new knowledge and to strengthen the firm's competency, whereas a profession-based organisation tends to use collaboration to gain market information and increase its ability to utilise R&D.;The study developed implications of both theoretical and managerial importance, identifying patterns of interrelationship between R&D institutions and key performance measures, and between R&D institutions and their collaboration mechanisms. Additionally, the main managerial implication could benefit R&D management practitioners, R&D managers, and R&D policymakers. Overall, the study's results demonstrate the importance of understanding the constraints of each R&D measurement context, i.e. levels of measurement, areas of measurement, and stages of R&D, for the performance measurement system.;Also, this study shows that each type of R&D institute may significantly be interrelated with other features, i.e. with key measures, and with key driver's mechanisms. Conducive to measuring and managing R&D performance efficiently, managers may realise the unique role of each type of R&D organisation (as well as its key measures and performance drivers) and design their performance measurement accordingly. Therefore, the benefits of this study may be seen as practical knowledge which could be employed to design R&D PM and, ultimately, to complement a strategic formulation to improve a firm's R&D performance

Evaluating an automated procedure of machine learning parameter tuning for software effort estimation

Software effort estimation requires accurate prediction models. Machine learning algorithms have been used to create more accurate estimation models. However, these algorithms are sensitive to factors such as the choice of hyper-parameters. To reduce this sensitivity, automated approaches for hyper-parameter tuning have been recently investigated. There is a need for further research on the effectiveness of such approaches in the context of software effort estimation. These evaluations could help understand which hyper-parameter settings can be adjusted to improve model accuracy, and in which specific contexts tuning can benefit model performance. The goal of this work is to develop an automated procedure for machine learning hyper-parameter tuning in the context of software effort estimation. The automated procedure builds and evaluates software effort estimation models to determine the most accurate evaluation schemes. The methodology followed in this work consists of first performing a systematic mapping study to characterize existing hyper-parameter tuning approaches in software effort estimation, developing the procedure to automate the evaluation of hyper-parameter tuning, and conducting controlled quasi experiments to evaluate the automated procedure. From the systematic literature mapping we discovered that effort estimation literature has favored the use of grid search. The results we obtained in our quasi experiments demonstrated that fast, less exhaustive tuners were viable in place of grid search. These results indicate that randomly evaluating 60 hyper-parameters can be as good as grid search, and that multiple state-of-the-art tuners were only more effective than this random search in 6% of the evaluated dataset-model combinations. We endorse random search, genetic algorithms, flash, differential evolution, and tabu and harmony search as effective tuners.Los algoritmos de aprendizaje automático han sido utilizados para crear modelos con mayor precisión para la estimación del esfuerzo del desarrollo de software. Sin embargo, estos algoritmos son sensibles a factores, incluyendo la selección de hiper parámetros. Para reducir esto, se han investigado recientemente algoritmos de ajuste automático de hiper parámetros. Es necesario evaluar la efectividad de estos algoritmos en el contexto de estimación de esfuerzo. Estas evaluaciones podrían ayudar a entender qué hiper parámetros se pueden ajustar para mejorar los modelos, y en qué contextos esto ayuda el rendimiento de los modelos. El objetivo de este trabajo es desarrollar un procedimiento automatizado para el ajuste de hiper parámetros para algoritmos de aprendizaje automático aplicados a la estimación de esfuerzo del desarrollo de software. La metodología seguida en este trabajo consta de realizar un estudio de mapeo sistemático para caracterizar los algoritmos de ajuste existentes, desarrollar el procedimiento automatizado, y conducir cuasi experimentos controlados para evaluar este procedimiento. Mediante el mapeo sistemático descubrimos que la literatura en estimación de esfuerzo ha favorecido el uso de la búsqueda en cuadrícula. Los resultados obtenidos en nuestros cuasi experimentos demostraron que algoritmos de estimación no-exhaustivos son viables para la estimación de esfuerzo. Estos resultados indican que evaluar aleatoriamente 60 hiper parámetros puede ser tan efectivo como la búsqueda en cuadrícula, y que muchos de los métodos usados en el estado del arte son solo más efectivos que esta búsqueda aleatoria en 6% de los escenarios. Recomendamos el uso de la búsqueda aleatoria, algoritmos genéticos y similares, y la búsqueda tabú y harmónica.Escuela de Ciencias de la Computación e InformáticaCentro de Investigaciones en Tecnologías de la Información y ComunicaciónUCR::Vicerrectoría de Investigación::Sistema de Estudios de Posgrado::Ingeniería::Maestría Académica en Computación e Informátic