Software Measurement Activities in Small and Medium Enterprises: an Empirical Assessment

An empirical study for evaluating the proper implementation of measurement/metric programs in software companies in one area of Turkey is presented. The research questions are discussed and validated with the help of senior software managers (more than 15 years’ experience) and then used for interviewing a variety of medium and small scale software companies in Ankara. Observations show that there is a common reluctance/lack of interest in utilizing measurements/metrics despite the fact that they are well known in the industry. A side product of this research is that internationally recognized standards such as ISO and CMMI are pursued if they are a part of project/job requirements; without these requirements, introducing those standards to the companies remains as a long-term target to increase quality

LAI Initiative on Systems Engineering Leading Indicators

Follow-on initiative from the Air Force/LAI Workshop on Systems Engineering for Robustnes

In Quest for Requirements Engineering Oracles: Dependent Variables and Measurements for (good) RE

Context: For many years, researchers and practitioners have been proposing various methods and approaches to Requirements Engineering (RE). Those contributions remain, however, too often on the level of apodictic discussions with- out having proper knowledge about the practical problems they propagate to address, or how to measure the success of the contributions when applying them in practical con- texts. While the scientific impact of research might not be threatened, the practical impact of the contributions is. Aim: We aim at better understanding practically relevant variables in RE, how those variables relate to each other, and to what extent we can measure those variables. This allows for the establishment of generalisable improvement goals, and the measurement of success of solution proposals. Method: We establish a first empirical basis of de- pendent variables in RE and means for their measurement. We classify the variables according to their dimension (e.g. RE, company, SW project), their measurability, and their actionability. Results: We reveal 93 variables with 167 dependencies of which a large subset is measurable directly in RE while further variables remain unmeasurable or have too complex dependencies for reliable measurements. We critically reflect on the results and show direct implications for research in the field of RE. Conclusion: We discuss a variety of conclusions we can draw from our results. For example, we show a set of first improvement goals directly usable for evidence-based RE research such as "increase flexibility in the RE process", we discuss suitable study types, and, finally, we can underpin the importance of replication studies to obtain generalisability

Towards the measurement of Enterprise Information Systems agility to support EIS improving projects

International audienceEnterprise information systems (EIS) are directly implied in the global performance of an organisation. Nevertheless, their potential rigidity in comparison with the required fast evolution of the supported organisation remains an important open research question. The proposed research work aims to define and evaluate the agility of an EIS, in order to assist both software engineers and business managers in EIS improvement projects. In particular, a framework is proposed to structure the different existing metrics on agility according to the improvements needs and the intrinsic characteristics of an information system

Using data mining to dynamically build up just in time learner models

Using rich data collected from e-learning systems, it may be possible to build up just in time dynamic learner models to analyze learners' behaviours and to evaluate learners' performance in online education systems. The goal is to create metrics to measure learners' characteristics from usage data. To achieve this goal we need to use data mining methods, especially clustering algorithms, to find patterns from which metrics can be derived from usage data. In this thesis, we propose a six layer model (raw data layer, fact data layer, data mining layer, measurement layer, metric layer and pedagogical application layer) to create a just in time learner model which draws inferences from usage data. In this approach, we collect raw data from online systems, filter fact data from raw data, and then use clustering mining methods to create measurements and metrics. In a pilot study, we used usage data collected from the iHelp system to create measurements and metrics to observe learners' behaviours in a real online system. The measurements and metrics relate to a learner's sociability, activity levels, learning styles, and knowledge levels. To validate the approach we designed two experiments to compare the metrics and measurements extracted from the iHelp system: expert evaluations and learner self evaluations. Even though the experiments did not produce statistically significant results, this approach shows promise to describe learners' behaviours through dynamically generated measurements and metric. Continued research on these kinds of methodologies is promising

Automating Cyber Analytics

Model based security metrics are a growing area of cyber security research concerned with measuring the risk exposure of an information system. These metrics are typically studied in isolation, with the formulation of the test itself being the primary finding in publications. As a result, there is a flood of metric specifications available in the literature but a corresponding dearth of analyses verifying results for a given metric calculation under different conditions or comparing the efficacy of one measurement technique over another. The motivation of this thesis is to create a systematic methodology for model based security metric development, analysis, integration, and validation. In doing so we hope to fill a critical gap in the way we view and improve a system’s security. In order to understand the security posture of a system before it is rolled out and as it evolves, we present in this dissertation an end to end solution for the automated measurement of security metrics needed to identify risk early and accurately. To our knowledge this is a novel capability in design time security analysis which provides the foundation for ongoing research into predictive cyber security analytics. Modern development environments contain a wealth of information in infrastructure-as-code repositories, continuous build systems, and container descriptions that could inform security models, but risk evaluation based on these sources is ad-hoc at best, and often simply left until deployment. Our goal in this work is to lay the groundwork for security measurement to be a practical part of the system design, development, and integration lifecycle. In this thesis we provide a framework for the systematic validation of the existing security metrics body of knowledge. In doing so we endeavour not only to survey the current state of the art, but to create a common platform for future research in the area to be conducted. We then demonstrate the utility of our framework through the evaluation of leading security metrics against a reference set of system models we have created. We investigate how to calibrate security metrics for different use cases and establish a new methodology for security metric benchmarking. We further explore the research avenues unlocked by automation through our concept of an API driven S-MaaS (Security Metrics-as-a-Service) offering. We review our design considerations in packaging security metrics for programmatic access, and discuss how various client access-patterns are anticipated in our implementation strategy. Using existing metric processing pipelines as reference, we show how the simple, modular interfaces in S-MaaS support dynamic composition and orchestration. Next we review aspects of our framework which can benefit from optimization and further automation through machine learning. First we create a dataset of network models labeled with the corresponding security metrics. By training classifiers to predict security values based only on network inputs, we can avoid the computationally expensive attack graph generation steps. We use our findings from this simple experiment to motivate our current lines of research into supervised and unsupervised techniques such as network embeddings, interaction rule synthesis, and reinforcement learning environments. Finally, we examine the results of our case studies. We summarize our security analysis of a large scale network migration, and list the friction points along the way which are remediated by this work. We relate how our research for a large-scale performance benchmarking project has influenced our vision for the future of security metrics collection and analysis through dev-ops automation. We then describe how we applied our framework to measure the incremental security impact of running a distributed stream processing system inside a hardware trusted execution environment

Identifying Factors Contributing Towards Information Security Maturity in an Organization

Information security capability maturity (ISCM) is a journey towards accurate alignment of business and security objectives, security systems, processes, and tasks integrated with business-enabled IT systems, security enabled organizational culture and decision making, and measurements and continuous improvements of controls and governance comprising security policies, processes, operating procedures, tasks, monitoring, and reporting. Information security capability maturity may be achieved in five levels: performing but ad-hoc, managed, defined, quantitatively governed, and optimized. These five levels need to be achieved in the capability areas of information integrity, information systems assurance, business enablement, security processes, security program management, competency of security team, security consciousness in employees, and security leadership. These areas of capabilities lead to achievement of technology trustworthiness of security controls, integrated security, and security guardianship throughout the enterprise, which are primary capability domains for achieving maturity of information security capability in an organization. There are many factors influencing the areas of capabilities and the capability domains for achieving information security capability maturity. However, there is little existing study done on identifying the factors that contribute to achievement of the highest level of information security capability maturity (optimized) in an organization. This research was designed to contribute to this area of research gap by identifying the factors contributing to the areas of capabilities for achieving the highest level of information security capability maturity. The factors were grouped under the eight capability areas and the three capability domains in the form of an initial structural construct. This research was designed to collect data on all the factors using an online structured questionnaire and analyzing the reliability and validity of the initial structural construct following the methods of principal components analysis (PCA), Cronbach Alpha reliability analysis, confirmatory factor analysis (CFA), and structural equation modeling. A number of multivariate statistical tests were conducted on the data collected regarding the factors to achieve an optimal model reflecting statistical significance, reliability, and validity. The research was conducted in four phases: expert panel and pilot study (first phase), principal component analysis (PCA) and reliability analysis (RA) of the factor scales (second phase), confirmatory factor analysis (CFA) using LISREL (third phase), and structural equation modeling (SEM) using LISREL (fourth phase). The final model subsequent to completing the four phases reflected acceptance or rejection of the eleven hypotheses defined in the initial structural construct of this study. The final optimized model was obtained with the most significant factors loading on the capability areas of information integrity, information security assurance, business enablement, security process maturity, security program management, competency of security team, security conscious employees, and security leadership, including the most significant factors loading the three capability domains of security technology trustworthiness, security integration, and security guardianship. All the eleven hypotheses were accepted as part of the optimal structural construct of the final model. The model provides a complex integrated framework of information security maturity requiring multi-functional advancements and maturity in processes, people, and technology, and organized security program management and communications fully integrated with the business programs and communications. Information security maturity is concluded as a complex function of multiple maturity programs in an organization leading to organized governance structures, multiple maturity programs, leadership, security consciousness, and risk-aware culture of employees

Building Better Relationships: Developing critically reflective practice when working preventively with domestic violence and abuse

Background Whilst the phenomenon of domestic abuse, violence against women and girls, the individual and social impact, is globally well documented, practice in this area is not. Responses are often confusing and contradictory, arising from an ill-defined area of professional practice that has a negative impact on those involved, including victims and practitioners who work with them. Aim The practice/research approach adopted for the study aims to develop professional knowledge and professional/interprofessional practice, through the development of critically reflective domestic abuse prevention practice. Method A qualitative study using Critical Participatory Action Research, limited to recruiting three practice/research sites, which included a range of practitioner/researchers from health, social care, and voluntary sector organisations. Data were analysed using a phronetic iterative approach (Tracy, 2020). Findings are specific to each research site, generated in the context of practice application. Findings Findings from the research indicate that providing a framework for critically reflective practice, enhances and develops critically reflective domestic abuse prevention practice. This is manifest in changes to i) language, ii) actions, and iii) relationships. Critically reflective practice in this field is enabled by practice architectures (Kemmis et al., 2014), arrangements that support its development and the actions produced, comprised of: courage, compassion, containment, responsibility, risk management, adaptability, awareness and reflexivity, facilitation, tools (for critical reflection) and time. The development of critically reflective domestic abuse prevention practice expanded space for action, increased wellbeing and created transformational relationships in the contextualised locations of the research sites. Conclusion The research has made a significant contribution to practice development, and learning, in the field of domestic abuse prevention, increasing knowledge of: 1) domestic abuse prevention work: its undervalued and hidden nature 2) places, in which critically reflective practice in this field take place, including the significance of compassion and containment 3) learning in relation to domestic abuse prevention/prevention work, and the importance of practice-based and praxis-focused education in this fiel