Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World

This report documents the program and the outcomes of GI-Dagstuhl Seminar 16394 "Software Performance Engineering in the DevOps World". The seminar addressed the problem of performance-aware DevOps. Both, DevOps and performance engineering have been growing trends over the past one to two years, in no small part due to the rise in importance of identifying performance anomalies in the operations (Ops) of cloud and big data systems and feeding these back to the development (Dev). However, so far, the research community has treated software engineering, performance engineering, and cloud computing mostly as individual research areas. We aimed to identify cross-community collaboration, and to set the path for long-lasting collaborations towards performance-aware DevOps. The main goal of the seminar was to bring together young researchers (PhD students in a later stage of their PhD, as well as PostDocs or Junior Professors) in the areas of (i) software engineering, (ii) performance engineering, and (iii) cloud computing and big data to present their current research projects, to exchange experience and expertise, to discuss research challenges, and to develop ideas for future collaborations

Enhancing the test and evaluation process: implementing agile development, test automation, and model-based systems engineering concepts

2020 Fall.Includes bibliographical references.With the growing complexity of modern systems, traditional testing methods are falling short. Test documentation suites used to verify the software for these types of large, complex systems can become bloated and unclear, leading to extremely long execution times and confusing, unmanageable test procedures. Additionally, the complexity of these systems can prevent the rapid understanding of complicated system concepts and behaviors, which is a necessary part of keeping up with the demands of modern testing efforts. Opportunities for optimization and innovation exist within the Test and Evaluation (T&E) domain, evidenced by the emergence of automated testing frameworks and iterative testing methodologies. Further opportunities lie with the directed expansion and application of related concepts such as Model-Based Systems Engineering (MBSE). This dissertation documents the development and implementation of three methods of enhancing the T&E field when applied to a real-world project. First, the development methodology of the system was transitioned from Waterfall to Agile, providing a more responsive approach when creating new features. Second, the Test Automation Framework (TAF) was developed, enabling the automatic execution of test procedures. Third, a method of test documentation using the Systems Modeling Language (SysML) was created, adopting concepts from MBSE to standardize the planning and analysis of test procedures. This dissertation provides the results of applying the three concepts to the development process of an airborne Electronic Warfare Management System (EWMS), which interfaces with onboard and offboard aircraft systems to receive and process the threat environment, providing the pilot or crew with a response solution for the protection of the aircraft. This system is representative of a traditional, long-term aerospace project that has been constantly upgraded over its lifetime. Over a two-year period, this new process produced a number of qualitative and quantitative results, including improving the quality and organization of the test documentation suite, reducing the minimum time to execute the test procedures, enabling the earlier identification of defects, and increasing the overall quality of the system under test. The application of these concepts generated many lessons learned, which are also provided. Transitioning a project's development methodology, modernizing the test approach, and introducing a new system of test documentation may provide significant benefits to the development of a system, but these types of process changes must be weighed against the needs of the project. This dissertation provides details of the effort to improve the effectiveness of the T&E process on an example project, as a framework for possible implementation on similar systems

Exploring Organizations\u27 Software Quality Assurance Strategies

Poor software quality leads to lost profits and even loss of life. U.S. organizations lose billions of dollars annually because of poor software quality. The purpose of this multiple case study was to explore the strategies that quality assurance (QA) leaders in small software development organizations used for successful software quality assurance (SQA) processes. A case study provided the best research design to allow for the exploration of organizational and managerial processes. The target population group was the QA leaders of 3 small software development organizations who successfully implemented SQA processes, located in Saint John, New Brunswick, Canada. The conceptual framework that grounded this study was total quality management (TQM) established by Deming in 1980. Face-to-face semistructured interviews with 2 QA leaders from each organization and documentation including process and training materials provided all the data for analysis. NVivo software aided a qualitative analysis of all collected data using a process of disassembling the data into common codes, reassembling the data into themes, interpreting the meaning, and concluding the data. The resulting major themes were Agile practices, documentation, testing, and lost profits. The results were in contrast to the main themes discovered in the literature review, although there was some overlap. The implications for positive social change include the potential to provide QA leaders with the strategies to improve SQA processes, thereby allowing for improved profits, contributing to the organizations\u27 longevity in business, and strengthening the local economy

Technical Debt Management: Definition of a Technical Debt Reduction Software Engineering Methodology for SMEs

In the past two decades, the metaphor of technical debt has gained significant importance in the field of software engineering. In general, the term is used to describe scenarios when instead of providing a proper solution for a given task, a sub-optimal implementation is used in order to gain short term benefits. Unfortunately, this kind of decisions can - and most of the time do - result in increased maintenance costs and poor evolvability in the long run. Over time, software practitioners further refined the initially source code-focused concept and started to apply the metaphor for a much wider range of software engineering inefficiencies, such as architectural defects, inappropriate documentation or low test coverage. Due to its similarity to financial debt, the analogy has also become a valuable communication tool in situations when there are less technical people involved in discussions. This master's thesis defines a technical debt reduction methodology, which can help SMEs to control the accumulation of technical debt. The proposed methodology can be thought of as a set of steps and good practices that facilitate the long-lasting productivity and profitability of SMEs. Since this field of research is relatively new, the need for publications addressing the topic is still rather high. Due to its numerous negative effects, it is crucial for companies to keep their debt levels as low as possible, which requires a systematic way of managing technical debt. Besides providing such a methodology, the document also intends to raise awareness about the nature and dangers of taking on unreasonable amounts of debt by examining the most important characteristics of the phenomenon. Finally, the thesis presents an industrial case study as well, which aims to showcase how some of the most necessary steps can be taken in practice

Improving Recurrent Software Development: A Contextualist Inquiry Into Release Cycle Management

Software development is increasingly conducted in a recurrent fashion, where the same product or service is continuously being developed for the marketplace. Still, we lack detailed studies about this particular context of software development. Against this backdrop, this dissertation presents an action research study into Software Inc., a large multi-national software provider. The research addressed the challenges the company faced in managing releases and organizing software process improvement (SPI) to help recurrently develop and deliver a specific product, Secure-on-Request, to its customers and the wider marketplace. The initial problem situation was characterized by recent acquisition of additional software, complexity of service delivery, new engineering and product management teams, and low software development process maturity. Asking how release management can be organized and improved in the context of recurrent development of software, we draw on Pettigrew’s contextualist inquiry to focus on the ongoing interaction between the contents, context and process to organize and improve release cycle practices and outcomes. As a result, the dissertation offers two contributions. Practically, it contributes to the resolution of the problem situation at Software Inc. Theoretically, it introduces a new software engineering discipline, release cycle management (RCM), focused on recurrent delivery of software, including SPI as an integral part, and grounded in the specific experiences at Software Inc

Uppskalning av agil systemutveckling: en flerfallstudie av ramverket Scaled Agile Framework

Agile software development methods have been extensively studied in the context for which they were originally developed: small, collocated and cross-functional teams. The proven benefits of agile methods have caused larger organizations and programs to take an interest in them, although adopting such methods at large is far more challenging. Several frameworks for adopting agile methods in a larger context exist but research surrounding them is scarce, consisting almost entirely of industry-produced experience reports. In this thesis we have studied the adoption of a popular scaling framework, the Scaled Agile Framework (SAFe), in two Finnish companies: Comptel, a telecom company; and NAPA, a company specialized in ship design and operation software. We sought to understand why these organizations chose to adopt SAFe, how they did it, what successes and challenges related to the adoptions the organizations encountered, and what effects the adoptions had. We reviewed existing literature in order to find out why and how other organizations had adopted SAFe. Using the results of the review we constructed a set of questions which we used in 16 interviews at the case organizations. The interviews spanned all organizational layers of SAFe: we interviewed people from leadership, middle management, and teams. The results of the interviews support many of the findings of the literature review, showing that planning the adoption in detail, training key personnel, and putting significant effort into the first planning session are key success factors in adopting SAFe. The interviews also showed that internal drivers of change are vital in establishing support for the adoption. Our results indicate that SAFe is a viable option for scaling agile software development, potentially improving visibility, collaboration and alignment.Agila systemutvecklingsmetoder blir allt vanligare och har studerats extensivt i den kontext för vilken de ursprungligen utformats: små självorganiserande och samordnade team. De påvisade fördelarna med agila metoder leder allt fler och större organisationer till att försöka implementera dem---dock växer svårighetsgraden i takt med organisationsstorleken. Ett flertal ramverk för uppskalning av agila metoder existerar, men akademisk forskning kring ämnet saknas nästan helt då majoriteten av litteraturen är fallstudier av varierande kvalité. I detta diplomarbete har vi studerat ibruktagningarna av det populära ramverket Scaled Agile Framework (SAFe) i två finska företag: Comptel, ett globalt telekommunikationsföretag; och NAPA, ett företag specialiserat på mjukvara för formgivning och anförande av fartyg. Vi ville förstå varför företagen valt att ibrukta SAFe, hur de gjort det, vilka problem och framgångsfaktorer de stötte på, samt vilka effekter ibruktaganden hade. Vi undersökte den existerande litteraturen för att få preliminära svar på de ovannämnda frågorna, och skapade en serie intervjufrågor på basis av resultaten. Dessa brukade vi i 16 intervjuer vid fallföretagen, där vi intervjuade ledare, mellanchefer samt medlemmar av team i ett försök att täcka alla de lager av organisationen som även SAFe täcker. Intervjuresultaten stöder många av litteraturstudiens resultat: noggrant planerande av ibruktagningen, utbildning av nyckelroller, samt en storsatsning på det första planeringstillfället är viktiga framgångsfaktorer för ibruktagandet. Intervjuerna visade också att interna pådrivare av förändringen är essentiella för att samla stöd för ibruktagandet. Våra resultat tyder på att SAFe är ett genomförbart alternativ för att uppskala agil systemutveckling, då det kan medföra ökad synlighet, ökat samarbete samt bättre anslutning mot ett gemensamt mål

A DMAIC Framework for Improving Software Quality in Organizations: Case Study at RK Company

Managing quality is a vital aspect in software development world, especially in the current business competition for fast delivery of feature rich products with high quality. For an organization to meet its intended level of excellence in order to ensure its success, a culture of quality should be built where every individual is responsible of quality and not just the software testing team. However, delivering software products with very few bugs is a challenging constraint that is usually sacrificed in order for a company to meet other management constraints such as cost, scope and scheduling. The purpose of this thesis is to apply six sigma DMAIC framework on 'RK’ company (name anonymized) in order to help software organizations focus on improving the quality of their software products. Different phases of DMAIC methodology are applied to one of the largest software applications for ‘RK’ company where critical to quality aspects were identified, production bugs were classified and measured, the causes of the large number of production bugs were specified leading to different improvement suggestions. Several metrics were proposed to help ‘RK’ company control its software development process to ensure the success of the project under study

Implementing Agile Development at Scale: An Industry Case Study

Agile software development methodologies are extremely popular. Their dynamic restructuring of the development process has been seen as the silver bullet for increasing the productivity of software development. A significant number of studies have analyzed the impact of implementing agile techniques. However these are mostly evaluated only in smaller team settings. There is very little reporting done on how agile development methods can be implemented at the team level and scaled up at the program/portfolio level in large software organizations. We present the results of an empirical study conducted at Pearson Education. The study focuses on the penetration of agile development in the organization, agile development practices followed at the team and program level and the perception of agile development by the people in diverse roles. The study shows that about 90% of the respondents use agile development. Of those working in agile development 13% work at the program level and 87% work at the team level. Similar to the practices at the team level, there are standard practices followed at the program level with varying rigor. Most view agile development favorably due to the benefits of agile development. Top benefits reported are improved communication between team members, quick releases and the increased flexibility to changes. Our analysis also indicates that among the population using the non-agile methods, 83% would like to switch to agile methods, while 11% of the agile users would like to switch to non-agile methods. Agile practices are followed more rigorously in larger teams. Respondents who only have experience working with agile methods practice agile techniques more rigorously and perceive it more positively. Respondents with training in agile methods are significantly more inclined to adhere to the process and have an overwhelmingly positive opinion about it. However, challenging conventional wisdom is the finding that experience does not impact the rigor or perception of agile methods. Dependencies among projects seem to have negative impact on the success at the program level due to the challenges in coordination. There is an increased need to focus on testing at the project level; however the rest of the aspects like estimation, prioritization, productivity and time tracking, reviews and continuous integration are working well at the project level. There can be an increased focus on some of the less rigorously used practices at the program level. As training seems to have a significant positive impact on the overall experience of agile development, it would help to increase the focus on training at an organizational level. In conclusion the data indicates that there is a way of successfully scaling up agile methods from the team/project level to the program level by following a disciplined approach. Teams and programs have dependencies, so better synchronization and coordination can be achieved if the agile methods are implemented across all the teams and programs. Training resources, defining and rigorously practicing agile techniques at the program and project level and reducing dependencies are key factors in the success of scaling agile methodologies

Technical Debt Decision-Making Framework

Software development companies strive to produce high-quality software. In commercial software development environments, due to resource and time constraints, software is often developed hastily which gives rise to technical debt. Technical debt refers to the consequences of taking shortcuts when developing software. These consequences include making the system difficult to maintain and defect prone. Technical debt can have financial consequences and impede feature enhancements. Identifying technical debt and deciding which debt to address is challenging given resource constraints. Project managers must decide which debt has the highest priority and is most critical to the project. This decision-making process is not standardized and sometimes differs from project to project. My research goal is to develop a framework that project managers can use in their decision-making process to prioritize technical debt based on its potential impact. To achieve this goal, we survey software practitioners, conduct literature reviews, and mine software repositories for historical data to build a framework to model the technical debt decision-making process and inform practitioners of the most critical debt items

Technical Debt Decision-Making Framework

Software development companies strive to produce high-quality software. In commercial software development environments, due to resource and time constraints, software is often developed hastily which gives rise to technical debt. Technical debt refers to the consequences of taking shortcuts when developing software. These consequences include making the system difficult to maintain and defect prone. Technical debt can have financial consequences and impede feature enhancements. Identifying technical debt and deciding which debt to address is challenging given resource constraints. Project managers must decide which debt has the highest priority and is most critical to the project. This decision-making process is not standardized and sometimes differs from project to project. My research goal is to develop a framework that project managers can use in their decision-making process to prioritize technical debt based on its potential impact. To achieve this goal, we survey software practitioners, conduct literature reviews, and mine software repositories for historical data to build a framework to model the technical debt decision-making process and inform practitioners of the most critical debt items