Phase 2: Investigation of Leading Indicators for Systems Engineering Effectiveness in Model-Centric Programs

Acquisition Research Program Sponsored Report SeriesSponsored Acquisition Research & Technical ReportsThis technical report summarizes the work conducted by Massachusetts Institute of Technology under contract award HQ0034-20-1-0008 during the performance period May 22, 2020 – July 31, 2021. Digital engineering transformation changes the practice of systems engineering, and drives the need to re-examine how engineering effectiveness is measured and assessed. Early engineering metrics were primarily lagging measures. More recently leading indicators have emerged that draw on trend information to allow for more predictive analysis of technical and programmatic performance of the engineering effort. By analyzing trends (e.g., requirements volatility) in context of the program’s environment and known factors, predictions can be forecast on the outcomes of certain activities (e.g., probability of successfully passing a milestone review), thereby enabling preventative or corrective action during the program. Augmenting a companion research study under contract HQ0034-19-1-0002 on adapting and extending existing systems engineering leading indicators, this study takes a future orientation. This report discusses how base measures can be extracted from a digital system model and composed as leading indicators. An illustrative case is used to identify how the desired base measures could be obtained directly from a model-based toolset. The importance of visualization and interactivity for future leading indicators is discussed, especially the potential role of visual analytics and interactive dashboards. Applicability of leading edge technologies (automated collection, visual analytics, augmented intelligence, etc.) are considered as advanced mechanisms for collecting and synthesizing measurement data from digital artifacts. This research aims to provide insights for the art of the possible for future systems engineering leading indicators and their use in decision-making on model-centric programs. Several recommendations for future research are proposed extending from the study.Approved for public release; distribution is unlimited.Approved for public release; distribution is unlimited

Investigation of Leading Indicators for Systems Engineering Effectiveness in Model-Centric Programs

Acquisition Research Program Sponsored Report SeriesSponsored Acquisition Research & Technical ReportsThis technical report summarizes the research conducted by Massachusetts Institute of Technology under contract award HQ0034-19-1-0002 during July 22, 2019 – August 31, 2021. Involved research team members include: Dr. Donna H. Rhodes, Principal Investigator; Dr. Eric Rebentisch, Research Associate; and Mr. Allen Moulton, Research Scientist. Systems engineering practice is evolving under the digital engineering paradigm, including use of model-based systems engineering and newer approaches such as agile. This drives a need to re-examine the existing use of metrics and leading indicators. Early engineering metrics were primarily lagging measures, whereas more recent leading indicators draw on trend information to provide more predictive analysis of technical and programmatic performance of the engineering effort. The existing systems engineering leading indicators were developed under the assumption of paper-based (traditional) systems engineering practice. This research investigates the model-based implications relevant to the existing leading indicators. It aims to support program leaders, transitioning to model-based engineering on their programs, in continued use of leading indicators. It provides guiding insights for how current leading indicators can be adapted for model-based engineering. The study elicited knowledge from subject matter experts and performed literature review in identifying these implications. An illustrative case was used to investigate how four leading indicators could be generated directly from a model-based toolset. Several recommendations for future research are proposed extending from the study. A companion research study (“phase 2”) under contract HQ0034-20-1-0008 provides insights for the art of the possible for future systems engineering leading indicators and their use in decision-making on model-centric programs. For completeness, selected background information and illustrative case are included in the technical reports in both studies. This research aims to provide insights for current practice within programs transforming to digital engineering, for continued use of systems engineering leading indicators. Several recommendations for future research are proposed extending from results of the study.Approved for public release; distribution is unlimited.Approved for public release; distribution is unlimited

Workshop Report Air Force/LAI Workshop on Systems Engineering for Robustness

Workshop repor

A Generalized Options-based Approach to Mitigate Perturbations in a Maritime Security System-of-Systems

Due to the complex and highly dynamic contexts in which systems operate nowadays, it has become crucial that, early in the architecting phase, System Architects take into account options to be utilized throughout the system's lifecycle to improve performance and lifecycle properties, such as flexibility. This paper introduces a preliminary approach that allows for the identification of relevant options, which are capable of mitigating perturbations negatively impacting a system of interest. The approach consists of the generation, evaluation and selection of relevant generalized options (enabling both changeability and robustness), and is demonstrated by application to a Maritime Security SoS case study. The inputs to the process are a list of desired design principles to implement in the system, and a list of perturbations that may affect the delivery of value to stakeholders (options are meant to mitigate perturbations). Four different metrics for option evaluation are proposed, together with techniques that can help during the process of selection of options.Massachusetts Institute of Technology. Systems Engineering Advancement Research Initiativ

The Big Picture: Historical View of Systems Thinking and Social Competencies in Research and Practice

MIT LAI and SEAri Research presentatio

Session on Revitalizing Systems Engineering

LAI 2004 Plenary Conference presentation agend

Defining System Changeability: Reconciling Flexibility, Adaptability, Scalability, and Robustness for Maintaining System Lifecycle Value

Designing and maintaining systems in a dynamic contemporary environment requires a rethinking of how systems provide value to stakeholders over time. Classically, two different approaches to promoting value sustainment may include developing either alterable or robust systems. The first accomplishes value delivery through altering the system to meet new needs, while the second accomplishes value delivery through maintaining a system to meet needs in spite of changes. The definitions of flexibility, adaptability, scalability, and robustness are shown to be different parts of the core concept of “changeability,” which can be described by three aspects: change agents, change effects, and change mechanisms. Cast in terms of system parameter changes, flexibility and adaptability are shown to relate to the origin of the change agent (external or internal to a system boundary respectively). Scalability and robustness, along with the additional property of modifiability, are shown to relate to change effects. The extent of changeability is determined by the number of possible change mechanisms available to the system as accepted by decision makers. Creating changeable systems, which can incorporate both classical notions of alterability and robustness, empowers systems to maintain value delivery over their lifecycle, in spite of changes in their contexts, thereby achieving value robustness to stakeholders over time

An Empirical Investigation of System Changes to Frame Links between Design Decisions and Ilities

Maintaining system performance in the presence of uncertainties in design and operating environments is both challenging and increasingly essential as system lifetimes grow longer. In response to perturbations brought on by these uncertainties, such as disturbances, context shifts, and shifting stakeholder needs, systems can continue to deliver value by being either robust or changeable. These lifecycle properties, sometimes called “ilities”, have been proposed as means to achieve system value sustainment in spite of changes in contexts or needs. Intentionally designing for these lifecycle properties is an active area of research, and no consensus has formed regarding how these and other “ilities” might trade off. This paper describes ongoing research that investigates empirical examples of system changes in order to characterize these changes and to develop a categorization scheme for framing and clarifying design approaches for proactively creating ilities in a system. Example categories from the data for system changes include: the perturbation trigger for the change, the type of agent executing the system change, and the valid lifecycle phase for execution. In providing a structured means to identify system change characteristics, this paper informs future research by framing possible relationships between ilities and design choices that enable them.Massachusetts Institute of Technology. Systems Engineering Advancement Research Initiativ

Evaluating system change options and timing using the epoch syncopation framework

Complex engineering systems face many unknowns with respect to their operating contexts and time-varying stakeholder needs over their lifespan. A useful means for partitioning this problem is to consider a set of static snapshots of contexts with accompanying stakeholder needs over fixed periods of time, herein called “epochs.” Designs can be optimized towards delivering stakeholder utility in a specific epoch or across a variety of epochs. In order to consider the uncertain sequence of epochs experienced by a system, the Epoch Syncopation Framework (ESF) is introduced in this paper. This framework, using Monte Carlo analysis and Markov probability matrices, analyzes the execution of potential system “change mechanisms,” which alter a system over time to respond to epoch shifts. Through an analysis of design tradespaces, the ESF takes into account performance, schedule, cost, and uncertainty regarding experienced epoch shifts. The intended contributions of the ESF include a set of useful baseline designs, desirable change mechanisms, and strategies for executing change mechanisms across a system lifespan. The ESF is demonstrated through an application to an existing dataset containing designs for a “space tug” satellite including its set of potential epochs.Massachusetts Institute of Technology. Systems Engineering Advancement Research Initiativ

Collaborative Systems Thinking: Towards an Understanding of Team-level Systems Thinking

As the engineering workforce ages, skills with long development periods are lost with retiring individuals faster than are younger engineers developing the skills. Systems thinking is one such skill. Recent research, (Davidz 2006), has shown the importance of experiential learning in systems thinking skill development. However, an engineering career begun today has fewer program experiences than in past decades because of extended program lifecycles and a reduction in the number of new large-scale engineering programs. This pattern is clearly visible in the aerospace industry, which (Stephens 2003) cites as already experiencing a systems thinking shortage. The ongoing research outlined in this paper explores systems thinking as an emergent property of teams. Collaborative systems thinking, a term coined by the authors to denote teamlevel systems thinking, may offer an opportunity to leverage and develop a skill in short supply by concentrating on the team in addition to the individual. This paper introduces the proposed definition for collaborative systems thinking, as developed by the authors, and the outlines the structure and progress of ongoing case research into the role of organizational culture and standard process usage in the development of collaborative systems thinking