NPS NRP Project PresentationMission Scenario Generation and Characterization to Support Acquisition Decisions for Long Range Precision Fires-Maritime (LRPF-M)Naval Surface Warfare Center (NSWC), Division DahlgrenThis research is supported by funding from the Naval Postgraduate School, Naval Research Program (PE 0605853N/2098). https://nps.edu/nrpChief of Naval Operations (CNO)Approved for public release. Distribution is unlimited.

Giammarco, Kristin M.

English

Calhoun, Institutional Archive of the Naval Postgraduate School

Mission Scenario Generation and Characterization to Support Acquisition Decisions for Long Range Precision Fires-Maritime (LRPF-M)

NPS NRP Project PosterMission Scenario Generation and Characterization to Support Acquisition Decisions for Long Range Precision Fires-Maritime (LRPF-M)Naval Surface Warfare Center (NSWC), Division DahlgrenThis research is supported by funding from the Naval Postgraduate School, Naval Research Program (PE 0605853N/2098). https://nps.edu/nrpChief of Naval Operations (CNO)Approved for public release. Distribution is unlimited.

Calhoun: The NPS Institutional ArchiveDSpace RepositoryFaculty and Researchers Faculty and Researchers' Publications2019-12Mission Scenario Generation andCharacterization to Support AcquisitionDecisions for Long Range PrecisionFires-Maritime (LRPF-M)Giammarco, Kristin M.Monterey, California: Naval Postgraduate Schoolhttp://hdl.handle.net/10945/70010This publication is a work of the U.S. Government as defined in Title 17, UnitedStates Code, Section 101. Copyright protection is not available for this work in theUnited States.Downloaded from NPS Archive: CalhounNaval PostgraduateSchoolResearch Questions• What are the alternative possible flows for a baseline mission, given events that can occur in the system’s environment?• Can the mission scenarios be characterized with durations, probabilities and/or costs to support acquisition decisions? A behavior model formally captured communication and decision flows among operations and control entities of a Navy/Marine Expeditionary Ship Interdiction System (NMESIS) Platoon operating in a littoral environment.Mission Scenario Generation and Characterization to Support Acquisition Decisions From 105 to 22 Alternative Flows, and Characterization • Logical and simplifying constraints reduced the number of valid scenarios while keeping a formal record of important assumptions• Implicit assumptions can be made explicit for all to understand• Constraints can be toggled on or off to admit or reject different combinations of events during validation of the scenarios• Assigned notional durations and probabilities to key events to estimate whole scenario characteristics‒ MP-Firebird calculations exclude zero-probability scenarios rejected by constraints (Quartuccio 2019)‒ Notional values were used to test the approach instead of experiential or historical dataConclusions & Further Research• MP modeling supports requirements discovery and analysis by providing scenario combinations that are unavailable in such numbers in a manual scenario generation process.• Quantify the value/savings (e.g., in time or dollars) of having exposed the assumptions, constraints and/or requirement needs for a system or program office. Researcher(s): Associate Professor Kristin Giammarco PhD, Faculty Associates-Research Ross Eldred and Ms. Marianna Jones; Graduate School of Engineering and Applied Sciences, Systems EngineeringAssociate Professor Mikhail Auguston PhD; Graduate School of Operational and Information Sciences, Computer ScienceTopic Sponsor:  Marine Corps Systems Command (MCSC)A “Global View” computes resulting model statistics.1 Mission, 105 Alternative Flows• A Monterey Phoenix (MP) behavior model exhaustively generated all possible combinations of alternative flows, e.g., how platoon:‒ deploys or fails to deploy‒ position remains concealed or exposed‒ is fully mission capable or partially mission capableExample Flow (Trace 6)NRP Project ID: NPS-19-N248-ADissertation: Quartuccio, J. (2019.)  Identification of behavior patterns in system of systems architectures (Doctoral dissertation).Distro A – Approved for public release; distribution is unlimited.

NPS NRP Executive SummaryMission Scenario Generation and Characterization to Support Acquisition Decisions for Long Range Precision Fires-Maritime (LRPF-M)Naval Surface Warfare Center (NSWC), Division DahlgrenThis research is supported by funding from the Naval Postgraduate School, Naval Research Program (PE 0605853N/2098). https://nps.edu/nrpChief of Naval Operations (CNO)Approved for public release. Distribution is unlimited.

Calhoun: The NPS Institutional ArchiveDSpace RepositoryFaculty and Researchers Faculty and Researchers' Publications2019-12Mission Scenario Generation andCharacterization to Support AcquisitionDecisions for Long Range PrecisionFires-Maritime (LRPF-M)Giammarco, Kristin M.Monterey, California: Naval Postgraduate Schoolhttp://hdl.handle.net/10945/70009This publication is a work of the U.S. Government as defined in Title 17, UnitedStates Code, Section 101. Copyright protection is not available for this work in theUnited States.Downloaded from NPS Archive: CalhounNPS NRP Executive Summary Mission Scenario Generation and Characterization to Support Acquisition Decisions Report Date: 12/31/19 Project Number (IREF ID): NPS-19-N248-A Naval Postgraduate School, Graduate School of Engineering and Applied Sciences      MONTEREY, CALIFORNIA MISSION SCENARIO GENERATION AND CHARACTERIZATION TO SUPPORT ACQUISITION DECISIONS   Period of Performance: 01/02/2019-12/30/2019  Researchers: Principal Investigator (PI): Associate Professor Kristin Giammarco, PhD, GSEAS Systems Engineering (SE) Department Additional Researcher(s): Associate Professor Mikhail Auguston, GSOIS Computer Science Department, Faculty Associate-Research Ross Eldred, GSEAS SE Department, and Faculty Associate-Research Marianna Jones, GSEAS SE Department Student Researcher(s): SE PhD Candidate John Quartuccio, GSEAS SE Department  Prepared for: Topic Sponsor Lead Organization: Marine Corps Systems Command (MCSC) Topic Sponsor Name: Mr. Craig T. Melton Topic Sponsor Contact Information: craig.melton@usmc.mil, 703-432-3950   Approved for public release; distribution is unlimited.  NPS NRP Executive Summary Mission Scenario Generation and Characterization to Support Acquisition Decisions Report Date: 12/31/19 Project Number (IREF ID): NPS-19-N248-A Naval Postgraduate School, Graduate School of Engineering and Applied Sciences  EXECUTIVE SUMMARY   Project Summary  Often, system requirements are shaped around operational scenarios that capture the anticipated uses for a system.  However, this method becomes problematic when the number of anticipated and tested scenarios are far less than the number of scenarios that the actual system may potentially encounter. During early design phases, these missed scenarios often result in missed requirements, causing systems to fail to meet needs and tacit expectations.  This research helps with conception and consideration of system requirements early in design, demonstrating a new and far more comprehensive approach for mission scenario generation and characterization. Using source data provided by the topic sponsor’s delegate, we created a Monterey Phoenix (MP) behavior model for a baseline littoral operation mission. We then enumerated dozens of alternative possible flows for that mission, given events that can occur in the system (a platoon), and in the system’s environment.  Our findings show how the MP model exposed assumptions about the operation, which were codified into formal logical and simplifying constraints. When these constraints are applied together, a reduction in the number of valid scenarios at scope 1 from 105 to 22 occurred, and when the constraints are considered separately, each can be used to drive the conception and consideration of mission and system requirements.     Keywords: scenario generation, requirements analysis, formal methods, behavior modeling, Monterey Phoenix, MP  Background MP (Auguston, 2009) is a Navy-developed formal language and approach for modeling systems, software, hardware, people, organizational, and/or environmental behaviors and their interactions with one another (Auguston & Whitcomb, 2010).  MP can be used to describe high-level operational processes, other business processes or architecture designs, and can support behavior descriptions down to a detailed design level (Auguston, Giammarco, Baldwin, Crump, & Farah-Stapleton, 2015).  It differs from other behavior modeling approaches by using lightweight formal methods (Jackson, 2012) to generate a scope-complete set of scenario variants from system behaviors and interactions within and among independently specified systems.  Scope-complete means the set of automatically generated scenarios is exhaustive up to a user-defined number of loop iterations (the scope limit), which is usually between 1 and 3.  This degree of completeness, even for small scopes, results in an entirely new way to conduct verification and validation (V&V) of human, system and software behavior models (Giammarco et al, 2018).     Through scope-complete behavior modeling early in the lifecycle, when system architecture issues are less expensive to find and fix, this work provides Marine Corps Systems Command (MCSC) with a more comprehensive approach for exposing latent assumptions, constraints and requirements pertaining to NPS NRP Executive Summary Mission Scenario Generation and Characterization to Support Acquisition Decisions Report Date: 12/31/19 Project Number (IREF ID): NPS-19-N248-A Naval Postgraduate School, Graduate School of Engineering and Applied Sciences their systems. Using MP automation for system architecture V&V results in the generation of more operational scenarios in less time, and with fewer errors, than can be achieved manually.  This more comprehensive set of scenarios results in broader consideration of potential situations the actual system may encounter, and earlier identification of requirements necessary for mission success. More generally, this work demonstrates how MP is used to expose assumptions about an operation, derive logical and simplifying constraints from those assumptions (Quartuccio, 2019), and use those constraints to drive system requirements that may have otherwise been overlooked.     The scope-complete scenario generation capability of MP was used to explore the following research questions: 1) What are the alternative possible flows for a baseline mission, given events that can occur in the system’s environment? 2) Can the mission scenarios be characterized with durations, probabilities and/or costs to support acquisition decisions?  Note: A limitation of this study is the availability of subject matter experts (SMEs) willing to volunteer their time in order to validate the model and its constraints. Therefore, the sample constraints applied to this model are only examples of constraints one might apply. Another limitation pertains to the assigning of probabilities to events; notional values were used in place of SME experiential or historical data.    Findings and Conclusions The research methodology used the MP-Firebird tool for scope-complete scenario generation and characterization. To begin the research, we collected the source data (Department of the Navy, 2017) from the topic sponsor’s delegate.  A draft MP model was created for a baseline scenario from the information in these documents, and the operational content of the model was internally validated by a former Navy Lieutenant (now faculty associate-research) and by a former Marine.  Once these SMEs were satisfied that we had a reasonable representation of a baseline scenario, we proceeded with modeling some alternatives to the baseline scenario that represented other “sunny day” cases as well as “rainy day” cases (off-nominal or failure scenarios). Our PhD student, John Quartuccio, contributed findings from his research on Bayesian belief networks for making probability calculations, as well as a pattern for object/sensor/processor/actor behaviors and interactions that applies to the MP model of littoral operations developed for this project.    Specifically, this research effort thoroughly explored our first research question, initially generating 105 alternative possible flows at scope 1 for a baseline littoral mission.  The mission model formally captured the communication and decision flows for a Navy/Marine Expeditionary Ship Interdiction System platoon.  Examples of alternatives include deployment or failure to deploy, platoon position remains concealed or exposed, and platoon is fully mission capable or partially mission capable.  The MP model exhaustively generated all possible combinations of alternatives subject to event coordination constraints. After the application of systematically discovered logical and simplification constraints, 22 valid scenarios remained at scope 1.  NPS NRP Executive Summary Mission Scenario Generation and Characterization to Support Acquisition Decisions Report Date: 12/31/19 Project Number (IREF ID): NPS-19-N248-A Naval Postgraduate School, Graduate School of Engineering and Applied Sciences Two important consequences of MP model structure are that 1) simple, implicit assumptions can be made explicit for all to understand and 2) assumptions coded as constraints can be toggled on or off to admit or reject different combinations of events during validation of the scenarios.  These findings led to the conclusion that MP modeling supports requirements discovery and analysis by providing a high number of scenario combinations which cannot be achieved in a manual scenario generation process.    Our secondary research question explored the assignment of notional durations to key events to estimate whole scenario durations, as well as notional probabilities, for the key alternatives to calculate whole scenario probabilities.  We verified a finding by systems engineering PhD student John Quartuccio that probabilities calculated by MP-Firebird do not account for zero-probability scenarios rejected by constraints (2019).  This important contribution can be used to enable computation of Bayesian-based scenario probability estimates that account for scenarios purged by constraints, such as those identified in this project.   Recommendations for Further Research MP-Firebird is a powerful tool for scenario generation that up to a user-defined scope of execution, is exhaustive.  Our research has resulted in an MP model of littoral operations with a clear connection to a reusable pattern of interactions, exposed some tacit assumptions about the operation that may not have otherwise been formally recognized, and advanced our understanding of probabilities in scope-complete behavior models. Potential future work includes further testing of the MP modeling approach on a real system at MCSC to see if it can expose real requirements, while also working to quantify the value (e.g., in time or dollars) of having exposed the assumptions, constraints and need for requirements to an actual program office.  A Naval Postgraduate School student working at MCSC would be an ideal candidate for involvement in this future work, as this relationship would help with the collection of source data for the actual system. Lastly, the completion of a standalone installation of MP software would facilitate running models of real systems in the MCSC work environment.    References Auguston, M. (2009). Monterey Phoenix, or how to make software architecture executable. Proceedings of the OOPSLA’09/Onward conference, OOPSLA Companion, pp.1031-1038 Auguston, M., Giammarco, K., Baldwin, W.C., Crump, J., & Farah-Stapleton, M.  (2015). Modeling and verifying business processes with Monterey Phoenix, Procedia Computer Science: 44, pp. 48-57.  Auguston, M. & Whitcomb, W. (2010). System architecture specification based on behavior models. Proceedings of the 15th ICCRTS Conference (International Command and Control Research and Technology Symposium), Santa Monica, CA. Doob, J. L. (1996.)  The development of rigor in mathematical probability (1900-1950). The American mathematical monthly, vol. 103, no. 7, pp. 586–595. Department of the Navy. (2017.) Littoral operations in a contested environment (Unclassified edition). Retrieved from https://mca-marines.org/wp-content/uploads/Littoral-Operations-in-a-Contested-Environment.pdf NPS NRP Executive Summary Mission Scenario Generation and Characterization to Support Acquisition Decisions Report Date: 12/31/19 Project Number (IREF ID): NPS-19-N248-A Naval Postgraduate School, Graduate School of Engineering and Applied Sciences Giammarco, K., Carlson, R., Blackburn, M., Auguston, M., Gehris, R., Jones, M., Allen, B., Shifflett, D., Quartuccio, J., Giles, K., Wolfgeher, C., Parker, G. (2018.) Verification and validation (V&V) of system behavior specifications. (Report No. SERC-2018-TR-116), Retrieved from https://sercuarc.org/publication/?id=203&pub-type=Technical-Report&publication=SERC-2018-TR-116-Verification+and+Validation+%28V%26V%29+of+System+Behavior+Specifications Jackson, D. (2012.) Software Abstractions: logic, language, and analysis. Cambridge, MA: MIT press. Quartuccio, J. (2019.)  Identification of behavior patterns in system of systems architectures (Doctoral dissertation).  Pending publication on https://calhoun.nps.edu/handle/10945/16  Acronyms Joint Force Maritime Component Commander    JFMCC Marine Corps Systems Command     MCSC Monterey Phoenix       MP subject matter expert      SME verification and validation      V&V    

https://calhoun.nps.edu/bitstream/handle/10945/70011/NPS-19-N248-A_NRP_Presentation_Giammarco.pdf?sequence=1

Mission Scenario Generation and Characterization to Support Acquisition Decisions for Long Range Precision Fires-Maritime (LRPF-M)

Abstract

Similar works

Full text

Available Versions

Calhoun, Institutional Archive of the Naval Postgraduate School

Calhoun, Institutional Archive of the Naval Postgraduate School

Calhoun, Institutional Archive of the Naval Postgraduate School