Panel #25: Unlocking the Value of Acquisition DataNaval Postgraduate SchoolApproved for public release; distribution is unlimited

Anton, Philip

Drezner, Jeffrey

Levedahl, Alexis

McKernan, Megan

Munson, Ken

Newberry, Sydne

English

Panel #25: Unlocking the Value of Acquisition DataApproved for public release; distribution is unlimited.Naval Postgraduate Schoo

Calhoun, Institutional Archive of the Naval Postgraduate School

Calhoun: The NPS Institutional ArchiveDSpace RepositoryReports and Technical Reports Faculty and Researchers' Publications2020-04-27Assessing Department of Defense Use of DataAnalytics and Enabling Data Management toImprove Acquisition OutcomesMcKernan, Megan; Drezner, Jeffrey; Munson, Ken; Anton,Philip; Newberry, Sydne; Levedahl, AlexisMonterey, California. Naval Postgraduate Schoolhttp://hdl.handle.net/10945/64753This publication is a work of the U.S. Government as defined in Title 17, UnitedStates Code, Section 101. Copyright protection is not available for this work in theUnited States.Downloaded from NPS Archive: CalhounApproved for public release; distribution unlimitedAssessing Department of Defense Use of Data Analytics and Enabling Data Management to Improve Acquisition Outcomes Philip Anton and Megan McKernan, RAND NDRINaval Postgraduate School’s 17th Annual Acquisition Research Symposium Image via md3d - Fotolia.Congressional Tasking:The briefing from the Secretary of Defense shall address:“The extent to which data analytics capabilities have been implemented within the military services, DOD laboratories, test centers, and [FFRDCs] to provide technical support for acquisition program management”“The amount of funding for intramural and extramural research and development activities to develop and implement data analytics capabilities in support of improved acquisition outcomes”“Whether the curriculum at the National Defense University, the Defense Acquisition University, and appropriate private-sector academic institutions includes appropriate courses on data analytics and other evaluation-related methods and their application to defense acquisitions”“Steps being taken to appropriately expose acquisition datain an anonymized fashion to researchers and analysts”“The potential to increase the use of analytical capabilities for acquisition programs and offices to improve acquisition outcomes”“Any potential improvements, based on private-sector best practices, in the efficiency of current data collection and analysis processes that could minimize collection and delivery of databy, from, and to government organizations”Joint Explanatory Statement of the Committee of Conference (H.R. 114-840, pp. 1125–1126) Accompanying the National Defense Authorization Act (NDAA) for fiscal year (FY) 2017, S. 2943.Extent TrainR&DOpportunitiesStatus QuoOpportunitiesStatus Quo Status Quo Status QuoDataDataDataIncrease Use Efficient DataExpose DataResearch approach addresses the breadth of Congressional topics: data, analysis, and improvement opportunities• Inputs:Discussions with select stakeholdersExisting literature, public budget exhibits, policies, legislation, training course catalogs and curricula; inventories of relative IT systemsExperience, knowledge, and judgement of authors, as noted• Study techniquesFunctional decomposition and mapping; workforce cost analysis; budget and solicitation analysis; literature review and analysis; analysis of industry best practices; analysis of DoD data management and analytic activities.• AssumptionsData analytics are a means for improving decisions• We lay out analytics that support specific decisions, but we generally do not directly study cause-and-effectAdvanced business analytics have some DoD applications• Found some application and explorations, but further applications require continued investigationResearch approach embraces the breadth of congressional interest with some limitations on study scope and depth• Broad scope driven by inclusive definitions of “acquisition” and “analytics” (see next slide) as well as the broad nature of Congress’s questions• Did not assess what specific acquisition data or data analytics are needed• A survey (a “data call”) was proposed to solicit specific examples of data analytics underway in the DoD acquisition community, but it was deemed infeasible within available time and resourcesIn lieu of a survey, primary inputs include discussions with select stakeholders; existing literature; public budget exhibits; policies and legislation; training catalogs and curricula; and inventories of relevant IT information systems• Experience, knowledge and author judgment was used to synthesize information and fill gaps in primary inputs, published research and other secondary dataWhere author judgment is applied, it is notedWe used the following broad definitions of Acquisition and Data Analytics based on DoD’s definition and Congressional languageAcquisitionThe conceptualization, initiation, design, development, test, contracting, production, deployment, integrated product support (IPS), modification, and disposal of weapons and other systems, supplies, or services (including construction) to satisfy DoD needs, intended for use in, or in support of, military missions. Defense Acquisition University (DAU), 2017Data AnalyticsData analysis, measurement, and other evaluation-related methods (i.e., techniques to assess and analyze data) to inform acquisition decisions, policymaking, program management, evaluation, and learning.(based on conference report elements)Bottom-Line Conclusions (in prose)Acquisition data analytics is extensive:• DoD Data and analytics supports acquisition decision-making through a wide range of acquisition functionsThose functions include fields used for decades: e.g., engineering, test and evaluation, cost estimating, auditing, and many othersThe data analytics employed by these applied functions range from simple empirical methods to advanced engineering and estimating approaches• It is difficult to precisely measure the magnitude of annual investment in data analytics, but rough macro-level, parametric estimates are in the billions of dollars.• Cases of some major acquisition program decisions that went counter to the extant data and analysis appear to have been made for other strategic reasons or drivers—not for broad lack of data analyticsStill, there is room for improvement:• Applying advanced commercial business data analytics to DoD acquisition program data is limited and mostly in the exploratory phase• Barriers to expanded data and analytics include:Some cases of inefficient collection and lack of sharing data due to cultural, security, and investment issuesLimited analytic desktop software for general staffWorkforce trained on job-specific data analytics rather than broader analytic skills• Progress has been made to:Apply commercial best practices to efficient data management and sharingAdd commercial business intelligence tools (e.g., Tableau, QlikSense) to information systems to facilitate data processing, analysis, and visualizationOffer new courses in data analytics at acquisition training institutions• Continued investments and progress is neededDoD appears to recognize this based on continued (albeit constrained) investmentsQ1. To what extent have data analytics capabilities been implemented?Findings:• The breadth of data analytics across acquisition functions gives a measure of extent (see next slide)• Spending indicates extent:One estimate: $11–15 billion/year on analytic-related workforceAbout $3 billion/year on IT systems for acquisition (FY 2019 budget exhibits)• Major IT systems supporting acquisition processes and data collection. Some have analytic layers.• Data analyses are often available, but final decisions reflect risks and other equitiesA breadth of DoD data analytics support acquisition functions and decisions• Strategy, planning, and budgeting• Data requirements and use cases• Authoritative sourcing• Collection, archiving, curation, and sharing• Security• Backup and recovery• Training and support• Data definitions and standards• Assessment, auditing, cleaning, and transformation• PurgingData Management (examples)General:Quantitative analysis, qualitative analysis, data mining, statistical analysis, machine learning (e.g., classification, clustering, outlier detection, filtering), semantic analysis (e.g., natural language processing, text analytics , sentiment analysis, and visual analysis (e.g., heat maps , time series plots, network graphs, spatial data mapping), etc.Applied:Market research, requirements analysis, mission analysis, technical maturity analysis, risk analysis and framing assumptions, DoD’s risk-management framework, affordability analysis, cost estimating, schedule analysis, analysis of alternatives, tradeoff analysis, budget analysis, legal and policy analysis, engineering analysis, T&E, security, quality analysis, supply chain analysis, reliability and life-cycle sustainment analysis, contracting analysis, production analysis, auditing, sustainment analysis, etc.Analytic Techniques (Examples) AcquisitionDecision-Making• Congress• Defense acquisition executive and others at OSD• Component Acquisition Executive• Program Executive Officer• Program ManagerKey EnablerDoD systems are evolving from isolated data to shared data with analytic layers; More advanced analytics are exploratoryDeveloping or FieldedResearchExample DoD SystemsMaturityIntegrated data, aggregation, and visualization toolsAdvancedData AnalyticsPreset calculations and dashboardsData integration for risk analysisFFRDCsText understanding from newsFFRDCsSupply chain network analysisDoD, FFRDCsPrediction and detection of program problemsDARPA, Labs, UARCs, UniversitiesDAMIR (Defense Acquisition Management Info Retrieval) DAVE/AL (Def. Acquisition Visibility Environment / Analytic Layer)OSD A&SOSD, ArmyPMRT (Project Management Resource Tools)CADE (Cost Assessment Data Enterprise)OSD CAPEAir ForceLIMS-EV (Logistics, Installations, and Mission Support—Enterprise View)Air ForceEVM-CR (Contract Performance)Data archives with simple analyticsRDAIS (R&D and Acquisition Information System )feedingfeedingfeeding*FPDS-NG (Fed. Procurement Data System--NextGen)DIBNow (Industrial Base)feedingOSDNavyFederal OSDWhile simpler systems remain, DoD‘s trend has been toward increased data sharing and more modern analytic toolsExploratory researchEvolving applicationsSET (Systems Engineering Transformation—Model-Based)NavyMaturing applicationsMature applicationsQ2. What is the potential to increase the use of analytic capabilities to improve acquisition outcomes?Findings: Options include:• Develop a data-analytics strategy between acquisition domains• Continue maturing data collection, access, and analytic layers on systems• Expand analytic tool availability• Create statutory authorities for external analyst protected access to data • Establish policy of full data access for analysts• Improved decision-maker incentives and understanding of data analytics may improve use of available data• Expand types of analysisPerspectives:• Commercial advanced data analytics do not always apply to governmentEven industry has problems implementing data analytics• Need more predictive indicators• Data analytics will not eliminate acquisition risksQ3. How much R&D funding is there to develop and implement data analytic capabilities?Findings:• FY 2019 DoD RDT&E budget request:Estimate: roughly $200 million (+/- 80%) across 31 program elements (PEs)• Found specific discussion related to data analytics for acquisition$520 million/year for IT systems (up from $313 million in FY 2017)• Provides data and sometimes have analytic layers• Some SBIR and STTR topics in January 2019 solicitations for acquisition data analytics• Found anecdotes of exploratory research on advanced analytics for acquisitionNotes:• This does not include R&D outside acquisition properMilitary operations, budgeting, requirements, or intelligence analysis in support of acquisition• Again, direct application of advanced commercial data analytics is challengingQ4. What are private-sector best practices that could minimize collection and delivery of data by, from, and to government organizations?Findings:• Private-Sector best practices include:Identify specific questions that leadership needs to answer in order to make informed decisionsPlan and prioritize what data are needed using an organization’s data strategyDefine data and establish common definitions between organizations using that dataDesignate single authoritative source for each datum or dataset, then share existing authoritative datasets between systems via technical meansEmphasize that data are corporate-wide assets, not owned by local unitsDoD adoption to date:• DoD is pursuing most of these practicesCommon PM tool suites are less common but being pursuedOther acquisition IT systems provide data feeds (e.g., contracting, auditing)Open systems and data sharing have reduced duplication and ensured common dataData-element sensitivity meta-labeling• Data sharing and ownership challenges remainOpportunities:• Continue pursuing relevant best practices from the private sector• Clarify that data are DoD-wide assets and address disincentives to data sharingClarify oversight extent and rolesContinue research on sensitivity upgrades from data-aggregationQ5. What steps are being taken to expose anonymized data to researchers and analysts?Findings:• Generally, DoD is not anonymizing acquisition data for various reasonsA counter-example is Personally Identifiable Information (PII) • DoD is:Identifying available dataImproving internal data transparency through technical meansAdding researcher and analyst accounts on data systemsRemoving classified and sensitive data elements before publishingPerspectives:• Anonymization has significant limitationsSome anonymization can be broken with modern correlation toolsMeta-data are required for correlation analysis and cannot be anonymizedData sensitivity is often not stored at the data element levelProcedures for categorizing and handling sensitive data are complicated, slow, and not well understood; incentives impede progressAggregating data can increase data sensitivity and classificationOpportunities:• Process data in a clean roomLimits use of anonymization-breaking tools• Possible option: blind analysisAlgorithms run in a black box, only showing the resultsChallenges include data cleaning and understandingResearch would be needed• (See prior slide for other data sharing opportunities)Q6. Do training institutions include appropriate courses on data analytics and other methods and their application to defense acquisitions?Findings:• Yes, DAU, NPS, NDU, AFIT, and partner universities and institutions* offer:Applied methods and tools coursesApplied data analytics coursesGeneric data analytics courses and electivesAdvanced analytics courses are predominantly at NPS and partner institutions• Enrollments at DAU in FY 2018 indicates a reasonable stratification:~150,000: Applied methods and tools courses~  60,000: Applied data analytics courses~    3,300: Generic data analytics courses• These courses should help staff understand how to request analysis and understand the resultsPerspectives:• Not everyone in acquisition can or should become a data scientist• On-the-job training is also important• Analytic layers on information systems are fairly intuitive and come with online help and training sessions• As in industry, few people understand both data science and the application area• Did not assess quality of coursesOpportunities:• Implement rotations in analytic-based offices• Better inform personnel of available analytics-based courses• Encourage analytics-based training* Georgia Tech, American University, George Mason University, Georgetown University, George Washington University, Johns Hopkins University, Stanford, University of Michigan, Google, IBM, and DoD Cyber Crime Center (DC3) Cyber Training AcademyPotential opportunities and actions (by stakeholder)Congress• Declare in 10 USC 2222(e) that acquisition data are “Common Enterprise Data” and thus must be shared across DoD• Make permanent the FFRDC access to sensitive data under Sec. 235 of the FY 2017 NDAAUSD(A&S) with CMO, CIO, CDOs, and SAEs• Address disincentives to data sharing• Enable DoD-wide access for appropriate analysts to sensitive data• Facilitate access to analytic tools through virtual computing environments and an embellished list of standard software approved for installation on DoD computers• Continue R&D on improved data and analytic systems and new applications in acquisition• Develop a data-analytics strategy across acquisition domainsDoD information managers• Continue pursuing PM and process suites with data outputs to feed oversight information systems• Continue maturing data collection, access, and analytic layers on data systems• Continue compiling and sharing catalog of available dataDefense acquisition training institutions• Continue offering data science courses and applied data analytics for staff, management, and rising leadersSteps for Future Research• For CongressIdentify DoD acquisition leadership structures that streamline acquisition while balancing decision incentives and authoritiesIdentify how statutes can be changed to give efficient access to sensitive data for UARCs, contractor labs, and SETA support contractor analysts while providing appropriate data protections• USD(A&S) with CMO, CIO, CDOs, SAEsIdentify how to address disincentives to data sharingDevelop policies and processes for analysis on data aggregation and classification upgradesDevelop policies and approaches for granting DoD-wide access to different DoD information systems for government and contractor analystsIdentify the minimum data needed at what level and for what purposes given costs and benefitsConduct detailed analysis to create a cross-domain DoD data-analytics strategy• For DoD acquisition training institutionsConduct an assessment of the quality and practical utility of data-analytics courses• New or expanded analysesExplore better ways to objectively separate effects of uncertainties and externalities in sustainment metricsRecognize and explore mission-level (versus program-level) analysisExamine better ways to utilize framing assumptions and their metricsConduct institutional performance analysisIdentify core data needed to answer important questionsMany of the options identified are concepts that require further research to develop specific implementation approaches.Acknowledgements• OSD sponsors and staffHon. Kevin Fahey, David Cadman, Brian Joseph, Mark Krzysko, Phil Rodgers, Nicolle Yoder, Rhonda Edwards, Mark Hogenmiller• DAUJames Woolsey, Leslie Deneault, Richard Hoeferkamp, Bill Parker, David Pearson, Mary Redshaw• Department of the ArmyRay Gagne, MAJ Thomas P. Dirienzo, Michael J. Gully, Michael A. Larsen, Alpesh Patel, COL Joseph W. Roberts, Kevin Samuels, COL Gregory Smith, Leo Smith• Department of the NavyBarbara (BJ) White-Olson, William Bray, CDR William Benham, Robert Borka, Charlotte Helan, Dale Moore, Linda Wack• Department of the Air ForceJohn Miller, Mildred Bonilla-Lucia, Ralph DiCicco, and Col. David J. Sanford• RANDFrank Camm, Scott Comes, Justin Grana, Kristin Lynch, Lauren Mayer, Igor Mikolic-Torreira, Brian Persons, Joel Predd, William Shelton, Jia Xu, Sarah Meadows, and Obaid Younossi

Assessing Department of Defense Use of Data Analytics and Enabling Data Management to Improve Acquisition Outcomes

https://calhoun.nps.edu/bitstream/handle/10945/64753/SYM-AM-20-050_slides.pdf?sequence=4

Assessing Department of Defense Use of Data Analytics and Enabling Data Management to Improve Acquisition Outcomes

Abstract

Similar works

Full text

Available Versions

Calhoun, Institutional Archive of the Naval Postgraduate School