Prepared for the Naval Postgraduate School, Monterey, CA 93943.Naval Postgraduate SchoolApproved for public release; distribution is unlimited.Approved for public release; distribution is unlimited

Housel, Tom

English

Approved for public release; distribution is unlimited.Approved for public release; distribution is unlimited.Prepared for the Naval Postgraduate School, Monterey, CA 93943.Naval Postgraduate Schoo

Calhoun, Institutional Archive of the Naval Postgraduate School

Calhoun: The NPS Institutional ArchiveDSpace RepositoryAcquisition Research Program Faculty and Researchers' Publications2021-05-21Evaluating Investments in Technology: Five ApproachesHousel, TomMonterey, California. Naval Postgraduate Schoolhttp://hdl.handle.net/10945/68219This publication is a work of the U.S. Government as defined in Title 17, UnitedStates Code, Section 101. Copyright protection is not available for this work in theUnited States.Downloaded from NPS Archive: CalhounEvaluating Investments in Technology: Five ApproachesDr. Tom HouselProfessorInformation Sciences DepartmentNaval Postgraduate SchoolPhone: 831-656-7657Email: tjhousel@nps.eduAgenda• Problem Space: Tech Investment lifecycle Issues• Continuous, Job Shop production models• 5 Analysis and Management Tools: • Balanced Scorecard (BSC)• Earned Value Management (EVM)• Knowledge Value Added (KVA)• Lean Six Sigma (L6𝛿𝛿)• Integrated Risk Management (IRM)• Selection Criteria to Consider• When to Apply 5 Approaches in the Tech Investment LifecycleProblem Space  • The problem is that current analysis and management tools based on continuous production are not adequate to address the evolving requirements of complex IT systems.• This is a problem because acquisitions of increasingly complex IT systems require a broader set of management and analysis tools to ensure successful acquisitions in a Job Shop production context.• The purpose of this study was to suggest a set of criteria for selecting management and analysis tools that could help acquisition professionals successfully navigate the acquisition lifecycle of Job Shop products.• This research is important because acquisition managers need a wider variety of tools to help them optimally analyze and manage their increasingly complex acquisition of IT based portfolios.Continuous, Job Shop• DOD acquisition economics: Job Shop, Continuous Production?• Continuous production: • Takes advantage of production learning curve• Lot size – very large• Commodity valuation• Job Shop:• Lot size of 1• Customized outputs• Higher premium on valueTech Investment Lifecycle Management and Control Techniques•Balanced Score Card (BSC)•Lean Six Sigma (L6δ)•Earned Value Management (EVM)•Knowledge Value Added (KVA)• Integrated Risk Management (IRM)Five Approaches: Selection Criteria to ConsiderLevel of Analysis Focus of Analysis Time to Perform AnalysisProcess: KVA, L6δ, BSC, IRMCost Savings: L6δ, EVM, BSC, IRMSchedule: EVMValue: KVA = outputsBSC, IRM: 3-6 monthsTask:  L6δ, EVM Cost Savings: L6δ, IRMValue = Cost+Schedule (on time)Cycle Time: L6δ, BSCEVM: 5-? months set up time (depends on requirements)Organization: BSC Strategic Competitive Advantages: BSC, IRMValue = Revenue: BSCStrategic Real Options for Investment in ITL6δ: 3-? months (depends on level of process complexity)KVA: 2 days – 1 month(depends on level of analysis)Five Approaches: When to Apply in the Tech Investment LifecyclePre-Investment Strategic Goal AlignmentImplementation Post ImplementationKVA (As-Is) BSC (Align strategywith performance metrics)EVM (Monitor cost and schedule, adjust as needed)KVA (Monitor ROI, ROK)L6(Id waste, value added)IRM (Identify the strategic options for IT investments)KVA (To-Be, ROI, ROK)L6( Assess and monitorcost, waste reduction)Other Other IRM (Use the project management toolswithin the IRM suite)OtherPre-Materiel Solutions AnalysisMateriel Solutions AnalysisTechnology Maturation and Risk ReductionEngineering and Manufacturing DevelopmentProduction and DevelopmentOperations and Support-Strategic goal alignment-Pre-investmentPre-Investment Pre-investment Implementation Implementation Post-implementationAligning Methodologies with Acquisition LifecycleMateriel Solutions AnalysisTechnology Maturation and Risk ReductionEngineering and Manufacturing DevelopmentProduction and Development Operations and SupportBSC IRM EVM EVM KVAIRM KVA IRM IRM L6σKVA L6σ KVA KVAL6σCombined Map to Show How the Five Methodologies Might Be Used in the 5000 Series PhasesBenefits and Challenges of the Five MethodologiesConclusions• Use the approach that fits with the selection criteria and point in the acquisition lifecycle (constrained by time and cost)• Use continuous production economics for mature, simple products• Use job shop, lot size of 1, economics for complex products that can be “intelligensized”• Need new common unit of value to track upside of value of intelligensizing military products to stay ahead in the Great Power Competition race• No current methodology in use provides adequate program value to risk-based forecastingBack-UP for Reference Only SlidesMSA TMRR EMD P&D O&SAnalysis of Alternatives Preliminary Design Review Complete detailed design Low rate initial productionLifecycle Sustainment Plan (LCSP)Initial funding estimates Capability Development DocumentSystem-level Critical Design Review (CDR)Initial Operational Test & Evaluation (IOT&E) System ModificationsTechnology Development Strategy Competitive prototypingEstablish project baseline with Performance Measurement Baseline (PMB)Full rate production decision SustainmentAcquisition Program Baseline (APB) establishedInitial and Full Operational Capability (IOC and FOC) DisposalLifecycle Key events and MilestonesFigure 1. The 5000 Series Acquisition Lifecycle Taken from Defense Acquisition Guidebook, 2017Balanced Scorecard (BSC)Balanced Scorecard Strategy Map• Is a strategic planning and management system that organizations use to:• Communicate what they are trying to accomplish• Align the day-to-day work that everyone is doing with strategy• Prioritize projects, products, and services• Measure and monitor progress towards strategic targets).http://www.balancedscorecard.org/BSC-Basics/About-the-Balanced-ScorecardLean Six SigmaLean Six Sigma• What is Lean Six Sigma?• Combination of 2 powerful processes improvement methods• Lean• Six Sigma• Designed to improves process performance:• Cost, waste, defects reduction• Cycle time reduction17https://goleansixsigma.com/what-is-lean-six-sigma/Combination of Lean and Six Sigmaprocess.https://www.greycampus.com/blog/quality-management/a-brief-introduction-to-lean-and-six-sigma-and-lean-six-sigmaSix Sigma: DMAIC19https://www.greycampus.com/blog/quality-management/a-brief-introduction-to-lean-and-six-sigma-and-lean-six-sigmaEarned Value Management (EVM)Earned Value Management (EVM) is a management tool intended to bring visibility to the planning for and execution of the technical, cost, and schedule aspects of acquisition programs to support decisions by program managers. EVM DefinitionEarned Value Management (EVM) is a project management tool that integrates the technical scope of work with schedule and cost elements for investment planning and control. It compares the amount of work accomplished in a given period with the amount of the work expected in that period. Differences in expectations are measured in both cost and schedule variancesEVM – Why & Whathttps://image.slidesharecdn.com/earnedvaluemanagement-090729045749-phpapp02/95/earned-value-management-7-638.jpg?cb=1422668891https://image.slidesharecdn.com/earnedvaluemanagement-090729045749-phpapp02/95/earned-value-management-7-638.jpg?cb=1422668891Knowledge Value Added (KVA)What is Knowledge Value-Added?• Knowledge Value Added (KVA) is a methodology for estimating the return on investment (ROI) between As-Is and To-Be process improvement • E.g., technology, automation improvement options• E.g., service process redesigns without technology, automation options• Provides productivity performance ratios (e.g., ROI) for core processes and their subprocesses• Provides productivity measure in terms of Return on knowledge (ROK) that is directly proportionate to ROI• Provides cost per common unit estimatesFundamental assumptions of KVA• Underlying Model: Change, Knowledge, and Value are ProportionateFundamental assumptions:1. If X = Y (P(X) = Y) no value has been added.2. “value” ∝ “change”3. “change” can be measured by the amount of knowledge required to make the change.So “value”∝ “change” ∝“amount of knowledge required to make the change”(Principle of replication)X P YInput OutputProcessKnowledge Value Added Steps• Step 1: Identify Core process and its subprocesses• E.g., Evidence Retrieval and Destruction Process• E.g., Access Evidence• Step 2: Identify subprocess outputs• E.g., Access Evidence  Item location• Step 3: Convert outputs into common units• E.g., Item location  Time to learn how to locate item (i.e., Learning Time – LT) • Step 4 (Using Spreadsheet Template): Estimate cost to produce the outputs• E.g., Cost to locate item (i.e., Work Time: WT)• Market Comparables = cost * 1.5• Step 5 (Using Spreadsheet Template): Generate ROK and/or ROI productivity ratios• E.g., ROK, ROI on Item location subprocessStrategy Mapping BSChttp://www.balancedscorecard.org/BSC-Basics/About-the-Balanced-S dBSC Development https://goleansixsigma.com/what-is-lean-six-sigma/Value Added Analysis31Value-Added: An activity is value-added if a customer is willing to pay for; it changes form, fit or function of a product or service; it converts input to output; it is not waste.Non-value Added (NVA): These activities are unnecessary: they provide no value for internal or external customers, and can be immediately eliminated. (sometimes called Type II NVA) Business Value Added (BVA): These activities provide no value to customers, but are necessary given current process limitations. Common examples are inspections, management approvals, most quality assurance activities; technical support activities. (sometimes called Type I NVA)Value Stream Analysis Generic Examplehttps://goleansixsigma.com/value-stream-mapping/Schedule Metricshttps://image.slidesharecdn.com/earnedvaluemanagement-090729045749-phpapp02/95/earned-value-management-7-638.jpg?cb=14226688915 Principles of Leanhttps://blogs.mtu.edu/improvement/2014/01/08/back-to-the-basics-5-principles-of-lean/Cost Metrics https://image.slidesharecdn.com/earnedvaluemanagement-090729045749-phpapp02/95/earned-value-management-7-638.jpg?cb=1422668891https://image.slidesharecdn.com/earnedvaluemanagement-090729045749-phpapp02/95/earned-value-management-7-638.jpg?cb=1422668891Are We Safer since 9/11/01?• Spent $1Tril since 9/11• “We’ve spent hundreds of billions since 9/11. The question is how much of that was wasted ….” (GAO Senior Auditor: The Atlantic, Sept 2016, p. 64)• FirstNet = Cost from $12 Bil to $47 Bil (GAO estimate) and may be 10-15 yrs in the future• BioWatch cost $200 mil. + $1 bil in maintenance fees (Canceled in 2014: it was a “dud”,  with no replacement of a better bio detection system, p. 65)• “If you’re shocked and scared and you know there’s a threat out there, you’ll do anything, spend anything, to deal with it…even if what you spend it on hasn’t been tested and you haven’t even set any standards to evaluate it.” (GAO Senior Auditor, p. 66)• Since 9/11/01: $100-$150Bil wasted on failed DHS projects, equipment (pg. 68)Part Retrieval and DistributionSteps 1 and 2: Identify Core Process, Subprocesses, and Subprocess Outputs DiagramPart LocationAccess Part DBRetrieve Part from InventoryPart RetrievalGive to requesterDistribute to Requestersub processsub processsub processoutput output outputPart Retrieval and Distribution Step 3: Convert Outputs Into Common UnitsNote: Convert subprocess outputs into common units using learning timeProcess DescriptionNumber of Employees Performing the SubprocessAutomation Cost (AC) =  Total Cost of Ownership per year / Number Subprocess Executions per year  Learning Time  (LT) in HrsAverage Hourly Labor Cost per Subprocess (AHCS)Percent (%) Subprocess Automated (PA)Times Performed In a Year (TPY)Average Time to Complete (ATC) hrsAccess Evidence Information 1  $                         5.00 35  $            15.00 50% 690 0.5Retrieve Evidence 1  $                       15.00 15  $            35.00 10% 690 0.33Sign out Evidence 1  $                       10.00 50  $            80.00 30% 690 0.5Part LocationAccess Part DBRetrieve Part from InventoryPart RetrievalGive to requesterDistribute to Requestersub processsub processsub processoutput output outputPart Retrieval and Distribution Step 4: Estimate cost to produce the outputsProcess DescriptionNumber of Employees Performing the SubprocessAutomation Cost (AC) =  Total Cost of Ownership per year / Number Subprocess Executions per year  Learning Time  (LT) in HrsAverage Hourly Labor Cost per Subprocess (AHCS)Percent (%) Subprocess Automated (PA)Times Performed In a Year (TPY)Average Time to Complete (ATC) hrsAccess Evidence Information 1  $                       5.00 35  $         15.00 50% 690 0.5Retrieve Evidence 1  $                    15.00 15  $         35.00 10% 690 0.33Sign out Evidence 1  $                    10.00 50  $         80.00 30% 690 0.5Part LocationAccess Part DBRetrieve Part from InventoryPart RetrievalGive to requesterDistribute to Requestersub processsub processsub processoutput output outputStep 5: Generate ROK and/or ROI productivity ratiosPart Retrieval and Distribution Step 5: Generate ROK and/or ROI productivity ratiosProcess DescriptionSingle Execution of a Subprocess Learning Time (SESLT)  = LT + (LT * PA)Total Learning Time Units (TLTU) per year =SESLT * TPYCost per year  =  AHCS*TPY*ATCSurrogate  Rev/Mrk Comp per year = Labor and Automation Cost * 1.5Denominator =  Cost per year  ROI and ROK Numerator = TLTU * Rev/unitROK = Numerator / DenominatorROI= (Numerator - Denominator) / DenominatorAccess Evidence Information 67                         45,885  $        6,900.00  $  10,350.00  $     6,900.00  $       27,711.4 402% 302%Retrieve Evidence 17                         11,385  $      11,500.00  $  17,250.00  $   11,500.00  $       6,875.77 60% -40%Sign out Evidence 95                         65,550  $      31,050.00  $  46,575.00  $   31,050.00  $     39,587.78 127% 27%Totals 178                                122,820  $         49,450.00  $    74,175.00  $     49,450.00  $       74,175.00 150% 50% CostValue ValueValue Value/CostMonetizationPart LocationAccess Part DBRetrieve Part from InventoryPart RetrievalGive to requesterDistribute to Requestersub processsub processsub processoutput output outputWhat is Return on Knowledge (ROK)? Return on Knowledge is a new organizational performance ratio  Numerator = amount of K(Value) required to reproduce process outputs How is this calculated? What does the resulting number represent? Denominator = cost to use K to produce output How is this calculated? What does the resulting number represent? ROK = Output/Input =  K Units/Input = Revenue/InputContinuous, Job Shop• DOD Acquisition economics: Job Shop, Continuous Production?• Continuous production: • Takes advantage of production learning curve• Lot size – very large• Commodity valuation• Job Shop:• Lot size of 1• Customized outputs• Higher premium on value• How do we use the four approaches with each of these models?

Evaluating Investments in Technology: Five Approaches

https://calhoun.nps.edu/bitstream/handle/10945/68219/SYM-AM-21-173.pdf?sequence=1

Evaluating Investments in Technology: Five Approaches

Abstract

Similar works

Full text

Available Versions

Calhoun, Institutional Archive of the Naval Postgraduate School