Proceedings Paper (for Acquisition Research Program)This paper describes applying ROI analysis principles for SOA performance management, creating Service-level Agreements (SLAs) to articulate agreements between the Government and external service providers, and managing SLAs through a governance framework (Hanf & Buck, 2009, March). This white paper highlights key findings of research undertaken by The MITRE Corporation (MITRE) and the resulting recommendations for (1) applying Return-on-Investment (ROI) analysis principles as the foundation for more effective performance management of Government Service-oriented Architecture (SOA), (2) creating comprehensive Service-level Agreements (SLAs) to articulate agreements between the Government and external service providers, and (3) managing SLAs through a governance framework (Oakley-Bogdewic & Buck, 2009; Hanf & Buck, 2009, March 25).  As illustrated in Figure 1, MITRE''s recommendations address the additional managerial complexity and uncertainty that SOA objectives and proposed solutions often create.Naval Postgraduate School Acquisition Research ProgramApproved for public release; distribution is unlimited

Buck, Kevin S.

Hanf, Diane P. M.

Symposium Presentation (for Acquisition Research Program)Approved for public release; distribution is unlimited.Naval Postgraduate School Acquisition Research Progra

Buck, Kevin

Hanf, Diane

Calhoun, Institutional Archive of the Naval Postgraduate School

Calhoun: The NPS Institutional ArchiveReports and Technical Reports All Technical Reports Collection2009-05-01Outcome-driven Service ProviderPerformance under Conditions ofComplexity and UncertaintyKevin Buckhttp://hdl.handle.net/10945/33430© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)Outcome‐Driven Service Provider Performance under Conditions of Complexity and UncertaintyKevin S. Buck, E523 (kbuck@mitre.org), 443‐636‐5380Diane Hanf, E060 (dhanf@mitre.org) , 339‐223‐5380SLA6th Annual Acquisition Research SymposiumThe Graduate School of Business & Public Policy, Naval Postgraduate SchoolMay 13‐14, 2009 (Monterey, California)Stakeholder-Driven Performance Management Mission-Oriented Investigation and Experimentation (MOIE)© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)2Agendan Purpose and Contextn Key Discussion Topics:n Conclusionsn Questions?SOA Performance ManagementSLAs for SOA  Performance ManagementSOA SLA GovernanceFederal Government Performance ManagementTo Address Complexity and Uncertainty© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)3Purpose and Context Purpose– Describe application of Return‐on‐Investment (ROI) analysis for SOA performance management– Discuss use of SLAs to articulate agreements between the Government and external service providers in SOA environments– Recommend an SLA Management governance framework Context Summarizes results‐to‐date of MITRE Corporation (MITRE) research efforts:– Stakeholder‐Driven Performance Managementn K. Buck, L. Oakley‐Bogdewic– SOA Performance Measures Expression in Performance‐Based Acquisition (PBA) Vehiclesn D. Hanf, K. Buck© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)4Performance Management – Federal Focus Objectives must be outcome‐driven– Complying with performance reporting is insufficient– Derive outcomes from stakeholder‐articulated statements Mission, vision, and strategic/transition Use a shared responsibility model– Program/portfolio/contract managers all have vested interest Federal government has a lackluster performance management track record– Clearest evidence: failed programs (failure to deliver or significant over‐budget/over‐schedule delivery) Performance management is challenging for the federal government– Challenges became greater with dependencies on non‐controlled resources, e.g., networks– Service‐Oriented Architecture (SOA) significantly increases dependencies Uncertainty and/or complexity increase Data and functions now added to the non‐controlled resource poolSOA Performance ManagementSLAs for SOA  Performance ManagementSOA SLA GovernanceFederal Government Performance ManagementTo Address Complexity and Uncertainty© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)5Performance Management – Federal Focus Key hypothesesComplexity Uncertainty  about the future# of moving parts and relationshipsFocus more on interdependencies  Chances of reducing uncertainty over timePlan for contingencies and garner flexibility% of DoD IT spending contracted outIf Then So,Contract performance impact on IT performanceManage IT delivery as a tight linkage between contract and overall performance= 79.3% = significant[1]Paul A. Strassmann, Budgeting DoD Information Technologies, 11September2008[1]© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)6SOA Performance Management SOA definition– An architectural approach used to build solutions– Consists of a set of services, service consumers, service producers and mechanisms for discovery and establishing contracts SOA helps organizations share resources, and it increases:– Acquisition complexity: multiple relationships to manage– Uncertainty regarding when resource needs change SOA performance management challenges:SOAP = Simple Object Access ProtocolUDDI = Universal Description, Discovery, and IntegrationWSDL = Web Services Description LanguageSOA Construct– Expecting savings without analysis– Understanding the SOA lifecycle– Measuring non‐fiscal returns– Managing expectations– Effectively managing SOA is resource‐intensive– Integrating multiple perspectives– Establishing stakeholder targetsSOA Performance ManagementSLAs for SOA  Performance ManagementSOA SLA GovernanceFederal Government Performance ManagementTo Address Complexity and UncertaintyMany more with Differing needs© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)7SOA Performance ManagementWithout credible and relevant benchmarks for what performance can be expected from your SOA participation, how will you assess SOA performance?MissionCurrent Capability Desired CapabilityPerformance GapROI-based investments selectionPerformance management planning(to include defining ROI based metrics)Performance Monitoring(as an on-going comparison of initially expected versus actual ROI)Post-Investment ReviewROI Analysis SOA investmentoption- List of priorities- List of relative desirability- Comparative customer satisfaction ratings- Balanced Scorecard ratings- Number of votes "for" and "against"(1)  What are the social consequences?(2)  What are the strategic implications?(3) What is the effect on employee morale?(4)  What are the political ramifications?(5)  Stoplight matrix of risk assessmentInvestment costs and benefits can be quantified, but not accurately monetized?(e.g., customer satisfaction)Investment costs and benefits can be monetized, but not easily?(e.g., productivity)Investment impacts cannot be accurately expressed monetarily?(e.g., degree of regulatory compliance)NoNoNoInvestment costs and benefits can be easily monetized?(e.g., salvage value)ProjectionsUncertainty AssessmentROI CalculationSuite of MetricsInternal Rate of Return (IRR)Net Present Value (NPV)Other Relevant Metrics Payback PeriodCost/Benefit RatioCost, Economic,and/or FinancialAnalysisYesYesYesYes QualitativeAssessmentFlexibilityCost SavingsTime SavingsManagement EfficienciesAgilityOpportunity to InnovateROI calculation =what we learned in finance classROI analysis = ROI calculation + assessment of intangiblesSOA Participant© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)8Applying SLAs to Manage SOA Service Provisioning SLAs are a means to establish performance‐related agreements between service providers and consumers An SLA is a formal negotiated agreement between two parties– A “contract” between customers and their service providers– Records the common understanding about service features Government experiences in applying SLAs:– Results have been mixed, often depending on whether the SLAs were:n Consistently applied, maintained and updatedn Managed with awareness of interdependent SLAs’ performance needsn Computer transaction‐based vice an accurate vehicle for consumer‐provider communication– “Losing the forest for the trees”n SLA administration focus vice continuous assessment as to whether desired outcomes are being achievedSOA Performance ManagementSLAs for SOA  Performance ManagementSOA SLA GovernanceFederal Government Performance ManagementTo Address Complexity and Uncertainty© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)9 Transactional versus Relational SLAs– Transactional Measures moment‐in‐time performance indicators                       (e.g., throughput), typically in an automated fashion– Relational Why performance levels are needed Parties’ agreement about performance standards  Rationale for thresholds Recourse for performance lapsesApplying SLAs to Manage SOA Service Provisioning© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)10Applying SLAs to Manage SOA Service Provisioning SLAs are most effective if stated in measurable terms and include– Stated expectations of outcomes– Derived Key Performance Indicators (KPIs) with outcome‐supported service levels– Defined measurement mechanisms and what determines “success”– Parties involved and their responsibilities– Reporting guidelines and requirements– Service provider incentives for meeting agreed‐upon levels of qualityMeasurableRelevantAchievableControllable Checklist for determining if SLAs are effectively supporting objectives:– SLAs articulate a definitive performance provider‐consumer agreement – SLAs allow for re‐negotiation if future conditions significantly change– SLAs clearly align with overall enterprise performance objectives Align decision frameworks, documents, and contract artifacts Involves diligent configuration control and communication© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)11 Key reasons for SLA failure:– Lack of well-defined requirements at the time of RFP issuance– When performance interdependencies exist, each party must have solid data on its own performance to counter challenges  Key SLA lessons learned [1] :– Agree to pre-existing service levelsn Some Government agencies agree that the required service levels will be set at pre-existing performance levels. By doing so, they preserve the current service that the new contract was designed to improve– Agree to service levels before contract awardn There is little incentive to uphold post-contract award agreements– Do not agree to fix service levels at initial provider performancen This reduces flexibility and incentive to get better performance if future needs are not known– State incentives appropriately n Ensure incentives reinforce positive behavior expectations– Don’t ask for the moonn Performance should be critically examined and consider cost drivers[1] “The Outsourcing Revolution, 2004: Protecting Critical Business Functions”, J. DelaneyApplying SLAs to Manage SOA Service Provisioning© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)12More key lessons learned– Less is more—make SLAs simple and familiar– Make SLAs measurable and actionablen Collect data to be acted onn Predetermine actions to be undertaken when metrics fall short– Detail the unusual areas and boiler plate the restn “Must haves” should be articulated in the contract itself– Describe methods for withholding/reducing feen Loss of business/productivity is rarely compensated directly- Typically, a rebate proportional to the shortfall is usedn SLAs typically include escalation procedures and conditions under which they are invoked– Incorporate contract language that allows SLAs to be changedn This language should tie to milestones as SLA changes may impactcost/scheduleApplying SLAs to Manage SOA Service Provisioning© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)13SOA SLA Governance SLA governance is the ongoing process of reviewing performance measures against stated goals and targets and reassessing SLA value The ultimate objectives of an SLA governance framework are to ensure:– Performance standards are established to meet overall contract goals– SLAs continue to describe performance deemed critical to overall outcomes– SLAs and performance measures are prioritized– All activities and surveillance are undertaken as effectively as possibleWhat we are trying to avoid with an SLA Governance Framework?Per the Government Accountability Office (GAO) in reviewing Navy-Marine Corps Intranet  (NMCI) performance:  “The Navy defined strategic goals for its NMCI program and developed a plan for measuring and reporting on achievement of these goals. However, the Navy did not implement this plan, choosing instead to focus on defining and measuring contractually specified SLAs.  Program officials did not have performance data to demonstrate progress in relation to strategic goals.  Without effective performance management, the Navy is increasing the risk that the program will continue to fall short of its goals and expected results.” GAO-07-51, December 2006, pp. 18-19.© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)14SOA SLA Governance SOA SLA governance goals and success driversGoalsPurposeProper construction, follow best practicesMap to outcomesIdentify interdependenciesFlexibility negotiatedObjectives re-visitedOutcomes re-validatedPriorities mappedRelative value identifiedIncentives match prioritiesRoles/responsibilities established and clearMonitors appropriately trainedMulti-disciplinary reviews/ committeesSuccessdrivers© 2009 The MITRE Corporation. All rights reservedApproved for Public Release (Case # 09-1742)15SOA SLA GovernanceGAO report [1] on NMCI effort strongly suggests that downside risk associated with SLAs is “getting lost in the trees for the forest”– Suggested SLA Governance Framework to ensure SLAs are effectively managed and align with expected outcomes:[1] GAO Report GAO-07-51.  “DOD Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting Goals and Satisfying Customers”PREPARE CREATE MONITORSLA Governance FrameworkUPDATEReview CurrentBusiness PracticesEstablish Goalsof EffortDefine how SLAfits in with goalsID anyInterdependencieswith other SLAsIdentify Relevant &Measurable KPI’sSet key roles& assignresponsibilitiesEstablish ReportingGuidelines Necessary updatesidentified and madeAll metricscontinually measured& monitoredKey roles are beingheld accountable forresponsibilitiesOverall effort ismoving forward:•On schedule•Benefits beingrealized•Collaboration byall partiesIdentify areas in SLA which requireupdates or changes 

Outcome-driven Service Provider Performance under Conditions of Complexity and Uncertainty

Proceedings Paper (for Acquisition Research Program)This paper describes applying ROI analysis principles for SOA performance management, creating Service-level Agreements (SLAs) to articulate agreements between the Government and external service providers, and managing SLAs through a governance framework (Hanf & Buck, 2009, March). This white paper highlights key findings of research undertaken by The MITRE Corporation (MITRE) and the resulting recommendations for (1) applying Return-on-Investment (ROI) analysis principles as the foundation for more effective performance management of Government Service-oriented Architecture (SOA), (2) creating comprehensive Service-level Agreements (SLAs) to articulate agreements between the Government and external service providers, and (3) managing SLAs through a governance framework (Oakley-Bogdewic & Buck, 2009; Hanf & Buck, 2009, March 25).  As illustrated in Figure 1, MITRE''s recommendations address the additional managerial complexity and uncertainty that SOA objectives and proposed solutions often create.Approved for public release; distribution is unlimited.Naval Postgraduate School Acquisition Research Progra

http://calhoun.nps.edu/bitstream/handle/10945/33430/NPS-AM-09-090.pdf?sequence=3&isAllowed=y

Outcome-driven Service Provider Performance under Conditions of Complexity and Uncertainty

Abstract

Similar works

Full text

Available Versions

Calhoun, Institutional Archive of the Naval Postgraduate School

Calhoun, Institutional Archive of the Naval Postgraduate School