Measurement issues in the evaluation of projects in a project portfolio

A common problem in multicriteria decision analysis is how to value projects in a project portfolio. Recently, there has been some attention given to a practice whereby analysts define a value function on some criterion by setting 0 as the value of the worst project. In particular, Clemen and Smith have argued this practice is not sound as it gives different results from the case where projects are "priced out", and makes a strong implicit assumption about the value of not doing a project. Moreover, it can be shown easily that the optimal choice set is not invariant with respect to rescalings such as those induced by the addition of a new item to the choice set, hence rank reversal can occur when this procedure is used. We provide a measurement theoretic account of the phenomenon and show that whether the admissible transformations of the underlying scale are similarity or affine (or other) transformations depends on the precise details of the value model used. We also provide an axiomatic impossibility result which illustrates an incompatibility between an idea that inaction has a value of zero and the admissibility of constant scale translations. We use our analysis to comment on the view of Clemen and Smith that the baseline problem is best dealt with by assigning project-specific scores for not doing particular projects

A Project Portfolio Management model adapted to non-profit organizations

As they strive towards greater professionalism in carrying out their activities, non-profit organizations (NPOs) have begun paying attention to project management. The non-profit sector (NPS) has also begun to adopt strategic planning techniques, thus making the acceptance of project portfolio management (PPM) methodology a natural consequence. This article aims to propose a project portfolio management model adapted to the context of NPOs

A Funder Journey: Scaling Up the Stone Family Foundation

This report follows the journey of one foundation from its inception to the present day. It highlights the work of an effective funder, and shares lessons that can be applied to funding more widely. The Stone Family Foundation is an example of how a relatively new and mid-sized foundation has looked to scale up and develop how it works and what it funds. The lessons learned from implementing the major grants program influenced the shape of its subsequent WASH funding. While this exact journey may be specific to the Stone Family Foundation, the general process -- starting out, planning, delivering, learning and then adapting -- will be relevant to other funders grappling with the same questions about how to become an effective funder

Lessons Learned and Next Steps in Energy Efficiency Measurement and Attribution: Energy Savings, Net to Gross, Non-Energy Benefits, and Persistence of Energy Efficiency Behavior

This white paper examines four topics addressing evaluation, measurement, and attribution of direct and indirect effects to energy efficiency and behavioral programs: Estimates of program savings (gross); Net savings derivation through free ridership / net to gross analyses; Indirect non-energy benefits / impacts (e.g., comfort, convenience, emissions, jobs); and, Persistence of savings

Benefits realisation management: panacea or false dawn?

Benefits Realisation Management (BRM) is becoming an increasingly important aspect of project and programme management. However, commentators have observed that the practice of BRM is often flawed, and have made suggestions as to how practice might be improved. This paper is concerned with the reasons why the implementation of BRM might not be straightforward, by focusing on the underlying assumptions. It will approach the issue by drawing on the author's experience from the 1990s and 2000s in working in the management of government-funded regeneration programmes in the UK. In this field there was a rigid benefits management framework, although it precedes the development of BRM. The paper will argue that there are important underlying conceptual issues in benefits management which have practical implications and need to be recognised in the development of theory for BRM

Methodological Challenges in Impact Evaluation: The Case of the Global Environment Facility (GEF)

In this paper, we explore some of the methodological challenges that evaluators face in assessing the impacts of complex intervention strategies. We illustrate these challenges, using the specific example of an impact evaluation of one of the six focal areas of the Global Environment Facility; its biodiversity program. The paper discusses how theory-based evaluation can provide a basis for meeting some of the challenges presented.

Software component testing : a standard and the effectiveness of techniques

This portfolio comprises two projects linked by the theme of software component testing, which is also often referred to as module or unit testing. One project covers its standardisation, while the other considers the analysis and evaluation of the application of selected testing techniques to an existing avionics system. The evaluation is based on empirical data obtained from fault reports relating to the avionics system. The standardisation project is based on the development of the BC BSI Software Component Testing Standard and the BCS/BSI Glossary of terms used in software testing, which are both included in the portfolio. The papers included for this project consider both those issues concerned with the adopted development process and the resolution of technical matters concerning the definition of the testing techniques and their associated measures. The test effectiveness project documents a retrospective analysis of an operational avionics system to determine the relative effectiveness of several software component testing techniques. The methodology differs from that used in other test effectiveness experiments in that it considers every possible set of inputs that are required to satisfy a testing technique rather than arbitrarily chosen values from within this set. The three papers present the experimental methodology used, intermediate results from a failure analysis of the studied system, and the test effectiveness results for ten testing techniques, definitions for which were taken from the BCS BSI Software Component Testing Standard. The creation of the two standards has filled a gap in both the national and international software testing standards arenas. Their production required an in-depth knowledge of software component testing techniques, the identification and use of a development process, and the negotiation of the standardisation process at a national level. The knowledge gained during this process has been disseminated by the author in the papers included as part of this portfolio. The investigation of test effectiveness has introduced a new methodology for determining the test effectiveness of software component testing techniques by means of a retrospective analysis and so provided a new set of data that can be added to the body of empirical data on software component testing effectiveness

Department of Homeland Security Science and Technology Directorate: Developing Technology to Protect America

In response to a congressional mandate and in consultation with Department of Homeland Security's (DHS) Science and Technology Directorate (S&T), the National Academy conducted a review of S&T's effectiveness and efficiency in addressing homeland security needs. This review included a particular focus that identified any unnecessary duplication of effort, and opportunity costs arising from an emphasis on homeland security-related research. Under the direction of the National Academy Panel, the study team reviewed a wide variety of documents related to S&T and homeland security-related research in general. The team also conducted interviews with more than 200 individuals, including S&T officials and staff, officials from other DHS component agencies, other federal agencies engaged in homeland security-related research, and experts from outside government in science policy, homeland security-related research and other scientific fields.Key FindingsThe results of this effort indicated that S&T faces a significant challenge in marshaling the resources of multiple federal agencies to work together to develop a homeland security-related strategic plan for all agencies. Yet the importance of this role should not be underestimated. The very process of working across agencies to develop and align the federal homeland security research enterprise around a forward-focused plan is critical to ensuring that future efforts support a common vision and goals, and that the metrics by which to measure national progress, and make changes as needed, are in place

Catalog of Approaches to Impact Measurement: Assessing Social Impact in Private Ventures

To inform action impact investors could take to measure impact in a coordinated manner, The Rockefeller Foundation commissioned the study of impact assessment approaches presented here.It is natural to hope to find a single, turnkey solution that can address all measurement needs. In this study we conducted a survey of impact investors and complemented it with seven years of experience in the field of impact investing to discover what these investors want from impact measurement, and conducted in-depth interviews with over twenty entities that have developed and implemented approaches to measuring impact. Our survey of existing approaches was thorough but surely is not comprehensive; however the approaches are a good representation of the current state of play. What we found is that there is not one single measurement answer. Instead the answer depends on what solution is most appropriate for a particular investor's "impact profile" defined as the investor's level of risk tolerance and desired financial return, the particular sector in which the investor operates, geography, and credibility level of information about impact that the investor requires