Evaluation Theory and Practice

One of the topics I was determined to discuss with my European colleagues at the Helsinki Conference was the use of theory in evaluation practice. I was thrilled to stumble upon several thought provoking discussions regarding the use of theory in evaluation. A common theme was the belief that teaching evaluation practitioners about theory was critical to a better future for the evaluation discipline. But why is theory so important? The session I addressed on this topic also involved Frans Leeuw, Evert Vedung and Gary Henry. I emerged from the session with some new (and old) insights about evaluation theory and evaluation practice. This brief article summarizes what I learnt – and what I said

Eliciting Preferences for Resource Allocation in Health Care

Willingness-to-pay (WTP) studies are increasingly being used in the evaluation of health care programmes and, although less frequently, for priority setting in health care. The usefulness of willingness-to-pay as a discriminatory tool for priority setting is considered in this paper for three different health care programmes in Ireland: cancer, cardiovascular and community care. While the resulting estimates are consistent with respondents’ rankings of the programmes, there is no statistical difference among the three programmes in terms of WTP. In considering marginal changes to existing health care programmes people consider their rankings of the programmes and the existing capacity of each programme. People are also more concerned with the gains to themselves of expanding various health care programmes than with wider issues of access or fairness.

Developing Capacity for International Developmental Evaluation

At the IDEAS Global Assembly this past May in the beautiful island of Barbados we engaged in discussions about improving international development evaluation capacity. With colleagues Tarek Azzam, Matthew Galen and John LaVelle, we discussed multiple initiatives ranging for the use of webinars, knowledge management systems, e-learning applications, massive open online courses, opportunities to use social media to enhance national and international conferences, and a range of university-based capacity development efforts. In this brief article we describe some of those initiatives

Remote Ergonomic Research in Space: Spacelab Findings and a Proposal

This paper discusses ergonomics research using remotely situated video camerasin spacecraft. Two prototype studies of crewmembers working in the micro-G environments aboard the first two flights of Spacelab are described. Various aspects of crew restraint, stabilization, manipulation of controls, and mobilization were observed, operationally defined, and quantified by observing videotaped scenes of Spacelab crewmembers. In the first study, four performance behaviors were quantified to provide estimates of their frequency of occurrence and variation over the course of each of the flights. The behaviors and their mean percent of observed times were: Hand-Hold 32.2%, Foot Restraint 35.3%, Translation 9.4%, and Struggle 3.7%. Because we observed that nearly a third of a crewmember\u27s time was spent inefficiently holding on With one hand while trying to work with the other, a second study was conducted exploring the use of foot restraints and hand stabilization. During 18 episodes of single-foot restraint, for example, there were 52 instances of hand stabilization and 135 instances of stabilization attempts with the other foot. The paper concludes with some defining charactenstics of adequate foot restraints, and a proposal for extending this research model to future spacecraft studies

The 2004 Claremont Debate: Lipsey vs. Scriven

While there is little disagreement about the need for, and value of, program evaluation, there remain major disagreements in the field about best practices (Donaldson & Lipsey, in press). For example, Donaldson and Scriven (2003) invited a diverse group of evaluators to Claremont in 2001 to share their visions for “how we should practice evaluation” in the new millennium. Theorists and practitioners discussed a wide range of views and evaluation approaches, many at odds with one another, on how best to improve evaluation practice (e.g., the experimental paradigm, evaluation as a transdiscipline, results-oriented management, empowerment evaluation, fourth generation evaluation, inclusive evaluation, theory-driven evaluation and the like). In response to some of the heated exchanges, Mark (2003) noted “it seems ironic when evaluators who espouse inclusion, empowerment, and participation would like to exclude, disempower, and see no participation by evaluators who hold different views.” Hefurther concluded that whatever peace has been achieved in the so-call quantitative-qualitative paradigm wars remains an uneasy peace

The international landscape of positive psychology research: A systematic review

Since positive psychology originated in 1998 as an organized stream of inquiry in theUnited States, it has inspired new theory and research on human flourishing across the world. The current systematic review presents an overview of (a) the prevalence of scientific research in positive psychology across five continents and 63 countries, (b) the characteristics of the research, including methodology and topics, and (c) the influence of positive psychology in expanding established lines of research in new ways. Through an analysis of 863 peer-reviewed positive psychology articles, this review attempts to map the international landscape of positive psychology research. Further, it responds to relevant critiques of the field, confirming some and dispelling others. Finally, recommendations are shared for future directions to build a more culturally responsive field of positive psychology that is committed to the advancement of flourishing and wellbeing in the global context

Scaling the heights of positive psychology: A systematic review of measurement scales

The volume of empirical research on positive psychology topics has grown substantially over the past two decades.  This review examines how constructs in positive psychology have been operationalized, measured, validated, cited, and applied to build the science. Based on an archive of 972 empirical articles linked to positive psychology, this review found that 762 articles used at least one measurement scale; 312 measures were created or adapted.  Findings reveal a wide range of scales being used to measure a variety of constructs, including scales on both life-enhancing and life-depleting constructs.  Key characteristics such as journals, constructs, and scale development and validation information are discussed.  There are some reliability analyses and validations occurring within the field, but the creation of new measures far outpaces the validation of existing measures.  Weaknesses such as multiple operationalizations may be rooted in inadequate discourse and synthesis.  We call for further cross-pollination for a more scientifically robust scholarship in positive psychology

The 2009 Claremont Debates: the Promise and Pitfalls of Utilization-focused and Empowerment Evaluation

Background: Hundreds of evaluators visit the Claremont Colleges in southern California each year to discuss a wide range of topics related to improving the quality of evaluation practice. Debates between thought leaders in the field have been one of the most popular and informative ways to advance understanding about how best to practice evaluation in contemporary times. Purpose: The purpose of this article is to provide a written transcript of the 2009 Claremont Evaluation Debates. The first debate is between Michael Quinn Patton and Michael Scriven on the promise and pitfalls of utilization-focused evaluation. The second debate is between David Fetterman, Michael Quinn Patton, and Michael Scriven on the promise and pitfalls of empowerment evaluation. Setting: The debates occurred at the Claremont Graduate University on August 23-24, 2009. Several hundred evaluators from around the world also viewed and participated in the debates via a live webcast

Does cancer type influence the impact of recurrence? : a review of the experience of patients with breast or prostate cancer recurrence

Objective: Patients will experience a plethora of issues when faced with a recurrence of their cancer. It is unclear if cancer type is a significant factor in how recurrence is experienced by an individual. The aim of the current review is to explore the evidence base and summarise the experiences of patients specifically with a recurrence of breast or prostate cancer (the most common for women and men, respectively) and then provide a comparison of these experiences. These experiences include the physical, psychological and psychosocial issues that arise at this time.  Methods: A systematic search was conducted of studies published between January 1994 and April 2019. Due to the mix of research designs used previously in the literature, this review was conducted in an integrative manner; allowing for inclusion of diverse research designs. Results were synthesised narratively, with data categorised according to physical, psychological, and psychosocial indices of quality of life. The review protocol was registered in the international database of prospective systematic reviews in health and social care- (CRD42019137381).  Results: Fifteen breast cancer and six prostate cancer articles were identified, each reporting one relevant study. Patients reported several negative issues at the time of a breast or prostate cancer recurrence. Similarities were found between cancer types, with physical problems such as fatigue, psychological issues including anxiety and depressive symptoms, and psychosocial concerns such as issues with healthcare professionals common in both cancers. Certain findings were inconsistent across studies, with some experiences differing between studies rather than due to cancer type.  Conclusions: Differences in the experience of recurrent cancer appear to be more heavily influenced by individual factors, rather than cancer type. Findings are confounded by gender; and should be considered preliminary. Effects of recurrence should be studied in samples where cancer type and gender are not confounded. Concerns are raised about available study quality and differing outcome measures in this interpretation. Care and support of the individual at the time of a cancer recurrence is a key focus. Future research suggestions with implications for clinical practise are included.Publisher PDFPeer reviewe