Evaluation Theory and Practice

One of the topics I was determined to discuss with my European colleagues at the Helsinki Conference was the use of theory in evaluation practice. I was thrilled to stumble upon several thought provoking discussions regarding the use of theory in evaluation. A common theme was the belief that teaching evaluation practitioners about theory was critical to a better future for the evaluation discipline. But why is theory so important? The session I addressed on this topic also involved Frans Leeuw, Evert Vedung and Gary Henry. I emerged from the session with some new (and old) insights about evaluation theory and evaluation practice. This brief article summarizes what I learnt – and what I said

Developing Capacity for International Developmental Evaluation

At the IDEAS Global Assembly this past May in the beautiful island of Barbados we engaged in discussions about improving international development evaluation capacity. With colleagues Tarek Azzam, Matthew Galen and John LaVelle, we discussed multiple initiatives ranging for the use of webinars, knowledge management systems, e-learning applications, massive open online courses, opportunities to use social media to enhance national and international conferences, and a range of university-based capacity development efforts. In this brief article we describe some of those initiatives

Remote Ergonomic Research in Space: Spacelab Findings and a Proposal

This paper discusses ergonomics research using remotely situated video camerasin spacecraft. Two prototype studies of crewmembers working in the micro-G environments aboard the first two flights of Spacelab are described. Various aspects of crew restraint, stabilization, manipulation of controls, and mobilization were observed, operationally defined, and quantified by observing videotaped scenes of Spacelab crewmembers. In the first study, four performance behaviors were quantified to provide estimates of their frequency of occurrence and variation over the course of each of the flights. The behaviors and their mean percent of observed times were: Hand-Hold 32.2%, Foot Restraint 35.3%, Translation 9.4%, and Struggle 3.7%. Because we observed that nearly a third of a crewmember\u27s time was spent inefficiently holding on With one hand while trying to work with the other, a second study was conducted exploring the use of foot restraints and hand stabilization. During 18 episodes of single-foot restraint, for example, there were 52 instances of hand stabilization and 135 instances of stabilization attempts with the other foot. The paper concludes with some defining charactenstics of adequate foot restraints, and a proposal for extending this research model to future spacecraft studies

The 2004 Claremont Debate: Lipsey vs. Scriven

While there is little disagreement about the need for, and value of, program evaluation, there remain major disagreements in the field about best practices (Donaldson & Lipsey, in press). For example, Donaldson and Scriven (2003) invited a diverse group of evaluators to Claremont in 2001 to share their visions for “how we should practice evaluation” in the new millennium. Theorists and practitioners discussed a wide range of views and evaluation approaches, many at odds with one another, on how best to improve evaluation practice (e.g., the experimental paradigm, evaluation as a transdiscipline, results-oriented management, empowerment evaluation, fourth generation evaluation, inclusive evaluation, theory-driven evaluation and the like). In response to some of the heated exchanges, Mark (2003) noted “it seems ironic when evaluators who espouse inclusion, empowerment, and participation would like to exclude, disempower, and see no participation by evaluators who hold different views.” Hefurther concluded that whatever peace has been achieved in the so-call quantitative-qualitative paradigm wars remains an uneasy peace

Scaling the heights of positive psychology: A systematic review of measurement scales

The volume of empirical research on positive psychology topics has grown substantially over the past two decades.  This review examines how constructs in positive psychology have been operationalized, measured, validated, cited, and applied to build the science. Based on an archive of 972 empirical articles linked to positive psychology, this review found that 762 articles used at least one measurement scale; 312 measures were created or adapted.  Findings reveal a wide range of scales being used to measure a variety of constructs, including scales on both life-enhancing and life-depleting constructs.  Key characteristics such as journals, constructs, and scale development and validation information are discussed.  There are some reliability analyses and validations occurring within the field, but the creation of new measures far outpaces the validation of existing measures.  Weaknesses such as multiple operationalizations may be rooted in inadequate discourse and synthesis.  We call for further cross-pollination for a more scientifically robust scholarship in positive psychology

The 2009 Claremont Debates: the Promise and Pitfalls of Utilization-focused and Empowerment Evaluation

Background: Hundreds of evaluators visit the Claremont Colleges in southern California each year to discuss a wide range of topics related to improving the quality of evaluation practice. Debates between thought leaders in the field have been one of the most popular and informative ways to advance understanding about how best to practice evaluation in contemporary times. Purpose: The purpose of this article is to provide a written transcript of the 2009 Claremont Evaluation Debates. The first debate is between Michael Quinn Patton and Michael Scriven on the promise and pitfalls of utilization-focused evaluation. The second debate is between David Fetterman, Michael Quinn Patton, and Michael Scriven on the promise and pitfalls of empowerment evaluation. Setting: The debates occurred at the Claremont Graduate University on August 23-24, 2009. Several hundred evaluators from around the world also viewed and participated in the debates via a live webcast

Rewarding Careers Applying Positive Psychological Science to Improve Quality of Work Life and Organizational Effectiveness

A hallmark of modern societies is the centrality of work and work-related activities (e.g., preparation for work and careers). Most adults are expected to spend the majority of their waking hours engaged in work- or career-focused endeavors into the foreseeable future (Donaldson, Gooler, & Weiss, 1998). A large body of research now suggests that work and careers are of primary importance, both socially and personally, for individuals across the globe. For example, the nature of one\u27s work often imposes a schedule and structure on one\u27s life, establishes patterns of social interaction, dictates economic status and well-being, provides others with a mean of judging one\u27s status and personal worth, and consequently becomes a major determinant of healthful adjustment and overall quality of life (Donaldson & Weiss, 1998). These conditions present psychologists with an array of settings to use psychological science to promote human welfare

Comparing and Contrasting a Program versus System Approach to Evaluation: The Example of a Cardiac Care System

This paper focuses on the application of systems thinking concepts to evaluate systems.  The terms systems and systems thinking concepts are first defined.  The use of systems thinking concepts in program and system evaluation are then highlighted.  It is noted that while there are methods available to assist evaluation practitioners in applying systems thinking concepts to program evaluation, there is a need for similar guidance in applying systems thinking concepts to evaluating systems.  System Evaluation Theory (SET) is then reviewed as one evaluation theory designed to apply systems thinking concepts to evaluating systems. A case illustration is presented to help teach practitioners how to apply SET’s three steps.  The discussion focuses on comparing the differences between evaluation questions answered by applying systems thinking concepts versus those using program logic models

Characteristics of Tobacco Users in the Lumber Industry

Cessation interventions for adult smokeless tobacco users may benefit from an improved understanding of the demographic, psychosocial, and tobacco-dependence characteristics of this group. In the current study, 143 employees of the Pacific Lumber Company were interviewed and completed questionnaires about their tobacco use product preference (smokeless tobacco only, cigarettes only, both, and former user), demographic, psychosocial, and tobacco-dependence characteristics. Results of a multivariate discriminant analysis revealed that smokeless-tobacco-only users were younger and reported in engaging in more exercise than did the other three groups; however, they also reported greater dependence on tobacco than did smokers. Formal cessation clinics similar to those that are being used effectively with smokers, and which are age appropriate, may be an effective treatment for adult smokeless tobacco users