Evaluation Policy and Evaluation Practice

Three issues for evaluation policy and practice are described: evaluation policy dimensions, evaluation policy instruments, and the political and economic environment for evaluation policy. Selected future directions are outlined, including the need to describe the evaluation policy landscape, further articulate an evaluation policy taxonomy, and develop and implement tactics for influencing evaluation policy, with particular attention to the role of professional associations

A Protocol of Systems Evaluation

National Science Foundation Systems Evaluation Grant No. EREC-053549

A Theory of Systems Evaluation

National Science Foundation Systems Evaluation Grant No. EREC-053549

Geophysical Applications for Arctic/Subarctic Transportation Planning

This report describes a series of geophysical surveys conducted in conjunction with geotechnical investigations carried out by the Alaska Department of Transportation and Public Facilities. The purpose of the study was to evaluate the value of and potential uses for data collected via geophysical techniques with respect to ongoing investigations related to linear infrastructure. One or more techniques, including direct-current resistivity, capacitive-coupled resistivity, and ground-penetrating radar, were evaluated at sites in continuous and discontinuous permafrost zones. Results revealed that resistivity techniques adequately differentiate between frozen and unfrozen ground, and in some instances, were able to identify individual ice wedges in a frozen heterogeneous matrix. Capacitive-coupled resistivity was found to be extremely promising due to its relative mobility as compared with direct-current resistivity. Ground-penetrating radar was shown to be useful for evaluating the factors leading to subsidence in an existing road. Taken as a whole, the study results indicate that supplemental geophysical surveys may add to the quality of a geotechnical investigation by helping to optimize the placement of boreholes. Moreover, such surveys may reduce the overall investigation costs by reducing the number of boreholes required to characterize the subsurface

Greater Than the Sum: Systems Thinking in Tobacco control

Tobacco control and public health have evolved into a complex set of interconnected and largely self-organizing systems. Their components include international, national, and local governmental agencies; individual advocacy groups; policy makers; health care professionals; nonprofit foundations; and the general population itself. The issues require the exploration of approaches and methodologies that speak to the evolving, dynamic nature of this systems environment. This monograph focuses on the first two years of the Initiative on the Study and Implementation of Systems (ISIS), which was funded by the National Cancer Institute to examine the potential for systems thinking in tobacco control and public health. ISIS explored the general idea of a systems thinking rubric encompassing a great variety of systems-oriented methodologies and approaches. Four approaches have particular promise for their applicability to tobacco control and public health and thus were chosen as areas for initial investigation: (1) organizing and managing as a system, (2) system dynamics and how to model those dynamics, (3) system networks and their analysis, and (4) systems knowledge and its management and translation. As a transdisciplinary effort that linked both tobacco control stakeholders and systems experts, ISIS combined a number of exploratory projects and case studies within these four approaches with a detailed examination of the potential for systems thinking in tobacco control. Its end product was a set of expert consensus guidelines for the future implementation of systems thinking and systems perspectives for tobacco control and public health.https://cancercontrol.cancer.gov/tcrb/monographs/18/index.htm

Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned.

The National Institutes of Health (NIH) Roadmap for Medical Research initiative, funded by the NIH Common Fund and offered through the Clinical and Translational Science Award (CTSA) program, developed more than 60 unique models for achieving the NIH goal of accelerating discoveries toward better public health. The variety of these models enabled participating academic centers to experiment with different approaches to fit their research environment.A central challenge related to the diversity of approaches is the ability to determine the success and contribution of each model. This paper describes the effort by the Evaluation Key Function Committee to develop and test a methodology for identifying a set of common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics. The project involved more than one-fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort

Complementary and conventional medicine: a concept map

BACKGROUND: Despite the substantive literature from survey research that has accumulated on complementary and alternative medicine (CAM) in the United States and elsewhere, very little research has been done to assess conceptual domains that CAM and conventional providers would emphasize in CAM survey studies. The objective of this study is to describe and interpret the results of concept mapping with conventional and CAM practitioners from a variety of backgrounds on the topic of CAM. METHODS: Concept mapping, including free sorts, ratings, and multidimensional scaling was used to organize conceptual domains relevant to CAM into a visual "cluster map." The panel consisted of CAM providers, conventional providers, and university faculty, and was convened to help formulate conceptual domains to guide the development of a CAM survey for use with United States military veterans. RESULTS: Eight conceptual clusters were identified: 1) Self-assessment, Self-care, and Quality of Life; 2) Health Status, Health Behaviors; 3) Self-assessment of Health; 4) Practical/Economic/ Environmental Concerns; 5) Needs Assessment; 6) CAM vs. Conventional Medicine; 7) Knowledge of CAM; and 8) Experience with CAM. The clusters suggest panelists saw interactions between CAM and conventional medicine as a critical component of the current medical landscape. CONCLUSIONS: Concept mapping provided insight into how CAM and conventional providers view the domain of health care, and was shown to be a useful tool in the formulation of CAM-related conceptual domains

Benefits of a College STEM Faculty Development Initiative: Instructors Report Increased and Sustained Implementation of Research-Based Instructional Strategies

The Summer Institutes on Scientific Teaching (SI) is a faculty development workshop in which science, technology, engineering, and mathematics (STEM) instructors, particularly from biology, are trained in the Scientific Teaching (ST) pedagogy. While participants have generally reported positive experiences, we aimed to assess how the SI affected participants’ teaching practices. Building on a previously developed taxonomy of ST practices, we surveyed SI participants from the 2004–2014 SI classes regarding specific ST practices. Participants’ self-reported use and implementation of ST practices increased immediately after SI attendance as well as over a longer time frame, suggesting that implementation persisted and even increased with time. However, instructors reported implementation gains for some practices more than others. The practices with the highest gains were engaging students in their own learning, using learning goals in course design, employing formative assessment, developing overarching course learning goals, representing science as a process, and facilitating group discussion activities. We propose that the ST practices showing the greatest gains may serve as beneficial focal points for professional development programs, while practices with smaller gains may require modified dissemination approaches or support structures

Advancing the Science of Team Science

The First Annual International Science of Team Science (SciTS) Conference was held in Chicago, IL April 22-24, 2010. This article presents a summary of the Conference proceedings. Clin Trans Sci 2010; Volume 3: 263-266