Program Evaluation in the Context of Debates in the Field: The Evaluation of PR-CETP

This paper rationalizes the selection of the concept of energy as the central theme of a new capstone course aimed at science education majors. It describes the goals of the course and the activities that preceded the course design and led to the selection of the topics, of the educational materials, and of the teaching methodologies. It presents a sequential description of the manner in which the conceptual knowledge of energy was to be developed. The specific experiments, interactive demonstrations and other educational materials utilized for the conceptual development of the concept of energy in context are described and referenced. The course objectives are described, as well as the instruments utilized to assess student learning. It also presents the activities utilized to assess the course, in addition to the modifications made to the course syllabus based on this assessment

Evaluating Program Impact: Our Approach to Performance Assessment

Discerning and communicating the impact of grantmaking and other programmatic contributions are essential to fulfilling the Rockefeller Brothers Fund's (RBF) mission as well as our commitment to stewardship, transparency, and accountability. The Fund's board and staff have found that engaging policymakers on the results and insights gained from our grantmaking, informing the public about our grantees' work, and attracting additional donors to promising institutions and approaches are key activities that help build a more just, sustainable, and peaceful world.In order to bring additional rigor to the Fund's approach to program impact assessment, a committee of RBF trustees and staff was established in March 2012. Based on our experience, the state of evaluation in philanthropy, and a review of literature and activity in the field, the Impact Assessment Committee developed a set of principles to guide our impact assessment approach, defined terms for the purposes of RBF discussions, established several points for evaluation activities in the life cycle of a grantmaking program, and identified opportunities to embed impact assessment in the Fund's regular institutional processes. The Fund establishes its programs in fields and places that reflect its mission and the evolution of its longstanding interests, along with an analysis of the changing global context. The key elements of the RBF's approach to assessing program impact are as follows:* The board approves program guidelines that lay the foundation for the Fund's grantmaking within a program. Guidelines include a preamble that presents the vision and rationale for each program, ambitious long-term goals, and strategies that articulate specific actions the Fund will support to achieve progress toward these goals. They provide guidance to staff and grantseekers about what the RBF is prepared to fund.* A program framework summary, derived from the guidelines, is developed for internal use and includes indicators of progress. These indicators identify anticipated changes in understanding, behavior, capacity, public engagement, or public policy that would demonstrate that program strategies are contributing to realizing program goals.* Within each program, evaluation activities occur on an ongoing basis. Monitoring of the field and of individual grants draws on regular staff engagement and grantee reporting; program reviews, conducted every three to five years by program staff, provide an opportunity to engage the board in a strategic review of progress—often resulting in updated program strategies; impact assessments are conducted by external consultants after five or more years as strategies mature.* The annual institutional calendar provides a variety of opportunities for the board and staff to discuss and review programmatic impact at different points each year and across several years.This approach to impact assessment reflects emerging practices in the field and is consistent with the Fund's values and grantmaking approaches. The committee believes that the approach effectively supports program learning, guides program development, and enhances the impact of the Fund's grantmaking

Educational Games in Geriatric Medicine Education: A Systematic Review

OBJECTIVE: To systematically review the medical literature to assess the effect of geriatric educational games on the satisfaction, knowledge, beliefs, attitudes and behaviors of health care professionals. METHODS: We conducted a systematic review following the Cochrane Collaboration methodology including an electronic search of 10 electronic databases. We included randomized controlled trials (RCT) and controlled clinical trials (CCT) and excluded single arm studies. Population of interests included members (practitioners or students) of the health care professions. Outcomes of interests were participants' satisfaction, knowledge, beliefs, attitude, and behaviors. RESULTS: We included 8 studies evaluating 5 geriatric role playing games, all conducted in United States. All studies suffered from one or more methodological limitations but the overall quality of evidence was acceptable. None of the studies assessed the effects of the games on beliefs or behaviors. None of the 8 studies reported a statistically significant difference between the 2 groups in terms of change in attitude. One study assessed the impact on knowledge and found non-statistically significant difference between the 2 groups. Two studies found levels of satisfaction among participants to be high. We did not conduct a planned meta-analysis because the included studies either reported no statistical data or reported different summary statistics. CONCLUSION: The available evidence does not support the use of role playing interventions in geriatric medical education with the aim of improving the attitudes towards the elderly.Society of General Internal Medicin

A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: a summary of significant discussions at the fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR)

The fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR) was held 5–6 November 2019 in The Hague, the Netherlands. ICASR is an interdisciplinary group whose goal is to maximize the use of technology for conducting rapid, accurate, and efficient systematic reviews of scientific evidence. The group seeks to facilitate the development and acceptance of automated techniques for systematic reviews. In 2018, the major themes discussed were the transferability of automation tools (i.e., tools developed for other purposes that might be used by systematic reviewers), the automated recognition of study design in multiple disciplines and applications, and approaches for the evaluation of automation tools

A Systematic Review of the Selected Evidences on the Effectiveness of Inter-professional Education (IPE) in Developing Interprofessional Learning Environment

Purpose: This systematic review aimed to provide available evidence in determining for the effectiveness of IPE as a modality in developing interprofessional learning environment for healthcare students.Methods: The computerized searches from 2009-2015 in ten electronic databases were performed. Two independent reviewers were consulted to assess the eligibility, level of evidences and methodological quality in each study. Result: nine out of ten studies were retrieved. These studies include (a) two RCT studies; scored eight and seven out of eight, (b) Three quasi-experimental pretestposttest design; scored seven, six, and six out of eight, (c) Three controlled before and after study; scored six, six, five and five out of eight, and (d) one controlled longitudinal; scored six out of eight respectively.Conclusion: Based on the systematic review, evidence showed that IPE was effective in building strong interprofessional learning environment. On the other hand, the authors recommend considering conduct of similar systematic review grounded on IPE with larger sample size within the health allied discipline

A systematic review of pedagogical approaches that can effectively include children with special educational needs in mainstream classrooms with a particular focus on peer group interactive approaches

The broad background to this review is a long history of concepts of special pupils and special education, and a faith in special pedagogical approaches. The rise of inclusive schools and some important critiques of special pedagogy (e.g. Hart, 1996; Norwich and Lewis, 2001; Thomas and Loxley, 2001) have raised the profile of teaching approaches that ordinary teachers can and do use to include children with special educational needs in mainstream classrooms. Inclusive education itself is increasingly conceived as being about the quality of learning and participation that goes on in inclusive schools rather than simplistic matters of where children are place

Developing educational materials about risks on social network sites: a design based research approach

Nearly all of today’s Western teenagers have a profile on a social network site (SNS). As many risks have been reported, researchers and governments have emphasized the role of school education to teach teenagers how to deal safely with SNSs. However, little is known about the specific characteristics which would make interventions effective. Therefore, the overall objective of this research aims to propose a list of validated theoretical design principles for future development of educational materials about risks on SNSs. This research goal was pursued through a design-based research procedure. Thereby targeting teenagers of secondary education in 8 separate studies, the different steps of the design-based research procedure have iteratively been completed. Firstly, a problem analysis was executed through 3 explorative studies, including an observational study, a theoretical evaluation of existing materials and a survey study. Secondly, initial solutions were developed and evaluated in practice through 5 quasi-experimental intervention studies. Thirdly, we reflected upon all the previous results to produce design principles. Finally, we conclude with an analysis of the design-based research methodology

Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach

Intelligent assistants are handling increasingly critical tasks, but until now, end users have had no way to systematically assess where their assistants make mistakes. For some intelligent assistants, this is a serious problem: if the assistant is doing work that is important, such as assisting with qualitative research or monitoring an elderly parent’s safety, the user may pay a high cost for unnoticed mistakes. This paper addresses the problem with WYSIWYT/ML (What You See Is What You Test for Machine Learning), a human/computer partnership that enables end users to systematically test intelligent assistants. Our empirical evaluation shows that WYSIWYT/ML helped end users find assistants’ mistakes significantly more effectively than ad hoc testing. Not only did it allow users to assess an assistant’s work on an average of 117 predictions in only 10 minutes, it also scaled to a much larger data set, assessing an assistant’s work on 623 out of 1,448 predictions using only the users’ original 10 minutes’ testing effort

Making Evaluation Work

Outlines how evaluation has been integrated into grantmaking practices, since the foundation embarked upon a new approach to the field of youth development in 2000