Dealing with Stakeholder Values in the Evaluation of Development Programs. A Methodological Framework for Mid Term Evaluation

In mid term program evaluations evaluators are often confronted with the double task of retrospectively judging the program's merit and worth while at the same time advising decision makers concerning future adjustments in courses of action. In such cases, it can be argued that it is particularly important that evaluators take into account the divergent views and needs of different stakeholder groups. In principle, program theory evaluation can constitute a sound basis for dealing with the double objective of retrospective judgment and proactive program improvement. However, as argued in the paper, current approaches in program theory evaluation may not be sufficiently equipped to systematically deal with divergent stakeholder values. Taking into account lessons from the literature on stakeholder values in evaluation, an alternative methodological framework is presented. The framework combines program theory evaluation with elements of multicriteria decision aid. An example is used to illustrate the framework.

Critical thinking and systems thinking: towards a critical literacy for systems thinking in practice

About the book: In reflective problem solving and thoughtful decision making using critical thinking one considers evidence, the context of judgment, the relevant criteria for making the judgment well, the applicable methods or techniques for forming the judgment, and the applicable theoretical constructs for understanding the problem and the question at hand. In this book, the authors present topical research in the study of critical thinking. Topics discussed include developing critical thinking through probability models; the promotion of critical thinking skills through argument mapping; an instructional model for teacher training in critical thinking; advanced academic literacy and critical thinking and critical thinking and higher education

Real-time Assessment, Prediction, and Scaffolding of Middle School Students’ Data Collection Skills within Physical Science Simulations

Despite widespread recognition by science educators, researchers and K-12 frameworks that scientific inquiry should be an essential part of science education, typical classrooms and assessments still emphasize rote vocabulary, facts, and formulas. One of several reasons for this is that the rigorous assessment of complex inquiry skills is still in its infancy. Though progress has been made, there are still many challenges that hinder inquiry from being assessed in a meaningful, scalable, reliable and timely manner. To address some of these challenges and to realize the possibility of formative assessment of inquiry, we describe a novel approach for evaluating, tracking, and scaffolding inquiry process skills. These skills are demonstrated as students experiment with computer-based simulations. In this work, we focus on two skills related to data collection, designing controlled experiments and testing stated hypotheses. Central to this approach is the use and extension of techniques developed in the Intelligent Tutoring Systems and Educational Data Mining communities to handle the variety of ways in which students can demonstrate skills. To evaluate students\u27 skills, we iteratively developed data-mined models (detectors) that can discern when students test their articulated hypotheses and design controlled experiments. To aggregate and track students\u27 developing latent skill across activities, we use and extend the Bayesian Knowledge-Tracing framework (Corbett & Anderson, 1995). As part of this work, we directly address the scalability and reliability of these models\u27 predictions because we tested how well they predict for student data not used to build them. When doing so, we found that these models demonstrate the potential to scale because they can correctly evaluate and track students\u27 inquiry skills. The ability to evaluate students\u27 inquiry also enables the system to provide automated, individualized feedback to students as they experiment. As part of this work, we also describe an approach to provide such scaffolding to students. We also tested the efficacy of these scaffolds by conducting a study to determine how scaffolding impacts acquisition and transfer of skill across science topics. When doing so, we found that students who received scaffolding versus students who did not were better able to acquire skills in the topic in which they practiced, and also transfer skills to a second topic when was scaffolding removed. Our overall findings suggest that computer-based simulations augmented with real-time feedback can be used to reliably measure the inquiry skills of interest and can help students learn how to demonstrate these skills. As such, our assessment approach and system as a whole shows promise as a way to formatively assess students\u27 inquiry

Impact Evaluations and Development: Nonie Guidance on Impact Evaluation

In international development, impact evaluation is principally concerned with final results of interventions (programs, projects, policy measures, reforms) on the welfare of communities, households, and individuals, including taxpayers and voters. Impact evaluation is one tool within the larger toolkit of monitoring and evaluation (including broad program evaluations, process evaluations, ex ante studies, etc.).The Network of Networks for Impact Evaluation (NONIE) was established in 2006 to foster more and better impact evaluations by its membership -- the evaluation networks of bilateral and multilateral organizations focusing on development issues, as well as networks of developing country evaluators. NONIE's member networks conduct a broad set of evaluations, examining issues such as project and strategy performance, institutional development, and aid effectiveness. By sharing methodological approaches and promoting learning by doing on impact evaluations, NONIE aims to promote the use of this more specific approach by its members within their larger portfolio of evaluations. This document, by Frans Leeuw and Jos Vaessen, has been developed to support this focus.For development practitioners, impact evaluations play a keyrole in the drive for better evidence on results and development effectiveness. They are particularly well suited to answer important questions about whether development interventions do or do not work, whether they make a difference, and how cost-effective they are. Consequently, they can help ensure that scarce resources are allocated where they can have the most developmental impact

Improving Qualitative Assessment In Higher Education

This dissertation considers in turn the role education plays in civil, democratic society; the role assessment plays in education; and the role theoretical constructs and cultural contexts play in assessment. Then, through literature review, document analysis, and interviews, the analysis investigates, identifies, and recommends grounded-theory-derived practices for improving qualitative assessment in higher education settings. The process of qualitative assessment is understood as being heuristic and continual, requiring re-examination and revision to maintain both its validity and reliability. To this end, rubrics are essential to efficiently and reliably assessing everything qualitative, whereas the realities of institutional culture and politics require adroit leadership from educators and administrators, drawing from manifest praxes in organizational theory, management theory, and political theory, to affect progressive change

How might teachers enable self-confidence? A review study

In the context of learner-centred learning and curricular reform, self-confidence is invoked as an important construct. However, there is no easily available research-informed guidance on what self-confidence means for the professional teacher. This study uses the analytic technique of Concept Analysis to review psychology and education literatures to provide a 'take-home' message for teachers. The review identifies conceptual artefacts (ideas, theories, concepts which explain, connect, predict or apply knowledge) that the teacher can appropriate in order to enable learner self-confidence. These conceptual artefacts are classified in three groups: characterising self-confidence; self-judgements of confidence; and factors that influence the development of self-confidence. The review finds self-confidence to be a robust and stable psychological construct, best promoted through teachers' attention to learners' development of knowledge and engagement in socially designed learning activities. It further finds that teachers' attention to activities which involve learners' self-regulation are of importance

Randomization does not help much, comparability does

Following Fisher, it is widely believed that randomization "relieves the experimenter from the anxiety of considering innumerable causes by which the data may be disturbed." In particular, it is said to control for known and unknown nuisance factors that may considerably challenge the validity of a result. Looking for quantitative advice, we study a number of straightforward, mathematically simple models. However, they all demonstrate that the optimism with respect to randomization is wishful thinking rather than based on fact. In small to medium-sized samples, random allocation of units to treatments typically yields a considerable imbalance between the groups, i.e., confounding due to randomization is the rule rather than the exception. In the second part of this contribution, we extend the reasoning to a number of traditional arguments for and against randomization. This discussion is rather non-technical, and at times even "foundational" (Frequentist vs. Bayesian). However, its result turns out to be quite similar. While randomization's contribution remains questionable, comparability contributes much to a compelling conclusion. Summing up, classical experimentation based on sound background theory and the systematic construction of exchangeable groups seems to be advisable

Ready to Engage? Urban Middle School Teachers’ Responsiveness to Targeted Engagement Interventions on Their Virtual Instructional Practices: An Action Research Study

Teachers’ effectiveness is associated with their instructional practices and is ultimately linked to students’ learning outcomes. In order to impact teachers’ effectiveness, schools focus substantial effort and resources on professional development led by an assumption that teachers’ classroom practices can be improved through targeted interventions. Even if this premise is correct, little information is available about how much a teacher’s practice may change through interventions, or which aspects of instructional practice are more receptive to improving teacher effectiveness (Garret et al., 2019). This study took place at an urban middle school and examined teachers’ responsiveness to targeted engagement intervention in their instructional practices during six weeks of virtual learning. These interventions were addressed through action research and consisted of professional development, coaching, and instructional feedback. There were six teacher participants in this study, three math, and three science. Data collected in this study contains observational field notes, coaching plans, coaching cycles, engagement frequency charts, professional development constructs, surveys, artifacts, and interviews. Findings from this study show: (a) positive responsiveness to teachers’ engagement interventions evidenced by increase in engagement practices during the length of the study; (b) increase in teachers’ perceptions about instructional feedback and professional development; (c) coaching with feedback grounded in data surfaced as most impactful intervention in this study; and; (d) engagement practices relevant to the socio-emotional and behavioral domain were least responsive to change; and (e) teachers’ beliefs and growth mindset drove the need in practice change. There was no evidence of practices in the behavioral engagement domain. Future recommendations of this are geared towards exploration into virtual environments that address: a) socio-emotional and behavioral engagement domains; b) student-teacher relatedness as referenced by Marzano and Pickering (2011), and c) deep understanding and high participation as referenced by Himmele’s (2011) Cognitive Engagement Model

Reflective Argumentation

Theories of argumentation usually focus on arguments as means of persuasion, finding consensus, or justifying knowledge claims. However, the construction and visualization of arguments can also be used to clarify one's own thinking and to stimulate change of this thinking if gaps, unjustified assumptions, contradictions, or open questions can be identified. This is what I call "reflective argumentation." The objective of this paper is, first, to clarify the conditions of reflective argumentation and, second, to discuss the possibilities of argument visualization methods in supporting reflection and cognitive change. After a discussion of the cognitive problems we are facing in conflicts--obviously the area where cognitive change is hardest--the second part will, based on this, determine a set of requirements argument visualization tools should fulfill if their main purpose is stimulating reflection and cognitive change. In the third part, I will evaluate available argument visualization methods with regard to these requirements and talk about their limitations. The fourth part, then, introduces a new method of argument visualization which I call Logical Argument Mapping (LAM). LAM has specifically been designed to support reflective argumentation. Since it uses primarily deductively valid argument schemes, this design decision has to be justified with regard to goals of reflective argumentation. The fifth part, finally, provides an example of how Logical Argument Mapping could be used as a method of reflective argumentation in a political controversy