Exploring the Issues: An Evaluation Literature Review

Finding ways to make evaluation more meaningful and more useful has been a key theme in the evaluation literature since the discipline began, and there is no shortage of discussion around improving evaluation among nonprofit practitioners. The topic has been a highlight at ONN's annual conference in recent years.However, much of the discussion around improving evaluation focuses on methodology, tools, and indicators.There has been less attention paid to who is asking and determining the questions of evaluation, such as who evaluation is for and what is its purpose. Consequently, the purpose of this background paper is to review the literature on evaluation use with a particular focus on systemic factors. In other words, we are interested in looking at the relationship between evaluation practice and the overall structure and function of the nonprofit sector in Ontario.We're interested in the policies and regulations that guide us, the roles played by various actors, theassumptions we make, the language we use, and the ways in which resources move through the sector. We're examining the purposes that evaluation serves, both overt and implicit. We want to learn more about the factors that make evaluations really useful, the issues that can get in the way of evaluations being useful, and ideas for improvement. Ultimately, our goal in this paper is to generate a broad vision to inform our project's final outcomes

Embracing complexity: theory, cases and the future of bioethics

This paper reflects on the relationship between theory and practice in bioethics, by using various concepts drawn from debates on innovation in healthcare research—in particular debates around how best to connect up blue skies ‘basic’ research with practical innovations that can improve human lives. It argues that it is a mistake to assume that the most difficult and important questions in bioethics are the most abstract ones, and also a mistake to assume that getting clear about abstract cases will automatically be of much help in getting clear about more complex cases. It replaces this implicitly linear model with a more complex one that draws on the idea of translational research in healthcare. On the translational model, there is a continuum of cases from the most simple and abstract (thought experiments) to the most concrete and complex (real world cases). Insights need to travel in both directions along this continuum—from the more abstract to the more concrete and from the more concrete to the more abstract. The paper maps out some difficulties in moving from simpler to more complex cases, and in doing so makes recommendations about the future of bioethics

Modern topics in theoretical nuclear physics

Over the past five years there have been profound advances in nuclear physics based on effective field theory and the renormalization group. In this brief, we summarize these advances and discuss how they impact our understanding of nuclear systems and experiments that seek to unravel their unknowns. We discuss future opportunities and focus on modern topics in low-energy nuclear physics, with special attention to the strong connections to many-body atomic and condensed matter physics, as well as to astrophysics. This makes it an exciting era for nuclear physics.Comment: 8 pages, 1 figure, prepared for the Nuclear Physics Town Hall Meeting at TRIUMF, Sept. 9-10, 2005, comments welcome, references adde

Process Evaluation Insights on Program Implementation

This white paper has the explicit intention to draw lessons learned from the past 30 years of energy efficiency program evaluation in order to facilitate improved program design and implementation going forward. The discussion in this white paper is developed based on interviews with 43 individuals who are either practitioners or users of process and market evaluation. In addition, we obtained references to published materials from our contacts and reviewed conference proceedings dating from 1992 to 2008, which resulted in a review of nearly 100 articles or reports documenting the results of, or commenting on, process and market evaluations

Conceptual modelling: framework, principles, and future research

The conceptual modelling task in a simulation project is very important and yet is still generally regarded as more of an art than a science. The meaning and nature of conceptual modelling are discussed and a framework set out. The overall aim should be to choose the best model for the project and conceptual modelling can be viewed as a difficult optimisation problem that can be tackled effectively using a creative search process that develops alternative models and predicts their performance throughout the project. An experiment relating model characteristics to some aspects of performance is described and this type of experiment may inform the process of predicting model performance. Based on advice from the literature and my own previous work on conceptual modelling 17 principles of conceptual modelling are suggested. Conceptual modelling research is still at an early stage and ideas for future research are proposed

Virtual Leadership: Required Competencies for Effective Leaders

There are countless books, articles, and resources available which attempt to identify the competencies and qualities of effective leaders. Traditionally, leaders have been at the center of a community, be it work, church, or social groups. In these communities, face-to-face meetings and close personal interaction have dominated the way leaders interact with their members. However, with the advent of the internet and the host of communication tools that followed, teams today are becoming increasingly dispersed and diverse. Studies are now being done to understand how leadership has or should evolve in order to meet the changing needs and demands of these new and different communities. Some argue that leadership in the virtual environment is simpler as fewer tools are available to virtual leaders. Others may argue that access to fewer tools makes virtual leadership more complex and challenging than traditional leadership. This paper will explore leadership in virtual settings and how it’s changing as more teams move away from traditional team environments. I’ll review the responsibilities and roles of virtual leaders in an effort to better highlight the core competencies needed in today’s virtual settings. I’ll also look at competencies required of global virtual leaders and I’ll address how these competencies can be cultivated to ensure leaders are more effective in leading teams in these new environments

The Health Status of Southern Children: A Neglected Regional Disparity

Purpose: Great variations exist in child health outcomes among states in the United States, with southern states consistently ranked among the lowest in the country. Investigation of the geographical distribution of children’s health status and the regional factors contributing to these outcomes has been neglected. We attempted to identify the degree to which region of residence may be linked to health outcomes for children with the specific aim of determining whether living in the southern region of the United States is adversely associated with children’s health status. Methods: A child health index (CHI) that ranked each state in the United States was computed by using statespecific composite scores generated from outcome measures for a number of indicators of child health. Five indicators for physical health were chosen (percent low birth weight infants, infant mortality rate, child death rate, teen death rate, and teen birth rates) based on their historic and routine use to define health outcomes in children. Indicators were calculated as rates or percentages. Standard scores were calculated for each state for each health indicator by subtracting the mean of the measures for all states from the observed measure for each state. Indicators related to social and economic status were considered to be variables that impact physical health, as opposed to indicators of physical health, and therefore were not used to generate the composite child health score. These variables were subsequently examined in this study as potential confounding variables. Mapping was used to redefine regional groupings of states, and parametric tests (2-sample t test, analysis of means, and analysis-of-variance F tests) were used to compare the means of the CHI scores for the regional groupings and test for statistical significance. Multiple regression analysis computed the relationship of region, social and economic indicators, and race to the CHI. Simple linear-regression analyses were used to assess the individual effect of each indicator. Results: A geographic region of contiguous states, characterized by their poor child health outcomes relative to other states and regions of the United States, exists within the “Deep South” (Mississippi, Louisiana, Arkansas, Tennessee, Alabama, Georgia, North Carolina, South Carolina, and Florida). This Deep-South region is statistically different in CHI scores from the US Census Bureau– defined grouping of states in the South. The mean of CHI scores for the Deep-South region was \u3e1 SD below the mean of CHI scores for all states. In contrast, the CHI score means for each of the other 3 regions were all above the overall mean of CHI scores for all states. Regression analysis showed that living in the Deep- South region is a stronger predictor of poor child health outcomes than other consistently collected and reported variables commonly used to predict children’s health. Conclusions: The findings of this study indicate that region of residence in the United States is statistically related to important measures of children’s health and may be among the most powerful predictors of child health outcomes and disparities. This clarification of the poorer health status of children living in the Deep South through spatial analysis is an essential first step for developing a better understanding of variations in the health of children. Similar to early epidemiology work linking geographic boundaries to disease, discovering the mechanisms/pathways/causes by which region influences health outcomes is a critical step in addressing disparities and inequities in child health and one that is an important and fertile area for future research. The reasons for these disparities may be complex and synergistically related to various economic, political, social, cultural, and perhaps even environmental (physical) factors in the region. This research will require the use and development of new approaches and applications of spatial analysis to develop insights into the societal, environmental, and historical determinants of child health that have been neglected in previous child health outcomes and policy research. The public policy implications of the findings in this study are substantial. Few, if any, policies identify these children as a high-risk group on the basis of their region of residence. A better understanding of the depth and breadth of disparities in health, education, and other social outcomes among and within regions of the United States is necessary for the generation of policies that enable policy makers to address and mitigate the factors that influence these disparities. Defining and clarifying the regional boundaries is also necessary to better inform public policy decisions related to resource allocation and the prevention and/or mitigation of the effects of region on child health. The identification of the Deep South as a clearly defined sub-region of the Census Bureau’s regional definition of the South suggests the need to use more culturally and socially relevant boundaries than the Census Bureau regions when analyzing regional data for policy development

How to Host a Data Competition: Statistical Advice for Design and Analysis of a Data Competition

Data competitions rely on real-time leaderboards to rank competitor entries and stimulate algorithm improvement. While such competitions have become quite popular and prevalent, particularly in supervised learning formats, their implementations by the host are highly variable. Without careful planning, a supervised learning competition is vulnerable to overfitting, where the winning solutions are so closely tuned to the particular set of provided data that they cannot generalize to the underlying problem of interest to the host. This paper outlines some important considerations for strategically designing relevant and informative data sets to maximize the learning outcome from hosting a competition based on our experience. It also describes a post-competition analysis that enables robust and efficient assessment of the strengths and weaknesses of solutions from different competitors, as well as greater understanding of the regions of the input space that are well-solved. The post-competition analysis, which complements the leaderboard, uses exploratory data analysis and generalized linear models (GLMs). The GLMs not only expand the range of results we can explore, they also provide more detailed analysis of individual sub-questions including similarities and differences between algorithms across different types of scenarios, universally easy or hard regions of the input space, and different learning objectives. When coupled with a strategically planned data generation approach, the methods provide richer and more informative summaries to enhance the interpretation of results beyond just the rankings on the leaderboard. The methods are illustrated with a recently completed competition to evaluate algorithms capable of detecting, identifying, and locating radioactive materials in an urban environment.Comment: 36 page

Improving Primo Usability and Teachability with Help from the Users

In the aftermath of a consortium migration to a shared cloud-based resource management and discovery system, a small college library implemented a web usability test to uncover the kinds of difficulties students had with the new interface. Lessons learned from this study led to targeted changes, which simplified aspects of searching, but also enhanced the librarians’ ability to teach more effectively. The authors discuss the testing methods, results, and teaching opportunities, both realized and potential, which arose from implementing changes