Chapter 5: Evaluation

The OTiS (Online Teaching in Scotland) programme, run by the now defunct Scotcit programme, ran an International e-Workshop on Developing Online Tutoring Skills which was held between 8–12 May 2000. It was organised by Heriot–Watt University, Edinburgh and The Robert Gordon University, Aberdeen, UK. Out of this workshop came the seminal Online Tutoring E-Book, a generic primer on e-learning pedagogy and methodology, full of practical implementation guidelines. Although the Scotcit programme ended some years ago, the E-Book has been copied to the SONET site as a series of PDF files, which are now available via the ALT Open Access Repository. The editor, Carol Higgison, is currently working in e-learning at the University of Bradford (see her staff profile) and is the Chair of the Association for Learning Technology (ALT)

Online Travel Service Quality: The Importance of Pre-Transaction Services

The Internet revolution has led to significant changes in the way travel agencies interact with customers. Travel websites are used to different degrees, and for a variety of combinations of pre-transaction, transaction and post-transaction services. A better understanding of how customers interact with online services will help providers improve service quality to levels that satisfy or even delight customers, and thus create loyalty. This article provides a comprehensive review of the literature on online service quality, applies the theory to online travel offerings, and reports on an empirical study of quality perceptions of pre-transaction services provided on three travel websites. Effects on customer perceived quality were measured for process and outcome dimensions of online services. Implications for the design of online travel services and suggestions for further research are formulated.Economics ;

Evaluating Program Impact: Our Approach to Performance Assessment

Discerning and communicating the impact of grantmaking and other programmatic contributions are essential to fulfilling the Rockefeller Brothers Fund's (RBF) mission as well as our commitment to stewardship, transparency, and accountability. The Fund's board and staff have found that engaging policymakers on the results and insights gained from our grantmaking, informing the public about our grantees' work, and attracting additional donors to promising institutions and approaches are key activities that help build a more just, sustainable, and peaceful world.In order to bring additional rigor to the Fund's approach to program impact assessment, a committee of RBF trustees and staff was established in March 2012. Based on our experience, the state of evaluation in philanthropy, and a review of literature and activity in the field, the Impact Assessment Committee developed a set of principles to guide our impact assessment approach, defined terms for the purposes of RBF discussions, established several points for evaluation activities in the life cycle of a grantmaking program, and identified opportunities to embed impact assessment in the Fund's regular institutional processes. The Fund establishes its programs in fields and places that reflect its mission and the evolution of its longstanding interests, along with an analysis of the changing global context. The key elements of the RBF's approach to assessing program impact are as follows:* The board approves program guidelines that lay the foundation for the Fund's grantmaking within a program. Guidelines include a preamble that presents the vision and rationale for each program, ambitious long-term goals, and strategies that articulate specific actions the Fund will support to achieve progress toward these goals. They provide guidance to staff and grantseekers about what the RBF is prepared to fund.* A program framework summary, derived from the guidelines, is developed for internal use and includes indicators of progress. These indicators identify anticipated changes in understanding, behavior, capacity, public engagement, or public policy that would demonstrate that program strategies are contributing to realizing program goals.* Within each program, evaluation activities occur on an ongoing basis. Monitoring of the field and of individual grants draws on regular staff engagement and grantee reporting; program reviews, conducted every three to five years by program staff, provide an opportunity to engage the board in a strategic review of progress—often resulting in updated program strategies; impact assessments are conducted by external consultants after five or more years as strategies mature.* The annual institutional calendar provides a variety of opportunities for the board and staff to discuss and review programmatic impact at different points each year and across several years.This approach to impact assessment reflects emerging practices in the field and is consistent with the Fund's values and grantmaking approaches. The committee believes that the approach effectively supports program learning, guides program development, and enhances the impact of the Fund's grantmaking

Improvements to the APBS biomolecular solvation software suite

The Adaptive Poisson-Boltzmann Solver (APBS) software was developed to solve the equations of continuum electrostatics for large biomolecular assemblages that has provided impact in the study of a broad range of chemical, biological, and biomedical applications. APBS addresses three key technology challenges for understanding solvation and electrostatics in biomedical applications: accurate and efficient models for biomolecular solvation and electrostatics, robust and scalable software for applying those theories to biomolecular systems, and mechanisms for sharing and analyzing biomolecular electrostatics data in the scientific community. To address new research applications and advancing computational capabilities, we have continually updated APBS and its suite of accompanying software since its release in 2001. In this manuscript, we discuss the models and capabilities that have recently been implemented within the APBS software package including: a Poisson-Boltzmann analytical and a semi-analytical solver, an optimized boundary element solver, a geometry-based geometric flow solvation model, a graph theory based algorithm for determining p$K_a$ values, and an improved web-based visualization tool for viewing electrostatics

Experimental Methods in IIR: The Tension between Rigour and Ethics in Studies Involving Users with Dyslexia

Designing user studies in the interactive information retrieval (IIR) paradigm on people with impairments may sometimes require different methodological considerations than for other users. Consequently, there may be a tension between what the community regards as being a rigorous methodology against what researchers can do ethically with their users. This paper discusses issues to consider when designing IIR studies involving people with dyslexia, such as sampling, informed consent and data collection. The conclusion is that conducting user studies on participants with dyslexia requires special considerations at all stages of the experimental design. The purpose of this paper is to raise awareness and understanding in the research community about experimental methods involving users with dyslexia, and addresses researchers, as well as editors and reviewers. Several of the issues raised do not only apply to people with dyslexia, but have implications when researching other groups, for instance elderly people and users with learning, cognitive, sensory or motor impairments

A Meta-Analysis of Procedures to Change Implicit Measures

Using a novel technique known as network meta-analysis, we synthesized evidence from 492 studies (87,418 participants) to investigate the effectiveness of procedures in changing implicit measures, which we define as response biases on implicit tasks. We also evaluated these procedures’ effects on explicit and behavioral measures. We found that implicit measures can be changed, but effects are often relatively weak (|ds| \u3c .30). Most studies focused on producing short-term changes with brief, single-session manipulations. Procedures that associate sets of concepts, invoke goals or motivations, or tax mental resources changed implicit measures the most, whereas procedures that induced threat, affirmation, or specific moods/emotions changed implicit measures the least. Bias tests suggested that implicit effects could be inflated relative to their true population values. Procedures changed explicit measures less consistently and to a smaller degree than implicit measures and generally produced trivial changes in behavior. Finally, changes in implicit measures did not mediate changes in explicit measures or behavior. Our findings suggest that changes in implicit measures are possible, but those changes do not necessarily translate into changes in explicit measures or behavior

Surveying Persons with Disabilities: A Source Guide (Version 1)

As a collaborator with the Cornell Rehabilitation Research and Training Center on Disability Demographics and Statistics, Mathematica Policy Research, Inc. has been working on a project that identifies the strengths and limitations in existing disability data collection in both content and data collection methodology. The intended outcomes of this project include expanding and synthesizing knowledge of best practices and the extent existing data use those practices, informing the development of data enhancement options, and contributing to a more informed use of existing data. In an effort to provide the public with an up-to-date and easily accessible source of research on the methodological issues associated with surveying persons with disabilities, MPR has prepared a Source Guide of material related to this topic. The Source Guide contains 150 abstracts, summaries, and references, followed by a Subject Index, which cross references the sources from the Reference List under various subjects. The Source Guide is viewed as a “living document,” and will be periodically updated

Determinants of Shopping Behavior of Urban Consumers

This study explores the influence of geo-demographic settings of commercial centers, customer attractions in shopping malls, and route to shopping of urban shoppers. The present research analyzes retailing patterns in urban areas in reference to customer orientation strategies, product search behavior and enhancing the customer value. Interrelationship among urban retailing, marketplace ambiance, conventional shopping wisdom of customers, long-term customer services, and technology led selling processes are also addressed in the study based on empirical survey. Broadly, this study makes contributions to the existing research in urban retailing towards factors determining shopping attractions, routes to shopping, and establishing the customer-centric strategies of the firms.Shopping mall, multi-channel retailing, consumer behavior, customer-centric strategy, market attractiveness, customer satisfaction

GEOSPATIAL ANALYSIS OF PEDIATRIC TUBERCULOSIS IN BOHOL, PHILIPPINES: DISEASE CLUSTERS AND ACCESS TO CARE

Worldwide, children represent approximately 10% of global tuberculosis (TB) cases. We recently reported a high tuberculin skin test (TST)-positive prevalence (355/5,476; weighted prevalence=6.4%) among children (\u3c15 years) throughout the island of Bohol, Philippines, with some geographically isolated communities having prevalence as high as 29%. In this study, we conducted a secondary geospatial and hot-spot analysis of this household-based cluster survey to assess the association between access to care (distance to a health care facility) and TST-positive prevalence. Our analysis indicated that villages with high TST-positive prevalence (≥10%) were significantly further from Provincial Health Office based in the capital city of Tagbilaran in time-distance (p=0.0001, r=0.2387) and kilometers (p=0.0011, r=0.3170). Similarly, prevalence was positively correlated with distance to the municipal Rural Health Unit (RHU), where most receive their medical care, in time-distance (p=0.0055; r=0.0415); however, the association was not significant in kilometers (p=0.3315; r=0.0715). Distance to health care facilities could represent an obstacle to access to care, thereby limiting diagnosis and treatment and resulting in increased risk of transmission to others in the community. Health care outreach in geographically isolated areas is critical to treatment and control of efforts for pediatric TB

Open TURNS: An industrial software for uncertainty quantification in simulation

The needs to assess robust performances for complex systems and to answer tighter regulatory processes (security, safety, environmental control, and health impacts, etc.) have led to the emergence of a new industrial simulation challenge: to take uncertainties into account when dealing with complex numerical simulation frameworks. Therefore, a generic methodology has emerged from the joint effort of several industrial companies and academic institutions. EDF R&D, Airbus Group and Phimeca Engineering started a collaboration at the beginning of 2005, joined by IMACS in 2014, for the development of an Open Source software platform dedicated to uncertainty propagation by probabilistic methods, named OpenTURNS for Open source Treatment of Uncertainty, Risk 'N Statistics. OpenTURNS addresses the specific industrial challenges attached to uncertainties, which are transparency, genericity, modularity and multi-accessibility. This paper focuses on OpenTURNS and presents its main features: openTURNS is an open source software under the LGPL license, that presents itself as a C++ library and a Python TUI, and which works under Linux and Windows environment. All the methodological tools are described in the different sections of this paper: uncertainty quantification, uncertainty propagation, sensitivity analysis and metamodeling. A section also explains the generic wrappers way to link openTURNS to any external code. The paper illustrates as much as possible the methodological tools on an educational example that simulates the height of a river and compares it to the height of a dyke that protects industrial facilities. At last, it gives an overview of the main developments planned for the next few years