Professional Decision-Making in Research (PDR): The validity of a new measure

In this paper, we report on the development and validity of the Professional Decision-Making in Research (PDR) measure, a vignette-based test that examines decision-making strategies used by investigators when confronted with challenging situations in the context of empirical research. The PDR was administered online with a battery of validity measures to a group of NIH-funded researchers and research trainees who were diverse in terms of age, years of experience, types of research, and race. The PDR demonstrated adequate reliability (alpha = .84) and parallel form correlation (r = .70). As hypothesized, the PDR was significantly negatively correlated with narcissism, cynicism, moral disengagement, and compliance disengagement; it was not correlated with socially desirable responding. In regression analysis, the strongest predictors of higher PDR scores were low compliance disengagement, speaking English as a native language, conducting clinical research with human subjects, and low levels of narcissism. Given that the PDR was written at an eighth grade reading level to be suitable for use with English as a second language participants and that only one-fourth of items focused on clinical research, further research into the possible roles of culture and research ethics training across specialties is warranted. This initial validity study demonstrates the potential usefulness of the PDR as an educational outcome assessment measure and a research instrument for studies on professionalism and integrity in research

Professional decision-making in medicine: Development of a new measure and preliminary evidence of validity

INTRODUCTION: This study developed a new Professional Decision-Making in Medicine Measure that assesses the use of effective decision-making strategies: seek help, manage emotions, recognize consequences and rules, and test assumptions and motives. The aim was to develop a content valid measure and obtain initial evidence for construct validity so that the measure could be used in future research or educational assessment. METHODS: Clinical scenario-based items were developed based on a review of the literature and interviews with physicians. For each item, respondents are tasked with selecting two responses (out of six plausible options) that they would choose in that situation. Three of the six options reflect a decision-making strategy; these responses are scored as correct. Data were collected from a sample of 318 fourth-year medical students in the United States. They completed a 16-item version of the measure (Form A) and measures of social desirability, moral disengagement, and professionalism attitudes. Professionalism ratings from clerkships were also obtained. A sub-group (n = 63) completed a second 16-item measure (Form B) to pilot test the instrument, as two test forms are useful for pre-posttest designs. RESULTS: Scores on the new measure indicated that, on average, participants answered 75% of items correctly. Evidence for construct validity included the lack of correlation between scores on the measure and socially desirable responding, negative correlation with moral disengagement, and modest to low correlations with professionalism attitudes. A positive correlation was observed with a clerkship rating focused on professionalism in peer interactions. CONCLUSIONS: These findings demonstrate modest proficiency in the use of decision-making strategies among fourth-year medical students. Additional research using the Professional Decision-Making Measure should explore scores among physicians in various career stages, and the causes and correlates of scores. Educators could utilize the measure to assess courses that teach decision-making strategies

Aligning objectives and assessment in responsible conduct of research instruction

Efforts to advance research integrity in light of concerns about misbehavior in research rely heavily on education in the responsible conduct of research (RCR). However, there is limited evidence for the effectiveness of RCR instruction as a remedy. Assessment is essential in RCR education if the research community wishes to expend the effort of instructors, students, and trainees wisely. This article presents key considerations that instructors and course directors must consider in aligning learning objectives with instructional methods and assessment measures, and it provides illustrative examples. Above all, in order for RCR educators to assess outcomes more effectively, they must align assessment to their learning objectives and attend to the validity of the measures used

Prognostic Launch Vehicle Probability of Failure Assessment Methodology for Conceptual Systems Predicated on Human Causal Factors

Lessons learned from past failures of launch vehicle developments and operations were used to create a new method to predict the probability of failure of conceptual systems. Existing methods such as Probabilistic Risk Assessments and Human Risk Assessments were considered but found to be too cumbersome for this type of system-wide application for yet-to-be-flown vehicles. The basis for this methodology were historic databases of past failures, where it was determined that various faulty human-interactions were the predominant root causes of failure rather than deficient component reliabilities evaluated through statistical analysis. This methodology contains an expert scoring part which can be used in either a qualitative or a quantitative mode. The method produces two products: a numerical score of the probability of failure or guidance to program management on critical areas in need of increased focus to improve the probability of success. In order to evaluate the effectiveness of this new method, data from a concluded vehicle program (USAF's Titan IV with the Centaur G-Prime upper stage) was used as a test case. Although the theoretical vs. actual probability of failure was found to be in reasonable agreement (4.46% vs. 6.67% respectively) the underlying sub-root cause scoring had significant disparities attributable to significant organizational changes and acquisitions. Recommendations are made for future applications of this method to ongoing launch vehicle development programs

The educational scorecard: The Start of our Journey

Against a backdrop of public sector cuts, increasing university fees and high youth unemployment, we are facing challenges in Higher Education to demonstrate the value of our courses. Assessing the value of learning, however, is not straight forward. This paper reports on a study of evaluation processes on a post-graduate, professionally accredited diploma delivered at a selection of post 1992 universities. The driver for the study was a concern that current evaluation processes do not fully demonstrate the value of the course nor take into account the needs of multiple stakeholders. The project included benchmarking University evaluation processes and conducting a dialogue with stakeholders. The study adopted a qualitative management research approach, involving: a review of current practice, comparison with a sample of equivalent courses, and consultation (in the form of focus groups and semi-structured interviews) with a sample of students and employers. The paper discusses findings and proposes recommendations for future evaluation procedures in the design of an ‘educational scorecard’ for the course. Kaplan and Norton’s ‘balanced scorecard (1996) concept was adapted to provide a mechanism to represent and balance the needs of different stakeholder groups in the education process. We argue that we cannot truly assess accountability and comparability without engaging a range of stakeholders, not only in soliciting their views on the outcome of the learning but also in the design and implementation of evaluation processes. The ‘educational scorecard’ presented in this paper is developed specifically for the post-graduate diploma in Human Resource Management (PDHRM) at Leeds Metropolitan University and the report concludes with some initial reflections on the benefits of adopting the scorecard methodology The proposed model is flexible and may be adapted for other HE institutions and courses

What are the impacts and cost-effectiveness of strategies to improve performance of untrained and under-trained teachers in the classroom in developing countries?

What are the impacts and cost effectiveness of strategies to improve performance of untrained and under-trained teachers in the classroom in developing countries

Parametric BIM-based Design Review

This research addressed the need for a new design review technology and method to express the tangible and intangible qualities of architectural experience of parametric BIM-based design projects. The research produced an innovative presentation tool by which parametric design is presented systematically. Focus groups provided assessments of the tool to reveal the usefulness of a parametric BIM-based design review method. The way in which we visualize architecture affects the way we design and perceive architectural form and performance. Contemporary architectural forms and systems are very complex, yet most architects who use Building Information Modeling (BIM) and generative design methods still embrace the two-dimensional 15th-century Albertian representational methods to express and review design projects. However, architecture cannot be fully perceived through a set of drawings that mediate our perception and evaluation of the built environment. The systematic and conventional approach of traditional architectural representation, in paper-based and slide-based design reviews, is not able to visualize phenomenal experience nor the inherent variation and versioning of parametric models. Pre-recorded walk-throughs with high quality rendering and imaging have been in use for decades, but high verisimilitude interactive walk-throughs are not commonly used in architectural presentations. The new generations of parametric and BIM systems allow for the quick production of variations in design by varying design parameters and their relationships. However, there is a lack of tools capable of conducting design reviews that engage the advantages of parametric and BIM design projects. Given the multitude of possibilities of in-game interface design, game-engines provide an opportunity for the creation of an interactive, parametric, and performance-oriented experience of architectural projects with multi-design options. This research has produced a concept for a dynamic presentation and review tool and method intended to meet the needs of parametric design, performance-based evaluation, and optimization of multi-objective design options. The concept is illustrated and tested using a prototype (Parametric Design Review, or PDR) based upon an interactive gaming environment equipped with a novel user interface that simultaneously engages the parametric framework, object parameters, multi-objective optimized design options and their performances with diagrammatic, perspectival, and orthographic representations. The prototype was presented to representative users in multiple focus group sessions. Focus group discussion data reveal that the proposed PDR interface was perceived to be useful if used for design reviews in both academic and professional practice settings

Towards the measurement of the perception of the responsibilities of the primary school deputy principal

The deputy principalship remains one of the least understood roles in the schools of contemporary education systems. Research which contributes to theory building about the deputy principalship has been hampered by the lack of survey instruments with known psychometric properties. This paper reports an exploratory study which uses latent trait theory to construct a variable which describes and conceptualises practitioner perspectives of the deputy principalship in the self managing school. The logic of constructing the variable is explained in terms of the requirements of the measurement model employed. A sample of 403 deputy principals, 179 principals and 138 teachers in government primary schools in Western Australia provided data for analysing the actual and ideal perceptions of these practitioners in terms of the variable as conceptualised. In this way, the variable provided the knowledge base for describing the \u27professional horizon\u27 of school practitioners with respect to the traditional and emergent facets of the deputy principalship. The outcomes of the analysis are considered for further research about mapping the responsibilities of the deputy principal in a changing environment

Why do countries join international literacy assessments? an actor-network theory analysis with case studies from Lao PDR and Mongolia.

International assessments are a growing educational phenomenon around the world, increasingly picking up in lower and middle income countries and entering the space of global educational governance (Fenwick et al. 2014). Following the success of the OECD’s first international assessments, the UNESCO Institute for Statistics (UIS) set out in 2003 to develop the Literacy Assessment and Monitoring Programme (LAMP) to measure adult literacy levels across lower and middle income countries in a context-sensitive way. As international organizations rationalize international assessments as essential tools for policy (Rizvi and Lingard 2010) target lower and middle income countries, researching the rationales behind these countries’ participation becomes an urgent area of investigation. In this thesis I enquire into what drives lower-middle income countries to join international assessment programmes through case studies of LAMP in the Lao PDR and Mongolia. Setting my research in the emerging field I define as International Assessment Studies, I argue that Lao PDR and Mongolia join international assessments for reasons that go beyond the need to inform policy (as stated by the UIS and the OECD) and to access foreign aid (Lockheed 2013). Different, and often contradictory interests are being played out through heterogeneous alliances (Latour 1996) which include human and non-human actors (including standardized testing instruments). Through the application of Actor-Network Theory, the data generated in my fieldwork suggests countries are joining the recent phenomenon of international assessments as a global ritual of belonging, comparing the gap with reference societies, and ‘scandalizing’ and ‘glorifying’ (i.e. statistically eliminating problems) with international data. The thesis suggests that understandings of governmentality need to be revised in light of the international and comparative character of educational governance. My findings have implications for understanding the politics of reception of international assessments, but also for the upcoming Programme for International Student Assessment (PISA) for Development which the OECD is in the process of developing – in a similar manner to LAMP – for lower and middle income countries