Putting theory oriented evaluation into practice

Evaluations of gaming simulations and business games as teaching devices are typically end-state driven. This emphasis fails to detect how the simulation being evaluated does or does not bring about its desired consequences. This paper advances the use of a logic model approach which possesses a holistic perspective that aims at including all elements associated with the situation created by a game. The use of the logic model approach is illustrated as applied to Simgame, a board game created for secondary school level business education in six European Union countries

The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations

Background: Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM.Methods: a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9.Results: The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a smaller SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2.Conclusions: An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use

Combining decision trees and stochastic curtailment for assessment length reduction of test batteries used for classification.

For classification problems in psychology (e.g., clinical diagnosis), batteries of tests are often administered. However, not every test or item may be necessary for accurate classification. In the current article, a combination of classification and regression trees (CART) and stochastic curtailment (SC) is introduced to reduce assessment length of questionnaire batteries. First, the CART algorithm provides relevant subscales and cutoffs needed for accurate classification, in the form of a decision tree. Second, for every subscale and cutoff appearing in the decision tree, SC reduces the number of items needed for accurate classification. This procedure is illustrated by post hoc simulation on a data set of 3,579 patients, to whom the Mood and Anxiety Symptoms Questionnaire (MASQ) was administered. Subscales of the MASQ are used for predicting diagnoses of depression. Results show that CART-SC provided an assessment length reduction of 56%, without loss of accuracy, compared with the more traditional prediction method of performing linear discriminant analysis on subscale scores. CART-SC appears to be an efficient and accurate algorithm for shortening test batteries. © The Author(s) 2013

Perfectionism and achievement goals in young Finnish ice-hockey players aspiring to make the Under-16 national team

Research on perfectionism suggests that is it useful to differentiate between perfectionistic strivings and perfectionistic concerns. Regarding the 2 x 2 achievement goal framework, the usefulness of this differentiation was recently demonstrated in a study with university student athletes (Stoeber, Stoll, Pescheck, & Otto, 2008, Study 2), in which it was found that perfectionistic strivings were associated with mastery-approach and performance-approach goals and perfectionistic concerns with mastery-avoidance, performance-approach, and performance-avoidance goals. Because the study was largely exploratory and only used non-elite athletes, the aim of the present research was to replicate and extend these findings by investigating a sample of 138 young, elite ice-hockey players, while adding further measures of perfectionism and using structural equation modelling (SEM) to confirm the relationships between perfectionistic strivings, perfectionistic concerns,and the 2 x 2 achievement goals. The SEM results showed that, in elite athletes also, perfectionistic strivings are associated with mastery-approach and performance-approach goals, whereas perfectionistic concerns are associated with masteryavoidance, performance-approach, and performance-avoidance goals. Our findings corroborate the importance of differentiating between perfectionistic strivings and perfectionistic concerns when studying perfectionism in sports, because only perfectionistic concerns (and not perfectionistic strivings) are associated with maladaptive patterns of achievement goals

Daily life stress and the cortisol awakening response : testing the anticipation hypothesis

Acknowledgments We thank Paul Stewart for his contribution to data collection and Dr Matthew Jones for programming the handheld computers. Author Contributions Conceived and designed the experiments: WS DJP. Performed the experiments: DJP. Analyzed the data: WS. Wrote the paper: WS DJP.Peer reviewedPublisher PD

Psychometric analysis of the scale for the predisposition to the occurrence of adverse events in nursing care provided in ICUS

OBJECTIVE: to present the result of the validity and reliability studies concerning the Scale for the Predisposition to the Occurrence of Adverse Events (EPEA). METHOD: construct validity was based on Principal Components Analysis. RESULTS: reliability verified through Cronbach's alpha indicated good reliability (structure α=0.80; process α=0.92). CONCLUSION: based on its psychometric indicators, the EPEA can be considered a valid measure to assess the attitudes of nurses in relation to factors that potentially lead to the occurrence of adverse events in ICUs

Evaluation of the health-related quality of life of children in Schistosoma haematobium-endemic communities in Kenya: a cross-sectional study.

BACKGROUND: Schistosomiasis remains a global public health challenge, with 93% of the ~237 million infections occurring in sub-Saharan Africa. Though rarely fatal, its recurring nature makes it a lifetime disorder with significant chronic health burdens. Much of its negative health impact is due to non-specific conditions such as anemia, undernutrition, pain, exercise intolerance, poor school performance, and decreased work capacity. This makes it difficult to estimate the disease burden specific to schistosomiasis using the standard DALY metric. METHODOLOGY/PRINCIPAL FINDINGS: In our study, we used Pediatric Quality of Life Inventory (PedsQL), a modular instrument available for ages 2-18 years, to assess health-related quality of life (HrQoL) among children living in a Schistosoma haematobium-endemic area in coastal Kenya. The PedsQL questionnaires were administered by interview to children aged 5-18 years (and their parents) in five villages spread across three districts. HrQoL (total score) was significantly lower in villages with high prevalence of S. haematobium (-4.0%, p<0.001) and among the lower socioeconomic quartiles (-2.0%, p<0.05). A greater effect was seen in the psychosocial scales as compared to the physical function scale. In moderate prevalence villages, detection of any parasite eggs in the urine was associated with a significant 2.1% (p<0.05) reduction in total score. The PedsQL reliabilities were generally high (Cronbach alphas ≥0.70), floor effects were acceptable, and identification of children from low socioeconomic standing was valid. CONCLUSIONS/SIGNIFICANCE: We conclude that exposure to urogenital schistosomiasis is associated with a 2-4% reduction in HrQoL. Further research is warranted to determine the reproducibility and responsiveness properties of QoL testing in relation to schistosomiasis. We anticipate that a case definition based on more sensitive parasitological diagnosis among younger children will better define the immediate and long-term HrQoL impact of Schistosoma infection

Developing autonomous learning in first year university students using perspectives from positive psychology

Autonomous learning is a commonly occurring learning outcome from university study, and it is argued that students require confidence in their own abilities to achieve this. Using approaches from positive psychology, this study aimed to develop confidence in first‐year university students to facilitate autonomous learning. Psychological character strengths were assessed in 214 students on day one at university. Two weeks later their top three strengths were given to them in study skills modules as part of a psycho‐educational intervention designed to increase their self‐efficacy and self‐esteem. The impact of the intervention was assessed against a control group of 40 students who had not received the intervention. The results suggested that students were more confident after the intervention, and that levels of autonomous learning increased significantly compared to the controls. Character strengths were found to be associated with self‐efficacy, self‐esteem and autonomous learning in ways that were theoretically meaningful