Thurstonian Scaling of Compositional Questionnaire Data

To prevent response biases, personality questionnaires may use comparative response formats. These include forced choice, where respondents choose among a number of items, and quantitative comparisons, where respondents indicate the extent to which items are preferred to each other. The present article extends Thurstonian modeling of binary choice data (Brown & Maydeu-Olivares, 2011a) to “proportion-of-total” (compositional) formats. Following Aitchison (1982), compositional item data are transformed into log-ratios, conceptualized as differences of latent item utilities. The mean and covariance structure of the log-ratios is modelled using Confirmatory Factor Analysis (CFA), where the item utilities are first-order factors, and personal attributes measured by a questionnaire are second-order factors. A simulation study with two sample sizes, N=300 and N=1000, shows that the method provides very good recovery of true parameters and near-nominal rejection rates. The approach is illustrated with empirical data from N=317 students, comparing model parameters obtained with compositional and Likert scale versions of a Big Five measure. The results show that the proposed model successfully captures the latent structures and person scores on the measured traits

Estimation and optimal designing under latent variable models for paired comparisons studies via a multiplicative algorithm

We consider:<BR/> 1. The problem of estimating the parameters of latent variable models such as the Bradley Terry or Thurstone Model by the method of maximum likelihood, given data from a paired comparisons experiment. The parameters of these models can be taken to be weights which are positive and sum to one;<BR/> 2. The problem of determining approximate locally optimal designs for good estimation of these parameters; i.e of determining optimal design weights which are also positive and sum to one

Defining the gap between research and practice in public relations programme evaluation - towards a new research agenda

The current situation in public relations programme evaluation is neatly summarized by McCoy who commented that 'probably the most common buzzwords in public relations in the last ten years have been evaluation and accountability' (McCoy 2005, 3). This paper examines the academic and practitioner-based literature and research on programme evaluation and it detects different priorities and approaches that may partly explain why the debate on acceptable and agreed evaluation methods continues. It analyses those differences and proposes a research agenda to bridge the gap and move the debate forward

Introducing a Pairwise Comparison Scale for UX Evaluations with Preschoolers

This paper describes the development and validation of a pairwise comparison scale for user experience (UX) evaluations with preschoolers. More particularly, the dimensionality, reliability and validity of the scale are discussed. The results of three experiments among almost 170 preschoolers show that user experience cannot be measured quantitatively as a multi-dimensional construct. In contrast, preschoolers’ UX should be measured directly as a one-dimensional higher order construct. This one-dimensional scale encompassing five general items proved to be internally consistent and valid providing evidence of a solid theory-based instrument to measure UX with preschoolers.status: publishe

On ordinal utility, cardinal utility, and random utility

Though the Random Utility Model (RUM) was conceived  entirely in terms of ordinal utility, the apparatus throughwhich it is widely practised exhibits properties of  cardinal utility.  The adoption of cardinal utility as a  working operation of ordinal is perfectly valid, provided  interpretations drawn from that operation remain faithful  to ordinal utility.  The paper considers whether the latterrequirement holds true for several measurements commonly  derived from RUM.  In particular it is found that  measurements of consumer surplus change may depart from  ordinal utility, and exploit the cardinality inherent in  the practical apparatus.

Adaptive Comparative Judgement: A Tool to Support Students’ Assessment Literacy

Comparative judgment in assessment is a process whereby repeated comparison of two items (e.g., assessment answers) can allow an accurate ranking of all the submissions to be achieved. In adaptive comparative judgment (ACJ), technology is used to automate the process and present pairs of pieces of work over iterative cycles. An online ACJ system was used to present students with work prepared by a previous cohort at the same stage of their studies. Objective marks given to the work by experienced faculty were compared to the rankings given to the work by a cohort of veterinary students (n=154). Each student was required to review and judge 20 answers provided by the previous cohort to a free-text short answer question. The time that students spent on the judgment tasks was recorded, and students were asked to reflect on their experiences after engaging with the task. There was a strong positive correlation between student ranking and faculty marking. A weak positive correlation was found between the time students spent on the judgments and their performance on the part of their own examination that contained questions in the same format. Slightly less than half of the students agreed that the exercise was a good use of their time, but 78% agreed that they had learned from the process. Qualitative data highlighted different levels of benefit from the simplest aspect of learning more about the topic to an appreciation of the more generic lessons to be learned

Visuo-spatial ability in colonoscopy simulator training

Visuo-spatial ability is associated with a quality of performance in a variety of surgical and medical skills. However, visuo-spatial ability is typically assessed using Visualization tests only, which led to an incomplete understanding of the involvement of visuo-spatial ability in these skills. To remedy this situation, the current study investigated the role of a broad range of visuo-spatial factors in colonoscopy simulator training. Fifteen medical trainees (no clinical experience in colonoscopy) participated in two psycho-metric test sessions to assess four visuo-spatial ability factors. Next, participants trained flexible endoscope manipulation, and navigation to the cecum on the GI Mentor II simulator, for four sessions within 1 week. Visualization, and to a lesser degree Spatial relations were the only visuo-spatial ability factors to correlate with colonoscopy simulator performance. Visualization additionally covaried with learning rate for time on task on both simulator tasks. High Visualization ability indicated faster exercise completion. Similar to other endoscopic procedures, performance in colonoscopy is positively associated with Visualization, a visuo-spatial ability factor characterized by the ability to mentally manipulate complex visuo-spatial stimuli. The complexity of the visuo-spatial mental transformations required to successfully perform colonoscopy is likely responsible for the challenging nature of this technique, and should inform training- and assessment design. Long term training studies, as well as studies investigating the nature of visuo-spatial complexity in this domain are needed to better understand the role of visuo-spatial ability in colonoscopy, and other endoscopic techniques