Misleading ‘quality’ measures in Higher Education : problems from combining diverse indicators that include subjective ratings and academic performance and costs

Quality indicators are often derived from weighted sums of diverse items, including ordinal Likert items. This procedure can be dangerously misleading because it takes no account of correlations among indicators. It also takes no account of whether indicators are input measures, e.g. prior achievement of incoming students, or outcome measures, e.g. proportion getting a good degrees or student satisfaction. UK Higher Education data for 04-05 were analyzed taking these issues into account. Multiple regression showed, unsurprisingly, that ‘bright’ students with high prior achievement did well on all outcome indicators. Getting a good degree was not influenced by any other measure. Completing a course was additionally positively associated with academic pay and spend on library and computing facilities. A good destination (not currently seeking work) was additionally positively associated with number of staff per student and vice-chancellor pay. Student satisfaction was additionally influenced, but negatively, with vice-chancellor pay. The implications for evaluating university quality are discusse

Signal detection theory, the approach of choice : model-based and distribution-free measures and evaluation

New and old methods of analyzing two-choice experiments with confidence ratings are evaluated. These include the theory of signal detectability (TSD), Luce's choice theory, nonparametric techniques based on areas under receiver-operating characteristic (ROC) ftmctions, and methods based on S' and Omega, proposed by Balakrishnan and his colleagues. New methods for assessing the bias of a complete ROC function are proposed, together with an additional area-based measure of response bias. Area measures of both sensitivity and bias proved the most consistent. Response bias for a full ROC function was larger than bias at the cut point and also provided additional information. Participants showed voluntary control of bias for all measures except Omega. Unequal variance versions of TSD and choice models gave similar fits to data, with the choice model closer to an equal variance version. Discrimination data from Balakrishnan (1999) formed the empirical test bed

Quality science from quality measurement: The role of measurement type with respect to replication and effect size magnitude in psychological research

Copyright: © 2018 Kornbrot et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.The quality of psychological studies is currently a major concern. The Many Labs Project (MLP) and the Open-Science-Collaboration (OSC) have collected key data on replicability and statistical effect sizes. We build on this work by investigating the role played by three measurement types: ratings, proportions and unbounded (measures without conceptual upper limits, e.g. time). Both replicability and effect sizes are dependent on the amount of variability due to extraneous factors. We predicted that the role of such extraneous factors might depend on measurement type, and would be greatest for ratings, intermediate for proportions and least for unbounded. Our results support this conjecture. OSC replication rates for unbounded, 43% and proportion 40% combined are reliably higher than those for ratings at 20% (effect size, w = .20). MLP replication rates for the original studies are: pro- portion = .74, ratings = .40 (effect size w = .33). Original effect sizes (Cohen’s d) are highest for: unbounded OSC cognitive = 1.45, OSC social = .90); next for proportions (OSC cogni- tive = 1.01, OSC social = .84, MLP = .82); and lowest for ratings (OSC social = .64, MLP = .31). These findings are of key importance to scientific methodology and design, even if the reasons for their occurrence are still at the level of conjecture.Peer reviewe

How was it for you? Psychophysics and the evaluation of student experience of e-learning

Evaluating the student experience of Higher Education has become a matter of national importance in several countries. For example, in England & Wales the National Student Survey (NSS) is administered on line to all students in the final year of their undergraduate degree. The NSS uses 5-point Likert scales, giving extent of agreement or disagreement with positive statements. There are 22 questions, covering 6 aspects of student experience. This presentation considers how psychophysical methods based on signal detection theory or Luceâ s choice theory can be used to analyze such data. Such methods can determine how well the questions discriminate different aspects of experience, as well as how favourably the students experience these aspects of their education. Particular emphasis is given to exploring discipline differences together with the effects of recent technologies, such as managed learning environments and web 2.0 social software

In the glimpse of an eye : decision making and vision

Rapid visual search, depending on iconic memory, is a core but controversial psychophysical topic. A key example is the claim by Horowitz & Wolfe (1998a) that ‘visual search has no memory. Their evidence is the effect of increasing search set size in a letter identification task. Search time per item was unimpaired when all letters were randomly relocated during the search. This paper presents additional analyse showing strong deleterious effects of randomly relocating letters, on error rates, and on total reaction time. Thus visual search does have a memory. A psychophysical information accrual model is presented to account for these data and other key studies on visual search. The model includes decision criteria as well as sensory parameters. Criterion adjustments, which depend on numbers of distractors, predict the lower mean search times and the lower error, rates observed for non-random presentations

The dog that didn't bark...interpreting non-significance

Hypothesis testing is a crucial component of science. This leads to guidelines (often ignored) in most disciplines including psychology. Unfortunately, most focus on significant effects. Non-significant effects are sidelined, in spite of their importance to scientific progress. This study reports a survey of practicing scientists on how they would report and interpret explicit scenarios with non-significant effects. There was no consensus on interpretation in terms of predicting future results. Respondants agreed about how to report the significance of a hypothesis test. Most chose not to report any descriptive statistics, or the sample size, or anything about power, or sufficient information to enable replication. These results shed light on statistical thinking and so should enable more useable guidelines. For non-significant effects, the importance of a priori power is emphasised

Impaired context maintenance in mild to moderately depressed students

Original article can be found at : http://www.informaworld.com/ Copyright Taylor & FrancisWe test the hypothesis that people with depression experience difficulties in maintaining task-relevant context information over longer periods of time using the AX version of the continuous performance task (AX-CPT). The AX-CPT requires that participants maintain a context cue (A) in an active state in order to respond correctly to a target cue (X) presented after a short delay. A total of 40 nondepressed and mild to moderately depressed students completed versions of the task with short (1-s) or long (10-s) interstimulus intervals (ISIs). Mildly depressed participants made significantly more context-dependent (BX) errors, unlike controls who made more errors on trials where good context processing would impair performance (AY). This pattern of errors was only evident in the long ISI condition, suggesting poor maintenance of contextual information.Peer reviewe

Raving about Ravens : modelling speed-accuracy in intelligence tests

The effect of time pressure on performance on intelligence tests is a long standing problem. In this study a computerised version of the Ravens Advanced Progressive Matrices was administered using 3 different forms of instructions: control, speed pressure, and accuracy pressure. Analyses used Rasch measures of participant ability and item difficulty, and the time each participant took to solve each problem. Raw scores were, surprisingly, more useful than Rasch measures. The time pressure group were faster but scored less well than the other two groups. Raw score had a small but significant correlation with total test time. Brighter participants took less time for easy items, but more time for hard items, which were both slower and more variable than easier items. Mean and SD were more consistent for total time than for either correct or error time. Effective models will need to incorporate these diverse result

New generation learners - does the learning environment match student expectations

Who exactly are our students and what characteristics do they have in terms of their interest in using technology for learning The study presented in brief below developed from a desire to know how far the--new generation of learners entering Higher Education had experienced technology and what their expectations might be for using technology in--their tertiary learning

Use of a haptic device by blind and sighted people : perception of virtual textures and objects

This paper describes a series of studies involving a haptic device which can display virtual textures and 3-D objects. The device has potential for simulating real world objects and assisting in the navigation of virtual environments (VEs). Three experiments investigated: (a) whether previous results from experiments using real textures could be replicated using virtual textures; (b) whether participants perceived virtual objects to have the intended size and angle; and (c) whether simulated real objects could be recognised. In all the experiments differences in perception by blind and sighted people were also explored. The results have implications for the future design of VEs in that it cannot be assumed that virtual textures and objects will feel to the user as the designer intends. A set of guidelines for the design of haptic interfaces and VEs are presented