Doing less but getting more: Improving forced-choice measures with IRT.

Multidimensional forced-choice (MFC) questionnaires typically show good validities and are resistant to impression management effects. However, they yield ipsative data, which distorts scale relationships and makes comparisons between people problematic. Depressed reliability estimates also led developers to create tests of potentially excessive length. We apply an IRT Preference Model to make more efficient use of information in existing MFC questionnaires. OPQ32i used for selection and assessment internationally is examined using this approach. The latent scores recovered from a much reduced number of MFC items are superior to the full test?s ipsative scores, and comparable to unbiased normative scores

Test-taker reactions to online completion of the OPQ32i

This study is part of an ongoing programme of research exploring the impact of online administration and, in particular, remote administration. Previous work (Bartram and Brown, 2002) showed that OPQ32i data are not biased in any way either by online administration or by the absence of a local test administrator. The present research considers candidates’ perceptions of completing the ipsative version of the Occupational Personality Questionnaire (OPQ32i) online. The present report describes the results from 54 feedback forms completed by test-takers after they had completed the OPQ32i online. All those taking OPQ32 online for one of SHL’s client organisations, SyBlue, were invited to complete a paper-and-pencil questionnaire after they had responded to the OPQ32. Every candidate received the feedback form and a pre-paid envelope to post it back to SHL. Questionnaires were completed anonymously, and there was no means of relating feedback responses to OPQ32 data. SyBlue is an executive search and recruitment consultancy specialising in middle to senior level appointments. The way in which SyBlue uses the psychometric assessment is unique in the headhunting market place. Whilst in theory other head-hunters could utilise similar objective techniques, on the whole it goes against the industry norm where head-hunters pride themselves on their ability to judge people over the phone and face to face in interview. In fact, research into interviewing suggests that most people are very poor at predicting future job success and those who interview for a living are not necessarily any better than anyone else

Putting the Person into Personality

The research has shown that is it possible to predict type membership with a high degree of accuracy using trait-based measures. For both the prototypical and the categorical cases, OPQ32 scales could be used to model type patterns and predict type membership with above 70% accuracy – far higher if one takes into account that type misclassifications are not random, but tend to be close to the ‘correct’ type. The research has both shown the value of using the Big 5 as a framework for summarising patterns of relationships between types and traits and also its limitations. However, accuracy of classification and nuances of interpretation of type differences are both considerably enhanced when the full 32 scales are considered. There is considerable value in being able to consider a person from both a person-centric view and from a trait-centric view – and to be able to flip between these perspectives. The value of starting from the trait-based measurement rather than a typology, however is that, as illustrated in the present work, it is possible to then consider a person in terms of two quite different typologies: one categorical and the other prototypical. This greatly adds to the richness of the interpretation one can obtain from a personality assessment and also helps to increase our understanding of types

Assessing the quality of tests : revision of the EFPA review model

Background: Diverse national and international organizations have been developing projects for many years to improve testing practices. The main goal of this paper is to present the revised model of the European Federation of Psychologists’ Associations (EFPA) for the evaluation of the quality of tests. This model aims to provide test users with rigorous information about the theoretical, practical and psychometric characteristics of tests, in order to enhance their use. Method: For the revision of the test review model, an EFPA task force was established, consisting of six European experts from different countries, who worked on the update of the previous European model, adapting it to the recent developments in the field of psychological and educational measurement. Results: The updated EFPA model provides for the comprehensive evaluation of tests. The first part describes test characteristics exhaustively, and in the second part, a quantitative and narrative evaluation of the most relevant psychometric characteristics of tests is presented. Conclusions: A revision of the European model for the description and evaluation of psychological and educational tests is presented. The revised model is analyzed in light of recent developments in the field

Trends in testing: highlights of a global survey

B

Let's Focus on Two-Stage Alignment Not Just on Overall Performance

Johnson et al. (2010) note the need for a vector defining the relationships between job components and overall job performance (OJP) in the job requirements matrix approach to synthetic validation. This need is also implicit in the job components validity approach that they discuss. In our view, relationships between job components and overall criteria such as OJP are central to synthetic validation, as the effectiveness of this approach depends on the generalizability of relationships between job and criterion components across jobs, organizations, and (of increasing importance) countries.</jats:p

Lack of consensus among competency ratings of the same occupation: Noise or substance?

Although rating differences among incumbents of the same occupation have traditionally been viewed as error variance in the work analysis domain, such differences might often capture substantive discrepancies in how incumbents approach their work. This study draws from job crafting, creativity, and role theories to uncover situational factors (i.e., occupational activities, context, and complexity) related to differences among competency ratings of the same occupation. The sample consisted of 192 incumbents from 64 occupations. Results showed that 25% of the variance associated with differences in competency ratings of the same occupation was related to the complexity, the context, and primarily the nature of the occupation's work activities. Consensus was highest for occupations involving equipment-related activities and direct contact with the public

The need for international guidelines on standards for test use : a review and international initiatives

Predstavljeni so vsi mednarodni, evropski in nacionalni projekti, povezani s standardi za uporabo testov, delo in raziskava delovne skupine za teste in testiranje v evropskem zduženju psihologov, evropske smernice za proces ocenjevanja, mednarodne smernice za uporabo testov

Big Five validity: Aggregation method matters

Correlations between Big Five personality factors and other variables have been examined in three different ways: direct scoring of items within a factor, application of a composite score formula, and taking the average of single-scale correlations. Those methods were shown to yield consistently different outcomes in four sets of data from sales-people and managers. Factor correlations with job performance were greatest for direct scoring, and were reduced by half when scale correlations were averaged. The insertion of previously suggested estimates into the composite score formula yielded intermediate correlations with performance. It is necessary to interpret summary accounts of correlations with a compound construct in the light of the aggregation method employed