Daouk, L.

McDowall, Almuth

Rust, J.

Birkbeck Institutional Research Online

John Rust, Almuth McDowall and Lina Daouk Navigating the test maze with confidence – a users’ guide The Psychometrics Centre was invited to deliver a workshop on test choice at the Test User Conference in May. Professor John Rust conducted a Needs Analysis before the event by distributing a brief e-questionnaire to ask what people wanted to see covered: a) Ability or personality? b) Development or assessment, or both? c) Applications in particular contexts: e.g. mapping onto competencies, assessment of teams? d) Assessment of particular groups: executives, young graduates? The delegates’ answers give a good overview of current concerns. The link between personality and development definitely is the flavour of the month. This is not surprising, given that recruitment in the UK is relatively slow, with key industries shedding staff as we write. Judging by the number of senior managers the UK buys in from other countries (remember the head of the Royal School of Nursing was recruited from overseas, causing an outcry here), the UK still has way to go in terms of effective development.  People also wanted to know more about how to inform themselves about different tests. Unlike Level A, which allows users to use any ability test, Level B is tied to a specific instrument. This can be problematic, as the instrument you are trained may not be the best for any given purpose. We were very clear that this workshop should give guidelines for process (read as: ‘how to find’) rather than judgemental (‘this test is better than that one’). Our model for informed test choice is: Draw up context specificationDecision for psychometricsSeek informationMake the right choiceCandidates?What for:Selection?Development?What to assess:Personality?Ability?Budget?Resources? Figure 1: A Model for Test Choice  The base decision is: “ Should we use psychometrics at all:”: giving people a questionnaire or test may not suit all situations. In a coaching or counselling situation certain information may be needed to establish rapport that can best be gained via a structured interview rather than a questionnaire.  Next, a context specification needs to be drawn up: a) What do I want to assess: personality (e.g. for team fit) or ability (still one of the best predictors of job performance)? b) Who are the candidates: young graduates, senior executives? c) What is the budget per candidate; are there suitably qualified people in the organisation to administer and interpret the tests, or do we need to buy consultancy time? d) The purpose: what will the information be used for – selection as in recruitment or promotion, or for solely developmental purposes?  All of these factors will have a bearing on test choice. To illustrate this point: delegates wanted more information on assessing young graduates for selection, as opposed to a more experienced candidate pool . We discussed the difficulties of assessing inexperienced candidates with work-based questionnaires (which can generate a ‘flat profile’) and the necessity to supplement this with skilled interviews and/ or application forms that assess relevant attributes with future-focused questions.  Next, we spend some time on a quick users’ guide on available sources of information. Test publishers’ web-sites provide good introductions to many instruments, and a web search on any tools can be a fruitful source of information. There are also specific on-line resources: the psych testing web-site run by the BPS (www.psychtesting.org.uk) which has reviews for a growing number of work-based tests, and the web-site run by the association of Business Test Publishers that allows searches for instruments (http://www.btpa.org/testing/testsearch.asp). The Mental measurement yearbook is now also available on-line (http://buros.unl.edu/buros/jsp/search.jsp). Reviews are available for a fee across all areas of testing: occupational, educational and clinical. Manuals are the most comprehensive source of detailed information on specific tests, but can usually not be accessed unless the user is qualified in the instrument. This is where a network of colleagues comes in handy: “what would/ do you use, and does it work?” We stressed that delegates should not rely on any one information source in isolation, but to combine information from as many informed sources as possible.  But how do you make the right choice? In order to get delegates thinking, we asked them to compare a ‘big five’ personality questionnaire, a ‘type’ instrument, an ability test, an integrity test and a 360 degree feedback instrument and to tell us the commonalities and  differences. The latter were easy: all these tests are based on different theoretical frameworks, and are used in different contexts. The similarities were not so obvious at first glance, but are absolutely crucial. Any such assessment should be guided by the four psychometric principles (for a full review see Rust & Golombok, 2000). It should be: a) reliable, across situations and people b) valid: measuring what it says on the tin c) free from bias: producing the same sorts of results for different groups d) standardised: administered and scored in a way that minimises unwanted error.  To conclude the session, we debated some of the current ‘hot topics’ that came out of the needs analysis. First, we explored the value of competencies and mapping tests against them. It can be a problem with such frameworks that labels are often solely descriptive. Take ‘effective communication’, for instance.  We used the example of a consultant in the Psychometrics Centre. Communication skills are crucial to our job, as we need them both on the academic side (teaching and research) and for consultancy (with our clients). However, the skills are very different in each context. Effective communication with students needs to be directive, as otherwise we’d never get the assignments in on time! Effective communication in an academic context requires a very different vocabulary that assumes much prior knowledge, and also has to be sensitive to other academics’ point of view. With our clients, external communication comprises an in-depth understanding of their organizational context and commercial requirements. Saying “but research says..” will not convince an HR manager to change his or her recruitment practices. Thus, it is often better to break down competencies to an actual behavioural level for a specific context, and then think about how best these behaviours could be assessed.  Another hot topic was assessing teams. There are instruments that operate specifically at the team level, but many other work-based questionnaires can also generate customised team-reports. Crucially, the effectiveness of using such tools for teams depends very much on the skill of the feedback giver as group dynamics introduce a different element to any feedback session and it takes skill and practice to be able to affect lasting change.  To conclude, we echoed what was said in one of the keynote addresses. Level A and B are a great stepping stone for qualified test use, but there is a great need for practitioners to learn about the applications of tests and questionnaires once back in the real world. We hope that our workshop helped to bridge this gap. Test choice is not rocket science, but it requires a structured and rigorous approach that should be deeply rooted in the application of the four psychometric principles.  Key reading Rust, J. & Golombok, S. (2000). Modern Psychometrics: Science of Psychological Assessment. Routledge  Web-sources: www.psychtesting.org.uk  http://buros.unl.edu/buros/jsp/search.jsp http://www.btpa.org/testing/testsearch.asp www.psychometricscentre.co.uk  If you would like to get in touch with the Psychometrics Centre, please telephone Rosaria de Santis on 020 70404502, m.de-santis@city.ac.uk. 

Navigating the test maze with confidence

Abstract

Similar works

Full text

Available Versions

Birkbeck Institutional Research Online

Name not available