21st Century…19th Century…6th Century BC Skills

During July I spent some time in Ireland; in the beautiful, south-west corner that is Kerry. Whilst there, I was reading a bit of the local history and came across some quite mind-boggling technology. In a gently shelving sandy cove by Waterville lies the remains of an astonishing leap of faith by a group of Victorian entrepreneurs. Sticking out of the cliff is the tattered end of a telegraph cable that ran across 2000 miles of ocean to Newfoundland and thereafter overland to New York. The south-west tip of Kerry (Valentia island) was chosen (in the 1850s) as the take-off point and ‘Hearts Content’ bay in Newfoundland the receiving point. But in between lay 2000 miles of wild ocean that is variously 2-3 miles deep

Tasks in Technology: An analysis of their purposes and effects

Introduction for the 2015 DATA Special Edition This paper was written in 1993 and was published in The International Journal of Technology and Design Education. 1994 Springer 4(3): 241-256. The paper concerns the nature of the tasks that initiate and drive technological activity. It is set in the context of two research projects that we conducted in TERU; the Assessment of Performance Unit project in Design & Technology (1985 to 1991) and the Economic and Social Research Council project “Understanding Technological Approaches” (UTA) (1992-1994). The former was a large scale national survey of performance in schools - involving tests on 10,000 learners in 700 schools, and the latter is a small scale study (80 learners in 20 schools) examining in detail the processes that learners engage in as they tackle technological tasks. However, the wider context of this paper concerns the English and Welsh National Curriculum (NC) implementation programme that had been launched in 1990. It caused a huge storm both in the curriculum generally and in design & technology (d&t) in particular. In the wider curriculum the assessment arrangements surrounding the Standard Assessment Tasks had been so badly designed that in 1992 teachers and schools had boycotted the whole process. And in d&t, the ‘Order’ that defined what teachers should do in the classroom/studio/workshop appeared to make very different demands on teachers than had previously been the case. The Order defined d&t in four ‘Attainment Targets’, the first of which (AT1) was ‘Identifying Needs and Opportunities’. This (at least) implied that learners themselves should be doing that ‘identifying’, and in 1990 that was far from common practice. At exactly this moment we undertook the ESRC: UTA project that enabled us to collect the data that would inform this issue. We followed in detail the tasks that teachers set or negotiated with learners and examined the consequences of these tasks on ’ subsequent actions

Examining the reliability of Adaptive Comparative Judgement (ACJ) as an assessment tool in educational settings

Conventional approaches to assessment involve teachers and examiners judging the quality of learners work by reference to lists of criteria or other ‘outcome’ statements. This paper explores a quite different method of assessment using ‘Adaptive Comparative Judgement’ (ACJ) that was developed within a research project at Goldsmiths University of London between 2004 and 2010. The method was developed into a tool that enabled judges to distinguish better/worse performances not by allocating numbers through mark schemes, but rather by direct, holistic, judgement. The tool was successfully deployed through a series of national and international research and development exercises. But game-changing innovations are never flaw-less first time out (Golley, Jet: Frank Whittle and the Invention of the Jet Engine, Datum Publishing, Liphook Hampshire, 2009; Dyson, Against the odds: an autobiography, Texere Publishing, Knutsford Cheshire, 2001) and a series of careful investigations resulted in a problem being identified within the workings of ACJ (Bramley, Investigating the reliability of Adaptive Comparative Judgment, Cambridge Assessment Research Report, UK, Cambridge, 2015). The issue was with the ‘adaptive’ component of the algorithm that, under certain conditions, appeared to exaggerate the reliability statistic. The problem was ‘worked’ by the software company running ACJ and a solution found. This paper reports the whole sequence of events—from the original innovation, through deployment, the emergent problem, and the resulting solution that was published at an international conference (Rangel Smith and Lynch in: PATT36 International Conference. Research & Practice in Technology Education: Perspectives on Human Capacity and Development, 2018) and subsequently deployed within a modified ACJ algorithm

Gender Differences in Technology Illuminated Through Test Performance (outcome) Data and ‘Realtime’, ‘On-task’ (process) Data

Introduction for the 2015 DATA Special Edition This paper was written in 1994 as an internal TERU paper – it has not previously been published. It draws from two research projects that gathered data on gender differences in performance in technology. As with the Tasks in Technology paper (also included in this Special Edition), the wider context was the early years of the National Curriculum and specifically concerning the Standard Assessment Tasks (SATs). We were aware of the sensitivity of the gender data, essentially that girls seriously outperformed boys and the concomitant concern that the tests themselves might contain implicit bias, so we undertook a systematic review of the data from our two TERU projects that could inform the matter. The first provided ‘outcome’ data from APU tests (15 year olds in 1988 – Kimbell et. al., 1991). The latter, derived from the Understanding Technological Approaches (UTA) project (Kimbell et. al., 1994) allowed us to crosscheck these data with ‘process’ data derived from classroom observations (across all school years from 1-11 in 1992/3 -). I focus on two specific aspects of gender performance that were highlighted in test findings: • concerning ‘active’ and ‘reflective’ response modes to tasks; • concerning design proposals in relation to ‘users’ and for ‘manufacture’

Schools Design Prize 1980

The fourth Schools Design Prize competition, organised by the Design Council and sponsored this year by Rolls Royce Limited, had its prizegiving ceremony at the Royal Institution on Wednesday, 12th November, 1980.The annual Design Prize, established in 1977 under the sponsorship of G.E.C., has provided ample evidence that young people, when challenged to solve a problem or satisfy a need, can produce designs that show remarkable imagination, ingenuity and creative ability. The development of these talents is not only of great educational importance, but also vital to the future well being of British manufacturing industry, and this year one of the major features of the competition has been the close collaboration between the entrants and the sponsors. Mr. Donald Pepper, Vice Chairman of Rolls Royce, in his introductory remarks at the prizegiving described this collaboration in which design engineers from Rolls Royce visited over 100 schools to discuss design projects with entrants and to offer their expertise and guidance on specific problems. The liaison he insisted was as important for industry as it was for the schools, for the long term health of the nation is dependent on fostering the potential that exists in many school design departments

Sharing and securing learners\u27 performance standards across schools

Assessing learners’ performance makes very different demands upon teachers depending on the purpose and the context of the assessment. But common to all assessment is some sense of what ‘quality’ looks like. Most often teachers engage in formative assessments in the classroom, and the familiar standards of the classroom are adequate for this purpose. However if teachers are to undertake external, nationally regulated assessment then some sense of a national standard of quality is required. But there are very limited mechanisms by which teachers can acquire this understanding, so they use their best judgement, and standards vary from school to school not because anyone is attempting to cheat the system but simply because they cannot know what the real national standard is. It is for this reason that regulated examination bodies follow some process such as the following from the State Examinations Commission (SEC) in Ireland. ‘… teacher estimated marks will be subjected to an in-school alignment process and later a national standardisation process’. (SEC, 2021). How much simpler it would all be if teachers had – as a matter of normal practice – access to, and familiarity with, work from a national sample of schools, not just their own classroom. Adaptive Comparative Judgement (ACJ) is an online assessment tool that has been used for some years, principally as a formative tool for learners (e.g. Bartholomew et al., 2018; 2019). This presentation reports on a study of the new ACJ Steady State tool from the same stable. The purpose of the new tool is to solve the problem of variable standards across schools by enabling teachers to make paired judgements of work from multiple schools and thereby evolve and agree standards of performance beyond their own school. The current study is operating in Ireland with a group of schools, a university, and the SEC. The anticipated outcomes include 1) better consistency of performance standards across schools in the research group and 2) greater understanding of and confidence in assessment judgements by the teachers. If ACJ has proved to be a powerful formative assessment tool for learners, ACJ Steady State is designed to be a formative assessment tool for teachers, helping to inform and support their assessment judgements