Broadfoot, Patricia

Oldfield, Alison

Sutherland, Rosamund

Timmis, Sue

English

Explore Bristol Research

Timmis, S., Broadfoot, P., Sutherland, R., & Oldfield, A. (2014).Learning analytics and technology enhanced assessment (TEA).University of Bristol.Publisher's PDF, also known as Version of recordLink to publication record on the Bristol Research PortalPDF-documentUniversity of Bristol – Bristol Research PortalGeneral rightsThis document is made available in accordance with publisher policies. Please cite only thepublished version using the reference above. Full terms of use are available:http://www.bristol.ac.uk/red/research-policy/pure/user-guides/brp-terms/Learning analytics and technology enhanced assessment (TEA)Rethinking  AssessmentDiscussion paper 4:Learning analytics has been defined as the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments  in which it occurs.This paper considers the following: What is learning analytics The potential of learning analytics The challenges of learning analytics The risks of learning analyticsKey recommendations Learning analytics should support students’ agency and development of positive identities rather than predict and determine them. Learning analytics data could have a negative impact on a student’s choices and identity when their achievements and characteristics are reflected back to them, predicted and visualised.  Educational institutions should be supported to develop the capacity to transform data analysis into valuable and empowering interventions. Learning analytics is not in itself the goal but could provide a basis for decision making for effective action. It should be recognised that the effectiveness of learning analytics depends on increased communication and knowledge sharing among commercial developers, researchers and practitioners. What is ‘Learning Analytics’?The current collection and analysis of large, complex data sets in education is both unprecedented and becoming more and more prevalent. Huge amounts of personal and academic data about individuals and institutions are being captured and used to support educational decision-making and denote achievement, which is increasingly data-driven and measured by standardised test data.1 The analysis and use of large datasets in education and other sectors is often assumed to facilitate better judgments, predict outcomes, and support preventative intervention. However, these possibilities of deeper understanding and better decision-making are tempered by a more complicated reality. Such datasets are often complex, subjective, difficult to understand and frequently overwhelm the capacity of individuals and organisations to use them effectively and sensibly. Indeed, there are significant challenges in how to manage, protect, interpret and purposefully use such educational data to support learning. The field of ‘learning analytics’ addresses these challenges.‘Learning analytics’ has been defined as: The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.2 The emerging field is seen to be one of the fastest growing areas of research related to education and technology. Its purpose revolves around managing and using data patterns to make recommendations for improving learning. To achieve this, learning analytics interrogates learner-based data interaction (using techniques such as predictive modelling, user profiling, adaptive learning, and social network analysis) to inform or prompt actions or decisions based on the results. Such decisions could include identifying the need for intervention, personalising support or promoting self-regulation of learning.3 The field is multi-layered and its impact can be seen to affect macro (government or international), meso (institution) and micro (individual learners) levels of action and intervention. Emerging sub-categories within the field provide feedback to different audiences, and some are listed here:  Action or nudge analytics – supports change by nudging or prompting people or institutions to take action. Social network analysis and social learning analytics – investigate social networks, their actors and their relationships, as well as focusing on learning that happens in networks such as participatory online cultures.  Disposition analytics – examines learner characteristics that can improve engagement and support for lifelong learning and future challenges. Disposition analytics relates to the dimensions of ‘learning power’ and the online self-reporting questionnaire called ELLI (Effective Lifelong Learning Inventory), which analyses and visually represents learning power profiles.4  The potential of learning analyticsThe field’s potential to support effective learning is being increasingly recognised. The capture of data trails to fine levels of granularity can illuminate learning activity and lead to recommendations that help teachers and learners assess learning activity.5 Intelligent recognition of verified data patterns also supports predictive modelling, which analyses learner-centred data in order to probabilistically predict future outcomes. These predictions can generate feedback suggesting adaptations, improvements or recommendations for learners, teachers, or wider institutions. These can include prompting a learner to self-assess their progress or informing teachers about students at risk of failing due to poor attendance or late submission of assignments. Learning analytics also supports assessment methods that can increase agency for the learners, such as the use of ‘learner-facing’ tools that represent data back to learners and support self-regulation of learning.6 The use of visual data analysis or visualisation (the graphical representation of multi-faceted complex data sets) demonstrates how such data can become ‘actionable’. For example, ‘dashboard’ systems allow students to monitor their own academic or behavioural activity and access relevant strategies and support. Teachers can also use ‘dashboards’ to compare student or class progress to predictive models of previous classes or performance.The challenges of learning analyticsThe enthusiasm surrounding this emergent field is tempered by debates about its purpose, impact, validity and ethics. At the heart of debates in the field are core questions about the purpose of learning analytics and its relationship to learner agency and control. A key challenge is the appropriate collection, protection and use of large data sets. There is a wide range of predictors and indicators that can be captured for learning analytics purposes, leading to significant debate within the learning analytics field on what data should be collected and analysed – and for what purpose.3 Deciding what information is and is not desirable to measure underpins the development of learning analytics values, tools and practices. Issues of data protection, ownership and privacy are also entirely relevant to this field and can cause learners and teachers to question both the immediate security of their data and who may eventually access information about their knowledge or competencies.1 These issues are gaining interest and call for greater transparency within the sector, as unease about learner rights and data ownership could impede enthusiasm and development of the field.5 Given the highly emotive and sensitive nature of providing feedback, the use of information to predict and evaluate students’ performance and trajectories also raises questions around ethical responsibilities: How do we ensure predictive models are valid and reliable? Does ‘predictive analytics’ simply inform students of who they are or allow them to form their own learning identities? Does the practice trade the potential aspirations of students for self-fulfilling and demotivating predictions of learning outcomes? These questions are especially pertinent when considering research that shows how ‘culturally transmitted beliefs about “the fixity of ability” have a direct, deleterious effect on students’ willingness to persist intelligently in the face of difficulty’.8Finally, the analysis of educational data must align with a pedagogically-based plan for effective and relevant action resulting from that analysis.4 An institution’s capacity to maintain a learning analytics system and design effective interventions is critical. An essential element of this is data literacy, or the ability to make sense of and effectively use data. While tools like ‘dashboards’ can offer learners a sense of control, users need to know what ‘good’ looks like in order to use these tools effectively.6 In this sense, human judgment and ability to derive meaning from the analysis of complex data plays an essential role in making learning analytics effective and relevant.1 Learning analytics can focus too strongly on the technological specifics of collecting and analysing information, rather than reflecting on how such data can and should be used. Furthermore, the pedagogical models used in learning analytics tend to mirror current educational priorities – such as mastering the curriculum and passing the course – and this emphasis ignores the complexity of learners’ relationships to education and risks alienating learners who are not disposed to learn.4 Possible risksLearning analytics offers exciting possibilities for education and assessment, particularly in the areas of personalising learning, empowering learners, improving data literacy and visualising feedback. However, the excitement is moderated by a number of challenges that need to be addressed by the educational community. These centre around issues of agency, purpose and the values behind learning analytics tools and practices. Data collection and the tools that measure it are not neutral and instead reflect what we consider to be ‘good’ education, pedagogy, and assessment practices, which are all fundamental issues at the heart of learning analytics development.9 Given the swiftness of learning analytics’ emergence and its expansion from higher education to the schools market, the field is attracting powerful commercial vendors who are developing data mining and analytics tools. Investment by large companies is needed in order that pedagogical processes can be developed, aimed at improving the process of learning rather than simply becoming another means of measuring performance. Ultimately, analysing data to improve learning requires tools that align with pedagogical principles alongside sound data analysis skills, ethical considerations and institutional capacities that can ensure value-driven and valuable interventions.1 Bienkowski, M., Feng, M., and Means, B. (2012) Enhancing Teaching and Learning through Educational Data Mining and Learning Analytics: An Issue Brief. US Department of Education, Office of Education Technology. 2 Ferguson, R. (2012) The State Of Learning Analytics in 2012: A Review and Future Challenges. Technical Report KMI-12-01, Knowledge Media Institute, The Open University. kmi.open.ac.uk/publications 3 Brown, M. (2012) ‘Learning analytics: Moving from  concept to practice,’ EDUCAUSE Learning Initiative brief. July 2012.4 Buckingham Shum, S. and Deakin Crick, R. (2012) ‘Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics’ Learning Analytics and Knowledge Conference (LAK’12), 29 April – 2 May 2012, Vancouver, BC, Canada.5 Siemens, G. and Long, P. (2011) Penetrating the fog: Analytics in learning and education. EDUCAUSE, 46 (5). educause.edu6 Siemens, G. (2012) ‘Learning Analytics: Envisioning a research discipline and a domain of practice’. LAK12: 2nd International Conference on Learning Analytics & Knowledge, 29 April – 2 May 2012, Vancouver.7 Bull, S., Quigley, S. and Mabbott, A. (2006) ‘Computer-based formative assessment to promote reflection and learner autonomy’. Engineering Education, 1 (1), pp. 8-18.8 JDweck, C. S. (1999). Self-Theories: Their role in motivation, personality, and development. Philadelphia, PA: The Psychology Press.9 Buckingham Shum, S. (2012) ‘Our Learning Analytics Are Our Pedagogy,’ Keynote Presentation at Expanding Horizons 2012, Macquarie University. Available from: slideshare.net/sbsDeciding what data should be collected and what is not desirable to measure, underpins the development of learning analytics tools and practicesLearning analytics needs to shift from a technology-focused orientation to one that uses data to take informed action on learning ISHAN LEWISGRACEEMILYCHLOEZARAFREYACHRISRYANJACKLIAMESMEAJAYZOECase study: Course SignalsCourse Signals (developed at Purdue University) is an example of a system which uses learning analytics to provide real-time feedback to students. Data is mined from multiple sources and statistical techniques are used to predict which students are at risk of failure. The system uses grades, demographic characteristics, past academic history and students’ effort (measured by their on-line activity within a learning management system). Based on the results of the predictive algorithms, tutors then develop an intervention programme that could include both on-line and face-to-face support. Arnold, K. E. & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase student success. Proceedings of the 2nd International Conference on Learning Analytics & Knowledge. New York: ACM.Rethinking Assessment2012/2013 Series of discussion papers Assessment is universally recognised as one of the most important – and powerful – elements of an educational experience. It is also seen as one of the hardest to reform. However, there is an increasingly accepted need for rethinking assessment if it is to keep up with current theoretical, cultural and technological developments affecting teaching and learning. Digital technologies open up new possibilities for more personalised, immediate and engaging assessment experiences. However, the use of digital technologies for assessment (referred to as ‘technology-enhanced assessment’) has yet to be ‘transformative’, with current practices either replicating traditional assessment methods or manifesting in pockets of innovation that are not widespread. How the potential of digital technologies can best support improved assessment practices and preferred educational outcomes is becoming an issue of increasing importance. An acknowledgement of the potential that digital technologies offer should recognise the complexity of the task, the many factors affecting successful educational change, and the significant ethical questions raised by the use of digital technologies in assessment. This series of discussion papers draw on a substantial review of literature which aimed to identify the different ways in which technology currently impacts on educational assessment practices and how it could contribute to a new vision for assessment.  The review of literature is available at:  bristol.ac.uk/education/research/sites/teaThe following discussion papers have been produced in order to highlight key issues  and questions identified by the review of literature:Paper 1: Transforming education through technology enhanced assessmentPaper 2: Integrating the formative and summative through technology  enhanced assessmentPaper 3: Exploiting the collaborative potential of technology enhanced  assessment in Higher EducationPaper 4: Learning analytics and technology enhanced assessmentPaper 5: Ethical issues in technology enhanced assessmentPaper 6: National standards and technology enhanced assessmentSeries authors:Patricia Broadfoot, Sue Timmis, Sarah Payton, Alison Oldfield, Rosamund SutherlandFor further information contact Sue Timmis at Sue.timmis@bristol.ac.ukAcknowledgements: We would like to thank Ruth Deakin Crick for her contribution to this paper.bristol.ac.uk/education4. Learning analytics and technology enhanced assessment (TEA)Graduate School of EducationUniversity of BristolHelen Wodehouse Building35 Berkeley SquareBristol BS8 1JATel: +44 (0)117 928 9000STELLARDESIGN: CW-DESIGN.CO.UKSupported by:

Learning analytics and technology enhanced assessment (TEA)

https://research-information.bris.ac.uk/ws/files/135276434/learninganalytics.pdf

Learning analytics and technology enhanced assessment (TEA)

Abstract

Similar works

Full text

Available Versions

Explore Bristol Research