Improving a tutor’s feedback assessment tool: transforming Open Mentor following two recent deployments

Evidence shows the vital role that the quality of feedback plays on students’ performance and on the overall increase of learning opportunities that good feedback creates for students. Based on this evidence, the Open University developed Open Mentor (OM), a system to support tutors enhance their feedback practice. Open Mentor Technology transfer (OMTetra), a JISC funded project, took OM and deployed it in two Higher Education institutions with the purpose of evaluating the process of transferability and continue the development of the tools available to tutors within the system. This paper describes the original OM and the enhancements identified after use and evaluations from tutors of the institutions involved

Addressing the Challenges of Assessment and Feedback in Higher Education: A collaborative effort across three UK Universities

Assessment has been identified as one of the major challenges faced by Higher Education Institutions (Whitelock, et al, 2007). As a response to the challenge, in a project funded by the Joint Information Systems Committee (JISC) Open Mentor (OM) was developed as a learning support tool for tutors to help them reflect on the quality of feedback given to their students on assignments submitted electronically. Its development was based on the fundamental theory that there was convincing evidence of systematic connections between different types of tutor comments and the level of attainment in an assignment (Whitelock, et al 2004). OM analyses, filters, and classifies tutor comments through an algorithm based on Bale’s Interaction Process. As a result, tutor’s feedback comments are classified into four categories namely: Positive reactions, Teaching points, Questions and Negative reactions. The feedback provided is analysed against an ideal number of feedback comments that an assignment given a mark of a specific band should have. Reports are provided in OM to support tutors in the task of reflecting on their feedback structure, content and style. The JISC-funded Open Mentor technology transfer (OMtetra) project is continuing the work initiated by the Open University implementing OM at the University of Southampton and King’s College London. OMtetra aims at taking up OM and extending its use by developing the system further and ultimately offering better support to tutors and students in the assessment process. A group of tutors from the University of Southampton and Kings’ College are at present using OM in their teaching and assessment. In this paper, we explore potential improvements to OM in three aspects: user interface, technology implementation and analysis algorithm design. For the user experience aspect suggested additions to OM include the creation of a simple entry form where tutors may validate the results of the analysis of the feedback comments. In addition, enhancements to OM will facilitate uploading of students and modules information into the system. Presently, OM utilises a built-in database of users that needs to be maintained separately from institutional systems. Improvements for this system feature include a more flexible authentication module which would simplify the deployment of the system in new environments and thus promote uptake by a larger number of institutions. In order to reach this goal, the system will be migrated to an open source framework which provides out-of-the-box integration with various authentication systems. The last to improve is the analysis algorithm. Currently, OM classifies tutors’ comments into four categories by applying an underlying text matching algorithm. This method could be improved if tutors are allowed to confirm comments’ classification through the OM interface and a free-text classification algorithm. As the number of users grow, so will the algorithm and analysis process, making it more comprehensive and intelligent as the keywords used during analysis are dynamically expanded. OMtetra is an on-going project with a lot of potential. We believe that the outcomes from the development and trial implementations of OM will contribute highly to the area of assessment in higher education

Regulating Artificial Intelligence and machine learning-enabled medical devices in Europe and the United Kingdom

Recent achievements in respect of Artificial Intelligence (AI) open up opportunities for new tools to assist medical diagnosis and care delivery. However, the typical process for the development of AI is through repeated cycles of learning and implementation, something that poses challenges to our existing system of regulating medical devices. Product developers face tensions between the benefits of continuous improvement/deployment of algorithms and keeping products unchanged. The latter more easily facilitates collecting evidence for safety assurance processes but sacrifices optimisation of performance and adaptation to user needs gained through learning-implementation cycles. The challenge is how to balance potential benefits with the need to assure their safety. Governance and assurance processes are needed that can accommodate real-time or near-real-time machine learning. Such an approach is of great importance in healthcare and other fields of application. AI has stimulated an intense process of learning as this new technology embeds in application contexts. The process is not only about the application of AI in the real world but also about the institutional arrangements for its safe and dependable deployment, including regulatory experimentation involving new market pathways, monitoring and surveillance, and sandbox schemes. We review the key themes, challenges and potential solutions raised at two stakeholder workshops and highlight recent attempts to adapt the laws for AI-enabled medical devices (AIeMD) with a special focus on the regulatory proposals in the UK and internationally. The UK regulatory trajectory shows signs of alignment with the US thinking, and yet the European Union model is still the most closely aligned framework.</p

Success for Boys: planning guide and core module

This publication does not have an abstract

Integrating Trust into the CyberCraft Initiative via the Trust Vectors Model

This research supports the hypothesis that the Trust Vector model can be modified to fit the CyberCraft Initiative, and that there are limits to the utility of historical data. This research proposed some modifications and expansions to the Trust Model Vector, and identified areas for future research

How can diagnostic assessment programs be implemented to enhance inter-professional collaborative care for cancer?

BackgroundInter-professional collaborative care (ICC) for cancer leads to multiple system, organizational, professional, and patient benefits, but is limited by numerous challenges. Empirical research on interventions that promote or enable ICC is sparse so guidance on how to achieve ICC is lacking. Research shows that ICC for diagnosis could be improved. Diagnostic assessment programs (DAPs) appear to be a promising model for enabling ICC. The purpose of this study was to explore how DAP structure and function enable ICC, and whether that may be associated with organizational and clinical outcomes.MethodsA case study approach will be used to explore ICC among eight DAPs that vary by type of cancer (lung, breast), academic status, and geographic region. To describe DAP function and outcomes, and gather information that will enable costing, recommendations expressed in DAP standards and clinical guidelines will be assessed through retrospective observational study. Data will be acquired from databases maintained by participating DAPs and the provincial cancer agency, and confirmed by and supplemented with review of medical records. We will conduct a pilot study to explore the feasibility of estimating the incremental cost-effectiveness ratio using person-level data from medical records and other sources. Interviews will be conducted with health professionals, staff, and referring physicians from each DAP to learn about barriers and facilitators of ICC. Qualitative methods based on a grounded approach will be used to guide sampling, data collection and analysis.DiscussionFindings may reveal opportunities for unique structures, interventions or tools that enable ICC that could be developed, implemented, and evaluated through future research. This information will serve as a formative needs assessment to identify the nature of ongoing or required improvements, which can be directly used by our decision maker collaborators, and as a framework by policy makers, cancer system managers, and DAP managers elsewhere to strategically plan for and implement diagnostic cancer services

Can Comprehensive Geriatric Assessment be delivered without the need for geriatricians? A formative evaluation in two perioperative surgical settings

Introduction The aim of this study was to design an approach to improving care for frail older patients in hospital services where Comprehensive Geriatric Assessment (CGA) was not part of the clinical tradition. Methods The intervention was based on the principles of CGA, using quality improvement methodology to embed care processes. Qualitative methods and coproduction were used to inform development of the intervention, which was directed towards the health care professionals involved in peri-operative/surgical cancer care pathways in two large UK teaching hospitals. A formative, qualitative evaluation was undertaken; data collection and analysis were guided by Normalisation Process Theory. Results The clinicians involved agreed to use the toolkit, identifying potential benefits including improved surgical decision making and delivery of interventions pre-operatively. However, sites concluded that pre-operative assessment was not the best place for CGA, and at the end of the 12-month trial, implementation was still nascent. Efforts competed against the dominance of national time-limited targets, and concerns relating to patients’ immediate treatment and recovery. Some participants involved in the peri-operative pathway felt that CGA required ongoing specialist input from geriatricians, but it was not clear that this was sustainable. Conclusions Clinical toolkits designed to empower non-geriatric teams to deliver CGA were received with initial enthusiasm, but did not fully achieve their stated aims due to the need for an extended period of service development with geriatrician support, competing priorities, and divergent views about appropriate professional domains.NIH

Addressing the challenges of assessment and feedback in higher education: a collaborative effort across three UK universities

Assessment has been identified as one of the major challenges faced by Higher Education Institutions (Whitelock, et al, 2007). As a response to the challenge, in a project funded by the Joint Information Systems Committee (JISC) Open Mentor (OM) was developed as a learning support tool for tutors to help them reflect on the quality of feedback given to their students on assignments submitted electronically. Its development was based on the fundamental theory that there was convincing evidence of systematic connections between different types of tutor comments and the level of attainment in an assignment (Whitelock, et al 2004). OM analyses, filters, and classifies tutor comments through an algorithm based on Bale’s Interaction Process. As a result, tutor’s feedback comments are classified into four categories namely: Positive reactions, Teaching points, Questions and Negative reactions. The feedback provided is analysed against an ideal number of feedback comments that an assignment given a mark of a specific band should have. Reports are provided in OM to support tutors in the task of reflecting on their feedback structure, content and style. The JISC-funded Open Mentor technology transfer (OMtetra) project is continuing the work initiated by the Open University implementing OM at the University of Southampton and King’s College London. OMtetra aims at taking up OM and extending its use by developing the system further and ultimately offering better support to tutors and students in the assessment process. A group of tutors from the University of Southampton and Kings’ College are at present using OM in their teaching and assessment. In this paper, we explore potential improvements to OM in three aspects: user interface, technology implementation and analysis algorithm design. For the user experience aspect suggested additions to OM include the creation of a simple entry form where tutors may validate the results of the analysis of the feedback comments. In addition, enhancements to OM will facilitate uploading of students and modules information into the system. Presently, OM utilises a built-in database of users that needs to be maintained separately from institutional systems. Improvements for this system feature include a more flexible authentication module which would simplify the deployment of the system in new environments and thus promote uptake by a larger number of institutions. In order to reach this goal, the system will be migrated to an open source framework which provides out-of-the-box integration with various authentication systems. The last to improve is the analysis algorithm. Currently, OM classifies tutors’ comments into four categories by applying an underlying text matching algorithm. This method could be improved if tutors are allowed to confirm comments’ classification through the OM interface and a free-text classification algorithm. As the number of users grow, so will the algorithm and analysis process, making it more comprehensive and intelligent as the keywords used during analysis are dynamically expanded. OMtetra is an on-going project with a lot of potential. We believe that the outcomes from the development and trial implementations of OM will contribute highly to the area of assessment in higher education

Mental health service use among mothers involved in public family law proceedings: linked data cohort study in South London 2007-2019

PURPOSE: Mental health problems and substance misuse are common among the mothers of children who experience court-mandated placement into care in England, yet there is limited research characterising these health needs to inform evidence-based policy. In this descriptive study, we aimed to generate evidence about the type, severity, and timing of mental health and substance misuse needs among women involved in public family law proceedings concerning child placement into care ('care proceedings'). METHODS: This is a retrospective, matched cohort study using linked family court and mental health service records for 2137 (66%) of the 3226 women involved in care proceedings between 2007 and 2019 in the South London and Maudsley NHS Mental Health Trust (SLaM) catchment area. We compared mental health service use and risk of dying with 17,096 female-matched controls who accessed SLaM between 2007 and 2019, aged 16-55 years, and were not involved in care proceedings. RESULTS: Most women (79%) were known to SLaM before care proceedings began. Women had higher rates of schizophrenia spectrum disorders (19% vs 11% matched controls), personality disorders (21% vs 11%), and substance misuse (33% vs 12%). They were more likely to have a SLaM inpatient admission (27% vs 14%) or to be sectioned (19% vs 8%). Women had a 2.15 (95% CI 1.68-2.74) times greater hazard of dying, compared to matched controls, adjusted for age. CONCLUSION: Women involved in care proceedings experience a particularly high burden of severe and complex mental health and substance misuse need. Women's increased risk of mortality following proceedings highlights that interventions responding to maternal mental health and substance misuse within family courts should offer continued, long-term support