Evaluating a virtual learning environment in the context of its community of practice

The evaluation of virtual learning environments (VLEs) and similar applications has, to date, largely consisted of checklists of system features, phenomenological studies or measures of specific forms of educational efficacy. Although these approaches offer some value, they are unable to capture the complex and holistic nature of a group of individuals using a common system to support the wide range of activities that make up a course or programme of study over time. This paper employs Wenger's theories of 'communities of practice' to provide a formal structure for looking at how a VLE supports a pre-existing course community. Wenger proposes a Learning Architecture Framework for a learning community of practice, which the authors have taken to provide an evaluation framework. This approach is complementary to both the holistic and complex natures of course environments, in that particular VLE affordances are less important than the activities of the course community in respect of the system. Thus, the VLE's efficacy in its context of use is the prime area of investigation rather than a reductionist analysis of its tools and components. An example of this approach in use is presented, evaluating the VLE that supports the undergraduate medical course at the University of Edinburgh. The paper provides a theoretical grounding, derives an evaluation instrument, analyses the efficacy and validity of the instrument in practice and draws conclusions as to how and where it may best be used

To fail is human: remediating remediation in medical education.

IntroductionRemediating failing medical learners has traditionally been a craft activity responding to individual learner and remediator circumstances. Although there have been moves towards more systematic approaches to remediation (at least at the institutional level), these changes have tended to focus on due process and defensibility rather than on educational principles. As remediation practice evolves, there is a growing need for common theoretical and systems-based perspectives to guide this work.MethodsThis paper steps back from the practicalities of remediation practice to take a critical systems perspective on remediation in contemporary medical education. In doing so, the authors acknowledge the complex interactions between institutional, professional, and societal forces that are both facilitators of and barriers to effective remediation practices.ResultsThe authors propose a model that situates remediation within the contexts of society as a whole, the medical profession, and medical education institutions. They also outline a number of recommendations to constructively align remediation principles and practices, support a continuum of remediation practices, destigmatize remediation, and develop institutional communities of practice in remediation.DiscussionMedical educators must embrace a responsible and accountable systems-level approach to remediation if they are to meet their obligations to provide a safe and effective physician workforce

Reuse as heuristic : from transmission to nurture in learning activity design

In recent years a combination of ever more flexible and sophisticated Web technologies and an explosion in the quantity of online content has sparked learning technologists around the world to pursue the promise of the 'reusable learning object' or RLO with the idea that RLOs could be reused in different educational contexts, thereby providing greater overall flexibility and return on investment. In 2002 the ACETS Project undertook a three-year study in the UK to investigate whether RLOs worked in practice and how the pursuit of reuse affected the teacher and their teaching. Teachers working in healthcare-related subjects in Higher and Further Education were asked to create an original learning design or activity from third-party digital resources and to reflect both on the process and its outcomes. The expectation was that teachers would be the ones selecting and reusing third-party materials. This paper describes how one of the ACETS exemplifiers reinterpreted this remit, challenged the anticipated transmissive model of learning, and instead, gave their students an opportunity to create their own original learning designs and learning activities from third-party digital resources. By describing the educational enhancements, the resulting heightened levels of critical thinking, and sensitivity to patient needs, 'reuse' will be shown to be an effective heuristic for student self-direction and professional development

The one minute mentor : a pilot study assessing medical students’ and residents’ professional behaviours through recordings of clinical preceptors’ immediate feedback

Introduction: The assessment of professional development and behaviour is an important issue in the training of medical students and physicians. Several methods have been developed for doing so. What is still needed is a method that combines assessment of actual behaviour in the workplace with timely feedback to learners. Goal: We describe the development, piloting and evaluation of a method for assessing professional behaviour using digital audio recordings of clinical supervisors’ brief feedback. We evaluate the inter-rater reliability, acceptability and feasibility of this approach. Methods: Six medical students in Year 5 and three GP registrars (residents) took part in this pilot project. Each had a personal digital assistant (PDA) and approached their clinical supervisors to give approximately one minute of verbal feedback on professionalism-related behaviours they had observed in the registrar’s clinical encounters. The comments, both in transcribed text format and audio, were scored by five evaluators for competence (the learner’s performance) and confidence (how confident the evaluator was that the comment clearly described an observed behaviour or attribute that was relevant). Students and evaluators were surveyed for feedback on the process. Results: Study evaluators rated 29 comments from supervisors in text and audio format. There was good inter-rater reliability (Cronbach α around 0.8) on competence scores. There was good agreement (paired t-test) between scores across supervisors for assessments of comments in both written and audio formats. Students found the method helpful in providing feedback on professionalism. Evaluators liked having a relatively objective approach for judging behaviours and attributes but found scoring audio comments to be time-consuming. Discussion: This method of assessing learners’ professional behaviour shows potential for providing both formative and summative assessment in a way that is feasible and acceptable to students and evaluators. Initial data shows good reliability but to be valid, training of clinical supervisors is necessary to help them provide useful comments based on defined behaviours and attributes of students. In addition, the validity of the scoring method remains to be confirmed

Evaluating a Virtual Learning Environment in Medical Education

The use of technology-supported teaching and learning in higher education has moved from a position of peripheral interest a few years ago to become a fundamental ingredient in the experience of many if not most students today. A major part of that change has been wrought by the widespread introduction and use of ‘virtual learning environments’ (VLEs). A defining characteristic of VLEs is that they combine a variety of tools and resources into a single integrated system. To use a VLE is not just to employ a single intervention but to change the very fabric of the students’ experience of study and the university. Despite this, much of the literature on VLEs has concentrated on producing typologies by listing and comparing system functions, describing small scale and short duration applications or providing speculative theories and predictions. Little attention has so far been paid to analysing what effects a VLE’s use has on the participants and the context of use, particularly across a large group of users and over a substantial period of time. This work presents the evaluation of a VLE developed and used to support undergraduate medical education at the University of Edinburgh since 1999. This system is called ‘EEMeC’ and was developed specifically within and in support of its context of use. EEMeC provides a large number of features and functions to many different kinds of user, it has evolved continuously since it was introduced and it has had a significant impact on teaching and learning in the undergraduate medical degree programme (MBChB). In such circumstances evaluation methodologies that depend on controls and single variables are nether applicable or practical. In order to approach the task of evaluating such a complex entity a multi-modal evaluation framework has been developed based on taking a series of metaphor-informed perspectives derived from the organisational theories of Gareth Morgan(Morgan 1997). The framework takes seven approaches to evaluation of EEMeC covering a range of quantitative and qualitative methodologies. These are combined in a dialectical analysis of EEMeC from these different evaluation perspectives. This work provides a detailed and multi-faceted account of a VLE-in-use and the ways in which it interacts with its user community in its context of use. Furthermore, the method of taking different metaphor-based evaluation perspectives of a complex problem space is presented as a viable approach for studying and evaluating similar learning support systems. The evaluation framework that has been developed would be particularly useful to those practitioners who have a pressing and practical need for meaningful evaluation techniques to inform and shape how complex systems such as VLEs are deployed and used. As such, this work can provide insights not just into EEMeC, but into the way VLEs are changing the environments and contexts in which they are used across the tertiary sector as a whole

Resetting the compass: exploring the implicit messages of orientation to a community-engaged medical school

Background: Although students’ transition into medical school is a critical step in their professional journey, orientation has been relatively under-researched, particularly with regard to its intersections with schools’ social missions. This paper reports on a study looking at the implicit messages of orientation to the Northern Ontario School of Medicine’s undergraduate program.Methods: An extended mixed methods study was conducted to look at different aspects of the School’s Orientation Week. The term “hidden curriculum” was used to shape inquiry, both in its broad sense of implicit educational experiences and messages and in its more specific sense of the educational messages sent by a medical school’s culture and activities. Data were collected using participant surveys, focus groups, and interviews. Transcripts and free-text survey responses were analyzed to identify underlying themes.Results: Orientation Week was generally well received and was generally perceived by different stakeholders (such as students, school leaders, and community members) as a positive and necessary undertaking. However, there were points of contention and confusion that created a hidden curriculum with respect to participants’ identities, both as students and as future health professionals.Conclusion: Orientation to undergraduate medical training can be successfully linked to a school’s social mission, but in doing so it can send complex and unintended messages to the participants that may be perceived quite differently based on their circumstances and expectations

The Calgary student run clinic in context: a mixed-methods case study

Background: Student Run Clinics (SRCs) provide students with clinical education while caring for underserved populations. While much of the research on SRCs comes from the USA, SRCs in other contexts need to be appraised in the context of the systems they interact with. This study explored how stakeholders in the University of Calgary’s SRC perceived its purpose and beneficiaries with respect to patients, students, undergraduate medical education, and its intersections within the healthcare system in Calgary.  Methods: Data came from the SRC’s EMR and stakeholder interviews at the Inn from the Cold (IFTC) shelter. Qualitative data were analyzed using standard grounded theory techniques. Results: There were 13 interviews - seven with student clinicians and six with preceptors and other stakeholders. Interviews highlighted the uncertainty of the SRCs role. Majority of participants saw the SRC as facilitating further access to other healthcare services, while some commented on its primarily education-focused role. Major limitations in the SRC’s scope of care and its integration with other services were identified. Conclusion: SRCs need to consider theiraccountabilities, both educational and healthcare-focused at individual and organization levels, in order to function as responsible healthcare providers in Calgary

Factors influencing medical student attrition and their implications in a large multi-center randomized education trial

Participant attrition may be a significant threat to the generalizability of the results of educational research studies if participants who do not persist in a study differ from those who do in ways that can affect the experimental outcomes. A multi-center trial of the efficacy of different computer-based instructional strategies gave us the opportunity to observe institutional and student factors linked to attrition from a study and the ways in which they altered the participation profile. The data is from a randomized controlled trial conducted at seven US medical schools investigating the educational impact of different instructional designs for computer-based learning modules for surgical clerks. All students undertaking their surgical clerkships at the participating schools were invited participate and those that consented were asked to complete five study measures during their surgery clerkship. Variations in study attrition rates were explored by institution and by participants' self-regulation, self-efficacy, perception of task value, and mastery goal orientation measured on entry to the study. Of the 1,363 invited participants 995 (73 %) consented to participate and provided baseline data. There was a significant drop in the rate of participation at each of the five study milestones with 902 (94 %) completing at least one of two module post-test, 799 (61 %) both module post-tests, 539 (36 %) the mid-rotation evaluation and 252 (25 %) the final evaluation. Attrition varied between institutions on survival analysis (p < 0.001). Small but statistically significant differences in self-regulation (p = 0.01), self-efficacy (p = 0.02) and task value (p = 0.04) were observed but not in mastery or performance goal orientation measures (p = NS). Study attrition was correlated with lower achievement on the National Board of Medical Examiners subject exam. The results of education trials should be interpreted with the understanding that students who persist may be somewhat more self-regulated, self-efficacious and higher achievers than their peers who drop out and as such do not represent the class as a whole