This is an ESCalate development project awarded to Kerry Shephard, University of Southampton in 2005. It looked at developing practical approaches to integrating a range of commonly available e-learning tools to facilitate wider use of the On-line Tutorial in staff development settings.  The aim of this project was to implement and assess the value of an online tutorial within staff development scenarios which are themselves encouraging e-learning engagement. Four tutorials were developed, each one addressing a different issue relating to the use of ICT (information and communication technology) in teaching: factors that limit the use of ICT to support student learning in UK HE, use of ICT to support widening participation, teaching strategies for e-learning, and a simulation of online student assessment

Furneaux, Clare

Haslam, Paul

Hutchings, Maggie

Shephard, Kerry

English

Digital Education Resource Archive

Evaluation Report; Tutorial 1  Prepared by the Tutorial Facilitator Dr Kerry Shephard.  October 2nd 2003 Summary This tutorial did achieve its broad aims and did allow participants to demonstrate achievement of its limited intended session outcomes. In addition it provided the project team with experience of on-line synchronous communication that should enable progression of the quality of learner support in subsequent tutorials. Introduction  The aim and intended Learning Outcomes for the session were;  Aim of session …to undertake a synchronous discussion on the issues that limit the use of ICT to support student learning in UK HE.  Session Outcomes … Participants will be able to demonstrate their own critical awareness of the use of ICT to support student learning in UK HE. … Participants will also be able to demonstrate involvement in online discussion and online negotiation about these issues.  Evaluation Sources 1. The session archive 2. Facilitator’s personal reflections 3. Participant contributions to the on-line discussion before and after the tutorial 4. Participant perceptions recorded on a short email Questionnaire. Participants were asked to indicate their evaluations of the Tutorial on a scale of 1 (strongly disagree that I attained this outcome) to 5 (strongly agree that I attained this outcome). Only one participant response was received (25%). Did the tutorial achieve its aims? The archive demonstrates that facilitator and participants did undertake a synchronous discussion on the issues that limit the use of ICT to support student learning in UK HE. Did participants achieve the intended ‘learning’ outcomes? The learning outcomes were carefully designed to be relatively undemanding. Neither intended outcome required participants to demonstrate learning or particular knowledge. The first sought to enable participants to demonstrate critical awareness. The second sought to enable participants to demonstrate involvement in discussion and negotiation.  All participants contributed to the discussion and all were sent a transcript of the discussion that would enable them to demonstrate involvement in discussion. There were 251 separate contributions including roughly equitable proportions from all sites. The extent and depth of contribution may not have pleased all participants but its existence is clearly demonstrable.  Negotiation is more difficult to confirm. The significant element of the tutorial that required negotiation was based on the attempt to collectively compile a list of ten factors and to negotiate a rank order for these factors. We did compile a list and we did attempt to agree an order but this latter aspect was limited by two major factors; lack of time and participant inability to use the Virtual Classroom’s whiteboard. Some agreement proved possible; for example one participant claimed “ditch the gov(-ernment) point and ac(-ademic) inertia and I think that we may be able to agree…”. One participant’s evaluation response sums up the level of attainment of this learning outcome “ 4 - better on involvement than negotiation”.   Did all participants demonstrate critical awareness of the use of ICT to support student learning in UK HE …. in the tutorial? …or in the post- tutorial discussion. It would not be easy to assess individuals on the extent of their demonstrated critical awareness but it is clear that all participants did demonstrate this to some extent even within the constraints of the tutorial itself.  Key contributions to confirm this included;  “making clear to the students the purpose of the e-lecture and how they were expected to engage with it”; “ How do we define a good streamed lecture? Where is the interaction going to take place?”; “ Yes, one of our progs (programmes)is run by educationalists who think that they can just transfer their good abilities to the on-line domain without learning new techniques. Consequently they’ve tried to use synchronous discussion –disastrously”. One Participant did doubt, however, that critical awareness could be demonstrated in synchronous computer mediated communication.   The Facilitator’s evaluation is that Tutorial 1 did unequivocally achieve its intended outcomes to a high degree. But this must be interpreted in relation to the relatively low level of expectation of what would be possible to achieve in this first tutorial and the extent to which the tutorial itself would be able to contribute to improvements in the abilities of participants. Participants within this tutorial almost certainly already had these abilities and skills. The tutorial gave them an opportunity to express them but not necessarily to develop them. Perhaps more precisely, the tutorial neither attempted to measurably improve participants’ abilities nor offered opportunities for them to demonstrate an improvement. But it was just one tutorial. This tutorial, situated within a programme with clearly defined entry requirements, learning outcomes and a well-aligned assessment, could almost certainly achieve measurable progress. Problems, solutions and conclusions from Tutorial 1 to pass on to Tutorial 2 Facilitator This tutorial clearly has accessory aims that related to the project rather than to the learner support of participants. It aimed to help discover aspects of the ‘synchronous on-line tutorial’ that worked and those that did not and to promote the development of all project members. Discussion after the tutorial focussed on these aspects and resulted in the production of a ‘Lessons Learned’ document attached below this section.  The aspects that the Tutorial-1-Facilitator thought needed particular attention in subsequent tutorials and discussion were;  Technical problems; problems with network bandwidth, firewalls and speed of interaction that seemed worse when NetMeeting or the shared browser was being used. The Facilitator urged perseverance at this stage.  Facilitation; The Facilitator urged other Facilitators to attend to good timing, to provide periodic summaries and to separate comments about the process of the tutorial (and the technology) from those of the topic of the tutorial.   Synchronisation; The Facilitator urged other Facilitators to attend to synchronisation between questions and responses in the tutorial discussion.   Whiteboard; The Facilitator urged other Facilitators to practice with this tool.  Depth of debate; The Facilitator urged other Facilitators to consider this in relation to the aims and objectives of this and of future tutorials.  Outcome of project; The Facilitator urged other Facilitators to consider the key question of what synchronous on-line tutorials are good for.  Coping without body language; The Facilitator urged other Facilitators to persevere with NetMeeting but probably realised at this stage that this was just not going to work for all partners!  Lessons Learned from Tutorial 1  Participant’s   Comments Category or theme Lesson to learn? From one facilitator to another. I could not get through to **** or to ****/**** by Netmeeting between 10 and 11....so I was very relieved that we all got together in the VC.  In addition, I spent time last week and from 8.30 this am trying to sort out the technical problems (esp re NetMeeting) and, though I was relieved others had had problems too so we couldn't use it, I did feel I'd wasted time and hassled busy folk unnecessarily. Was also disappointed not to get to grips with a new technology - let's hope we manage that for Friday.  It was not possible to show the streamed video yet this had operated properly on every test run.  Technical problems Worth pursuing with NetMeeting. When and if we get it working reliably it might help us as an introductory meeting medium as intended. I think that there are still firewall problems at Reading but it is worth **** trying to contact **** and Kerry on Friday  (I do not think that we can call****).  I suspect that the group browser fails to operate when there are more than 2 users. We do need to test this but to have backups available. Some conversations were not rapid...some were very slow and I could not tell if this was due to slow internet connections or to unwilling participants.  This question of order and permissions seems to be a significant one. **** and I sat alongside each other talking as we went along and this will have influenced our degree of interaction online compared with f2f. (We plan to use different terminals alongside each other next time). But it also makes us think of the principles of interaction in f2f seminars and to ask if those norms of interaction in f2f are relevant here? Would we expect each individual to make responses to every statement made in class? This issue brings to mind the optimum number of participants in an online tutorial. What would be our ideal number? What would have happened if there had been more than 4-5 participants interacting? Participation  The transcript shows that participation was good from all partners. We will probably get more confident.  Some discussions flowed well and I did feel that I (as facilitator) could step back and watch it happening.   I tried to separate comments about the on-line process from comments that related to the tutorial...I think that this was OK.  The balance between participant interaction and tutor guidance was not an easy one without the benefit of f2f cues. There were times when we were not sure whose turn it was, when to say something and when to wait. Kerry did a grand job in rescuing us and refocusing discussion, which really helped us.  We did manage to have discussion, punctuated by brief summaries from the facilitator. We also did manage to work collaboratively on a task.   Facilitation Facilitators need to: attend to good timing (to move the discussion between topics), give  periodic summaries and separate comments about the discussion from comments about the tutorial process.   I suggest that the Facilitator’s comments about process should occur in (Brackets). The expected problem of questions and answers arriving out of synch did occur. I felt that all participants did experience this and learn from it as intended. Participants did suggest solutions (eg adding RE; topic) that we did work with for a while. I hope that we do adopt a system in the second tutorial that we all buy into.  Reading thru the transcript just now: a) explains to me some things I didn't understand, as I was getting messages in a different order at the time and this explains some of the confusion I expressed at certain points.  The chat medium led very quickly to a sense of both confusion and urgency. Finding responses were out of step with co-participants contributions was disconcerting.  Establishing ground rules (eg. Use of RE and subject) as we went along to assist the out of sequence threads of discussion was a good idea. This leads me to think about pre-planning synchronisation We do need to perfect a process of synchronisation. Some systems allow for speakers to request and pass on a virtual ‘baton’.  I think that we should carry on using Re. to indicate the topic of a contribution. Also to try hard to keep our contributions short so that they do not take too much time to type. and student support issues. We might want to recommend a longer lead in time in gentle synchronous chat and establishing ground rules before the serious business of engaging in e-learning (along the lines of Gilly Salmon's five stage model for asynchronous e-moderating)? The whiteboard needs practice to make it work..but for simple tasks it can work. (I think that all participants did manage to do something construcive to the Whiteboard).  I couldn't interact at all with the Whiteboard - and didn't realise others, apart from Kerry, were doing so. Confusing - but could be sorted out, I assume. What did I need to do to be able to use the Whiteboard, Kerry?  The group whiteboard seemed to have a mind of its own. Initially it was not clear if the facilitator was managing this or if it was open to participants to contribute. Suddenly it was there and words were appearing while we were still adding ideas in the Chat window. We did not manage to input any text on the whiteboard but look forward to having another go.  whiteboard We all need to practice with the Whiteboard unless ****, **** and **** assure us that they will not use it. A key element of these tutorials is to encourage collaboration on Tasks. The Whiteboard is one way to do this. The time went very fast. I did have to move the group on to ensure that the tutorial addressed all of the topics and finished within 1 hour.  It also shows me points I just didn't register when chatting - eg the point (****'s, I think) about the government - I have no recollection of seeing it appear on screen in the chat. I did scroll up from time to time (in the longish gaps) but didn't see this point.  The experience reminded me of learning to drive. Your mind and senses are trying to engage with lots of different things all at the same time. The content and interaction (the road ahead) can be lost in trying to understand and master the medium.  The text box in the VC is not large enough..it is difficult to keep track of a rapid conversation.  Timing and speed Time moves rapidly in an active synchronous tutorial! I suspect  it would drag if we did not make the contributions that we did!  There is not really enough time to develop in depth arguments. See the comments below in Depth of Debate. We found ourselves needing to respond more rapidly and abbreviating and curtailing responses to send them as quickly as possible (with feelings of can't think and type quickly enough).   The time went incredibly quickly and there was a sense of unfinished business when the session was drawn to a close, but your time keeping was great, thanks Kerry.  The first is about the level of sophistication we were able to display in understanding the nature of entries and negotiating tricky areas. I think the transcript shows a couple of examples of interchanges that were difficult to decode at any depth. The first was the debate over whether academic exposition would benefit from being more measured (and possibly prosaic - there was a reference to institutions not being schools), rather than idiosyncratic and potentially less organised. The second was the discussion of the willingness of academics in general to embrace technology in their preparation. I would have liked to have gone into both at greater depth but felt constrained by the medium.   We did discuss the Bad Lecture video clip, however - though didn't do it seriously enough, I suspect, for Kerry. Were you disappointed with the flippancy there, Kerry? You tried valiantly to keep us on task - wonder if that would have been easier f2f - I suspect so.  The look of the thing - I'm always horrified by my typos in synchronous chat. Also you feel the pressure to submit a reply, so they are short. When you add to that the problem following/replying to threads, it can all appear a bit unthought -out and, well, unacademic.  Discussion on this task was also not trivial. One participant felt strongly that some content was not right and this was attended to.  There seemed to be a lack of space and time to share considered responses on ideas within the session. This Depth of debate Clearly there are concerns about the depth of debate. I do not think right now that we will improve this in a synchronous setting.   Compare the average word count of contributions in the synchronous tutorial to those in the asynchronous discussion….does this illustrate the dept of argument possible? (Hey ****…is this your academic area?).  On the other hand.. asynchronous discussion took 4 days to get 4 responses from 4 sites. Many discussions work on the expectation that participants will lok in and make a contribution 3 times each week. Our first tutorial had 251 separate contributions from 4 sites. That’s more than 60 contributions each. On the basis of 3 contributions each week per participant…our one-hour tutorial had as many contributions as 20 weeks of asynchronous discussion.   (never mind the quality, feel the width!)  Of course the comparison is not really fair and leads into the comments below about the purpose of reflects ****'s and ****'s comments about the pedagogical and learning value of the medium and in what circumstances it might be appropriate to use it. It seemed OK for brainstorming and sharing ideas but we needed more time for analysing and prioritising our responses. synchronous tutorials.  More seriously pedagogically - we didn't really discuss the Warburton article, did we? It's got a lot in it - esp all the figures and I felt we didn't get to grips with it much. A f2f discussion, with a clear agenda, would have been more fruitful.  The second is about the need to define the reasons we opt to use a particular medium. During our synchronous discussion, I found myself questioning the circumstances under which I might be able to use it to improve the quality of learning. I could only see myself using it to discuss an artefact or paper which we might all be working on. I also felt that a telephone conference between a group of our size might have yielded as much.  All these points reinforce for me ****'s query - what would the use of synchronous chat be in teaching? Very limited, I think - esp if you have a larger group, where it gets impossible to follow threads. Much better to go for asynchrounous discussion, which at least gets round most of the points above (though not 5 & 7).  But (and this was my worry re publication) there is a written Outcome of project What is the synchronous tutorial good for? We need to work on this as the tutorials develop.   Tutorial 1 had clear session outcomes. I think that they were less demanding than ****’s are for Tutorial 2./ I guess that, in the absence of an assessment I do need to ask each participant if they felt that they achieved the session outcomes. (I will send an email).  Perhaps the lesson learned is that we must have reasonable expectations of what we will achieve? record for all to see - which you don't have after a f2f chat. So that's another worry/distracting thought at the back of your mind as you chat.  I agree with **** about worrying about not knowing who the other folk are and how you are coming across. I got concerned that **** and ****weren't saying much at one point and worried that they might be feeling out of it. Were you?  Without body language, it was difficult to identify seriousness, playfulness etc and I was certainly reluctant to engage robustly for fear of giving offence unintentionally.  Coping without body language I still have hopes that the introductory NetMeeting sessions will allow us to operate without visual and audio cues…the alternative is more technologically demanding technology!   

The on-line tutorial: developing and evaluating resources and disseminating experience

Abstract

Similar works

Full text

Available Versions

Digital Education Resource Archive