Kambouri, Maria

Mellar, Harvey

Pavlou, Victoria

Sanderson, Mariana

UCL Discovery

 ICT and adult literacy and numeracy – what counts as evidence? Harvey Mellar, Maria Kambouri, Mariana Sanderson, and Victoria Pavlou1 Institute of Education, University of London Background and aims of the study The Skills for Life strategy proposes an important role for ICT in adult literacy and numeracy provision. ICT is seen as a motivator, as opening up wider access to learning, and as providing new ways of teaching and learning. The Institute of Education has carried out a number of evaluation studies of the use of ICT in the teaching of adult literacy and numeracy (see http://www.ioe.ac.uk/hgm/research/SkillsforLife/). This work in common with other studies (e.g. Hopey 1998) was principally based on surveys, and interviews with managers, tutors and learners, together with some testing of learning gains. Whilst this research has found some positive signs, it has also suggested there is a long way to go if the expectations of the impact of ICT on learning for this group of learners are to be met. Research is needed in order to support a move beyond present practice, and to find more effective ways of using ICT to improve learning for learners with basic skills needs. One element that was clearly lacking from previous studies was a detailed account of what tutors actually do when they are using ICT for adult literacy and numeracy. The aim of this present study, therefore, was to carry out detailed classroom observations in order to collect, review and analyse existing practice, examining how the use of ICT has changed teaching and learning in Basic Skills, and so to begin to more closely identify the factors involved in effective teaching with ICT in the area of adult literacy and numeracy. Once this study is complete then it is expected that in a second stage of the project that a series of hypotheses about effective use can be generated which will then be tested through intervention studies. Methodology Preparation (July to August 2002) Identification of centres and tutor-researchers The study was carried out with centres forming part of the East London Pathfinder Consortium. We identified a group of eight tutor-researchers who would cover the range of uses of ICT that we wanted to see, and who would be willing to work with the project. These eight tutor-researchers together with the Pathfinder Co-ordinator formed an integral part of the research team, something that we feel greatly added to the validity of the data. The tutor-researchers were involved at every level of the project: setting the project goals, contributing to the development of the observation criteria, testing out the criteria, carrying out observations of other tutors’ classrooms, and giving feedback on the results. There were three one day workshops during the project in which the whole team participated. The tutor-researchers were provided with five days of teaching cover to allow them to attend these meetings and to carry out observations of each others’ classrooms.                                             1 Together with the East London Pathfinder Co-ordinator: Jenny Barrett, and Tutor-Researchers: Judith Beer, Sean Gubbins, Jenik Nazarian, Jane Peckham, Camilla Nightingale, Eugene Smith, Graciano Soares and Peter Townsend  As a pilot study, we videotaped three classroom sessions before the first workshop. First workshop The aim of the first workshop was to share the tutor-researchers’ current practice, and to discuss the issues and problems that needed to be addressed. The videos that had been made of teaching sessions as part of the pilot study formed a very useful resource for discussion both of present practice and research issues. Research issues The first workshop enabled us to define the issues that the research team saw as relevant in addressing the general project aim “to carry out detailed observational studies in classrooms in order to collect, review and analyse existing practice, examining how the use of ICT has changed teaching and learning in Basic Skills, and so to begin to more closely identify the factors involved in effective teaching with ICT in the area of adult literacy and numeracy”. The issues identified were: Curriculum issues Learner issues Relationship of ICT and content objectives Learning preferences Learning objectives of tutors/learning goals of learners Individual and collaborative working Accreditation of ICT as well as literacy and numeracy Feedback Difference between ESOL and literacy Literacy level of the learners  Issues specific to THIS audience (adults/Basic Skills)   ICT issues Teacher issues ICT as a possible barrier  Relation of ICT to teacher style ICT as a basic skill – digital literacies Classroom management ICT as a subject in itself – the vocabulary of ICT Meta-language of the classroom Breakdown Management of open learning Development of observation methods (August – September 2003) Second tutor workshop The second workshop was principally concerned with the issues of data capture and observation methods. While videotaping was presented as a possible way of collecting data, after discussion with the tutors this method was abandoned, as being impracticable in some classrooms because of their size, and unacceptable to a large number of Muslim students in the classes being observed.   It was decided that each tutor would prepare a pre-session report outlining the learning objectives and the place of ICT within their lessons, that lessons would be viewed on three or four occasions, with at least one observation undertaken by one of the tutor-researchers, and whilst most of the researcher observations would be of the class as a whole, one of the observations for each class would look principally at the learners. A series of observational schedules were drawn up, based on the issues identified in the previous workshop. Observation schedules and observations The main classroom observation schedule went through a number of developmental steps starting with a rather general design and developing into a quite detailed design (for the final version see Appendix 1). The schedule had a section for structured coding to be carried out at five-minute intervals, and section for a more open-ended narrative account together with a column for comments linking the observations to the learning objectives identified in the pre-session report. These instruments were initially piloted on the videotapes of sessions that we had recorded and then in a number of classroom observations.  A definition of the coding for this structured element of the observation is presented in the form of a systemic network (for an account of systemic networks see Bliss et al 1983) in Appendix 2. The coding rules were: • Code predominant activity over last five minutes • Code from the left, and from top to bottom, in the network • ALWAYS code all three variables: tutor activity, learner activity, and equipment used • ICT Use  o If any ICT use then code ALL the variables: style of use, type of use, users, and purpose o If ICT problems occur then code ICT Problems  The systemic network notation used here employs three symbols:       A  B  C Code each of the variables A, B, C    D E F Select one of D, E, F   Repeat selection as often as necessary  This coding was intended to take just a few seconds at the end of each five minute interval, and was not intended to be a major distraction from the narrative coding. In order to enable this to happen an element of practice was necessary, but once the researchers were familiar with the coding this process was found to produce few difficulties. This level of detailed structured recording had a number of advantages: it enabled us to maintain a degree of objectivity in capturing the main outlines of the classroom activities, it acted as a trigger to the observer to reflect on what was happening in the classroom at regular intervals, acted as a structure for their accounts, and acted as an index which enabled us to look for examples of specific behaviours within the narrative accounts. The most problematic instrument to develop was the one for tutor-researchers to observe other tutors. It was not felt appropriate that they should use the complex observational schedule used by the researchers that we have described above, as the training necessary to use it was not justified for the small number of observations that they would be undertaking. A pro-forma for observing critical events was initially proposed, but the tutor-researchers opted instead to record their observations in a more open way, recording what they saw as important in the session they were observing. The team generated a guidance sheet which was given to the observers. Data collection (October 2002 to January 2003) Data collection Eight tutors were observed working with 11 different classes. On average each class was observed three times over a two-month period in order to gain a view of the learners’ developing practice with ICT, and a further observation session was devoted to observing the learners in the same class. Observations were carried out both by research officers, and by tutor-researchers. Thirty-five classroom observations (constituting a total of 774 five-minute observation units) were made, 11 of these were carried out by tutor-researchers, and additionally 12 learner observations were made.  Before the observation sessions the tutors completed a pre-session report and provided a lesson plan. After an observation a short summary account of the classroom observation was created and sent to the tutor for comments. It had been hoped in this way to generate a negotiated account of the session, but in practice there were relatively few comments. It was felt that it would be best to interview tutors immediately or shortly after the session observed, but this turned out to be very difficult in practice because of the time constraints on the tutors. Tutor-researchers varied greatly in the way in which they approached the classroom observations, most tutor-researchers took an observing role, but some worked with groups of learners. The style of reporting varied, some were timed narratives, others were carefully organized around specific themes. All, however, found the experience useful from the perspective of their own professional development, and almost all reported that it was the first time that they had actually seen another tutor teaching with ICT. Third workshop A third workshop was carried out towards the end of the observations in January 2003 and gave a chance for the research team to reflect on the data collection process (and in particular on the observational instruments) and on the data collected. The team collectively developed thumbnail accounts of each of the classrooms observed, calling on evidence from the researchers, the tutor-researchers as observers, and the tutor-researchers as teachers. This served to deepen the understanding of what was happening in the classrooms and to orient the researchers to the data before starting on the data analysis. Data analysis Whilst this paper is concerned principally with the data collection phase of the study, it may be useful to briefly sketch the forms of analysis that were later carried out. Frequency tables and charts were constructed from the data derived from the structured observations. Comparisons were made between tutors and between curriculum areas, and cross-tabulations between variables examined to find significant interactions. The data derived from the open ended observations were summarised using the scheme given in Appendix 3. The results of the two forms of analysis were then compared and combined. Discussion This paper has detailed the procedures by which we collected evidence about the use of ICT to teach literacy, ESOL and numeracy students in basic skills adult education classes. Previous studies had focussed principally on interviews with tutors and learners about the effects this practice had had on them, and there were no previous accounts of what was actually taking place in the classrooms. It was important for the validity of this study that we base these observations on the concerns of the tutors themselves, hence we derived the research questions and some of the observation instruments in conjunction with the tutor-researchers. This we felt gave us the best possible chance of deriving data which was relevant to their concerns. The decision to code in five minute intervals according to a structured coding scheme was taken after a literature review of classroom observation methods, and a series of pilot studies. This enabled the generation of a systemic network, which in itself is a useful picture of the variables involved in the interaction. Structured observation, we felt, would allow some element of distance in the account, and would enable more than one observer to collect similar data. It has allowed us to capture some of the more behavioural surface features of the classroom and derive numerical measures that were useful for seeing  differences and similarities between classes. Not all features of this coding turned out to be as useful as we had hoped, but on the whole the method yielded useful data. The data collected by these structured observations was supplemented by more open ended observations which led to a fuller picture of what was happening in the classrooms, and hence to the possibility of making comparisons between teaching styles and use of ICT.  References Bliss, J., M. Monk, et al. (1983). Qualitative data analysis for educational research: a guide to uses of systemic networks. London, Croom Helm. Hopey, C.E. (1998). Technology, basic skills, and adult education: getting ready and moving forward. Ohio: ERIC Clearinghouse on Adult, Career, and Vocational Education, Ohio State University. Mellar, H., Kambouri, M., Wolf, A., Goodwin, T., Hayton, A., Koulouris, P. and Windsor, V. (2001). Research into the effectiveness of learning through ICT for people with basic skills needs June 2001. Sheffield: Ufi. Summary available at: http://www.ufiltd.co.uk/press/papers/literacyguide.pdf (accessed 25 Sept 2003) Mellar, H. and Kambouri, M. (in press). ‘Learning and teaching adult basic skills with digital technology: research from the UK’. In A. Brown and N. Davis (eds), ICT and education: World Yearbook 2004: Digital technologies, communities and education. London: Kogan Page. Pratt, D.D. et al (1998). Five Perspectives on Teaching in Adult and Higher Education. Malabar, FL: Krieger  Appendix 1 - Observation Schedule                       Research observers record Code:    Time IC SY IC TY IC US IC PU IC CP IC CU CC TU CC SA CC EQ ACTION COMMENT Learning goal 0930-940                                                                                            Code   Numbers  Research Observers report Software details & other material Site  Enrolled  Classroom details Class tutor   Usual   Class&time  Today   ICT use  explanation of headings  Class context explanation of headings    ICSY  Style of use  CCTU Tutor activities   ICTY  Type of use CCSA  learner activities  ICUS  Users CCEQ Non- IC equipment  ICPU  Purpose     ICCP  Computer problems  Tutor observer: Research observer  ICCU  IC problems for user   name:   name:     Appendix 2 - Systemic Network   Appendix 3 - Pro forma for summaries of observational data 1. Learning goals a. Describe the learning objectives of the sessions – this should be derived from the tutors pre-session report, lesson plans, interviews, meetings, and any feedback b. Provide any evidence of matches or mismatches between the tutor’s learning objectives and the learners’ needs/goals/objectives? c. Describe how the teaching of ICT is integrated with the learning objectives. Describe how the teaching of ICT is integrated into the lesson plans. Describe the relationship of ICT and the content learning objectives, for example, which is seen as primary. d. Describe any evidence of ICT being viewed as a basic skill, or as a subject in itself. Describe any evidence of progression in ICT skills being considered. e. For ESOL or literacy classes provide any evidence of planning the use of ICT that takes into account the differing needs of these two audiences. f. Describe the forms of assessment used for ICT and for literacy, numeracy and ESOL as appropriate. Provide any evidence that this impacts upon the design of the teaching sessions. 2. Teaching a. Describe the teaching in the three sessions based on observation sheets etc b. Provide any evidence about the tutor’s perspective on teaching. (Refer to Pratt’s classification: i. The Transmission Perspective: Effective Delivery of Content ii. The Apprenticeship Perspective: Modelling Ways of Being iii. The Developmental Perspective: Cultivating Ways of Thinking iv. The Nurturing Perspective: Facilitating Self-efficacy v. The Social Reform Perspective: Seeking a Better Society?) c. Describe the tutor’s style of teaching. Describe the way in which task instructions are given to the class. In particular provide any evidence of possible interactions between the use of ICT and the tutor’s style teaching – does the tutor assimilate ICT to their normal teaching style, or does the use of ICT influence and change the tutor’s teaching style? d. Describe how are the necessary ICT skills are taught. e. Describe how the tutor manages the classroom, and provide any evidence of ways in which this may be affected by the use of ICT. f. Describe to what extent what happened in the classroom corresponded with the tutor’s original plans. Provide any evidence that ICT impacted on this.   g. Provide any evidence that the relationship between ICT and content teaching in the learning objectives was achieved and to what extent this changed in practice. 3. ICT a. Describe way ICT was used in the classroom, based on observation sheets etc. b. Provide evidence of any situations where ICT may have acted as a barrier to learning or teaching. c. Describe any instances of technological failure and the consequences of this. 4. Learning a. Describe how the learners worked with each other, in particular comment upon the extent to which learners worked alone or co-operated either on their own initiative or on the instruction of the tutor. b. Describe any incidents that highlight the importance of feedback from the ICT to the learners. This feedback might be instructional feedback (e.g. ‘correct’, ‘wrong’) or might be the responsiveness of the system (e.g. drawing a graph). c. Provide any evidence of incidents in which the use of ICT may have matched learners’ preferences for modes of learning (e.g. preference for visual over textual explanations). d. Provide any evidence of the effect of the literacy level of learners impacting (either positively or negatively) upon their use of  ICT. e. Provide any evidence of the effect of the ICT skills levels of learners impacting (either positively or negatively) upon their learning of literacy, numeracy or ESOL. 

What counts as evidence for what purposes in research in adult literacy

http://discovery.ucl.ac.uk/10000707/1/Mellar_Kambouri_et_al_2003.pdf

What counts as evidence for what purposes in research in adult literacy

Abstract

Similar works

Full text

Available Versions

UCL Discovery