Effects of a self-assessment procedure on VET teachers' competencies in coaching students' reflection skills

Ideally, teachers are professionals who take responsibility for their work and choices made. Teachers are supposed to respond to new developments by experimenting with new forms of education and educational contents and to reflect on these. It is important that teachers continuously develop themselves and demonstrate a professional attitude towards their work. Self-assessment as a tool for learning fits really well into the conception of the teacher as a professional. Self-assessment makes teachers responsible for their own learning and is regarded as an essential feature of professional practice. In the literature, self-assessment is frequently described as a promising method for teacher learning. Self-assessment is defined in this dissertation as an activity in which teachers apply criteria and standards to their own work and make judgments with respect to the extent to which they have met these criteria and standards. Self-assessment not necessarily needs to be an isolated or individual activity; frequently peers are used to give feedback or are other measures undertaken to enhance learning through self-assessment. It is argued that self-assessment can lead to new insights or raise awareness of aspects of practice that need to be improved, stimulate the monitoring capacities of teachers, and enhance their understanding of what constitutes good practice. Despite the popularity and potential benefits of self-assessment, the theoretical and empirical bases supporting these claims are as yet rather underdeveloped. To better understand the possibilities, merits and shortcomings of teachers’ self-assessment as a tool for learning, there is an urgent need for research into its effects. Aim of the different studies in this dissertation was to gain insight in the effects of a developed self-assessment procedure used by teachers working in the two higher streams of Vocational Education and Training (VET) in the sector Health. This procedure was meant to foster teachers’ competence development in coaching students’ reflection skills. Reflection is an important skill for VET students, particularly for students in the Health sector. Teachers, however, experience coaching students’ reflection as an important but difficult competence. It is supposed that focusing the self-assessment procedure on improving teachers’ competencies in coaching students’ reflection skills fulfils a need. The central problem of this dissertation was the following: what are the effects of self-assessment on teachers’ competencies in coaching Vocational Education and Training students’ reflection skills? To address this problem, the following research questions were answered in this dissertation: 1. How can a useful self-assessment procedure be developed for VET teachers for coaching students’ reflection skills? 2. How do VET teachers use a self-assessment procedure that has been developed for coaching students´ reflection skills? 3. How are VET teachers’ competencies in coaching students’ reflection skills rated by themselves, their colleagues and which trends in scoring are visible over time? 4. What and how do VET teachers learn and intend to learn from (repeatedly) being engaged in a self-assessment procedure used to develop their competencies in coaching students’ reflection skills? 5. How do VET teachers value the different aspects of the self-assessment procedure which they used to develop their competencies in coaching students’ reflection skills? These research questions were answered in five different studies. The first question pertained to the development of a self-assessment procedure, the other questions to determining the effects of the developed self-assessment procedure. In the second chapter, the first research question pertaining to the development of a self-assessment procedure for fostering teachers’ competencies to coach Vocational Education and Training (VET) students’ reflection skills was answered. Through formulating design principles derived from the literature on quality criteria for teacher assessment and conditions for teacher learning a useful self-assessment procedure was developed consisting of the following elements: 1. a tool based on criteria and standards to be used by teachers to assess their own coaching competencies of students’ reflection skills; 2. feedback from peers on observed lessons by using the same tool; 3. a report written by the teachers in which they reflect on their competencies, feedback from their colleagues, set goals for future action and describe the effort needed for this; 4. feedback from colleagues on this reflective report and, if necessary, the possibility to make adjustments. Criteria and standards for self-assessment were iteratively formulated through consulting both teachers (N=40) and literature. Theoretical perspectives provided the starting point for developing the criteria and standards reflecting the what and when of coaching of reflection skills by students. Practical perspectives derived from the teachers yielded additional criteria and directions for improving with regard to the how of coaching. The result of this development process was a set of 23 criteria. Standards were formulated on a four point scale, indicating the extent to which a particular coaching behavior was realized. Through founding criteria (and standards) not only on theory but also on teachers’ own practice, it was attempted to contextualize the self-assessment procedure and to realize teachers’ feelings of ownership over the criteria. By including feedback from a colleague and writing a reflection report into the procedure for self-assessment and organizing this procedure repeatedly, it was attempted to encourage learning through self-assessment. Before starting the procedure for the first time, the teachers were trained how to undertake the self-assessment procedure. Based on this first study it could be concluded that it was possible to design an adequate and useful self-assessment procedure for fostering teachers’ learning. A first impression of teachers’ use of the self-assessment procedure indicated that feedback from a colleague appeared to be helpful for assessing oneself and that teachers performed the self-assessment procedure differently, particularly regarding the use of criteria. In the studies presented in the chapters 3 to 6, the effects of the self-assessment procedure were investigated. In these studies 24 teachers from two schools participated. From one school 20 teachers from two teams participated; from the other school 4 teachers. The sample consisted of 9 male and 15 female teachers. Their teaching experience ranged from just having finished teacher education to 28 years. All teachers were also involved in the development of the self-assessment procedure and they all undertook this procedure in three different assessment rounds (moments) with three months in between. The teachers performed these self-assessment rounds in couples. The study presented in Chapter 3 addressed the question how teachers used the different aspects of the self-assessment procedure. To answer this question the focus was on (1) how teachers used criteria and standards, (2) how they gave and received feedback, and (3) how they reflected on their competencies in coaching students’ reflection skills. For this purpose, completed self-assessments forms, video-taped feedback conversations with colleagues and written reflective reports of the 24 teachers, collected during the first assessment round, were analyzed by using category systems for each data source. It was found that teachers’ use of the self-assessment procedure could be characterized by: (1) slightly positive assessments of teachers about their own performance as well as those they received from their colleagues, (2) constructive peer feedback that was generally accepted by the teachers who were assessed, and (3) clear and informative reflective reports by the teachers mainly focusing on their own actions resulting from the self-assessment. Results of this study indicated that when used consequently and systematically, criteria and standards have added value for giving feedback and writing a reflective report. However, the results also indicated that additional measures need to be undertaken to broaden and deepen the self-assessors’ reflections and to make colleagues feedback more critical. In general, this study showed that teachers can benefit from a self-assessment procedure and improve their competencies in coaching students’ reflection skills. In Chapter 4, the third research question was addressed. This question was as follows: How are VET teachers’ competencies in coaching students’ reflection skills rated by themselves, by their colleagues and which trends in scoring are visible over time? To answer this question, 72 self-assessments and 72 colleague assessments from three assessment rounds were analyzed. The reliability and validity of the rating instrument were checked by conducting several analyses. Internal consistency of the scales for the what, when, and how of coaching students’ reflection skills appeared to be satisfactory. It also appeared from correlations between the what, when and how scales that the scales measured distinct, but related aspects of coaching students’ reflection skills. To answer the research questions, first a descriptive analysis was performed on the three scales per moment (means and standard deviations). Next, absolute and relative difference scores for the what, when and how of coaching students’ reflection skills were computed. Then, the scores of the self-assessors, colleagues and the difference scores were analyzed via a repeated-measures ANOVA. Finally, Scheffé’s post-hoc test was used to determine where the differences between the mean scores were significant and effect sizes were calculated. It was found that colleagues perceived the teachers’ coaching competencies slightly more positive than the teachers themselves. It also appeared that teachers considered themselves reasonably competent in how to coach students’ reflection skills and moderately competent in the what and when of coaching students reflection skills. Furthermore, it became clear that large individual differences existed in the perceptions of self-assessors and colleagues of competencies of coaching students’ reflection skills and in the development of these differences over time. Differences between the judgments of self-assessors and their colleagues were found to be small and did not develop over time. Large differences between couples were found though. This study led to the conclusion that a repeated self-assessment did not lead to significant changes over time, but did make meaningful differences visible between teachers in self-assessments and colleague-assessments as well as differences between individual teachers and in changes over time. In Chapter 5, the research question pertaining to what and how teachers learn and intent to learn from self-assessment was addressed. For this purpose, 69 reflective reports of the 24 teachers from the three repeated self-assessment rounds were analysed. To analyze the reflective reports, a category system was developed in an iterative process of moving back and forth between predefined concepts from the literature and data. The final category system consisted of five main categories: "awareness", "new insight/idea", "confirmation of an idea", "intentions for learning", and "planned learning activities". The fifth main category, planned learning activities, was further specified with ‘doing’, ‘experimentation’, ‘reflection’, ‘feedback’, ‘observation’ and ‘studying sources’. The reliability of the system was calculated by assessing the inter-rater reliability. The category systems appeared to be sufficiently reliable. To determine what and how teachers learn and intend to learn, all the 69 reflective reports were analyzed separately by using the category system. Absolute frequencies were calculated for every coding category across the different reflective reports written per self- assessment round. Also, changes in learning outcomes and intended outcomes over time were determined by comparing absolute frequencies of every coding category across the three self-assessment rounds. Findings revealed that being engaged in the self-assessment procedure mainly led to raised awareness of new aspects of coaching students’ reflection skills. To a lesser degree, teachers felt they had learned new theoretical/practical insights concerning coaching students’ reflection skills. Regarding the what, when and how of coaching these skills, teachers mainly reported learning outcomes pertaining to what of coaching students’ reflection skills and on conditions for coaching these skills (how). Learning outcomes showed a cyclical progression consisting of raised awareness, followed by new ideas/insights and confirmation of these ideas, leading to raised awareness again. Finally, it turned out that the teachers predominantly learned from the self-assessment in a performance-oriented way. Based on this study, it was concluded that repeatedly undertaking the self-assessment procedure was useful for the teachers’ learning, but mainly led to awareness and learning to improve performance. To encourage more meaningful or meaning-oriented learning, additional measures in the self-assessment procedure seem to be necessary. Chapter 6 reported on a study that focused on how teachers valued their experiences of being engaged in the self-assessment procedure and what aspects of this procedure they perceived as useful. For this purpose, an item-based questionnaire was used that covered elements of the self-assessment procedure, its underlying design and its impact on teachers’ learning. The 32 items could be scored on the following five-point scale: 1) totally invaluable, 2) not valuable, 3) neutral, 4) valuable, 5) very valuable, and 6) not applicable (n/a). In this study, the position was taken that something that is perceived as valuable based on experiences is a strong indication for its usefulness as well. An exploratory factor analysis with orthogonal rotation (varimax) was conducted on the 32 items. Based on the scee plot, we decided to distinguish three scales which, together, explained 55% of the variance. The items that clustered on the same component suggested that factor 1 represented ‘impact and organization’, factor 2 ‘assessment and feedback’, and factor 3 ‘autonomy’. The internal consistency of the scales (Cronbach’s alpha) appeared to be high. To investigate discriminant validity, scale correlations were calculated. It appeared that the scales were related, though sufficiently distinctive. It was found that all three scales were scored positive by respondents. In general, teachers positively weighted the self-assessment procedure against their efforts and learning outcomes as a result of assessing themselves. In particular, the assessment and feedback component was valued quite positively. It can be concluded that the development of the self-assessment procedure via design principles, based on conditions for learning and quality criteria of assessment led to positive perceptions of teachers’ usefulness of the self-assessment procedure. In Chapter 7, the most important findings and conclusions from the five studies were presented. The self-assessment procedure as developed and investigated in this dissertation can be characterized as a potentially beneficial and useful tool for fostering teachers’ learning. This overall conclusion is based on the following findings. First, criteria and standards appeared to be beneficial and useful for teachers’ learning in the sense that they directed teachers’ attention, provided them with a vocabulary and with the possibility to structure or guide their thoughts when making these explicit in their reflective reports. However, for future self-assessments, it is important to deal adequately with teachers’ tendencies to use high scores when assessing themselves or their colleagues. Second, teachers valued receiving feedback from their colleagues as very positive for their learning, but additional measures are needed for teachers to promote them to be a critical friend. Third, the added value of writing a reflective report for teachers’ learning was based on the necessity to explicate learning outcomes, intended outcomes and plans to realize them. Fourth, including criteria and standards, feedback, and writing a reflective report, as well as undertaking the self-assessment procedure repeatedly, is not sufficient enough for promoting teachers’ meaning-oriented learning. Additional measures for learning through self-assessment seem necessary to stimulate this kind of learning by teachers and to develop their further understanding of their own competencies of coaching students’ reflection skills. Subsequently, the most important findings and conclusions were discussed. This discussion focused on important assumptions underlying the promising nature of self-assessment. The first assumption was that self-assessment is attractive because it is efficient and easy to apply. Results of this thesis indicate that for effective self-assessment in terms of learning gains, an extensive and time-consuming preparation seems necessary, that it takes time for teachers to get used to their role as self-assessor and critical friend and that the development of a self-assessment procedure in itself is difficult and time-consuming. Finally, arguing about the effectiveness of self-assessment requires a norm. It depends on the norm and related expectations about the norm whether self-assessment can be regarded as an attractive means for fostering teachers’ learning. Setting such norms are personal- and context bound and thus not easy to do. The second assumption underlying the promising nature of self-assessment was that self-assessment motivates teachers to learn because it makes them responsible for and owner of their own learning. Based on the positive appreciations of teachers of self-assessment it seems reasonable to assume that self-assessment motivates teachers to develop their competencies in coaching. However, teachers demonstrated different ways of being responsible for their learning and differed in what and how they learned. Based on these findings, it can be questioned to what extent the emphasis on teachers’ ownership and responsibility is really useful for teachers’ learning or to what extent this has to be nuanced for specific teachers under certain conditions. The third assumption underlying the promising nature of self-assessment was that self-assessment provides teachers with feedback to develop their own competencies to coach students’ reflection. Effective feedback answers three questions: "where are you going?", "how are you going?", and "how do you get there?". Self-assessment provides meaningful answers to where they are going; however, the question remains whether a meaningful answer is the right answer. Are teachers themselves able to determine how they are going and to generate the proper feedback for themselves? What and how teachers learn to coach students’ reflection from self-assessment is influenced by what teachers already know about it. This dissertation showed that it is possible for teachers to determine how to get there when it pertains to basic aspects of coaching students’ reflection skills, such as how to coach and conditions for coaching. For more difficult aspects of coaching it seems difficult to decide how to get there. The fourth assumption underlying the promising nature of self-assessment was that self-assessment provides teachers with the knowledge and skills to start a more systematic process of informal learning. It is argued that realizing such a process of informal learning is not self-evident. Not all teachers feel the will or need to self-regulate their learning, have the opportunity to self-regulate their learning in the school, and develop the capacity to self-regulate their own learning. Finally, it was discussed how accounting for the implementation of self-assessment turns out in practice. It appeared difficult and time-consuming to integrate the self-assessment procedure in the schools’ Human Resources Policy. Finally, hours made available for undertaking the self-assessment procedure were made at the cost of other activities teachers needed to perform. The studies presented in this dissertation aimed at gaining insight in the effects of self-assessment for the professional development of VET teachers regarding their coaching competencies in coaching students’ reflection skills. This dissertation provided knowledge and understanding of what contributes to the usefulness of self-assessment. It also provided schools with a self-assessment procedure to be used by teachers to foster their competencies in coaching students’ reflection skills and made explicit what can be understood as ‘good’ coaching of students’ reflection. Limitations to the study presented in this dissertation pertained to its descriptive and exploratory nature,, the absence of pre- and post-test and/or a control group in the design, and the participation of a limited number of teachers from a specific type of vocational education. Also, teacher learning was only investigated by analyzing reflective reports, being a reflection of teachers’ perceptions, and the analysis of the data was limited to an analysis across cases. Future research might focus on the role of feedback of self-assessment and individual differences between teachers in how they use and what they learn from self-assessment

Feeding a monster or doing good? Mining industrial design student feedback at large

In this paper, we describe work in progress, a study of a large database of written university teacher feedback given to Industrial Design students throughout the semester, over several years. This large corpus is captured technically in a database, but conceptually organized into several feedback moments per semester, which will be analyzed using a newly developed framework targeting Industrial Design education at the university level. Essential aspects of this paper focus on the translation of related work in the area of feedback mechanisms for higher education into a tailored framework for feedback in the area of Industrial Design, and the translation of the existing corpus of data into indicators of feedback quality and how feedback is received and further on used by students in their learning process. Outcomes will be used to further investigate how the quality of feedback evolves and to develop better tools for feedback. With this paper we aim at opening the discussion about our intended process and methodology, as well as inspiration for aspects we have missed so far

A Crowdsourced GNSS and PPP Reference Network

Due to its requirements for well-positioned base stations and lengthy times-to-first-fix, the Precise Point Positioning (PPP) techniques for decimeter positioning haven\u27t found widespread application in navigation in smartphones. In principle, a global network of PPP base stations can provide decimeter-level accuracy to various positioning applications, including smartphone-based navigation. However, a global network of PPP base stations that can monitor and gather satellite information is expensive to build and maintain. This disclosure describes a crowdsourced, dense, PPP network and ephemeris-coverage map that can enable GNSS chipsets (e.g., utilized in smartphones and other mobile devices) to obtain decimeter-level positioning accuracy