Peer Assessment Based on Ratings in a Social Media Course

Peer assessment is seen as a powerful supporting tool to achieve scalability in the evaluation of complex assignments in large courses, possibly virtual ones, as in the context of massive open online courses (MOOCs). However, the adoption of peer assessment is slow due in part to the lack of ready-to-use systems. Furthermore, the validity of peer assessment is still under discussion. In this paper, in order to tackle some of these issues, we present as a proof-of-concept of a novel extension of Graasp, a social media platform, to setup a peer assessment activity. We then report a case study of peer assessment using Graasp in a Social Media course with 60 master's level university students and analyze the level of agreement between students and instructors in the evaluation of short individual reports. Finally, to see if both instructor and student evaluations were based on appearance of project reports rather than on content, we conducted a study with 40 kids who rated reports solely on their look. Our results convey the fact that unlike the kid evaluation, which shows a low level of agreement with instructors, student assessment is reliable since the level of agreement between instructors and students was high

Engaging Qualities: factors affecting learner attention in online design studios

This study looks at the qualities of learner-generated online content, as rated by experts, and how these relate to learners’ engagement through comments and conversations around this content. The work uploaded to an Online Design Studio by students across a Design and Innovation Qualification was rated and analysed quantitatively using the Consensual Assessment Technique (CAT). Correlations of qualities to comments made on this content were considered and a qualitative analysis of the comments was carried out. It was observed that design students do not necessarily pay attention to the same qualities in learner-generated content that experts rate highly, except for a particular quality at the first level of study. The content that students do engage with also changes with increasing levels of study. These findings have implications for the learning design of online design courses and qualifications as well as for design institutions seeking to supplement proximate design studios with Online Social Network Services

PeerWise - The Marmite of Veterinary Student Learning

PeerWise is a free online student-centred collaborative learning tool with which students anonymously author, answer, and evaluate multiple choice questions (MCQs). Features such as commenting on questions, rating questions and comments, and appearing on leaderboards, can encourage healthy competition, engage students in reflection and debate, and enhance their communication skills. PeerWise has been used in diverse subject areas but never previously in Veterinary Medicine. The Veterinary undergraduates at the University of Glasgow are a distinct cohort; academically gifted and often highly strategic in their learning due to time pressures and volume of course material. In 2010-11 we introduced PeerWise into 1st year Veterinary Biomolecular Sciences in the Glasgow Bachelor of Veterinary Medicine and Surgery programme. To scaffold PeerWise use, a short interactive session introduced students to the tool and to the basic principles of good MCQ authorship. Students were asked to author four and answer forty MCQs throughout the academic year. Participation was encouraged by an allocation of up to 5% of the final year mark and inclusion of studentauthored questions in the first summative examination. Our analysis focuses on engagement of the class with the\ud tool and their perceptions of its use. All 141 students in the class engaged with PeerWise and the majority contributed beyond that which was stipulated. Student engagement with PeerWise prior to a summative exam was positively correlated to exam score, yielding a relationship that was highly significant (p<0.001). Student perceptions of PeerWise were predominantly positive with explicit recognition of its value as a learning and revision tool, and more than two thirds of the class in agreement that question authoring and answering reinforced their learning. There was clear polarisation of views, however, and those students who did not like PeerWise were vociferous in their dislike, the biggest criticism being lack of moderation by staff

Supporting Answerers with Feedback in Social Q&A

Prior research has examined the use of Social Question and Answer (Q&A) websites for answer and help seeking. However, the potential for these websites to support domain learning has not yet been realized. Helping users write effective answers can be beneficial for subject area learning for both answerers and the recipients of answers. In this study, we examine the utility of crowdsourced, criteria-based feedback for answerers on a student-centered Q&A website, Brainly.com. In an experiment with 55 users, we compared perceptions of the current rating system against two feedback designs with explicit criteria (Appropriate, Understandable, and Generalizable). Contrary to our hypotheses, answerers disagreed with and rejected the criteria-based feedback. Although the criteria aligned with answerers' goals, and crowdsourced ratings were found to be objectively accurate, the norms and expectations for answers on Brainly conflicted with our design. We conclude with implications for the design of feedback in social Q&A.Comment: Published in Proceedings of the Fifth Annual ACM Conference on Learning at Scale, Article No. 10, London, United Kingdom. June 26 - 28, 201

Providing behaviour awareness in collaborative project courses

Several studies show that awareness mechanisms can contribute to enhance the collaboration process among students and the learning experiences during collaborative project courses. However, it is not clear what awareness information should be provided to whom, when it should be provided, and how to obtain and represent such information in an accurate and understandable way. Regardless the research efforts done in this area, the problem remains open. By recognizing the diversity of work scenarios (contexts) where the collaboration may occur, this research proposes a behaviour awareness mechanism to support collaborative work in undergraduate project courses. Based on the authors previous experiences and the literature in the area, the proposed mechanism considers personal and social awareness components, which represent metrics in a visual way, helping students realize their performance, and lecturers intervene when needed. The trustworthiness of the mechanisms for determining the metrics was verified using empirical data, and the usability and usefulness of these metrics were evaluated with undergraduate students. Experimental results show that this awareness mechanism is useful, understandable and representative of the observed scenarios.Peer ReviewedPostprint (published version

Providing behaviour awareness in collaborative project courses

Several studies show that awareness mechanisms can contribute to enhance the collaboration process among students and the learning experiences during collaborative project courses. However, it is not clear what awareness information should be provided to whom, when it should be provided, and how to obtain and represent such information in an accurate and understandable way. Regardless the research efforts done in this area, the problem remains open. By recognizing the diversity of work scenarios (contexts) where the collaboration may occur, this research proposes a behaviour awareness mechanism to support collaborative work in undergraduate project courses. Based on the authors previous experiences and the literature in the area, the proposed mechanism considers personal and social awareness components, which represent metrics in a visual way, helping students realize their performance, and lecturers intervene when needed. The trustworthiness of the mechanisms for determining the metrics was verified using empirical data, and the usability and usefulness of these metrics were evaluated with undergraduate students. Experimental results show that this awareness mechanism is useful, understandable and representative of the observed scenarios.Peer ReviewedPostprint (published version

Can Teacher Training in Classroom Management Make a Difference for Children's Experiences in Preschool? A Preview of Findings from the Foundations of Learning Demonstration

Early evaluation results from Newark, NJ, show that Foundations of Learning improved teachers' classroom management and productivity, reduced children's conflict with peers, and engaged students in the learning tasks of preschool. The intervention was implemented in Head Start programs, community-based child care centers, and public schools

If you build it, will they come? How researchers perceive and use web 2.0

Over the past 15 years, the web has transformed the way we seek and use information. In the last 5 years in particular a set of innovative techniques – collectively termed ‘web 2.0’ – have enabled people to become producers as well as consumers of information. It has been suggested that these relatively easy-to-use tools, and the behaviours which underpin their use, have enormous potential for scholarly researchers, enabling them to communicate their research and its findings more rapidly, broadly and effectively than ever before. This report is based on a study commissioned by the Research Information Network to investigate whether such aspirations are being realised. It seeks to improve our currently limited understanding of whether, and if so how, researchers are making use of various web 2.0 tools in the course of their work, the factors that encourage or inhibit adoption, and researchers’ attitudes towards web 2.0 and other forms of communication. Context: How researchers communicate their work and their findings varies in different subjects or disciplines, and in different institutional settings. Such differences have a strong influence on how researchers approach the adoption – or not – of new information and communications technologies. It is also important to stress that ‘web 2.0’ encompasses a wide range of interactions between technologies and social practices which allow web users to generate, repurpose and share content with each other. We focus in this study on a range of generic tools – wikis, blogs and some social networking systems – as well as those designed specifically by and for people within the scholarly community. Method: Our study was designed not only to capture current attitudes and patterns of adoption but also to identify researchers’ needs and aspirations, and problems that they encounter. We began with an online survey, which collected information about researchers’ information gathering and dissemination habits and their attitudes towards web 2.0. This was followed by in-depth, semi-structured interviews with a stratified sample of survey respondents to explore in more depth their experience of web 2.0, including perceived barriers as well as drivers to adoption. Finally, we undertook five case studies of web 2.0 services to investigate their development and adoption across different communities and business models. Key findings: Our study indicates that a majority of researchers are making at least occasional use of one or more web 2.0 tools or services for purposes related to their research: for communicating their work; for developing and sustaining networks and collaborations; or for finding out about what others are doing. But frequent or intensive use is rare, and some researchers regard blogs, wikis and other novel forms of communication as a waste of time or even dangerous. In deciding if they will make web 2.0 tools and services part of their everyday practice, the key questions for researchers are the benefits they may secure from doing so, and how it fits with their use of established services. Researchers who use web 2.0 tools and services do not see them as comparable to or substitutes for other channels and means of communication, but as having their own distinctive role for specific purposes and at particular stages of research. And frequent use of one kind of tool does not imply frequent use of others as well