The Rapid Adoption of SMARTER Teamwork Tools: the System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork

The Rapid Adoption of SMARTER Teamwork Tools: the System for Management, Assessment, Research, Training, Education, and Remediation for TeamworkThe rapid adoption of Team-Maker and the Comprehensive Assessment of Team MemberEffectiveness (CATME) tools for team formation and peer evaluation make it possible to extendtheir success to have a significant impact on the development of team skills in higher education.As of October 19, 2014, the web-based systems have been used by more than 300,000 studentsof more than 6000 faculty at more than 1100 institutions in 59 countries—the figure belowshows the growth of the user base at the end of September.This paper and its accompanying poster will describe progress toward broadening the scope ofthose tools into a complete system for the management of teamwork in undergraduate education.The System for the Management, Assessment, Research, Training, Education, and Remediationof Teamwork (SMARTER Teamwork) has three specific goals: 1) to equip students to work inteams by providing them with training and feedback, 2) to equip faculty to manage student teamsby providing them with information and tools to facilitate best practices, and 3) to equipresearchers to understand teams by broadening the system’s capabilities to collect additionaltypes of data so that a wider range of research questions can be studied through a secureresearcher interface. The three goals of the project support each other in hierarchical fashion:research informs faculty practice, faculty determine the students’ experience, which, if wellmanaged based on research findings, equips students to work in teams. Our strategies forachieving these goals are based on a well-accepted training model that has five elements:information, demonstration, practice, feedback, and remediation.The paper that will be submitted and the poster presented at the conference will focus on newfeatures of the system, the development of training materials, and the deployment of a partnerwebsite that shares information about the SMARTER tools for teamwork and provides basicinformation about teamwork and team management. The observed growth is both penetrationinto the original target market and growth into unexpected markets

Board # 114 : Progress toward Optimizing Student Team Skill Development using Evidence-Based Strategies

The broad goal of this work is to study the effectiveness of various teamwork training interventions. This research requires the use of a common model of teamwork and a system for training, collecting ratings data, and providing feedback. We will leverage the NSF’s prior investment in the CATME system, which meets the research criteria and automates some of the data collection and feedback, which will aid in executing the research protocol consistently. Seven empirical studies will determine the effect sizes of training, practice in teams, practice rating, and feedback interventions on cognitive development (improvement of team skills) and metacognitive development (improvement of self- and peer-evaluation skills). Outcomes. We focus both on cognitive skills related to team-member effectiveness and on metacognitive skills that enable competent self- and peer-evaluation of team members’ effectiveness. An intermediate knowledge-level outcome affects both—developing an improved cognitive model of teamwork. Students must learn what skills are necessary for effective teamwork to be able to develop and evaluate them. Strategies. To achieve these outcomes, we have several strategies. Frame-of-reference training, which is well-established and empirically supported, will align students’ cognitive model of teamwork with ours by teaching students the ways team members can contribute effectively to teams in the five key areas summarized earlier. Experience working in teams and evaluating teamwork will improve team skills and self- and peer-evaluation skills. Experience in teams increases as students work on multiple teams. Rating practice will be accomplished by showing students descriptions or videotapes of fictitious team members and having them rate the contributions these fictitious team members make, in addition to rating themselves and their real teammates following work in teams. Finally, we will examine how the degree to which and manner in which feedback on team skills is provided affect student outcomes. This presentation (Executive Summary and Poster) will provide a valuable update on this project, share various lessons for classroom practice, and provide guidance to other faculty who seek to use CATME in their research

Optimizing Student Team Skill Development Using Evidence-Based Strategies

The critical importance of effective teamwork in engineering is widely recognized. Surprisingly, however, relatively little is known about how to develop teamwork skills in higher education classes, including what factors contribute to effective teamwork, their relative importance in a team\u27s overall performance, and the underlying individual and interpersonal dynamics. Increasing numbers of engineering instructors are adopting instructional practices relying on teamwork, yet many instructors simply form student teams and hope the members individually and collectively learn on their own how to work in teams and succeed in their task(s). Instructors do this because they do not have guidance for a better approach. This research project aims to address this gap in faculty knowledge. The empirical studies conducted as part of this project build on research in engineering education, cognitive psychology, social psychology, and other fields in a coordinated large-scale research project that will provide faculty with needed knowledge and tools to ensure that students learn team skills. The research team is conducting seven separate studies measuring the impact of teamwork training, experience working in teams, practice rating the teamwork of fictitious team members, and giving and receiving peer feedback. The research is measuring each of these effects in real teams on three learning outcomes: improved teamwork knowledge, improved ability to evaluate teamwork, and improved ability to function effectively in teams. These studies will result in practical recommendations for time-pressed faculty to implement

SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork

SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for TeamworkThe rapid adoption of Team-Maker and the Comprehensive Assessment of Team MemberEffectiveness (CATME) tools for team formation and peer evaluation make it possible to extendtheir success to have a significant impact on the development of team skills in higher education.The web-based systems have been used by more than 110,000 students of more than 2400faculty at more than 500 institutions internationally—the figure below shows the growth of theuser base. 2400 The system has had 113,373 unique student users. 2200 Fitted curves are third order. 2000 1800 Faculty and staff 1600 1400 1200 Number of 1000 users 800 Institutions 600 400 Aug Oct. 2005 2012 200 0 0 1 2 3 4 5 6 7 Years since software was releasedThis paper and its accompanying poster will describe strategies for broadening the scope of thosetools into a complete system for the management of teamwork in undergraduate education. TheSystem for the Management, Assessment, Research, Training, Education, and Remediation ofTeamwork (SMARTER Teamwork) has three specific goals: 1) to equip students to work inteams by providing them with training and feedback, 2) to equip faculty to manage student teamsby providing them with information and tools to facilitate best practices, and 3) to equipresearchers to understand teams by broadening the system’s capabilities to collect additionaltypes of data so that a wider range of research questions can be studied through a secureresearcher interface. The three goals of the project support each other in hierarchical fashion:research informs faculty practice, faculty determine the students’ experience, which, if wellmanaged based on research findings, equips students to work in teams. Our strategies forachieving these goals are based on a well-accepted training model that has five elements:information, demonstration, practice, feedback, and remediation.The paper that will be submitted and the poster presented at the conference will focus on newfeatures of the system, the development of training materials, and the deployment of a partnerwebsite that shares information about the SMARTER tools for teamwork and provides basicinformation about teamwork and team management

Stimulus vs memory based performance rating : two cognitive models

M.S.C. Michael Yor

Justifying Aggregation with Consensus-Based Constructs: A Review and Examination of Cutoff Values for Common Aggregation Indices

Management researchers often use consensus-based composition models to examine the antecedents and effects of higher-level constructs. Typically, researchers present three indices, rwg, ICC(1), and ICC(2), to demonstrate agreement and consistency among lower-level units when justifying aggregation. Nevertheless, researchers debate what values for these indices are sufficient. This study examines the distributional characteristics of ICCs and rwg values from three sources: the multilevel literature, a large multinational sample of student teams, and a large sample of randomly generated “pseudo teams.” Our results support existing cutoff criteria for ICCs but suggest that generally accepted values for rwg may, under certain circumstances, reflect pseudo-agreement (i.e., agreement observed among two raters not attributable to the same target). Thus, when there is minimal between-group variance (i.e., low ICCs), it is difficult to determine whether high rwg values reflect agreement or pseudo-agreement. Based on these findings, we provide recommendations to help researchers interpret aggregation indices

SMARTER teamwork system for management, assessment, research, training, education, and remediation for teamwork /

Report on CATM