Engineering Faculty Perspectives on the Nature of Quality Teaching

There is wide agreement that teaching quality matters in higher education, but faculty have varied ideas about the definition of quality. Faculty definitions of quality teaching were coded using an existing framework. The most common definition of teaching quality (held by 49% of participants) is associated with elitism and restricted access—the best way to improve education is to admit better students. These faculty focus on education as “knowledge transfer” and “learning content.” Another 38% of faculty had a transformational perspective, more focused on process than content, valuing “empowering students,” “developing students,” and “creating an environment for learning.” These faculty refer to pedagogies of engagement such as active learning. The only other prevalent definition of quality (30% of faculty) focused on “fitness for purpose,” characterized by terms such as “ability to meet specific legitimate learning objectives” and “mastery of learning outcomes.” This work provides guidance to faculty development efforts

Developing an Instrument to Assess the Effects of Pre-College Engineering Participation on the Experiences of First-Year Engineering Students

In this Complete Research paper, we describe the development of a survey instrument to measure the ways that students experience the transition from pre-college engineering activities to first-year engineering programs. As the number of opportunities to study and do engineering prior to matriculation in an undergraduate engineering program increases, first-year engineering students draw from a diverse range of pre-college engineering experiences that affect their transition to studying engineering at a university. The instrument utilizes a theoretical framework developed via a phenomenographic interview process that identified five distinct ways students experience the transition from pre-college to university engineering. These range from foreclosure or a feeling of entrapment in engineering, to frustration, to tedium, to connection, to the ability to help others be successful in first-year engineering. Utilizing the interview data that informed the development of these categories, we identified statements associated with each of these ways of experiencing the transition from pre-college to university engineering and used these statements to develop an initial instrument consisting of 65 Likert-type questions on students’ experiences combined with detailed questions on both the types pre-college engineering experiences the students participated in and the content of these experiences. Validation of the initial instrument involved multiple rounds of feedback from experts in both pre-college and first-year engineering education, followed by an initial administration of the instrument to the first-year student population at two universities. Analysis of these results showed that overall the instrument had good reliability, however we identified 15 low functioning items for removal, reducing the total number of items to 50. Upon completion of the development process, we administered the final instrument again to another population of first-year engineering students. Analysis of these results using Exploratory Factor Analysis yields components that align well with most elements of the aforementioned theoretical framework. However, we identified several additional independent factors related to ways that students experience disconnects or frustration when transitioning from pre-college to first-year engineering programs. Pre-college engineering is growing, but students arrive in first-year engineering programs with varying levels of prior exposure to engineering. Understanding how pre-college experiences affect students’ transitions to engineering will provide valuable data for both the creators and instructors of pre-college and first-year engineering curricula, and facilitate better alignment between these interrelated spheres of engineering education

Board # 114 : Progress toward Optimizing Student Team Skill Development using Evidence-Based Strategies

The broad goal of this work is to study the effectiveness of various teamwork training interventions. This research requires the use of a common model of teamwork and a system for training, collecting ratings data, and providing feedback. We will leverage the NSF’s prior investment in the CATME system, which meets the research criteria and automates some of the data collection and feedback, which will aid in executing the research protocol consistently. Seven empirical studies will determine the effect sizes of training, practice in teams, practice rating, and feedback interventions on cognitive development (improvement of team skills) and metacognitive development (improvement of self- and peer-evaluation skills). Outcomes. We focus both on cognitive skills related to team-member effectiveness and on metacognitive skills that enable competent self- and peer-evaluation of team members’ effectiveness. An intermediate knowledge-level outcome affects both—developing an improved cognitive model of teamwork. Students must learn what skills are necessary for effective teamwork to be able to develop and evaluate them. Strategies. To achieve these outcomes, we have several strategies. Frame-of-reference training, which is well-established and empirically supported, will align students’ cognitive model of teamwork with ours by teaching students the ways team members can contribute effectively to teams in the five key areas summarized earlier. Experience working in teams and evaluating teamwork will improve team skills and self- and peer-evaluation skills. Experience in teams increases as students work on multiple teams. Rating practice will be accomplished by showing students descriptions or videotapes of fictitious team members and having them rate the contributions these fictitious team members make, in addition to rating themselves and their real teammates following work in teams. Finally, we will examine how the degree to which and manner in which feedback on team skills is provided affect student outcomes. This presentation (Executive Summary and Poster) will provide a valuable update on this project, share various lessons for classroom practice, and provide guidance to other faculty who seek to use CATME in their research

Optimizing Student Team Skill Development Using Evidence-Based Strategies

The critical importance of effective teamwork in engineering is widely recognized. Surprisingly, however, relatively little is known about how to develop teamwork skills in higher education classes, including what factors contribute to effective teamwork, their relative importance in a team\u27s overall performance, and the underlying individual and interpersonal dynamics. Increasing numbers of engineering instructors are adopting instructional practices relying on teamwork, yet many instructors simply form student teams and hope the members individually and collectively learn on their own how to work in teams and succeed in their task(s). Instructors do this because they do not have guidance for a better approach. This research project aims to address this gap in faculty knowledge. The empirical studies conducted as part of this project build on research in engineering education, cognitive psychology, social psychology, and other fields in a coordinated large-scale research project that will provide faculty with needed knowledge and tools to ensure that students learn team skills. The research team is conducting seven separate studies measuring the impact of teamwork training, experience working in teams, practice rating the teamwork of fictitious team members, and giving and receiving peer feedback. The research is measuring each of these effects in real teams on three learning outcomes: improved teamwork knowledge, improved ability to evaluate teamwork, and improved ability to function effectively in teams. These studies will result in practical recommendations for time-pressed faculty to implement

SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork

The rapid adoption of Team-Maker and the Comprehensive Assessment of Team Member Effectiveness (CATME), tools for team formation and peer evaluation, make it possible to extend their success to have a significant impact on the development of team skills in higher education. The web-based systems are used by over 700 faculty at over 200 institutions internationally. This paper and its accompanying poster will describe strategies for broadening the scope of those tools into a complete system for the management of teamwork in undergraduate education. The System for the Management, Assessment, Research, Training, Education, and Remediation of Teamwork (SMARTER Teamwork) has three specific goals: 1) to equip students to work in teams by providing them with training and feedback, 2) to equip faculty to manage student teams by providing them with information and tools to facilitate best practices, and 3) to equip researchers to understand teams by broadening the system’s capabilities to collect additional types of data so that a wider range of research questions can be studied through a secure researcher interface. The three goals of the project support each other in hierarchical fashion: research informs faculty practice, faculty determine the students’ experience, which, if well managed based on research findings, equips students to work in teams. Our strategies for achieving these goals are based on a well-accepted training model that has five elements: information, demonstration, practice, feedback, and remediation. Different outcomes are expected for each group of people. For the students, both individual outcomes, such as student learning, and team outcomes, such as the development of shared mental models, are expected. For the faculty, individual outcomes such as faculty learning and faculty satisfaction are expected. The outcomes for researchers will be community outcomes, that is, benefits for stakeholders outside the research team, such as generating new knowledge for teaming theory and disseminating best practices. Measuring these outcomes is the basis for the project’s evaluation plan

SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork

SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for TeamworkThe rapid adoption of Team-Maker and the Comprehensive Assessment of Team MemberEffectiveness (CATME) tools for team formation and peer evaluation make it possible to extendtheir success to have a significant impact on the development of team skills in higher education.The web-based systems have been used by more than 110,000 students of more than 2400faculty at more than 500 institutions internationally—the figure below shows the growth of theuser base. 2400 The system has had 113,373 unique student users. 2200 Fitted curves are third order. 2000 1800 Faculty and staff 1600 1400 1200 Number of 1000 users 800 Institutions 600 400 Aug Oct. 2005 2012 200 0 0 1 2 3 4 5 6 7 Years since software was releasedThis paper and its accompanying poster will describe strategies for broadening the scope of thosetools into a complete system for the management of teamwork in undergraduate education. TheSystem for the Management, Assessment, Research, Training, Education, and Remediation ofTeamwork (SMARTER Teamwork) has three specific goals: 1) to equip students to work inteams by providing them with training and feedback, 2) to equip faculty to manage student teamsby providing them with information and tools to facilitate best practices, and 3) to equipresearchers to understand teams by broadening the system’s capabilities to collect additionaltypes of data so that a wider range of research questions can be studied through a secureresearcher interface. The three goals of the project support each other in hierarchical fashion:research informs faculty practice, faculty determine the students’ experience, which, if wellmanaged based on research findings, equips students to work in teams. Our strategies forachieving these goals are based on a well-accepted training model that has five elements:information, demonstration, practice, feedback, and remediation.The paper that will be submitted and the poster presented at the conference will focus on newfeatures of the system, the development of training materials, and the deployment of a partnerwebsite that shares information about the SMARTER tools for teamwork and provides basicinformation about teamwork and team management

The effect of college costs and financial aid on access to engineering

Financial factors such as tuition costs and financial aid have substantial influences on college access. Prior studies have examined how financial factors influence cohort patterns of incoming students. Our study adds to that body of work by studying institutional differences in the effect of college costs and financial aid on access. We particularly focus on engineering students and explore access of an important underrepresented group in engineering—students of low socioeconomic status. We utilize two large databases: the Integrated Postsecondary Education Data System (IPEDS) and the Multiple Institution Database for Investigation of Engineering Longitudinal Development (MIDFIELD). We employ descriptive statistics, nonparametric test, and a difference-in-difference regression model to determine the relationship between financial factors and engineering cohort patterns. We demonstrate the inability of grant aid to match the pace of rising tuition and fees, and identify different trends between institutions with and without merit-based scholarships. The adoption of merit-based scholarship was positively correlated to in-state student enrollment, engineering first-time student enrollment, and the fraction of students with high socioeconomic status. Compared to the overall institutional effect of merit-based scholarships, engineering experienced a larger increase in the fraction of students with high socioeconomic status. The scholarship effect was not consistently related to in-state students’ SAT scores. Variations in significance and direction exist in the results across institutions

Engineering ethics curriculum incorporation methods and results from a nationally administered standardized examination background, literature, and research methods /

The ethics literature within the engineering arena is long on opinions, but short on evidence as to the most effective curriculum models for incorporating an understanding of professional and ethical responsibility. Research related to professional ethics has primarily focused on assessment of student learning, rather than evaluation of curriculum integration methods. A limited number of studies have been published that compare two methods of curriculum integration, yet no rigorous studies that compare multiple methods of curriculum incorporation are known to exist. Without clear evidence of best methods, the debate will continue, and there will be no assurance that the methods currently in use are the most effective. Within this paper, a recently completed research program is described that evaluated the methods of assimilating ethics into the engineering curriculum to determine if a relationship exists between the curriculum models and the outcome on a nationally administered examination, engineering-specific standardized examination. The study’s population was engineering students during the time period between October 1996 and April 2005 enrolled at nine academic institutions in the southeast United States for which valuable data are available. A mixed-methods (quantitative and qualitative) research program was designed and executed. The qualitative aspects of the study focused on research questions related to the impetus and considerations given to curriculum changes made by the twenty-three discrete engineering programs that participated in the study. The qualitative research questions were investigated through a process of semi-structured interviews conducted with program representatives and evaluation of an extensive number of ABET Self-Study accreditation documents. Once the curriculum models utilized by the participating programs were identified and defined for the chronological limits of the study, a quantitative process was implemented to compare the curriculum models to performance on the ethics section of the Fundamentals of Engineering Examination. A student-level dataset of subject scores was obtained for all administrations of the Fundamentals of Engineering Examination for each of the participating programs. Statistical techniques were utilized to evaluate the relationship between curriculum methods and examination performance. This paper will provide a statement of the perceived educational issues and a comprehensive summary of the applicable literature. A detailed discussion of the study’s design and implemented methods will be presented. Subsequent publications will present the findings, discussion, and implications resulting from the completed study. This study was executed to fulfill dissertation research requirements associated with doctoral program in Engineering Education at Purdue University

How to assign individualized scores on a group project: an empirical evaluation

One major challenge in using group projects to assess student learning is accounting for the differences of contribution among group members so that the mark assigned to each individual actually reflects their performance. This research addresses the validity of grading group projects by evaluating different methods that derive individualized scores from group work. Both Monte Carlo simulation and real test data analyses were conducted. The four investigated methods are the within-group adjustment method, the partial adjustment method, the between-group adjustment method, and the expected contribution adjustment method. For all methods, a weighting factor is computed based on the peer and self ratings of contributions to the group project by group members. This study finds that individual differences have to be taken into account if group grades are going to be assigned and utilized for evaluating individual performance at all. Adjusting contribution differences based on peer and self ratings could be an effective way to improve the validity of group grades. Among the four studied methods, adjusting both the within-group and between-group contribution differences is the most effective, and is thus recommended for classroom use

Journeys into pre-college engineering a comparison of practices and policies in Australia and the United States /

Backround: In recent years, national reports in both Australia and the United States have called for increasing the number of engineers as a means of ensuring national prosperity. Both countries have also identified that this goal begins with primary and secondary schools through both increasing the number of students with the required science and mathematics abilities to be successful in engineering and developing programs to introduce students to engineering. While technology education has long been a part of the school curricula of both countries, the formal inclusion of engineering represents a relatively recent phenomenon. The development and definition of pre-college engineering is rapidly changing in both countries, presenting numerous opportunities to learn from comparisons of the approaches taken in Australia and the United States. Purpose: To explore the similarities and differences in the treatment of pre-college engineering in the USA and Australia. Design/Method: This research is based on reviews of national and state reports addressing pre-college engineering education in Australia and the United States. We compare how engineering content is being incorporated into national and state curriculum frameworks. Finally, we review the different types and providers of pre-college engineering activities, and explore the inclusion of pre-college engineering in the engineering education research community of each country. Results: Similarities exist across the two countries in focusing on the shortage of engineers in the labour market and the role that engineers play in maintaining national security and prosperity. There is variation in the inclusion of engineering in the educational programs of different states. However, the United States has a more developed pre-college engineering ecosystem, including a greater presence in the engineering education community. Conclusions: Based on the results of this research, a tremendous opportunity exists for the Australian engineering education community to bring a greater research focus on P-12 engineering activities. Many states in Australia have developed 11th and 12th year engineering courses that could serve as a model for schools in the United States interested in implementing similar programs. Finally, national curriculum frameworks affecting engineering are in the process of being revised in both countries, which will have a significant effect on P-12 engineering in the schools in upcoming years