Cox, Monica F

Hahn, Jeeyeon

Kulkarni, Asawaree Ajit

McNeill, Nathan

English

Purdue E-Pubs

Purdue UniversityPurdue e-PubsSchool of Engineering Education Graduate StudentSeries School of Engineering Education2010Developing A Global Real Time Assessment ToolFor The Teaching Enhancement Of EngineeringGraduate Teaching AssistantsNathan McNeillPurdue UniversityJeeyeon HahnPurdue UniversityAsawaree Ajit KulkarniPurdue UniversityMonica F. CoxPurdue University - Main Campus, mfc@purdue.eduFollow this and additional works at: http://docs.lib.purdue.edu/enegsPart of the Engineering Education CommonsThis document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact epubs@purdue.edu foradditional information.McNeill, Nathan; Hahn, Jeeyeon; Kulkarni, Asawaree Ajit; and Cox, Monica F., "Developing A Global Real Time Assessment Tool ForThe Teaching Enhancement Of Engineering Graduate Teaching Assistants" (2010). School of Engineering Education Graduate StudentSeries. Paper 19.http://docs.lib.purdue.edu/enegs/19AC 2010-904: DEVELOPING A GLOBAL REAL-TIME ASSESSMENT TOOL FORTHE TEACHING ENHANCEMENT OF ENGINEERING GRADUATE TEACHINGASSISTANTSMonica Cox, Purdue UniversityMonica F. Cox, Ph.D., is an Assistant Professor in the School of Engineering Education at PurdueUniversity. She obtained a B.S. in mathematics from Spelman College, a M.S. in industrialengineering from the University of Alabama, and a Ph.D. in Leadership and Policy Studies fromPeabody College of Vanderbilt University. Teaching interests relate to the professionaldevelopment of graduate engineering students and to leadership, policy, and change in science,technology, engineering, and mathematics education. Primary research projects explore thepreparation of engineering doctoral students for careers in academia and industry and thedevelopment of engineering education assessment tools. She is a NSF Faculty Early Career(CAREER) award winner and is a recipient of a Presidential Early Career Award for Scientistsand Engineers (PECASE). Jeeyeon Hahn , Purdue UniversityJeeyeon Hahn is a PhD student in the School of Aeronautics and Astronautics at PurdueUniversity. She obtained B.S. and M.S. in Aeronautical and Astronautical Engineering fromPurdue University. Her primary research interests relate to the Fatigue of Structures andMaterials. Nathan McNeill, Purdue UniversityNathan McNeill is a Ph.D. candidate in the School of Engineering Education at PurdueUniversity. He obtained a B.S. in engineering from Walla Walla University and an M.S. inmechanical engineering from The Georgia Institute of Technology. His current research workfocuses on the outcomes of globally oriented engineering education experiences. He has also beeninvolved in the development of tools for the direct observation of pedagogy employed inlaboratory teaching environments. Asawaree Kulkarni, Purdue UniversityAsawaree A. Kulkarni, graduated from Purdue University's College of Technology with a degreeof M.S. in Computer and Information Technology. She obtained her B.S. in ComputerEngineering from University of Pune, India. Her research interests mainly consist of UnifiedCommunications and VoIP Security and her Master's thesis was titled "Performance Evaluationof an Open Source Speaker Recognition System under the effect of Speech Compression in a realworld VoIP Environment". As a graduate research assistant she worked on projects under theareas of Steganography and Image Processing. Asawaree worked with Purdue's School ofEngineering Education to design and develop software for the Global Real-time Assessment Toolfor Teaching Enhancement (G-RATE). © American Society for Engineering Education, 2010 Page 15.375.1Developing a Global Real-time Assessment Tool for the Teaching Enhancement of Engineering Graduate Teaching Assistants   Abstract  In 1999, the VaNTH Observation System (VOS), a direct observation system that records instructor and student interactions in classrooms using the dimensions of the “How People Learn” (HPL) framework was developed to provide pedagogical feedback to bioengineering faculty at four research universities. Since its development, the VOS has been adapted for use in first year engineering laboratories taught by the graduate teaching assistants (GTAs) at a large Midwestern university. After piloting the VOS within this new environment in 2007, researchers noted some limitations. To overcome these limitations, a new pedagogical observation system, called the Global Real-time Assessment Tool for Teaching Enhancement (G-RATE), has been developed to provide GTAs with multidimensional feedback about their teaching. This paper describes the evolution of the G-RATE system, provides an overview of each section of the instrument, and discusses future steps for the development of this tool.  Introduction  Seymour et al.1 note the importance of pedagogical feedback for graduate teaching assistants (GTAs) in a variety of disciplines. In a comparison of GTAs teaching in innovative and traditional course environments, the authors identified GTA responsibilities in traditional environments to include grading, working closely with undergraduate, and serving as secondary resources for students enrolled in the course. Within the innovative environments, however, GTAs were found to be troubleshooters, liaisons between faculty supervisors and students, and collaborators with faculty in the teaching of students enrolled in the course.   Because of the important roles that GTAs play in both traditional and innovative environments, feedback to GTAs about the extent to which they are fulfilling their teaching roles is needed. Although such feedback has been traditionally provided by GTAs’ coordinators or supervisors2, by GTA self-reflections2, 3, by undergraduate students’ evaluations4, or by other surveys4, direct measures of GTA effectiveness are needed also. For this reason, this paper describes a new pedagogical system, the Global Real-time Assessment Tool for Teaching Enhancement (G-RATE), which has been developed to provide GTAs with multidimensional feedback about their teaching. This paper describes the evolution of the G-RATE system, provides an overview of each section of the instrument, and discusses future steps for the development of this tool.     Page 15.375.2Background  Prior to developing the G-RATE, researchers used the VaNTH Observation System (VOS)5 to collect real-time, in-class data about GTAs’ pedagogical practices in first-year engineering laboratories at a Midwestern university over the course of two semesters. Such observations were needed to understand what data the VOS did and did not capture in laboratory environments, and how this data differed from that collected in classroom environments for which the VOS was designed. Of the four components of the VOS, the Classroom Interaction Observation (CIO) portion was used to collect data using who, to whom, what, how, and media categories (Figure 1), since data were recorded in four to six second intervals. Although  the codes for each category are described in detail in Harris and Cox5, a pictorial representation of the five categories is displayed in Figure 1. A revised version of the CIO is located on a Personal Digital Assistant (PDA) that was used by trained classroom observers (i.e., three graduate assistants and one undergraduate researcher).    Figure 1. Revised data collection system for the VaNTH Observation System (VOS)   To collect data with the CIO portion of the VOS, observers sat in three-hour lab classes and observed a GTA supervising thirty undergraduate engineering students. The observer recorded observations of the instruction at regular intervals by selecting corresponding codes under each category, thereby resulting in the creation of hundreds of code strings per instructional session.   Despite this collection of code string data, the use of the VOS in laboratory settings was hampered primarily by two factors6. First, the VOS required considerable training to use. Careful training was required to achieve accuracy in four- to six- second coding intervals. Second, not all Page 15.375.3categories were intuitive. While the who, to whom, and media categories were straightforward, the what and how categories required additional interpretation by the observer depending on the types of classroom interactions occurring in the classroom. Also, the how category, which contains the four HPL dimensions of knowledge-centered (K), learner-centered (L), assessment-centered (A), and community-centered (C), along with classroom organization (O), was difficult to assess because of the subjective nature of this items.  Based upon these observation using the VOS, researchers suggested that the following elements be incorporated into an improved tool.: ≠ The tool should be easy to use and not require extensive observer training.  ≠ A complete description of the codes and what they depict should be integrated into the tool so that users do not have to memorize the codes and their definitions. ≠ The who and to whom categories should be customizable according to the particular environments (i.e., laboratory or classroom) where the tool is to be used. ≠ A help function should be provided to help users. Example questions and definitions would correspond to these options.  Development of the G-RATE  In response to the needs listed previously, the Global Real-time Assessment Tool for Teaching Enhancement (G-RATE) was developed. Figure 2 shows the start page of the G-RATE, from which a user selects his/her role, or function, in using the tool. The G-RATE consists of the Observer, Student, Graduate Teaching Assistant, Researcher, and Administrator functions. The descriptions of these functions are: ≠ Administrator: An administrator can modify the observation parameters of the G-RATE (e.g., modifying the code strings of the observation categories and selecting appropriate questions for GTAs and undergraduate students). This occurs at the beginning of the lab session. ≠ Observer: An observer can record the observational data that reflect GTAs’ instructional practices in elements of the How People Learn (HPL) framework7 during lab sessions.  ≠ Student: Students who are enrolled in the observed lab session can answer the formative and/or summative survey items which provide feedback to the instructional GTA. This occurs during the last ten minutes of the lab session. ≠ GTA: The GTAs can access to the reflective items that can be used for self-assessment of their teaching after the lab session. ≠ Researcher: A researcher can access to data which are collected from the other G-RATE functions without overriding modifications. The researcher accesses these profiles after all other data have been collected. Page 15.375.4Training materials can be accessed by clicking on the buttons at the bottom of the interface. These tutorial sessions provide training in system operation as well as instruction about the HPL framework, the theoretical framework on which the G-RATE instrument is based.     Figure 2. Start page of the G-RATE  System Requirements of G-RATE The G-RATE was designed using the Microsoft Visual C# programming language. The G-RATE requires the Windows XP or Vista operating system and 3 MB free hard disk space as well as an eight inch or larger screen. Microsoft Excel and Microsoft Access are required for generating observation reports.  Current status  At present, only the development of the observer, administrator, and training functions has been completed. Figure 3 shows the page that is used for recording observational data using the observer function. The G-RATE uses descriptive codes rather than abbreviated codes that were used with the VOS interface (e.g., “higher order question” rather than “HQ”). To help observers learn the codes, tooltip and help functions have been provided. Clicking on the exclamation mark beside a code produces a popup tooltip that explains what code means and provides examples of how it is to be used. A user can also access the help function at any time during observation for general help on such as interface use or information on the How People Learn framework.  Another capability which has been added to the G-RATE tool is an automatic coding interval option. With this option, the data that the observer enters is automatically stored at preset time intervals. This helps observers to enter data at precise intervals.  Page 15.375.5 Figure 3. Observer function page of the G-RATE  The administrator function allows a user to customize some observational interface features (Figure 4). Under the index tab of the administrator function, the supervisor of the observations can modify the elements of the who and to whom categories to meet the conditions of the environment in which the tool is being used. In addition, on the user page tab, the supervisor can set a password so that only authorized users can access the database in which data is store.     Figure 4. Administrator function page of the G-RATE  The help function provides training materials to help users familiarize themselves with the G-RATE and the How People Learn framework (Figure 5). The help window can be loaded at any time by pressing a help button that is located on every window in the interface.  Page 15.375.6 r  Figure 5. Help window of the G-RATE  Future Work  Although initial work has occurred on the G-RATE, several tasks remain. First, the remaining parts of the G-RATE (i.e., the researcher, undergraduate, and GTA functions) are yet to be fully developed. Second, in order to provide feedback from various groups to GTAs and to study their performance for certain periods, the databases of each function that are stored under each GTA will be combined as an interlocking database. Third, a simple feedback report template, which can easily be interpreted by someone who has no extensive knowledge of the HPL framework, will be developed. Once the report template is developed, observation reports will be generated by clicking on an ‘Analyze’ button on the observer function page. Finally, when the system is fully developed, the G-RATE will be tested and validated in a first-year engineering laboratory course.  Conclusions  The G-RATE is a tool that provides multidimensional direct observational feedback to GTAs on the instructional interactions that they are using teaching in a classroom. The G-RATE is an adaptation of the earlier VOS tool and provides flexible and easy-to-use interface which allows the tool to be used outside of traditional classroom environments. The G-RATE be also be used to collect data from a variety of stakeholders including undergraduate students, GTAs, researchers, and administrators in various observation environments.   Acknowledgements  The work has been funded by National Science Foundation grant # 0648380.  Page 15.375.7References  1. Seymour, E., Partners in Innovation: Teaching Assistants in College Science Courses. Lanham, MD: Rowman & Littlefield, 2005. 2. Prieto, L.R. and S.A. Meyers, The Teaching Assistant Training Handbook: How To Prepare TAs for Their Responsibilities. Stillwater, OK: New Forums Press, 2001. 3. Robinson, J.B., "New Teaching Assistants Facilitate Active Learning in Chemistry Laboratories: Promoting Teaching Assistant Learning through Formative Assessment and Peer Review," Journal of Graduate Teaching Assistant Development, vol. 7, pp. 147-162, 2000. 4. Black, B. and M. Kaplan, A guidebook for University of Michigan graduate student instructors: Center for Research on Learning and Teaching, 1998. 5. Harris, A.H. and M.F. Cox, "Developing an Observation System to Capture Instructional Differences in Engineering Classrooms," Journal of Engineering Education, vol. 92, pp. 329-336, 2003. 6. McNeill, N.J., M.F. Cox, H.A. Diefes-Dux, T. Medley, and J. Hayes, "Development of an Instrument to Collect Pedagogical Data from Graduate Teaching Assistants within Engineering Laboratories," in American Society for Engineering Education Annual Conference & Exposition Pittsburgh, PA, 2008. 7.            Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds). How People Learn: Brain, Mind,                 Experience, and School. Washington, DC: National Academy Press, 1999.    Page 15.375.8

Developing A Global Real Time Assessment Tool For The Teaching Enhancement Of Engineering Graduate Teaching Assistants

https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1018&context=enegs

Developing A Global Real Time Assessment Tool For The Teaching Enhancement Of Engineering Graduate Teaching Assistants

Abstract

Similar works

Full text

Available Versions

Purdue E-Pubs