Stark, Philip B.

English

Eastern Illinois University

Journal of Collective Bargaining in the AcademyVolume 0 National Center Proceedings 2015 Article 48April 2015Teaching evaluations: class act or class action?Philip B. StarkUC BerkeleyFollow this and additional works at: http://thekeep.eiu.edu/jcbaPart of the Collective Bargaining Commons, and the Higher Education CommonsThis Proceedings Material is brought to you for free and open access by The Keep. It has been accepted for inclusion in Journal of Collective Bargainingin the Academy by an authorized editor of The Keep. For more information, please contact tabruns@eiu.edu.Recommended CitationStark, Philip B. (2015) "Teaching evaluations: class act or class action?," Journal of Collective Bargaining in the Academy: Vol. 0 , Article48.Available at: http://thekeep.eiu.edu/jcba/vol0/iss10/48iching evaluations: class act or class action? Teaching evaluations: class act or class action? Philip B. Stark Department of Statistics University of California, Berkeley National Center for the Study of Collective Bargaining in Higher Education and the Professions Annual Conference Hunter CoUege New York, NY 19-20 April 2015 p://www.stat.berkeley.edu/~stark/Seminars/setNCSCB15.htm Page 1 of 3/24/2( 1Stark: Teaching evaluations: class act or class action?Published by The Keep, 2015iching evaluations: class act or class action? What do SET measure? No consensus. • SET scores are highly correlated with students' grade expectations Marsh & Cooper, 1980; Short et al., 2012; Worthington, 2002 p : //www.stat.berkeley .edu/~stark/Seminars/setNCSCB 15.htm Page 2 of 2 / 19 3/24/2( 2Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/48:iching evaluations: class act or class action? What do SET measure? No consensus. • SET scores are highly correlated with students' grade expectations Marsh & Cooper, 1980; Short et al., 2012; Worthington, 2002 • SET scores & enjoyment scores very strongly correlated Stark, unpublished, 2014 p : //www.stat.berkeley.edu/~stark/Seminars/setNCSCB15.htm Page 3 of 3 / 19 3/24/2( 3Stark: Teaching evaluations: class act or class action?Published by The Keep, 20151ching evaluations: class act or class action? What do SET measure? No consensus. • SET scores are highly correlated with students' grade expectations Marsh & Cooper, 1980; Short et al., 2012; Worthington, 2002 • SET scores & enjoyment scores very strongly correlated Stark, unpublished, 2014 • SET can be predicted from the students' reaction to 30 seconds of silent video of the instructor; physical attractiveness matters Ambady & Rosenthal, 1993 p://www.stat.berkeley .edu/~stark/Seminars/setNCSCB 15.htm Page 4 of 4 I 19 3/24/2( 4Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/48:iching evaluations: class act or class action? What do SET measure? No consensus. • SET scores are highly correlated with students' grade expectations Marsh & Cooper, 1980; Short et al., 2012; Worthington, 2002 • SET scores & enjoyment scores very strongly correlated Stark, unpublished, 2014 • SET can be predicted from the students' reaction to 30 seconds of silent video of the instructor; physical attractiveness matters Ambady & Rosenthal, 1993 • gender, ethnicity, & the instructor's age matter Anderson & Miller, 1997; Basow, 1995; Boring, 2014; Cramer & Alexitch , 2000; Marsh & Dunkin, 1992; McNell et al., 2014; Wachtel, 1998; Weinberg et al. , 2007; Worthington, 2002 p : //www.stat.berkeley.edu/~stark/Seminars/setNCSCB15.htm Page 5 of 5 I 19 3/24/2( 5Stark: Teaching evaluations: class act or class action?Published by The Keep, 2015:i.ching evaluations: class act or class action? What do SET measure? No consensus. • SET scores are highly correlated with students' grade expectations Marsh & Cooper, 1980; Short et al., 2012; Worthington, 2002 • SET scores & enjoyment scores very strongly correlated Stark, unpublished, 2014 • SET can be predicted from the students' reaction to 30 seconds of silent video of the instructor; physical attractiveness matters Ambady & Rosenthal, 1993 • gender, ethnicity, & the instructor's age matter Anderson & Miller, 1997; Basow, 1995; Boring, 2014; Cramer & Alexitch , 2000; Marsh & Dunkin, 1992; McNell et al., 2014; Wachtel, 1998; Weinberg et al., 2007; Worthington, 2002 • omnibus questions about curriculum design, effectiveness, etc., appear most influenced by factors unrelated to learning Worthington, 2002 p://www.stat.berkeley.edu/~stark/Seminars/setNCSCB 15 .htm Page 6 of 6 / 19 3/24/2( 6Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/48iching evaluations: class act or class action? The gold standard: Randomized, controlled experiments p://www.stat.berkeley.edu/~stark/Seminars/setNCSCB15.htm Page 7 of 7 I 19 3/24/2( 7Stark: Teaching evaluations: class act or class action?Published by The Keep, 20151ching evaluations: class act or class action? The gold standard: Randomized, controlled experiments Carrell & West, 2008 United States Air Force Academy assigns students to instructors at random in core courses, including follow-on courses. All sections have identical syllabi and exams. Student evaluations are positively correlated with contemporaneous professor value-added and negatively correlated with follow-on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning (introductory course professor value-added in follow-on courses) . p://www.stat.berkeley .edu/~stark/Seminars/setNCSCB 15 .htm Page 8 of 8 / 19 3/24/2( 8Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/481ching evaluations: class act or class action? The gold standard: Randomized, controlled experiments Carrell & West, 2008 United States Air Force Academy assigns students to instructors at random in core courses, including follow-on courses. All sections have identical syllabi and exams. Student evaluations are positively correlated with contemporaneous professor value-added and negatively correlated with follow-on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning (introductory course professor value-added in follow-on courses). Braga, Paccagnella, & Pellizzari, 2011 Randomized assignment of students to instructors at Bocconi University, Milan in other words, teachers who are associated with better subsequent performance receive worst evaluations from their students. p://www.stat.berkeley.edu/- stark/Seminars/setNCSCB15.htm Page 9 of 3/24/2( 9Stark: Teaching evaluations: class act or class action?Published by The Keep, 2015iching evaluations: class act or class action? Page 10 of McNeil, Driscoll & Hunt, 2014: Gender Bias NC State online course. Adjective F-M Randomized assignments of Caring -0.47 students into 4 groups. 2 instructors, 1 male 1 Consistent -0.57 female. Enthusiastic -0.76 Each instructor was identified Fair -0.47 to students by actual gender in 1 section, false gender in 1 Feedback -0.46 section . Regardless of actual gender, Helpful -0 .35 substantially higher ratings Knowledgeable -0.67 when each instructor was identified as male, even for Praise -0.61 "objective" measures, e.g ., speed of returning Professional -0 .80 homework. 5-point scale. Prompt -0.61 Respectful -0.22 Responsive -0 .61 10 / 19 p://www.stat.berkeley.edu/~stark/Seminars/setNCSCB15.htm 3/24/2( 10Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/48:iching evaluations: class act or class action? Page 11 of Boring, 2014: more evidence of gender bias Male students in particular tend to give higher overall satisfaction scores to male teachers , rewarding them for their perceived higher quality in course delivery style . . . . Male teachers can increase their SET scores by investing more effort in the characteristics that male students tend to value more. However, female teachers must invest more effort improving the teaching dimensions in which students tend to perceive a slight comparative advantage for women , i.e. course structure, organization and teaching material. ... The results suggest that better teaching is not necessarily measured by SETs. p://www. stat.berkeley.edu/~stark/Seminars/setNCSCB15 .htm 11 / 19 3/24/2( 11Stark: Teaching evaluations: class act or class action?Published by The Keep, 20151ching evaluations: class act or class action? Lauer, 2012: Student comments knotty, too Survey of 185 students, 45 faculty at Rollins College, Winter Park, Florida I once believed that narrative comments on course evaluation forms were straightforward and useful. p://www.stat.berkeley.edu/~stark/Seminars/setNCSCB 15 .htm Page 12 of 12 / 19 3/24/2( 12Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/48:i.ching evaluations: class act or class action? Lauer, 2012: Student comments knotty, too Survey of 185 students, 45 faculty at Rollins College, Winter Park, Florida I once believed that narrative comments on course evaluation forms were straightforward and useful. Faculty & students ascribe quite different meanings to words such as "fair," "professional ," "organized," "challenging," & "respectful." p://www. stat.berkeley.edu/~stark/Seminars/setNCSCB15.htm Page 13 of 13 / 19 3/24/2( 13Stark: Teaching evaluations: class act or class action?Published by The Keep, 2015:iching evaluations: class act or class action? Lauer, 2012: Student comments knotty, too Survey of 185 students, 45 faculty at Rollins College, Winter Park, Florida I once believed that narrative comments on course evaluation forms were straightforward and useful. Faculty & students ascribe quite different meanings to words such as "fair," "professional," "organized," "challenging," & "respectful." not fair means ... student% instructor % plays favorites 45.8 31 .7 grading problematic 2.3 49.2 work is too hard 12.7 0 won't "work with you" on problems 12.3 0 other 6.9 19 p: //www.stat.berkeley.edu/~stark/Seminars/setNCSCB 15 .htm Page 14 of 14 / 19 3/24/2( 14Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/48:iching evaluations: class act or class action? Page 15 of Benton & Cashin, 2012: exemplar SET apologists It is difficult to get a man to understand something, when his salary depends upon his not understanding it! -Upton Sinclair • Widely cited, unrefereed technical report from a business that sells SET; flawed statistics p : //www.stat.berkeley .edu/~stark/Serninars/setNCSCB 15.htm 15 / 19 3/24/2( 15Stark: Teaching evaluations: class act or class action?Published by The Keep, 20151ching evaluations: class act or class action? Page 16 of Benton & Cashin, 2012: exemplar SET apologists It is difficult to get a man to understand something , when his salary depends upon his not understanding it! -Upton Sinclair • Widely cited , unrefereed technical report from a business that sells SET; flawed statistics • Rebut straw man positions: 0 Students cannot make consistent judgments. 0 Student ratings are just popularity contests. 0 Students will not appreciate good teaching until they are out of college a few years. 0 Students just want easy courses. 0 Student feedback cannot be used to help improve instruction. p://www.stat.berkeley.edu/~stark/Seminars/setNCSCB15 .htm 16 / 19 3/24/2( 16Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/48iching evaluations: class act or class action? Page 17 of Benton & Cashin, 2012: exemplar SET apologists It is difficult to get a man to understand something , when his salary depends upon his not understanding it! -Upton Sinclair • Widely cited, unrefereed technical report from a business that sells SET; flawed statistics • Rebut straw man positions: 0 Students cannot make consistent judgments. 0 Student ratings are just popularity contests. 0 Students will not appreciate good teaching until they are out of college a few years. 0 Students just want easy courses. 0 Student feedback cannot be used to help improve instruction. • The two non-absolutist statements they reject are demonstrably true: 0 Student ratings are unreliable and invalid. 0 The time of day the course is offered affects ratings. p://www.stat.berkeley.edu/~stark/Seminars/setNCSCB 15.htm 17 I 19 3/24/2( 17Stark: Teaching evaluations: class act or class action?Published by The Keep, 2015:i.ching evaluations: class act or class action? Benton & Cashin, 2012: exemplar SET apologists It is difficult to get a man to understand something , when his salary depends upon his not understanding it! -Upton Sinclair Page 18 of • Widely cited , unrefereed technical report from a business that sells SET; flawed statistics • Rebut straw man positions: 0 Students cannot make consistent judgments. 0 Student ratings are just popularity contests . 0 Students will not appreciate good teaching until they are out of college a few years . 0 Students just want easy courses. 0 Student feedback cannot be used to help improve instruction. • The two non-absolutist statements they reject are demonstrably true: 0 Student ratings are unreliable and invalid . 0 The time of day the course is offered affects ratings. • The remaining statement they reject is true , in my experience as a teacher and department chair: 0 Emphasis on student ratings has led to grade inflation. See also Ewing , 2012 ; lsely & Singh , 2005; Krautmann & Sander, 1999; 18 / 19 McPherson . 2006 p://www.stat.berkeley.edu/- stark/Seminars/setNCSCB 15 .htm 3/24/2( 18Journal of Collective Bargaining in the Academy, Vol. 0, Iss. 10 [2015], Art. 48http://thekeep.eiu.edu/jcba/vol0/iss10/48:iching evaluations: class act or class action? Page 19 of Recommendations 1. Drop omnibus items about "overall teaching effectiveness" and "value of the course" 2. Do not average or compare averages of SET scores: Such averages do not make sense statistically. Instead , report the distribution of scores, the number of responders, and the response rate . 3. Responders are not a random sample and there's no reason their responses should be representative of the class as a whole : do not extrapolate. 4. Pay attention to student comments but understand their limitations and heed differences in language usage. 5. Avoid comparing teaching effectiveness across courses of different types, levels, sizes, functions , or disciplines. 6. Use teaching portfolios as part of the review process. 7. Use classroom observation as part of milestone reviews. 8. To improve teaching and evaluate teaching fairly and honestly, spend time observing teaching & teaching materials. p://www.stat.berkeley.edu/~stark/Seminars/setNCSCB 15.htm 19 / 19 3/24/2( 19Stark: Teaching evaluations: class act or class action?Published by The Keep, 2015

Teaching evaluations: class act or class action?

The Keep

https://thekeep.eiu.edu/cgi/viewcontent.cgi?article=1505&amp;context=jcba

Teaching evaluations: class act or class action?

Abstract

Similar works

Full text

Available Versions

Eastern Illinois University

The Keep