Evaluating the extent of a large-scale transformation in gateway science courses

We evaluate the impact of an institutional effort to transform undergraduate science courses using an approach based on course assessments. The approach is guided by A Framework for K-12 Science Education and focuses on scientific and engineering practices, crosscutting concepts, and core ideas, together called three-dimensional learning. To evaluate the extent of change, we applied the Three-dimensional Learning Assessment Protocol to 4 years of chemistry, physics, and biology course exams. Changes in exams differed by discipline and even by course, apparently depending on an interplay between departmental culture, course organization, and perceived course ownership, demonstrating the complex nature of transformation in higher education. We conclude that while transformation must be supported at all organizational levels, ultimately, change is controlled by factors at the course and departmental levels

Aligning Practice to Policies: Changing the Culture to Recognize and Reward Teaching at Research Universities

Recent calls for improvement in undergraduate education within STEM (science, technology, engineering, and mathematics) disciplines are hampered by the methods used to evaluate teaching effectiveness. Faculty members at research universities are commonly assessed and promoted mainly on the basis of research success. To improve the quality of undergraduate teaching across all disciplines, not only STEM fields, requires creating an environment wherein continuous improvement of teaching is valued, assessed, and rewarded at various stages of a faculty member’s career. This requires consistent application of policies that reflect well-established best practices for evaluating teaching at the department, college, and university levels. Evidence shows most teaching evaluation practices do not reflect stated policies, even when the policies specifically espouse teaching as a value. Thus, alignment of practice to policy is a major barrier to establishing a culture in which teaching is valued. Situated in the context of current national efforts to improve undergraduate STEM education, including the Association of American Universities Undergraduate STEM Education Initiative, this essay discusses four guiding principles for aligning practice with stated priorities in formal policies: 1) enhancing the role of deans and chairs; 2) effectively using the hiring process; 3) improving communication; and 4) improving the understanding of teaching as a scholarly activity. In addition, three specific examples of efforts to improve the practice of evaluating teaching are presented as examples: 1) Three Bucket Model of merit review at the University of California, Irvine; (2) Evaluation of Teaching Rubric, University of Kansas; and (3) Teaching Quality Framework, University of Colorado, Boulder. These examples provide flexible criteria to holistically evaluate and improve the quality of teaching across the diverse institutions comprising modern higher education

Characterizing college science instruction: The Three-Dimensional Learning Observation Protocol

The importance of improving STEM education is of perennial interest, and to this end, the education community needs ways to characterize transformation efforts. Three-dimensional learning (3DL) is one such approach to transformation, in which core ideas of the discipline, scientific practices, and crosscutting concepts are combined to support student development of disciplinary expertise. We have previously reported on an approach to the characterization of assessments, the Three-Dimensional Learning Assessment Protocol (3D-LAP), that can be used to identify whether assessments have the potential to engage students in 3DL. Here we present the development of a companion, the Three-Dimensional Learning Observation Protocol (3D-LOP), an observation protocol that can reliably distinguish between instruction that has potential for engagement with 3DL and instruction that does not. The 3D-LOP goes beyond other observation protocols, because it is intended not only to characterize the pedagogical approaches being used in the instructional environment, but also to identify whether students are being asked to engage with scientific practices, core ideas, and crosscutting concepts. We demonstrate herein that the 3D-LOP can be used reliably to code for the presence of 3DL; further, we present data that show the utility of the 3D-LOP in differentiating between instruction that has the potential to promote 3DL from instruction that does not. Our team plans to continue using this protocol to evaluate outcomes of instructional transformation projects. We also propose that the 3D-LOP can be used to support practitioners in developing curricular materials and selecting instructional strategies to promote engagement in three-dimensional instruction

Adapting Assessment Tasks To Support Three-Dimensional Learning

As chemists, we understand that science is more than a set of disconnected facts. It is a way of investigating and understanding our natural world that involves things like asking questions, analyzing data, identifying patterns, constructing explanations, developing and using models, and applying core concepts to other situations. This paper uses the concept of three-dimensional (3D) learning, presented in <i>A Framework for K-12 Science Education</i>, to reconceptualize and develop assessment items that require students to integrate chemistry core ideas with scientific practices and crosscutting concepts. Developing 3D assessments from scratch is time-consuming and beyond the scope of most faculty work. Here we present an alternate approach: We provide a detailed description of ways in which instructors can take current assessment questions and modify them to align with three-dimensional learning by focusing on the evidence that is sought about what students know and can do with their knowledge

Aligning Practice to Policies: Changing the Culture to Recognize and Reward Teaching at Research Universities.

Recent calls for improvement in undergraduate education within STEM (science, technology, engineering, and mathematics) disciplines are hampered by the methods used to evaluate teaching effectiveness. Faculty members at research universities are commonly assessed and promoted mainly on the basis of research success. To improve the quality of undergraduate teaching across all disciplines, not only STEM fields, requires creating an environment wherein continuous improvement of teaching is valued, assessed, and rewarded at various stages of a faculty member's career. This requires consistent application of policies that reflect well-established best practices for evaluating teaching at the department, college, and university levels. Evidence shows most teaching evaluation practices do not reflect stated policies, even when the policies specifically espouse teaching as a value. Thus, alignment of practice to policy is a major barrier to establishing a culture in which teaching is valued. Situated in the context of current national efforts to improve undergraduate STEM education, including the Association of American Universities Undergraduate STEM Education Initiative, this essay discusses four guiding principles for aligning practice with stated priorities in formal policies: 1) enhancing the role of deans and chairs; 2) effectively using the hiring process; 3) improving communication; and 4) improving the understanding of teaching as a scholarly activity. In addition, three specific examples of efforts to improve the practice of evaluating teaching are presented as examples: 1) Three Bucket Model of merit review at the University of California, Irvine; (2) Evaluation of Teaching Rubric, University of Kansas; and (3) Teaching Quality Framework, University of Colorado, Boulder. These examples provide flexible criteria to holistically evaluate and improve the quality of teaching across the diverse institutions comprising modern higher education

Comparison of two exams from each discipline, displaying the percentage of exam points assigned to tasks coded with scientific practices, crosscutting concepts, and core ideas.

<p>This representation shows that although there are few zero dimensional tasks in physics, the vast majority of the tasks address core ideas, and the other two dimensions are almost never elicited.</p

Comparison of two exams characterized using the 3D-LAP.

<p>The first row of each diagram shows the question number. In the last three rows, blue, green, and red shaded cells, indicate there is evidence for a scientific practice, crosscutting concept, or core idea, respectively. Questions 21–23 on the Chemistry B exam are constructed response. All other questions shown are selected response.</p