Data Driven Course Improvements: Using Artifact Analysis to Conquer ABET Criterion 4

This evidence based practice describes a process to evaluate a course within the spirit of ABET Criteria 4, continuous improvement. Faculty and staff are often asked to collaborate on the design and instruction of core engineering courses. Over time, these courses may evolve to accommodate new subject matter, pedagogical approaches, political and personal preferences, or other criteria as dictated by a dynamic group of stakeholders. Many changes originate from a clearly defined need or mandate, while others may sneak in without a full analysis of the course. Repeated and often subtle changes compound to have a significant impact on the course, creating a narrative reflecting the intents of the faculty and the concerns of the institution as course goals and methods are updated in each subsequent semester. This paper describes a process to employ engineering education research methods to describe the nature, development, implications, and motivation behind of course changes. We define a six step process focused on the use of artifact analysis to provide instructional teams with concrete historical data, allowing them to better understand the structure of their course and how it has changed over time. A case study examining a large-format, First Year Engineering course is included at a part of this paper, providing context and serving to describe the process in action. The case study includes methodological choices, analysis, and findings as a guide to practitioners seeking to follow or further develop our process for gathering data. The data produced can be used to inform future changes to the course design to ensure alignment of the course objectives, assessment, and pedagogy, while at the same time systematically meeting the requirements of ABET Criteria 4

Work in Progress: Rigorously Assessing the Anecdotal Evidence of Increased Student Persistence in an Active, Blended, and Collaborative Mechanical Engineering Environment

This work in progress describes an ongoing study of an active, blended, and collaborative (ABC) course environment used in a core mechanical engineering course. This course has built on the growing body of literature citing active learning (Freeman et al., 2014), blended structures (Bowen & Ithaka, 2012), and collaborative engagement (Jeong & Chi, 2007) as positive influences on college and university science, technology, engineering, and math (STEM) outcomes. For the last six years, “Dynamics”, a core mechanical engineering course at a large public university, has utilized in-class activities, frequently-watched problem-solving videos, and a collaborative blog space to realize an ABC environment. On one key metric of course success, the rate of students who drop, fail, or withdraw from (DFW), the course has experienced near-constant improvements since the ABC structures were introduced. In this study, the authors utilize rigorous longitudinal methods to determine whether this drop in DFW rates can be directly attributed to increased implementation of ABC features. The authors hypothesize that as instructors become accustomed to the ABC environment and increase the level of in-class activity, use of blended resources, and collaboration, the likelihood of DFW in each subsequent year would drop. However, in the same time period, each subsequent class entered with higher levels of performance on proxy measures for prior knowledge. We therefore build a logistic regression model to predict individual-level DFW and determine whether the anecdotal drops in DFW that we observe can be attributed to the expansion of the ABC environment. More specifically, we predict likelihood of DFW based on students’ prior knowledge (grade in the preceding course, SAT math score), key demographics (gender, race/ethnicity), the semester and year they took Dynamics, their instructor, their year in school, and their major. We test for year fixed effects {year_t, t = 1, 2, ..., 7} to determine whether odds ratios for DFW consistently and significantly decrease over time. We also test for instructor effects, in particular for differences between the instructors who were involved in the design and development of the ABC environment and more independent instructors who only partially implemented the ABC course components. We anticipate results that will provide more rigorous, less biased, and efficient estimates for the individual- and class-level components that explain variance in DFW rates. These results would provide immediate implications for the next phase of our work, as we assess the next on-term implementation of the course in 2016. Our findings would also have long-term significance for other classes in mechanical engineering and related disciplines and for classes at other institutions that are considering implementing a comprehensive ABC learning environment

Analyzing an Abbreviated Dynamics Concept Inventory and Its Role as an Instrument for Assessing Emergent Learning Pedagogies

The Dynamics Concept Inventory (DCI) is a validated assessment tool commonly used to evaluate student growth within core, gateway-level mechanics courses. This research explored the evaluative use of this tool within the context of Freeform – an emergent course system that buttresses active class meetings with blended and collaborative virtual learning environments, themselves founded upon extensive multimedia content and interactive forums – at Purdue University. The paper specifically considers a number of related issues including: (i) the thoughtful development (via expert content validation) and statistical reliability of an abbreviated DCI instrument, which is more amenable to in-class implementation than the much longer full DCI; (ii) the correlation of abbreviated-DCI performance with exam scores and final course grades for a dynamics course using the Freeform framework with an emphasis on both conceptual understanding and traditional problem-solving skills; and (iii) various inter-section performance metrics in a preliminary study on how an implementation of the abbreviated-DCI may help elucidate the impact of the instructor within the Freeform framework. The results of these analyses supported the validity and reliability of the abbreviated DCI tool, and demonstrated its usefulness in a formal research setting. The preliminary study suggested that the Freeform framework might normalize differences in instructor pedagogical choices and student performance across class sections. These findings indicate that the abbreviated DCI holds promise as a research instrument and lay the groundwork for future inquiry into the impact of the Freeform instructional framework on students and instructors alike

Transforming a Dynamics Course to an Active, Blended, and Collaborative Format: Focus on the Faculty

Mechanical engineering programs are increasingly applying educational research by transforming courses to be more interactive and to use a blend of online and face-to-face materials. However, the process of an existing course adopting these new practices is not well studied, and even less is understood about the faculty experience from on-boarding to delivery of a new curriculum or pedagogy. In this study, we follow the translation of an active, blended, and collaborative (ABC) curriculum for a core dynamics course from a large public university (where the ABC curriculum was developed) to a small private university. We use interpretive phenomenology to focus on the lived experience of the instructor newly implementing these course materials, format, and pedagogical approach. Specifically, we address the following research questions: (1) What is the lived experience of a mechanical engineering instructor at a different institution as she adopts and adapts the provided materials and format? (2) How does the experience of this instructor evolve throughout the semester? We use rich qualitative data to understand the experience of the instructor, who taught this course in its prior format and, in Fall 2015, taught the “off-term” core dynamics course via the new ABC structure. Through weekly reflection prompts, pre- and post-semester interviews, and supplementary process data (e.g., notes of weekly meetings between the new implementer and ABC team at the large public university), we describe and characterize the multi-faceted instructor experience. This includes her experience learning about the curriculum and online tools, implementing the class and adjusting her teaching practices, and assessing her students’ engagement with the course and understanding of dynamics concepts. Our findings suggest further areas of inquiry for studies of faculty practices around curriculum adoption, including probing opportunities for cross-institutional collaborations to share materials and transform courses, interrogating variation in mechanical engineering department and student cultures, and studying sources of faculty development and support throughout the course transformation process

Student Implementation Experiences in Blended Learning: A Phenomenographic and Narrative Analysis to Inform Pedagogical Innovation

In this dissertation, I argue that there is value in treating students as implementors during processes of educational innovation. I lay the groundwork for this argument through a review of literature comparing best practices in the implementation of innovations in higher education with best practices from active learning, blended learning, and collaborative learning research. This is followed by a phenomenographic and narrative analysis: a deliberate combination of phenomenography and narrative analysis methods for the interpretation of data and representation of findings, leveraging the strengths of each approach to account for the other’s shortcomings. The result of this work is an outcome space containing a hierarchical framework typical of phenomenography describing the various ways in which the participating students experienced implementation within the context of a blended learning environment called Freeform. The presentation of this framework is followed by a series of constructed narratives which contextualize how the hierarchical framework may be evidenced in student experiences of implementation in higher education. The hierarchical framework contains six categories of description: Circumstantial Non-Adoptive, Circumstantial Adoptive, Preferential Non-Adoptive, Preferential Adoptive, Adaptive, and Transformative. Proceeding from Circumstantial Non-Adoptive and Circumstantial Adoptive to Transformative, each subsequent category of the model characterizes implementation experiences that are increasingly impacted by students’ own self-awareness of their personal learning needs and subsequent self-directed learning behavior. This represents a departure from previous implementation research in engineering education for a number of reasons. First, it demonstrates that there is value in considering students’ roles as implementors of educational innovations, rather than tacitly treating them as subjects to be implemented upon. Second, the use of the word “circumstantial” intentionally acknowledges that the external (environmental) factors that influence implementation can be distinct to individual implementors while remaining contextual in nature. Third, it demonstrates that the processes of implementation which students undergo can lead to concrete changes in learning behavior that extend beyond the scope of the implementation itself. Narrative analysis is used to develop a series of narratives that embody the implementation experiences communicated by student participants. These narratives are constructed using disparate ideas, reflections, and tales from a variety of participants, emplotting representative characters within constructed stories in a way that retains the student perspective without adhering too closely to any individual participant’s reported experience. This approach serves two goals: to encourage readers to reflect on how the categories of the hierarchical framework can be demonstrated in students’ experiences, and to reinforce the fact that individual students can exhibit implementation experiences and behaviors that are characteristic of multiple categories of the framework simultaneously. It is important to remember that the categories included in the framework are not meant to characterize students themselves, but rather to characterize their interactions with specific pedagogical innovations. The study concludes by interpreting these results in light of literature on implementation and change, proposing new models and making suggestions to faculty to inform the future implementation of educational innovations. Faculty are encouraged to treat students as implementors, and to exercise best practices from implementation literature when employing educational innovations in the classroom. This includes adopting practices that inform, empower, and listen to students, intentionally employing strategies that allow students to exercise their own agency by understanding and utilizing innovations effectively