Development and analysis of assessments that promote sensemaking in physics

Doctor of PhilosophyDepartment of PhysicsJames LavertyAssessments are an integral part of academic environments and can present opportunities for students to make sense of novel contexts using their existing ideas. Assessments also provide insights on students’ learning and the efficacy of the pedagogical practices. Consequently, physics education research shares a storied history of developing research-based assessments (RBAs) that support students' sensemaking. However, contemporary studies have noted several shortcomings of the existing RBAs such as: (i) lack of clarity for instructors in interpreting students’ scores to make modifications to their instruction, (ii) misalignment between the content of the assessments and the local learning goals of the instructors, (iii) scarcity of standardized assessments for undergraduate physics, and (iv) the need to shift the focus of RBAs from ``knowing" to ``doing" physics. Researchers have also called for probing the contextual factors that influence students' sensemaking in physics. In light of these observations, I present methodological and theoretical approaches to developing and analyzing next-generation RBAs. I introduce the development process of a novel RBA -- the Thermal and Statistical Physics Assessment (TaSPA) -- which focuses on assessing the ``doing" aspect of physics along with providing actionable feedback for instructors to modify their courses. Additionally, this assessment allows instructors to choose what they wish to assess, thereby bridging the gap between assessment objectives and the local learning goals of the instructors. I elucidate the leveraging of existing theoretical and design frameworks in the development of TaSPA and how the interplay of these frameworks addresses some of the shortcomings of the contemporary assessments. This diagnostic represents a paradigm shift in how assessments are envisioned and designed by the discipline-based education research community. I also complement the contemporary literature by theoretically exploring assessment task features that increase the likelihood of students sensemaking in physics. I identify the task features by first noting the salient characteristics of the sensemaking process as described in the science education literature. Existing theoretical ideas from cognitive psychology, education, and philosophy of science are then leveraged in unpacking the task features which elicit the characteristics of sensemaking. Furthermore, I leverage Conjecture Mapping -- a framework from design-based research -- to articulate how the proposed task features elicit the desired outcome of sensemaking. I argue that to promote sensemaking, tasks should cue students to unpack the underlying mechanism of a real-world phenomenon by coordinating multiple representations and by physically interpreting mathematical expressions. The proof of concept of this idea is then presented through analysis of students' reasoning about the tasks embodying the proposed features. The analysis is then extended to provide an explicit account of the intertwining between modeling and sensemaking processes. The analyses reveal that particular aspects of modeling and sensemaking processes co-occur. For instance, the priming on the ``given'' information from the problem statement constituted the students' engagement with their mental models, and their attempts to resolve inconsistencies in understanding involved the use of external representations. I find that barriers experienced in modeling can inhibit students' sustained sensemaking. Major contributions of this work include: (i) elucidating a methodological approach in developing an RBA that promotes the ``doing'' aspect of physics; (ii) demonstrating an agent-based perspective in exploring assessment task features; (iii) operationalizing conjecture mapping in the context of task design in physics; (iv) introducing a methodology extendable to unpack task features which can elicit other valued epistemic practices; and (v) an explicit framework-based unpacking of the association between modeling and sensemaking. This dissertation opens up avenues for future explorations such as extending the proposed methodology in developing RBAs for other upper-division physics courses such as quantum mechanics. The presented theoretical and methodological approaches can also be extended in exploring features of the assessment tasks that promote additional epistemic practices such as argumentation and modeling. Researchers can also explore expanding the proposed list of task features, and the accompanying constraints (if any)

Student and AI responses to physics problems examined through the lenses of sensemaking and mechanistic reasoning

Several reports in education have called for transforming physics learning environments by promoting sensemaking of real-world scenarios in light of curricular ideas. Recent advancements in Generative-Artificial Intelligence has garnered increasing traction in educators' community by virtue of its potential in transforming STEM learning. In this exploratory study, we adopt a mixed-methods approach in comparatively examining student- and AI-generated responses to two different formats of a physics problem through the cognitive lenses of sensemaking and mechanistic reasoning. The student data is derived from think-aloud interviews of introductory students and the AI data comes from ChatGPT's solutions collected using Zero shot approach. The results highlight AI responses to evidence most features of the two processes through well-structured solutions and student responses to effectively leverage representations in their solutions through iterative refinement of arguments. In other words, while AI responses reflect how physics is talked about, the student responses reflect how physics is practiced. Implications of these results in light of development and deployment of AI systems in physics pedagogy are discussed

Towards a content-based epistemic measure in physics

We elaborate on a new approach of assessing content-based epistemic clarity of college physics students in terms of their ability to discriminate between different epistemic warrants for propositions in a chained argument in physics. A threefold classification (nominal, physical, and mathematical) of warrants is used, with each class split into a number of distinct epistemic categories. The assessment tool consists of a questionnaire, illustrated here for a standard undergraduate physics derivation, in which the student chooses the appropriate epistemic category (from an appended list) for each proposition in the given derivation. The tool includes a brief supplementary orientation note for clarifying the meaning of different categories. It is tested for reliability and internal consistency. Correlation of content knowledge, ascertained independently, with epistemic clarity is investigated. Surveys on large and diverse samples of physics undergraduates carried out with or without the orientation note reveal a variety of epistemic errors, some common among all groups of students, and some that are good discriminators. The epistemic measure obtained from such a tool is, however, limited in scope by its narrow definition. The construct validity of the tool and the pedagogic relevance of the work are discussed

Hands in mind: learning to write with both hands improves inhibitory control, but not attention

Embodied cognition theories predict that changing motor control would change cognitive control, as cognition is considered to emerge from action in this theoretical approach. We tested this prediction, by examining the attention and cognitive control capabilities of a group of school students (12-13-year-olds) trained to write using both hands (experimental group, N=28), compared to a group of age- matched children (control group, N=33) who did not receive such training. The key tasks used were the attentional network test (ANT) task and the hearts and flowers (HF) task. Results from the ANT task showed that there was no significant difference in the three attentional networks between the groups. However, results from the HF task showed that the experimental group had better inhibitory control. This second result provides support to the embodied cognition prediction that cognitive control and motor control are related, and the former can be changed to some extent by changing the latter