How do I assess an amorphous outcome? A new take on critical thinking.

The ability to think critically is an essential skill for the next generation of leaders. Post-secondary institutions are required to teach and assess critical thinking (CT) in our programs, but what is CT exactly and how do you assess it? This research project focused on finding a conceptualization of CT that can help us approach these questions from a new angle and provide opportunities to assess CT development. Many existing ideas of CT contradict each other, creating confusion about what CT is and what it looks like in practice. A common CT framework can alleviate this miscommunication and provide a platform on which structured assessment tools can be built. The Model of Integrated Thinking Skills (MITS) was created by the primary author to fill this gap by connecting recurring themes about CT found in the literature. MITS was validated through an online survey completed by students, faculty, and staff from various disciplines at the University of Guelph. Our results suggest that the framework is accurate/complete, applicable to all disciplines, and enables the reader to distinguish CT from other intellectual skills. In this workshop, participants will discuss the challenges around teaching/assessing CT, familiarize themselves with MITS and explore how it relates to their experiences with modern science education. We will also talk about the struggles associated with facilitating CT development in students

Students take the wheel! How a private blog enabled undergraduate researchers to guide a large project.

There is growing advocacy for including the student voice in educational research and initiatives. Students involved in such projects are often transformed through the process, finishing their participation with new insights and awareness of the curriculum they experience. One way to document the students’ experience and activity is through the use of private blogging software. In 2015, a large discrete learning outcomes project was completed with the help of three undergraduate co-op students. The aim of this project was to map the learning outcomes of eight courses into a learning outcomes framework using real assessments from the courses. The logistics of the project required excellent communication, task delegation, and time management of the students. The students used private online blogging software to document their progress on the project. Through a qualitative secondary data analysis, we analyzed the blogs and identified a number of emergent themes that suggest how the blogging software may have contributed to the students’ success in the project. In this presentation, we will share the main findings of our secondary analysis of the students’ use of the blogging platform. Although the sample of students is small, the dataset itself consists of over 100 blog posts. Our intention was to conduct an in-depth qualitative analysis of the use of the blog for a complex project, to explore how blogs could be used by students to work as a productive team. Participants will have the opportunity to discuss their thoughts and reflect on student engagement opportunities within their practice

Engaging Student Stakeholders in Developing a Learning Outcomes Assessment Framework

Learning outcomes assessment and alignment contribute to the transparency, quality, and progression of a program. We set forth a learning outcomes framework that aligns learning outcomes at the course, major, program, and university levels. Senior undergraduate students were recruited to analyze assessments from eight core courses required for Molecular and Cellular Biology (MCB) majors at the University of Guelph. This analysis was conducted to achieve two goals: (a) to develop tools to assess learning outcomes in the MCB Department, and (b) to incorporate insights shared by the student perspective. Almost 1,600 Individual questions and their attributes were coded, compiled, and linked into the learning outcomes framework. The students then connected the questions to course concepts and assigned a cognitive domain indicated by Bloom’s Taxonomy level. After training and calibration, two undergraduate students evaluated all questions in the eight core courses with an average of 93.2% ± 1.6% (n=8) agreement between evaluators. These data were used to generate assessment profiles for individual courses and as an aggregate to provide insights regarding the program. This work makes constructive use the learning outcomes framework and illustrates the importance of leveraging undergraduate student perspectives in discussions of learning outcomes in higher education

Development and validation of a critical thinking framework

Critical thinking (CT) is essential in many higher-order thinking skills and should be enhanced by higher education programs. The Model of Integrated Thinking Skills (MITS) was developed to help students developing CT, assist educators teaching CT, and satisfy the following three criteria: 1) completely and objectively describe CT, 2) distinguish CT from other skills, and 3) be used by all disciplines.  A validation survey determined how MITS is perceived with respect to the three criteria.  Most participants perceived that MITS meets the three criteria, suggesting that MITS can assist student development of CT

Using existing assessments to track longitudinal development of students’ problem solving skills

Do we really know how our students develop transferrable skills? Do they? Learning outcomes assessment (LOA) data are most commonly collected via national surveys, alumni surveys / focus groups, and locally developed surveys (MacFarlane & Brumwell, 2016; Williams, 2014). While these methods collect important affective information about students and alumni, they are not well-suited to measure students’ actual achievement and are often disconnected from the classroom where student skills are developed (Campbell & Cabrera, 2011; Gordon, Ludlum, & Hoey, 2008; Porter, 2011, 2013). Detailed LOA data describing students’ actual skills and perceived skills are needed to facilitate targeted improvements to courses and programs and enhance the LOA initiative (Klemenčič & Chirikov, 2015). In this session, we present “The Framework”: a strategy that coordinates three essential perspectives of LOA (intention, achievement, and perception) in the evaluation of a program. Focussing on problem solving for its pilot use, The Framework uses a novel problem solving profile tool to characterize the problem solving challenge of assignments, a program-level developmental rubric for problem solving, and a student survey to collect students’ perceptions of their own problem solving development. To the best of our knowledge, The Framework is original and has not been established elsewhere. Using pilot data from the 2018-2019 academic year, we invite you to explore how The Framework generates both course and program data educators were previously unable to access and how these data can change the questions we ask about our courses/programs and shift our conceptualizations of curriculum mapping. The principles of The Framework are transferrable to any undergraduate context

Using a Project Blog to Promote Student Learning and Reflection

Research on the use of blogs in higher education suggests that students who blog as part of a course requirement demonstrate increased reflection, heightened feelings of connection, and increased course-related knowledge. Blogging provides a promising medium for students to engage in reflection on their participation in a cooperative education work term. Blogging can help guide a project by providing an encouraging place for team members to communicate clearly, document their work and other resources, and brainstorm ideas. In this paper, we describe a qualitative analysis of blog posts written by three students during a co-op placement. The students used the blog extensively and with multiple purposes: to reflect, to communicate, to brainstorm ideas, to evaluate literature, and to document team creations. We discuss the characteristics of the project and blog that may have contributed to its extensive use and provide recommendations for individuals interested in implementing a blog in similar projects.Selon la recherche, les étudiants en enseignement supérieur utilisant un blogue pour satisfaire aux exigences des cours montrent un niveau élevé de réflexion, de sentiment d’appartenance et de connaissances liées au cours. Le blogage fournit aux étudiants un médium prometteur pour s’engager dans la réflexion durant un stage coopératif. Le blogue peut guider un groupe en encourageant une communication claire, une documentation des ressources et un libre partage des idées. Dans ce document, nous décrivons une analyse qualitative des articles de blogue de trois étudiants en stage. Les étudiants ont largement utilisé le blogue : pour communiquer, pour faire des « remue-méninges, » pour évaluer la littérature et pour documenter des créations communes. Nous discutons les caractéristiques du projet qui pourraient avoir eu un impact sur l’utilisation répandue du blogue, avant de proposer des conseils à ceux qui s’intéressent à la mise en œuvre du blogage dans de tels projets

Intention and Perception: Designing questions for online assessments.

During the session participants will be guided through a process to develop and evaluate assessment components that target higher order cognitive skills. Building on this base, participants will engage in a process of creating new questions that align with the learning outcomes (LO) for a course. As a group, we will discuss online quizzes and how we can embrace their unique characteristics while designing questions. Itinerary: I. Relationship between LO and assessment and its significance Activity 1: Recognizing misalignment. II. Question alignment Progress from creation to reflection to revision III. The online perspective Discussion: Characteristics of an online quiz Transitioning current assessments to the online setting Embracing these traits in question design Activity 2: Create a question that aligns with a given CLO, reflect and revise. Discuss

Aligning Assessments as a Process in Program Evaluation

Assessment and learning outcome alignment contributes to the transparency, quality and progression of a program. In this presentation, we will share a process used to evaluate the alignment of assessments with both course content and the learning outcomes framework. This framework links learning outcomes at each level of the institution, from individual courses to majors, programs, and the university. We can use this framework in online settings to collect data at each learning outcome level. The pilot project at the University of Guelph will show how this data analysis can guide discussions about the improvement of program curriculum by exposing misalignments

sj-docx-1-aje-10.1177_10982140241234841 - Supplemental material for Mapping Evaluation Use: A Scoping Review of Extant Literature (2005–2022)

Supplemental material, sj-docx-1-aje-10.1177_10982140241234841 for Mapping Evaluation Use: A Scoping Review of Extant Literature (2005–2022) by Michelle Searle, Amanda Cooper, Paisley Worthington, Jennifer Hughes, Rebecca Gokiert and Cheryl Poth in American Journal of Evaluation</p