Canner, Judith

Dahlen, Sarah P.C.

Gage, Ondine

Graff, Nelson

Gregg, Shar

Shapiro, Dan

Waldrup-Paterson, Vivian

Wood, Swarup E.

English

Digital Commons @ CSUMB (California State University, Monterey Bay)

California State University, Monterey Bay Digital Commons @ CSUMB Library Faculty Publications and Presentations Library 2020 Engaging Faculty in Assessment of Institutional Learning Outcomes Judith Canner Sarah P.C. Dahlen Ondine Gage Nelson Graff Shar Gregg See next page for additional authors Follow this and additional works at: https://digitalcommons.csumb.edu/lib_fac This Article is brought to you for free and open access by the Library at Digital Commons @ CSUMB. It has been accepted for inclusion in Library Faculty Publications and Presentations by an authorized administrator of Digital Commons @ CSUMB. For more information, please contact digitalcommons@csumb.edu. Authors Judith Canner, Sarah P.C. Dahlen, Ondine Gage, Nelson Graff, Shar Gregg, Dan Shapiro, Vivian Waldrup-Paterson, and Swarup E. Wood Engaging Faculty in Assessment of Institutional Learning Outcomes  Judith Canner, Sarah P.C. Dahlen, Ondine Gage, Nelson Graff, Shar Gregg, Dan Shapiro, Vivian Waldrup-Paterson, and Swarup E. Wood  As regional accreditation agencies place more emphasis on direct assessment of student achievement and demonstrated use of assessment results to improve teaching and learning, engaging faculty in assessment is becoming increasingly important; however, it is a common trope in higher education that faculty do not want to participate in institution-level assessment (Emil and Cress 2014; Hutchings 2010). California State University, Monterey Bay (CSUMB) has developed a process for institutional learning outcomes assessment using reflective faculty engagement that contradicts that trope. Our process has been guided by our belief that assessment should  ● Be directed by faculty ● Recognize both faculty expertise and potential for growth ● Emphasize the centrality of increasing student learning ● Foster dialogue and community-building ● Produce usable results This article describes our process and presents data on how participation has impacted faculty.   Directed by faculty. At CSUMB, assessment of student learning outcomes is directed by the faculty through the Academic Senate Assessment Committee whose voting members consist of faculty representatives from each college. In response to our regional accreditor’s requirement that we assess student achievement of the “five core competencies” (critical thinking, information literacy, quantitative reasoning, written communication, and oral communication), CSUMB decided at the outset to design an institution-level assessment process that prioritized faculty engagement. In 2013 the Assessment Committee appointed a faculty member to the position of Critical Thinking Coordinator to assemble an interdisciplinary team of faculty to assess critical thinking at the institutional level. Since then, coordinators for each of the core competencies have been chosen and each core competency assessed twice.   Faculty expertise and growth. With support from the Center for Teaching, Learning, and Assessment and Communication Across the Disciplines, each core competency faculty coordinator used the following process: ● Six to eight “assessment scholars” with experience teaching the core competency were selected from the faculty in response to a general call.  ● Initial meetings focused on studying the AAC&U VALUE Rubrics and modifying them for our institutional context.  ● Assessment scholars were provided professional development on assessment best practices (Suskie 2018).  ● Over 2 - 4 days, the coordinator and assessment scholars assessed embedded student work drawn from key courses from across the curriculum, with two faculty members scoring each student artifact.  ● Results were disseminated through the Academic Senate and used to identify professional development needs.  Recruiting faculty for these projects recognized and utilized their disciplinary expertise while simultaneously providing them professional development in assessment best practices. Because projects were conducted over the summer, participants were compensated at the standard daily rate for the institution.   Centrality of student learning. Key to engaging faculty was communicating the institution’s assessment philosophy and how it influenced the design of the institutional assessment projects. Rather than trying to meet scholarly criteria for “good” data, we emphasized producing usable data that catalyze conversations and action (Roscoe 2017). Like Barrette (2017), our process incorporates “active learning and reflection; context-specific content related to normal work contexts, questions, and problems; and expert input and opportunities to interact with peers” (p. 6), always centered on improving student achievement of institutional learning outcomes.   Dialogue and community building. The following activities were used to promote dialogue and community building during the assessment project: ● Each assessment project started with a half day of norming and discussion.  ● Three-hour scoring sessions were punctuated with opportunities for faculty pairs to discuss differing assessments of individual student artifacts (when present) followed by whole-group discussions to share observations, experiences, and insights.  ● Each day ended with a written reflection in which assessment scholars responded to a set of open-ended prompts based on Brookfield’s critical incident questionnaire (Brookfield 1995).  ● Each day began by discussing the prior day’s reflections and, if needed, more norming.  ● The assessment project ended with a final reflection that provided participants the opportunity to comment on the assessment process, the rubrics, and their own learning.  Like others who use assessment in this way (e.g., Martin and Sisson 2018), this assessment work has led to valuable discussions of assessment, the core competencies, and professional development opportunities for the entire faculty around assignment design, teaching analysis, and teaching each of the core competencies.   Impacts on assessment scholars. Between summer 2014 and summer 2017, 40 assessment scholars (consisting of lecturers and tenure-line faculty) have participated in 7 institutional assessment projects. To determine the impacts on the participants, the authors read all responses to the daily and final reflections, identified major themes, and then coded and analyzed the responses. In addition, a follow-up survey was sent to all assessment scholars in December 2017 and those responses (25/40 or 63%) were also coded and analyzed. Results are summarized in Table 1.    In general, faculty responses to daily reflections and the post-assessment survey revealed that participants valued the experience because of the opportunity to reflect on pedagogy and teaching practice. Conversations addressed elements of the rubric that faculty had difficulty scoring or patterns they had noticed in student work, but they also ranged widely into other topics about teaching, learning, and assessment. Because the rubrics name some agreed-upon criteria for tracing intellectual work in student artifacts, faculty found them useful springboards for discussions about teaching those skills. Faculty also valued the opportunity for social interaction as well as the food, both of which were important contributors to community building and feeling appreciated by their peers and the institution for their contributions.  Beyond the interpersonal benefits and general pedagogical discussions and other areas identified (Table 1), faculty comments revealed that they saw the scoring sessions as an opportunity to develop both shared expectations regarding the quality of student work and deeper understandings of the core competencies themselves. Representative comments include, “I hadn’t really carefully thought through the various elements of [critical thinking]. This experience has helped me to begin understanding how to unpack these in the CT process” and “It led me to examine my lessons and assignments carefully to ensure that I am teaching students the skills they need to graduate with proficiency in the areas focused on the [learning outcome].” Further, in the post-assessment surveys, faculty identified ongoing benefits including continued improvements to their course design and assignments, teaching, and interactions with students and colleagues (Table 1).  Using results. Central to our goal of engaging our faculty more broadly, in both doing assessment and closing the loop, is emphasizing “holistic alignment” (Jankowski and Marshall 2017) in which we explicitly work to use assessment of institutional learning outcomes to align not only course and program outcomes, but also assignments, pedagogy, and co-curricular learning. These assessment projects and results prompted the following: ● Refinement of the rubrics and development of core competency assignment guides and rubric guides (TLA 2018).  ● Workshops on assignment design, teaching analysis, and incorporating information literacy into oral communication tasks.  ● Professional development evidence-based practices that promote students’ reading, writing, critical thinking, and communication skills.  ● Collaboration with professional and student tutors in the learning center to better support student development of the core competencies. ● Engagement of STEM faculty in Writing Enriched Curriculum (WEC) development.  All together, our assessment process exemplifies one of Hutchings’ (2010) main recommendations for engaging faculty in assessment: “Create Campus Spaces and Occasions for Constructive Assessment Conversation and Action” (p. 15).   Conclusion. The positive faculty responses were certainly influenced by the fact that participants were self-selected and paid for their work. However, some of us who have participated in differently structured assessment projects in which the same was true but the experience less engaging know that the particular design of this process had a positive influence. In fact, several assessment scholars noted that they were surprised by how engaging and helpful the process was for them. Nevertheless, the challenge remains on how to reach the majority of faculty. These projects are an important and necessary first step.   Jankowski and Marshall (2017) highlight a NILOA policy statement in which the argument is made that institutions should “focus on improvement and compliance will take care of itself.”  In emphasizing faculty engagement in the design of assessment projects and working to integrate and align three philosophies of assignment -- measurement, compliance, and student-centered learning (Jankowski, 2017) -- the institution continues to build a culture of assessment for improvement that satisfies accreditation requirements while maintaining a central focus on using assessment to increase student learning and success.  References  Brookfield, S.D. 1995. Becoming a Critically Reflective Teacher. San Francisco: Jossey-Bass. Emil, S., and C. Cress. 2014. Faculty Perspectives on Programme Curricular Assessment: Individual and Institutional Characteristics that Influence Participation Engagement. Assessment & Evaluation in Higher Education, 39(5), 531-552. Hutchings, P. 2010. Opening Doors to Faculty Involvement in Assessment. NILOA Occasional Paper, 4. Jankowski, N. A. 2017. Moving Toward a Philosophy of Assessment. Assessment Update, 29(3), 10-11. Jankowski, N. A. and D. W. Marshall. 2017. Degrees That Matter: Moving Higher Education to a Learning Systems Paradigm. Stylus Publishing. Martin, E. C., and P. J. Sisson. 2018. Meaningful and Sustainable Assessment at Middlesex Community College. Assessment Update, 30(1), 10-11. Roscoe, D.D. 2017. Toward an Improvement Paradigm for Academic Quality. Liberal Education. Washington, D.C.: Association of American Colleges and Universities. Retrieved from https://www.aacu.org/liberaleducation/2017/winter/roscoe Suskie, L. 2018. Assessing Learning: A Common Sense Guide. John Wiley & Sons. TLA. 2018. Assignment Guides and Rubrics for Institutional Learning Outcomes. Retrieved from https://csumb.edu/tla/ilo-assignment-guides-rubrics-and-threshold-concepts   Table 1. Analyses of daily reflections and post reflection survey. Daily Reflections (n = 40) Question(s) Theme  % affirmative responses for theme  Representative quote(s) At what moment(s) during today’s work did you feel most engaged with what was happening?  What is something you experienced during today’s work that you found most affirming or helpful?  Social interaction 80% “I enjoy collaborating with faculty. It allows for sharing teaching styles, insights, and strategies. It allows me to think more deeply about course and assignment design to benefit student learning.” Norming 74% “While we all operate within a certain set of standards, the norming process reminds us that we don't always share identical values. Deliberately applying a rubric can help to diminish these personal distinctions.” Rubric 63% “Writing assignments and corresponding rubrics can be utilized as powerful teaching tools. The assignments and feedback given to students across this university should be rigorous, clear, and consistent.” What is something you experienced during today’s work that you find most puzzling or confusing.  Rubric 61% “We’re still fleshing out the rubric, so there is a snootch of confusion - but that’s kind of the point of this whole process.” Post-Assessment Surveys (n = 25) Questions  % affirmative responses Representative quote(s) How did participation in the ILO1 assessment work affect your... Course/ syllabus design 72% “I think more about the overall course rather than the assignments first - that is, what do I want my students to get out of the course.” Assignment design 88% “Not only did I change wording in assignments to more directly target the learning outcomes. I also dropped an assignment that did not directly address learning outcomes and added two other assignments that allow students to think more critically and use information literacy skills.” Teaching and other interactions with students 92% “I was able to communicate oral and written communication goals in an informed, confident manner. I particularly appreciated that I was speaking from a platform that included my peers rather than being based only on my personal training and perception.” Work or other conversations with faculty/staff 80% “I've become more bold in pushing for greater clarity from my colleagues in their work with students -- and for more opportunities for collaboration to achieve more parity in the work we assign.”  Judith Canner is an associate professor of statistics, Sarah P.C. Dahlen is the library assessment  coordinator, Ondine Gage is an assistant professor of education, Nelson Graff is the director of communication across the disciplines, Shar Gregg is a lecturer and oral communication coordinator, Dan Shapiro is Interim Associate Vice President for Academic Planning and Institutional Effectiveness , and Swarup E. Wood is the interim director of general education at California State University, Monterey Bay.  

Engaging Faculty in Assessment of Institutional Learning Outcomes

https://digitalcommons.csumb.edu/cgi/viewcontent.cgi?article=1015&context=lib_fac

Engaging Faculty in Assessment of Institutional Learning Outcomes

Abstract

Similar works

Full text

Available Versions

Digital Commons @ CSUMB (California State University, Monterey Bay)