Kramer, Philip I.

English

College of Saint Benedict and Saint John’s University: DigitalCommons@CSB/SJU

College of Saint Benedict and Saint John's University DigitalCommons@CSB/SJU OARCA Staff Publications Office of Academic Review and Curricular Advancement 2005 Systemwide assessment of Utah's general education courses Philip I. Kramer College of Saint Benedict/Saint John's University, pkramer@csbsju.edu Follow this and additional works at: https://digitalcommons.csbsju.edu/oarca_pubs  Part of the Educational Assessment, Evaluation, and Research Commons Recommended Citation Kramer, P. I. (2005). Systemwide assessment of Utah's general education courses. Assessment Update, 17(2), 4-6. This Article is brought to you for free and open access by DigitalCommons@CSB/SJU. It has been accepted for inclusion in OARCA Staff Publications by an authorized administrator of DigitalCommons@CSB/SJU. For more information, please contact digitalcommons@csbsju.edu. 4 Assessment Update • March–April 2005 • Volume 17, Number 2IN 1998, THE UTAH STATE BOARD OFRegents ordered a sample administra-tion of American College Testing’sCollegiate Assessment of Academic Proficiency (CAAP), a test of generalknowledge, for Utah System of HigherEducation (USHE) students. The pilot testwas administered to 3,148 Utah collegestudents who were completing theirsophomore year at one of Utah’s ninepublic colleges or universities.Although Utah college and universitystudents scored above the national aver-age, the results of the CAAP were consid-ered a disappointment. Faculty throughoutthe state believed that the CAAP was aninadequate test of general education be-cause (1) the test was not aligned withcourse content or objectives; (2) theCAAP did not correlate with students’grade point averages; (3) the findingswere too general and failed to offer anyguidance to faculty; and (4) the CAAPwas not administered to students who hadtaken any specific courses in common orcompleted similar degree requirements(Utah State Board of Regents, 2000,p. 11). Consequently, the Regents’ TaskForce for General Education, an ad hocfaculty committee charged by the regentswith examining general education in thestate, asked the regents to allow facultymembers to create their own assessment instruments to measure the general edu-cation knowledge of Utah college anduniversity students.The regents agreed with members ofthe task force and authorized the group tocreate and administer course-relevant,content-embedded assessment instru-ments. However, before assessment in-struments were designed, the task forcefirst had to determine the meaning of gen-eral education. After months of work, thefaculty representatives from throughoutthe state agreed on a set of nine generaleducation competences. This set estab-lished the minimum level of knowledgethat students were required to attain aftercompleting the second year of general ed-ucation courses at any of the public insti-tutions of higher education in the state.Nine competences were established in (1)quantitative literacy, (2) writing, (3) socialsciences, (4) humanities, (5) life sciences,(6) physical sciences, (7) fine arts, (8)technology and computers, and (9) Amer-ican institutions.1After agreeing on general educationcompetences, statewide faculty began todevelop the pilot assessment. Much of theorganization and implementation of thepilot assessment emanated from facultydisciplinary subcommittees. In 2000, a re-quest for proposals had been sent from thetask force to the chief academic officers(CAOs) on the nine campuses, inviting theCAOs to participate in the assessmentprocess and to nominate faculty represen-tatives from four disciplines (economics,history, political science, and mathemat-ics) to represent their respective cam-puses. Over a period of several months,nominated faculty met in small discipli-nary groups and developed a bank ofcourse-relevant, content-embedded testquestions.After coming to general agreementabout the importance of measuring value-added knowledge, faculty in the disci-plinary subcommittees decided to usemultiple-choice pretest and posttest ques-tions. Faculty were aware of the limita-tions of multiple-choice tests; however,creating other types of test questions—forexample, essay questions—was not possi-ble due to time and financial constraints.The mathematics faculty committee re-fused to use multiple-choice questions. In-stead, this committee elected to use a testcomprising true-false and fill-in-the-blankquestions. Perhaps justifiably differingfrom the other three disciplines, mathe-matics faculty insisted on the methodolog-ical ability to evaluate students’ thoughtSystemwide Assessment of Utah’s General Education CoursesPhilip I. KramerFaculty representatives agreed on a set of nine general education competences, which established the minimum level of knowledge that students were required to attain after completing the second year of general education courses at any of the public institutions of higher education in the state.Assessment Update • March–April 2005 • Volume 17, Number 2 5processes on the pilot assessment. Thus,students taking the mathematics portion ofthe pilot assessment test were administeredquestions on which they were required toshow proof of their answers.A bank of possible test questions wasdeveloped for each of the four pretestsand posttests. The stated goal was to mea-sure the value added to a student’s knowl-edge base after successfully completing acourse in the discipline. Faculty in eachdiscipline at each institution were free toselect the test questions from the bank ofquestions developed for their area by thedisciplinary subcommittee. Thus, al-though all of the disciplinary tests weresimilar at the different institutions, theywere not the same.The general education pilot assessmentwas administered to 2,141 students at thenine USHE institutions at the beginning andthe end of the spring 2001 semester. Pre-test and posttest scores were analyzed from699 College Algebra (Mathematics 1050)students, 164 U.S. Economic History (Eco-nomics 1740) students, 1,207 AmericanCivilization (History 1700) students, and71 U.S. National Government (PoliticalScience 1100) students. There were dra-matic improvements in student learningbetween the beginning and the end of thesemester. Improvement in scores frompretest to posttest ranged from a low of 36percent in American Civilization to a highof 169 percent in mathematics.A subsequent study was conducted to examine the purposes of the pilot as-sessment, how the pilot assessment wasorganized and implemented, what meth-odological issues were involved in creatingthe pilot assessment, and what recom-mendations participants in the study hadfor future assessment endeavors. Thisstudy used purposeful sampling of partic-ipants. In purposeful samples, the partici-pants who are selected have intimateknowledge of the phenomenon beingstudied. In addition, in this study, partici-pants were chosen who were “consideredto be influential, prominent and/or well-informed people,” in accordance withguidelines by Marshall and Rossman(1999, p. 113) for designing effectivequalitative studies. They were selected asresearch participants because of theirknowledge of and, in some cases, eventheir expertise in the creation of the pilotassessment that occurred in the spring2001 semester.Data collection methods included in-depth interviews of fourteen faculty par-ticipants, field note analysis, and analysisof public documents. The analyses ofpublic documents and field notes wereused to triangulate the analysis of the par-ticipant interviews. Data analysis com-bined the three separate but concurrentactivities of data reduction, data display,and conclusion drawing and verification(Miles and Huberman, 1994).The following highlights characterizethe major findings of the study:Purposes of the General Education Pilot AssessmentThe major purposes for the pilot assess-ment were found to be the following:• To demonstrate accountability• To improve teaching and learning• To compare USHE institutions• To demonstrate student, faculty, institu-tion, or system performance• To measure and report the gain in stu-dent academic growth• To articulate or accredit curricula,courses, degrees, or programs for stu-dent transfer or matriculation• To respond to the concern that compe-tence-based assessment conducted atother institutions (for example, WesternGovernors University) could adverselyaffect how USHE teaching and learningare measuredOrganization and Implementation of Pilot AssessmentThe following were identified as majorfeatures in organizing and implementingthe pilot assessment:• A standardized, norm-referenced na-tional assessment instrument was re-jected by statewide faculty.• The faculty designed a pilot assessmentthat was related to the USHE general ed-ucation curriculum, linked to the curric-ular goals of the USHE, and embeddedin regular course examinations.• Four faculty disciplinary subcommitteeswere asked to design the assessment in-struments.• The task force decided to report onlyaggregated systemwide assessment databecause members believed that anony-mity would increase participation, andthey were concerned that faculty, aca-demic departments, or institutionscould be punished for poor assessmentresults.• Senior scholars took an active role in thedesign of the pilot assessment.• The faculty of the ad hoc Regents’ TaskForce for General Education and facultyin general took the lead in designing andconducting the pilot assessment.Methodological Issues of the Pilot AssessmentSome of the major methodological issuesincluded the following:• Developing a bank of mostly multiple-choice test questions from which indi-vidual faculty could select test questionsfor use in pretests and posttests• Using course commonalities in the pilotassessment• Using pretests and posttests in the pilotassessment• Deciding to embed the pilot assessmentposttest in regular final examinations• Addressing concerns about the possibleeffect that students’ knowledge of pre-test results would have on their test-taking ability during the pilot posttest• Addressing student participation issuesin taking the test• Addressing concerns about test validityand reliabilityRecommendations for Future AssessmentParticipants were asked to offer recom-mendations for future general educationassessment in the USHE. Participant rec-ommendations included the following:• Recognize that assessment is importantin demonstrating accountability, demon-6 Assessment Update • March–April 2005 • Volume 17, Number 2strating the improvement of teachingand learning, and augmenting institu-tional accreditation.• Bring faculty together to communicateabout goals, objectives, curriculum, andassessment.• Provide funding for assessment.• Clarify assessment goals and objectives.• Link assessment to faculty development,accountability, and the USHE masterplan for higher education.• Include all stakeholders in the designand implementation of assessment.• Report assessment results to all stake-holders.Recommendations for Future ResearchBased on the results of this study, the fol-lowing recommendations for researchwere made:• Investigate the effects of the control thatvarious stakeholder groups have inplanning, designing, implementing, andreporting on assessment. Specifically,examine how control is exercised andshared and to what extent being in con-trol or having power influences inter-stakeholder relationships and assessmentoutcomes.• Examine the issues of stakeholderanonymity and faculty fear of punish-ment.Recommendations for PracticeBased on the results of this study, the fol-lowing recommendations for practitionerswere made:• Although linking accountability with theimprovement of teaching and learninghas many advantages, the current distrustand misunderstanding among some as-sessment stakeholders in Utah are suchthat postponing the link is recommendedfor the immediate future. Before a linkcan be established, Utah’s assessmentstakeholders need to discuss assessmentissues, including the responsibility andconsequences of teaching and learning;assessment purpose, design, implemen-tation, and reporting; and stakeholderparticipation and control.• An assessment advisory board of stake-holder representatives should be created.A major goal of the board should be toguide future assessment efforts in Utah.Specifically, the board would be chargedwith determining the overall design of an assessment by establishing thepurpose of the assessment, appointingindividuals and groups to design and im-plement the assessment, and determin-ing how and to whom the assessmentresults would be reported. An externalconsultant would be recruited and hiredby the Utah State Board of Regents toserve as an independent analyst of theassessment process and to certify the as-sessment results.• Faculty should be in charge of the de-sign of the assessment instrument butnot the entire assessment process. Fac-ulty know the curriculum, curriculargoals, course content, and how to mea-sure curricular outcomes better than anyother stakeholder group. Thus, facultycan best design what is to be measuredand how it should be measured.• Multiple assessment measures shouldbe used. The validity, reliability, andcredibility of Utah’s systemwide as-sessment will increase when multiplemeasures are used. Measures could in-clude multiple-choice examinations,essay examinations, portfolios, faculty-student interviews, and national stan-dardized examinations. A case has beenmade for using both criterion-referencedand norm-referenced examinations andrandomly selecting students from eachUSHE institution to take them.• Systemwide assessment should reportnot only the aggregated system resultsbut also the disaggregated institutionaland academic department results. How-ever, before assessment results are dis-aggregated, the advisory board shouldaddress concerns about anonymity, pun-ishment, and faculty resistance. Aca-demic integrity and quality and the needfor a transparent system demand that re-sults be reported at several levels. AcknowledgmentThis article is based on my doctoral dis-sertation, “Planning, Designing, andConducting Systemwide Assessment inHigher Education: A Case Study ofUtah’s General Education Pilot Assess-ment.” University of Utah, May 2003,Salt Lake City.Note1. Successfully completing an AmericanInstitutions course is a USHE gradua-tion requirement that students may sat-isfy by completing either Economics1740 (U.S. Economic History), History1700 (American Civilization), or Polit-ical Science 1100 (U.S. National Gov-ernment).ReferencesMarshall, C., and Rossman, G. B. Design-ing Qualitative Research. (3rd ed.)Thousand Oaks, Calif.: Sage, 1999.Miles, M. B., and Huberman, A. M. Qual-itative Data Analysis: An ExpandedSourcebook. (2nd ed.) Thousand Oaks,Calif.: Sage, 1994.Utah State Board of Regents. BiennialAssessment and Accountability Report.Salt Lake City: Utah State Board ofRegents, 2000.Philip I. Kramer is assistant professor in the Department of Educational Leadership and Foundations and the Department of Teacher Education at the University of Texas at El Paso.

Systemwide assessment of Utah\u27s general education courses

https://digitalcommons.csbsju.edu/cgi/viewcontent.cgi?article=1015&amp;context=oarca_pubs

Systemwide assessment of Utah\u27s general education courses

Abstract

Similar works

Full text

Available Versions

College of Saint Benedict and Saint John’s University: DigitalCommons@CSB/SJU