The authors explore readily available ways to assess library services such as inter-library loan fill rates, student satisfaction surveys, circulation rates of newly acquired material, and assessment of the bibliographies of student papers.  They also discuss their experience with administrating the LibQual+ survey

Sanders, Martha Rice

Varnet, Harvey

English

HELIN Digital Commons

Asking Better Questions: Small-Scale Assessment Measures That Inform Ongoing Work

DigitalCommons@Providence

Providence CollegeDigitalCommons@ProvidenceLibrary Faculty and Staff papers Phillips Memorial LibraryDecember 2004Asking Better Questions: Small-Scale AssessmentMeasures That Inform Ongoing WorkHarvey VarnetProvidence CollegeMartha Rice SandersHELIN Consortium, martha.sanders@iii.comFollow this and additional works at: http://digitalcommons.providence.edu/facstaff_pubsThis Article is brought to you for free and open access by the Phillips Memorial Library at DigitalCommons@Providence. It has been accepted forinclusion in Library Faculty and Staff papers by an authorized administrator of DigitalCommons@Providence. For more information, please contactmcaprio1@providence.edu, hposey@providence.edu.Varnet, Harvey and Sanders, Martha Rice, "Asking Better Questions: Small-Scale Assessment Measures That Inform Ongoing Work"(2004). Library Faculty and Staff papers. 3.http://digitalcommons.providence.edu/facstaff_pubs/3Asking Better Questions: Small-Scale Assessment Measures That Inform Ongoing Work   By Harvey Varnet and Martha Rice Sanders  The late Dick Palmer, professor of library and information science at Simmons College, loved words.  He had served in the navy as an intelligence officer, code-breaking Japanese during World War II.  Dick loved to challenge his classes to define the word knowledge. His favorite definition was “that information which allows us to ask better questions.”  Dr. Palmer was a pioneer in information science and, though he didn’t know it at the time, he was in front of the assessment movement in higher education.  He always asked “what for?” – if you didn’t have an answer, he quickly followed that remark with “why?”  The approach was gentle but firm and purposeful.  In small and medium sized colleges and universities, undertaking library assessment is definitely a challenge.  First, there’s the thought that this could be a major undertaking, and who has the time or on-staff expertise to do this in a meaningful way.  Getting staff interested in this topic is not always easy. Then there’s the complexity of linking anything you do in the library to the overall campus assessment program (if indeed one is fully formulated).  In the northeast, NEASC (the New England Association of Schools and Colleges) has been the prime driver in the assessment movement.  On their web site,1 the Commission on Institutions of Higher Education outlines standards and other policies, including an important “Policy on Institutional Effectiveness.”  NEASC has attempted to assist accreditation site visitors by formulating what it calls “pilot site visit questions.2” While the questions relating to Standard 7, Library and Information Resources, are helpful, they are limited in scope.  One area these questions do ask site visitors to consider is the development of information literacy skills – meaning, for most of us, bibliographic instruction or some library and/or institutional attempt to foster such skills.  This is an area where many of our colleagues do an excellent job of measuring their effectiveness.  Otherwise, what to do?  The culture of evidence One less intimidating way to look at assessment is to see our efforts as part of the greater culture of evidence. Peggy Maki, a noted authority on assessment and the Director of Assessment at the American Association for Higher Education, talks about support units and how outcomes assessment, while helpful, may not always fit our operations.  However, libraries perform lots of services, with accompanying data/statistics, that can inform our decision-making.  In Ms. Maki’s words, aggregating the data, analyzing it, and then doing something with it – asking better questions - is an important aspect in helping to create the culture of evidence that what we are doing is meaningful and worthwhile. Small-Scale Measures  At Providence College, we’ve tried to adopt techniques that tell us something about ourselves.  On an annual basis, we look at interlibrary loan fill-rate.  This data is easy to gather, as it is primarily OCLC-based.  The data tell us that we are very efficient at getting requests from students and faculty out to other colleges – less than one day, typically.  The chart below also shows us how, over time, the fill rates for copies (mostly journal articles) and monographs are remarkably consistent.  This allows our service staff to tell students and faculty, with some confidence, what they can expect if and when we don’t have an item and it is not available nearby in a sister consortium library.  Interlibrary Loan Fill Rates - Borrowing  When considering budgeting challenges, we occasionally look at book use rates.  Our process is to review whether the books we acquired two fiscal years ago circulated or not.  This tells us in some measure that we are buying the books that are needed to support the curriculum.  With a campus that is not research driven but research interested, we surely acquire books for their long-term value.  However, our mission is to provide an undergraduate, liberal arts education, so our purchasing decisions are made with this in mind.  One might ask, what is ‘success’ when looking at use rates.  One interesting way to look at this is via Richard Trueswell’s3 seminal work years ago that outlined the 80/20 rule – over time, 80% of circulation tends to come from 20% of a collection.  Our fiscal year 2002 acquisitions circulated as indicated below – not bad, considering the compressed time period of analysis and the oft-reported general overall downward trend in book circulation.   Book Circulation – FY 2002 Purchases  0 circulations = 3,261 1-5 = 2,875     5+ = 56  Total acquisitions = 6,192  Another small-scale approach we’ve taken is to ask faculty for the bibliographies of papers or presentations that their seminar students take or from major departmental courses.  We’ve had limited success at getting these bibliographies, but each set we have looked at has proven useful.  Our goal in the review of the bibliographies is to determine if, within the library or our consortium, we can provide 80% of the materials the students needed.  This approach also deals in reality – we look at actual resources used, versus speculation that “they don’t have anything I need to do my course work.”    Our analysis of bibliographies in social work benefited both areas.  The perception that we had very few journal subscriptions was incorrect – we were able to prove this – and this helped change the tone of our discussions with the department.   We reviewed the senior seminar papers for two years and discovered that we were able to supply 62.5% of the citations in 2001 and 87% in 2002.  When the department came up for its professional accreditation, we were an active and welcomed partner in the process.  The library was cited as a program strength.  In music, we were able to review the bibliographies for two courses.  In each course, we held over 90% of the books used and 100% of the journals cited.  This data helped dispel the perception within the Music Dept. faculty that the library held none of the titles needed by students for doing their research.  In addition, the library purchased one book for the collection that was accessed by a student through ILL for their paper.  In chemistry, analysis of student seminar citations revealed that we held or provided access to 77% of the journals cited.  For books, 46% were available in the consortium, though we held none used by students.  This led to an interesting dialogue with the chair of the department.  After reviewing what we did not hold, the chair indicated that we ought not to buy the works cited.  The chair indicated that though useful for students’ papers, the books were not a particularly good fit for our collection.  Larger-Scale Measures Another aspect of operational analysis is benefiting from the overall assessment work that the college undertakes.  Similar to many institutions, Providence College administers the Student Satisfaction Inventory (SSI) – in our case, to freshmen and seniors.  The Noel-Levitz4 SSI has two library related questions:  Library staff are helpful and approachable, and Library resources and services are adequate.  While these questions may not be ideally stated (for many of us, this is actually four and not two questions), the results give us a look at how our primary clientele generally perceive us. The SSI uses the gap analysis technique for assessment, with the ‘gap’ being the difference between students’ ratings of importance and their ratings of satisfaction, each on a 1-7 point scale (7 is best).  We have data from three such SSI surveys (1997, 2000, 2003).  In 2003, for the staff question, students tend to rate us well, with a ‘gap’ of 0.26.  For the resources and services question, students aren’t quite as satisfied – our ‘gap’ is 1.02.  In both cases our scores are slightly ahead of our national peers.  And, in both cases, the ‘gap’ has decreased from the previous assessment, a trend in the right direction.  Our scores for the staff question also rank us as one of the highest rated service/support areas on campus. Because the college has an ongoing assessment program, we received institutional support in 2003 to participate in the LibQual+ survey5. This nationally-normed survey was developed at Texas A & M University in joint cooperation with the Association of Research Libraries.  This survey measures four areas – Affect of Service, Access to Information, Personal Control, and Library as Place.  Our library was built in 1969, and we knew it was badly in need of updating and renovating.  What we learned from LibQual+ was that students felt strongly about the library as place, in our case in the negative.  The radar chart produced in our results summary is very red – not a good color to see, but a clear message to the library and college administration.     We began a long-term renovation process in the summer of 2003, completing the library’s main floor in the spring of 2004.  Our plan is to participate in Libqual+ next year, again surveying sophomores and juniors to determine if their perceptions have changed.    As the library improved physically, it seemed to us that students were coming in to use the library in larger numbers.  We keep evening attendance headcounts, so we decided to compare the current (renovation) year to the previous year’s counts.  The chart below shows how attendance was up for each week analyzed.  The counts for each year were taken during the week before Thanksgiving, in the first two weeks of December, and for all of January.  Library Attendence  We reported this information to the college administration, and they in turn reported it to the Board of Trustees at its late January meeting.  There was general recognition that the renovations were well worth the expense involved and the Board indicated its support for the next phase of renovations.  The future We plan to continue with a mix of assessment activities.  We are now planning to look much more closely at our instruction program – how are students and faculty perceiving its usefulness?  We teach between 50 to 70 classes per year and we need to know more from the students as to whether or not the information they receive has utility and value.  We are also interested in some general assessment of the high-profile activity of reference desk service.  We’re working with our campus assessment coordinator on a point-of-use survey for reference, hoping that we’ll get useful feedback from users.  While we get anecdotal information about reference assistance, this information tends to be at the extremes – either someone very pleased or quite the opposite.  We’re hopeful that more user-response data will give us a balanced picture of this primary and important service.  We will continue to analyze and review interlibrary loan data and book use rates.  We also plan to review course-related bibliographies from those faculty willing to share.  This work has proven to be very beneficial and has qualitatively changed the discussions we have regarding library services with our teaching colleagues.  In our next review of bibliographies, we’ll attempt to see if there is any correlation between the grade students earn on their papers and their use of refereed, professional sources. The college will continue to do the SSI survey on a regular basis, and we’ll extract that data to see how students are perceiving us in general.  And finally, we are hopeful for institutional support this coming year to do another LibQual+ survey.  The cost for this activity is quite different from all our other assessment efforts, but the return on investment is well worth the effort.  In its “Standards Revision Discussion Paper,” NEASC notes:  “Evidence is fundamental to institutional decision-making…  Increasingly, as will be reflected in the revised Standards, the Commission expects that institutions gather and use evidence as the basis for this evaluation and improvement.”6 At Providence College, we in the library are attempting to take this evidence-based approach seriously.  We’re confident our ‘mixed’ approach to analysis is on track.  The blend of analytical activities we’ve undertaken has helped to make us a better campus service.  This ‘mixed’ approach is also on the kind of scale where the evaluation work doesn’t overwhelm any other ongoing activity.  To conclude with more wisdom from Dick Palmer, “Just count – it’s amazing what the numbers tell you about your prejudices.”  Providence College profile Providence College is a primarily undergraduate, liberal arts, Catholic institution that is committed to fostering academic excellence through the sciences and humanities.  The college enrolls approximately 4,200 FTE undergraduate students and 330 FTE graduate students.  There are 250 faculty offering degree programs in over 40 areas.  Harvey Varnet is library director, e-mail: hvarnet@providence.edu, and Martha Rice Sanders is coordinator of collections services and catalog librarian, e-mail: msanders@providence.edu, at Providence College in Rhode Island  © 2005 Harvey Varnet and Martha Rice Sanders C&RL News, June 2005, Vol. 66, No. 6, p. 461-465  1. New England Association of Schools and Colleges web site: http://www.neasc.org/cihe/assessment.htm2 NEASC Pilot Questions web site:   http://www.neasc.org/cihe/site_visit_questions.PDF3 Those unfamiliar with Dr. Trueswell’s work should review one or more of the following citations. Trueswell, Richard, Some behavioral patterns of library users: the 80/20 rule.  Wilson Library Bulletin. 1969; 43: 458-461.    Trueswell, Richard, User circulation satisfaction vs. size of holdings at three academic libraries,   College and Research Libraries. May 1969; 30(3): 204-213. Trueswell, Richard,  Determining the optimal number of volumes for a library's core collection.  . Libri (International Library Review). 1976; 19(1): 49-60. 4. The corporate web site for Noel-Levitz, Inc. can be found at:    http://www.noellevitz.com/solutions/retention/satisfaction/index.aspTo review gap analysis use, see “The Measurement of Gaps in Service Quality” by Robert Lewis and David Klein (pages 33 – 38) in The Service Challenge: Integrating for Competitive Advantage, Chicago, American Marketign Association, 1987.  5. The underpinnings of LibQual+ are based on the pioneering “servqual” work of A. Parasuman and his colleagues. See Zeithaml, Valarie A.,  Parasuraman, A., and Berry, Leonard L., Delivering Quality Service: Balancing Customer Perceptions and Expectations, New York, Free Press, 1990.  For Parasuman et al., the only opinion that matters is the customers – all others are irrelevant.  To read further on LibQual+, see the following: Score Norms for Improving Library Service Quality: A LibQUAL+(TM) Study.  Cook, C., Heath, F. and Thompson, B.,  portal: Libraries and the Academy 2, no. 1 (2002) 13-26.      Reliability and Structure of LibQUAL+™ Scores: Measuring Perceived Library Service Quality. Thompson, B., Cook, C., and Thompson, R.L.,  portal: Libraries and the Academy 2, no. 1(2002): 3-12.     ARL Index and Other Validity Correlates of LibQUAL+™ Heath, F., Cook, C., Kyrillidou, M. and Thompson, B., portal: Libraries and the Academy 2, no. 1 (2002): 27-42.  The web site for Libqual+ can be fount at:   http://www.libqual.org6. NEASC’s Standards Revision Discussion Paper (issued in December 2003) can be found at: http://www.neasc.org/cihe/revisions/summary_of_significant_changes.pdfThe quote from the paper is on page three, second bullet.  

https://digitalcommons.providence.edu/cgi/viewcontent.cgi?article=1002&amp;context=facstaff_pubs

Asking Better Questions: Small-Scale Assessment Measures That Inform Ongoing Work

Abstract

Similar works

Full text

Available Versions

HELIN Digital Commons

DigitalCommons@Providence