Hamasu, Claire

Kelly, Betsy

Journal of the Medical Library Association

PubMed

Claire Hamasu

Betsy Kelly

Crossref

Journal of the Medical Library Association JMLA

Assessment and evaluation is not a gut feeling: integrating assessment and evaluation into library operations

Digital Commons@Becker

EDITORIALAssessment and evaluation is not a gut feeling: integrating assessmentand evaluation into library operations*{DOI: http://dx.doi.org/10.3163/1536-5050.101.2.001The authors love assessment andevaluation, and we believe itshould be integrated into all as-pects of library programming.There. It’s been said. Reaching thispoint in our relationship withassessment and evaluation hasdefinitely had its ups and downs,and we still struggle with it. Sowhy speak out about this now?The current economic environmentmakes it more important than everfor libraries to showcase theircontributions to their institutions.Several hospital libraries have beenclosed in the National Network ofLibraries of Medicine, MidConti-nental Region (NN/LM MCR), inthe last few years. The VeteransAdministration Hospitals, antici-pating boomer librarians’ retire-ments are looking at large numbersof vacancies with the possibility ofno replacements. While academichealth sciences librarians are notanticipating their libraries beingclosed, they are fighting for re-placements for departing and re-tiring librarians and for budgets tooffer services to support morecomplex needs such as data man-agement and curation, systematicreviews, and provision of clinicalinformation at the point of care.Health care reform is requiringmore accountability. When assess-ment and evaluation are pervasive,librarians have the evidence toeffectively communicate how eachpart of the library contributes tothe success of the institution.The authors believe in the value oflibrarians and libraries and that theycontribute to the mission of thehospitals, health centers, universi-ties, and organizations they serve.Librarians have stories to tell, butthe data currently collected tend tofocus on isolated factors that do nottell the whole story. Librarians aredoing ourselves a disservice by notproviding data that show how wesupport meaningful use and dem-onstrate how library resources andservices contribute to the institution-al mission. Health sciences librari-ans must ask: What do librariansreally contribute to our institutions?Are patients receiving better carebecause of what librarians are do-ing? Which activities best supportour users? How does our trainingenhance lifelong learning skills?How do librarians contribute to theresearch process? What infrastruc-ture is needed for these activities?Once these questions have beenasked, librarians must decide whatmetrics will help us address them.Several terms related to thisdiscussion require definition. As-sessment and evaluation are oftenused interchangeably but they are,in fact, different activities. Byassessment, the authors mean theprocess of continually reviewingwork in progress. By evaluation,we mean review done at a point intime allowing the evaluator tomake judgments or draw conclu-sions about the impact andachievement of the desired out-comes of an action. Value is a termfrequently used when discussing alibrary’s contribution to the parentorganization. The word value im-plies merit, importance, or worth.In the context of evaluation, we arealso speaking of value-added, val-ue that exceeds expectations. Andfinally, there are metrics, a set ofaccepted or agreed upon measures.Many things can be measured, butmetrics are relevant and significantto the question at hand. Metrics areusually, but not always, quantifi-able. The assessment and evalua-tion practices of two organizations,discussed below, illustrate the ap-plication of these terms and thevalue that can be derived when theprocess is pervasive through theorganization.National Network of Libraries ofMedicine,MidContinentalRegionIn 2001, the National Library ofMedicine (NLM) awarded the NN/LM contract to the Spencer S. EcclesHealth Sciences Library at the Uni-versity of Utah. With this awardcame a new model of service to theregion, one that incorporated part-nering with Resource Libraries(leading academic libraries) in eachstate and establishing staff at eachof the Resource Libraries to carryout the NN/LM five-year contract.This new program was chal-lenged to not only coordinate adistributed staff, but to prove tothe NLM and to members of theregion that this new model wouldwork. Staff wanted to show notthat we were better than a central-ized program, but that we could beas effective in addressing the needsof the region’s constituents. Thisdesire to show impact was writteninto the proposal. We realized thatwe were at a critical moment in ourhistory and that, since we were atthe beginning of a first-time con-tract, we had the opportunity tocollect measures that could be laterused to show improvement. Li-braries can establish any day astheir first day and collect baselinedata to start measuring their effec-tiveness, but measures must betaken, or there is no way to chartprogress.To learn who our members are,we have gathered information onthem once every five years: informa-tion about their staffing, place andinvolvement in their parent organi-zations, budgets, collections, teach-ing, technology, outreach, and use ofvarious NLM resources, as well asprograms or services that havegained importance among libraries* This project has been funded in whole orin part with federal funds from the Depart-ment of Health and Human Services,National Institutes of Health, NationalLibrary of Medicine, under contractno. HHS-N-276-2011-00006-C with the Uni-versity of Utah Spencer S. Eccles HealthSciences Library.{ Based on a presentation at MLA ’11, the111th Annual Meeting of the MedicalLibrary Association; Minneapolis, MN;May 2011.J Med Lib Assoc 101(2) April 2013 85since the last assessment. In additionto providing details on our mem-bers, this assessment also alerts us tothe ways in which their libraryenvironments are changing. Wehave a core set of topics (e.g.,technology), but within each topic,the measures may change over time.For example while connectivity wasmeasured by bandwidth in 2003,today the metric is whether institu-tional policy blocks access to onlineservices and resources such as socialmedia.The NN/LM MCR program fo-cuses on six areas of outreach:assessment and evaluation, libraryadvocacy, health information liter-acy, education, technology, andmember services. Each of theseareas develops an annual plan toreach the five-year goals estab-lished by NLM. We regularlyassess whether we carried out theactivities that we set for ourselvesand evaluate to what extent wemet the indicators for project out-comes. For example, in the last fewyears, the technology program hasjudged its effectiveness by thenumber of members who adopteda technology that was promotedor taught. Our health informationliteracy initiatives have been mea-sured by the feedback receivedfrom collaborating organizationswho recount what we did toimprove their access to informa-tion. These stories, which relate theimpact we have on the region, arereported back to NLM so that theyunderstand the value we are add-ing to the region. Based on ourexperience, we know that ourmetrics provide information onour impact on our user populationand that by making assessmentand evaluation pervasive, the or-ganization and all of its programsprovide a holistic view of what weare doing to achieve the NN/LMmission and goals and the progressin achieving them over the life ofthe contract.Bernard Becker Medical LibraryAt Washington University in St.Louis, the Bernard Becker MedicalLibrary’s paper forms have givenway to databases where morecomplex metrics can be recordedand a variety of data collected andmanipulated. To collect informa-tion that truly documents the useand value of the library, the libraryrevised the Library Statistics Ap-plication created by the Universityof New Mexico [1]. More of thelibrary’s collections are electronic,making traditional gate count, cir-culation, and shelving statisticsirrelevant. Library Strategic Direc-tions 2012–2015, the library’s three-year strategic plan, addresses eachof the four missions of the Schoolof Medicine [2]. It outlines eighteenindicators for evaluating progressin seven strategic directions. TheBecker statistics software providesa way to demonstrate the breadthand depth of services providedby the staff. Many of the metricsneeded to demonstrate progress onthe strategic plan are captured inthe statistics database. Staff knowwhat services we are providing,how often, and for whom. Wecapture names and email addressesof those we work with and anec-dotal or qualitative notes aboutthe interaction. These data can bequickly analyzed and pulled intoMicrosoft Access reports that sum-marize how both the librarians andthe paraprofessionals contribute tothe library’s strategic plan and theSchool of Medicine’s mission. Theanalysis is included in the library’sannual report that accompanies ourbudget request to administration.Data collection began in late 2010 sowe cannot yet demonstrate whetherproviding this information to theadministration has had an impact.However, the consistency and de-tail of the collected data haveallowed us to illustrate that thelibrary’s services are evolving andgrowing and that the library isinvolved and invested in the edu-cation, research, patient care, andoutreach mission of the School ofMedicine.Libraries have traditionally col-lected data about resources andservices and have primarily usedit to compare themselves with otherlibraries. However, these data donot provide the whole story of theimpact the library has on its owninstitution nor how the librarycontributes to the institutional mis-sion. Think of the parable fromIndia of the elephant and the blindmen. By touching a single part ofthe elephant, each blind man un-derstands the elephant as the partthat he touched. Most librariesconduct evaluation the same way.They take a single service (clinicallibrarianship) or a single depart-ment (systems) and evaluate pro-ductivity, fulfillment of need, oruser satisfaction. From this evalua-tion, all that can be known about thelibrary is the trunk or leg or tusk. Inher article, ‘‘The Postmodern Li-brary in an Age of Assessment,’’Ray describes the modern library asbeing characterized by ‘‘predictabil-ity, certainty, control, absolutetruths, and order’’ [3]. Health sci-ences libraries are no longer in thatmodern age. Today is the age of thepostmodern library, characterizedby ‘‘subjectivity and multiple reali-ties; plurality and diversity’’ [3].The results of evaluation effortsneed to reflect this complexity, sothat a multidimensional story canbe related to our stakeholders, andlibrarians can better understand thecomplexity of our own libraries.Accrediting organizations andgovernment entities expect highereducation and health care to dem-onstrate how institutions meettheir missions and improve thequality of patient care and theeducational experience. Librariesare having similar demands placedon them. Librarians need to beleaders in demonstrating how toeffectively gather and use informa-tion to illustrate and promote ourvalue. To become leaders, healthsciences librarians need to inte-grate assessment and evaluationinto our organizations, collect datafor metrics that define our value,and report the evidence of ourvalue to our stakeholders.The authors look forward tothe time envisioned by Ray whenassessment and evaluation ‘‘is as-similated into the daily work ofeach staff member. Assessment isno longer viewed as an externallyimposed chore or a detour fromnormal routines. It becomes a self-organizing principle and helpscreate an organizational climateEditorial86 J Med Lib Assoc 101(2) April 2013that encourages inquiry, explora-tion and reflection’’ [3].Claire Hamasu, MLS, chamasu@,RML4.utah.edu, Associate Director,National Network of Libraries of Med-icine, MidContinental Region, SpencerS. Eccles Health Sciences Library,University of Utah, 10 North 1900East, Salt Lake City, UT 84112-5890;Betsy Kelly, MLS, MBA, betsy.kelly@wustl.edu, Associate Director,Bernard Becker Medical Library, andAssessment and Evaluation Coordinator,National Network of Libraries of Med-icine, MidContinental Region, Wash-ington University, 660 South EuclidAvenue, St. Louis, MO 63110References1. Swamback S. Library statistics ap-plication. 2008 Dec 18. In: Lobo vault[Internet]. Albuquerque (NM): Univer-sity of New Mexico [cited 22 Aug2012]. ,http://repository.unm.edu/handle/1928/7566..2. Washington University, School ofMedicine, Bernard Becker Medical. Li-brary strategic directions 2012–2015 [In-ternet]. St. Louis (MO): Washington Uni-versity in St. Louis. 40 p. [cited 22 Aug2012]. ,https://becker.wustl.edu/sites/default/files/BeckerReport_low.pdf..3. Ray KL. The postmodern library in anage of assessment [Internet]. ACRL TenthNational Conference; Denver (CO); 15–18Mar 2001: Association of College andResearch Libraries [cited 22 Aug 2012].,http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/pdf/kray.pdf..optimized for your library.EditorialJ Med Lib Assoc 101(2) April 2013 87

http://digitalcommons.wustl.edu/cgi/viewcontent.cgi?article=1044&context=becker_pubs

Assessment and evaluation is not a gut feeling: integrating assessment and evaluation into library operations

Abstract

Similar works

Full text

Available Versions

Crossref

Digital Commons@Becker