This presentation describes a number of qualitative assessment practices that can help technical services managers assess their effectiveness. Strategies include process improvement initiatives, customer service surveys, focus groups, benchmarking, and more

Mugridge, Rebecca L.

Poehlmann, Nancy M.

English

University at Albany, State University of New York (SUNY): Scholars Archive

University at Albany, State University of New YorkScholars ArchiveUniversity Libraries Faculty Scholarship University Libraries2014Assessment Strategies for Technical ServicesRebecca L. MugridgeUniversity at Albany, State University of New York, rmugridge@albany.eduNancy M. PoehlmannUniversity at Albany, State University of New York, npoehlmann@albany.eduFollow this and additional works at: http://scholarsarchive.library.albany.edu/ulib_fac_scholarPart of the Cataloging and Metadata CommonsThis Presentation is brought to you for free and open access by the University Libraries at Scholars Archive. It has been accepted for inclusion inUniversity Libraries Faculty Scholarship by an authorized administrator of Scholars Archive. For more information, please contactscholarsarchive@albany.edu.Recommended CitationMugridge, Rebecca L. and Poehlmann, Nancy M., "Assessment Strategies for Technical Services" (2014). University Libraries FacultyScholarship. 5.http://scholarsarchive.library.albany.edu/ulib_fac_scholar/5       Assessment Strategies for Technical Services  Rebecca L. Mugridge Nancy Poehlmann SUNYLA June 12, 2014 Agenda • Assessment landscape • Our research on assessment • Goals of assessment • Five assessment methods • Workflow assessment • Customer surveys • Focus groups • Benchmarking • Periodic reviews  Assessment landscape   • ACRL Value of Academic Libraries report • LibQual™  • ARL biennial assessment conferences • ARL SPEC Kit on Library Assessment • ALCTS e-forum on assessment   SPEC kit on Library Assessment  • Spec Kit 303 (Stephanie Wright and Lynda S. White) • Assessment of technical services activities addressed in one question: • “Please indicate which of the following departments/units your library has assessed since 2002 and what methodologies were used for those assessment” SPEC Kit on Library Assessment: Cataloging  • Cataloging: Number of respondents: 62 • Surveys: 4.8% • Qualitative methods: 9.7% • Statistics collection and analysis: 69% • Usability: 1.6% • Other (Benchmarking, Unit cost analyses, Balanced Scorecard, Process improvement): 14.5% • Have not assessed: 24%  SPEC Kit on Library Assessment: Acquisitions  • Acquisitions: Number of respondents: 62 • Surveys: 14.5% • Qualitative methods: 13% • Statistics collection and analysis: 74% • Usability: 0% • Other: 13% • Have not assessed: 21%  SPEC Kit on Library Assessment: Preservation  • Preservation: Number of respondents: 61 • Surveys: 13% • Qualitative methods: 13% • Statistics collection and analysis: 57% • Usability: 0% • Other: 8% • Have not assessed: 33%  ALCTS E-Forum on Statistics and Assessment  • What statistics are collected by technical services and how are they collected? • The differences between collecting statistics and providing meaningful reports. • How do technical services utilize statistics to assess the effectiveness of their operations? What benchmarks are used to define success? • How do technical services operations factor into efforts to define the value of the library to the parent organization or community? • http://www.ala.org/alcts/confevents/past/e-forum   Technical Services Assessment  • Rebecca L. Mugridge, “Technical Services Assessment: A Survey of Pennsylvania Academic Libraries” Library Resources and Technical Services 58:2 (April 2014): 100-110. • Survey sent to 120 PA academic libraries • 63 responses • 53% response rate  Technical Services Assessment: Research Findings  • Methods of assessment included: • Gathering statistics: 84% • Gathering input from non-technical services staff: 44% • Conducting customer service surveys: 25% • Benchmarking: 19% • Suggestion box: 12% • Focus groups: 10%  Technical Services Assessment: Research Findings    • Departments/functions assessed: • Cataloging/Metadata: 57% • Acquisitions: 57% • Electronic Resources Management: 45% • Preservation: 26% Goals of assessment  • Streamline or improve processes • Make better decisions • Lower costs • Reallocate staff or other resources • Identify activities and services that can be eliminated • Inform strategic planning activities • Communicate with customers or administration  Assessment activities  Five methods of assessment that we’re going to discuss in this presentation:  • Workflow analysis and assessment with a facilitator • Customer surveys • Interviews or focus groups • Quality assessment • Benchmarking   Workflow analysis and assessment  • Most-commonly reported form of assessment in the library literature • Examples: • Assessment of technical services workflow • Assessment of cataloging and database maintenance • Workflow assessment and redesign • Streamlining work between acquisitions and cataloging • Assessment of shelf-ready services CQI at Penn State  • Penn State’s model is based on Continuous Quality Improvement, using a five-step model: • Where are we now? • Where should we be in the future? • How will we know when we get there? • How far do we have to go? • How do we get there? • http://www.psu.edu/president/pia/cqi/planning_model1.pdf   CQI improvement teams  • CQI Improvement Teams need: • A clear process • Support from a sponsor • An administrative leader for the team • A facilitator • We used the Fast Track approach: • http://www.psu.edu/president/pia/innovation/fasttrack.pdf  When is CQI useful?  • When useful: • Multiple units • Complex workflow • Workflow has been in place for a long time • Differences of opinion exist about how to address workflow changes How CQI works  • Start with a list of issues or questions • Include all stakeholders in process • Kick off meeting with sponsors • Make an effort to understand the current process • Identify areas for improvement • Map new process and report back to sponsors • Follow up assessment FastTrack CQI team 1  • Video processing for Media Technology Support Services (AV rental/booking/support for classrooms across university)  • Cataloging • Acquisitions • Media Technology Support • Looked at workflow between the three units • Goal to decrease processing time and increase efficiencies  FastTrack CQI team 2  • Catalyst: Reorganization of government documents processing • Stakeholder departments: • Cataloging • Acquisitions and serials • Social Sciences Library • U.S., PA, UN, Canadian, EU, etc.   FastTrack CQI team results  • Streamlined processes with fewer hand-offs • Greater efficiencies • Faster turn-around times (Acquisition to Shelf) • Better understanding of workflow • Improved documentation • Greater confidence in established processes Customer service survey Benefits of surveys:  • They can tell you something that you don’t know • They can corroborate something that you already suspect • They demonstrate to your customers that you care what they think • They can serve as a public relations or marketing tool • They can be used to support change, request funding, or pursue further assessment efforts  Customer service survey Example 1   • Cataloging and Metadata Services (2011) • Queried subject and campus libraries • Not anonymous • One survey response per library  Customer service survey Example 1 survey questions  • At which branch, subject, or campus library do you work? • What services do we provide to your unit? • How happy are you with the following aspects of this service: • Speed of services • Quality of services • Speed of response to reported problems • If you wish, describe specific service experiences in detail. • Do you feel that you know to whom to talk about service issues as they arise? [Y/N]  Customer service survey Example 1 survey questions • How comfortable do you feel with the process of asking for help? • Not comfortable • Somewhat comfortable • Very comfortable • Are you able to find information or documentation on the Cataloging and Metadata Services website? [Y/N] • Describe your process for asking questions about cataloging services. • If you could see one new service provided to your library by Cataloging and Metadata Services, what would it be? • Do you have any additional comments?  Customer service survey Example 2  • Departmental Survey (2012) • All library employees surveyed • Anonymous • General and specific questions • Open ended questions  Customer service survey: Example 2 survey questions • Rate your overall satisfaction with the services provided by [the unit] for each of the following aspects: • Accuracy • Completeness • Effectiveness • Efficiency • Timeliness • Have you ever had any interactions with [the unit’s] website? • Do you have any suggestions about the currently provided services or any new services that you would like to see offered by [the unit] • How often do you interact with [the unit]? Interviews or focus groups  • Informally as part of a periodic “checking in” with customers • Example: Periodic meetings with subject library staff • Most included all of their staff available at the meeting time • Results: better communication with our customers and greater comfort level with asking questions  Research on focus groups in cataloging  • K.C. Elhard and Qiang Jin, “Shifting focus: Assessing cataloging service through focus groups,” Library Collections, Acquisitions, & Technical Services 28:2 (2004): 196-204.  Sample focus group topics • Do you find it easy to communicate with the three cataloging units? • Do you ever use the Technical Service Division Web page to identify appropriate contacts to resolve problems? • Are maintenance problems you encounter quickly resolved to your satisfaction? • What do you find most confusing about what the cataloging units do? • What do we do in cataloging that is the most helpful to your library? • What one service would you like to see cataloging provide which is not currently provided? • Is there anything else you would like to tell us about cataloging? Activity #1 If you were going to conduct a customer service survey or a focus group at your library, what burning questions would you like to have answered? Quality assessment: Example 1 Annual cataloging reviews:  • Each cataloging team conducts own review • Develop own process • Write report • What was the process? • Training needs identified? • Policy issues identified? • Overall assessment of the process itself? Quality assessment: Example 2 Training assessment:  • Yale University: retrospective conversion project • Retraining program  • Quality assurance testing • Evaluated for a period of time • Major vs. minor errors • Made decisions based on the results • Could be conducted yearly or as needed  Quality assessment: Benefits  • Ensures that everyone is following policies and procedures • Identify training needs • Team building exercise • Open discussions • Non-threatening questioning • Good management practice • Builds confidence Benchmarking Definition:  Benchmarking is the process of comparing one’s own policies, procedures or other factors, e.g., statistics, to other institutions for evaluative purposes or to determine best practices Statistical benchmarks • Mechael D. Charbonneau, “Production Benchmarks for Catalogers in Academic Libraries: Are We There Yet?” Library Resources & Technical Services 49:1 (2005), 40-48. • Findings: • Defining production benchmarks for cataloging doesn’t work very well • Cataloging is highly specialized and can’t be quantified in the same way as mechanized-based measurements. • J. Buschman and F.W. Chickering, “A Rough Measure of Copy Cataloging Productivity in the Academic Library.” Library Philosophy and Practice, http://digitalcommons.unl.edu/libphilprac/139  • It is possible to determine benchmarks for copy cataloging  Benchmarking as an Assessment Tool  • Rebecca L. Mugridge and Nancy M. Poehlmann, “Benchmarking as an Assessment Tool for Cataloging” (Manuscript under review). • Survey conducted on AUTOCAT • 92 completed surveys • 20 libraries reported using benchmarking (22%) • 9/10 libraries reported that they planned to use benchmarking again within the next five years   Goals of benchmarking: Research Findings  • Improve or streamline processes: 72% • Make better decisions:61% • Improve services: 33% • Reallocate staff or other resources: 33% • Explore offering new services: 22% • Inform strategic planning activities: 22% Information collected: Research Findings  • Procedures: 82% • Statistics: 73% • Policies: 55% • Staffing levels: 36% • Best practices: 55%  Goals and results: Research Findings Goal or Result Libraries that selected this as a goal of benchmarking (n=18) Libraries that selected this as a result of benchmarking (n=10) Improve or streamline processes 13 (72.2 percent) 7 (70 %) Make better decisions 11 (61.1 percent) 5 (50 %) Improve services 6 (33.3 percent) 3 (30 %) Reallocate staff or other resources 6 (33.3 percent) 3 (30 %) Explore offering new services 4 (22.2 percent) 1 (10 %) Inform strategic planning activities 4 (22.2 percent) 1 (10 %) Reporting benchmarking results: Research findings   • Informational report to library administration: 60% • Annual report: 30% • Assessment report: 20% • Presentations: 10% Benchmarking pros and cons: Research findings  • Advantages • Improve performance • Generate ideas • Encourage a continuous improvement mindset • Disadvantages/Challenges • Apples to oranges comparisons • It’s difficult to identify a peer group Assess the assessment  • Some assessment efforts may prove to be more effective than others • Did the assessment effort give you the information you need to meet your goals? • If not, you may choose another approach or refine your current approach • Document and share the results (internally, and if possible, externally)  Need for further research  • Research on specific assessment methods • Assessment of cataloging and technical services in different types of libraries: how are our needs different; how are they similar? • Qualitative benchmarks for technical services • Share experiences at conferences and other venues • Share experiences through case studies  • Technical services assessment toolkit Conclusion  • There are many ways that we can conduct meaningful assessment of cataloging activities: • Workflow analysis and assessment with an outside facilitator • Customer surveys • Interviews or focus groups • Internal evaluation, assessment, or reviews • Benchmarking  Activity #2  Given the opportunity, what would you assess at your library and why? What method or methods would you use? Questions? Rebecca L. Mugridge   Associate Director for Technical Services and Library Systems University at Albany Libraries LI-B34E University Library 1400 Washington Ave. Albany, NY 12222 email:  rmugridge@albany.edu  phone:  (518) 442-3631  Nancy Poehlmann Head, Cataloging Services University at Albany Libraries LI-B35A University Library 1400 Washington Ave. Albany, NY 12222 email:  npoehlmann@albany.edu  phone:  (518) 442-3889     

Assessment Strategies for Technical Services  (Presentation)

https://scholarsarchive.library.albany.edu/cgi/viewcontent.cgi?article=1004&context=ulib_fac_scholar

Assessment Strategies for Technical Services (Presentation)

Abstract

Similar works

Full text

Available Versions

University at Albany, State University of New York (SUNY): Scholars Archive