This poster will address the development process of a qualitative evaluation tool to aid in the thorough analysis of library resources at the University of Maryland. Specifically, our project looks at the use and added value of this tool for the building, reflecting, and analyzing connections between qualitative and quantitative data. This will allow for more meaningful justifications of budgetary decisions than compared to cost and use metrics alone. Given the necessity for meticulous review of continuing resources, our project addresses a request for enhanced transparency from the university faculty and library oversight body and serves as a useful tool for accountability and justification of impactful decisions for stakeholders internally and externally. We will discuss the extant literature and the need for this type of tool, the development process including the output planning and data input format, the initial reception of the project, and future goals and planning for our initial usag

DePope, Leigh Ann

Frank, Cindy

Ritchie, Stephanie

Sly, Jordan

English

Digital Repository at the University of Maryland

Jordan S. Sly, Leigh Ann DePope, Cindy Frank, Stephanie RitchieUniversity of Maryland Libraries, jsly@umd.edu, ldepope@umd.edu, cfrank@umd.edu, sritchie@umd.edu The Time has come…To Build, Reflect, and Analyze Connections Between Qualitative and Quantitative DataWe would like to thank the organizers of the Charleston Conference; the research and subject librarians of the University of Maryland Libraries who provided valuable feedback on the project; Dan Mack, Associate Dean for Collections and Maggie Saponaro, Collection Development Strategies for their continued support and encouragement for this work.To address the specific questions we are looking to answer, very little literature provides direct guidance or evidence towards developing a qualitative model for the evaluation of library resources. The majority of related literature tends to fall into one of two categories: 1) literature about the usability and evaluation of database construction or 2) the use of statistical information to guide decisions. Additionally, much of the literature in this area tends to be outdated and therefore focuses on evaluation criteria unnecessary for our current needs. That said, the following review of literature addresses the trends we were able to determine and the basis of evidence on which to develop our current proposed plan as well as a future direction for this project.From the outset, librarians have been working to develop methods to assess the value of expensive and difficult electronic databases and resources. Large (1989) details several evaluation criteria for databases: scope, authority, accuracy, uniqueness, comprehensiveness, retrieval capabilities, and support services. Johnson (1996) stipulates the “higher relative costs and greater financial risk of electronic materials” which make evaluation an important criteria for the acquisition and retention of these materials. As such, Johnson considers initial and continuing support costs in the criteria outlined. Additionally, Johnson includes lists of selection criteria for materials in general and electronic resources in particular. Tables for both cost and non-cost criteria are provided. Jacso (1997) provides a very thorough review of database content evaluation with an insistence that content aspects of quality are more important than other criteria which may be used to evaluate databases. Aspects of content quality include subject scope, composition, source and journal coverage, geographic, language and time-period coverage, currency, accuracy, consistency and completeness. Literature exploring each one of these elements is summarized and Jacso finally concludes that database evaluation will increase and become more difficult. Further, Jacso (2001) builds on these basic notions, of quality and adds the criteria of recency and inclusion of new information. These formulations largely look at the database from a structural perspective and usability focus on now far outdated materials.  Due to the age of this article, many of the examples and some of the criteria (e.g., times of availability, system of charging for use) are outdated to the point of being useless. Building from these early examples, Metz (2000) discusses pricing, particularly in regard to discounts born from cooperative acquisition and volume of users. Metz also tackling licensing considerations including rights of libraries, and negotiation of reasonable terms to meet use needs such as for instructional purposes. Functional elements such as interface, system maintenance, ADA compliance, and other technical considerations are covered. Metz also tackling archiving or perpetual rights considerations including allowances for archival copies of materials. Natarajan (2003) discusses general criteria for electronic resources divided into primary and secondary considerations. Primary considerations include relevance, content, scope, organization of materials and quality factors such as authority, content, and unique attributes. Secondary considerations include cost, access and technical support and requirements. Database quality criteria originally determined by the now-defunct Southern California Online Users Group are further enumerated in 10 different areas: consistency, coverage/scope, timeliness, accuracy, accessibility/ease of use, integration, output/exporting, documentation, customer support, and value -to-cost ratio. Importantly, Dalton and McNicol (2004) established the need for the mixed method approach to understanding the true value of particular materials for the goal of sustaining lifelong learning using the eVALUEd method outlined in their paper. Using this method, the authors argue, a more holistic view of the resource and a more accurate picture of the user are developed, the value of which is the ability to make more informed decisions. Crucially, the researchers highlight the need for a statistically sound method for analyzing the qualitative results in order to use the data in an effective way. If not exactly in response to previous studies, other researchers such as Kyrillidou and Giersch (2004) highlight a “need for new measures” particularly noting satisfaction of users as a key qualitative metric offered by respondents to the “Scottsdale Survey,” made up of responses from 22 Association of Research Libraries’ library representatives. Unfortunately, the e-metrics added to the ARL as a result of the survey feedback were mainly quantitative in nature. Studies such as these indicate a shift in the understanding of evaluation metrics, however and serve as a balancing point between the simple evaluation of a database from a construction-level to the more robust assessment of material as needed in the on-going balancing of cost against performance for the users. This shift is clear in Powers (2006) wherein the researcher identifies several criteria for database evaluation projects. She splits the data criteria into two types: known data and evaluation. Known data maybe quantitative e.g, use or cost per use, or qualitative like title or peer comparisons. Evaluation data would be criteria ranked or given value by reviewers. Content, uniqueness of content, ease of use, instruction usefulness, quality and need are some of the evaluation criteria suggested by Powers. Questions for each criterion help guide evaluation decisions. Like Natarajan, Gebhard (2010) addresses many of the critical issues surrounding both the challenges of determining appropriate metrics and the desires of such a project in finding robust methods to assess digital resources. In her model, Gebhard sets out to understand the value of eight different database aggregators to art history researchers. As such, what Gebhard is looking to understand is more than the pure intrinsic value of access, but also the value added to each resource by nature of its uniqueness, usability, and availability of proprietary material. Through a questionnaire matrix, Gebhard assesses the unique value of multiple databases against their competitors in order to determine the true value of each resource. What is clear through this selective literature review is a narrative indicating the increased need for the pairing of statistical data and user-added, qualitative feedback to hopefully find, as Gebhard attempted, an evidence-based, multidimensional model to assess the true value of a given resource. Like most university library systems, the University of Maryland has been finding ways to maintain research collections including subscriptions to databases of scholarly material with an increasingly shrinking or flat budget. Due to the extraordinary rate of inflation with these materials, librarians have had to make difficult cuts to valuable subscriptions. Retention and deselection decisions are contentious and can lead to problems for all campus library users. This often leads to competitions and resource hoarding in order to avoid difficult and rigorous assessments of the materials. Typically metrics are used to gauge a database’s use in terms of rate of accession and cost-per-use of the material. Our project aims to add a qualitative assessment matrix to this process in order to enhance the meaning and context of these simple numbers in order to aid in the decision-making process and to provide a new level of accountability to the process by making the criteria for decisions transparent and available to colleagues and campus stakeholders. This poster reflects our beta-stage of development including our proofs of concepts, trials, and feedback from our colleagues. Mission: Provide librarians with collections management responsibilities the tools for a thorough analysis of resources in a systematic and robust wayGoal: To be able to maintain a database of qualitative data to facilitate meaningful, accurate, and descriptive analysis and assessments of library resourcesSo what?: Our project looks at the use and added value of this tool for the building, reflecting, and analyzing connections between qualitative and quantitative data. This will allow for more meaningful justifications of budgetary decisions than compared to cost and use metrics alone. Given the necessity for meticulous review of continuing resources, our project addresses a request for enhanced transparency from the university faculty and library oversight body and serves as a useful tool for accountability and justification of impactful decisions for stakeholders internally and externally.A handful of organizations have attempted to create tools to evaluate databases and other electronic resources using qualitative criteria. We have reviewed these tools and selected some features as possible templates or practices for our databases evaluation work. eDesiderata - http://edesiderata.crl.edu/eDesiderarta functions as a source of market intelligence information for research libraries to evaluate and manage electronic resources. Some of the seeds of this project came from the now defunct Joint Information Systems Committee (JISC) Academic Database Assessment Tool, a tool to assess database journal coverage. Jasco (2010) provides a good overview of this tool and qualitative data available within.eVALUEd - http://www.evalued.bcu.ac.uk/eVALUEd provides a set of tools to help academic libraries develop their own evaluation practices for electronic resources. Some best practices around themes such as user experience, planning, management, and impact offer templated qualitative data collection and analysis tools. Dalton and McNicol (2004) share details about the development and use of this tool. Journal Review Tool - James Madison University (JMU)The Journal Review Tool is an online survey tool to collect structured feedback from the JMU campus community including qualitative and quantitative measures. Duncan, C.J., & O’Gara, G.M. (2015) details the development and use of this tool to review electronic resources.Description and MethodsInitially, we set out to create a simple spreadsheet to record simple data. It became clear quickly, however, that a more dynamic system was needed. As such, we put together a Qualtrics survey with a series of questions allowing librarians to both accurately and more comprehensively describe resources. For ease of use, the survey asks both direct questions about the qualities of the resources as well as open-ended areas for user input. This design accomplishes two distinct goals:1) The directed questions give librarians a chance to evaluate materials on an equal footing across disciplines by providing a clear set of valuable criteria  2) The Freedom to express “X” factors and value-added components of a resource including relative use formulas to indicate high departmental impact despite relatively small usage numbers (e.g. a small Classics department with nearly 100% usage, vs. a large engineering department with a lower impact percentage yet higher usage numbers) As shown  in the survey provided below, these questions seek to mainly address the following concerns:● Scope and completeness of content ● Core users and their institutional and research needs● Exclusivity of content ● Resource mechanics● Resource restrictions (and how this conflicts with other needs)● Peer institution ownership                                     early model using a scoring method to assess valueEach record is organized by the title of the resource. Using the clipboard icon on the right-hand side of the screen, we can drill-down into the specific record to view the full description provided by the librarian. We can look at these results both as an individual record for a specific resource and as a full-view of all resources evaluated which allows decision-makers to understand a fuller context for the resource within the university’s holdings.Because of the importance of this project and the cross-departmental nature of its purpose, we greatly value the feedback we have received from our colleagues in the libraries. The feedback has generally fit into the following areas:TrepidationSome colleagues have seen this project as a potential threat to librarian expertise and have expressed fears relating to what they view as a restrictive set of criteria.ExcitementOther colleagues have provided the exact opposite response indicating that they are happy that resources are given a clear set of metrics for equal evaluation.UnderstandingThe majority of the feedback we have received has been generally positive, but with minor reservations or suggestions for improvement. Very few librarians have expressed true trepidation and it has been clear that consistent, accurate, and illustrative communications is a vital component to this project. This is especially true as aspects of this project add work to the complicated job of resource evaluation. This added work, however, addresses crucial aspects needed for evidence-based practice and accountability to our campus stakeholders supporting the decisions made about our shared resources. ● One of the key findings through this project was the complexity and the vast scope inherent in determining the true value of a given resource. ● Further workshop and focus test questions and response matrices for usability and feedback quality/usefulness for all stakeholders and librarians● Continue to work to develop a tool that balances librarian intuition with direct evidence and data to give an accurate picture of a given resource without removing librarian agency. ● Hone the instrument to best capture the  “true value indicator.” This measurement would include both quantitative statistics and qualitative measurements coded for scoring. ● Create a workable and scalable database for capturing, storing, and recalling data on resources with librarian access to use the data for comparisons and resource discussions● Visualization and display models for resource discussions with campus stakeholders in a readable and contextualized wayThere is still work to be done to ensure the long-term usefulness of this tool. Additionally, more work is required to better define a scoring and coding criteria to ensure fairness and equity across the resources being evaluated. One of our biggest obstacles has been, and likely will continue to be, messaging and librarian buy-in as some librarians have questioned the need for such a tool and have expressed concern that this tool will remove aspects of librarian agency. We have designed this tool not to take away from core areas of librarian knowledge, but to better express these factors for a non-library audience and to help justify difficult decisions to campus stakeholders by illustrating a fuller picture of the resource by combining quantitative usage data with qualitative feedback and to do so in a fair and equal way across subject areas. This project works to contextualize the sometimes misleading and never complete picture that the vendor-supplied usage data provides by investigating the relative weight of these numbers by providing more specifics about the users (and the use of the resource balanced by the size of a department), the importance to the users, the scope within the library’s collection policies, aspects of usability, vendor relationships, competing products, and more. Equipped with more data, librarians can more effectively and fairly defend resources from cancellation or support deaccession decisions with further depth and demonstrable evidence of the resources’ impact and their own expertise.This is a sample spreadsheet with databases that demonstrate the potential assessment process https://docs.google.com/spreadsheets/d/1ZnXCext0xVLRUO0eAOkuIGzZoeItnZTnakpYGGIS1sM/edit#gid=0Basic descriptive aspects of the resource●  Fully owned content (i.e., UM Libraries own rights in perpetuity) ● Content able to be acquired through ILL  ● Open access - content freely available  ● Exclusivity - content only available through this database  ● Primary source  ●  Unique secondary material  ● Provides media, maps, and/or other non-text content  ● Provides access to data or data sets  ● Provides full-text content ● Abstracting and indexing only ● Facilitates full-text linking (i.e., Find@UMD button available within the database)  ● Metrics or other citation usage tools (i.e., impact factors, cited by, etc.)  ● Specialized search features (i.e., ChemDraw structure searching) ● Internal applications or other specialized tools  Costs● Does this resource have continuing costs associated with it?● Is this resource acquired through a consortial agreement?● Is this resource acquired through a partnership with/funds from an academic department?● Is this resource acquired through a gift fund?Technical issues● Does this resource have frequent technical difficulties or problems in accessing its content?Relative usage by● What is the ratio of usage to the number of faculty in the department or School? (usage #s supplied by Collection Services and faculty #s at https://reports.umd.edu)● What is the ratio of usage to the number of students in the department or School?  (usage #s supplied by Collection Services and student #s at https://reports.umd.edu)Peer comparisons● Is this resource available at USMAI (local consortia) partner institutions?● Is this resource available at BTAA (national consortia) partner institutions?● Is this resource available at other libraries in the Mid-Atlantic region?● Is this resource available at libraries of subject disciplinary peer institutions?Accreditation needs and Curricular fit● Is the resource needed for Accreditation?○ If yes, please explain.  ● Is the resource needed for Instruction?● Is the content and ease of access appropriate for an undergraduate introduction to the subject?● Is the content and ease of access appropriate for undergraduate expertise in the subject?● Is the content and ease of access appropriate for graduate-level research in the subject?● Does the resource support area/regional studies pertaining to particular geographical, national/federal, or cultural regions?● Does the resource support diversity and inclusion efforts? (i.e., stated mission of diversity, diverse content, diverse authors, etc.)Licensing issues● Are simultaneous users allowed?● Are there any licensing restrictions?● Was this resource difficult to license?● Is the vendor easy to work with?● Has the vendor changed policies or done anything to make using or licensing this resource difficult?Aspects of Usability● Is this resource easy for users to learn how to use?● Why or why not is this database easy to learn for users?

The Time Has Come...To Build, Reflect, and Analyze Connections Between Qualitative and Quantitative Data

http://drum.lib.umd.edu/bitstream/1903/25269/1/Sly%2c%20De%20Pope%2c%20Frank%2c%20Ritchie%2c%20Qualitative%20and%20Quantitative%20Resource%20Assessment.pdf

The Time Has Come...To Build, Reflect, and Analyze Connections Between Qualitative and Quantitative Data

Abstract

Similar works

Full text

Available Versions

Digital Repository at the University of Maryland