Frizzell, Matthew

Porter, Seth

English

DigitalCommons@Kennesaw State University

Georgia Library QuarterlyVolume 55Issue 2 Spring 2018 Article 104-1-2018Assessing Instructional Initiatives and Servicesthrough Program EvaluationSeth PorterPrinceton University, sthporter@gmail.comMatthew FrizzellGeorgia Institute of Technology, matthew.frizzell@library.gatech.eduFollow this and additional works at: https://digitalcommons.kennesaw.edu/glqPart of the Library and Information Science CommonsThis Article is brought to you for free and open access by DigitalCommons@Kennesaw State University. It has been accepted for inclusion in GeorgiaLibrary Quarterly by an authorized editor of DigitalCommons@Kennesaw State University. For more information, please contactdigitalcommons@kennesaw.edu.Recommended CitationPorter, Seth and Frizzell, Matthew (2018) "Assessing Instructional Initiatives and Services through Program Evaluation," GeorgiaLibrary Quarterly: Vol. 55 : Iss. 2 , Article 10.Available at: https://digitalcommons.kennesaw.edu/glq/vol55/iss2/10Assessing Instructional Initiatives and Services through Program Evaluation  By Seth M. Porter and Matthew Frizzell  Introduction  This paper is inspired by an Atlanta Area Bibliographic Instruction Group (AABIG) presentation and discussion that aimed to introduce an overview of program evaluation as a method for assessing organizational effectiveness. At AABIG, the authors discussed program evaluation, including a contextual overview, when to use it, how to use it, the tools required, and a case study reflecting these methods and tools. The following discussion will reflect this format.   The paper is broken into the following sections: (a) a literature review that will give the reader a brief introduction to program evaluation process; (b) the best practices for implementing the empirical analysis of library programs; (c) a case study based on program evaluation efforts at Georgia Institute of Technology (Georgia Tech); and (d) the conclusion will tie the overview, methods and tools, and best practices into a coherent overview on the importance of adopting a program evaluation mindset in the academic library.  Literature Review  Porter (in press) described program evaluation as:   A systematic application of scientific methods to design, implement, improve, or measure the outcomes of a programs (Rossi & Freeman, 1993). Most importantly the systematic nature of program evaluation creates a framework for collection analyses of data that is used to measure the effectiveness and outcomes of a specific program, treatment, or service (Center for Disease Control [CDC], 2017). The term “program” can mean many different actions, treatments, or services. These include media campaigns, education services, instructional programs, public policies, and research projects (CDC, 1999; Bingham & Felbringer, 2002). Essentially, program evaluation is a systematic, scientific approach to assess the effectiveness, efficiency, and outcomes of specific programs (Bingham & Felbringer, 2002).   When we talk about program evaluation based on this definition, we don’t mean a rigid binary process. It is more of an agile and empirical approach to academic library assessment. While we believe it is a much needed approach to assessment in academic libraries, it is not the only approach. That said, we believe it can add empirical rigor to a formal library assessment program. Throughout the next section we will give a very brief introduction on the best practices in program evaluation. These best practices will illustrate the holistic but empirical nature of program evaluation.   The Gold Standard. When approaching program evaluation, the gold standard is experimental design. Experimental methods are completely randomized and the participants are chosen by chance. In program evaluation and assessment in academic libraries this is the most valuable type of assessment. However, it can be difficult to implement because of the need for a random implementation, and a control and treatment group (Bingham & Felbringer, 2002).  Testing. The first place to start is the pre-test post-test. When implementing a pre-test post-test, individuals are chosen randomly for a treatment or control group. There are many 1Porter and Frizzell: Assessing Instructional Initiatives and Services through Program EvaluationPublished by DigitalCommons@Kennesaw State University, 2018ways to do this, but in an academic library the numbers are usually small so you can pull a name out of a hat or flip a coin (Bingham & Felbringer, 2002). Next, have both groups take a pre-test to measure current levels of content knowledge.  After the initial tests, the treatment group will take part of the program. In an academic library, for example, that could be a pilot program or an information literacy session or for-credit course. The control group will not take part in the pilot program or information literacy sessions. When the program is complete both groups are tested on content knowledge and outcomes are analyzed (Bingham & Felbringer, 2002; Porter, in press).  The Next Best Thing. The gold standard is not always possible, or when it is possible, it is too expensive to implement. Nevertheless, you can still implement empirical assessment programs through quasi-experimental design. Basically, quasi-experimental design attempts to create a random controlled trial through other methods (Rossi & Freeman, 1993; Bingham & Felbringer, 2002).  This is not as empirical as a true experimental design, because the evaluation must identify, classify, and measure all variables of the experimental and control group to attempt to create a true experiment (Rossi & Freeman, 1993). That said, you can still use many of the program evaluation tools that are used in the gold standard of experimental design; for example, the pre-test post-test design could be crafted under a quasi-experimental design (McNamara, 2017; Environmental Protection Agency [EPA], 2017). An example of this in an academic library would be an artificial control and treatment group; for example, different sections of the same class. The experimenter could implement a pre-test post-test methodology to test the effect of a program implementation. While this isn’t as pure as an experimental design, it is a plausible replacement.  Cost Effectiveness Analysis. A cost effectiveness analysis (CEA) is a natural fit within academic library assessment. It is a tool used to understand the resource allocation through the projected output and testing of a specific program (Bingham & Felbringer, 2002). Essentially a cost effectiveness analysis analyzes the potential implementation of a program based on the comparison of potential needed resources and what the expected outcomes of the program will be.   The value of a cost effectiveness analysis in an academic library is that you can use tools to measure holistic impacts and benefits that do not have an economic or financial cost (Metz, 2007). That is what a CEA does and why it is a great tool to use in academic libraries (Bingham & Felbringer, 2002).  Case Study  At the Georgia Tech Price Gilbert Library, we are currently undergoing not just a major renovation of our physical spaces, but also reevaluating our services and how we have traditionally done business as part of a process we call Library Next.  The next section looks at an example of using program evaluation methodologies to determine whether we are delivering services in an effective manner. This example will likely look familiar to the reader insofar as these are the types of calculations we do daily as librarians to effectively use shrinking resources. Part of what we would like to focus on is using a rigorous methodology and some of the formalized mechanics of program evaluation in these instances.   Example A: At Georgia Tech we had for years provided reference help that was combined with circulation at one service point. As librarians retired or took positions elsewhere, and jobs were not backfilled, it became necessary to prioritize how librarians spent 2Georgia Library Quarterly, Vol. 55, Iss. 2 [2018], Art. 10https://digitalcommons.kennesaw.edu/glq/vol55/iss2/10their time, and the decision was made to split out these services.  Initially we moved to a tiered service with librarians on call and seated nearby which then eventually morphed into its own service area called the Expert Consultation Center (ECC). While undergoing extensive renovations at the library, space, and how it is being utilized, was especially important because the usable area had been significantly reduced.  So in the case of the expert consultation center there are two constraints that are our cost factors: librarian’s time and the physical space. In order to evaluate whether this is the most effective use of those resources, we built a tally to record service interactions. When designing the tally, though, we wanted to include other metrics as well in order to gain better insight into variables such as the complexity of the questions, time spent, nature of the questions, and time when the interaction occurred.  Recording these specific data points is key to helping us better understand our cost effectiveness analysis and answer other related questions. Is our service cost effective between certain hours but not others? Are there certain types of questions which aren’t cost effective for librarians to be answering when signage or training staff can better meet this need?  After collecting and analyzing Expert Consultation Center data from September 2016 to January 2017 we were able to evaluate the program to determine if continuing its operations makes sense for Georgia Tech. For that period of time we saw a total of 173 questions asked over a span of 264 hours that the area was staffed which works out to 0.655 questions per hour. The peak times were between 12–2 p.m., with 68% of the questions coming during this time period. Qualtrics was used to create a custom made report for Georgia Tech but any method available to your organization that can tally data should suffice. We also rated the questions that were asked at the ECC on how complex they were: 0 was a directional question, 1 finding materials in the catalog or research guide, all the way to 5, which was an in depth consultation, pulling from multiple databases, or reviewing patron created content. Here we found 53% of the questions were simple category 0 questions, 11% were category 1, 11% category 2, 14% category 3, 6% category 4, and 5% category 5.  Other metrics measured were whether the patron was a repeat user of the ECC and if they were referred to their subject librarian afterwards with “yes” garnering only 9% and 7% respectively. We also included a qualitative open text field for librarian comments in case any interesting trends stood out or if the data was inconclusive at the end of the study.  When removing the simple questions from the total consultations, the Expert Consultation Center only received 0.30 questions per hour with data trends not indicating significant increases as time went on. Clearly, this indicates an ineffective use of resources and space, so the service was discontinued. In many instances where the data was less conclusive, the evaluation could have extended for a longer period of time to help accurately make an assessment, but in this case the ongoing cost didn’t justify ongoing service.   Conclusion  Detailed throughout this discussion the strengths of program evaluation as an addition to academic libraries assessment programs is illustrated through best practices and tools, such as experimental design, quasi-experimental design, and cost-effectiveness analysis. This systematic nature of program evaluation can be used to measure the effectiveness and outcomes of library services. In taking this empirical and scientific approach to library assessment, libraries can internally and externally improve their programs, services, and messaging. 3Porter and Frizzell: Assessing Instructional Initiatives and Services through Program EvaluationPublished by DigitalCommons@Kennesaw State University, 2018While this is an empirical and positivist approach, we do not dismiss other valid assessment approaches like ethnographic research, document analysis, content analysis, qualitative interviews, survey based research, case study approaches, and programs that combine these into a fluent portfolio such as service design. However, it is important to understand there are other potential methods that will add real value that can be empirically tested and communicated to stakeholders within the library and without, and program evaluation is one of these methods that is an addition to a library assessment portfolio. As demonstrated in the case study, not only do these methods help make judicious allocation of resources but they also allow an organization to explain policy decisions to internal and external stake holders in objective terms. In this case the decision to move away from a reference point of service approach was unpopular internally and the data from a program evaluation methodology made for a conclusive decision rather than relying on anecdotal evidence.  Seth Porter is Head of Stokes Library and Research Center for Woodrow Wilson School of Public Policy at Princeton University  Matthew Frizzell is Assessment Coordinator Librarian at Georgia Institute of Technology   4Georgia Library Quarterly, Vol. 55, Iss. 2 [2018], Art. 10https://digitalcommons.kennesaw.edu/glq/vol55/iss2/10References  Bingham, R., & Felbringer, C. (2002). Evaluation in practice: A methodological approach. New York, NY: Chatham House Publishers.  Center for Disease Control. (2017). Program Performance and Evaluation Office. Retrieved from https://www.cdc.gov/eval/framework/  Center for Disease Control. (1999). Framework for program evaluation in public health. Retrieved from https://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm  Environmental Protection Agency. (2017). Program evaluation and performance measurement at the EPA. Retrieved from https://www.epa.gov/evaluate/program-evaluation-and-performance-measurement-epa  McNamara, C. (2017). Basic guide to program evaluation. Retrieved from https://www.unm.edu/~egrong/web/docs/R3_Basic%20Guide%20to%20Program%20Evaluation.pdf  Metz, A. (2007). Why conduct a program evaluation? Five reasons why evaluation can help an out-of-school time program. Research-to-Results, #2007-31. Retrieved from https://cyfar.org/sites/default/files/Child_Trends-2007_10_01_RB_WhyProgEval.pdf  Porter, S. (in press). Using program evaluation as a proxy for assessment: Diffusion from policy literature to improve academic program assessment. In M. Britto & K. M. Kinsley (Eds.), Academic Libraries and the Academy: Strategies and Approaches to Demonstrate Your Value. Chicago, IL: ACRL.   Rossi, P., & Freeman, H. (1993). Evaluation: A systematic approach. (5th ed.). Newbury Park, CA: Sage Publications, Inc. 5Porter and Frizzell: Assessing Instructional Initiatives and Services through Program EvaluationPublished by DigitalCommons@Kennesaw State University, 2018

Assessing Instructional Initiatives and Services through Program Evaluation

https://digitalcommons.kennesaw.edu/cgi/viewcontent.cgi?article=2118&amp;context=glq

Assessing Instructional Initiatives and Services through Program Evaluation

Abstract

Similar works

Full text

Available Versions

DigitalCommons@Kennesaw State University