Program evaluations can serve a number of functions. From a marketing standpoint, one important function is the generation of information upon which a program or  product  can be modified. A case study approach is used to show two methods for gathering information for the purposes of program modification. Both are relatively easy to use and involve minimum costs

Gitelson, Richard J.

English

Bowling Green State University: ScholarWorks@BGSU

Visions in Leisure and Business Volume 3 Number 2 Article 6 1984 Program Modification Through Program Evaluation Richard J. Gitelson University of North Carolina at Chapel Hill Follow this and additional works at: https://scholarworks.bgsu.edu/visions Recommended Citation Gitelson, Richard J. (1984) "Program Modification Through Program Evaluation," Visions in Leisure and Business: Vol. 3 : No. 2 , Article 6. Available at: https://scholarworks.bgsu.edu/visions/vol3/iss2/6 This Article is brought to you for free and open access by the Journals at ScholarWorks@BGSU. It has been accepted for inclusion in Visions in Leisure and Business by an authorized editor of ScholarWorks@BGSU. PROGRAM MODIFICATION THROUGH PROGRAM EVALUATION BY DR. RICHARD J. GITELSON, ASSISTANT PROFESSOR CURRICULUM IN RECREATION ADMINISTRATION UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL CHAPEL HILL, NORTH CAROLINA 27514 ABSTRACT Program evaluations can serve a number of functions. From a marketing standpoint, one important function is the generation of information upon which a program or "product" can be modified. A case study approach is used to show two methods for gathering information for the purposes of program modification. Both are relatively easy to use and involve minimum costs. PROGRAM MODIFICATION THROUGH PROGRAM EVALUTION INTRODUCTION The previous article described the process by which an idea can be translated into a "product" to be offered by a recreation department. By emphasizing a "marketing" approach, i.e. one in which the potential clientele plays a major role in the design phase, a recreation department theoretically maximizes its chances of providing a successful program. However, to paraphrase a famous quote, "the best laid plans of the most conscientious recreation programmer often go astray." One of the many purposes of program evaluation is to determine what, if anything, has gone wrong. More importantly, a program evaluation can help us determine what to do to improve our recreation "product". The two methods presented in this paper have been chosen for a number of reasons. Both methods are easy to implement, both are inexpensive, and most importantly, both provide information that can be interpreted without a degree in statistics. 14 IMPORTANCE-PERFORMANCE ANALYSIS A product, whether a recreation program or a bar of soap, is made up of a number of attributes, such as its price, where it is distributed, how it is promoted, and the actual benefits that are derived from its use. Martilla and James have developed an evaluation technique that is based on the assumption that " ••• consumer satisfaction is a function of both expectations of certain important attributes and judgments of attribute performance." (4, p�771 What makes the technique so attractive is that the results can be displayed easily in the form of a two-dimensional grid, which makes it easy to interpret the results. This evaluation process, which is referred to as importance-performance analysis, was recently used to evaluate a roadrace (125 participants) sponsored by a local recreation department. Their experience illustrates how the method provides information that can be used to modify a particular program. The first step in the process involved identifying the key features of the marketing mix of the particular program being evaluted. This can be done by using some of the techniques described in the last article. For example, focused or unfocused interviews with potential participants or "experts" can uncover important attributes. Also, previous research may provide a starting list of features to be included in the analysis. In the present case, the department began with a list of attributes that had been developed by Guadagnolo et al. (3) (for some examples of theattributes, see Table 1). After reviewing the list with local runners and department personnel, only a few minor changes were made to the original list. The participants were then asked after the race to rate these attributes in two separate stages. First, they were asked to rate the general importance of each measure. Then, they were asked to rate how the program actually performed in terms of each attribute. The mean scores and the rating scales that were used are shown in Table 1. Other rating scales may be used. For example, Martilla and James use four point scales in their example. A two dimensional grid was then used to graphically display the results. One axis is formed by the importance rating, and the other axis is formed by the performance rating. Some of the actual results from the race have been plotted in Table 2. The graph forms four quadrants. Martilla and James used the following terminology to describe the quadrants. Examples from Table 2 will be used to illustrate how the graph can be interpreted. 1. Concentrate Here. In the present example, the participants rated restrooms as very important (6.36 on a scale of 1 to 7, with 7 being very important), while they rated the actual performance as being less than satisfactory. This indicates an area which the programmers should continue to concentrate their efforts. 15 2. Keep!!£ The Good work. Participants rated starting proceduresas being very Tinpor"tant--rc; them and also indictated that the race promoters had done a good job in this area. This indicates an area which the programmers should continue to concentrate their efforts. 3. Possible Overkill. Participants do not feel that pre-race events are very important. However, the race rated very high on their attribute. This is one area where the race organizers can possible spend less effort in the future race and still have a successful event. Of course, as with any evaluation process, the final decision as to what modifications are made, if any, should be based on all pertinent information. If one pre-race event is a show case for the county commissioners, the event might be held regardless of the importance of that event to the participants. 4. Low Priority. The participants in this race were not particular�satisfied with the change-shower facilities. However, this was also not rated as being very important to the participants. Thus, if time and resources permit, this is an area that might be worked on, although it can be considered low priority. This evaluation process involved about five hours of work to prepare the questionnaire. Copying the questionnaire and postage cost approximately $10. Costs were kept to a minimum in the present example by including the questionnaire in the same envelope that was used to mail out the race results. Also, the decision was made to not include return postage on the self-addressed return envelope that was included with the questionnaire. The results {47 participants returned the questionnaire) were tabulated on a computer at the local university, although the results could have been calculated on a microcomputer or hand calculator. Another advantage to this technique, is that the evaluator can look at different groups of participants. For example, the race organizer wanted to see if there were differences between males and females as to what was important and how the actual race was evaluated. In the present example, there were no significant differences. However, the technique allows these comparisons to be made. For a more detailed look at this approach, it is recommended that you write Frank Guadagnolo (3) at The Pennsylvania State University for a copy of their analysis of the Great Race. USING YOUR RECREATION COMMISSION FOR PROGRAM EVALUATION According to Connolly, "A good evaluation is one that paints a picture of how well a program functions at one point in time and how the program may be improved for the future". (2, p.34) This first technique certainly "paints" such a picture. The following example, developed by members of a local recreation commission, also provides the programer with information that can be used in determining if program modifications are needed. 16 This technique, although simple to use, is based on the assumption that information should be gathered in a number of different ways. As such, it conforms to the notion that program evaluation should rely on multiple sources of information. (1) In the fall of 1982, the recreation commission formed a committee to help the recreation department evaluate its programs. The committee was to be cdmposed of three commission members and two members of the department staff. A number of programs would be chosen by the committee to be analyzed each year. A questionnaire was developed (see Table 3) that would be distributed to participants (and parents in the case of youth programs). In addition to the questionnaire, each commission member was also assigned a particular program, and was expected to actually interview participants and program instructors. Additional information was gathered using department records and attendance records. A number of suggestions have been generated by using this technique as to how programs can be modified. For example, it was discovered that the publicity had not been adequate for the Pee Wee basketball program, and that the instructor to child ratio was too high. Modifications were made and the overall satisfaction with the program is now excellent. Not only do the commission members now feel that they are making a valid contribution to the department's programs, but the staff gets valuable information that can be used to modify programs. CONCLUSION Program evaluation can perform many functions. From a marketing standpoint, one important function is to provide information that can be used for program modification. Both of the above methods are relatively easy to use and involve low monetary costs for the department. Although program evaluation is usually portrayed as a rather complex undertaking, these methods show that the main prerequisite for the program evaluation is a desire to learn how well one is doing, with the goal being to insure that our program efforts are the best that they can be. REFERENCES 1. c. Bullock, "Interactionist Evaluators, Look for "What Is" Not What Should Be", Parks and Recreation, Vol. 17(2), pp. 37-39, 1982. 2. P. Connelly, Evaluation's Accountability, Parks� Recreation, Vol. 17 Critical 17(2), pp. Role in Agency 34-36, 1982.3. F. B. Guadagnolo, R. B. Warnick, and D. L. Kerstettler,Importance-Performance Marketing Analysis of a Ten-Kilometer �, Department of Recreation and Parks, PennsylvanTa State University. For a copy, write to Frank Guadagnolo, Department of Recreation and Parks, 267Recreation Building, Pennsylvania State University, University Park, Pnnsylvania, 16082. 4. J. A. Martilla and J. Analysis, Journal£! Marketing, Vol. c. James,41(1), pp.18 Importance-Performance 77-79, 1977.1. 2. 3. 4. s. 6. TABLE ONE SCORES FOR SELECTED ATTRIBUTES OF THE OCTOBERFEST ROAD RACE A BFeature Importance Performance Change/shower facilities 3.73 3.50 Restrooms 6.36 3.97 Pre-race events 3.64 5.13 Starting procedures 6.20 5.92 Water stations 5.08 6.10 Number of joggers 3.60 5.00 Quadrant Low Priority Concentrate Possible Overkill Good Work Good Work Low Priority A The means could range from 1 to 7. Martilla and James recommend that medians be used, although they indicate it makes little difference as to which statistic is used. B The response categories for the importance questionnaire ranged from very important (7) to not important (1). The response categories from the performance questionnaire were (1) Terrible, (2) Unhappy, (3) Mostly dissatisfied, (4) Mixed, (5) Mostlysatisfied, (6) Pleased, (7) Delighted19 I M 0 R T A N c E 7 TABLE TWO EXAMPLES OF GRAPH SHOWING IMPORTANCE AND PERFORMANCE SCORES B D Keep Up The 6 Concentrate E Good work Here 5 4 I Al F c 3 ILow Priority IPossible 2 overkill l 1 2 3 4 5 6 l Performance A. Change shower facilities {see Table One for Scores)B. Restroomsc. Pre-Race eventsD. Starting proceduresE. water stationsF. Number of joggers20 7 TABLE THREE QUESTIONNAIRE CATEGORIES USED FOR INTERVIEWS BY RECREATION COMMISSION -------------------------------------------------------------------------EVALUATION AREAS -------------------------------------------------------------------------1. Training of Personnel2. Guidance on Policies and Procedures3. Condition of equipment and supplies4. Instructor to child (participant) ration5. Supervisor's Accessibility6. Cleanliness of the facility7. Safety conditions8. Appropriateness of facility9. Publicity10. Program Benefits (including learning experiences, enjoyment,etc.)11. Instructor's attitude and knowledge12. Scheduling of program13. Quality of supervision-------------------------------------------------------------------------21 

Program Modification Through Program Evaluation

https://scholarworks.bgsu.edu/cgi/viewcontent.cgi?article=1127&amp;context=visions

Program Modification Through Program Evaluation

Abstract

Similar works

Full text

Available Versions

Bowling Green State University: ScholarWorks@BGSU