Every five years, unit offices of Virginia Cooperative Extension (VCE) must submit a situation analysis (SA) report. The purpose of the SA report is to identify the current issues at the local level to develop future educational programs. The primary objective of this study was to develop a methodology that could be used to assess the quality of SA reports in Virginia. We have evaluated 196 SA reports from VCE to begin development of an assessment. Results of testing of the tool identify areas for continued development

Kuri, Subrato K

Lambur, Michael

Vines, Karen A

English

Clemson University: TigerPrints

Journal of Extension Volume 58 Number 3 Article 23 Identifying Key Constructs and Measurements to Assess the Situation Analysis Reports Subrato K. Kuri Virginia Tech, subrato@vt.edu Karen A. Vines Virginia Tech, kvines@vt.edu Michael Lambur Virginia Cooperative Extension, lamburmt@vt.edu This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License. Recommended Citation Kuri, S. K., Vines, K. A., & Lambur, M. (2021). Identifying Key Constructs and Measurements to Assess the Situation Analysis Reports. Journal of Extension, 58(3). Retrieved from https://tigerprints.clemson.edu/joe/vol58/iss3/23 This Tools of the Trade is brought to you for free and open access by TigerPrints. It has been accepted for inclusion in Journal of Extension by an authorized editor of TigerPrints. For more information, please contact kokeefe@clemson.edu. Identifying Key Constructs and Measurements to Assess the Situation Analysis Reports Cover Page Footnote We are thankful to the Virginia Cooperative for providing the opportunity to use the SA reports as a basis for this project. We also extend our heartfelt thanks to the independent reviewers who assisted with the assessment of the tool. Moreover, we are extremely grateful to Md. Jamal Hossain, Ph.D. Researcher at the Dept. of Social Statistics and Demography, University of Southampton United Kingdom and Statistical Application and Collaboration Group (SAIG) of the Department of Statistics at Virginia Tech for their support for statistical consultation. This tools of the trade is available in Journal of Extension: https://tigerprints.clemson.edu/joe/vol58/iss3/23   Identifying Key Constructs and Measurements to Assess the Situation Analysis Reports   Introduction and Purpose Situation analysis (SA) reports, based on local needs assessment, identify critical current issues to develop relevant educational programs that meet local needs (Bayer et al., 2020; McCawley, 2009; and Teuteberg & Cummins, 2017). McCawley (2009) defines needs assessment as a “systematic approach to studying the state of knowledge, ability, interest, or attitude of a defined audience or group involving a particular subject” (p. 3). Local Extension offices in Virginia are required to develop a SA report every five years. Agents are instructed to follow this process for developing their SA report: 1) Plan the local process, 2) develop a unit profile, 3) identify needs from a community and resident perspective, and 4) interpret data and decide on program direction (Lambur, n.d.). Forest and Baker (1994) recommend that SA reports should include the following items. • A description of the current condition • Identification of needs, problems, opportunities, and/or emerging issues and supporting data • Indicators of severity or scope of need • Benchmark data against which later impact measurements can be compared • Primary audience(s), numbers, and geographic locations • Identification of gaps between “what is” and “what could be” and needed research  Our interest was in finding a way to evaluate the quality of the Virginia Cooperative Extension (VCE) SA. After conducting a review of the literature, we did not find an established method for assessing SA reports. Mike Lambur, who oversees the SA process in VCE, reviews the SA reports and provides feedback to the units. However, this review occurs after the reports are submitted, is subjective and summative, improving the quality of future reports but not the current report. Thus, the primary objective of this project was to develop a methodology that could be used by Extension practitioners at the local level to objectively assess the quality of their SA reports. This will help contribute to proper program planning, implementation, and reduce the waste of resources for inappropriate programming. In addition, a well-done SA report is a product provided by VCE that can be used by local and state governments and to build collaboration with other organizations.  Methods and Data Sources We initially studied 102 SA of VCE submitted in 2013 to understand the style, contents, similarities, and dissimilarities of the format of SA reports. Then we developed a draft rubric to assess the SA reports based on the previously identified styles, contents, and formats of 2013 SA reports. We identified seven categories to evaluate within the SA reports (see Table 1). We weighted each section of the SA reports differently based on their contribution to the report. The VCE administrators internally reviewed the draft rubric. We revised the tool based on their feedback.        Table 1 Weighting of Categories in VCE SA Reports  Category Weight (%) Overall Writing Style 15 Preliminary Materials 05 Introduction 10 Unit Profile 20 Community Perspective or Methodology 20 Major Issues 20 References and Appendices 10  A total of 25 criteria were defined for all seven categories (Appendix A). We developed a five-point rating scale and associated quality indicators for each criterion. SA reports were reviewed and criteria were assigned a score. The revised tool was used to assess the 2018 SA reports submitted by 94 unit offices of VCE. These reports were then shared with the district directors and unit coordinators to help them understand the strengths and weaknesses of their SA reporting process.    To validate the tool for use across a wide range of users, we selected five SA reports and asked nine individuals to assess these reports. Reviewers included an undergraduate student and one graduate student, one IT director of college, one associate director, two district directors, one county coordinator, and two professional lecturers of VCE. For purposes of this assessment, we exported the tool into Virginia Tech QualtricsXM (https://virginiatech.ca1.qualtrics.com/) and shared individual links for each report with the selected reviewers. After obtaining all evaluation reports, we exported the data for analysis using Statistical Package for Social Science (SPSS 16.0) to conduct the Kruskal-Wallis Test.   Findings  The developed tool is presented in Appendix A. None of the reviewers indicated any challenges using the tool. Table 2 shows the results of the usability of the tools by the independent reviewers. Table 3 shows that the scores for ‘community perspectives’ and ‘major issues’ sections are significantly different from reviewer to reviewer. Moreover, the total scores of 3 reports (i.e., except County B and County E) are significantly different Figure 1  Methodological Approach used for Tool Development   across the reviewers. These also differ from the scores assigned by the developer. Thus, we may conclude that our tool can be used to evaluate the SA reports. However, there is variability in some areas and some reports from one reviewer to the next. Community perspective and major issues are the two sections, which show most disagreement between reviewers. These areas need further review and edit.  Table 2  Comparison Between Reviewers’ Scores for SA Reports of Test Counties Sections of the report County A County B County C County D County E χ2 p χ2 p χ2 p χ2 p χ2 p Overall Writing Style 13.75 .088 11.63 .168 10.90 .208 28.09 .000 11.71 .165 Preliminary materials 1.09 .998 1.01 .998 .73 .999 2.40 .966 .88 .999 Introduction .51 .776 6.59 .581 12.00 .151 8.02 .432 11.31 .185 Unit profile 11.51 .174 3.04 .932 14.66 .067 20.98 .007 13.20 .105 Community Perspective 35.03 .000 20.55 .008 39.02 .000 40.30 .000 15.19 .055 Major issues 21.95 .005 14.67 .066 22.34 .004 18.19 .020 16.38 .037 References and Appendices 11.29 .186 7.28 .507 11.15 .193 4.73 .786 13.65 0.91 Total Score 28.52 0.000 10.63 .224 33.92 .000 55.74 .000 14.96 .060 Note. N = 9; degrees of freedom (df) = 08; χ2 = Chi-square Value of Kruskal Wallis Test; p = Asymptotic significance value;  p ≤ .05 (significance at 95% level of confidence) p ≤ .01 (significance at 99% level of confidence)    Table 3  Level of Agreements of the Selected Parameters (i.e., Questions)  Questions (Scores) Level of Agreement County A County B County C County D County E Q 14. Description of the sample(s) or sources of input for data collection Poor (0 and 1) 44.4 88.9 11.1 55.6 44.4 Fair (2) 22.2 11.1 66.7 22.2 33.3 Good (3) 33.3 0 22.2 22.2 22.2 Excellent (4) 0 0 0 0 0 Q 15. Process for selecting participants Poor (0 and 1) 44.4 100.0 33.3 55.6 44.4 Fair (2) 22.2 0 44.4 11.1 22.2 Good (3) 33.3 0 22.2 33.4 22.2 Excellent (4) 0 0 0 0 11.1   Questions (Scores) Level of Agreement County A County B County C County D County E Q 16. Methods indicate that data collected from both traditional and non-traditional stakeholders of VCE, including underserved audiences Poor (0 and 1) 44.4 88.9 44.4 55.6 55.6 Fair (2) 33.3 11.1 33.3 11.1 11.1 Good (3) 22.2 0 11.1 22.2 22.2 Excellent (4) 0 0 11.1 11.1 11.1 Q 17. Indicates data collection methods with the sample size for each data collecting method. Response rate should be included for surveys Poor (0 and 1) 66.7 100.0 55.6 44.4 55.6 Fair (2) 22.2 0 22.2 0 22.2 Good (3) 11.1 0 11.1 33.3 11.1 Excellent (4) 0 0 11.1 11.1 11.1 Q 18. Timeline of data collection provided Poor (0 and 1) 77.8 100.0 33.3 55.6 66.7 Fair (2) 22.2 0 44.4 22.2 22.2 Good (3) 0 0 22.2 22.2 11.1 Excellent (4) 0 0 0 0 0 Q 19. Various data analysis techniques are used Poor (0 and 1) 33.3 100.0 55.6 44.4 66.7 Fair (2) 44.4 0 11.1 22.2 33.3 Good (3) 11.1 0 33.3 11.1 0 Excellent (4) 11.1 0 0 22.2 0 Q 20. Identified issues are prioritized Poor (0 and 1) 66.7 44.4 33.3 11.1 11.1 Fair (2) 11.1 0 22.2 11.1 44.4 Good (3) 22.2 55.6 22.2 22.2 22.2 Excellent (4) 0 0 22.2 55.6 22.2 Q 21. Methods used for identifying priorities are clearly defined Poor (0 and 1) 44.4 100.0 44.4 22.2 55.6 Fair (2) 33.3 0 22.2 22.2 22.2 Good (3) 11.1 0 11.1 33.3 22.2 Excellent (4) 11.1 0 22.2 22.2 0 Q 22. Summary of each prioritized issues includes sources of supporting data and stakeholder’s connection to that issue Poor (0 and 1) 55.6 66.7 33.3 11.1 55.6 Fair (2) 11.1 22.2 0 33.3 22.2 Good (3) 33.3 11.1 44.4 33.3 22.2 Excellent (4) 0 0 22.2 22.2 0  Recommendations We are pleased that we have developed a draft tool which can be refined to evaluate VCE SA reports at the local level. However, some refinement is necessary to increase the consistency of this tool across reviewers. There is needed to better define the criteria across the 25 criteria in general. In addition, questions in the community perspectives and major issues sections need to be simplified to increase understanding. We recommend the tool be   tested again after revision for accuracy as well as usability by providing a situation where reviewers provide verbal feedback in addition to the assigned scores.  Application and Importance for Cooperative Extension System We believe this tool is promising and can be improved to provide formative evaluation of the quality of VCE SA reports. In the future, long, unclear, and complex questions should be simplified, defined, and made more concise for better usability of this tool. After revision, unit coordinators can use the tool as a checklist in the development of the SA to provide more formative evaluation, improving the quailty of the SA report. The tool may be adapted by other Extension organizations as well. The revised tool can also be used for identifying the training needs of the staff. Based on the assessment scores, VCE will be able to target training to specific units.   References Bayer, R., Mitchell-Hawkins, V., & Travis, A. (2020). Needs Assessment Tool Kit. Journal of Extension, 58 (1), Article v58-1tt1. https://joe.org/joe/2020february/tt1.php?utm_source=email&utm_medium=email&utm_campaign=2020february Forest, L., & Baker, H. (1994). The program planning process. In D. J. Blackburn, (Ed.). Extension handbook: Processes and practices (pp. 86-99). Toronto, Canada: Thompson Educational Publishing. Glen, S. (2016, February 24). Kruskal Wallis H Test: Definition, Examples & Assumptions. StatisticsHowTo.com: Elementary Statistics for the rest of us!. https://www.statisticshowto.com/kruskal-wallis/ Lambur, M. (n.d.). VCE Situation Analysis: Module Overview. Virginia Cooperative Extension Intranet. https://www.intra.ext.vt.edu/support/process/situation.html McCawley, P. (2009). Methods for conducting an educational needs assessment: Guidelines for Cooperative Extension System professionals (University of Idaho Extension BUL 870). Retrieved from https://www.extension.uidaho.edu/publishing/pdf/BUL/BUL0870.pdf Teuteberg, D., & Cummins, M. (2017). Using community assessments to improve 4-H youth development programming efforts. Journal of Extension, 55(4), Article 4TOT9. https://joe.org/joe/2017august/tt9.php     Appendix A Situation Analysis Assessment Tool    Assessment Plan for Situation Analysis Report of Virginia Cooperative Extension Report Provided by:  Name of the Locality/Unit:                                            District:  Year:   Directions  1. Read through entire situation analysis report being reviewed.  2. Complete the scoring by section. 3. Score every item. 4. The scoring procedure for each category of the rubric should be interpreted as follows:  Absent (0) Items are missing from the situation analysis report Poor (1) Inappropriate items with incomplete materials and technique/methods Adequate (2) Items added with very broad/vague materials with no proper techniques/methods  Good (3) Items added with complete materials with no proper techniques/methods Excellent (4) Items added with complete materials with proper techniques/methods  Items that are included in the situation analysis report but are not in the appropriate section should still receive credit and be scored in the appropriate section of the rubric. Thus, the reports scored ranged from 0 to 25 are categorized as poor-quality reports. Fair category reports are scored from 25 to 50. Good and excellent category reports should obtain a score of 51 to 75 and more than 75 respectively.  General Guidelines A good situation analysis report should include the following: • A current and active Extension Leadership Council (ELC) or other stakeholder groups to help guide the situation analysis process. • An orientation/training for the ELC/stakeholder group on the situation analysis process. • The ELC review of the existing situation analysis plan to develop an initial plan, formulate the methods to assess the community needs, analyze the unit profile, interpret findings, and finalize the reports. • Generation of a list of key issues, or issue areas, from the unit profile data and community inputs. • Identify priority issues in the unit supported by data and indicate those that are currently being addressed by VCE,    • Identify additional issues that could be addressed by VCE if given additional resources or those that are outside the scope of VCE’s ability to address. Situation Analysis Assessment The categories below correspond to different sections of the situation analysis report. Each item within a category provides a specific criterion for the quality of work. The five-code scale is intended for numerical scoring. For each item, please refer to the situation analysis report and check the box corresponding to your assessment of the item.   Absent Poor Adequate Good Excellent Category Total (CT) Conversion Value Category/Criteria 0 1 2 3 4   Overall (15% of total score) Conversion rate: (15×CT)/20 1. Quality of writing (i.e., clarity, consistency of voice, correct grammar, proper spelling)        2. Quality of report as a communication tool (i.e., free from scientific jargons and understandable to mass audiences)      3. Content presented in a logical order      4. All of the required sections of the report are included      5. Evidence of ELC members and (or) external stakeholder involvement      Preliminary Materials of the Report (5% of total score) Conversion rate: (5×CT)/12 6. Title page with the names of contributors (authors, editors, etc.)        7. Table of contents (including all sections of the report with page numbers, list of tables, list of figures, appendices, etc.)      8. List of abbreviations and acronyms      Introduction (10% of total score) Conversion rate: (10×CT)/8 9. Executive summary of the report/ abstract           10. The purpose of this report      Unit Profile (20% of total score) Conversion rate: (20×CT)/12   11. Unit profile developed with the most appropriate and current data sources         12. The unit profile should reflect the population, education, income, business, economy, social and administrative institutes, etc. of the locality      13. Source(s) of data          Absent Poor Adequate Good Excellent Category Total (CT) Conversion Value Category/Criteria 0 1 2 3 4   Community Perspective or Methodology (20% of total score) Conversion rate: (20×CT)/24 14.  Description of the sample(s) or sources of input for data collection        15.  Process for selecting participants      16. Methods indicate that data collected from both traditional and non-traditional stakeholders of VCE, including underserved audiences       17. Indicates data collection methods with the sample size for each data collecting method. Response rate should be included for surveys      18. Timeline of data collection provided      19. Various data analysis techniques are used      Major Issues (20% of total score) Conversion rate: (20×CT)/12 20. Identified issues are prioritized         21. Methods used for identifying priorities are clearly defined      22. Summary of each prioritized issues includes sources of supporting data and stakeholder’s connection to that issue      References and Appendices (10% of total score) Conversion rate: (10×CT)/12 23. References are cited in the text and listed at the conclusion of the report        24. Data collection tools are included      25. Quality of the instruments used for data collection (e.g., question type, formatting)      Grand Total (Sum of all Conversion Value)   Category Score (%) Obtained Score Excellent  >75  Good  51-75  Fair  25-50  Poor  0-25    

Identifying Key Constructs and Measurements to Assess the Situation Analysis Reports

https://tigerprints.clemson.edu/cgi/viewcontent.cgi?article=1114&context=joe

Identifying Key Constructs and Measurements to Assess the Situation Analysis Reports

Abstract

Similar works

Full text

Available Versions

Clemson University: TigerPrints