Gilmore, Tracy

Moyo, Lesley

English

Eastern Michigan University: Digital Commons@EMU

-THE QUALTRICS TEMPO: CHECK THE PULSE OF THE CLASS...- LOEX-2013       155 Moyo (Director for Library Research and Instructional Services) and Gilmore (Collections Assessment Librarian) Virginia Tech  [Blacksburg, VA]     THE QUALTRICS TEMPO: CHECK THE PULSE OF THE CLASS USING QUALTRICS RESEARCH SUITE LESLEY MOYO AND TRACY GILMORE INTRODUCTION  The Qualtrics Tempo interactive presentation demonstrated the use and effectiveness of the Qualtrics Research Suite to collect and analyze data in an instruction setting. The session objectives were to simplify lesson assessment and evaluation and demonstrate how to generate real-time assessment data using Qualtrics. Participants were led through a robust hands-on session covering basic survey creation and administration, report generation, and data analysis. Qualtrics is an online survey tool used to quickly create, distribute, view and analyze response results graphically. It consists of three major domains: the Qualtrics Research Suite —a comprehensive suite of tools for creating and distributing online surveys, with integral data analysis and reporting features, Site Intercept—an online marketing and research tool that allows you to dynamically interact with targeted website visitors, and Qualtrics 360—a repository of corporate measurement and evaluation tools. In this LOEX interactive session, the Qualtrics Research Suite was employed as the platform for assessing library instruction. In a library instruction setting, the Qualtrics software allows instructors to send students to an online survey where they can provide feedback on their learning experience. The system allows for immediate analysis of their responses to assess session effectiveness and learning outcomes. With a focus on academic institutions (Leland, 2011), the Qualtrics Research Suite is great for gathering data quickly and offers functionality such as advanced survey logic and data analysis support for cross-tabulation and regression analysis. Qualtrics subscription is available at both individual and institutional levels. At Virginia Tech, the service is available free to all Virginia Tech students, faculty, and staff via an institutional account. Individual free accounts with limitations are also freely available to anyone (http://www.qualtrics.com/).  Strong proponents of library instruction (Oakleaf, 2009, 2011; Sobel & Wolf, 2011; Whitlock & Nanavati, 2013), stress the importance of pervasive assessment and evaluation. Three foundational areas are needed to support establishment of strong assessment programs in library instruction: first, understanding assessment and evaluation; second, establishing an assessment infrastructure using an appropriate assessment tool or system; and third, knowledge and application of basic research methods to analyze data and apply the results to make decisions about specific library sessions or instructional programs. Through an interactive workshop, the session used the Qualtrics Research Suite to demonstrate not only the importance of these three areas, but also the strategies for operationalizing them. In particular, the workshop simulated a real learning situation by designing a real lesson that incorporated everything from learning objectives, to lesson content that engaged participants in learning how to use the Qualtrics Research Suite, to assessment activities that generated data on the effectiveness of the session. The session emphasized not only the immediacy of the results, but also the seamless progression of assessment when the recommended approach is followed. To ensure full participation in the LOEX interactive session, participants without devices were provided an iPad or a NetBook to use. In addition, they were also able to observe the lesson on the projected presenter screen. Furthermore, all session information and materials were posted online for participants to review and use for up to six months following this session. Prior to the start of the session, participants were also provided the lesson plan and other learning material for their perusal. This allowed them a few minutes to acquaint themselves and engage with materials.  THE INTERACTIVE SESSION  Prior to the hands-on session, participants were directed to the Qualtrics website (http://www.qualtrics.com/) to 156       LOEX-2013   -MOYO AND GILMORE- set up free personal accounts. The provided iPads and Netbooks were pre-loaded with a link to Qualtrics, the class outline, and the online lesson evaluation.  Participants were first taught how to use Qualtrics to create and administer surveys for real-time library instruction evaluation. The main Qualtrics Research Suite components were covered, including: Survey builder—how to create and edit the survey, and how to distribute the survey via a URL. During the lesson, participants could follow along and either re-create the same survey on their device, or create an alternate survey based on their individual institutional perspective. This made for very enthusiastic questions and discussion as participants explored survey options that met their unique needs. To demonstrate data analysis techniques, participants were asked to provide feedback on their experience and perception of the interactive session via a short Qualtrics survey that had been created beforehand. The survey included three types of questions; Likert Scale, multiple choice for quick response, and two open-ended questions for qualitative feedback. This feedback generated ample evaluation data that was used to demonstrate the data analysis features inherent in the Qualtrics system, and how to employ those features to generate usable results that can inform decisions on library instruction.  Before statistical data analysis techniques were taught, the session interactively explored the various options available for instant report generation, including PDF, Powerpoint, Excel, and Word reporting options. This was followed by the data analysis and review, beginning with descriptive statistics, and followed by more robust statistical analyses and tests, including correlational tests, and cross-tabs, t-tests, and mention of other features that facilitate more advanced tests such as regression analyses. Throughout this component of the workshop, participants could view how their feedback translated into usable data through the examples of the reports and data tables generated instantaneously by the Qualtrics system.  The following is a detailed outline of the workshop content and sequence as covered in the interactive session. Creating and Editing Surveys Of the three options (Quick Survey Builder, Create From Copy, and Survey Library) available to create a survey, the Quick Survey Builder was selected for this lesson; it is the fastest way to begin creating and editing a survey. Step-by-step, items such as creating a survey name and folder were explained, followed by an overview of the Qualtrics work space and tabs.  The next step in the process included an overview of Qualtrics editing tools. To demonstrate question types, editing and formatting, participants created their own Library Instruction Survey from a list of preset questions (see Appendix B). The question variety included:  • Multiple Choice • Automatic Choice (true/false, yes/no) • Likert Scale  • Question Matrix • Open ended questions with text box/text area.  A hands-on tour and overview of the Qualtrics Question Pane was given along with explanations on how to create, edit, and format each question or response type. Detailed examples, including the location of specific tools within the question pane, were outlined in a power point handout included with the online materials.  The Powerpoint can be viewed via the LOEX 2013 Material website at https://sites.google.com/a/vt.edu/loex-2013-material/qualtrics-powerpoint.  Managing the Survey Flow After question creation and editing were completed, an overview and demonstration of the Survey Flow tools was given. The Qualtrics Survey Flow are a collection of powerful survey logic tools that allow the user to create blocks of questions, change the navigation or presentation of questions,  and  to create various survey experiences that are based on respondents answers. Included in this suite of tools are: • Numbering and Ordering • Display logic • Skip logic  • Branch logic • Block randomization Due to time constraints not all of the available features were covered. However, participants were able to experiment with several of the survey flow logic tools. Participants changed the numbering and ordering of questions, created a block of questions, and experimented with Skip Logic.  Throughout these exercises, they posed relevant questions that spurred ongoing discussions and interactivity.  Survey Distribution The final step in the survey creation process was survey activation and distribution.  The options for survey distribution are outlined in the menu within the Distribute Survey tab.  Included among the survey distribution options are: distribution via social media, in-page pop-ups, email, and URL links. This session adopted the URL link option. Once participants had selected this option and activated their surveys they each received an active anonymous survey link that they could share or follow in order to access the survey and provide requested feedback.  LESSON EVALUATION  When the hands-on lesson and associated exercises were completed, participants were directed to the Evaluation Survey link to evaluate their class experience (see Appendix C). -THE QUALTRICS TEMPO: CHECK THE PULSE OF THE CLASS...- LOEX-2013       157 The survey provided real-time collection of data and was used to demonstrate how Qualtrics can be used as an in-class assessment tool, with immediate results. Participants were given several minutes to complete the survey and provide their feedback.  Data analysis of the responses provided by participants was demonstrated using the Qualtrics data analysis and presentations features. Participants first received a basic overview and demonstration of the View Reports tabs and functionality. Items highlighted in this section included: • Accessing and Navigating reports • Viewing individual responses • Exporting and sharing reports • Integrating charts and graphs in the result sets • Using the Cross Tabs option to conduct statistical tests Data Analysis Disclosure - The presenters pre-populated some of data for the sake of expediency and demonstration purposes, however 45 responses were obtained from the session participants. The basic tests that were conducted in this segment include descriptive statistics providing such data as variance and standard deviation, a correlational test using cross-tabs, and interpretation of the results. Although time constraints prevented detailed exploration of advanced statistical analysis features inherent in Qualtrics, an overview of the basic features provided an entre into this area, and a starting point for participants who would wish to explore these tools further.  USEFUL RESOURCES  Qualtrics is a robust online survey platform that offers a multitude of sophisticated services that far exceed the ability to teach within a 45-minute session.  In cognizance of this fact, the presenters recommended additional resources that participants could use to learn more about Qualtrics. Included among the recommended resources are great online training resources provided by Qualtrics: • Learn Qualtrics in five steps: http://www.qualtrics.com/university/researchsuite/learn-qualtrics-in-5-steps • Video clips at: http://www.qualtrics.com/university/researchsuite/ • Survey Design Tips: http://www.qualtrics.com/university/researchsuite/resources-to-help-you • Free eBooks on basic marketing research: http://www.qualtrics.com/university/researchsuite/ebooks CONCLUSION  By the end of the interactive session, anecdotal observations and feedback provided by the participants, as well as the questions asked during the session indicated that participants were able to: 1. Use Qualtrics to create and administer an online survey for class assessment and evaluation purposes (CAT). 2. Conduct simple data analysis (correlational study) using Qualtrics’ Cross-tabs. 3. Generate instant Qualtrics reports from survey data using Qualtrics’ reporting system. Thus, confirming that the session objectives were met. Several participants requested and were provided information on how they might adopt Qualtrics at personal on institutional levels. The feedback received during evaluation of the lesson included suggestions to offer additional session of this workshop at other professional venues.  REFERENCES Leland, E. (2011). A few good online survey tools. Idealware. Retrieved from http://www.idealware.org/articles/fgt_online_surveys.php  Oakleaf, M. (2009). The information literacy instruction assessment cycle: A guide for increasing student learning and improving librarian instructional skills. Journal of Documentation, 65(4), 539-560. doi: 10.1108/00220410910970249  Oakleaf, M. (2011). Are they learning? Are we? Learning outcomes and the academic library. Library Quarterly, 81(1), 61-82.  Sobel, K., & Wolf, K. (2011). Updating your tool belt: Redesigning assessments of learning in the library. Reference & User Services Quarterly, 50(3), 245-258.  Qualtrics (2013). Q University. http://www.qualtrics.com/university/researchsuite/basic-building/editing-questions/overview-and-tutorial-video (accessed May 2013).  Whitlock, B., & Nanavati, J. (2013). A systematic approach to performative and authentic assessment. Reference Services Review, 41(1), 32-48.  doi: 10.1108/00907321311300866 158       LOEX-2013   -MOYO AND GILMORE- APPENDIX A  LOEX 2013 MATERIAL   THE QUALTRICS TEMPO: CHECK THE PULSE OF THE CLASS USING QUALTRICS SURVEY SOFTWARE FOR REAL-TIME ASSESSMENT AND EVALUATION PART I – Interactive Lesson  Lesson Objectives By the end of this session, participants will: 1.       Be able to create and administer basic surveys using Qualtrics 2.       Be able to generate instant Qualtrics reports from survey data using Qualtrics’ reporting system 3.       Be able to conduct simple data analysis using Qualtrics’ cross-tabs   Instruction Session (Hands-on session) A.   Overview of Qualtrics - Value of Qualtrics in library instruction & research Useful resources for learning Qualtrics Learn Qualtrics in five steps: http://www.qualtrics.com/university/researchsuite/learn-qualtrics-in-5-steps) Also 5 video clips at: http://www.qualtrics.com/university/researchsuite/   1.       Easy to implement Qualtrics as part of the instruction/research infrastructure 2.       Instant results 3.       Data can be crunched to answer questions as questions arise 4.       Many types of data analysis features B.      Login into Qualtrics (Qualtrics.com) individual accounts institutional accounts 1.       Free accounts 2.       Paid accounts C.     Create survey (from survey builder, from copy, from template) Select survey builder 1.       Name and folder 2.       Question blocks, Question items, add, delete 3.       Types of questions/responses 4.       Edit individual questions (create/remove questions) 5.       Rich text editor D.     Edit survey 1.       Numbering/ordering 2.       Display logic 3.       Skip logic E.      Distribute survey 1.       Activate survey 2.       Anonymous survey    PART II - LESSON EVALUATION   Go to this link: https://virginiatech.qualtrics.com/SE/?SID=SV_50SEHT0F5cTW8W9     (The link is bookmarked on the netbooks/iPads) 1.       I have previous experience of assessing Library Instruction 2.       This was the first time I was introduced to Qualtrics 3.       This lesson was useful in learning some basics about Qualtrics 4.       As a result of this session I am likely to use Qualtrics in future assessment efforts 5.       I have the necessary quantitative research methods skills for analyzing Qualtrics data -THE QUALTRICS TEMPO: CHECK THE PULSE OF THE CLASS...- LOEX-2013       159 6.       I would like to learn more about Qualtrics and its potential for assessing library instruction 7.       Here are my other comments   PART III - Data Analysis & Review      Viewing Results 1.       Sorting results by field 2.       Descriptive statistics a.       Graphs and Tables 3.       Creating subgroups Generating reports 1. PDF 2. PowerPoint 3.       Excel Basic data analysis 1.       Correlation 2.       (mention of other available data analysis features: t-tests, regression, etc.)   PART IV – Summary  ·         Recap ·         Q&A ·         Wrap-up   160       LOEX-2013   -MOYO AND GILMORE- APPENDIX B   LIBRARY INSTRUCTION SURVEY Q1 Indicate your academic level  Freshman  Sophomore  Junior  Senior Q2 This is my first library instruction session  True  False Q3 Overall how would you rate this session?  Strongly Agree Agree Neither Agree nor Disagree Disagree Strongly Disagree I found the information to be useful.           The instructor presented the information clearly.           The instructor made good use of examples and illustrations.           I will use the information from this course in the future.            Q4 Here are my other comments.     -THE QUALTRICS TEMPO: CHECK THE PULSE OF THE CLASS...- LOEX-2013       161 APPENDIX C   EVALUATION SURVEY Q1 I have previous experience in assessing library instruction.  True  False Q2 This was the first time I was introduced to Qualtrics.  True  False Q3   Strongly Agree Agree Neither Agree nor Disagree Disagree Strongly Disagree This lesson was useful in learning some basics about Qualtrics           As a result of this session I am likely to use Qualtrics in future assessment efforts           I have the necessary quantitative research methods skills for analyzing Qualtrics data           I would like to learn more about Qualtrics and its potential for assessing library instruction             Q4 Here are my other comments.    162       LOEX-2013     

The Qualtrics Tempo: Check the Pulse of the Class Using Qualtrics Research Suite

https://commons.emich.edu/cgi/viewcontent.cgi?article=1029&context=loexconf2013

The Qualtrics Tempo: Check the Pulse of the Class Using Qualtrics Research Suite

Abstract

Similar works

Full text

Available Versions

Eastern Michigan University: Digital Commons@EMU