A survey of research methods used to study student and faculty use of digital libraries and digital resources across a number of related studies.National Science Foundation DUE 1049537 & 1049531not peer reviewe

Iverson, Ellen

McMartin, Flora

Morgan, Glenda

Morrill, Joshua

O'Connell, Kristin

Illinois Digital Environment for Access to Learning and Scholarship Repository

  AEA - 2012 Evaluating Faculty and Student Use of Digital Resources for Teaching and Learning Presenters: Flora McMartin, Broad-based Knowledge, flora.mcmartin@gmail.com Glenda Morgan, University of Illinois Urbana, gmorgan@illinois.edu Josh Morrill, Morrill Solutions Research (MSR), joshua@morrillsolutions.com Ellen Iverson, Carleton College, eiverson@carleton.edu Kristin O’Connell, Carleton College, koconnell@carleton.edu   American Evaluation Association Annual Meeting, Minneapolis, Oct 2012              Copyright, creative Commons Attribution Reqd WHY STUDY DIGITAL RESOURCES?  How people interact with scholarly content is changing  Roots in NSDL and other digital library initiatives  People had only studied how faculty used specific collections or how students did research papers  No one knew how faculty or students  found & used materials A STUDY BEGINS…A STUDY EVOLVES General Faculty Baseline Faculty Studies in Context General Student Baseline Faculty Use of Digital Resources Physics Community Social Science Community Geoscience Community Student Digital Strategies Each evolution of our original study opened us up to new techniques and new potential for the data and data collection FOCUS  GROUPS Faculty Use of Digital Resources Student Use of Digital Resources Starting With Focus Groups Faculty (2006) What did we know? ---Not much. RQ1 - How do faculty use online materials in teaching? RQ2 - How do materials align w/ faculty work patterns? RQ3 - What makes online materials useful for teaching?  Students (2011) What did we know? ---A little. RQ1 - How do students use digital learning resources?  RQ2 - Why do students use these resources?  RQ3 - What is the impact of this use on students’ learning? RQ4 - What are the barriers to their use?   Focus Group Findings Faculty (2006) • Digital Resources over 'Learning Objects' • Personal definitions of digital libraries (DLs) varied widely • Personal web-pages, currated collections, browsers  • Very few people knew about NSDL (or other DLs) • Barriers to use • Google • Information overload • Concern about copyright and use • Not invented here   Focus Group Findings Students (2011) • Very information literate (savvy) • Used Web as supplement to class materials (text books still very important) • Social networking important, but most worked alone • Iterative use of Wikipedia - Google - friends - textbooks  • Advice • Students are over surveyed   How Focus Groups Informed Our Surveys  Faculty(2006)  • Language –    **Digital Resource** • Barriers    Students (2011)  • Demographics • Study/learning habits affect choice/use  • Reach to ALL students   Both directly informed the survey process that emerged from the focus groups. SURVEYS Faculty Use of Digital Resources Student Use of Digital Resources Survey Administration Faculty (2006) Large Sample Attained  N=4,479 BUT administration relied on complex, high-touch, messy administration. Campuses were the (sometimes reluctant) go-betweens. Students (2011) US Students Marketing Sample Vendor Large Sample Attained  N=1,749 Went more directly to students.  Nice variation in sample.  Used Marketing vendor for sample.  MUCH EASIER! Student Sample Lemonade We do not have a variable to select students. From our panel  Sample Vendor (Survey Monkey Audience) Student Sample Lemonade We do not have a variable to select students. From our panel  OK- select on a limited age range (18-30) and we will ask in the survey if they are past/ current/ never been students. US Sample Vendor (Survey Monkey Audience) We now have 3 useful groups to compare:    - Current students (full time part time, etc.)   - Past students / Alumni   - Never students/ Never went to college Survey Findings (“6-Word Mottos”) Faculty (2006) Lots of data, lots of comparisons. Experienced a bit of “paralysis by analysis”  Some of the most interesting relationships were the most complex to explain. Students (2011) Limited, shorter survey.  Borrowed market research techniques/ analysis to help make findings more useable/ approachable. “Many chefs make lots of soup!” “Develop frameworks to understand the results” Ambivalent	Learners	48%	of	Sample	This segment addresses learning problems using a plan (at least they believe that they have a plan).  But, mostly, they do not feel strongly about their learning.  They are confident in their ability to find information, but do not enjoy studying nor do they have a need to learn.  This is the largest learner segment from the sample.  Adap ve	Learners	26%	of	Sample	This segment exhibits a lot of characteristics of “ideal” learners (They solve problems with a plan, they are systematic, they set goals, they ask for help if they experience a problem, they enjoy studying and have a need to learn).  A differentiator in this group is that there is more variance around setting specific times to study.  For example, this could be a learner who studies in a hallway whenever they had some free time.   Free	Form	Learners		13%	of	Sample	This group is not systematic in their learning, and do not solve problems with plans.  But they are willing to change what they do when presented with new information (may speak to an experiential type of learner).  This group also feels like they have a need to learn, but are among the least likely to set aside specific time to study.   Time	Sensi ve	Learners	11%	of	Sample	This segment is similar to the adaptive learners in many ways (use a plan, are systematic, etc), but they are just not quite as strong in these skills. Directionally they are identical to adaptive learners.  The other key difference is that this group is the most likely to set specific times to study, and least likely to ask for assistance with a problem. This is also the smallest learner segment.   Early Student Findings… LINKING INTERVIEWS WITH SURVEY METHODS Faculty Use of Digital Resources Social Science Geoscience Using Interviews with Surveys Focus groups Survey Survey Interviews All STEM faculty Social Science faculty Geoscience faculty Survey Web analytics and interviews Linking Interviews with Survey methods  Social Science Faculty Study Design Survey • Use digital resources • Assess quantitative literacy  Qualitative Study • Approach teaching • Identify potential website users Linking Interviews with Survey methods  Social Science Faculty Study Design   Survey  • Administered in 2010  • Sent to 3280  faculty (stratified sample*) • 1037 responses (32% response rate)  Qualitative Study • Interviewed 27 survey respondents in Fall 2011 • Interviewed additional two-year and economics faculty (5 participants in Spring 2012)  Social Science Interview Study   Linking Interviews with Survey methods  Survey Interviews Social Science faculty • Detect differences between faculty at different institutional types  • Learn more about the teaching practices of instructors at different institutions  • Understand more specifically how faculty used digital resources to support their teaching  Making sense of survey data Linking Interviews with Survey methods  Survey: • Reducing student anxiety with using quantitative data not important for setting goals for course Interviews: • Student anxiety with quantitative data was major challenge ID target audience Linking Interviews with Survey methods  Persona 1: 2YC Instructor (non-methods/econ) • Teach quantitative skills at a basic level • Know about percentages  • Be able to interpret a graph • For graduation: basic quantitative skills • Pedagogy: Small group activities that engage • Looking for short activities on an ideal website Linking Interviews with Survey methods  Persona 2: Instructor at a four-year institution (non-methods/econ) • Teach quantitative skills at an advanced level • Statistical tests • Data analysis • For graduation: advanced quantitative skills • Pedagogy: Lecture and deliver content easily • Looking for relevant short video clips, blog posts, to add in to their lectures on an ideal website      Linking Interviews with Survey methods  3rd mixed method approaches Focus groups Survey Survey Interviews All STEM faculty Social Science faculty Geoscience faculty Survey Web analytics and interviews Linking Interviews with Survey methods  Case Studies Web Log Excerpt for a Single Deep Session • 11:53:13 GET/resources/2304.html 11:53:13 GET/resources/23072.html • 11:53:22 GET/redirect.php?r=http://geomechanics.geol.pdx..... • 11:54:00 GET/resources/2304.html • 11:54:01 GET/resources/23072.html • 11:54:03 GET/NAGTWorkshops/ • 11:54:03 GET/NAGTWorkshops/index.html • 11:54:13 GET/NAGTWorkshops/careerprep/index.html  Corresponding narrative • “This is a visit to the career collection that begins with a search to a structural geology handout . . .” Linking Interviews with Survey methods  1. Six months of website use analyzed  Case Studies 2. National survey responses to teaching approaches and publishing history 3. Analysis of previous interview responses 4. New telephone protocol linking website use to six months of teaching  Linking Interviews with Survey methods  Next steps • Classroom observation using Reformed Teaching Observation Protocol • Collection and analysis of syllabi and assignment prompts • Student assessment using Geoscience Literacy Exam (GLE) Linking Interviews with Survey methods  Thanks to the National Science Foundation for Support • NSF DUE award no. 1049537 & 1049531   

Evaluating Faculty and Student Use of Digital Resources for Teaching and Learning

IDEALS @ Illinois

https://www.ideals.illinois.edu/bitstream/handle/2142/38402/Morgan-AEA-2012.pdf?sequence=2

Evaluating Faculty and Student Use of Digital Resources for Teaching and Learning

Abstract

Similar works

Full text

Available Versions

Illinois Digital Environment for Access to Learning and Scholarship Repository

IDEALS @ Illinois