Presentation at the annual NSDL Conference, Washington DC, November 2010unpublishe

Hoelter, Lynette

Iverson, Ellen

Mason, Bruce

McMartin, Flora

Morgan, Glenda

Morrill, Joshua

Wolf, Alan

English

Illinois Digital Environment for Access to Learning and Scholarship Repository

Learning About Your NSDL Collection's Use and UsersGlenda Morgan, University of IllinoisFlora McMartin, Broad-based KnowledgeEllen Iverson, SERC - Carleton CollegeBruce Mason, University of OklahomaLynette Hoelter, TWD – University of MichiganJosh Morrill – Morrill SolutionsAlan Wolf – University of Wisconsin-MadisonAnd a Cast of Thousands!12/16/2010 NSDL PI Meeting, Washington DC 2“In God we trust. All others bring data.”               W. Edwards DemmingThe Evaluation Dilemma Stakeholders and funders want to see results (often undefined) You want to make informed decisions about what’s working and what’s not in your collectionBUT, You don’t have the resources or time to conduct evaluations on your own12/16/2010 3NSDL PI Meeting, Washington DCRevising the NSDL Faculty Survey to Meet Your Collection’s Evaluation Needs  Review the development of the survey instrument Discuss the process of revising the survey instrument to meet your evaluation needs  Share experiences of ComPADRE and Teaching With Data PIs in revising and using the survey instrument Review when and how the survey instrument might work for you Lay the groundwork for a follow on meeting (December 16) where we can consult with you around your use of the survey instrument12/16/2010 4NSDL PI Meeting, Washington DCA Short Project History Initial tool developed in 2006  Purpose: To learn how instructors use digital resources Used and tested on national audience of over 4,000 Individual questions tested for validity and reliability Revised and tested for use in two collections ComPADRE Teaching With Data12/16/2010 5NSDL PI Meeting, Washington DCComPADREHow the survey would meet our needs Need to know current practices In order to support and/or change educational practices in physics Although we've had a few informal surveys of our users and potential users at the beginning of ComPADRE, we wanted another independent view. It was important to learn about the practices of our potential users because we thought they were  different, used less technology than many of those who we hear from most often.  On a purely practical level, this survey brought in a lot more assessment muscle than we had available. 12/16/2010 NSDL PI Meeting, Washington DC 6ComPADRE AudienceWe had done a series of Surveys and Requests for Comment the first 4 years of ComPADRE. This was our next effort Building the Survey Needed to make the Survey Shorter Add Physics/ComPADRE specific issues Included High School teachers Results of the Survey Process Interesting audience data from our samples Strong audience, Expected resources, R1’s don’t care Surprise – Young faculty’s perception of DR somewhat lower Follow-up Question – New Faculty Workshop12/16/2010 7NSDL PI Meeting, Washington DCTeaching With Data - Creating Questions Refining the survey required us to think about What we really wanted to know (began with the analyses) This step helped greatly in clarifying and cementing the goals of both projects Where/How social science instruction differs from that in the “hard” sciences and what modifications to  questions/response options were needed How to convey to respondents such concepts as digital resource (and which types were important), quantitative literacy, and “(re)use” of resources Whether any modifications should be made to original questions – including demographics – based on data from the previous surveys Good question and questionnaire design (unbiased, not double-barreled, reducing respondent burden)12/16/2010 8NSDL PI Meeting, Washington DCThe Rest of the Process, Part 1 Pre-tested instrument with a group of sociology instructors and librarians and “debriefed” question-by-question afterward Revised instrument based on suggestions; initial group reviewed again Pre-tested online instrument Created sampling frame Goal: obtain enough cases from schools in each Carnegie classification for comparative analyses Aimed for 200+ faculty in each category, resulting in a sample of approximately 340012/16/2010 9NSDL PI Meeting, Washington DCThe Rest of the Process, Part 2 Ran survey in August, 2 reminders; ended in early October Response rate of 32% for those with valid email addresses12/16/2010 NSDL PI Meeting, Washington DC 10Survey Results Will… Give a baseline of digital resource use in the social sciences  Allow comparisons with faculty responses from 2 previous surveys Shed light on where instructors are finding resources and for what they use them to achieve Provide information about how instructors think (or don’t think) about QL (methods, statistics and “substantive” classes Help TwD in its marketing strategies – giving us language for describing the materials; showing how instructors think about their use so we can align that with the “sales pitch” more clearly Inform the outreach team so they can target their presentations and other outreach efforts12/16/2010 11NSDL PI Meeting, Washington DCEvolution of A Survey12INNOVATIONInitial Faculty SurveyComPADRE SurveyQuantitative Literacy SurveyAPPLICATIONSurvey QuestionsNew Measurement. Testing ideas that may or may not ultimately work (Google vs. DL). A large-scale baseline survey.  Typically longStrategic application of survey Knowledge. Meshing with preexisting surveys, groups or processes.  Results in shorter survey and largely more interpretable findings.12/16/2010 NSDL PI Meeting, Washington DCIs a Survey Right For You?Getting the most from the survey development processDo you have enough information to start formulating questions that would be understandable to your audience?13Do you have enough people who can be contacted to make a survey worthwhile?To what degree do you want new directions (innovation)?  To what degree do you want adherence to existing -or past-information?WWhat is Essential for you to know?What would be “nice to know”?Survey Consult YES or NO?12/16/2010 NSDL PI Meeting, Washington DCActually Doing the Survey You’ve got the instrument, now what? 2 issues Sample Delivery12/16/2010 NSDL PI Meeting, Washington DC 14Sample Audience What do you want to know? Population What kind of a sample do you want to use? Identifying them?12/16/2010 NSDL PI Meeting, Washington DC 15Delivery Online vs phone vs paper vs in-person How will you distribute the survey? Will this bias the result?12/16/2010 16NSDL PI Meeting, Washington DCQuestions?12/16/2010 17NSDL PI Meeting, Washington DCSupport and Follow UpFor More information about the survey: December 16, 2010 - Virtual brown bag hosted by the NSDL Resource Center.  Or contact us directly Morgan: gmorgan@illinois.edu Josh: joshua@morrillsolutions.com Flora: floramcmartin@gmail.com Ellen: eiverson@carleton.edu Alan: alanwolf@wisc.eduFor Information about ComPADRE or TWD Bruce: bmason@ou.edu Lynette: lhoelter@isr.umich.edu12/16/2010 18NSDL PI Meeting, Washington DC

Learning About Your NSDL Collection's Use and Users

IDEALS @ Illinois

Learning About Your NSDL Collection's Use and Users

Abstract

Similar works

Full text

Available Versions

Illinois Digital Environment for Access to Learning and Scholarship Repository

IDEALS @ Illinois