TheWeb Experiment List (http://genpsylab-wexlist.unizh.ch/), a free Web-based service for the recruitment of participants in Internet-based experiments, is presented. The Web Experiment List also serves as a searchable archive for the research community. It lists more than 250 links to and descriptions of current and past Web experiments. Searches can be conducted by area of research, language, type of study, date, and status (active vs. archived). Data from log file analyses reveal an increasing use of the Web Experiment List and provide a picture of the distribution of the use of the Web experiment method across disciplines. On a general theoretical note, Web services are discussed as a viable software alternative to the traditional program forma

Lengler, Ralph

Reips, Ulf-Dietrich

RERO DOC Digital Library

Behavior Research Methods2005, 37 (2), 287-292The number of experiments conducted via the World-Wide Web appears to have grown exponentially since 1995,when psychologists began to take advantage of the newstandard for HTML that allows for convenient data col-lection (Musch & Reips, 2000). Internet-based researchhas become a new topic in psychology, and many re-searchers are looking for services that support this typeof research.In this article, we present the Web Experiment List(http://genpsylab–wexlist.unizh.ch/), a Web-based ser-vice for the recruitment of participants in Internet-basedexperiments. The Web Experiment List is also a search-able archive for the research community. It lists morethan 250 links to and descriptions of current and pastWeb experiments. Figure 1 shows the list’s main page.Two Types of Software: Programs and Web ServicesA number of tools have been developed for Internet-based experimenting that form a general framework ofreference for the methodology. These tools can be groupedinto two general classes of “software”: programs andWeb services.1 Programs follow the traditional format.They need to be installed on a computer and run locally.The working of the program depends on the computer’sconfiguration, which may vary considerably over time(as other software is installed) and from user to user. Dif-ferent types of operating systems may not allow a user toinstall the software at all. Upgrades and updates may benecessary. However, the user is in control of the service andindependent of a connection to the Internet. An example ofa tool for Internet-based experimenting (in this case forWeb-based decision-making experiments) of the programtype is WebDIP (Schulte-Mecklenbeck & Neun, 2005).Web services, on the other hand, run on a server thatis connected to the Internet. Users access it via a Webbrowser and can use it only while they are connected tothe Internet. Because the functionality of Web browsersis less dependent on the operating system (sometimesthey are even referred to as being platform independent),all who access a Web service are likely to see and experi-ence almost the same interface (but see, e.g., Dillman &Bowker, 2001, for browser-related problems in Internet-based research). Web services spare the user from up-grading and updating, since this is done by the Web ser-vice administrators at the server. Nothing is installed onthe user’s computer, saving space and time.The Web Experiment List is such a Web service, be-cause the highly frequent interactive exchange of manypeople in different roles (researchers, participants, stu-dents, historians, administrators) with the Web site couldnot possibly be organized via the program concept. TheWeb Experiment List is even part of a whole set of sev-eral general Web services for Web-based experimenta-tion that were developed to complement each other. Theseare: WEXTOR2 (Reips & Neuhaus, 2002), the Web Ex-periment List, the Web Experimental Psychology Lab(Reips, 2001), and Scientif ic LogAnalyzer (Reips &Stieger, 2004). This set of tools covers every step in theInternet-based research process: developing a Web ex-periment, recruiting participants, and analyzing the data.Recruiting Internet ParticipantsSome of the best places for recruitment are institu-tionalized Web sites for Internet-based experimenting,287 Copyright 2005 Psychonomic Society, Inc.The article is based on a presentation given at the “Tools for Internet-Based Research” symposium at the 34th Annual Meeting of the Societyfor Computers in Psychology, Minneapolis, November 18, 2004. Cor-respondence concerning this article should be addressed to U.-D. Reips,Psychologisches Institut, Universität Zürich, Rämistr. 62, 8001 Zurich,Switzerland (e-mail: u.reips@psychologie.unizh.ch). The Web Experiment List:A Web service for the recruitment of participantsand archiving of Internet-based experimentsULF-DIETRICH REIPS and RALPH LENGLERUniversity of Zurich, Zurich, SwitzerlandThe Web Experiment List (http://genpsylab-wexlist.unizh.ch/), a free Web-based service for the re-cruitment of participants in Internet-based experiments, is presented. The Web Experiment List alsoserves as a searchable archive for the research community. It lists more than 250 links to and descrip-tions of current and past Web experiments. Searches can be conducted by area of research, language,type of study, date, and status (active vs. archived). Data from log file analyses reveal an increasing useof the Web Experiment List and provide a picture of the distribution of the use of the Web experimentmethod across disciplines. On a general theoretical note, Web services are discussed as a viable soft-ware alternative to the traditional program format.288 REIPS AND LENGLERsuch as (in addition to the the Web Experiment List) theWeb Experimental Psychology Lab and the Psychologi-cal Research on the Net list by John Krantz, who pub-lished one of the first Web experiments (Krantz, Ballard,& Scher, 1997). Every month, these Web sites are visitedby thousands of potential participants (Reips, 2001). Alink on a Web experiment site may also serve as an archive,as an example for future studies, as a resource for stu-dents of psychological research, and as a reference in pub-lications.Some of the links to these and other sites on the Webthat contain large collections of psychological studiesthat are currently in progress are Psychological Researchon the Net (by John Krantz), http://psych.hanover.edu/research/exponnet.html; Web Experimental PsychologyLab (by Ulf-Dietrich Reips), http://www.psychologie.unizh.ch/sowi/Ulf/Lab/WebExpPsyLab.html; DecisionResearch Center (by Michael Birnbaum), http://psych.fuller ton.edu/mbirnbaum/decisions/thanks.htm; Lab-United –International Online-Research (by AnjaBerger and Mirko Wendland), http://www.lab-united.com/; and Social Psychology Network (by Scott Plous),http://www. socialpsychology.org/expts.htm.People coming to a study via one of these Web researchsites are true volunteers who have already decided thatthey want to take part in one or more psychology stud-ies. Consequently, the concerns one might have with re-spect to students who are participating only to fulfill anassignment are relieved with this source of participants.How to Link an Experiment to the Web Experiment ListNear the top of the main page at http://genpsylab-wexlist.unizh.ch/, there is a link called Add your ownstudy to the list that opens a form. Figure 2 shows this entryform, which an experimenter must fill out to put a Webexperiment on the Web Experiment List. The entry form contains a number of fields and op-tions. For example, entries about the author’s name, theinstitution, the e-mail of the author, the title and a briefdescription of the experiment have to be provided in thelanguage chosen in the respective menu (currently, thedescriptions can be in English and German). The user isasked to provide the URL, the date up to which the studyis to be marked as active, and any important remarks—for example, whether the study can only be accessedFigure 1. Main page of the Web Experiment List Web site, with lists of new, older active, and archivedWeb experiments in English and German. The page also contains links to the entry form for one’s ownstudy and to other information about Internet-based experimenting.WEB EXPERIMENT LIST 289with a certain plug-in or Web browser. Because the formis shared with a second Web site (the Web Survey List)the method of study also needs to be specified: Web ex-periment, Web survey with experimental component, orWeb survey. From a drop-down menu, the user can choosethe number of referred visits (visitors accessing thestudy via the Web Experiment List) before an automati-cally generated e-mail will be sent back to him or her. Inaddition, the user is offered the choice of having the Webexperiment also linked to the Web Experimental Psy-chology Lab (Reips, 2001), which generates much highervisitor traffic but has stricter conditions of ratificationof linked Web experiments.After clicking the “Submit” button, the user is pre-sented with a confirmation page that summarizes the en-tered information in the same format as that in which itwill be displayed on the study’s page in the Web Experi-ment List (see Figure 3). This Web page contains the fol-lowing information and options: a statement as to whethereverything has been inserted successfully; whether thestudy will be displayed as being currently active orarchived; the area of research chosen (e.g., “CognitivePsychology” or “Internet Science”); the submissiondate; the method of the study; the author, the author af-filiation, the title of the study, and a brief description ofthe experiment; special remarks, such as whether thestudy works with all Web browsers; a link that returnsthe user to the previous page if a change of entries is de-sired; and a link that allows the user to submit the exper-iment in the other language (names and other language-independent entries need not be retyped). The user canalso close the window or go back to the Web ExperimentList’s main page.After a Web experiment has been submitted, an auto-matically generated e-mail is sent to the user that con-firms the submission of the entry. A second e-mail is sentto the list manager, informing him of the new entry. Thelist manager then (roughly) evaluates the experiment forobvious usability problems and in light of guidelines forInternet-based experimenting (e.g., Reips, 2002). If nec-Figure 2. The request for new entry form, with fields for information about the Web studyand configurable options.290 REIPS AND LENGLERessary, he requests additional information or points outincompatibilities and ethical or methodological issues,before linking the study to the publicly accessible list.This routine check and the listing is free of charge.Features and OptionsThe Web Experiment List can be collapsed or shownin full. It shows recently added Web experiments sepa-rately from older ones, and it shows for which experimentsdata are being actively collected.One of the more important features is the search andsort option in the Web Experiment List. Figure 4 showshow an array of drop-down menu and radio buttons al-lows for combined searches. Using this option, it is easyto, let’s say, find all new and active social psychology ex-periments that are in English.On a critical note, one might argue that a list or portalformat may not be ideal for all user types (researchers insearch of participants, participants in search of Web-based studies, teachers and students on a quest for edu-cational material about psychological experiments, mem-bers of the scientific community in search of an archive,etc.). However, with the search feature, the Web Experi-ment List already contains a second way of finding Webexperiments within the site. Also, to date, no other waysof making a large collection of Web experiments acces-sible to the public have been successfully launched onthe Internet. Empirically, like the other lists mentionedearlier, the Web Experiment List stands as a well-usedWeb site and gathers some interesting statistics regard-ing the types of Web experiments conducted and ac-cessed, as will be shown in the next section.Usage Data and DescriptivesTwo thirds of submitted Web experiments are in En-glish only, about 27% are in German only, and 6% areconducted in both languages. The ratio of click-throughsto the German version is about four fifth of the partici-pants clicking through to the English version of the samestudies. Currently, about 10% of the experiments are ac-tive; all others are archived. The most frequently accessedexperiment among the more than 250 listed Web exper-iments was accessed more than 1,200 times (about 750times in the German version and about 450 times in theFigure 3. The feedback screen that is shown after a study is entered.The study description is displayed in the very same format as would beseen after one clicks a study link on the main page.WEB EXPERIMENT LIST 291English version). About 100 Web experiments are incognitive psychology, about 90 in social psychology,about 25 in perception, and about 10 each in Internet sci-ence, personality, and clinical psychology, with very fewin developmental psychology and neuropsychology.New studies are currently being added at a rate of aboutone to three per week.Visits to the Web site have increased from about 35 toabout 40 per day during the last 7-month period we eval-uated, excluding visits by search engine spiders and ro-bots. Top-level domains that clearly indicate 1 of 64countries of originating access (not .net or .com etc.) re-veal that most visitors came from Switzerland, Germany,the United States, the United Kingdom, and Japan. Therewere about three visits a day in which the Web Experi-ment List was found via a search engine. Of these, 80%came via Google, 10% via Yahoo, and 10% via othersearch engines. Top search terms were “experiment” (andword derivations), “psychology,” “list,” “web,” and “sur-vey.” The list of search phrases also shows that the WebExperiment List was often found when people searchedfor the name of a particular researcher.An analysis of paths taken through the Web site showsthat many visitors first clicked on the link at the bottom ofthe initially short page to expand the view to the full list.Length of visits was between 15 sec and 14 min for mostvisitors, with an average of a little more than 3.5 min. Mak-ing a new entry to the Web Experiment List takes about3:20 min, on average. Reading the description of one of theWeb experiments takes about 1:10 min on average, and theaverage view time for the archive is about 1:50 min. Usingthe search engine takes about half a minute, on average.Discussion and OutlookA Web service was presented that is designed to be ofhelp to Web researchers who wish to freely recruit par-ticipants on the Internet for their Internet-based experi-ments. With its vast archive, the service offers examplesof completed studies for those who seek to learn—namely, teachers and students of psychology. Readers ofempirical reports that are based on Internet-based ex-periments may visit to get detailed “participant experi-ences.” And last but not least, the Web Experiment Listserves people from the larger population who wish tofind and be part of the exciting process of research, whootherwise might never have had the opportunity to par-ticipate in a study.Future developments of the Web Experiment List willinclude the buildup of a participant pool that will beavailable as an additional source of participants to thoseresearchers who link their Web experiments to the list.We also contemplate adding more features to the searchpage—for example, searching by popularity or even fulltext search. Because some Web experiments do not re-main on line for display after data collection has finishedand, therefore, the archiving function of the list is contin-uously challenged, a feature may be added that takes a“snapshot” of the experiments at the time of submission.Recently, we set up a sister Web site for Web-basedsurveys, the Web Survey List. Although there was an ob-vious need for such a list of surveys, given the higherbase rate of this study type on the Internet and the fre-quent requests for listing we received, it is still too earlyto assess whether this new list will be a success. We cer-tainly hope that both the Web Survey List and the WebExperiment List will be or continue to be of help to thecommunity of researchers who have begun to use the In-ternet for the study of human behavior.REFERENCESBirnbaum, M. H., & Reips, U.-D. (2005). Behavioral research and datacollection via the Internet. In R. W. Proctor & K.-P. L. Vu (Eds.), Thehandbook of human factors in Web design (pp. 471-492). Mahwah,NJ: Erlbaum.Figure 4. All listed studies are searchable by category via the interface that is dis-played in this figure.292 REIPS AND LENGLERDillman, D. A., & Bowker, D. K. (2001). The Web questionnaire chal-lenge to survey methodologists. In U.-D. Reips & M. Bosnjak (Eds.),Dimensions of Internet science (pp. 159-178). Lengerich, Germany:Pabst.Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the resultsof laboratory and World-Wide Web samples on the determinants offemale attractiveness. Behavior Research Methods, Instruments, &Computers, 29, 264-269.Musch, J., & Reips, U.-D. (2000). A brief history of Web experiment-ing. In M. H. Birnbaum (Eds.), Psychological experiments on the In-ternet (pp. 61-87). San Diego: Academic Press.Reips, U.-D. (2001). The Web Experimental Psychology Lab: Fiveyears of data collection on the Internet. Behavior Research Methods,Instruments, & Computers, 33, 201-211.Reips, U.-D. (2002). Standards for Internet-based experimenting. Ex-perimental Psychology, 49, 243-256.Reips, U.-D., & Neuhaus, C. (2002). WEXTOR: A Web-based tool forgenerating and visualizing experimental designs and procedures. Be-havior Research Methods, Instruments, & Computers, 34, 234-240.Reips, U.-D., & Stieger, S. (2004). Scientific LogAnalyzer: A Web-based tool for analyses of server log files in psychological research. Be-havior Research Methods, Instruments, & Computers, 36, 304-311.Schmidt, W. C. (2000). The server-side of psychology Web experi-ments. In M. H. Birnbaum (Ed.), Psychological experiments on theInternet (pp. 285-310). San Diego: Academic Press.Schulte-Mecklenbeck, M., & Neun, M. (2005). WebDiP - A tool forinformation search experiments on the WWW. Behavior ResearchMethods, 37, 293-300.NOTES1. Despite a certain similarity, the distinction between programs andWeb services differs from another important one in Internet-based re-search: client-side versus server-side (Birnbaum & Reips, 2005; Reips,2002; Schmidt, 2000). Client-side and server-side processes are di-rected by a central agent (e.g., a Web service); programs and Web ser-vices are central agents.2. http://psych-wextor.unizh.ch/wextor/.(Manuscript received January 1, 2005; revision accepted for publication May 20, 2005.)

The Web Experiment List: A web service for the recruitment of participants and archiving of Internet-based experiments

http://doc.rero.ch/record/315428/files/13428_2010_Article_BF03192696.pdf

The Web Experiment List: A web service for the recruitment of participants and archiving of Internet-based experiments

Abstract

Similar works

Full text

Available Versions

RERO DOC Digital Library