Interval-level measurement with visual analogue scales in Internet-based research: VAS Generator

The present article describes VAS Generator (www.vasgenerator.net), a free Web service for creating a wide range of visual analogue scales that can be used as measurement devices in Web surveys and Web experimentation, as well as for local computerized assessment. A step-by-step example for creating and implementing a visual analogue scale with visual feedback is given. VAS Generator and the scales it generates work independently of platforms and use the underlying languages HTML and JavaScript. Results from a validation study with 355 participants are reported and show that the scales generated with VAS Generator approximate an interval-scale level. In light of previous research on visual analogue versus categorical (e.g., radio button) scales in Internet-based research, we conclude that categorical scales only reach ordinal-scale level, and thus visual analogue scales are to be preferred whenever possibl

The Web Experiment List: A web service for the recruitment of participants and archiving of Internet-based experiments

TheWeb Experiment List (http://genpsylab-wexlist.unizh.ch/), a free Web-based service for the recruitment of participants in Internet-based experiments, is presented. The Web Experiment List also serves as a searchable archive for the research community. It lists more than 250 links to and descriptions of current and past Web experiments. Searches can be conducted by area of research, language, type of study, date, and status (active vs. archived). Data from log file analyses reveal an increasing use of the Web Experiment List and provide a picture of the distribution of the use of the Web experiment method across disciplines. On a general theoretical note, Web services are discussed as a viable software alternative to the traditional program forma

Ein national gefördertes Onlinelabor als Infrastruktur für die Forschung: Ergebnisse einer Meinungs- und Bedarfserhebung

Wir stellen die Ergebnisse einer Meinungs- und Bedarfserhebung zu einem national geförderten Online-Labor für die Forschung in Psychologie und verwandten Disziplinen vor. Ein solches Online-Labor sollte einen großen Teilnehmerpool, Zugang zu und Schnittstellen mit Software zur Durchführung Internet-gestützter Studien, edukative Komponenten (Methodenberatung) sowie eine Anbindung an ein Datenarchiv zur Verfügung stellen. Die Meinungsund Bedarfserhebung wurde unter wissenschaftlich tätigen Psychologen sowie anderen potenziell an einer solchen Forschungsinfrastruktur interessierten Forschenden durchgeführt. Über 85 Prozent der Befragten unterstützt ein solches Online-Labor.We present the results of an opinion and needs assessment for a nationally funded online laboratory to aid research in psychology and in related disciplines. Such an online laboratory should provide a large participant pool, should offer access to and interfaces with software to conduct Internet-based studies, provide education on research methodologies, and should connect with a data archive. The assessment was conducted among psychologists working in academia and among other researchers potentially interested in a research infrastructure of this kind. More than 85 percent of the respondents support such an online laboratory

Webdatanet : Innovation and quality in web-based data collection

The article discusses the development of WEBDATANET established in 2011 which aims to create a multidisciplinary network of web-based data collection experts in Europe. Topics include the presence of 190 experts in 30 European countries and abroad, the establishment of web-based teaching and discussion platforms and working groups and task forces. Also discussed is the scope of the research carried by WEBDATANET. In light of the growing importance of web-based data in the social and behavioral sciences, WEBDATANET was established in 2011 as a COST Action (IS 1004) to create a multidisciplinary network of web-based data collection experts: (web) survey methodologists, psychologists, sociologists, linguists, economists, Internet scientists, media and public opinion researchers. The aim was to accumulate and synthesize knowledge regarding methodological issues of web-based data collection (surveys, experiments, tests, non-reactive data, and mobile Internet research), and foster its scientific usage in a broader community

Subjective assessment of global picture-wise just noticeable difference

The picture-wise just noticeable difference (PJND) for a given image and a compression scheme is a statistical quantity giving the smallest distortion that a subject can perceive when the image is compressed with the compression scheme. The PJND is determined with subjective assessment tests for a sample of subjects. We introduce and apply two methods of adjustment where the subject interactively selects the distortion level at the PJND using either a slider or keystrokes. We compare the results and times required to those of the adaptive binary search type approach, in which image pairs with distortions that bracket the PJND are displayed and the difference in distortion levels is reduced until the PJND is identified. For the three methods, two images are compared using the flicker test in which the displayed images alternate at a frequency of 8 Hz. Unlike previous work, our goal is a global one, determining the PJND not only for the original pristine image but also for a sequence of compressed versions. Results for the MCL-JCI dataset show that the PJND measurements based on adjustment are comparable with those of the traditional approach using binary search, yet significantly faster. Moreover, we conducted a crowdsourcing study with side-by-side comparisons and forced choice, which suggests that the flicker test is more sensitive than a side-by-side comparison

Relaxed forced choice improves performance of visual quality assessment methods

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.In image quality assessment, a collective visual quality score for an image or video is obtained from the individual ratings of many subjects. One commonly used format for these experiments is the two-alternative forced choice method. Two stimuli with the same content but differing visual quality are presented sequentially or side-by-side. Subjects are asked to select the one of better quality, and when uncertain, they are required to guess. The relaxed alternative forced choice format aims to reduce the cognitive load and the noise in the responses due to the guessing by providing a third response option, namely, “not sure”. This work presents a large and comprehensive crowdsourcing experiment to compare these two response formats: the one with the “not sure” option and the one without it. To provide unambiguous ground truth for quality evaluation, subjects were shown pairs of images with differing numbers of dots and asked each time to choose the one with more dots. Our crowdsourcing study involved 254 participants and was conducted using a within-subject design. Each participant was asked to respond to 40 pair comparisons with and without the “not sure” response option and completed a questionnaire to evaluate their cognitive load for each testing condition. The experimental results show that the inclusion of the “not sure” response option in the forced choice method reduced mental load and led to models with better data fit and correspondence to ground truth. We also tested for the equivalence of the models and found that they were different. The dataset is available at http://database.mmsp-kn.de/cogvqa-database.html