Using protocol analysis to explore the creative requirements engineering process

Protocol analysis is an empirical method applied by researchers in cognitive psychology and behavioural analysis. Protocol analysis can be used to collect, document and analyse thought processes by an individual problem solver. In general, research subjects are asked to think aloud when performing a given task. Their verbal reports are transcribed and represent a sequence of their thoughts and cognitive activities. These verbal reports are analysed to identify relevant segments of cognitive behaviours by the research subjects. The analysis results may be cross-examined (or validated through retrospective interviews with the research subjects). This paper offers a critical analysis of this research method, its approaches to data collection and analysis, strengths and limitations, and discusses its use in information systems research. The aim is to explore the use of protocol analysis in studying the creative requirements engineering process.<br /

SHAPA: An interactive software tool for protocol analysis applied to aircrew communications and workload

As modern transport environments become increasingly complex, issues such as crew communication, interaction with automation, and workload management have become crucial. Much research is being focused on holistic aspects of social and cognitive behavior, such as the strategies used to handle workload, the flow of information, the scheduling of tasks, the verbal and non-verbal interactions between crew members. Traditional laboratory performance measures no longer sufficiently meet the needs of researchers addressing these issues. However observational techniques are better equipped to capture the type of data needed and to build models of the requisite level of sophistication. Presented here is SHAPA, an interactive software tool for performing both verbal and non-verbal protocol analysis. It has been developed with the idea of affording the researchers the closest possible degree of engagement with protocol data. The researcher can configure SHAPA to encode protocols using any theoretical framework or encoding vocabulary that is desired. SHAPA allows protocol analysis to be performed at any level of analysis, and it supplies a wide variety of tools for data aggregation, manipulation. The output generated by SHAPA can be used alone or in combination with other performance variables to get a rich picture of the influences on sequences of verbal or nonverbal behavior

An idiographic approach to the fluctuation of appraisals and coping during a trapshooting competition

Events occurring during competition can impact athletes differently and influence their cognitive states and emotional states. Therefore, appraisal and coping processes are individual and can be understood better using an idiographic approach. The purpose of this study is two-fold: 1) to describe the nature of the fluctuation of emotional states and coping processes during the competition and 2) to propose an idiographic and ecologically valid method of study of these processes through the use of verbal protocols and sequential analysis. One master world-class elite (58 years old and 28 years of competitive experience) and one master 4th-category regional level trapshooter (59 years old and 30 of experience) participated in this study. Participants completed an affect grid after each shot during two competitions of the national trapshooting championship. Each competition was composed of 6 sets of 25 shots. After each set, participants provided verbal reports using a delayed verbal protocol procedure. This procedure consisted of identifying critical moments within the competition, and reporting thoughts and feelings immediately before and after each critical moment. Verbal reports were transcribed verbatim and coded according to Lazarus’ cognitive-motivational-relational theory of emotion. Units of information were submitted to event sequential analysis to determine the probability of occurrence of paired-events. The elite level athlete reported a stable pattern of pleasure and arousal levels, while the non-elite athlete reported greater fluctuation of emotional states. It was found great inter- and intra-individual variability depending on the context, but patterns of appraisal and coping were identified through sequential analysis

A frame signature matrix for analysing and comparing interaction design behaviour

Protocol studies are an established method to investigate design behaviour. In the context of a project to investigate novice interaction design (ID) behaviour across protocols and cultures, we found that existing design behaviour analysis frameworks did not provide reliable results. This paper describes the development of a new approach to analyse and compare ID behaviour using verbal protocols. We augment Schön’s basic design and reflection cycle with construction of a frame signature matrix and analogical categorisation coding. We demonstrate this approach by comparing two protocols of novice interaction designers in Botswana. The initial findings indicate that this approach increases consistency and accuracy of coding, and that there are different degrees of reframing for the design problem and solutions

Cognitive Science and Psychology

The protocol algorithm abstracted from a human cognizer's own narrative in the course of doing a cognitive task is an explanation of the corresponding mental activity in Pylyshyn's (1984) virtual machine model of mind. Strong equivalence between an analytic algorithm and the protocol algorithm is an index of validity of the explanatory model. Cognitive psychologists may not find the index strong equivalence useful as a means to ensure that a theory is not circular because (a) research data are also used as foundation data, (b) there is no justification for the relationship between a to﷓be﷓validated theory and its criterion of validity, and (c) foundation data, validation criterion and to﷓be﷓validated theory are not independent in cognitive science. There is also the difficulty with not knowing what psychological primitives are

Estimating intra-rater reliability on an oral english proficiency test from a Bilingual Education Program

Este estudio tiene como objetivo presentar los resultados de una investigación la cual pretendía estimar el nivel de confiabilidad intra-evaluador en un examen de suficiencia oral en inglés, y determinar los diferentes factores internos y externos que afectan la consistencia del evaluador. Los participantes involucrados en el desarrollo de este estudio fueron dos profesores encargados de evaluar la sección de habla de un examen de suficiencia administrado en la Licenciatura en Bilingüismo con énfasis en inglés. Se calculó un coeficiente de correlación con el fin de establecer la consistencia de los evaluadores mientras que un protocolo verbal retrospectivo fue llevado a cabo para recopilar información acerca de los factores que influyen en la confiabilidad del evaluador. Los resultados sugieren que hay un alto nivel de confiabilidad intra-evaluador en el examen de suficiencia en cuanto el coeficiente de correlación arrojó valores superiores a .80. No obstante, aspectos relacionados con la falta de adhesión a los criterios de la rúbrica, la relación evaluador-estudiante, las condiciones físicas, y la presión y responsabilidad del evaluador para dar una nota precisa fueron identificados como factores que afectan la consistencia del evaluador. Finalmente, se proporcionaron algunas implicaciones procedentes de esta investigación

Assessing the prosody of minimally to nonverbal children with autism

A procedure for assessing basic prosodic perception and production abilities of minimally to nonverbal children and adolescents with autism spectrum disorder is described (AP: Assessment of Prosody). The procedure consists of three sections: an optional primer phase, a learning phase, and an assessment phase. It includes the assessment of both the perception of basic pitch accent structure distinctions (low versus high) as well as elicits expressive productions of these contrasts. The goal of the procedure is to evaluate the extent to which this population can perceive and produce prosodic distinctions. The overarching aim is to create a pre and post assessment to quantify prosodic competence and performance of minimally to nonverbal children and adolescents who are eligible for music-motor based intervention therapies (i.e. AMMT: Auditory Motor Mapping Therapy). Current and future versions of the assessment are discussed.Published versio

Neonatal azithromycin administration to prevent infant mortality: study protocol for a randomised controlled trial.

IntroductionBiannual mass azithromycin distribution to children aged 1-59 months has been shown to reduce all-cause mortality. Children under 28 days of age were not treated in studies evaluating mass azithromycin distribution for child mortality due to concerns related to infantile hypertrophic pyloric stenosis (IHPS). Here, we report the design of a randomised controlled trial to evaluate the efficacy and safety of administration of a single dose of oral azithromycin during the neonatal period.Methods and analysisThe Nouveaux-nés et Azithromycine: une Innovation dans le Traitement des Enfants (NAITRE) study is a double-masked randomised placebo-controlled trial designed to evaluate the efficacy of a single dose of azithromycin (20 mg/kg) for the prevention of child mortality. Newborns (n=21 712) aged 8-27 days weighing at least 2500 g are 1:1 randomised to a single, directly observed, oral dose of azithromycin or matching placebo. Participants are followed weekly for 3 weeks after treatment to screen for adverse events, including IHPS. The primary outcome is all-cause mortality at the 6-month study visit.Ethics and disseminationThis study was approved by the Institutional Review Boards at the University of California, San Francisco in San Francisco, USA (Protocol #18-25027) and the Comité National d'Ethique pour la Recherche in Ouagadougou, Burkina Faso (Protocol #2018-10-123). The findings of this trial will be presented at local, regional and international meetings and published in open access peer-reviewed journals.Trial registration numberNCT03682653; Pre-results