Ethnographic Observation in the Usability Evaluation of Computer Assisted Data Collection

The impact of computer aided interview on data quality has been systematically evaluated by statistical institutes in diversecountries. This work aims to show the importance of field observation in the evaluation of computer assisted data collection.The method shown here was proposed to be applied to the context of IBGE, the Institute of Brazil central administration thatperforms the Census. This method (Scenario and Task Based Interviews - STBI) was designed to be implemented with theparticipation of interviewers who use personal digital assistants to perform statistical data collection for demographicresearch. Authors analyzed the usability of the application developed to support Continuous National Household SampleSurvey (Continuous PNAD). The proposed method represented a mix of four approaches to evaluation. This work gives aspecial emphasis to describe the part of the method that concerns the application of ethnographic observations and itsconclusions

User evaluation outside the lab: the trial of Físchlár-News

A user study of Físchlár-News system was conducted in Spring 2004 with 16 users, each user using the system for a 1-month period. Físchlár-News is an experimental online news archive that incorporates various automatic content-based video indexing techniques and a news story recommender algorithm to process and index the daily 9 o’clock broadcast news from TV and allows its users to browse, search, be recommended, and play news stories on a conventional web browser. Pre and post-trial questionnaires, interaction logging and incident diary methods collected both qualitative and quantitative usage data during the trial period. While the details of the findings from this evaluation is reported elsewhere, in this paper we report the details of the methodology taken and our experience of conducting this evaluation

CAL evaluation: Future directions

Formal, experimental methods have proved increasingly difficult to implement, and lack the capacity to generate detailed results when evaluating the impact of CAL on teaching and learning. The rigid nature of experimental design restricts the scope of investigations and the conditions in which studies can be conducted It has also consistently failed to account for all influences on learning. In innovative CAL environments, practical and theoretical development depends on the ability fully to investigate the wide range of such influences. Over the past five years, a customizable evaluation framework has been developed specifically for CAL research. The conceptual approach is defined as Situated Evaluation of CAL (SECAL), and the primary focus is on quality of learning outcomes. Two important principles underpin this development. First, the widely accepted need to evaluate in authentic contexts includes examination of the combined effects of CAL with other resources and influential aspects of the learning environment. Secondly, evaluation design is based on a critical approach and qualitative, case‐based research. Positive outcomes from applications of SECAL include the easy satisfaction of practical and situation‐specific requirements and the relatively low cost of evaluation studies. Although there is little scope to produce generalizable results in the short term, the difficulty of doing so in experimental studies suggests that this objective is difficult to achieve in educational research. A more realistic, longer‐term aim is the development of grounded theory based on common findings from individual cases

Collective awareness platforms and digital social innovation mediating consensus seeking in problem situations

In this paper we show the results of our studies carried out in the framework of the European Project SciCafe2.0 in the area of Participatory Engagement models. We present a methodological approach built on participative engagements models and holistic framework for problem situation clarification and solution impacts assessment. Several online platforms for social engagement have been analysed to extract the main patterns of participative engagement. We present our own experiments through the SciCafe2.0 Platform and our insights from requirements elicitation

EVALUATION OF CURRICULUM 2013 IMPLEMENTATION IN ELEMENTARY SCHOOLS

This research aims to contribute several ideas towards the implementation of the 2013 curriculum for elementary school teachers in Palembang. This descriptive qualitative research used interviews and questionnaires as data collection techniques. The results of the 2013 Curriculum implementation evaluation research were related to the constraints of teachers in implementing the 2013 curriculum. The benefits of this research are the data analyzed with arguments descriptively. The results of the study found obstacles and obstacles to the implementation of the 2013 curriculum for schools and teachers towards the mastery of science and technology, the creation of instructional media, evaluation of learning and the limitations of facilities and infrastructure so that the implementation of the 2013 curriculum could only be carried out in 2018/2019. Solutions are needed in the form of providing adequate training and workshops for teachers and education personnel. Provide the necessary facilities and infrastructure and develop technology-based assessment systems

Interdisciplinary systematic review: does alignment between system and design shape adoption and use of barcode medication administration technology?

BACKGROUND: In order to reduce safety risks associated with medication administrations, technologies such as barcode medication administration (BCMA) are increasingly used. Examining how human factors influence adoption and usability of this technology can potentially highlight areas for improvement in design and implementation. OBJECTIVE: To describe how human factors related determinants for BCMA have been researched and reported by healthcare and human-computer interaction disciplines. DATA SOURCES: The Cumulative Index of Nursing, and Allied Health Literature, PubMed, OVID MEDLINE and Google Scholar. STUDY ELIGIBILITY CRITERIA: Primary research published from April 2000 to April 2020, search terms developed to identity different disciplinary research perspectives that examined BCMA use, used a human factors lens and were published in English. SYNTHESIS METHODS: Computerised systematic searches were conducted in four databases. Eligible papers were systematically analysed for themes. Themes were discussed with a second reviewer and supervisors to ensure they were representative of content. RESULTS: Of 3707 papers screened, 11 were included. Studies did not fit neatly into a clinical or human-computer interaction perspective but instead uncovered a range of overlapping narratives, demonstrating consensus on the key themes despite differing research approaches. Prevalent themes were misaligned design and workflow, adaptation and workarounds, mediating factors, safety, users' perceptions and design and usability. Inadequate design frequently led to workarounds, which jeopardised safety. Reported mediating factors included clarity of user needs, pre/post implementation evaluations, analysis of existing workarounds and appropriate technology, infrastructure and staffing. LIMITATIONS: Most studies were relatively small and qualitative, making it difficult to generalise findings. CONCLUSION: Evaluating interdisciplinary perspectives including human factors approaches identified similar and complementary enablers and barriers to successful technology use. Often, mediating factors were developed to compensate for unsuitable design; a collaborative approach between system designer and end users is necessary for BCMA to achieve its true safety potential

Beyond Automated Assessment: Building Metacognitive Awareness in Novice Programmers in CS1

The primary task of learning to program in introductory computer science courses (CS1) cognitively overloads novices and must be better supported. Several recent studies have attempted to address this problem by understanding the role of metacognitive awareness in novices learning programming. These studies have focused on teaching metacognitive awareness to students by helping them understand the six stages of learning so students can know where they are in the problem-solving process, but these approaches are not scalable. One way to address scalability is to implement features in an automated assessment tool (AAT) that build metacognitive awareness in novice programmers. Currently, AATs that provide feedback messages to students can be said to implement the fifth and sixth learning stages integral to metacognitive awareness: implement solution (compilation) and evaluate implemented solution (test cases). The computer science education (CSed) community is actively engaged in research on the efficacy of compile error messages (CEMs) and how best to enhance them to maximize student learning and it is currently heavily disputed whether or not enhanced compile error messages (ECEMs) in AATs actually improve student learning. The discussion on the effectiveness of ECEMs in AATs remains focused on only one learning stage critical to metacognitive awareness in novices: implement solution. This research carries out an ethnomethodologically-informed study of CS1 students via think-aloud studies and interviews in order to propose a framework for designing an AAT that builds metacognitive awareness by supporting novices through all six stages of learning. The results of this study provide two important contributions. The first is the confirmation that ECEMs that are designed from a human-factors approach are more helpful for students than standard compiler error messages. The second important contribution is that the results from the observations and post-assessment interviews revealed the difficulties novice programmers often face to developing metacognitive awareness when using an AAT. Understanding these barriers revealed concrete ways to help novice programmers through all six stages of the problem-solving process. This was presented above as a framework of features, which when implemented properly, provides a scalable way to implicitly produce metacognitive awareness in novice programmers

ECSCW 2013 Adjunct Proceedings The 13th European Conference on Computer Supported Cooperative Work 21 - 25. September 2013, Paphos, Cyprus

This volume presents the adjunct proceedings of ECSCW 2013.While the proceedings published by Springer Verlag contains the core of the technical program, namely the full papers, the adjunct proceedings includes contributions on work in progress, workshops and master classes, demos and videos, the doctoral colloquium, and keynotes, thus indicating what our field may become in the future