A Usability Evaluation Framework for

Currently, more than two billions people access the Web for various purposes. The majority are people without programming or modelling background. Part of these people (called end-users) also likes to create their own Web applications to meet their daily needs. Mashup Makers are tools to create such end-user’s Web applications. As such, Mashup Makers could become the dominant environment for end-user development of Web applications. Existing Mashup Makers promise that creating a Web Mashup is very easy and just a matter of a few mouse clicks. However, there is no evidence that this is indeed the case. On the contrary, research has already revealed usability problems with Mashup Makers. Therefore, this thesis concentrates on the usability of Mashup Makers as development environments for Web applications for end-users. Usability is a key issue for the success of software artifacts, and especially if the artifacts are intended for non-technical users. Therefore, we target the achievement of a consolidated approach, model, and framework for the evaluation of the usability of Mashup Makers for end-users. Such a framework will not only allow evaluating the usability of existing Mashup Makers, but it will also provide key issues concerning usability (ie usability impact factors) that developers of Mashup Makers and of other future end-user development tools can take into consideration when developing new tools

Proceedings of the Sixth Danish Human-Computer Interaction Research Symposium.

Proceedings of theSixth Danish Human-Computer Interaction Research Symposium.Aarhus, Denmark, November 15, 200

The impact of computer interface design on Saudi students’ performance on a L2 reading test

A thesis submitted to the University of Bedfordshire in partial fulfillment of the requirements for the degree of Doctor of PhilosophyThis study investigates the effect of testing mode on lower-level Saudi Arabian test-takers’ performance and cognitive processes when taking an L2 reading test on computer compared to its paper-based counterpart from an interface design perspective. An interface was developed and implemented into the computer-based version of the L2 reading test in this study, which was administered to 102 Saudi Arabian University students for quantitative analyses and to an additional eighteen for qualitative analyses. All participants were assessed on the same L2 reading test in two modes on two separate occasions in a within-subject design. Statistical tests such as correlations, group comparisons, and item analyses were employed to investigate test-mode effect on test-takers’ performance whereas test-takers’ concurrent verbalizations were recorded when taking the reading test to investigate their cognitive processes. Strategies found in both modes were compared through their frequency of occurrence. In addition, a qualitative illustration of test-takers cognitive behavior was given to describe the processes when taking a lower-level L2 reading test. A mixed-method approach was adhered to when collecting data consisting of questionnaires think-aloud protocols, and post-experimental interviews as main data collection instruments. Results on test-takers’ performance showed that there was no significant difference between the two modes of testing on overall reading performance, however, item level analyses discovered significant differences on two of the test’s items. Further qualitative investigation into possible interface design related causes for these differences showed no identifiable relationship between test-takers’ performance and the computer-based testing mode. Results of the cognitive processes analyses showed significant differences in three out of the total number of cognitive processes employed by test-takers indicating that test-takers had more difficulties in processing text in the paper-based test than in the computer-based test. Both product and process analyses carried out further provided convincing supporting evidence for the cognitive validity, content validity, and context validity contributing to the construct validity of the computer-based test used in this study

How users read translated web pages: occupational and purpose-based differences

Aquest estudi vol demostrar diferències en els patrons de lectura d'una pàgina web en base a les diferències ocupacionals dels participants i la finalitat de lectura. La recerca es realitza amb 20 participants dividits en dos grups: un grup de lectors professionals, on els participants són professionals de la lectura com traductors, editors i correctors, i un grup de lectura recreativa, on les seves professions no impliquen lectura detallada, com xefs, enginyers i personal militar. Tots els participants van completar quatre tasques dissenyades amb quatre finalitats de lectura: 1) sense objectiu específic, 2) per estudiar el tema, 3) per obtenir informació, i 4) per compartir informació. El text de partida en anglès presenta iOS 7. Per al test es va usar la versió web oficial en coreà, després d'inserir-li cinc tipus d'error segons el model d'avaluació LISA. Es va gravar la pantalla dels participants amb programari específic i es van usar Protocols en Veu Alta (TAP) per als seus informes verbals. L'anàlisi suggereix que els errors de traducció es perceben de diferent manera quan la finalitat lectora i la professió dels lectors varien. El grup de lectura professional va aplicar una lectura lineal i meticulosa, mentre que el grup de lectura recreativa va optar per una lectura circular i no rigorosa. El primer grup va detectar gran quantitat d'errors, mentre que el segon va mostrar un baix percentatge de detecció d'errors. Malgrat això, el grup professional mostra major nivell de tolerància als errors de traducció en el procés de comprensió i el grup recreatiu major nivell de frustració a la falta de comprensió. L'autoritat de la companyia encarregada de la web va exercir gran influència en el nivell de confiança del grup de lectura recreativa.Este estudio investiga diferencias en la lectura de una página web traducida en base a las diferencias ocupacionales de los participantes y la finalidad de lectura. La investigación se realiza con 20 participantes divididos en dos grupos: un grupo de lectores profesionales, cuyos participantes son profesionales de la lectura como traductores, editores y correctores, y un grupo de lectura recreativa, cuyas profesiones no implican lectura detallada, como chefs, ingenieros y personal militar. Todos los participantes completaron cuatro tareas diseñadas con cuatro finalidades de lectura: 1) sin objetivo específico, 2) para estudiar el tema, 3) para obtener información, y 4) para compartir información. Para estudiar como los lectores perciben los errores de traducción y la relación entre patrones de lectura y finalidad lectora, se introdujeron cinco tipos de errores de traducción en una web. Se grabó la pantalla de los participantes con software específico y se usaron Protocolos en Voz Alta (TAP) para sus informes verbales. El análisis sugiere que los errores de traducción se perciben de diferente manera cuando la finalidad lectora y la profesión de los lectores varían. El grupo de lectura profesional aplicó una lectura lineal y concienzuda, mientras que el grupo de lectura recreativa optó por una lectura circular y no rigurosa. El primer grupo detectó gran cantidad de errores, mientras que el segundo mostró un bajo porcentaje de detección de errores. Pese a ello, el grupo profesional muestra mayor nivel de tolerancia a los errores de traducción en el proceso de comprensión y el grupo recreativo mayor nivel de frustración a la falta de comprensión. La autoridad de la compañía encargada de la web ejerció gran influencia en el nivel de confianza del grupo de lectura recreativa.As a part of process-oriented research efforts, the study aimed to test occupational differences and purpose-based differences in reading pattern of a translated web page. The research used two reading groups with ten participants in each: a heavy-reading group, whose participants were reading professionals such as translators, editors and proofreaders, and a light-reading group, whose professions did not involve intensive reading, such as chefs, engineers, and military personnel. The participants in both groups completed four different tasks that were designed to provoke four different reading purposes: 1) reading without a specific task, 2) reading for studying subject matter, 3) reading for retrieving information, and 4) reading for sharing information. In order to learn more about how readers perceive translation errors and to test the relations between the reading patterns and the reading purposes, five different types of translation errors were planted in a web page. The participants’ screen activity was recorded by a screen recording software, and TAP is used for their verbal reports. In-depth analysis of the results suggests that occupations and reading purposes have meaningful impacts on the reading patterns of the translated web page. The heavy-reading group displayed very strict bottom-up approaches, with linear and thorough reading, whereas the light-reading group showed relaxed top-down approaches, with circular and not-thorough reading. The heavy group showed critical attitudes in detecting errors, while the light-reading group showed extremely low error-detection rates, with relaxed attitudes. Surprisingly, in spite of the critical error detection pattern, the tolerance level of translation errors in the process of comprehension was much higher in the heavy-reading group, and the frustration level of incomprehension was much higher in the light-reading group. The authority of the company producing the web page also heavily influenced the trust level of the light-reading group

A study of methods of evaluating multimedia materials for language learning

There is a long tradition of paper based materials evaluation in ELT, but at this juncture, a scarcity of studies on ELT Multimedia (MM) materials evaluation. Such studies as have been undertaken have tended to adopt the perspective of the materials developer rather than the end user. But there have been no developed studies of evaluation methods which could be adopted by potential users. Despite calls being made for systematic evaluation, not many proposals have been developed, and there has been little exploration of potential best practice, or of the 'goodness of fit' between methods and evaluation puposes. This study aims to investigate evaluation methods in order to establish best practice in the evaluation of multimedia CALL applications, with a focus on learners' experience and opinions, and with the aim of enabling potential teacher-users of CALL materials to gauge the suitability of materials for their learners. Chapter one of this thesis provides a rationale for this study and an overview of the background to this research. Chapter two presents a review of literature undertaken in four domains: educational evaluation and research methods; Human Computer Interaction (HCI) usability evaluation methods; ELT materials evaluation; and studies ofCALL materials in use. Chapter three focuses on the design and conduct of the study by explaining how some methods of evaluation were trialled in a pilot study and four were selected for adoption and analysis in the main study. The methods selected were Foeu.r Groups, Rttrospective Protorolr, PLUM and SUM! Questionnaires and Activity Monitonitg. To determine the qualities and limitations of these methods, a set of criteria was developed from the literature on software usability evaluation methods in HCI and a broader literature on educational evaluation and research methods. The four data study chapters (4-7) each discuss one of the chosen methods and descnbe how the method was ,6perationalised in an evaluation of learner responses to multimedia software. The final chapter draws together the discussion of the findings and presents different proposals for best practice. The focus in the discussion of findings is on how the chosen methods performed according to the set of criteria. The findings confirm that focus groups and questionnaires are quick and efficient methods whereas retrospective protocols and activity monitoring provide more detailed and protracted data. Teacher evaluators can be guided by the objectives of their evaluation to explore different combinations ofthese methods. Participants in the pilot were 12 ESL students from the University of Warwick and in the main study 45 Freshman/sophomore students from a university in Pakistan. The materials used to operationalise methods were the EASE CD-ROMS listening to Lectures and Seminar Skills 1: Presentations. The research contributes to the field in undertaking an in-depth and extensive study of evaluation methods applicable to CALL materials, whi~ adopt a leamer-centred perspective, and conform to sound principles within educational evaluation, yet which draw on practice in the field of HCI, since this expertise is so relevant in the rapid development of multimedia materials for use in ELT. Moreover, by developing the composite set of core criteria this study has created a tool which practitioners in the field can use to select most appropriate methods for their particular evaluation purposes

E-learning in higher education: designing for diversity

"This research was conducted to compare methods of e-learning accessibility evaluation that may be applied in a higher education context. Results of ""objective"" accessibility evaluation of e-learning technologies using automated tools were compared to results of ""subjective"" accessibility evaluation with student participants. It was found that objective and subjective accessibility evaluation of e-learning technologies both yield useful, albeit different, information. To further explore subjective accessibility evaluation, results and student perceptions were compared following moderated and unmoderated testing sessions. Neither the efficiency of completing tasks in a sample online course nor the number of accessibility problems detected were deemed significantly affected by the format of the testing session. However, most students preferred to participate in an unmoderated testing session where they felt less self-conscious and as though they could interact more naturally with the technology. Findings from this study point to the importance of considering not only objective accessibility evaluation and accessibility guideline conformance as measures of the accessibility of e-learning technologies, but also the subjective experiences of students as they engage with the technologies. There is also value in taking a holistic approach towards evaluating e-learning accessibility by considering the accessibility of learning outcomes (factoring in the learning context to the evaluation) in addition to the accessibility of individual e-learning technologies. Because accessibility is a variable that is important to all students, and not just students with disabilities, it is critical that institutions of higher education work with a variety of stakeholders to determine not only how best to evaluate e-learning accessibility, but also how to ensure that the results of accessibility evaluation are widely disseminated in a manner that is likely to have a broad impact on enhancing e-learning accessibility for diverse student populations.

Mapping reading strategies in an electronic efl learning environment

Tese (doutorado) - Universidade Federal de Santa Catarina, Centro de Comunicação e Expressão. Curso de Pós-Graduação em Letras/Inglês e Literatura CorrespondenteEsta pesquisa teve como objetivo principal fazer o mapeamento das estratégias de leitura utilizadas por alunos brasileiros de inglês em um ambiente eletrônico instrucional. Duas outras questões também fizeram parte da pesquisa, uma questionando se houve aumento de motivação com o uso do formato hipertextual. A perspectiva dos participantes da pesquisa foi o ponto de partida para tal análise. O outro questionamento girava em torno do aumento ou não da demanda cognitiva sobre os sujeitos devido ao uso da plataforma hipertextual. Doze sujeitos participaram do experimento o qual utilizou um aplicativo versando sobre estratégias básicas de leitura especialmente desenvolvido para esta pesquisa. Os resultados e a análise dos dados foram baseados na adaptação feita por Kauer e colaboradores (1999) do modelo de interação de sete estágios de Norman (1988). Um protocolo verbal, um questionário e as observações da pesquisadora foram os instrumentos utilizados para análise dos dados obtidos. Setenta e um estágios/estratégias foram identificados dentro de um contexto altamente decisório. A análise dos dados sugeriu que de acordo com a percepção dos participantes houve um aumento da motivação. A presente pesquisa buscou suporte para tal percepção em fatores tais como estilo de aprendizagem, objetivos estabelecidos, nível de competência de conteúdo e da interface, dentre outros. Também não foi constatada sobrecarga cognitiva sobre os participantes devido ao uso do formato hipertextual. Como consequência, nenhum efeito colateral significativo, como desorientação ou frustração, ficou evidenciado

A metacognitive feedback scaffolding system for pedagogical apprenticeship

This thesis addresses the issue of how to help staff in Universities learn to give feedback with the main focus on helping teaching assistants (TAs) learn to give feedback while marking programming assignments. The result is an innovative approach which has been implemented in a novel computer support system called McFeSPA. The design of McFeSPA is based on an extensive review of the research literature on feedback. McFeSPA has been developed based on relevant work in educational psychology and Artificial Intelligence in EDucation (AIED) e.g. scaffolding the learner, ideas about andragogy, feedback patterns, research into the nature and quality of feedback and cognitive apprenticeship. McFeSPA draws on work on feedback patterns that have been proposed within the Pedagogical Patterns Project (PPP) to provide guidance on structuring the feedback report given to the student by the TA. The design also draws on the notion of andragogy to support the TA. McFeSPA is the first Intelligent Tutoring System (ITS) that supports adults learning to help students by giving quality feedback. The approach taken is more than a synthesis of these key ideas: the scaffolding framework has been implemented both for the domain of programming and the feedback domain itself; the programming domain has been structured for training TAs to give better feedback and as a framework for the analysis of students’ performance. The construction of feedback was validated by a small group of TAs. The TAs employed McFeSPA in a realistic situation that was supported by McFeSPA which uses scaffolding to support the TA and then fade. The approach to helping TAs become better feedback givers, which is instantiated in McFeSPA, has been validated through an experimental study with a small group of TAs using a triangulation approach. We found that our participants learned differently by using McFeSPA. The evaluation indicates that 1) providing content scaffolding (i.e. detailed feedback about the content using contingent hints) in McFeSPA can help almost all TAs increase their knowledge/understanding of the issues of learning to give feedback; 2) providing metacognitive scaffolding (i.e. each level of detailed feedback in contingent hint, this can also be general pop-up messages in using the system apart from feedback that encourage the participants to give good feedback) in McFeSPA helped all TAs reflect on/rethink their skills in giving feedback; and 3) when the TAs obtained knowledge about giving quality feedback, providing adaptable fading of TAs using McFeSPA allowed the TAs to learn alone without any support