Toward a document evaluation methodology: What does research tell us about the validity and reliability of evaluation methods?

Although the usefulness of evaluating documents has become generally accepted among communication professionals, the supporting research that puts evaluation practices empirically to the test is only beginning to emerge. This article presents an overview of the available research on troubleshooting evaluation methods. Four lines of research are distinguished concerning the validity of evaluation methods, sample composition, sample size, and the implementation of evaluation results during revisio

Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

To intervene or not to intervene: An investigation of three think-aloud protocols in usability testing

This paper presents the results of a study investigating the use of three think-aloud methods in website usability testing: the concurrent think-aloud, the speech-communication, and the active intervention methods. These three methods were compared through an evaluation of a library website, which involved four points of comparison: overall task performance, test participants’ experiences, the quantity and quality of usability problems discovered, and the cost of employing the methods. Data were collected from 60 individuals, with 20 participants allocated to each testing method, who were asked to complete a set of nine experimental tasks. The results of the study revealed that the three variations enabled the identification of a similar number of usability problems and types. However, the active intervention method was found to cause some reactivity, modifying participants’ interaction with the interface, and negatively affecting their feelings towards the evaluator. The active intervention method also required much greater investment than did the other two methods in terms of evaluators' time

Rethinking thinking aloud: A comparison of three think-aloud protocols

This paper presents the results of a study that compared three think-aloud methods: concurrent think-aloud, retrospective think-aloud, and a hybrid method. The three methods were compared through an evaluation of a library website, which involved four points of comparison: task performance, participants' experiences, usability problems discovered, and the cost of employing the methods. The results revealed that the concurrent method outperformed both the retrospective and the hybrid methods in facilitating successful usability testing. It detected higher numbers of usability problems than the retrospective method, and produced output comparable to that of the hybrid method. The method received average to positive ratings from its users, and no reactivity was observed. Lastly, this method required much less time on the evaluator's part than did the other two methods, which involved double the testing and analysis time

Are two pairs of eyes better than one? A comparison of concurrent think-aloud and co-participation methods in usability testing

This paper presents the results of a study that aimed to compare the traditional concurrent think-aloud protocol with the co-participation method to determine the benefit of adding an additional participant to the testing session. The two methods were compared through an evaluation of a library website, and their relative validity and utility were measured using four points of comparison: overall task performance, test participants’ experiences, quantity and quality of problems discovered, and the cost of employing each method. The results of the study show significant differences between the performances of the two types of testing methods. The co-participation method was evaluated more positively by users and led to the detection of a greater number of minor usability problems. This method, however, was found to require a greater investment of time and effort on the part of the evaluator in comparison to the classical method. This study found no difference between the methods in terms of task performance

When to ask participants to think aloud:A comparative study of concurrent and retrospective think-aloud methods

This paper presents the results of a study that compared two think-aloud usability testing methods: the concurrent think-aloud and the retrospective think-aloud methods. Data from task performance, testing experience, and usability problems were collected from 30 participants equally distributed between the two think-aloud conditions. The results suggest that while the thinking aloud method had no impact on task performance and testing experience, participants using the concurrent think-aloud method reported a larger number of problems with the test interface than participants using the retrospective think-aloud method. These findings suggest a reason for preferring the concurrent think-aloud method to the retrospective one

Role of context in usability evaluations: A review

Usability is often defined as the ability of a system to carry out specific tasks by specific users in a specific context. Usability evaluation involves testing the system for its expected usability. Usability testing is performed in natural environment (field) or artificial environment (laboratory). The result of usability evaluation is affected by the environment in which it is carried out. Previous studies have focused only on the physical environment (lab and field) effect on the results but rarely focused on the effect of social environment (people present during testing). Therefore, this study aims to review how important it is to take context into account during usability evaluation. Context is explored through the theory of behaviour settings, according to which behaviour of individuals is strongly influenced by the physical as well as the social environment in which they function. The result of this review indicates that the physical and social context plays a substantial role in usability evaluations. Further, it also suggests that the usability evaluation model should encompass context as an important component in the framework.Comment: 10 pages, 2 tables, ACIJ- Vol3.No.2 March 201

Deceiving user interface elements - user study on dark patterns

Large portion of websites and applications want to influence users’ behaviour. Persuasive designs can guide users towards choices they desire but there are also designs that don’t have the users’ best interests in mind. These design choices are called dark patterns and they aim to manipulate users towards choices that don’t serve the users’ best interests. At the same time, many online service providers want to offer best user experience possible for their users. Since the purposes of these two concepts are somewhat contradictory it is worth examining how dark patterns affect user experience. Previous research has studied users’ perspective on dark patterns and it has been found that dark patterns can cause negative effects such as annoyance. However, no research has examined dark patterns’ effects specifically to user experience. This research examines the effects of dark patterns trough user experience model. This research was conducted as qualitative research. First, a literature review presents the findings of previous research on dark patterns and user experience. The used data collection method in this research was concurrent think-aloud technique. This specific technique has not been used before to study users’ perceptions on dark patterns. The think-aloud technique was used in user studies where participants performed tasks and verbalized their thoughts at the same time. The participants performed five tasks that included different dark patterns. The results of this research support the findings of previous research. It was found that that dark patterns can affect user experience negatively. This research indicates that dark patterns affect mostly the pragmatic quality of user experience. Especially the lack of autonomy reduces pragmatic quality. These effects cause negative consequences in emotional and behavioural level. However, the effects vary between dark pattern types and strategies. Also, individual differences on users’ expectations and standards affect how dark patterns are perceived. This research suggests that prevalence of dark patterns can make users used to dark patterns and thus reduce the effects on user experience.Suuri osa nettisivuista ja sovelluksista haluaa vaikuttaa käyttäjien toimintaan. Suostuttelevat suunnittelumallit voivat ohjata käyttäjiä kohti heidän haluamia valintoja, mutta on olemassa myös malleja, jotka eivät ole käyttäjien etujen mukaisia. Näitä suunnittelumalleja kutsutaan tummiksi kuvioiksi ja ne pyrkivät manipuloimaan käyttäjiä kohti valintoja, jotka eivät ole heidän etujensa mukaisia. Samaan aikaan monet verkossa toimivat palveluntarjoajat haluavat tarjota käyttäjilleen parhaan mahdollisen käyttäjäkokemuksen. Koska näiden kahden konseptin tarkoitukset ovat hieman ristiriitaisia on syytä tarkastella miten tummat kuviot vaikuttavat käyttäjäkokemukseen. Aikaisemmat tutkimukset ovat tutkineet tummia kuvioita käyttäjien näkökulmasta ja on löydetty, että tummat kuviot voivat aiheuttaa negatiivisia vaikutuksia kuten ärsytystä. Ei kuitenkaan ole tehty tutkimusta, joka tutkisi tummien kuvioiden vaikutuksia nimenomaan käyttäjäkokemukseen. Tämä tutkimus tutkii tummien kuvioiden vaikutuksia käyttäjäkokemusmallin läpi. Tämä tutkimus toteutettiin laadullisena tutkimuksena. Aluksi, kirjallisuuskatsaus esittelee aikaisempien tutkimuksien löydöksiä tummista kuvioista ja käyttäjäkokemuksesta. Tämän tutkimuksen käytetty tiedonkeruumenetelmä oli samanaikaisen ääneen ajattelun tekniikka. Tätä tiettyä tekniikkaa ei ole aiemmin käytetty tutkittaessa käyttäjien käsityksiä tummista kuvioista. Ääneen ajattelun tekniikkaa käytettiin käyttäjätutkimuksissa, joissa osallistujat suorittivat tehtäviä ja samalla sanallistivat ajatuksiaan. Osallistujat suorittivat viisi tehtävää, jotka sisälsivät erilaisia tummia kuvioita. Tämän tutkimuksen tulokset tukevat aikaisempia tutkimuksia. Tutkimuksessa todettiin, että tummat kuviot voivat vaikuttaa käyttäjäkokemukseen negatiivisesti. Tämä tutkimus osoittaa, että tummat kuviot vaikuttavat enimmäkseen käyttäjäkokemuksen pragmaattiseen laatuun. Erityisesti autonomian puute alentaa pragmaattista laatua. Nämä vaikutukset aiheuttavat negatiivisia seurauksia niin emotionaalisella kuin käyttäytymisenkin tasolla. Vaikutukset kuitenkin vaihtelevat tummien kuvioiden tyyppien ja strategioiden välillä. Myös käyttäjien yksilölliset eroavaisuudet odotuksissa ja standardeissa vaikuttavat siihen miten tummat kuviot koetaan. Tämä tutkimus ehdottaa, että tummien kuvioiden yleisyys voi saada käyttäjät tottumaan niihin ja näin vähentää kuvioiden vaikutusta käyttäjäkokemukseen