Player experience is an important area of research in digital games. The findings are crucial for the developers, designers and reviewers of games, allowing for the better understanding of player experience whilst playing digital games. Questionnaires are a way to directly measure the reported experiences of players. This approach in games research, however, is challenging for new researchers because of the proliferation of questionnaires available. The problem is knowing which questionnaires are measuring what aspect of experience. This paper sets out the need for positioning the various questionnaires in relation to each other. We list all the current available questionnaires to measure engagement whilst playing digital games. We, therefore, argue that further investigation on these questionnaires is needed to produce better quality questionnaires and reduce confusion amongst player experience researchers

Cairns, P.

Denisova, A.

Nordin, A. I.

City Research Online

              City, University of London Institutional RepositoryCitation: Nordin, A. I., Denisova, A. and Cairns, P. (2014). Too Many Questionnaires: Measuring Player Experience Whilst Playing Digital Games. Paper presented at the The Seventh York Doctoral Symposium on Computer Science and Electronics, 30 Oct 2014, York, UK. This is the published version of the paper. This version of the publication may differ from the final published version. Permanent repository link:  http://openaccess.city.ac.uk/21353/Link to published version: Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.City Research Online:            http://openaccess.city.ac.uk/            publications@city.ac.ukCity Research OnlineToo Many Questionnaires: Measuring PlayerExperience Whilst Playing Digital GamesA. Imran Nordin, Alena Denisova, and Paul CairnsDepartment of Computer Science, University of York, UKAbstract. Player experience is an important area of research in dig-ital games. The findings are crucial for the developers, designers andreviewers of games, allowing for the better understanding of player ex-perience whilst playing digital games. Questionnaires are a way to di-rectly measure the reported experiences of players. This approach ingames research, however, is challenging for new researchers because ofthe proliferation of questionnaires available. The problem is knowingwhich questionnaires are measuring what aspect of experience. This pa-per sets out the need for positioning the various questionnaires in relationto each other. We list all the current available questionnaires to measureengagement whilst playing digital games. We, therefore, argue that fur-ther investigation on these questionnaires is needed to produce betterquality questionnaires and reduce confusion amongst player experienceresearchers.1 Introduction and MotivationTerms such as fun, flow, presence and immersion amongst others have been usedto describe the experience of playing digital games [1,2]. Though there are manyobjective (e.g. physiological assessment, such as heart rate measurements, elec-tromyography (EMG), electrodermal activity (EDA), etc.) and subjective (e.g.interviews and focus groups) ways of assessing player experience, they must allat some point reference the subjective nature of individuated experiences. Ques-tionnaires are a useful research method to directly quantify the subjective playerexperience because they are both easy to deploy and provide a standardised testfor quantifying a particular aspect of experience under consideration [3]. Addi-tionally, questionnaires allow players to express their subjective experience albeitwithin the parameters set by the items of the questionnaire.Questionnaire is a technique that allows participants to convey their thoughtsand feelings within the framework of set questions. These questions act asprompts to the participant – allowing them to consider specific aspects of theirgaming experience. More importantly, the written questionnaire also ensuresthat the same specific aspects are considered by all participants, hence, offeringconsistency and uniformity.However, there are a few drawbacks of using questionnaires to measure playerexperience. Aside from the more obvious problems, such as participants nottaking a questionnaire seriously, there is also a less evident and more profoundproblem – namely the wording of the questions themselves that reduce the facevalidity [3], and equally the way in which (and the scale upon which [4]) theparticipants answer them.As there are many aspects of player experience, there are many questionnairesthat are currently used in order to measure them. Some questionnaires take abroad brush approach looking at all aspects of gaming experience [5,6], otherstake a specific aspect, such as immersion [7] or motivation [8]. In one sense thisproliferation of questionnaires helps researchers to home in on the aspect ofconcern, but at the same time it can be confusing as to whether questionnairesthat purport to measure the same thing (or even apparently different things) arein fact measuring the same thing.This is not to say that there should only be one questionnaire. The varietyin questionnaires is necessary to allow a nuanced focus on different aspects ofgames. But where, for example, different questionnaires claim to be measuringengagement, they ought to produce consistent and correlated results.2 Player Engagement QuestionnairesExisting models of player experience use their own questionnaires to measurethe overall engagement when playing a game based on certain aspects that in-fluence the game enjoyment. A summary of the most widely used questionnairesis presented in Table 1.Such a large number of existing questionnaires poses a challenge for newresearchers, who may not necessarily be familiar with every specific detail ofeach theory. Choosing one of them is therefore often based on their availability –many of these questionnaires are not available publicly, or it may be needlesslychallenging to obtain some of them. So eventually only those ones that are easilyaccessible tend to be used for measuring player experience.Moreover, in order to obtain reliable results, the data needs to be gatheredusing a reliable questionnaire. Unfortunately, some of existing questionnaires arenot statistically validated, and are eventually avoided as they are presumed tonot be trustworthy.Gaming Engagement Questionnaire (GEQ) [5] and Immersive ExperienceQuestionnaire (IEQ) [7], are prominent examples of questionnaires set up in asimilar way in order to evaluate player experience. Amongst other questionnairesreviewed in table 1, these two are available publicly and have more similaritiesbetween them, than the rest of them. The GEQ was initially developed to assessthe impact of deep engagement in violent video games. The questionnaire itselfconsists of 19 positively worded questions answered on a five-point Likert scale:the higher the score that the user gives for each question, the more engaged theyare deemed to be. The formulation of the questionnaire puts engagement ona single dimension that ranges up from immersion to flow. This questionnaire,however, has relatively little empirical validation that has been undertaken toestablish the reliability of the questionnaire in part because of its (relatively)recent introduction to the field.Questionnaire ComponentsFlow Questionnaire [9]Clear goalsHigh concentrationReduced self-consciousnessDistorted sense of timeDirect and immediate feedbackBalance between ability level and challengeA sense of personal controlIntrinsically rewarding activityPresence Questionnaire [10]Control factorSensory factorDistractionRealism factorImmersive ExperienceQuestionnaire (IEQ) [7]Emotional involvementCognitive involvementReal world dissociationChallengeControlGameFlow Questionnaire [11]ConcentrationA sense of challengePlayer skillsControlClear goalsFeedbackSocial interactionImmersionGame EngagementQuestionnaire (GEQ) [5]AbsorptionFlowPresenceImmersionPlayer Experience of NeedsSatisfaction (PENS) [12]CompetenceAutonomyRelatednessPresence (Immersion)Social Presence in GamingQuestionnaire (SPGQ) [13]Psychological involvement (empathy)Psychological involvement (negative feelings)Behavioural engagementTable 1. Questionnaires measuring user engagement whilst playing digital games.On the other hand, the IEQ is a widely used questionnaire in determiningthe levels of immersion experienced by players. It has been tested much moreempirically across a far-reaching array of different scenarios and game types, forexample [14,15,16]. Similarly to the GEQ, it uses five-point Likert scale ques-tions for measuring player experience, but is specifically focused on the notionof immersion when playing games. Unlike the GEQ, the IEQ uses both posi-tive statements and negative statements. Each positively worded statement hasa negatively worded counterpart, adding an additional layer of accuracy. Theoverall score is composed of the summary of the results from the positive ques-tions, as well as the inverted results of the negative ones. The development ofthe IEQ also suggested that there are five factors underlying immersion, but inpractice immersion is also treated as a single dimension with the factors lendingaspect for interpretation of results.3 Proposed Design for the Future WorkAs mentioned earlier, several challenges arise, particularly for the new playerresearchers, when investigating player experience while playing digital games.Usability also appears to be an issue that affects questionnaires [17].In this position paper, we, therefore, are proposing our idea to investigate twoof the main questionnaires used for measuring the experience of playing digitalgames, namely the Immersive Experience Questionnaire (IEQ) [7] and the GameEngagement Questionnaire (GEQ) [5] to test whether they are both measuringthe same experience. Given the similar emphasis of these two questionnaires, itseems reasonable that they are in fact addressing the same underlying aspect ofplayer experience. However, the question is whether this is in fact the case.Work is underway to test the hypothesis that the IEQ scores are correlatedto the GEQ scores. The design of the experiment will involve the manipulationof player experience by using music, as in work of Sanders et al. [16]. Playerexperience will be measured using both questionnaires, and the results obtainedusing each questionnaire will be compared. We believe the results will provide aninsight to address any potential problem of having multiple measurement tools.References1. R. Bernhaupt, M. Eckschlager, and M. Tscheligi, “Methods for evaluating games:how to measure usability and user experience in games?” in Proceedings of the in-ternational conference on Advances in computer entertainment technology. ACM,2007, pp. 309–310.2. E. Brown and P. Cairns, “A grounded investigation of game immersion,” in CHI ’04extended abstracts on Human factors in computing systems, ser. CHI EA ’04, 2004,pp. 1297–1300. [Online]. Available: http://doi.acm.org/10.1145/985921.9860483. A. Adams and A. L. Cox, Research methods for human-computer interaction.Cambridge, UK: Cambridge University Press, 2008, ch. Questionnaires, in-depthinterviews and focus groups.4. E. A. Greenleaf, “Improving rating scale measures by detecting and correcting biascomponents in some response styles.” Journal of Marketing Research, 1992.5. J. H. Brockmyer, C. M. Fox, K. A. Curtiss, E. McBroom, K. M. Burkhart, and J. N.Pidruzny, “The development of the game engagement questionnaire: A measure ofengagement in video game-playing,” Journal of Experimental Social Psychology,vol. 45, no. 4, pp. 624–634, 2009.6. H. Qin, P. Rau, and G. Salvendy, “Measuring player immersion in the computergame narrative,” Intl. Journal of Human–Computer Interaction, vol. 25, no. 2, pp.107–133, 2009.7. C. Jennett, A. L. Cox, P. Cairns, S. Dhoparee, A. Epps, T. Tijs, and A. Walton,“Measuring and defining the experience of immersion in games,” InternationalJournal of Human-Computer Studies, vol. 66, no. 9, pp. 641 – 661, 2008. [Online].Available: http://www.sciencedirect.com/science/article/pii/S10715819080004998. J. P. Gee, “What video games have to teach us about learning and literacy,” Com-puters in Entertainment (CIE) - Theoretical and Practical Computer Applicationsin Entertainment, vol. 1(1), 2002.9. M. Csikszentmihalyi, The flow experience and its significance for human psychol-ogy. Cambridge University Press, Cambridge, UK, 1998, ch. In: Optimal experi-ence: psychological studies of flow in consciousness, pp. 15–35.10. B. Witmer and M. Singer, “Measuring presence in virtual environments: A presencequestionnaire,” Presence, vol. 7, no. 3, pp. 225–240, 1998.11. P. Sweetser and P. Wyeth, “Gameflow: A model for evaluating player enjoymentin games,” ACM Computers in Entertainment, vol. 3 (3), 2005.12. R. M. Ryan, C. S. Rigby, and A. Przybylski, “The motivational pull of video games:A self-determination theory approach,” Motivation and Emotion, vol. 30, no. 4, pp.344–360, 2006.13. Y. de Kort, W. IJsselsteijn, and K. Poels, “Digital games as social presence tech-nology: Development of the social presence in gaming questionnaire (spgq),” inProceedings of PRESENCE 2007: The 10th International Workshop on Presence,2007, pp. 195–203.14. A. Cox, P. Cairns, P. Shah, and M. Carroll, “Not doing but thinking: the roleof challenge in the gaming experience,” in Proceedings of the 2012 ACM annualconference on Human Factors in Computing Systems. ACM, 2012, pp. 79–88.15. M. Thompson, A. Nordin, and P. Cairns, “Effect of touch-screen size on gameimmersion,” BCS HCI 2012, 2012.16. T. Sanders and P. Cairns, “Time perception, immersion and music in videogames,”in Proceedings of the 24th BCS Interaction Specialist Group Conference. BritishComputer Society, 2010, pp. 160–167.17. P. Cairns, “A commentary on short questionnaires for assessing usability,” Inter-acting with Computers, p. iwt019, 2013.

Too Many Questionnaires: Measuring Player Experience Whilst Playing Digital Games

http://openaccess.city.ac.uk/21353/1/YDS_2014%28Final%29.pdf

Too Many Questionnaires: Measuring Player Experience Whilst Playing Digital Games

Abstract

Similar works

Full text

Available Versions

City Research Online