Towards a new ITU-T recommendation for subjective methods evaluating gaming QoE

This paper reports on activities in Study Group 12 of the International Telecommunication Union (ITU-T SG12) to define a new Recommendation on subjective evaluation methods for gaming Quality of Experience (QoE). It first resumes the structure and content of the current draft which has been proposed to ITU-T SG12 in September 2014 and then critically discusses potential gaming content and evaluation methods for inclusion into the upcoming Recommendation. The aim is to start a discussion amongst experts on potential evaluation methods and their limitations, before finalizing a Recommendation. Such a recommendation might in the end be applied by non -expert users, hence wrong decisions in the evaluation design could negatively affect gaming QoE throughout the evaluation

Methodologies for evaluating the playability of mobile games:systematic literature review

Tiivistelmä. The gaming industry has been growing rapidly during the past years due to the interest of the new generations in mobile gaming. To deliver a great experience for the gamers, it is required for the gaming companies to produce games that are challenging but at the same time easy to play. To achieve this, it is required to understand the factors that affect the gaming experience. Playability is a term that is used to understand the usability of a game and its experience. The purpose of this thesis was to understand what is known related to the playability of mobile games and to identify the methodologies that are used by the community to evaluate this phenomenon. To find the answers to these questions, it was performed a systematic literature review (SLR) using the databases Scopus, IEEE Xplore, and Web of Science. After conducting the SLR, 1,390 studies related to the playability of mobile games were found from which 27 were identified as primary studies of this research. From the data collected from the primary studies, there were identified 12 different methodologies that are used for evaluating the playability of mobile games. The methodologies that are most suitable to assess the playability of mobile games are heuristic evaluation and playtesting. Other methodologies can be used for evaluating the playability of mobile games, but they must include a set of heuristics that allows evaluating the playability. The limitations of the research were mentioned, and it was proposed topics for future research of this field. The contribution of this thesis is the summarizing of the current methodologies that are used to understand and evaluate the playability of mobile games. The results of this thesis are valuable for game developers, game designers, and game usability practitioners

Flow Experience as a Quality Measure in Evaluating Physically Activating Collaborative Serious Games

The measurement of the subjective playing experience is important part of the game development process. The enjoyment level that a serious game offers is a key factor in determining whether a player will be engaged in the gameplay and achieve the objectives of the game. In this paper we report the results of a game design process in which two prototypes of a collaborative exergame were studied. The main aim of the paper is to explore to what extend the measurement of flow experience can facilitate the game evaluation and design process. Alltogether 102 junior high school students participated in two user experience studies and played collaborative exergames designed to teach soft skills. Playing experience was measured with a flow questionnaire, playing behavior was observed and some of the players were interviewed. The results showed that flow experience can be used to evaluate the overall quality of the gameplay and it provides a structured approach to consider the quality of the game. However, flow does not provide detailed information about the shortages of the game and thus complementary methods is needed to identify the causes. The results also indicated that flow experience was independent of gender that supports its use in quality measurement

Benchmarking framework for serious games in project management

Abstract. Serious games are games that are developed with educational goals instead of entertainment at the core. There have been several different ways serious games have been assessed, but often these targets a specific part of the game. This thesis aims to construct a benchmarking framework for evaluating serious games. The evaluated games come from the project management domain, and as such, the framework is directed to said area. The evaluations of the games are done through action research orientation, with benchmarking as the method to evaluate games and the framework itself. The construction of the framework is based on an exploratory literature review. The gathered framework consists of the following evaluation areas: game session, mechanics, playability and flow, project management and costs. The evaluation of serious games can not be based on the game elements alone. However, consideration must be given to the materials related to the game and the game’s fit to the learning event. Often the marketing for serious games was focused on the subject materials of the game, while the mechanics and playability of the game were left to lower attention. More research is needed on the usability of the framework and the evaluations on the current state of project management serious games. The developed framework’s core could also be tested against different serious game domains.Projektijohtamisen vakavien pelien benchmarkkaus arviointimalli. Tiivistelmä. Vakavia pelejä kehittäessä oppimistavoitteet ovat keskiössä viihdearvon sijasta. Ajan saatossa on muotoiltu lukuisia eri tapoja arvioida vakavia pelejä, mutta usein nämä arviot kohdistuvat yksittäiseen osa-alueeseen pelissä. Tämän diplomityön tavoitteena on muodostaa benchmarkkaus arviointimalli vakavien pelien arvioimiseksi. Arvioitavat pelit kerätään projektijohtamisen alueelta. Tämän vuoksi arviointimallin opetuspelien aiheeseen liittyvät osa-alueet kohdistuvat projektijohtamisen osa-alueelle. Arvioinneissa hyödynnetään toimintatutkimuksen menetelmiä täydentämään benchmarkkauksen toimintatapoja. Tutkimuksessa arviointi kohdistuu niin peleihin kuin malliin itseensä. Pelien benchmarkkaus arviointimalli koostettiin tutkivan kirjallisuuskatsauksen pohjilta. Arviointimalli kohdistuu viidestä osa-alueesta: pelitapahtuma, pelin mekaniikka, pelattavuus ja pelin luontevuus, projektijohtaminen ja kustannukset. Pelien arviointi ei voi pohjata pelkästään pelin osa-alueisiin, vaan on välttämätöntä tarkastella myös pelin soveltuvuutta pelitapahtumaan ja käytettävissä oleviin resursseihin. Usein vakavien pelien markkinointi keskittyy pelin opetettavan aiheen ympärille, jättäen pelin mekaniikan ja pelattavuuden toissijaisiksi. Kehitettyä benchmarkkaus arviointimallia tulisi arvioida lisää jatkotutkimuksissa. Lisäksi arviointimallia voisi soveltaa myös muilla aloilla, kuin projektijohtamisessa

A survey of Game Usability Practices in Northern European Game Companies

In today’s heavily competitive game market, it has become very important to make a game that stands out from the other games. Even though the game design might not be very unique, a game can still offer a better playing experience by having a better usability than a similar game with worse usability. The human-computer interaction research could provide game companies with the practical tools and methods needed to improve the game usability. This study investigates the views of the Northern European game companies on the concept of game usability, the extent to which the game companies utilize usability methods, and the methods they use. Data from two surveys conducted in Northern European game companies were analyzed; the unit of analysis of the survey data was company. The respondents – professionals with different roles in game development – regarded usability as a broad concept and rated it as having high importance in games. Game companies used multitude of usability methods, but these methods and their usage have not stabilized. Surprisingly, only a few game companies used heuristic evaluation, mostly employing in-house game usability heuristics

Evaluating 2D and 3D serious games: The significance of student-player characteristics

Τα σοβαρά παιχνίδια (ψηφιακά εκπαιδευτικά παιχνίδια μαθησιακού σκοπού)  κερδίζουν ολοένα και περισσότερο το ενδιαφέρον πολλών μελετητών της μαθησιακής διαδικασίας όλων των εκπαιδευτικών επιπέδων. Σημαντικές πτυχές της εφαρμογής τους στην εκπαίδευση σχετίζονται με τα χαρακτηριστικά τους, τη συνάφειά τους με την παιδαγωγική και τις απόψεις των μαθητών. Αυτή η ποσοτική έρευνα επικεντρώθηκε στα χαρακτηριστικά των φοιτητών παιδαγωγικών τμημάτων (φύλο, επιστημονικό υπόβαθρο, προηγούμενη εμπειρία με παιχνίδια και δεξιότητες ΤΠΕ) και στους παράγοντες που επηρεάζουν και διαμορφώνουν την μαθησιακή εμπειρία τους όταν παίζουν σοβαρά παιχνίδια δύο (2D) και τριών (3D) διαστάσεων, όπως είναι η εμβύθιση, η απόλαυση, η υποκειμενική αντίληψη των μαθητών σχετικά με τη χρησιμότητά τους για τη βελτίωση της γνώσης, την επάρκεια της αφήγησης, τον ρεαλισμό, την ανατροφοδότηση, την οπτικοακουστική επάρκεια, τη συνάφειά τους με τα προσωπικά ενδιαφέροντά τους, τη σαφήνεια του στόχου τους, την ευκολία χρήσης τους, την καταλληλότητα του μαθησιακού υλικού και τα κίνητρα. Συνολικά 542 φοιτητές συμμετείχαν στη μελέτη. Σειρά μη παραμετρικών αναλύσεων (Mann-Whitney U tests και Kruskal-Wallis H tests) αποκάλυψε ότι οι φοιτητές προτίμησαν το 2D παιχνίδι. Το φύλο, το επιστημονικό υπόβαθρο και οι δεξιότητες στις ΤΠΕ δεν είχαν αντίκτυπο στις απόψεις τους. Από την άλλη πλευρά, οι φοιτητές με υψηλή ικανότητα στη χρήση των παιχνιδιών φαίνεται να έχουν καλύτερη μαθησιακή εμπειρία, καθώς οι βαθμολογίες τους στους περισσότερους παράγοντες ήταν υψηλότερες σε σύγκριση με τους φοιτητές με χαμηλότερη ικανότητα. Αυτά τα αποτελέσματα μας οδήγησαν να προσανατολιστούμε στην εξέταση άλλων πτυχών/δεξιοτήτων των φοιτητών πέρα από τα συνήθη δημογραφικά στοιχεία, όπως είναι η αυτορρύθμιση, η χωρική ικανότητα και η νοητική περιστροφή και να ερευνήσουμε τη δυνατότητα των σοβαρών παιχνιδιών να βελτιώσουν αυτές τις δεξιότητες.Serious games are gaining ground in all educational levels. Important aspects of their implementation in education are related to their features, their relevance to pedagogy and the learners' views. This quantitative research focused on characteristics of pedagogical departments’ students (gender, scientific background, prior gaming and ICT skills) and factors are considered to affect and shape their learning experience when playing 2D and 3D serious games (immersion, enjoyment, perceived usefulness-knowledge improvement, perceived narratives' adequacy, perceived realism, perceived feedback's adequacy, perceived audiovisual adequacy, perceived relevance to personal interests, perceived goal's clarity, perceived ease of use, adequacy of the learning material and motivation). A total of 542 university students participated in the study. The data analysis revealed that they preferred the 2D game. Their gender, scientific background and ICT skills did not have an impact on their views. On the other hand, students with high game-playing competence seem to have a better learning experience, as their scores in most of the factors were higher compared to the ones with lower game-playing competence. Those results led us to consider other aspects of learners beyond ordinary demographics such as self-regulation, spatial cognition and mental rotation and to examine the evidence that serious games can improve these skills

Advancing performability in playable media : a simulation-based interface as a dynamic score

When designing playable media with non-game orientation, alternative play scenarios to gameplay scenarios must be accompanied by alternative mechanics to game mechanics. Problems of designing playable media with non-game orientation are stated as the problems of designing a platform for creative explorations and creative expressions. For such design problems, two requirements are articulated: 1) play state transitions must be dynamic in non-trivial ways in order to achieve a significant level of engagement, and 2) pathways for players’ experience from exploration to expression must be provided. The transformative pathway from creative exploration to creative expression is analogous to pathways for game players’ skill acquisition in gameplay. The paper first describes a concept of simulation-based interface, and then binds that concept with the concept of dynamic score. The former partially accounts for the first requirement, the latter the second requirement. The paper describes the prototype and realization of the two concepts’ binding. “Score” is here defined as a representation of cue organization through a transmodal abstraction. A simulation based interface is presented with swarm mechanics and its function as a dynamic score is demonstrated with an interactive musical composition and performance

User perception of gaming element effectiveness in a corporate learning application

This Conversion Masters in Information Technology thesis gathered users' perceptions about eight gaming elements to determine their effectiveness on aspects of playability, enjoyment and intrinsic motivation needed in a gamified corporate learning application. The study focused on user opinions about a Progress Bar, Individual Leaderboard, Departmental Leaderboard, Timer, In-Game Currency, Badges, Storyline/Theme and Avatar. A gamification application containing these gaming elements was designed and developed to make the evaluation. The application entailed users learning four Information Technology Infrastructure Library (ITIL) processes needed to manage an information technology department in a telecommunications company. The application design process considered the business goals, rules, target behaviours, time limits, rewards, feedback, levels, storytelling, interest, aesthetics, replay or do-overs, user types, activity cycles, fun mechanisms and development tools needed to create a coherent, addictive, engaging and fun user experience. Player types were determined using the Brainhex online survey. Federoff's Game Playability Heuristics model was used to measure the users' perceptions about the playability of the application. Sweetser and Wyeth's Gameflow model was used to measure perceptions about the gaming elements' contribution toward creating an enjoyable experience. Malone and Lepper's Taxonomy of Intrinsic Motivation for Learning was used to measure the gaming elements' ability to promote an intrinsically motivating learning environment. Masterminds, Achievers, Conquerors and Seekers were the most prominent player types found in the Brainhex online survey for which the gamification application design then catered. The staff in the department play-tested the application to evaluate the gaming elements. Overall the Storyline/Theme, suited to Seekers and Masterminds, ranked as the most effective gaming element in this study. The users perceived artwork as an essential component of a gamified learning application. The Individual Leaderboard, suited to Conquerors, ranked very closely as the second most effective gaming element. The Storyline/Theme and Individual Leaderboard both performed the strongest against the criteria measuring the playability. The Storyline/Theme was by far the strongest from a gameflow perspective and the Individual Leaderboard from a motivation perspective. The Avatars ranked the worst across all the measurement criteria. Based on quiz results, 86 percent of the staff in the department had learned the material from the gamified training prototype developed in this work. The findings from this study will therefore serve as input for developing a full-scale gamification learning application

Measuring Fun: A Case Study in Adapting to the Evolving Metrics of Player Experience

The gaming industry and the concept of gamification have altered the way many developers and users approach interactive products. As social gaming demographics expand to what was previously considered “casual” audiences, more users expect an enjoyable experience from their digital applications and games. Developers now request more detailed subjective descriptions of satisfaction and the player experience from user-experience (UX) practitioners. Focusing on how fun a product is for users/players requires subjective, situationally dependent metrics rather than traditional UX efficiency metrics. The UX discipline is still constructing a comprehensive ecology of the player experience and how to measure it. This article contributes to that ecology by detailing a case in which our team conducted a usability test on a new video game peripheral. Our client’s primary concern dealt with how fun experienced gamers found the device. As our test progressed, we encountered a number of fun-related participant behaviors that led us to develop new metrics beyond our initial planned metrics. These new metrics helped us and our client better define and discuss enjoyability. Our case, in conjunction with a detailed definition and review of player experience and UX scholarship, shows the importance of adopting metrics contextually specific to the video-game product and player group when measuring fun is the primary goal

Academic methods for usability evaluation of serious games: a systematic review

In the last years, there has been an increasing interest in the design of video games as a tool for education, training, health promotion, socialization, etc. Usability, which is a key factor in any video game, becomes even more important in these so-called Bserious games^, where the users’ special characteristics should be considered, and the game efficacy depends on the users’ adherence and engagement. However, evaluation of the usability of this kind of games requires a redefinition of techniques, methods and even terminology. In this paper, we elicit six research questions and conduct a systematic review of the scientific literature, which resulted in the selection of 187 papers that contained the most relevant responses. The conclusions of this systematic review illustrate the general status of current academic usability evaluations of these games and the main trends in the selection of methodologies and how are they applied. This view may be a very valuable foundation for future researchMinisterio de Ciencia e Innovación PROCUR@-IPT-2011-1038-90000