Unmediated Interaction: Communicating with Computers and Embedded Devices as If They Are Not There

Although computers are smaller and more readily accessible today than they have ever been, I believe that we have barely scratched the surface of what computers can become. When we use computing devices today, we end up spending a lot of our time navigating to particular functions or commands to use devices their way rather than executing those commands immediately. In this dissertation, I explore what I call unmediated interaction, the notion of people using computers as if the computers are not there and as if the people are using their own abilities or powers instead. I argue that facilitating unmediated interaction via personalization, new input modalities, and improved text entry can reduce both input overhead and output overhead, which are the burden of providing inputs to and receiving outputs from the intermediate device, respectively. I introduce three computational methods for reducing input overhead and one for reducing output overhead. First, I show how input data mining can eliminate the need for user inputs altogether. Specifically, I develop a method for mining controller inputs to gain deep insights about a players playing style, their preferences, and the nature of video games that they are playing, all of which can be used to personalize their experience without any explicit input on their part. Next, I introduce gaze locking, a method for sensing eye contact from an image that allows people to interact with computers, devices, and other objects just by looking at them. Third, I introduce computationally optimized keyboard designs for touchscreen manual input that allow people to type on smartphones faster and with far fewer errors than currently possible. Last, I introduce the racing auditory display (RAD), an audio system that makes it possible for people who are blind to play the same types of racing games that sighted players can play, and with a similar speed and sense of control as sighted players. The RAD shows how we can reduce output overhead to provide user interface parity between people with and without disabilities. Together, I hope that these systems open the door to even more efforts in unmediated interaction, with the goal of making computers less like devices that we use and more like abilities or powers that we have

Improving Big Data Visual Analytics with Interactive Virtual Reality

For decades, the growth and volume of digital data collection has made it challenging to digest large volumes of information and extract underlying structure. Coined 'Big Data', massive amounts of information has quite often been gathered inconsistently (e.g from many sources, of various forms, at different rates, etc.). These factors impede the practices of not only processing data, but also analyzing and displaying it in an efficient manner to the user. Many efforts have been completed in the data mining and visual analytics community to create effective ways to further improve analysis and achieve the knowledge desired for better understanding. Our approach for improved big data visual analytics is two-fold, focusing on both visualization and interaction. Given geo-tagged information, we are exploring the benefits of visualizing datasets in the original geospatial domain by utilizing a virtual reality platform. After running proven analytics on the data, we intend to represent the information in a more realistic 3D setting, where analysts can achieve an enhanced situational awareness and rely on familiar perceptions to draw in-depth conclusions on the dataset. In addition, developing a human-computer interface that responds to natural user actions and inputs creates a more intuitive environment. Tasks can be performed to manipulate the dataset and allow users to dive deeper upon request, adhering to desired demands and intentions. Due to the volume and popularity of social media, we developed a 3D tool visualizing Twitter on MIT's campus for analysis. Utilizing emerging technologies of today to create a fully immersive tool that promotes visualization and interaction can help ease the process of understanding and representing big data.Comment: 6 pages, 8 figures, 2015 IEEE High Performance Extreme Computing Conference (HPEC '15); corrected typo

A Mixed Method Approach for Evaluating and Improving the Design of Learning in Puzzle Games

Despite the acknowledgment that learning is a necessary part of all gameplay, the area of Games User Research lacks an established evidence based method through which designers and researchers can understand, assess, and improve how commercial games teach players game-specific skills and information. In this paper, we propose a mixed method procedure that draws together both quantitative and experiential approaches to examine the extent to which players are supported in learning about the game world and mechanics. We demonstrate the method through presenting a case study of the game Portal involving 14 participants, who differed in terms of their gaming expertise. By comparing optimum solutions to puzzles against observed player performance, we illustrate how the method can indicate particular problems with how learning is structured within a game. We argue that the method can highlight where major breakdowns occur and yield design insights that can improve the player experience with puzzle games

GAMESPECT: A Composition Framework and Meta-Level Domain Specific Aspect Language for Unreal Engine 4

Game engine programming involves a great number of software components, many of which perform similar tasks; for example, memory allocation must take place in the renderer as well as in the creation routines while other tasks such as error logging must take place everywhere. One area of all games which is critical to the success of the game is that of game balance and tuning. These balancing initiatives cut across all areas of code from the player and AI to the mission manager. In computer science, we’ve come to call these types of concerns “cross cutting”. Aspect oriented programming was developed, in part, to solve the problems of cross cutting: employing “advice” which can be incorporated across different pieces of functionality. Yet, despite the prevalence of a solution, very little work has been done to bring cross cutting to game engine programming. Additionally, the discipline involves a heavy amount of code rewriting and reuse while simultaneously relying on many common design patterns that are copied from one project to another. In the case of game balance, the code may be wildly different across two different games despite the fact that similar tasks are being done. These two problems are exacerbated by the fact that almost every game engine has its own custom DSL (domain specific language) unique to that situation. If a DSL could showcase the areas of cross cutting concerns while highlighting the ability to capture design patterns that can be used across games, significant productivity savings could be achieved while simultaneously creating a common thread for discussion of shared problems within the domain. This dissertation sought to do exactly that- create a metalanguage called GAMESPECT which supports multiple styles of DSLs while bringing aspect-oriented programming into the DSL’s to make them DSAL (domain specific aspect languages). The example cross cutting concern was game balance and tuning since it’s so pervasive and important to gaming. We have created GAMESPECT as a language and a composition framework which can assist engine developers and game designers in balancing their games, forming one central place for game balancing concerns even while these concerns may cross different languages and locations inside the source code. Generality was measured by showcasing the composition specifications in multiple contexts and languages. In addition to evaluating generality and performance metrics, effectiveness was be measured. Specifically, comparisons were made between a balancing initiative when performed with GAMESPECT vs a traditional methodology. In doing so, this work shows a clear advantage to using a Metalanguage such as GAMESPECT for this task. In general, a line of code reduction of 9-40% per task was achieved with negligible effects to performance. The use of a metalanguage in Unreal Engine 4 is a starting point to further discussions concerning other game engines. In addition, this work has implications beyond video game programming. The work described highlights benefits which might be achieved in other disciplines where design pattern implementations and cross-cutting concern usage is high; the real time simulation field and the field of Windows GUI programming are two examples of future domains

NPC AI System Based on Gameplay Recordings

Hästi optimeeritud mitte-mängija tegelased (MMT) on vastaste või meeskonna kaaslastena üheks peamiseks osaks mitme mängija mängudes. Enamus mänguroboteid on ehitatud jäikade süsteemide peal, mis võimaldavad vaid loetud arvu otsuseid ja animatsioone. Kogenud mängijad suudavad eristada mänguroboteid inimmängijatest ning ette ennustada nende liigutusi ja strateegiaid. See alandab mängukogemuse kvaliteeti. Seetõttu, eelistavad mitme mängijaga mängude mängijad mängida pigem inimmängijate kui MMTde vastu. Virtuaalreaalsuse (VR) mängud ja VR mängijad on siiani veel väike osa mängutööstusest ja mitme mängija VR mängud kannatavad mängijabaasi kaotusest, kui mänguomanikud ei suuda leida teisi mängijaid, kellega mängida. See uurimus demonstreerib mängulindistustel põhineva tehisintellekt (TI) süsteemi rakendatavust VR esimese isiku vaates tulistamismängule Vrena. Teemamäng kasutab ebatavalist liikumisesüsteemi, milles mängijad liiguvad otsiankrute abil. VR mängijate liigutuste imiteerimiseks loodi AI süsteem, mis kasutab mängulindistusi navigeerimisandmetena. Süsteem koosneb kolmest peamisest funktsionaalsusest. Need funktsionaalsused on mängutegevuse lindistamine, andmete töötlemine ja navigeerimine. Mängu keskkond on tükeldatud kuubikujulisteks sektoriteks, et vähendada erinevate asukohal põhinevate olekute arvu ning mängutegevus on lindistatud ajaintervallide ja tegevuste põhjal. Loodud mängulogid on segmenteeritud logilõikudeks ning logilõikude abil on loodud otsingutabel. Otsingutabelit kasutatakse MMT agentide navigeerimiseks ning MMTde otsuste langetamise mehanism jäljendab olek-tegevus-tasu kontseptsiooni. Loodud töövahendi kvaliteeti hinnati uuringu põhjal, millest saadi märkimisväärset tagasisidet süsteemi täiustamiseks.A well optimized Non-Player Character (NPC) as an opponent or a teammate is a major part of the multiplayer games. Most of the game bots are built upon a rigid system with numbered decisions and animations. Experienced players can distinguish bots from hu-man players and they can predict bot movements and strategies. This reduces the quality of the gameplay experience. Therefore, multiplayer game players favour playing against human players rather than NPCs. VR game market and VR gamers are still a small frac-tion of the game industry and multiplayer VR games suffer from loss of their player base if the game owners cannot find other players to play with. This study demonstrates the applicability of an Artificial Intelligence (AI) system based on gameplay recordings for a Virtual Reality (VR) First-person Shooter (FPS) game called Vrena. The subject game has an uncommon way of movement, in which the players use grappling hooks to navigate. To imitate VR players’ movements and gestures an AI system is developed which uses gameplay recordings as navigation data. The system contains three major functionality. These functionalities are gameplay recording, data refinement, and navigation. The game environment is sliced into cubic sectors to reduce the number of positional states and gameplay is recorded by time intervals and actions. Produced game logs are segmented into log sections and these log sections are used for creating a look-up table. The lookup table is used for navigating the NPC agent and the decision mechanism followed a way similar to the state-action-reward concept. The success of the developed tool is tested via a survey, which provided substantial feedback for improving the system

Using models of baseline gameplay to design for physical rehabilitation

Modified digital games manage to drive motivation in repetitive exercises needed for motor rehabilitation, however designing modifications that satisfy both rehabilitation and engagement goals is challenging. We present a method wherein a statistical model of baseline gameplay identifies design configurations that emulate behaviours compatible with unmodified play. We illustrate this approach through a case study involving upper limb rehabilitation with a custom controller for a Pac-Man game. A participatory design workshop with occupational therapists defined two interaction parameters for gameplay and rehabilitation adjustments. The parameters' effect on the interaction was measured experimentally with 12 participants. We show that a low-latency model, using both user input behaviour and internal game state, identifies values for interaction parameters that reproduce baseline gameplay under degraded control. We discuss how this method can be applied to systematically balance gamification problems involving trade-offs between physical requirements and subjectively engaging experiences.Comment: 19 pages, 10 figure

Generalised Player Modelling : Why Artificial Intelligence in Games Should Incorporate Meaning, with a Formalism for so Doing

General game-playing artificial intelligence (AI) has recently seen important advances due to the various techniques known as ‘deep learning’. However, in terms of human-computer interaction, the advances conceal a major limitation: these algorithms do not incorporate any sense of what human players find meaningful in games. I argue that adaptive game AI will be enhanced by a generalised player model, because games are inherently human artefacts which require some encoding of the human perspective in order to respond naturally to individual players. The player model provides constraints on the adaptive AI, which allow it to encode aspects of what human players find meaningful. I propose that a general player model requires parameters for the subjective experience of play, including: player psychology, game structure, and actions of play. I argue that such a player model would enhance efficiency of per-game solutions, and also support study of game-playing by allowing (within-player) comparison between games, or (within-game) comparison between players (human and AI). Here we detail requirements for functional adaptive AI, arguing from first-principles drawn from games research literature, and propose a formal specification for a generalised player model based on our ‘Behavlets’ method for psychologically-derived player modelling.Peer reviewe

Player Modeling

Player modeling is the study of computational models of players in games. This includes the detection, modeling, prediction and expression of human player characteristics which are manifested through cognitive, affective and behavioral patterns. This chapter introduces a holistic view of player modeling and provides a high level taxonomy and discussion of the key components of a player\u27s model. The discussion focuses on a taxonomy of approaches for constructing a player model, the available types of data for the model\u27s input and a proposed classification for the model\u27s output. The chapter provides also a brief overview of some promising applications and a discussion of the key challenges player modeling is currently facing which are linked to the input, the output and the computational model

The Use of digital games to enhance the physical exercise activity of the elderly : a case of Finland

According to the World Health Organization (WHO), population ageing is a global phenomenon, which brings both challenges and opportunities for society. The current longer expected lifespan can create opportunities for the elderly to contribute in many ways to their families and communities. However, it greatly depends on their quality of life, which is affected by many factors, including physical and functional health, social well-being, and cognitive abilities. The WHO (2012) states that physical health is one of the indicators for the elderly’s quality of life, and it declines with increasing age. Participation in regular physical exercises can help the elderly improve their physical and mental health, and this has been aided by the use of modern technologies to promote the elderly’s physical and functional health. Of these latest technologies, digital games have shown promise to improve and enhance the elderly’s physical activities through fun and engaging gameplay. The literature highlights that some commercial games in the market (e.g. Microsoft Kinect- Sports and Nintendo Wii Sports games) have the potential to improve the elderly’s physical health such as gait, balance, and fall prevention. However, researchers argue that these commercial games are not designed specifically for the elderly and their physical exercise activities. They state that most commercial games are not user-friendly for the elderly whose functional and physical abilities are limited due to their advanced years. The literature points out that more studies need to be undertaken to understand the usability and usefulness of digital games for physical exercise activities so that game designers can create elderly-friendly digital games in the future. In Finland, the government has been focusing on promoting healthy ageing and increasing home care services for the elderly. In recent years, Finnish researchers have used digital games to promote older Finns’ healthy and active ageing. The existing literature, whilst showing the potential of digital games for elderly Finns’ physical health, also acknowledges further research is needed particularly in the context of Finland. Thus, in this study, we aimed at investigating digital games to specifically assess their applications for older Finns’ physical activities, focusing on the quality of users’ experiences, and their reported ease of use and perceived usefulness. We used the mixed methods approach, which applies both qualitative and quantitative research methods. The study design included four stages: requirements gathering, analysis and design, prototyping, and evaluation. Firstly, we conducted pre-studies to elicit users’ requirements. This was followed by the analysis of the resulting data to identify trends and patterns, which fuelled ideas in the brainstorming game design and development phases. The final product was a digital game-based physical exercise called the Skiing Game. We then evaluated the Skiing Game in Finland with 21 elderly Finns (M=7, F=14, Average Age =76). By using questionnaires, observation, and interviews, we investigated user experiences, focusing on the game’s usability, and usefulness for enhancing the physical activity and wellbeing of the elderly. We also conducted a comparative test of the Skiing Game in Japan with 24 elderly Japanese participants (M=12, F=12, Average Age = 72) to further understand non-Finnish elderly users’ experiences. The findings from the usability study of the Skiing Game in Finland demonstrated that elderly Finns had a positive experience in the gameplay, and their motivation was noticeably high. It also confirmed that elderly Finns have a genuine interest in digital game-based exercises and strong intentions to play digital games as a form of physical exercise in the future. Although prior to the study most of them had negative views and misconceptions about digital games, after the gameplay their attitudes were decidedly positive. They acknowledged that whilst playing digital games could be an alternative way of exercising for them their use would primarily be when they don’t have access to their usual non-digital physical exercise. The Japanese usability of the Skiing Game showed that the elderly Japanese people also had positive user experiences in playing digital games, and also intend to use them in the future. Similarly, after playing the game they reported that their attitudes towards digital games become positive, and indicated playing digital games could be an alternative way of exercising. Although the comparison of the two studies suggests that the elderly Finns had relatively more positive experiences whilst playing the Skiing Game, compared to their Japanese counterparts, in general, both groups had a positive experience in the gameplay and showed interest in digital games as an alternative exercise. Based on the usability lessons learned from these two studies, recommendations for practitioners and designers regarding improvements in game design and development are made in this report. Implementing these modifications into future designs and further development of digital games for the elderly will improve their commercial viability and user uptake. The findings from this study can provide valuable insights, particularly for Finnish policymakers and healthcare practitioners who are keen to introduce digital games into the aged-care sector in Finland. The studies have also provided valuable insights into the optimal methods for introducing Finnish digital games to international markets, in particular, digital games tailored specifically for the physical exercise needs and motivations of the elderly. By taking into consideration the limitations of the study, we provide our future studies and further improvements of the game to be conducted