Merging the Real and the Virtual: An Exploration of Interaction Methods to Blend Realities

We investigate, build, and design interaction methods to merge the real with the virtual. An initial investigation looks at spatial augmented reality (SAR) and its effects on pointing with a real mobile phone. A study reveals a set of trade-offs between the raycast, viewport, and direct pointing techniques. To further investigate the manipulation of virtual content within a SAR environment, we design an interaction technique that utilizes the distance that a user holds mobile phone away from their body. Our technique enables pushing virtual content from a mobile phone to an external SAR environment, interact with that content, rotate-scale-translate it, and pull the content back into the mobile phone. This is all done in a way that ensures seamless transitions between the real environment of the mobile phone and the virtual SAR environment. To investigate the issues that occur when the physical environment is hidden by a fully immersive virtual reality (VR) HMD, we design and investigate a system that merges a realtime 3D reconstruction of the real world with a virtual environment. This allows users to freely move, manipulate, observe, and communicate with people and objects situated in their physical reality without losing their sense of immersion or presence inside a virtual world. A study with VR users demonstrates the affordances provided by the system and how it can be used to enhance current VR experiences. We then move to AR, to investigate the limitations of optical see-through HMDs and the problem of communicating the internal state of the virtual world with unaugmented users. To address these issues and enable new ways to visualize, manipulate, and share virtual content, we propose a system that combines a wearable SAR projector. Demonstrations showcase ways to utilize the projected and head-mounted displays together, such as expanding field of view, distributing content across depth surfaces, and enabling bystander collaboration. We then turn to videogames to investigate how spectatorship of these virtual environments can be enhanced through expanded video rendering techniques. We extract and combine additional data to form a cumulative 3D representation of the live game environment for spectators, which enables each spectator to individually control a personal view into the stream while in VR. A study shows that users prefer spectating in VR when compared with a comparable desktop rendering

Relevant Independent Variables on MOBA Video Games to Train Machine Learning Algorithms

Popularity of MultiplayerOnlineBattle Arena (MOBA)video gameshas grown considerably, its popularity as well as the complexity of their playability, have attracted the attention in recent years of researchers from various areas of knowledge and in particular how they have resorted to different machine learning techniques. The papers reviewed mainly look for patterns in multidimensional data sets. Furthermore, these previous researches do not present a way to select the independent variables(predictors)to train the models. For this reason, this paper proposes a listof variables based on the techniques used and the objectives of the research. It allows to provide a set of variables to find patterns applied in MOBA videogames.In order to get the mentioned list,the consulted workswere groupedbythe used machine learning techniques, ranging from rule-based systems to complex neural network architectures. Also, a grouping technique is applied based on the objective of each research proposed

Many-screen viewing: collaborative consumption of television media across multiple devices

The landscape of television is changing. Modern Internet enabled sets are now capable computing devices offering new forms of connectivity and interaction to viewers. One development enabled by this transition is the distribution of auxiliary content to a portable computing device, such as a mobile phone or tablet, working in concert with the television. These configurations are enabled by second screen applications that provide relevant content in synchronisation with the programme on a nearby television set. This thesis extends the notion of second screen to arrangements that incorporate multiple mobile devices working with the television, utilised by collocated groups of participants. Herein these arrangements are referred to as ‘many-screen’ television. Two many-screen applications were developed for the augmentation of sports programming in preparation of this thesis; the Olympic Companion and MarathOn Multiscreen Applications. Both of these applications were informed by background literature on second screen television and wider issues in HCI multiscreen research. In addition, the design of both applications was inspired by the needs of traditional and online broadcasters, through an internship with BBC Research and Development and involvement in a YouTube sponsored project. Both the applications were evaluated by collocated groups of users in formative user studies. These studies centred on how users share and organise what to watch, incorporate activity within the traditionally passive television viewing experience and the integration of user-generated video content in a many-screen system. The primary contribution of this thesis is a series of industry validated guidelines for the design of many-screen applications. The guidelines highlight issues around user awareness devices, content and other user’s actions, the balance between communal and private viewing and the appropriation of user-generated content in many-screen watching

Many-screen viewing: collaborative consumption of television media across multiple devices

The landscape of television is changing. Modern Internet enabled sets are now capable computing devices offering new forms of connectivity and interaction to viewers. One development enabled by this transition is the distribution of auxiliary content to a portable computing device, such as a mobile phone or tablet, working in concert with the television. These configurations are enabled by second screen applications that provide relevant content in synchronisation with the programme on a nearby television set. This thesis extends the notion of second screen to arrangements that incorporate multiple mobile devices working with the television, utilised by collocated groups of participants. Herein these arrangements are referred to as ‘many-screen’ television. Two many-screen applications were developed for the augmentation of sports programming in preparation of this thesis; the Olympic Companion and MarathOn Multiscreen Applications. Both of these applications were informed by background literature on second screen television and wider issues in HCI multiscreen research. In addition, the design of both applications was inspired by the needs of traditional and online broadcasters, through an internship with BBC Research and Development and involvement in a YouTube sponsored project. Both the applications were evaluated by collocated groups of users in formative user studies. These studies centred on how users share and organise what to watch, incorporate activity within the traditionally passive television viewing experience and the integration of user-generated video content in a many-screen system. The primary contribution of this thesis is a series of industry validated guidelines for the design of many-screen applications. The guidelines highlight issues around user awareness devices, content and other user’s actions, the balance between communal and private viewing and the appropriation of user-generated content in many-screen watching

Decision Making Skill and Complex Problem Solving in Team Sports

This thesis aimed to enhance understanding of the nature of knowledge bases possessed by elite sports performers which underpin perceptual-cognitive and decision making skills. Two main theories were considered; Active Control of Thought (ACT*) and Representational Redescription (RR). The purpose of Study 1 was to examine the anticipatory ability of elite and non-elite players in football and hockey. The results indicated that elite players in both sports were quicker and more accurate in their expectation of pass destination. Study 2 aimed to understand the extent to which knowledge is transferable. The results indicated that elite players’ knowledge is relatively domain specific although some elements of underlying task strategy may transfer. The objective of Study 3 was to explore the means by which elite and non-elite players in football and hockey identify and differentiate between possible decisions. Results showed elite players’ rationale was based on deeper theoretical principles whilst non-experts utilised relatively superficial information and naïve theories. Study 4 focussed on problem representations of elite and non-elite football players. Results revealed elite players’ representations were more pertinent, connected and articulated in a more effective manner. Overall, the findings from the current thesis provide advanced understanding of the knowledge bases responsible for perceptual-cognitive and decision making skill, and such understanding may assist attempts to enhance athletes’ performance and support future research

Rational Agent Architecture to Recommend which Item to Buy in MOBA Videogames

Los videojuegos multijugador de arena de batalla en línea (MOBA), es un genero de videojuegos que durante la última década han ganado popularidad en la escena competitiva de los E-Sports. Este incremento en su popularidad y la complejidad propia de los mismos han llamado la atención de investigadores en todas las áreas del conocimiento, incluyendo la Inteligencia Artificial. Dichos investigadores han utilizado una amplia variedad de técnicas de Aprendizaje de Maquina buscando mejorar la experiencia de diversos usuarios -jugadores novatos, jugadores expertos, espectadores, entre otros- a través de modelos de predicción, sistemas de recomendación y, aunque se han utilizado técnicas de optimización; estas últimas han sido las menos utilizadas en los videojuegos tipo MOBA. Por ello, el presente trabajo de investigación propone la arquitectura de un agente racional capaz de recomendar a un jugador que objeto comprar para aumentar sus probabilidades de ganar una partida, utilizando una técnica de optimización para la generación de recomendaciones. En la arquitectura propuesta, el agente percibe su ambiente con la información disponible en el API del videojuego League of Legends -uno de los MOBA mas populares actualmente-. Tal información es interpretada por una Regresión Logística que durante las etapas tempranas del juego demostró tener una precisión alrededor de 0.975. A su vez, la técnica de optimización seleccionada para generar la sugerencia fue GRASP; en promedio cada sugerencia es generada en 0.36 segundos, estas sugerencias durante la experimentación lograron aumentar la probabilidad de ganar una partida en promedio 5.2x.Multiplayer online battle arena (MOBA) video games are a genre of video games that during the last decade have gained popularity in the competitive E-Sports scene. This increase in popularity and MOBA’s complexity have attracted the attention of researchers in all areas of knowledge, including Artificial Intelligence (AI). AI researchers have used a wide variety of Machine Learning techniques seeking to improve the experience of various users - novice players, expert players, spectators, among others - through prediction models, recommendation systems and optimization algorithms. However, optimization algorithms have been the least used in MOBA videogames. For that reason, this research proposes the architecture of a rational agent capable of recommending to a player what item to buy to increase his probabilities of winning a game, using an optimization technique for generating recommendations. In the proposed architecture, the agent perceives his environment with the information available in the API of League of Legends -currently, one of the most popular MOBA videogames -. Such information is interpreted by a Logistic Regression that during the early stages of the game was shown to have an accuracy around 0.975. Additionally, the optimization technique selected to generate the suggestion was GRASP. On average each suggestion is generated in 0.36 seconds. During experimentation, these suggestions increase the probability of winning a game on average 5.2x.Magíster en Inteligencia ArtificialMaestrí

Rational Agent Architecture to Recommend which Item to Buy in MOBA Videogames

Los videojuegos multijugador de arena de batalla en línea (MOBA), es un genero de videojuegos que durante la última década han ganado popularidad en la escena competitiva de los E-Sports. Este incremento en su popularidad y la complejidad propia de los mismos han llamado la atención de investigadores en todas las áreas del conocimiento, incluyendo la Inteligencia Artificial. Dichos investigadores han utilizado una amplia variedad de técnicas de Aprendizaje de Maquina buscando mejorar la experiencia de diversos usuarios -jugadores novatos, jugadores expertos, espectadores, entre otros- a través de modelos de predicción, sistemas de recomendación y, aunque se han utilizado técnicas de optimización; estas últimas han sido las menos utilizadas en los videojuegos tipo MOBA. Por ello, el presente trabajo de investigación propone la arquitectura de un agente racional capaz de recomendar a un jugador que objeto comprar para aumentar sus probabilidades de ganar una partida, utilizando una técnica de optimización para la generación de recomendaciones. En la arquitectura propuesta, el agente percibe su ambiente con la información disponible en el API del videojuego League of Legends -uno de los MOBA mas populares actualmente-. Tal información es interpretada por una Regresión Logística que durante las etapas tempranas del juego demostró tener una precisión alrededor de 0.975. A su vez, la técnica de optimización seleccionada para generar la sugerencia fue GRASP; en promedio cada sugerencia es generada en 0.36 segundos, estas sugerencias durante la experimentación lograron aumentar la probabilidad de ganar una partida en promedio 5.2x.Multiplayer online battle arena (MOBA) video games are a genre of video games that during the last decade have gained popularity in the competitive E-Sports scene. This increase in popularity and MOBA’s complexity have attracted the attention of researchers in all areas of knowledge, including Artificial Intelligence (AI). AI researchers have used a wide variety of Machine Learning techniques seeking to improve the experience of various users - novice players, expert players, spectators, among others - through prediction models, recommendation systems and optimization algorithms. However, optimization algorithms have been the least used in MOBA videogames. For that reason, this research proposes the architecture of a rational agent capable of recommending to a player what item to buy to increase his probabilities of winning a game, using an optimization technique for generating recommendations. In the proposed architecture, the agent perceives his environment with the information available in the API of League of Legends -currently, one of the most popular MOBA videogames -. Such information is interpreted by a Logistic Regression that during the early stages of the game was shown to have an accuracy around 0.975. Additionally, the optimization technique selected to generate the suggestion was GRASP. On average each suggestion is generated in 0.36 seconds. During experimentation, these suggestions increase the probability of winning a game on average 5.2x.Magíster en Inteligencia ArtificialMaestrí

Localisation and tracking of stationary users for extended reality

In this thesis, we investigate the topics of localisation and tracking in the context of Extended Reality. In many on-site or outdoor Augmented Reality (AR) applications, users are standing or sitting in one place and performing mostly rotational movements, i.e. stationary. This type of stationary motion also occurs in Virtual Reality (VR) applications such as panorama capture by moving a camera in a circle. Both applications require us to track the motion of a camera in potentially very large and open environments. State-of-the-art methods such as Structure-from-Motion (SfM), and Simultaneous Localisation and Mapping (SLAM), tend to rely on scene reconstruction from significant translational motion in order to compute camera positions. This can often lead to failure in application scenarios such as tracking for seated sport spectators, or stereo panorama capture where the translational movement is small compared to the scale of the environment. To begin with, we investigate the topic of localisation as it is key to providing global context for many stationary applications. To achieve this, we capture our own datasets in a variety of large open spaces including two sports stadia. We then develop and investigate these techniques in the context of these sports stadia using a variety of state-of-the-art localisation approaches. We cover geometry-based methods to handle dynamic aspects of a stadium environment, as well as appearance-based methods, and compare them to a state-of-the-art SfM system to identify the most applicable methods for server-based and on-device localisation. Recent work in SfM has shown that the type of stationary motion that we target can be reliably estimated by applying spherical constraints to the pose estimation. In this thesis, we extend these concepts into a real-time keyframe-based SLAM system for the purposes of AR, and develop a unique data structure for simplifying keyframe selection. We show that our constrained approach can track more robustly in these challenging stationary scenarios compared to state-of-the-art SLAM through both synthetic and real-data tests. In the application of capturing stereo panoramas for VR, this thesis demonstrates the unsuitability of standard SfM techniques for reconstructing these circular videos. We apply and extend recent research in spherically constrained SfM to creating stereo panoramas and compare this with state-of-the-art general SfM in a technical evaluation. With a user study, we show that the motion requirements of our SfM approach are similar to the natural motion of users, and that a constrained SfM approach is sufficient for providing stereoscopic effects when viewing the panoramas in VR