Games and Brain-Computer Interfaces: The State of the Art

BCI gaming is a very young field; most games are proof-of-concepts. Work that compares BCIs in a game environments with traditional BCIs indicates no negative effects, or even a positive effect of the rich visual environments on the performance. The low transfer-rate of current games poses a problem for control of a game. This is often solved by changing the goal of the game. Multi-modal input with BCI forms an promising solution, as does assigning more meaningful functionality to BCI control

Tangible user interfaces : past, present and future directions

In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in or- der to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This paper examines the existing body of work on Tangible User In- terfaces. We start by sketching the history of tangible user interfaces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frame- works and taxonomies. We also discuss conceptual foundations of TUIs including perspectives from cognitive sciences, phycology, and philoso- phy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limita- tions of TUIs and chart directions for future research

3D (embodied) projection mapping and sensing bodies : a study in interactive dance performance

This dissertation identifies the synergies between physical and virtual environments when designing for immersive experiences in interactive dance performances. The integration of virtual information in physical space is transforming our interactions and experiences with the world. By using the body and creative expression as the interface between real and virtual worlds, dance performance creates a privileged framework to research and design interactive mixed reality environments and immersive augmented architectures. The research is primarily situated in the fields of visual art and interaction design. It combines performance with transdisciplinary fields and intertwines practice with theory. The theoretical and conceptual implications involved in designing and experiencing immersive hybrid environments are analyzed using the reality–virtuality continuum. These theories helped frame the ways augmented reality architectures are achieved through the integration of dance performance with digital software and reception displays. They also helped identify the main artistic affordances and restrictions in the design of augmented reality and augmented virtuality environments for live performance. These pervasive media architectures were materialized in three field experiments, the live dance performances. Each performance was created in three different stages of conception, design and production. The first stage was to “digitize” the performer’s movement and brain activity to the virtual environment and our system. This was accomplished through the use of depth sensor cameras, 3D motion capture, and brain computer interfaces. The second stage was the creation of the computational architecture and software that aggregates the connections and mapping between the physical body and the spatial dynamics of the virtual environment. This process created real-time interactions between the performer’s behavior and motion and the real-time generative computer 3D graphics. Finally, the third stage consisted of the output modality: 3D projector based augmentation techniques were adopted in order to overlay the virtual environment onto physical space. This thesis proposes and lays out theoretical, technical, and artistic frameworks between 3D digital environments and moving bodies in dance performance. By sensing the body and the brain with the 3D virtual environments, new layers of augmentation and interactions are established, and ultimately this generates mixed reality environments for embodied improvisational self-expression.Radio-Television-Fil

Natural User Interface for Education in Virtual Environments

Education and self-improvement are key features of human behavior. However, learning in the physical world is not always desirable or achievable. That is how simulators came to be. There are domains where purely virtual simulators can be created in contrast to physical ones. In this research we present a novel environment for learning, using a natural user interface. We, humans, are not designed to operate and manipulate objects via keyboard, mouse or a controller. The natural way of interaction and communication is achieved through our actuators (hands and feet) and our sensors (hearing, vision, touch, smell and taste). That is the reason why it makes more sense to use sensors that can track our skeletal movements, are able to estimate our pose, and interpret our gestures. After acquiring and processing the desired – natural input, a system can analyze and translate those gestures into movement signals

Advancing computational biophysics with Virtual Reality

Modelos computacionais são ferramentas poderosas para explorar as propriedades de sistemas biológicos complexos. Na neurociência computacional, permitir fácil exploração e visualização computacional desses modelos é crucial para o progresso do campo. Nos últimos anos, os sistemas de visualização 3D e o hardware de realidade virtual tornaram-se mais acessíveis e isso abre uma janela de oportunidade para os serviços de visualização. O principal problema atual da visualização 3D diz respeito à usabilidade (ou seja, navegação e seleção). Durante esta dissertação, hipotetizaremos que a substituição do 3D por VR irá (1) superar os problemas de usabilidade mencionados e, eventualmente, (2) aumentar a eficácia dos utilizadores em relação às questões do campo de estudo (neurociência). Para avaliar os resultados do trabalho desenvolvido nesta dissertação, será realizada uma experiência de duas partes, em que um grupo de indivíduos deverá executar um conjunto de tarefas pré-determinadas e avaliar sua experiência usando 3D na primeira e VR na última parte. Além da autoavaliação da experiência, dados como tempo de conclusão e correção da tarefa também serão usados para quantificar a eficácia do método de visualização. Dada a experiência mencionada, um protótipo de uma aplicação (baseada na Web) com visualização de Realidade Virtual deve ser desenvolvido. A visualização 3D será fornecida por uma framework de código aberto baseada na Web, chamada Geppetto. Cada uma das decisões tomadas no desenvolvimento do protótipo será analisada adequadamente neste documento, bem como a literatura científica que servirá de base quando necessário. Além do estudo da Realidade Virtual propriamente dita, também serão analisados métodos padronizados para a visualização de informações (neuro) científicas. A solução proposta procurará constituir uma base de trabalho sólida e suficientemente genérica a ser aplicada, não apenas no contexto da neurociência, mas também em vários outros contextos onde a visualização de modelos através de Realidade Virtual poderá ser bem-sucedida.Computational models are powerful tools for exploring the properties of complex biological systems. In computational neuroscience, allowing easy computational exploration and visualization of this models is crucial for the progress of the field. In recent years, Virtual Reality hardware and visualization systems have become more affordable and this opens a window of opportunity for visualization services. The current major problem of 3D visualization concerns usability (i.e., navigation and selection). During this dissertation, we will hypothesize that the replacement of 3D for VR will (1) overcome the usability issues mentioned and eventually (2) boost user effectiveness regarding field of study (neuroscience) concerns. In order to evaluate the results of the work developed under this dissertation, a two-part experiment will be carried out where a group of individuals must perform a set of predetermined tasks and evaluate their experience using 3D in the first and VR in the last part. Besides the self-evaluation of the experiment, data such as completion time and task correctness will also be used to quantify the effectiveness of the visualization method. Given the aforementioned experiment, a prototype of a (web-based) application with Virtual Reality visualization shall be developed. The 3D visualization will be provided by a web-based open-sourced framework called Geppetto. Each of the decisions made in the development of the prototype will be properly analyzed in this document, as well as the scientific literature that will serve as a basis when necessary. Besides the study of Virtual Reality itself, standard methods with respect to the visualization of (neuro)scientific information will also be analyzed. The proposed solution will seek to constitute a solid and sufficiently generic work base to be applied, not only in the scope of neuroscience, but also in several other contexts where visualization through VR might be successful

Prevalence of haptic feedback in robot-mediated surgery : a systematic review of literature

© 2017 Springer-Verlag. This is a post-peer-review, pre-copyedit version of an article published in Journal of Robotic Surgery. The final authenticated version is available online at: https://doi.org/10.1007/s11701-017-0763-4With the successful uptake and inclusion of robotic systems in minimally invasive surgery and with the increasing application of robotic surgery (RS) in numerous surgical specialities worldwide, there is now a need to develop and enhance the technology further. One such improvement is the implementation and amalgamation of haptic feedback technology into RS which will permit the operating surgeon on the console to receive haptic information on the type of tissue being operated on. The main advantage of using this is to allow the operating surgeon to feel and control the amount of force applied to different tissues during surgery thus minimising the risk of tissue damage due to both the direct and indirect effects of excessive tissue force or tension being applied during RS. We performed a two-rater systematic review to identify the latest developments and potential avenues of improving technology in the application and implementation of haptic feedback technology to the operating surgeon on the console during RS. This review provides a summary of technological enhancements in RS, considering different stages of work, from proof of concept to cadaver tissue testing, surgery in animals, and finally real implementation in surgical practice. We identify that at the time of this review, while there is a unanimous agreement regarding need for haptic and tactile feedback, there are no solutions or products available that address this need. There is a scope and need for new developments in haptic augmentation for robot-mediated surgery with the aim of improving patient care and robotic surgical technology further.Peer reviewe