Research on Application of Cognitive-Driven Human-Computer Interaction

Human-computer interaction is an important research content of intelligent manufacturing human factor engineering. Natural human-computer interaction conforms to the cognition of users' habits and can efficiently process inaccurate information interaction, thus improving user experience and reducing cognitive load. Through the analysis of the information interaction process, user interaction experience cognition and human-computer interaction principles in the human-computer interaction system, a cognitive-driven human-computer interaction information transmission model is established. Investigate the main interaction modes in the current human-computer interaction system, and discuss its application status, technical requirements and problems. This paper discusses the analysis and evaluation methods of interaction modes in human-computer system from three levels of subjective evaluation, physiological measurement and mathematical method evaluation, so as to promote the understanding of inaccurate information to achieve the effect of interaction self-adaptation and guide the design and optimization of human-computer interaction system. According to the development status of human-computer interaction in intelligent environment, the research hotspots, problems and development trends of human-computer interaction are put forward

Narratives of ocular experience in interactive 360° environments

The purpose of this research project was to examine how immersive digital virtual technologies have the potential to expand the genre of interactive film into new forms of audience engagement and narrative production. Aside from addressing the limitations of interactive film, I have explored how interactive digital narratives can be reconfigured in the wake of immersive media. My contribution to knowledge stems from using a transdisciplinary synthesis of the interactive systems in film and digital media art, which is embodied in the research framework and theoretical focal point that I have titled Cynematics (chapter 2). Using a methodology that promotes iterative experimentation I developed a series of works that allowed me to practically explore the limitations of interactive film systems that involve non-haptic user interaction. This is evidenced in the following series of works: Virtual Embodiment, Narrative Maze, Eye Artefact Interactions and Routine Error - all of which are discussed in chapter 4 of this thesis. Each of these lab experiments collectively build towards the development of novel interactive 360° film practices. Funneling my research towards these underexplored processes I focused on virtual gaze interaction (chapters 4-6), aiming to define and historically contextualise this system of interaction, whilst critically engaging with it through my practice. It is here that gaze interaction is cemented as the key focus of this thesis. The potential of interactive 360° film is explored through the creation of three core pieces of practice, which are titled as follows: Systems of Seeing (chapter 5), Mimesis (chapter 6), Vanishing Point (chapter 7). Alongside the close readings in these chapters and the theoretical developments explored in each are the interaction designs included in the appendix of the thesis. These provide useful context for readers unable to experience these site-specific installations as virtual reality applications. After creating these systems, I established terms to theoretically unpack some of the processes occurring within them. These include Datascape Mediation (chapter 2), which frames agency as a complex entanglement built on the constantly evolving relationships between human and machine - and Live-Editing Practice (chapter 7), which aims to elucidate how the interactive 360° film practice designed for this research leads to new way of thinking about how we design, shoot and interact with 360° film. Reflecting on feedback from exhibiting Mimesis I decided to define and evaluate the key modes of virtual gaze interaction, which led to the development of a chapter and concept referred to as The Reticle Effect (chapter 6). This refers to how a visual overlay that is used to represent a user's line of sight not only shapes their experience of the work, but also dictates their perception of genre. To navigate this, I combined qualitative and quantitative analysis to explore user responses to four different types of gaze interaction. In preparing to collect this data I had to articulate these different types of interaction, which served to demarcate the difference between each of these types of gaze interaction. Stemming from this I used questionnaires, thematic analysis and data visualisation to explore the use and response to these systems. The results of this not only supports the idea of the reticle effect, but also gives insight into how these different types of virtual gaze interaction shape whether these works are viewed as games or as types of interactive film. The output of this allowed me to further expand on interactive 360° film as a genre of immersive media and move beyond the realm of interactive film into new technological discourses, which serves to validate the nascent, yet expansive reach of interactive 360° film as a form of practice. The thesis is concluded by framing this research within the wider discourse of posthuman theory as given that the technologies of immersive media perpetuate a state of extended human experience - how we interact and consider the theories that surround these mediums needs to be considered in the same way. The practice and theory developed throughout this thesis contribute to this discourse and allow for new ways of considering filmic language in the wake of interactive 360° film practice

16th Sound and Music Computing Conference SMC 2019 (28–31 May 2019, Malaga, Spain)

The 16th Sound and Music Computing Conference (SMC 2019) took place in Malaga, Spain, 28-31 May 2019 and it was organized by the Application of Information and Communication Technologies Research group (ATIC) of the University of Malaga (UMA). The SMC 2019 associated Summer School took place 25-28 May 2019. The First International Day of Women in Inclusive Engineering, Sound and Music Computing Research (WiSMC 2019) took place on 28 May 2019. The SMC 2019 TOPICS OF INTEREST included a wide selection of topics related to acoustics, psychoacoustics, music, technology for music, audio analysis, musicology, sonification, music games, machine learning, serious games, immersive audio, sound synthesis, etc

Augmented reality and scene examination

The research presented in this thesis explores the impact of Augmented Reality on human performance, and compares this technology with Virtual Reality using a head-mounted video-feed for a variety of tasks that relate to scene examination. The motivation for the work was the question of whether Augmented Reality could provide a vehicle for training in crime scene investigation. The Augmented Reality application was developed using fiducial markers in the Windows Presentation Foundation, running on a wearable computer platform; Virtual Reality was developed using the Crytek game engine to present a photo-realistic 3D environment; and a video-feed was provided through head-mounted webcam. All media were presented through head-mounted displays of similar resolution to provide the sole source of visual information to participants in the experiments. The experiments were designed to increase the amount of mobility required to conduct the search task, i.e., from rotation in the horizontal or vertical plane through to movement around a room. In each experiment, participants were required to find objects and subsequently recall their location. It is concluded that human performance is affected not merely via the medium through which the world is perceived but moreover, the constraints governing how movement in the world is controlled

Accessibility and Inclusiveness of New Information and Communication Technologies for Disabled Users and Content Creators in the Metaverse

Despite the proliferation of Blockchain Metaverse projects, the inclusion of physically disabled individuals in the Metaverse remains distant, with limited standards and regulations in place. However, the article proposes a concept of the Metaverse that leverages emerging technologies, such as Virtual and Augmented Reality, and the Internet of Things, to enable greater engagement of disabled creatives. This approach aims to enhance inclusiveness in the Metaverse landscape. Based on the findings, the paper concludes that the active involvement of physically disabled individuals in the design and development of Metaverse platforms is crucial for promoting inclusivity. The proposed framework for accessibility and inclusiveness in Virtual, Augmented, and Mixed realities of decentralised Metaverses provides a basis for the meaningful participation of disabled creatives. The article emphasises the importance of addressing the mechanisms for art production by individuals with disabilities in the emerging Metaverse landscape. Additionally, it highlights the need for further research and collaboration to establish standards and regulations that facilitate the inclusion of physically disabled individuals in Metaverse projects

Network streaming and compression for mixed reality tele-immersion

Bulterman, D.C.A. [Promotor]Cesar, P.S. [Copromotor

VR for Cultural Heritage. A VR-WEB-BIM for the future maintenance of Milan’s Cathedral.

The work presented here is the final step of a multidisciplinary research project conducted on the Milan Cathedral for eight years (2008–2015). Three main topics, consequentially related, will be here addressed: (i) the survey of the structure, meant to update the old drawings; (ii) the construction of an accurate and detailed 3D model to be used to produce measurements at a 1:20–1:50 representation scale; (iii) the development of a Building Information System (BIM) to collect all the data relating to the restoration projects, as well as all information relating to past, current and future maintenance activities of the cathedral. The result of this research project is a complex and accurate digital 3D model of the main spire of the cathedral and of other parts of the building. This model can be visualized, navigated and used by the Veneranda Fabbrica technicians as an info-data catalogue, thanks to a common web browser connected with the remote BIM System Server and the modelling software where ad hoc I/O plugins are implemented. The last step of this long project was to take advantage of the nascent potential of immersive visualization techniques and to transpose the BIM system in a VR environment, thus obtaining two main results. The first was a high-appeal visualization system that allows a virtual visit of the Main Spire of the cathedral, the building’s highest part that has been closed to visitors since the beginning of the XX century. The second was the possibility to use this technology to virtually explore the cathedral from a technical point of view: by using an immersive visualization technology, operators can improve their understanding of the structure and obtain real-time information about the state of conservation, including current and past maintenance activities, in a sort of “augmented reality system in a virtual environment”

Development of an intelligent personal assistant to empower operators in industry 4.0 environments

Mestrado de dupla diplomação com a UTFPR - Universidade Tecnológica Federal do ParanáIndustry 4.0 brings a high level of automation to industrial environments and changes the way companies operate, both in operational aspects and in human relations. It is important to define the role of the millions of operators affected in this new socioeconomic paradigm, integrating new technologies and empowering the workforce to take advantage of aspects such as the flexibility and versatility that human operators bring to production lines. To advance the implementation of this objective, this work proposes the development of an intelligent personal assistant, using concepts of human-in-the-loop cyber-physical systems and context awareness, to assist operators during manufacturing tasks, providing the necessary information for the fulfillment of operations and verifying the accuracy to inform them about possible errors. The implementation is divided in two parts. The first part focuses on an application that supports the real-time operations that can be present in the industry, such as pick and place in warehouses and the assembly of complex equipment on an assembly line. Through an interface, the instruction is given and, using artificial vision techniques with images coming from an IntelRealsense camera, it verifies if the operation is being correctly performed. The gathering of this information occurs in a context awareness algorithm, fulfilling the requirement of intelligent personal assistant and providing feedback to the operator so that the tasks are performed with efficiency and lower incidence of errors. The second part includes the training of these operators in an immersive environment through a virtual reality equipment such as the Oculus Go. The immersive scenario, developed in Unity3D, uses as a model the real workbench, bringing the possibility of performing these trainings in any environment and excluding the need to use real equipment, which could be damaged by the user’s inexperience. The results achieved during the validation tests performed in these two parts, commenting on the strengths, challenges and failures found in the system in general. These results are also qualitatively compared with traditional applications of the proposed case studies in order to prove the fulfillment of the objectives proposed in this work. Finally, the usability test is presented, which provides data on weak points in the user experience for possible improvements in future work.A indústria 4.0 traz um nível elevado de automação a ambientes industriais e muda a forma em que empresas funcionam, tanto em aspectos operacionais quanto em relações humanas. É importante a definição do papel dos milhões de operadores afetados neste novo paradigma socioeconômico, fazendo a integração das novas tecnologias e capacitando a mão de obra para fazer proveito de aspectos como a flexibilidade e versatilidade que operadores humanos trazem às linhas de produção. Para avançar a implementação deste objetivo, este trabalho propõe o desenvolvimento de uma assistente pessoal inteligente, utilizando conceitos de human-in-the-loop cyberphysical systems e context awareness, para auxiliar operadores durante tarefas de manufatura, provendo informações necessárias para o cumprimento de operações e verificando a acurácia para informá-lo sobre possíveis erros. A implementação está dividida em duas partes. A primeira parte foca em uma aplicação de operações em tempo real que podem estar presentes na indústria como pick-andplace em armazéns e a montagem de equipamentos complexos em uma linha de montagem. Através de uma interface é dada a instrução a ser realizada e, utilizando técnicas de visão artificial, com imagens vindas de uma câmera IntelRealsense, verifica se a operação está sendo corretamente executada. A junção dessas informações ocorre em um algoritmo de context awareness, cumprindo o requisito de assistente pessoal inteligente e fornecendo o feedback ao operador para que as tarefas sejam realizadas com eficiência e menor incidência de erros. Já a segunda parte engloba o treinamento destes operadores em um ambiente imersivo através de um equipamento de realidade virtual como o Oculus Go. O cenário, desenvolvido no Unity3D, utiliza como modelo a bancada real, trazendo a possibilidade de se realizar esses treinamentos em qualquer ambiente, excluindo a necessidade da utilização de equipamentos reais e possíveis danos originados de inexperiência do usuário. Os resultados apresentam os testes de validação realizados nestas duas partes, comentando os pontos fortes, desafios e falhas encontradas no sistema em geral. Estes resultados também são comparados qualitativamente com aplicações tradicionais dos casos de estudo propostos de forma a comprovar o cumprimento dos objetivos propostos neste trabalho. Por fim, é apresentado o teste de usabilidade que fornece dados em pontos fracos na experiência de usuários para possíveis melhorias em futuros trabalhos