Enabling Robot Grasping using Mixed Reality

The rapid advancements in Robotics and Mixed Reality (MR) have opened new avenues for intuitive human-robot interaction. In this thesis, an intuitive and accessible robot grasping application is developed using MR to enable programming using the operator’s hand movements, reducing the technical complexity associated with traditional programming. The developed application leverages the strengths of MR to provide users with an immersive and intuitive environment. It includes powerful tools such as QR code recognition for quick deployment of virtual objects or the utilization of a Virtual Station that can be placed at any desired location, allowing remote and safe control of the robot. Three modes have been implemented, including manual target placing and thorough editing of its properties, path recording of the user's hand trajectory, and real-time replication of the operator's hand movements by the robot. To assess the effectiveness and intuitiveness of the developed application, a series of user tests are presented. These evaluations include user feedback and task completion time compared to traditional programming methods, which provide valuable insights into the application's usability, efficiency, and user satisfaction. The intuitiveness of the developed application democratizes robot programming, expanding accessibility to a wider range of users, including inexperienced operators and students

Sim2real and Digital Twins in Autonomous Driving: A Survey

Safety and cost are two important concerns for the development of autonomous driving technologies. From the academic research to commercial applications of autonomous driving vehicles, sufficient simulation and real world testing are required. In general, a large scale of testing in simulation environment is conducted and then the learned driving knowledge is transferred to the real world, so how to adapt driving knowledge learned in simulation to reality becomes a critical issue. However, the virtual simulation world differs from the real world in many aspects such as lighting, textures, vehicle dynamics, and agents' behaviors, etc., which makes it difficult to bridge the gap between the virtual and real worlds. This gap is commonly referred to as the reality gap (RG). In recent years, researchers have explored various approaches to address the reality gap issue, which can be broadly classified into two categories: transferring knowledge from simulation to reality (sim2real) and learning in digital twins (DTs). In this paper, we consider the solutions through the sim2real and DTs technologies, and review important applications and innovations in the field of autonomous driving. Meanwhile, we show the state-of-the-arts from the views of algorithms, models, and simulators, and elaborate the development process from sim2real to DTs. The presentation also illustrates the far-reaching effects of the development of sim2real and DTs in autonomous driving

Possibilities for decision science in the metaverse

As the next generation of the internet, the metaverse will be an immersive, interactive and persistent, three-dimensional world that merges both physical and virtual environments. Its associated advanced technologies (e.g., digital twins, avatar self-representation, virtual reality, augmented reality, extended reality, brain-computer interface, artificial intelligence, machine learning, Internet-of-Things, blockchain technology) are already functioning, and people are able to experience and make decisions in immersive virtual environments. In this paper, we first briefly assess the similarities and differences in the judgment and decision making (JDM) situations that people may face in the metaverse compared to their physical environments. Next, we discuss how human interaction with metaverse-related advanced technologies may affect the cognitive processes of perception, attention, memory and reasoning that underlie human JDM, as well as the subsequent effects of these processes on JDM, in both the virtual and physical worlds. Finally, we highlight some opportunities that the metaverse may afford decision scientists, given the availability of integrated, digitalized data collected from users (or research participants), and some of the challenges that lie therein. We structure these opportunities around the five key metaverse domains: digitalization, virtualization, social networks, virtual worlds, and data integration. As a research platform, the metaverse can help drive a paradigm shift away from traditional approaches to JDM research and enable decision scientists to test and consolidate existing theories as well as advance new ones, while conducting research that has social benefits

Exploring the use of smart glasses, gesture control, and environmental data in augmented reality games

Abstract. In the last decade, augmented reality has become a popular trend. Big corporations like Microsoft, Facebook, and Google started to invest in augmented reality because they saw the potential that it has especially with the rising of the consumer version of the head mounted displays such as Microsoft’s HoloLens and the ODG’s R7. However, there is a gap in the knowledge about the interaction with such devices since they are fairly new and an average consumer cannot yet afford them due to their relatively high prices. In this thesis, the Ghost Hunters game is described. The game is a mobile augmented reality pervasive game that uses the environment light data to charge the in-game “goggles”. The game has two different versions, a smartphone and smart glasses version. The Ghost Hunters game was implemented for exploring the use of two different types of interactions methods, buttons and natural hand gestures for both smartphones and smart glasses. In addition to that, the thesis sought to explore the use of ambient light in augmented reality games. First, the thesis defines the essential concepts related to games and augmented reality based on the literature and then describes the current state of the art of pervasive games and smart glasses. Second, both the design and implementation of the Ghost Hunters game are described in detail. Afterwards, the three rounds of field trials that were conducted to investigate the suitability of the two previously mentioned interaction methods are described and discussed. The findings suggest that smart glasses are more immersive than smartphones in context of pervasive AR games. Moreover, prior AR experience has a significant positive impact on the immersion of smart glasses users. Similarly, males were more immersed in the game than females. Hand gestures were proven to be more usable than the buttons on both devices. However, the interaction method did not affect the game engagement at all, but surprisingly it did affect the way users perceive the UI with smart glasses. Users that used the physical buttons were more likely to notice the UI elements than the users who used the hand gestures

Optimization of Human Processes in A Greenhouse Aeroponic Potato Farming System Using 3D Simulation And Virtual Reality

As global demand to industrialize food production increases, Aeroponics farming emerges as a sustainable approach to yielding higher food growth with minimal resources. This thesis addresses two aspects for advancing industrial scale aeroponic potato farming; Optimizing human process flow and configuring an effective layout in a greenhouse farm. The first research question addresses how human processes and task affects productivity in an aeroponic greenhouse farm and how it can be enhanced, by monitoring workflow patterns and identifying bottlenecks. The second question explores layout configurations suitable for large scale aeroponic potato farm produce. Through different layout comparisons, this study identifies the optimal layout con-figuration that maximizes practicality, facility effectiveness, space utilization and work efficiency. This study is simulation-based using 3D designs and virtual reality. The method used in this thesis is based on Discrete Event Simulation (DES) a modelling technique that represents the behaviour of a production system by simulating individual events that occur at certain points in time. A proposed solution is creating an aeroponic greenhouse farm layout using DES software, inserting 3D elements and configuring some human process modelling. The first layout is vertical having longer aisles and having an efficient workflow process. The second layout takes a horizontal shape with shorter aisles making work efficient and easy to manoeuvre. The findings offer valuable insights into how the right layout configuration can improve workers' efficiency. Results showing basic human performance of both greenhouse farm layout and simulation can be viewed using virtual reality glasses. The result demonstrates that the use of 3D modelling and simulation is beneficial in optimizing process flow and visualizing layouts when incorporated with VR to inspect models via immersive experience

Smart Factory Using Virtual Reality and Online Multi-User: Towards a Metaverse for Experimental Frameworks

Virtual reality (VR) has been brought closer to the general public over the past decade as it has become increasingly available for desktop and mobile platforms. As a result, consumer-grade VR may redefine how people learn by creating an engaging “hands-on” training experience. Today, VR applications leverage rich interactivity in a virtual environment without real-world consequences to optimize training programs in companies and educational institutions. Therefore, the main objective of this article was to improve the collaboration and communication practices in 3D virtual worlds with VR and metaverse focused on the educational and productive sector in smart factory. A key premise of our work is that the characteristics of the real environment can be replicated in a virtual world through digital twins, wherein new, configurable, innovative, and valuable ways of working and learning collaboratively can be created using avatar models. To do so, we present a proposal for the development of an experimental framework that constitutes a crucial first step in the process of formalizing collaboration in virtual environments through VR-powered metaverses. The VR system includes functional components, object-oriented configurations, advanced core, interfaces, and an online multi-user system. We present the study of the first application case of the framework with VR in a metaverse, focused on the smart factory, that shows the most relevant technologies of Industry 4.0. Functionality tests were carried out and evaluated with users through usability metrics that showed the satisfactory results of its potential educational and commercial use. Finally, the experimental results show that a commercial software framework for VR games can accelerate the development of experiments in the metaverse to connect users from different parts of the world in real time.Universidad Cooperativa de Colombia-Cali, Colombia INV278

Flight controller synthesis via deep reinforcement learning

Traditional control methods are inadequate in many deployment settings involving autonomous control of Cyber-Physical Systems (CPS). In such settings, CPS controllers must operate and respond to unpredictable interactions, conditions, or failure modes. Dealing with such unpredictability requires the use of executive and cognitive control functions that allow for planning and reasoning. Motivated by the sport of drone racing, this dissertation addresses these concerns for state-of-the-art flight control by investigating the use of deep artificial neural networks to bring essential elements of higher-level cognition to bear on the design, implementation, deployment, and evaluation of low level (attitude) flight controllers. First, this thesis presents a feasibility analyses and results which confirm that neural networks, trained via reinforcement learning, are more accurate than traditional control methods used by commercial uncrewed aerial vehicles (UAVs) for attitude control. Second, armed with these results, this thesis reports on the development and release of an open source, full solution stack for building neuro-flight controllers. This stack consists of a tuning framework for implementing training environments (GymFC) and firmware for the world’s first neural network supported flight controller (Neuroflight). GymFC’s novel approach fuses together the digital twinning paradigm with flight control training to provide seamless transfer to hardware. Third, to transfer models synthesized by GymFC to hardware, this thesis reports on the toolchain that has been released for compiling neural networks into Neuroflight, which can be flashed to off-the-shelf microcontrollers. This toolchain includes detailed procedures for constructing a multicopter digital twin to allow the research and development community to synthesize flight controllers unique to their own aircraft. Finally, this thesis examines alternative reward system functions as well as changes to the software environment to bridge the gap between simulation and real world deployment environments. The design, evaluation, and experimental work summarized in this thesis demonstrates that deep reinforcement learning is able to be leveraged for the design and implementation of neural network controllers capable not only of maintaining stable flight, but also precision aerobatic maneuvers in real world settings. As such, this work provides a foundation for developing the next generation of flight control systems

Mixed reality and BIM applied to construction health and safety

Dissertação de mestrado em European Master in Building Information ModellingNa construção, as operações no local são complexas. Os trabalhadores da construção civil realizam várias atividades distintas, cada uma frequentemente associada a uma infinidade de riscos. Esses riscos incluem muitos trabalhadores nas proximidades das áreas operacionais onde essas atividades ocorrem. Como tal, o setor da construção é conhecido por registar um elevado índice de acidentes. Apesar do grande esforço na análise de soluções de projeto, planeamento de atividades, educação e consciencialização dos intervenientes no local, é necessário melhorar. A experiência de campo mostra que a gestão da saúde e segurança no local de construção é principalmente manual e fragmentada. Além disso, a falta de integração entre o mundo digital e o mundo físico, especialmente a interação entre a análise de dados e as medidas correspondentes, leva à fragmentação e duplicação de dados e à estagnação entre as diferentes fases do ciclo de vida da construção. As situações em um estaleiro de obra são dinâmicas e mudam constantemente em termos de instalações, equipamentos, materiais, pessoas, disposição do local e vários outros componentes em diferentes fases. Isso aumenta a complexidade da monitorização e gestão de segurança dos estaleiros de obras. Nesta dissertação é proposto o desenvolvimento de um esquema para aplicação de um Digital Twin (gémeo digital) para procesos BIM durante a fase de construção – enriquecido com riscos de construção – permitindo coletar dados de sensores no local, melhorando assim a monitorização e registro do comportamento dos trabalhadores da construção. Este sistema pode ser utilizado para criar alertas em tempo real para um ou vários trabalhadores, contribuindo para mitigar o número de acidentes nesta indústria. Além de abordar a lacuna entre a operação no local e a gestão em escritório, esta proposta permite criar um banco de dados para analisar incidentes e informar decisões com base em um modelo BIM. Este modelo também pode ser facilmente atualizado para melhor representar cada momento de uma etapa de construção (4D BIM), servindo como uma importante ferramenta para quem colabora neste espaço físico.In construction, site operations are complex. Construction workers perform a diversity of activities, each often associated with a multitude of risks. Those risks are generally extended to many workers in the vicinity of the operational areas where the activities happen. As such, the construction sector is known for recording a high rate of accidents. Despite the large effort in the analysis of design solutions, planning of activities, education, and awareness of site agents, improvements are required. Our field experience shows that live field CHS management is mostly manual and fragmented. Furthermore, the lack of integration between the digital world and the physical world, especially the interaction between data analysis and the corresponding measures, has led to information fragmentation and data duplication, and stagnation between the different phases of the construction life cycle. events on a construction site are dynamic and constantly changing in terms of facilities, equipment, materials, people, site layout, and various other components at different stages. This increases the complexity of safety monitoring and management of construction sites. We propose the development of a framework for a Digital Twin application based on a BIM model representing the conditions during the construction stage. This is further enriched with hazards and allows to collection of data from sensors on-site to improve the monitoring and recording behaviour of construction workers. This framework can be used to create live alerts to one or multiple workers, contributing to mitigating the number of casualties in the industry. Beyond addressing the gap between on-site operation and office controls, Beyond addressing the gap between on-site operation and office controls, our proposal allows creating a database to analyse incidents and inform decisions based on a on the data stores in the BIM model. This model can also be easily updated to better represent each moment of the construction stage (4D BIM), serving as an important tool for the training of those who collaborate in its physical space