Deep-Learning-Incorporated Augmented Reality Application for Engineering Lab Training

Deep learning (DL) algorithms have achieved significantly high performance in object detection tasks. At the same time, augmented reality (AR) techniques are transforming the ways that we work and connect with people. With the increasing popularity of online and hybrid learning, we propose a new framework for improving students’ learning experiences with electrical engineering lab equipment by incorporating the abovementioned technologies. The DL powered automatic object detection component integrated into the AR application is designed to recognize equipment such as multimeter, oscilloscope, wave generator, and power supply. A deep neural network model, namely MobileNet-SSD v2, is implemented for equipment detection using TensorFlow’s object detection API. When a piece of equipment is detected, the corresponding AR-based tutorial will be displayed on the screen. The mean average precision (mAP) of the developed equipment detection model is 81.4%, while the average recall of the model is 85.3%. Furthermore, to demonstrate practical application of the proposed framework, we develop a multimeter tutorial where virtual models are superimposed on real multimeters. The tutorial includes images and web links as well to help users learn more effectively. The Unity3D game engine is used as the primary development tool for this tutorial to integrate DL and AR frameworks and create immersive scenarios. The proposed framework can be a useful foundation for AR and machine-learning-based frameworks for industrial and educational training

Situated storytelling with SLAM enabled augmented reality

This paper addresses the feasibility of situated storytelling using Simultaneous Localisation and Mapping (SLAM) enabled augmented reality (AR) on a mobile phone. We specifically focus on storytelling in the heritage context as it provides a rich environment for stories to be told in. We conducted expert interviews with several museum and heritage sites to identify major themes for storytelling in the heritage context. These themes informed the development of an AR based storytelling application for a mobile phone. We evaluated the application in a user study and gained further insight into the factors that users appreciate in AR based storytelling. From these insights we derive several high level design guidelines that may inform future system development for situated storytelling, especially in the heritage context