Compact Lens Assembly for the Teleportal Augmented Reality System

A compact, ultra-light and high-performance projection optics lens assembly using Diffractive Optics Technology, plastic and glass optics, and Aspheric optics has been designed with only four elements. The lens may be included as is, or scaled to be included in all instruments using conventional Double-Gauss lens forms. The preferred lens is only 20 mm in diameter and 15 mm long has a weight of only 8 g. Thus, two lenses for head mounted stereo displays is only 16 g. Such a stereo display, known as a head-mounted projective display (HMPD), consists of a pair of miniature projection lenses, beam splitters, and miniature displays mounted on the helmet, and retro-reflective sheeting materials placed strategically in the environment

Head Mounted Projection Display with a Wide Field of View

A ultra-wide field of view head mounted display has been realized by integrating an ARC display component having a greater than about 70 degrees field of retro-reflection with an optical tiling display which provides a greater than about 80 degrees horizontal field of view by about 50 degrees vertical field of view whereby an overall binocular horizontal field of view greater than about 120 degrees is realized with a greater resolution than about 2 arc minutes. There is also taught a method of providing a wide field of view head mounted display by the steps of: combining an ARC display component and an optical tiling display; and, integrating said component and said tiling display whereby an overall field of view greater than about 120 degrees is realized

A Wearable Head-mounted Projection Display

Conventional head-mounted projection displays (HMPDs) contain of a pair of miniature projection lenses, beamsplitters, and miniature displays mounted on the helmet, as well as a retro-reflective screen placed strategically in the environment. We have extened the HMPD technology integrating the screen into a fully mobile embodiment. Some initial efforts of demonstrating this technology has been captured followed by an investigation of the diffraction effects versus image degradation caused by integrating the retro-reflective screen within the HMPD. The key contribution of this research is the conception and development of a mobileHMPD (M-HMPD). We have included an extensive analysis of macro- and microscopic properties that encompass the retro-reflective screen. Furthermore, an evaluation of the overall performance of the optics will be assessed in both object space for the optical designer and visual space for the possible users of this technology. This research effort will also be focused on conceiving a mobile M-HMPD aimed for dual indoor/outdoor applications. The M-HMPD shares the known advantage such as ultralightweight optics (i.e. 8g per eye), unperceptible distortion (i.e. ≤ 2.5%), and lightweight headset (i.e. ≤ 2.5 lbs) compared with eyepiece type head-mounted displays (HMDs) of equal eye relief and field of view. In addition, the M-HMPD also presents an advantage over the preexisting HMPD in that it does not require a retro-reflective screen placed strategically in the environment. This newly developed M-HMPD has the ability to project clear images at three different locations within near- or far-field observation depths without loss of image quality. This particular M-HMPD embodiment was targeted to mixed reality, augmented reality, and wearable display applications

Development of a Training Tool for Endotracheal Intubation: Distributed Augmented Reality

The authors introduce a tool referred to as the Ultimate Intubation Head (UIH) to train medical practitioners’ hand-eye coordination in performing endotracheal intubation with the help of augmented reality methods. In this paper we describe the integration of a deployable UIH and present methods for augmented reality registration of real and virtual anatomical models. The assessment of the 52 degrees field of view optics of the custom-designed and built head-mounted display is less than 1.5 arc minutes in the amount of blur and astigmatism, the two limiting optical aberrations. Distortion is less than 2.5%. Preliminary results of the registration of a physical phantom mandible on its virtual counterpart yields less than 3mm rms. in registration. Finally we describe an approach to distributed visualization where a given training procedure may be visualized and shared at various remote locations. Basic assessments of delays within two scenarios of data distribution were conducted and reported

Optical versus video see-through mead-mounted displays in medical visualization

We compare two technological approaches to augmented reality for 3-D medical visualization: optical and video see-through devices. We provide a context to discuss the technology by reviewing several medical applications of augmented-reality research efforts driven by real needs in the medical field, both in the United States and in Europe. We then discuss the issues for each approach, optical versus video, from both a technology and human-factor point of view. Finally, we point to potentially promising future developments of such devices including eye tracking and multifocus planes capabilities, as well as hybrid optical/video technology

Scalable multi-view stereo camera array for real world real-time image capture and three-dimensional displays

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2004.Includes bibliographical references (leaves 71-75).The number of three-dimensional displays available is escalating and yet the capturing devices for multiple view content are focused on either single camera precision rigs that are limited to stationary objects or the use of synthetically created animations. In this work we will use the existence of inexpensive digital CMOS cameras to explore a multi- image capture paradigm and the gathering of real world real-time data of active and static scenes. The capturing system can be developed and employed for a wide range of applications such as portrait-based images for multi-view facial recognition systems, hypostereo surgical training systems, and stereo surveillance by unmanned aerial vehicles. The system will be adaptable to capturing the correct stereo views based on the environmental scene and the desired three-dimensional display. Several issues explored by the system will include image calibration, geometric correction, the possibility of object tracking, and transfer of the array technology into other image capturing systems. These features provide the user more freedom to interact with their specific 3-D content while allowing the computer to take on the difficult role of stereoscopic cinematographer.Samuel L. Hill.S.M

Scene understanding through semantic image segmentation in augmented reality

Abstract. Semantic image segmentation, the task of assigning a label to each pixel in an image, is a major challenge in the field of computer vision. Semantic image segmentation using fully convolutional neural networks (FCNNs) offers an online solution to the scene understanding while having a simple training procedure and fast inference speed if designed efficiently. The semantic information provided by the semantic segmentation is a detailed understanding of the current context and this scene understanding is vital for scene modification in augmented reality (AR), especially if one aims to perform destructive scene augmentation. Augmented reality systems, by nature, aim to have a real-time modification of the context through head-mounted see-through or video-see-through displays, thus require efficiency in each step. Although there are many solutions to the semantic image segmentation in the literature such as DeeplabV3+, Deeplab DPC, they fail to offer a low latency inference due to their complex architectures in aim to acquire the best accuracy. As a part of this thesis work, we provide an efficient architecture for semantic image segmentation using an FCNN model and achieve real-time performance on smartphones at 19.65 frames per second (fps) while maintaining a high mean intersection over union (mIOU) of 67.7% on Cityscapes validation set with our "Basic" variant and 15.41 fps and 70.3% mIOU on Cityscapes test set using our "DPC" variant. The implementation is open-sourced and compatible with Tensorflow Lite, thus able to run on embedded and mobile devices. Furthermore, the thesis work demonstrates an augmented reality implementation where semantic segmentation masks are tracked online in a 3D environment using Google ARCore. We show that the frequent calculation of semantic information is not necessary and by tracking the calculated semantic information in 3D space using inertial-visual odometry that is provided by the ARCore framework, we can achieve savings on battery and CPU usage while maintaining a high mIOU. We further demonstrate a possible use case of the system by inpainting the objects in 3D space that are found by the semantic image segmentation network. The implemented Android application performs real-time augmented reality at 30 fps while running the computationally efficient network that was proposed as a part of this thesis work in parallel

투명한 매질에서의 광 경로 분석을 이용한 집약적 3차원 디스플레이

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2017. 2. 이병호.본 박사학위 논문에서는 광학적으로 투명한 매질에서의 광 경로 분석을 바탕으로 집약적인 3차원 디스플레이 시스템을 구현하는 접근 방법에 대하여 논의한다. 3차원 영상 장치를 구성하는 요소와 시청자 사이의 물리적인 거리를 줄이는 것은 집약적인 3차원 디스플레이 시스템을 구현하는 직관적인 방법이다. 또한, 기존 시스템의 크기를 유지하면서 더 많은 양의 3차원 영상 정보를 표현하는 것 또한 집약적 3차원 디스플레이 시스템을 의미한다. 높은 대역폭과 작은 구조를 가진 집약적 3차원 디스플레이 시스템을 구현하기 위하여 다음의 두 가지 광학 현상을 이용한다. 등방성 물질에서의 전반사 특성과 이방성 물질에서의 복굴절 특성이다. 가시광 영역에서 빛을 투과시키는 두 매질의 고유 광학 특성을 기존의 3차원 디스플레이 시스템에 적용하기 위하여 광 경로 추적을 통하여 분석한다. 광 도파로의 전반사 특성은 집약적 다중 투사 3차원 디스플레이 시스템을 구현하기 위하여 사용한다. 투사 광학계의 영상 정보는 광 도파로로 입사, 내부에서 전반사를 통하여 진행하고, 이에 수평 투사 거리는 광 도파로의 두께로 제한된다. 다수의 전반사 이후 영상 정보는 광 도파로의 출사 면을 통해 빠져나가고, 렌즈에 의하여 최적 시청 지점에서 시점을 형성한다. 광 도파로 내부에서의 광 경로를 등가 모델을 통하여 조사하고, 이를 통해 다수의 투사 광학계로부터 생성된 다수의 시점 영상이 왜곡되는 것을 분석하고 보정한다. 10개의 시점을 제공하는 집약적 다중 투사 3차원 디스플레이 시스템을 통해 제안된 방법을 검증한다. 향상된 대역폭 특성을 가진 다중 투사 3차원 디스플레이와 다중 초점 헤드 마운트 디스플레이 구현을 위한 이방성 판을 이용한 편광 다중화 방법을 제안한다. 빛의 편광 상태, 이방성 판의 광축 방향에 따라 광 경로가 달라진다. 측면 방향으로의 광 경로 전환은 다중 투사 3차원 디스플레이 기술과 결합하여 시점을 측면 방향으로 두 배로 증가시킨다. 깊이 방향으로의 광 경로 전환은 헤드 마운트 디스플레이에서 다중 초점 기능을 구현한다. 광 경로 추적 시뮬레이션을 통해 이방성 판의 모양, 광축, 파장 등의 다양한 파라미터 변화에 따른 광 경로 전환을 분석한다. 각각의 기능에 맞도록 설계된 이방성 판과 편광 회전자를 실시간으로 결합하여, 다중 투사 3차원 디스플레이와 다중 초점 헤드 마운트 디스플레이의 대역폭이 2배 증가한다. 각 시스템에 대한 시작품을 제작하고, 제안된 방법을 실험적으로 검증한다. 본 논문에서는 광 도파로와 복굴절 물질을 이용하여 그 광 경로를 분석, 대형의 다중 투사 3차원 디스플레이 시스템과 개인 사용자의 헤드 마운트 디스플레이 시스템의 크기를 감소시키고, 표현 가능한 정보량을 증가시키는 방법을 제안한다. 광 도파로와 이방성 판은 기존의 3차원 디스플레이 시스템과 쉽게 결합이 가능하며, 제안된 방법은 향후 소형뿐만 아니라 중대형 3차원 디스플레이 시스템의 집약화에 기여할 수 있을 것으로 기대된다.This dissertation investigates approaches for realizing compact three-dimensional (3D) display systems based on optical path analysis in optically transparent medium. Reducing the physical distance between 3D display apparatuses and an observer is an intuitive method to realize compact 3D display systems. In addition, it is considered compact 3D display systems when they present more 3D data than conventional systems while preserving the size of the systems. For implementing compact 3D display systems with high bandwidth and minimized structure, two optical phenomena are investigated: one is the total internal reflection (TIR) in isotropic materials and the other is the double refraction in birefringent crystals. Both materials are optically transparent in visible range and ray tracing simulations for analyzing the optical path in the materials are performed to apply the unique optical phenomenon into conventional 3D display systems. An optical light-guide with the TIR is adopted to realize a compact multi-projection 3D display system. A projection image originated from the projection engine is incident on the optical light-guide and experiences multiple folds by the TIR. The horizontal projection distance of the system is effectively reduced as the thickness of the optical light-guide. After multiple folds, the projection image is emerged from the exit surface of the optical light-guide and collimated to form a viewing zone at the optimum viewing position. The optical path governed by the TIR is analyzed by adopting an equivalent model of the optical light-guide. Through the equivalent model, image distortion for multiple view images in the optical light-guide is evaluated and compensated. For verifying the feasibility of the proposed system, a ten-view multi-projection 3D display system with minimized projection distance is implemented. To improve the bandwidth of multi-projection 3D display systems and head-mounted display (HMD) systems, a polarization multiplexing technique with the birefringent plate is proposed. With the polarization state of the image and the direction of optic axis of the birefringent plate, the optical path of rays varies in the birefringent material. The optical path switching in the lateral direction is applied in the multi-projection system to duplicate the viewing zone in the lateral direction. Likewise, a multi-focal function in the HMD is realized by adopting the optical path switching in the longitudinal direction. For illuminating the detailed optical path switching and the image characteristic such as an astigmatism and a color dispersion in the birefringent material, ray tracing simulations with the change of optical structure, the optic axis, and wavelengths are performed. By combining the birefringent material and a polarization rotation device, the bandwidth of both the multi-projection 3D display and the HMD is doubled in real-time. Prototypes of both systems are implemented and the feasibility of the proposed systems is verified through experiments. In this dissertation, the optical phenomena of the TIR and the double refraction realize the compact 3D display systems: the multi-projection 3D display for public and the multi-focal HMD display for individual. The optical components of the optical light-guide and the birefringent plate can be easily combined with the conventional 3D display system and it is expected that the proposed method can contribute to the realization of future 3D display systems with compact size and high bandwidth.Chapter 1 Introduction 10 1.1 Overview of modern 3D display providing high quality 3D images 10 1.2 Motivation of this dissertation 15 1.3 Scope and organization 18 Chapter 2 Compact multi-projection 3D displays with optical path analysis of total internal reflection 20 2.1 Introduction 20 2.2 Principle of compact multi-projection 3D display system using optical light-guide 23 2.2.1 Multi-projection 3D display system 23 2.2.2 Optical light-guide for multi-projection 3D display system 26 2.2.3 Analysis on image characteristics of projection images in optical light-guide 34 2.2.4 Pre-distortion method for view image compensation 44 2.3 Implementation of prototype of multi-projection 3D display system with reduced projection distance 47 2.4 Summary and discussion 52 Chapter 3 Compact multi-projection 3D displays with optical path analysis of double refraction 53 3.1 Introduction 53 3.2 Principle of viewing zone duplication in multi-projection 3D display system 57 3.2.1 Polarization-dependent optical path switching in birefringent crystal 57 3.2.2 Analysis on image formation through birefringent plane-parallel plate 60 3.2.3 Full-color generation of dual projection 64 3.3 Implementation of prototype of viewing zone duplication of multi-projection 3D display system 68 3.3.1 Experimental setup for viewing zone duplication of multi-projection 3D display system 68 3.3.2 Luminance distribution measurement of viewing zone duplication of multi-projection 3D display system 74 3.4 Summary and discussion 79 Chapter 4 Compact multi-focal 3D HMDs with optical path analysis of double refraction 81 4.1 Introduction 81 4.2 Principle of multi-focal 3D HMD system 86 4.2.1 Multi-focal 3D HMD system using Savart plate 86 4.2.2 Astigmatism compensation by modified Savart plate 89 4.2.3 Analysis on lateral chromatic aberration of extraordinary plane 96 4.2.4 Additive type compressive light field display 101 4.3 Implementation of prototype of multi-focal 3D HMD system 104 4.4 Summary and discussion 112 Chapter 5 Conclusion 114 Bibliography 117 Appendix 129 초 록 130Docto

Towards Highly-Integrated Stereovideoscopy for \u3ci\u3ein vivo\u3c/i\u3e Surgical Robots

When compared to traditional surgery, laparoscopic procedures result in better patient outcomes: shorter recovery, reduced post-operative pain, and less trauma to incisioned tissue. Unfortunately, laparoscopic procedures require specialized training for surgeons, as these minimally-invasive procedures provide an operating environment that has limited dexterity and limited vision. Advanced surgical robotics platforms can make minimally-invasive techniques safer and easier for the surgeon to complete successfully. The most common type of surgical robotics platforms -- the laparoscopic robots -- accomplish this with multi-degree-of-freedom manipulators that are capable of a diversified set of movements when compared to traditional laparoscopic instruments. Also, these laparoscopic robots allow for advanced kinematic translation techniques that allow the surgeon to focus on the surgical site, while the robot calculates the best possible joint positions to complete any surgical motion. An important component of these systems is the endoscopic system used to transmit a live view of the surgical environment to the surgeon. Coupled with 3D high-definition endoscopic cameras, the entirety of the platform, in effect, eliminates the peculiarities associated with laparoscopic procedures, which allows less-skilled surgeons to complete minimally-invasive surgical procedures quickly and accurately. A much newer approach to performing minimally-invasive surgery is the idea of using in-vivo surgical robots -- small robots that are inserted directly into the patient through a single, small incision; once inside, an in-vivo robot can perform surgery at arbitrary positions, with a much wider range of motion. While laparoscopic robots can harness traditional endoscopic video solutions, these in-vivo robots require a fundamentally different video solution that is as flexible as possible and free of bulky cables or fiber optics. This requires a miniaturized videoscopy system that incorporates an image sensor with a transceiver; because of severe size constraints, this system should be deeply embedded into the robotics platform. Here, early results are presented from the integration of a miniature stereoscopic camera into an in-vivo surgical robotics platform. A 26mm X 24mm stereo camera was designed and manufactured. The proposed device features USB connectivity and 1280 X 720 resolution at 30 fps. Resolution testing indicates the device performs much better than similarly-priced analog cameras. Suitability of the platform for 3D computer vision tasks -- including stereo reconstruction -- is examined. The platform was also tested in a living porcine model at the University of Nebraska Medical Center. Results from this experiment suggest that while the platform performs well in controlled, static environments, further work is required to obtain usable results in true surgeries. Concluding, several ideas for improvement are presented, along with a discussion of core challenges associated with the platform. Adviser: Lance C. Pérez [Document = 28 Mb