An affordable surround-screen virtual reality display

Building a projection-based virtual reality display is a time, cost, and resource intensive enterprise andmany details contribute to the final display quality. This is especially true for surround-screen displays wheremost of them are one-of-a-kind systems or custom-made installations with specialized projectors, framing, andprojection screens. In general, the costs of acquiring these types of systems have been in the hundreds and evenmillions of dollars, specifically for those supporting synchronized stereoscopic projection across multiple screens.Furthermore, the maintenance of such systems adds an additional recurrent cost, which makes them hard to affordfor a general introduction in a wider range of industry, academic, and research communities.We present a low-cost, easy to maintain surround-screen design based on off-the-shelf affordable componentsfor the projection screens, framing, and display system. The resulting system quality is comparable to significantlymore expensive commercially available solutions. Additionally, users with average knowledge can implement ourdesign and it has the added advantage that single components can be individually upgraded based on necessity aswell as available funds

Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions

Calibration Methods for Head-Tracked 3D Displays

Head-tracked 3D displays can provide a compelling 3D effect, but even small inaccuracies in the calibration of the participant’s viewpoint to the display can disrupt the 3D illusion. We propose a novel interactive procedure for a participant to easily and accurately calibrate a head-tracked display by visually aligning patterns across a multi-screen display. Head-tracker measurements are then calibrated to these known viewpoints. We conducted a user study to evaluate the effectiveness of different visual patterns and different display shapes. We found that the easiest to align shape was the spherical display and the best calibration pattern was the combination of circles and lines. We performed a quantitative camera-based calibration of a cubic display and found visual calibration outperformed manual tuning and generated viewpoint calibrations accurate to within a degree. Our work removes the usual, burdensome step of manual calibration when using head-tracked displays and paves the way for wider adoption of this inexpensive and effective 3D display technology

Projector-Based Augmentation

Projector-based augmentation approaches hold the potential of combining the advantages of well-establishes spatial virtual reality and spatial augmented reality. Immersive, semi-immersive and augmented visualizations can be realized in everyday environments – without the need for special projection screens and dedicated display configurations. Limitations of mobile devices, such as low resolution and small field of view, focus constrains, and ergonomic issues can be overcome in many cases by the utilization of projection technology. Thus, applications that do not require mobility can benefit from efficient spatial augmentations. Examples range from edutainment in museums (such as storytelling projections onto natural stone walls in historical buildings) to architectural visualizations (such as augmentations of complex illumination simulations or modified surface materials in real building structures). This chapter describes projector-camera methods and multi-projector techniques that aim at correcting geometric aberrations, compensating local and global radiometric effects, and improving focus properties of images projected onto everyday surfaces

HOLOGRAPHICS: Combining Holograms with Interactive Computer Graphics

Among all imaging techniques that have been invented throughout the last decades, computer graphics is one of the most successful tools today. Many areas in science, entertainment, education, and engineering would be unimaginable without the aid of 2D or 3D computer graphics. The reason for this success story might be its interactivity, which is an important property that is still not provided efficiently by competing technologies – such as holography. While optical holography and digital holography are limited to presenting a non-interactive content, electroholography or computer generated holograms (CGH) facilitate the computer-based generation and display of holograms at interactive rates [2,3,29,30]. Holographic fringes can be computed by either rendering multiple perspective images, then combining them into a stereogram [4], or simulating the optical interference and calculating the interference pattern [5]. Once computed, such a system dynamically visualizes the fringes with a holographic display. Since creating an electrohologram requires processing, transmitting, and storing a massive amount of data, today’s computer technology still sets the limits for electroholography. To overcome some of these performance issues, advanced reduction and compression methods have been developed that create truly interactive electroholograms. Unfortunately, most of these holograms are relatively small, low resolution, and cover only a small color spectrum. However, recent advances in consumer graphics hardware may reveal potential acceleration possibilities that can overcome these limitations [6]. In parallel to the development of computer graphics and despite their non-interactivity, optical and digital holography have created new fields, including interferometry, copy protection, data storage, holographic optical elements, and display holograms. Especially display holography has conquered several application domains. Museum exhibits often use optical holograms because they can present 3D objects with almost no loss in visual quality. In contrast to most stereoscopic or autostereoscopic graphics displays, holographic images can provide all depth cues—perspective, binocular disparity, motion parallax, convergence, and accommodation—and theoretically can be viewed simultaneously from an unlimited number of positions. Displaying artifacts virtually removes the need to build physical replicas of the original objects. In addition, optical holograms can be used to make engineering, medical, dental, archaeological, and other recordings—for teaching, training, experimentation and documentation. Archaeologists, for example, use optical holograms to archive and investigate ancient artifacts [7,8]. Scientists can use hologram copies to perform their research without having access to the original artifacts or settling for inaccurate replicas. Optical holograms can store a massive amount of information on a thin holographic emulsion. This technology can record and reconstruct a 3D scene with almost no loss in quality. Natural color holographic silver halide emulsion with grain sizes of 8nm is today’s state-of-the-art [14]. Today, computer graphics and raster displays offer a megapixel resolution and the interactive rendering of megabytes of data. Optical holograms, however, provide a terapixel resolution and are able to present an information content in the range of terabytes in real-time. Both are dimensions that will not be reached by computer graphics and conventional displays within the next years – even if Moore’s law proves to hold in future. Obviously, one has to make a decision between interactivity and quality when choosing a display technology for a particular application. While some applications require high visual realism and real-time presentation (that cannot be provided by computer graphics), others depend on user interaction (which is not possible with optical and digital holograms). Consequently, holography and computer graphics are being used as tools to solve individual research, engineering, and presentation problems within several domains. Up until today, however, these tools have been applied separately. The intention of the project which is summarized in this chapter is to combine both technologies to create a powerful tool for science, industry and education. This has been referred to as HoloGraphics. Several possibilities have been investigated that allow merging computer generated graphics and holograms [1]. The goal is to combine the advantages of conventional holograms (i.e. extremely high visual quality and realism, support for all depth queues and for multiple observers at no computational cost, space efficiency, etc.) with the advantages of today’s computer graphics capabilities (i.e. interactivity, real-time rendering, simulation and animation, stereoscopic and autostereoscopic presentation, etc.). The results of these investigations are presented in this chapter

Impacto da calibração num LiDAR baseado em visão estereoscópica

Every year 1.3 million people die due to road accidents. Given that the main culprit is human error, autonomous driving is the path to avert and prevent these numbers. An autonomous vehicle must be able to perceive its surroundings, therefore requiring vision sensors. Of the many kinds of vision sensors available, the three main automotive vision sensors are cameras, RADAR and LiDAR. LiDARs have the unique capability of capturing a high-resolution point cloud, thus enabling 3D object detection. However, current LiDAR technology is still immature and expensive, which makes it unattractive to the automotive market. We propose an alternative LiDAR concept – the LiDART – that is able to generate a point cloud simply resorting to stereoscopic vision and dot projection. LiDART takes advantage of mass-produced components such as a dot pattern projector and a stereoscopic camera rig, thus inherently overcoming problems in cost and maturity. Nonetheless, LiDART has four key challenges: noise, correspondence, centroiding and calibration. This thesis focuses on the calibration aspects of LiDART and aims to investigate the systematic error introduced by standard calibration techniques. In this work, the quality of stereoscopic calibration was assessed both experimentally and numerically. The experimental validation consisted in assembling a prototype and calibrating it using standard calibration techniques for stereoscopic vision. Calibration quality was assessed by estimating the distance to a target. As for numerical assessment, a simulation tool was developed to cross-validate most experimental results. The obtained results show that standard calibration techniques result in a considerable systematic error, reaching 30% of the correct distance. Nonetheless, the estimated error depends monotonically on distance. Consequently, the systematic error can be significantly reduced if better calibration methods, specifically designed for the application at hand, are used in the future.Todos os anos 1.3 milhões de pessoas perdem a vida devido a acidentes de viação. Dado que a principal razão por detrás destes trágicos números é o erro humano, o caminho para prevenir perder tantas vidas passa pela condução autónoma. Um veículo autónomo deve ser capaz de observar o cenário envolvente. Para tal, são necessários sensores de visão. Dos vários sensores de visão disponiveis no mercado, os três principais sensores de visão automotivos são a câmara, o RADAR e o Li- DAR. O LiDAR tem a capacidade única de capturar uma nuvem de pontos com alta resolução, permitindo assim deteção de objetos em 3D. Contudo, a tecnologia por detrás de um LiDAR é atualmente dispendiosa e imatura, o que tem dificultado a adoção por parte de fabricantes de automóveis. Este trabalho propõe um conceito de LiDAR alternativo – o LiDART – capaz de gerar uma nuvem de pontos recorrendo simplesmente a visão estereoscópica e à projeção de pontos. O LiDART tem a vantagem de se basear em componentes produzidos em massa, tais como um projector de pontos e uma câmara estereoscópica, ultrapassando assim os problemas de custo e maturidade. Não obstante, o LiDART tem quatro desafios principais: ruído, correspondência, estimação de centróide e calibração. Esta tese foca-se nas características de calibração do LiDART, tendo como objectivo investigar o erro sistemático introduzido por técnicas de calibração comuns. A qualidade da calibração foi avaliada experimentalmente e numericamente. A validação experimental consistiu em montar um protótipo e calibrá-lo de várias maneiras. A qualidade da calibração foi então avaliada através da estimação da distância a um alvo. Relativamente à parte numérica, desenvolveu-se uma ferramenta de simulação para validar grande parte dos resultados experimentais. Os resultados obtidos mostram que técnicas de calibração comuns resultam num erro sistemático considerável, chegando a 30% da distância correta. Porém, o erro de estimação varia monotonicamente com a distância. Consequentemente, o erro sistemático pode ser reduzido significativamente se melhores métodos de calibração, especialmente pensados para a aplicação em questão, forem aplicados no futuro.Mestrado em Engenharia Eletrónica e Telecomunicaçõe

Flexible calibration of a stereo vision system by active display

Abstract Camera calibration plays a fundamental role for 3D computer vision since it is the first step to recover reliable metric information from 2D images. The calibration of a stereo-vision system is a two-step process: firstly, the calibration of the individual cameras must be carried out, then the two individual calibrations are combined to retrieve the relative placement between the two cameras, and to refine intrinsic and extrinsic parameters. The most commonly adopted calibration methodology uses multiple images of a physical checkerboard pattern. However, the process is time-consuming since the operator must move the calibration target into different positions, typically from 15 to 20. Moreover, the calibration of different optical setups requires the use of calibration boards, which differ for size and number of target points depending on the desired working volume. This paper proposes an innovative approach to the calibration, which is based on the use of a conventional computer screen to actively display the calibration checkerboard. The potential non-planarity of the screen is compensated by an iterative approach, which also estimate the actual screen shape during the calibration process. The use of an active display greatly enhances the flexibility of the stereo-camera calibration process since the same device can be used to calibrate different optical setups by simply varying number and size of the displayed squared patterns

MistForm: adaptive shape changing fog screens

We present MistForm, a shape changing fog display that can support one or two users interacting with either 2D or 3D content. Mistform combines affordances from both shape changing interfaces and mid-air displays. For example, a concave display can maintain content in comfortable reach for a single user, while a convex shape can support several users engaged on individual tasks. MistForm also enables unique interaction possibilities by exploiting the synergies between shape changing interfaces and mid-air fog displays. For instance, moving the screen will affect the brightness and blurriness of the screen at specific locations around the display, creating spaces with similar (collaboration) or different visibility (personalized content). We describe the design of MistForm and analyse its inherent challenges, such as image distortion and uneven brightness on dynamic curved surfaces. We provide a machine learning approach to characterize the shape of the screen and a rendering algorithm to remove aberrations. We finally explore novel interactive possibilities and reflect on their potential and limitations

INFORMATION TECHNOLOGY FOR NEXT-GENERATION OF SURGICAL ENVIRONMENTS

Minimally invasive surgeries (MIS) are fundamentally constrained by image quality,access to the operative field, and the visualization environment on which thesurgeon relies for real-time information. Although invasive access benefits the patient,it also leads to more challenging procedures, which require better skills andtraining. Endoscopic surgeries rely heavily on 2D interfaces, introducing additionalchallenges due to the loss of depth perception, the lack of 3-Dimensional imaging,and the reduction of degrees of freedom.By using state-of-the-art technology within a distributed computational architecture,it is possible to incorporate multiple sensors, hybrid display devices, and3D visualization algorithms within a exible surgical environment. Such environmentscan assist the surgeon with valuable information that goes far beyond what iscurrently available. In this thesis, we will discuss how 3D visualization and reconstruction,stereo displays, high-resolution display devices, and tracking techniques arekey elements in the next-generation of surgical environments