70 research outputs found
Creating 3D models using reconstruction techniques
Thesis (MSc)--Stellenbosch University, 2018.ENGLISH ABSTRACT :Virtual reality models of real world environments have a number of compelling
applications, such as preserving the architecture and designs of older buildings. This process can be achieved by using 3D artists to reconstruct the
environment, however this is a long and expensive process. Thus, this thesis
investigates various techniques and approaches used in 3D reconstruction of
environments using a single RGB-D camera and aims to reconstruct the 3D
environment to generate a 3D model. This would allow non-technical users to
reconstruct environments and use these models in business and simulations,
such as selling real-estate, modifying pre-existing structures for renovation and
planning. With the recent improvements in virtual reality technology such as
the Oculus Rift and HTC Vive, a user can be immersed into virtual reality
environments created from real world structures. A system based on Kinect
Fusion is implemented to reconstruct an environment and track the motion of
the camera within the environment. The system is designed as a series of selfcontained subsystems that allows for each of the subsystems to be modified,
expanded upon or easily replaced by alternative methods. The system is made
available as an open source C++ project using Nvidia’s CUDA framework to
aid reproducibility and provides a platform for future research. The system
makes use of the Kinect sensor to capture information about the environment.
A coarse-to-fine least squares approach is used to estimate the motion of the
camera. In addition, the system employs a frame-to-model approach that uses
a view of the estimated reconstruction of the model as the reference frame and
the incoming scene data as the target. This minimises the drift with respect
to the true trajectory of the camera. The model is built using a volumetric
approach, with volumetric information implicitly stored as a truncated signed
distance function. The system filters out noise in the raw sensor data by using a bilateral filter. A point cloud is extracted from the volume using an orthogonal ray caster which enables an improved hole-filling approach. This
allows the system to extract both the explicit and implicit structure from the
volume. The 3D reconstruction is followed by mesh generation based on the
point cloud. This is achieved by using an approach related to Delaunay triangulation, the ball-pivot algorithm. The resulting system processes frames at
30Hz, enabling real-time point cloud generation, while the mesh generation occurs offline. This system is initially tested using Blender to generate synthetic
data, followed by a series of real world tests. The synthetic data is used to test
the presented system’s motion tracking against the ground truth. While the
presented system suffers from the effects of drift over long frame sequences, it
is shown to be capable of tracking the motion of the camera. This thesis finds
that the ball pivot algorithm can generate the edges and faces for synthetic
point clouds, however it performs poorly when using the noisy synthetic and
real world data sets. Based on the results obtained it is recommended that the
obtained point cloud be preprocessed to remove noise before it is provided to
the mesh generation algorithm and an alternative mesh generation technique
should be employed that is more robust to noise.AFRIKAANSE OPSOMMING : Modelle in virtuele realiteit van werklike omgewings het ’n aantal belangrike
toepassings, soos byvoorbeeld die behoud van die argitektuur en ontwerpe van
geskiedkundig belangrike geboue. 3D kunstenaars kan ingespan word om omgewings te modelleer, maar dit is ’n lang en duur proses. Hierdie proefskrif
ondersoek verskillende tegnieke en benaderings wat gebruik word in die 3D
rekonstruksie van omgewings deur gebruik van ’n enkele RGB-D kamera en
beoog om die 3D rekonstruksie van die omgewing te omskep in ’n 3D-model.
Hierdie sal nie-tegniese gebruikers toelaat om self modelle te skep van omgewings, en om hierdie modelle te gebruik in besigheid toepassings en simulasies,
soos byvoorbeeld die verkoop van vaste eiendom, die wysiging van bestaande
strukture vir beplanning en opknapping. Met die onlangse tegnologiese verbeteringe in die veld van virtuele realiteit soos, byvoorbeeld, die Oculus Rift
en HTC Vive, kan ’n gebruiker geplaas word in ’n virtuele omgewing wat geskep was vanaf strukture in die werklike wˆereld. ’n Stelsel gebaseer op Kinect
Fusion word ge¨ımplementeer om ’n omgewing te rekonstrueer en die beweging
van die kamera binne die omgewing te volg. Die stelsel is ontwerp as ’n reeks
selfstandige modules wat die afsonderlike aanpassing, uitbreiding of vervanging
van die modules vergemaklik. Die stelsel word beskikbaar gestel as ’n open
source C++ projek met behulp van Nvidia se CUDA raamwerk om reproduseerbaarheid te bevorder en bied ook ’n platform vir toekomstige navorsing.
Die stelsel maak gebruik van die Kinect-sensor om inligting oor die omgewing
vas te vang. ’n Grof-tot-fyn kleinste kwadraat benadering word gebruik om
die beweging van die kamera te skat. Daarbenewens gebruik die stelsel ’n
beeld-tot-model benadering wat gebruik maak van die beraamde rekonstruksie van die model as die verwysingsraamwerk en die inkomende toneel data as
die teiken. Dit verminder die drywing ten opsigte van die ware trajek van die
kamera. Die model word gebou met behulp van ’n volumetriese benadering,met volumetriese inligting wat implisiet gestoor word as ’n verkorte getekende
afstandfunksie. Die stelsel filter ruis in die ruwe sensor data uit deur om ’n
bilaterale filter te gebruik. ’n Puntwolk word uit die volume onttrek deur ’n ortogonale straalvolger te gebruik wat die vul van gate in die model verbeter. Dit
laat die stelsel toe om die eksplisiete en implisiete struktuur van die volume te
onttrek. Die 3D-rekonstruksie word gevolg deur maasgenerasie gebaseer op die
puntwolk. Dit word behaal deur ’n benadering wat verband hou met Delaunay
triangulasie, die bal wentelings algoritme, te gebruik. Die resulterende stelsel verwerk beelde teen 30Hz, wat intydse-puntwolkgenerasie moontlik maak,
terwyl die maasgenerering aflyn plassvind. Hierdie stelsel word aanvanklik getoets deur om met Blender sintetiese data te genereer, gevolg deur ’n reeks
werklike wˆereldtoetse. Die sintetiese data word gebruik om die stelsel se afgeskatte trajek teenoor die korrekte trajek te vergelyk. Terwyl die stelsel ly aan
die effekte van wegdrywing oor langdurige intreevideos, word dit getoon dat
die stelsel wel die lokale beweging van die kamera kan volg. Hierdie proefskrif
bevind dat die bal wentelingsalgoritme die oppervlaktes en bygaande rande
vir sintetiese puntwolke kan genereer, maar dit is sterk gevoelig vir ruis in
sintetiese en werklike datastelle. Op grond van die resultate wat verkry word,
word aanbeveel dat die verkrygde puntwolk vooraf verwerk word om ruis te
verwyder voordat dit aan die maasgenereringsalgoritme verskaf word, en ’n
alternatiewe maasgenereringstegniek moet gebruik word wat meer robuust is
ten opsigte van ruis
Recommended from our members
3D (embodied) projection mapping and sensing bodies : a study in interactive dance performance
This dissertation identifies the synergies between physical and virtual environments when designing for immersive experiences in interactive dance performances. The integration of virtual information in physical space is transforming our interactions and experiences with the world. By using the body and creative expression as the interface between real and virtual worlds, dance performance creates a privileged framework to research and design interactive mixed reality environments and immersive augmented architectures. The research is primarily situated in the fields of visual art and interaction design. It combines performance with transdisciplinary fields and intertwines practice with theory. The theoretical and conceptual implications involved in designing and experiencing immersive hybrid environments are analyzed using the reality–virtuality continuum. These theories helped frame the ways augmented reality architectures are achieved through the integration of dance performance with digital software and reception displays. They also helped identify the main artistic affordances and restrictions in the design of augmented reality and augmented virtuality environments for live performance. These pervasive media architectures were materialized in three field experiments, the live dance performances. Each performance was created in three different stages of conception, design and production. The first stage was to “digitize” the performer’s movement and brain activity to the virtual environment and our system. This was accomplished through the use of depth sensor cameras, 3D motion capture, and brain computer interfaces. The second stage was the creation of the computational architecture and software that aggregates the connections and mapping between the physical body and the spatial dynamics of the virtual environment. This process created real-time interactions between the performer’s behavior and motion and the real-time generative computer 3D graphics. Finally, the third stage consisted of the output modality: 3D projector based augmentation techniques were adopted in order to overlay the virtual environment onto physical space. This thesis proposes and lays out theoretical, technical, and artistic frameworks between 3D digital environments and moving bodies in dance performance. By sensing the body and the brain with the 3D virtual environments, new layers of augmentation and interactions are established, and ultimately this generates mixed reality environments for embodied improvisational self-expression.Radio-Television-Fil
Towards 3D Scanning from Digital Images by Novice Users
The uptake of hobbyist 3D printers is being held back, in part, due to the barriers associated with creating a computer model to be printed. One way of creating such a computer model is to take a 3D scan of a pre-existing object using multiple digital images of the object showing the object from different points of view. This document details one way of doing this, with particular emphasis on camera calibration: the process of estimating camera parameters for the camera that took an image.
In common calibration scenarios, multiple images are used where it is assumed that the internal parameters, such as zoom and focus settings, are fixed between images and the relative placement of the camera between images needs to be estimated. This is not ideal for a novice doing 3D scanning with a “point and shoot” camera where these internal parameters may not have been held fixed between images. A common coordinate system between images with a known relationship to real-world measurements is also desirable.
Additionally, in some 3D scanning scenarios that use digital images, where it is expected that a trained individual will be doing the photography and internal settings can be held constant throughout the process, the images used for doing the calibration are different from those that are used to do the object capture.
A technique has been developed to overcome these shortcomings. It uses a known printed sheet of paper, called the calibration sheet, that the object to be scanned sits on so that object acquisition and camera calibration can be done from the same image. Each image is processed independently with reference to the known size of the calibration sheet so the output is automatically to scale and minor camera calibration errors with one image do not propagate and affect estimates of camera calibration parameters for other images. The calibration process developed is also one that will work where large parts of the calibration sheet are obscured
Towards Real-time Mixed Reality Matting In Natural Scenes
In Mixed Reality scenarios, background replacement is a common way to immerse a user in a synthetic environment. Properly identifying the background pixels in an image or video is a dif- ficult problem known as matting. Proper alpha mattes usually come from human guidance, special hardware setups, or color dependent algorithms. This is a consequence of the under-constrained nature of the per pixel alpha blending equation. In constant color matting, research identifies and replaces a background that is a single color, known as the chroma key color. Unfortunately, the algorithms force a controlled physical environment and favor constant, uniform lighting. More generic approaches, such as natural image matting, have made progress finding alpha matte solutions in environments with naturally occurring backgrounds. However, even for the quicker algorithms, the generation of trimaps, indicating regions of known foreground and background pixels, normally requires human interaction or offline computation. This research addresses ways to automatically solve an alpha matte for an image in realtime, and by extension a video, using a consumer level GPU. It does so even in the context of noisy environments that result in less reliable constraints than found in controlled settings. To attack these challenges, we are particularly interested in automatically generating trimaps from depth buffers for dynamic scenes so that algorithms requiring more dense constraints may be used. The resulting computation is parallelizable so that it may run on a GPU and should work for natural images as well as chroma key backgrounds. Extra input may be required, but when this occurs, commodity hardware available in most Mixed Reality setups should be able to provide the input. This allows us to provide real-time alpha mattes for Mixed Reality scenarios that take place in relatively controlled environments. As a consequence, while monochromatic backdrops (such as green screens or retro-reflective material) aid the algorithm’s accuracy, they are not an explicit requirement. iii Finally we explore a sub-image based approach to parallelize an existing hierarchical approach on high resolution imagery. We show that locality can be exploited to significantly reduce the memory and compute requirements of previously necessary when computing alpha mattes of high resolution images. We achieve this using a parallelizable scheme that is both independent of the matting algorithm and image features. Combined, these research topics provide a basis for Mixed Reality scenarios using real-time natural image matting on high definition video sources
Recommended from our members
Mobile depth sensing technology and algorithms with application to occupational therapy healthcare
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonThe UK government is striving to shift its current healthcare delivery model from clini-cian–oriented services, to that of patient and self–care–oriented intervention strategies. It seeks to do so through Information Communication (ICT) and Computer Mediated Re-ality Technologies (CMRT) as a key strategy to overcome the ever–increasing scarcity of healthcare resources and costs. To this end, in the UK the use of paper–based information systems have exhibited their limitations in providing apposite care. At the national level, The Royal College of Occupational Therapists (RCOT) identify home visits and modifica-tions as key levers in a multifactorial health programme to evaluate interventions for older people with a history of falling or are identified as being prone to falling. Prescribing Assistive Equipment (AE) is one such mechanism that seeks to reduce the risk of falling whilst promoting the continued independence of physical dexterity and mobility in older adults at home. In the UK, the yearly cost of falls is estimated at £2.3 billion. Further evidence places a 30% to 60% abandonment rate on prescribed AE by and large due to a ‘poor fit’ and measurement inaccuracies.
To remain aligned with the national strategy, and assist in the eradication of measurement inaccuracies, this thesis employs Mobile Depth Sensing and Motion Track-ing Devices (MDSMTDs) to assist OTs in in the process of digitally measuring the extrin-sic fall–risk factors for the provision of AE. The quintessential component in this assess-ment lies in the measurement of fittings and furniture items in the home. To digitise and aid in this process, the artefact presented in this thesis employs stereo computer–vision and camera calibration algorithms to extract edges in 3D space. It modifies the Sobel–Feldman convolution filter by reducing the magnitude response and employs the camera intrinsic parameters as a mechanism to calculate the distortion matrix for interpolation between the edges and the 3D point cloud. Further Augmented Reality User Experience (AR-UX) facets are provided to digitise current state of the art clinical guidance and over-lay its instructions onto the real world (i.e., 3D space).
Empirical mixed methods assessment revealed that in terms of accuracy, the arte-fact exhibited enhanced performance gains over current paper–based guidance. In terms of accuracy consistency, the artefact can rectify measurement consistency inaccuracies, but there are still a wide range of factors that can influence the integrity of the point-cloud in respect of the device’s point-of-view, holding positions and measurement speed. To this end, OTs usability, and adoption preferences materialise in favour of the artefact. In conclusion, this thesis demonstrates that MDSMTDs are a promising alterna-tive to existing paper–based measurement practices as OTs appear to prefer the digital–based system and that they can take measurements more efficiently and accurately
- …