Recent trends, technical concepts and components of computer-assisted orthopedic surgery systems: A comprehensive review

Computer-assisted orthopedic surgery (CAOS) systems have become one of the most important and challenging types of system in clinical orthopedics, as they enable precise treatment of musculoskeletal diseases, employing modern clinical navigation systems and surgical tools. This paper brings a comprehensive review of recent trends and possibilities of CAOS systems. There are three types of the surgical planning systems, including: systems based on the volumetric images (computer tomography (CT), magnetic resonance imaging (MRI) or ultrasound images), further systems utilize either 2D or 3D fluoroscopic images, and the last one utilizes the kinetic information about the joints and morphological information about the target bones. This complex review is focused on three fundamental aspects of CAOS systems: their essential components, types of CAOS systems, and mechanical tools used in CAOS systems. In this review, we also outline the possibilities for using ultrasound computer-assisted orthopedic surgery (UCAOS) systems as an alternative to conventionally used CAOS systems.Web of Science1923art. no. 519

Identifying Problems Associated with Focus and Context Awareness in 3D Modelling Tasks

Creating complex 3D models is a challenging process. One of the main reasons for this is that 3D models are usually created using software developed for conventional 2D displays which lack true depth perspective, and therefore do not support correct perception of spatial placement and depth-ordering of displayed content. As a result, modellers often have to deal with many overlapping components of 3D models (e.g. vertices, edges, faces, etc.) on a 2D display surface. This in turn causes them to have difficulties in distinguishing distances, maintaining position and orientation awareness, etc. To better understand the nature of these problems, which can collectively be defined as ‘focus and context awareness’ problems, we have conducted a pilot study with a group of novice 3D modellers, and a series of interviews with a group of professional 3D modellers. This article presents these two studies, and their findings, which have resulted in identifying a set of focus and context awareness problems that modellers face in creating 3D models using conventional modelling software. The article also provides a review of potential solutions to these problems in the related literature

The wildfire dataset: enhancing deep learning-based forest fire detection with a diverse evolving open-source dataset focused on data representativeness and a novel multi-task learning approach

This study explores the potential of RGB image data for forest fire detection using deep learning models, evaluating their advantages and limitations, and discussing potential integration within a multi-modal data context. The research introduces a uniquely comprehensive wildfire dataset, capturing a broad array of environmental conditions, forest types, geographical regions, and confounding elements, aiming to reduce high false alarm rates in fire detection systems. To ensure integrity, only public domain images were included, and a detailed description of the dataset’s attributes, URL sources, and image resolutions is provided. The study also introduces a novel multi-task learning approach, integrating multi-class confounding elements within the framework. A pioneering strategy in the field of forest fire detection, this method aims to enhance the model’s discriminatory ability and decrease false positives. When tested against the wildfire dataset, the multi-task learning approach demonstrated significantly superior performance in key metrics and lower false alarm rates compared to traditional binary classification methods. This emphasizes the effectiveness of the proposed methodology and the potential to address confounding elements. Recognizing the need for practical solutions, the study stresses the importance of future work to increase the representativeness of training and testing datasets. The evolving and publicly available wildfire dataset is anticipated to inspire innovative solutions, marking a substantial contribution to the fieldPostprint (published version

Reconstructive archaeology: in situ visualisation of previously excavated finds and features through an ongoing mixed reality process

Featured ApplicationThis automatic 3D reconstructive process currently underway supplies archaeologists with a mixed reality (MR) technique that allows them to interactively visualise 3D models representing formerly extracted finds, and to position such models over the features still present at the archaeological site.Archaeological excavation is a demolishing process. Rather few elements outlast extractive operations. Therefore, it is hard to visualise the precise location of unearthed finds at a previously excavated research area. Here, we present a mixed reality environment that displays in situ 3D models of features that were formerly extracted and recorded with 3D coordinates during unearthing operations. We created a tablet application that allows the user to view the position, orientation and dimensions of every recorded find while freely moving around the archaeological site with the device. To anchor the model, we used physical landmarks left at the excavation. A series of customised forms were created to show (onscreen) the different types of features by superimposing them over the terrain as perceived by the tablet camera. The application permits zooming-in, zooming-out, querying for specific artefacts and reading metadata associated with the archaeological elements. When at the office, our environment enables accurate visualisations of the 3D geometry concerning previously unearthed features and their spatial relationships. The application operates using the Swift programming language, Python scripts and ARKit technology. We present here an example of its use at Les Cottes, France, a palaeolithic site where thousands of artefacts are excavated out of six superimposed layers with a complex conformation.NWOVI.C.191.070Human Origin

INFORMATION TECHNOLOGY FOR NEXT-GENERATION OF SURGICAL ENVIRONMENTS

Minimally invasive surgeries (MIS) are fundamentally constrained by image quality,access to the operative field, and the visualization environment on which thesurgeon relies for real-time information. Although invasive access benefits the patient,it also leads to more challenging procedures, which require better skills andtraining. Endoscopic surgeries rely heavily on 2D interfaces, introducing additionalchallenges due to the loss of depth perception, the lack of 3-Dimensional imaging,and the reduction of degrees of freedom.By using state-of-the-art technology within a distributed computational architecture,it is possible to incorporate multiple sensors, hybrid display devices, and3D visualization algorithms within a exible surgical environment. Such environmentscan assist the surgeon with valuable information that goes far beyond what iscurrently available. In this thesis, we will discuss how 3D visualization and reconstruction,stereo displays, high-resolution display devices, and tracking techniques arekey elements in the next-generation of surgical environments

REAL-TIME CAPTURE AND RENDERING OF PHYSICAL SCENE WITH AN EFFICIENTLY CALIBRATED RGB-D CAMERA NETWORK

From object tracking to 3D reconstruction, RGB-Depth (RGB-D) camera networks play an increasingly important role in many vision and graphics applications. With the recent explosive growth of Augmented Reality (AR) and Virtual Reality (VR) platforms, utilizing camera RGB-D camera networks to capture and render dynamic physical space can enhance immersive experiences for users. To maximize coverage and minimize costs, practical applications often use a small number of RGB-D cameras and sparsely place them around the environment for data capturing. While sparse color camera networks have been studied for decades, the problems of extrinsic calibration of and rendering with sparse RGB-D camera networks are less well understood. Extrinsic calibration is difficult because of inappropriate RGB-D camera models and lack of shared scene features. Due to the significant camera noise and sparse coverage of the scene, the quality of rendering 3D point clouds is much lower compared with synthetic models. Adding virtual objects whose rendering depend on the physical environment such as those with reflective surfaces further complicate the rendering pipeline. In this dissertation, I propose novel solutions to tackle these challenges faced by RGB-D camera systems. First, I propose a novel extrinsic calibration algorithm that can accurately and rapidly calibrate the geometric relationships across an arbitrary number of RGB-D cameras on a network. Second, I propose a novel rendering pipeline that can capture and render, in real-time, dynamic scenes in the presence of arbitrary-shaped reflective virtual objects. Third, I have demonstrated a teleportation application that uses the proposed system to merge two geographically separated 3D captured scenes into the same reconstructed environment. To provide a fast and robust calibration for a sparse RGB-D camera network, first, the correspondences between different camera views are established by using a spherical calibration object. We show that this approach outperforms other techniques based on planar calibration objects. Second, instead of modeling camera extrinsic using rigid transformation that is optimal only for pinhole cameras, different view transformation functions including rigid transformation, polynomial transformation, and manifold regression are systematically tested to determine the most robust mapping that generalizes well to unseen data. Third, the celebrated bundle adjustment procedure is reformulated to minimize the global 3D projection error so as to fine-tune the initial estimates. To achieve a realistic mirror rendering, a robust eye detector is used to identify the viewer\u27s 3D location and render the reflective scene accordingly. The limited field of view obtained from a single camera is overcome by our calibrated RGB-D camera network system that is scalable to capture an arbitrarily large environment. The rendering is accomplished by raytracing light rays from the viewpoint to the scene reflected by the virtual curved surface. To the best of our knowledge, the proposed system is the first to render reflective dynamic scenes from real 3D data in large environments. Our scalable client-server architecture is computationally efficient - the calibration of a camera network system, including data capture, can be done in minutes using only commodity PCs

Widening the view angle of auto-multiscopic display, denoising low brightness light field data and 3D reconstruction with delicate details

This doctoral thesis will present the results of my work into widening the viewing angle of the auto-multiscopic display, denoising light filed data the enhancement of captured light filed data captured in low light circumstance, and the attempts on reconstructing the subject surface with delicate details from microscopy image sets. The automultiscopic displays carefully control the distribution of emitted light over space, direction (angle) and time so that even a static image displayed can encode parallax across viewing directions (light field). This allows simultaneous observation by multiple viewers, each perceiving 3D from their own (correct) perspective. Currently, the illusion can only be effectively maintained over a narrow range of viewing angles. We propose and analyze a simple solution to widen the range of viewing angles for automultiscopic displays that use parallax barriers. We insert a refractive medium, with a high refractive index, between the display and parallax barriers. The inserted medium warps the exitant lightfield in a way that increases the potential viewing angle. We analyze the consequences of this warp and build a prototype with a 93% increase in the effective viewing angle. Additionally, we developed an integral images synthesis method that can address the refraction introduced by the inserted medium efficiently without the use of ray tracing. Capturing light field image with a short exposure time is preferable for eliminating the motion blur but it also leads to low brightness in a low light environment, which results in a low signal noise ratio. Most light field denoising methods apply regular 2D image denoising method to the sub-aperture images of a 4D light field directly, but it is not suitable for focused light field data whose sub-aperture image resolution is too low to be applied regular denoising methods. Therefore, we propose a deep learning denoising method based on micro lens images of focused light field to denoise the depth map and the original micro lens image set simultaneously, and achieved high quality total focused images from the low focused light field data. In areas like digital museum, remote researching, 3D reconstruction with delicate details of subjects is desired and technology like 3D reconstruction based on macro photography has been used successfully for various purposes. We intend to push it further by using microscope rather than macro lens, which is supposed to be able to capture the microscopy level details of the subject. We design and implement a scanning method which is able to capture microscopy image set from a curve surface based on robotic arm, and the 3D reconstruction method suitable for the microscopy image set

A Fast and Robust Extrinsic Calibration for RGB-D Camera Networks

From object tracking to 3D reconstruction, RGB-Depth (RGB-D) camera networks play an increasingly important role in many vision and graphics applications. Practical applications often use sparsely-placed cameras to maximize visibility, while using as few cameras as possible to minimize cost. In general, it is challenging to calibrate sparse camera networks due to the lack of shared scene features across different camera views. In this paper, we propose a novel algorithm that can accurately and rapidly calibrate the geometric relationships across an arbitrary number of RGB-D cameras on a network. Our work has a number of novel features. First, to cope with the wide separation between different cameras, we establish view correspondences by using a spherical calibration object. We show that this approach outperforms other techniques based on planar calibration objects. Second, instead of modeling camera extrinsic calibration using rigid transformation, which is optimal only for pinhole cameras, we systematically test different view transformation functions including rigid transformation, polynomial transformation and manifold regression to determine the most robust mapping that generalizes well to unseen data. Third, we reformulate the celebrated bundle adjustment procedure to minimize the global 3D reprojection error so as to fine-tune the initial estimates. Finally, our scalable client-server architecture is computationally efficient: the calibration of a five-camera system, including data capture, can be done in minutes using only commodity PCs. Our proposed framework is compared with other state-of-the-arts systems using both quantitative measurements and visual alignment results of the merged point clouds

Supporting Focus and Context Awareness in 3D Modeling Using Multi-Layered Displays

Although advances in computer technology over the past few decades have made it possible to create and render highly realistic 3D models these days, the process of creating these models has remained largely unchanged over the years. Modern 3D modeling software provide a range of tools to assist users with creating 3D models, but the process of creating models in virtual 3D space is nevertheless still challenging and cumbersome. This thesis, therefore, aims to investigate whether it is possible to support modelers more effectively by providing them with alternative combinations of hardware and software tools to improve their 3D modeling tasks. The first step towards achieving this goal has been to better understand the type of problems modelers face in using conventional 3D modeling software. To achieve this, a pilot study of novice 3D modelers, and a more comprehensive study of professional modelers were conducted. These studies resulted in identifying a range of focus and context awareness problems that modelers face in creating complex 3D models using conventional modeling software. These problems can be divided into four categories: maintaining position awareness, identifying and selecting objects or components of interest, recognizing the distance between objects or components, and realizing the relative position of objects or components. Based on the above categorization, five focus and context awareness techniques were developed for a multi-layer computer display to enable modelers to better maintain their focus and context awareness while performing 3D modeling tasks. These techniques are: object isolation, component segregation, peeling focus, slicing, and peeling focus and context. A user study was then conducted to compare the effectiveness of these focus and context awareness techniques with other tools provided by conventional 3D modeling software. The results of this study were used to further improve, and evaluate through a second study, the five focus and context awareness techniques. The two studies have demonstrated that some of these techniques are more effective in supporting 3D modeling tasks than other existing software tools