Intraoperative Navigation Systems for Image-Guided Surgery

Recent technological advancements in medical imaging equipment have resulted in a dramatic improvement of image accuracy, now capable of providing useful information previously not available to clinicians. In the surgical context, intraoperative imaging provides a crucial value for the success of the operation. Many nontrivial scientific and technical problems need to be addressed in order to efficiently exploit the different information sources nowadays available in advanced operating rooms. In particular, it is necessary to provide: (i) accurate tracking of surgical instruments, (ii) real-time matching of images from different modalities, and (iii) reliable guidance toward the surgical target. Satisfying all of these requisites is needed to realize effective intraoperative navigation systems for image-guided surgery. Various solutions have been proposed and successfully tested in the field of image navigation systems in the last ten years; nevertheless several problems still arise in most of the applications regarding precision, usability and capabilities of the existing systems. Identifying and solving these issues represents an urgent scientific challenge. This thesis investigates the current state of the art in the field of intraoperative navigation systems, focusing in particular on the challenges related to efficient and effective usage of ultrasound imaging during surgery. The main contribution of this thesis to the state of the art are related to: Techniques for automatic motion compensation and therapy monitoring applied to a novel ultrasound-guided surgical robotic platform in the context of abdominal tumor thermoablation. Novel image-fusion based navigation systems for ultrasound-guided neurosurgery in the context of brain tumor resection, highlighting their applicability as off-line surgical training instruments. The proposed systems, which were designed and developed in the framework of two international research projects, have been tested in real or simulated surgical scenarios, showing promising results toward their application in clinical practice

A method for the assessment of time-varying brain shift during navigated epilepsy surgery

Image guidance is widely used in neurosurgery. Tracking systems (neuronavigators) allow registering the preoperative image space to the surgical space. The localization accuracy is influenced by technical and clinical factors, such as brain shift. This paper aims at providing quantitative measure of the time-varying brain shift during open epilepsy surgery, and at measuring the pattern of brain deformation with respect to three potentially meaningful parameters: craniotomy area, craniotomy orientation and gravity vector direction in the images reference frame

Unmanned Ground Robots for Rescue Tasks

This chapter describes two unmanned ground vehicles that can help search and rescue teams in their difficult, but life-saving tasks. These robotic assets have been developed within the framework of the European project ICARUS. The large unmanned ground vehicle is intended to be a mobile base station. It is equipped with a powerful manipulator arm and can be used for debris removal, shoring operations, and remote structural operations (cutting, welding, hammering, etc.) on very rough terrain. The smaller unmanned ground vehicle is also equipped with an array of sensors, enabling it to search for victims inside semi-destroyed buildings. Working together with each other and the human search and rescue workers, these robotic assets form a powerful team, increasing the effectiveness of search and rescue operations, as proven by operational validation tests in collaboration with end users

RealDex: Towards Human-like Grasping for Robotic Dexterous Hand

In this paper, we introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns, enriched by multi-view and multimodal visual data. Utilizing a teleoperation system, we seamlessly synchronize human-robot hand poses in real time. This collection of human-like motions is crucial for training dexterous hands to mimic human movements more naturally and precisely. RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios. Moreover, we introduce a cutting-edge dexterous grasping motion generation framework, which aligns with human experience and enhances real-world applicability through effectively utilizing Multimodal Large Language Models. Extensive experiments have demonstrated the superior performance of our method on RealDex and other open datasets. The complete dataset and code will be made available upon the publication of this work

Pictures in Your Mind: Using Interactive Gesture-Controlled Reliefs to Explore Art

Tactile reliefs offer many benefits over the more classic raised line drawings or tactile diagrams, as depth, 3D shape, and surface textures are directly perceivable. Although often created for blind and visually impaired (BVI) people, a wider range of people may benefit from such multimodal material. However, some reliefs are still difficult to understand without proper guidance or accompanying verbal descriptions, hindering autonomous exploration. In this work, we present a gesture-controlled interactive audio guide (IAG) based on recent low-cost depth cameras that can be operated directly with the hands on relief surfaces during tactile exploration. The interactively explorable, location-dependent verbal and captioned descriptions promise rapid tactile accessibility to 2.5D spatial information in a home or education setting, to online resources, or as a kiosk installation at public places. We present a working prototype, discuss design decisions, and present the results of two evaluation studies: the first with 13 BVI test users and the second follow-up study with 14 test users across a wide range of people with differences and difficulties associated with perception, memory, cognition, and communication. The participant-led research method of this latter study prompted new, significant and innovative developments

Novel PET Systems and Image Reconstruction with Actively Controlled Geometry

Positron Emission Tomography (PET) provides in vivo measurement of imaging ligands that are labeled with positron emitting radionuclide. Since its invention, most PET scanners have been designed to have a group of gamma ray detectors arranged in a ring geometry, accommodating the whole patient body. Virtual Pinhole PET incorporates higher resolution detectors being placed close to the Region-of-Interest (ROI) within the imaging Field-of-View (FOV) of the whole-body scanner, providing better image resolution and contrast recover. To further adapt this technology to a wider range of diseases, we proposed a second generation of virtual pinhole PET using actively controlled high resolution detectors integrated on a robotic arm. When the whole system is integrated to a commercial PET scanner, we achieved positioning repeatability within 0.5 mm. Monte Carlo simulation shows that by focusing the high-resolution detectors to a specific organ of interest, we can achieve better resolution, sensitivity and contrast recovery. In another direction, we proposed a portable, versatile and low cost PET imaging system for Point-of-Care (POC) applications. It consists of one or more movable detectors in coincidence with a detector array behind a patient. The movable detectors make it possible for the operator to control the scanning trajectory freely to achieve optimal coverage and sensitivity for patient specific imaging tasks. Since this system does not require a conventional full ring geometry, it can be built portable and low cost for bed-side or intraoperative use. We developed a proof-of-principle prototype that consists of a compact high resolution silicon photomultiplier detector mounted on a hand-held probe and a half ring of conventional detectors. The probe is attached to a MicroScribe device, which tracks the location and orientation of the probe as it moves. We also performed Monte Carlo simulations for two POC PET geometries with Time-of-Flight (TOF) capability. To support the development of such PET systems with unconventional geometries, a fully 3D image reconstruction framework has been developed for PET systems with arbitrary geometry. For POC PET and the second generation robotic Virtual Pinhole PET, new challenges emerge and our targeted applications require more efficiently image reconstruction that provides imaging results in near real time. Inspired by the previous work, we developed a list mode GPU-based image reconstruction framework with the capability to model dynamically changing geometry. Ordered-Subset MAP-EM algorithm is implemented on multi-GPU platform to achieve fast reconstruction in the order of seconds per iteration, under practical data rate. We tested this using both experimental and simulation data, for whole body PET scanner and unconventional PET scanners. Future application of adaptive imaging requires near real time performance for large statistics, which requires additional acceleration of this framework

A bank of unscented Kalman filters for multimodal human perception with mobile service robots

A new generation of mobile service robots could be ready soon to operate in human environments if they can robustly estimate position and identity of surrounding people. Researchers in this field face a number of challenging problems, among which sensor uncertainties and real-time constraints. In this paper, we propose a novel and efficient solution for simultaneous tracking and recognition of people within the observation range of a mobile robot. Multisensor techniques for legs and face detection are fused in a robust probabilistic framework to height, clothes and face recognition algorithms. The system is based on an efficient bank of Unscented Kalman Filters that keeps a multi-hypothesis estimate of the person being tracked, including the case where the latter is unknown to the robot. Several experiments with real mobile robots are presented to validate the proposed approach. They show that our solutions can improve the robot's perception and recognition of humans, providing a useful contribution for the future application of service robotics