Magnetic-Visual Sensor Fusion-based Dense 3D Reconstruction and Localization for Endoscopic Capsule Robots

Reliable and real-time 3D reconstruction and localization functionality is a crucial prerequisite for the navigation of actively controlled capsule endoscopic robots as an emerging, minimally invasive diagnostic and therapeutic technology for use in the gastrointestinal (GI) tract. In this study, we propose a fully dense, non-rigidly deformable, strictly real-time, intraoperative map fusion approach for actively controlled endoscopic capsule robot applications which combines magnetic and vision-based localization, with non-rigid deformations based frame-to-model map fusion. The performance of the proposed method is demonstrated using four different ex-vivo porcine stomach models. Across different trajectories of varying speed and complexity, and four different endoscopic cameras, the root mean square surface reconstruction errors 1.58 to 2.17 cm.Comment: submitted to IROS 201

A novel hybrid 3D endoscope zooming and repositioning system : design and feasibility study

Background: Manipulation of the endoscope during minimally invasive surgery is a major source of inconvenience and discomfort. This report elucidates the architecture of a novel one-hand controlled endoscope positioning device and presents a practicability evaluation. Methods and materials: Setup time and total surgery time, number and duration of the manipulations, side effects of three-dimensional (3D) imaging, and ergonomic complaints were assessed by three surgeons during cadaveric and in vivo porcine trials. Results: Setup was accomplished in an average (SD) of 230 (120) seconds. The manipulation time was 3.87 (1.77) seconds for angular movements and 0.83 (0.24) seconds for zooming, with an average (SD) of 30.5 (16.3) manipulations per procedure. No side effects of 3D imaging or ergonomic complaints were reported. Conclusions: The integration of an active zoom into a passive endoscope holder delivers a convenient synergy between a human and a machine-controlled holding device. It is shown to be safe, simple, and intuitive to use and allows unrestrained autonomic control of the endoscope by the surgeon

An intelligent real time 3D vision system for robotic welding tasks

MARWIN is a top-level robot control system that has been designed for automatic robot welding tasks. It extracts welding parameters and calculates robot trajectories directly from CAD models which are then verified by real-time 3D scanning and registration. MARWIN's 3D computer vision provides a user-centred robot environment in which a task is specified by the user by simply confirming and/or adjusting suggested parameters and welding sequences. The focus of this paper is on describing a mathematical formulation for fast 3D reconstruction using structured light together with the mechanical design and testing of the 3D vision system and show how such technologies can be exploited in robot welding tasks

Improved GelSight Tactile Sensor for Measuring Geometry and Slip

A GelSight sensor uses an elastomeric slab covered with a reflective membrane to measure tactile signals. It measures the 3D geometry and contact force information with high spacial resolution, and successfully helped many challenging robot tasks. A previous sensor, based on a semi-specular membrane, produces high resolution but with limited geometry accuracy. In this paper, we describe a new design of GelSight for robot gripper, using a Lambertian membrane and new illumination system, which gives greatly improved geometric accuracy while retaining the compact size. We demonstrate its use in measuring surface normals and reconstructing height maps using photometric stereo. We also use it for the task of slip detection, using a combination of information about relative motions on the membrane surface and the shear distortions. Using a robotic arm and a set of 37 everyday objects with varied properties, we find that the sensor can detect translational and rotational slip in general cases, and can be used to improve the stability of the grasp.Comment: IEEE/RSJ International Conference on Intelligent Robots and System