Mastoidectomy is one of the most common surgical procedures relating to the petrous bone. In this paper we describe our preliminary results in the realization of a virtual reality mastoidectomy simulator. Our system is designed to work on patient-specific volumetric object models directly derived from 3D CT and MRI images. The paper summarizes the detailed task analysis performed in order to define the system requirements, introduces the architecture of the prototype simulator, and discusses the initial feedback received from selected end users.17-2

Agus, Marco

Giachetti, Andrea

Gobbetti, Enrico

Nigel, John W.

Stone, Robert J.

Zanetti, Gianluigi

Zorcolo, Antonio

Mastoidectomy is one of the most common surgical procedures relating to the petrous bone. In this paper we describe our preliminary results in the realization of a virtual reality mastoidectomy simulator. Our system is designed to work on patient-specific volumetric object models directly derived from 3D, CT and MRI images. The paper summarizes the detailed task analysis performed in order to define the system requirements, introduces the architecture of the prototype simulator, and discusses the initial feedback received from selected end users.</p

John, Nigel W.

University of Birmingham Research Portal

Mastoidectomy simulation with combined visual and haptic feedback

MARCO AGUS

ENRICO GOBBETTI

GIANLUIGI ZANETTI

NIGEL W. JOHN

ROBERT J. STONE

GIACHETTI, Andrea

Catalogo dei prodotti della ricerca

P-arch

Mastoidectomy Simulation with CombinedVisual and Haptic FeedbackMarco Agus, EE1, Andrea Giachetti, Ph.D. 1,  Enrico Gobbetti, Ph.D. 1 ,Gianluigi Zanetti, Ph.D. 1, Antonio Zorcolo, Dipl.-Inform1    Nigel W. John, Ph.D. 2, Robert J. Stone, Prof3.1CRS4, Uta (CA) Italy – http://www.crs4.it2Manchester Visualization Centre, University of  Manchester, Manchester,UK,3Virtual Presence Ltd., Manchester, UKAbstract. Mastoidectomy is one of the most common surgical procedures relatingto the petrous bone. In this paper we describe our preliminary results in therealization of a virtual reality mastoidectomy simulator. Our system is designed towork on patient-specific volumetric object models directly derived from 3D CT andMRI images. The paper summarizes the detailed task analysis performed in order todefine the system requirements, introduces the architecture of the prototypesimulator, and discusses the initial feedback received from selected end users.1. IntroductionMastoidectomy is one of the most common surgical procedures relating to the petrousbone. It consists of removal of the air cavities just under the skin behind the ear itself, andit is performed for chronic infection of the mastoid air cells (mastoiditis). It is a surgicalprocedure undertaken by a wide range of surgeons in everyday practice. The site anatomyis widely variant. The main risks are related to the detection and avoidance of the facialnerve and of aberrant jugular veins (or branches) and to the resection of adequate amountsof the mastoid air cells. The ability to rehearse the procedure using patient-specific data isextremely rare. A VR simulator realistically mimicking a patient-specific operatingenvironment addresses this shortcoming. A number of groups are working toward this goal[4][5][6][7].In this paper we describe our preliminary results in the realization of a virtual realitymastoidectomy simulator designed to work on patient-specific volumetric object modelsdirectly derived from 3D CT and MRI images. The simulator runs on a multiprocessing PCplatform and provides realistic visual and haptic feedback, including secondary effectssuch as the obscuring of the operational site due to the accumulation of bone dust andother burring debris.The rest of the paper is organized as follows: first, we describe the detailed task analysisperformed to define the system requirements; we then provide a brief description of thearchitecture of the prototype simulator, and we conclude with a discussion of the firstresults obtained and a view of our future work.1. Human-Centered Design A detailed task analysis, following ISO 13407 [3], has been carried out in order toidentify the essential ergonomic components [9]. The analysis involved a review ofexisting documentation, training aids, and video recordings, interviews with experiencedoperators, as well as direct observation of the procedure being performed in theater. In the typical mastoidectomy surgical setup, Fig. 1, the ENT surgeon looks at the regionof interest through a stereoscopic microscope and holds in his hands a high speed burr anda sucker. These tools are used, respectively, to cut the bone and to remove water (used tocool the burr bit) and bone paste produced by the mixing of bone dust with water.Subjective analysis of video records, together with in-situ observations highlighted acorrelation between drilling behaviours and type and depth of bone. In the case of initialcortex burring and recess preparation for, e.g., a cochlea implant receiver/stimulator, drilltip/burr motions of around 0.8 cm together with sweeps over 2-4 cm were evident, as werefine flexion and extension movements of the forefinger and thumb around the drill. Shorter(1-2 cm) motions with rapid lateral strokes characterized the post-cortex mastoidectomy.For deeper drilling, ~1 cm,- strokes down to 1 or 2 mm were evident with more of a"polishing"motion quality, guided using the contours from prior drill procedures."Static"drill handling was also noted, eroding bone tissue whilst maintaining minimalsurface pressure. As for the visual effect of the drill on the surface of the bone, the task analysis hilightedthat the graphical process must simulate drill site obscuration by bone dust paste, becauseits absence would reduce the importance placed by a trainee on the need for regularirrigation and suction. Realistic and meaningful bleeding is a perennial problem for VRresearchers. We have concluded that, visually, the actual drill representation needs only bequite simple, and it is felt that representing the spinning of the cutter or diamond burr isunnecessary. What is considered necessary, from a functional standpoint, is an effectivecollision detection mechanism which not only copes with increased resolution as thevirtual drill proceeds deeper into the temporal bone, but is also capable of generating errorstates when (for example) a large burr is inserted into a narrow drill site. As for the nature of the technology required for displaying drill, drill site, bone, and soon, there is no conclusive evidence or support for the premise that the use of a stereoscopicsystem will aid performance in this case. Binocular viewing systems are deployed in theoperating theatre and used by surgeons, and so binocular imaging should be available tothe simulator. However, the wearing of any form of stereoscopic display, such as a head-mounted display or liquid crystal shutter glasses should be avoided. The surgeon or traineeFigure 1: The typical mastoidectomy surgical setup: the ENT surgeon looks at the region interested bythe procedure via a stereoscopic microscope and holds in his hands a high speed burr and a sucker,that he uses, respectively, to cut the bone and to remove bone paste produced by the mixing of bonedust with water.does not want to use cumbersome eyewear that is not necessary for carrying on the realprocedure. We make the hypothesis that, if the simulation achieves a reasonable level offidelity, then the combination of high-resolution images and haptic feedback will, morethan likely, suffice. As well as the visual and 6-DOF input/3-DOF haptic feedback for drill simulation(including high frequency vibration), the training system might also be enhanced by theinclusion of audio effects. Some surgeons suggest that they are able to detect subtlechanges in sound depending on the nature of the bone they are working with (eg. cortex vs.petrous). However, this quality is considered to be "overkill" in a training system such asthat being considered here. 2. Prototype System ArchitectureWe have built a prototype system, simulating the effects identified by the task analysis.The system is running on a dual PC platform (Fig. 2). It exploits both message passing andshared-memory parallelism to meet the performance constraints imposed by the humanperceptual system. The system uses a volumetric approach, with the initial configuration ofthe model directly derived from patient CT data. The final version will use volumetrictissue probability maps derived from CT, MRI and MRA data using a multi-dimensionalclassification technique [8]. The data is initially replicated on the two machines. The firstmachine is dedicated to the high-frequency tasks: haptic device handling, one for theFigure 2: Decoupled simulation architecture. The system uses a volumetric approach, with the initialconfiguration of the model directly derived from patient CT data. The data is initially replicated onthe two machines. The first machine is dedicated to the high-frequency tasks: haptic device handlingand bone removal simulation. The second machine concurrently runs at 10-20 Hz the low-frequencytasks: bone removal, fluid evolution and visual eedback. The two machines are synchronized usingone-way message passing with a dead reckoning protocol.dominant hand controlling the burr and the irrigator, and the other controlling the sucker,as well as bone removal simulation. These tasks require at least 1 kHz update frequencybecause of the need of simulating hard contacts. The second PC concurrently runs at 10-20Hz the low-frequency tasks: bone dust evolution and visual feedback. The two machinesare synchronized using one-way message passing with a dead reckoning protocol.In our volumetric description of the scene, voxels labeled as bone must bone must reactto the manipulators through the haptic feedback devices, but they do not evolve unless theyare removed by burring. In the data replicated in the machine dedicated to low frequencytasks, further values are introduced in the volume labelling voxels occupied by dust, bloodand water. These values are used directly by the volume rendering thread.The "fast"subsystem performs the burring simulation, i.e. the force feedback calculationand the bone removal from the dataset, sending the force value to the haptic devices, andsending information on manipulator positions and bone removed to the "slow"subsystem. This task is extremely difficult to perform at over 1 Khz. We have thus organized oursimulation so that each time step is divided into two sub-steps. The first sub-step estimatesthe bone material deformation and the resulting elastic forces, given the relative position ofthe burr with respect to the bone. The second sub-step estimates the local rate of cutting ofthe bone by using a postulated energy balance between the mechanical work performed bythe burr motor and the energy needed to cut the bone, which is assumed to be proportionalto the bone mass removed.The "flow" subsystem performs the visual simulation of bone dust and fluid dynamicsas well as the visual rendering of the scene. We are modeling the dust/fluid dynamics usingwhat essentially amounts to a hybrid particles/sand pile model. The visual rendering subsystem must operate within the timing constraints imposed bythe human perceptual system (i.e. a latency of less than 300 ms, and a frequency above 10-15 Hz). We reach this goal by using a parallel processing approach, which exploits thecapabilities of current graphics PC architectures. In our system, the renderer is totallydecoupled from the simulator and the tracking system, and runs at its own frequency. Therendering system is based on a volumetric approach. We use texture mapping and alphablending for a back to front reconstruction of the scene. Shading effects are implementedby exploiting the register combiner OpenGL extension on most modern commoditygraphic boards. Surgical instruments are rendered as polygons, and combined with thevolumetric rendering of the rest of the scene using Z-buffering.3. Implementation and ResultsOur current configuration is the following:- a single-processor PIII/600 MHz with 256 MB PC100 RAM for the high-frequencytasks; two threads run in parallel: one for the haptic loop (1 KHz), and one for sendingvolume and instruments position updates to the other machine;- a dual-processor PIII/800 MHz with 512 MB PC800 RAM and a NVIDIA GeForce 2GTS and running a 2.4 linux kernel, for the low frequency tasks. Three threads arecontinuously running on this machine: one to receive volume and position updates, oneto simulate bone removal and fluid evolution, and one for visual rendering;- a Phantom Desktop haptic device for the dominant hand; the device is connected to thesingle processor PC. It provides 6DOF tracking and 3DOF force feedback for theburr/irrigator;- a Phantom 1.0 haptic device for the non-dominant hand; the device is connected to thesingle processor PC. It provides 6DOF tracking and 3DOF force feedback for thesucker;- a N-vision VB30 binocular display for presenting images to the user. The VB-30contains a small high resolution LCD display and is connected to the S-VGA output ofthe dual processor PC.We have gathered initial feedback about the prototype system from the Otolaryngologysurgeons that are collaborating with this project. The performance of the prototype is sufficient to meet the timing constraints for displayand force-feedback effects, even though the computational and visualization platform ismade only of affordable and widely accessible components. We are currently using avolume of 256x256x128 8-bit cubical voxels (0.3 mm side) to represent the region wherethe operation takes place. The force-feedback loop is running at 1 KHz using a 5x5x5 gridround the tip of the instruments for force computations. The computation needed for forceevaluation and bone erosion takes typically 20 usec, and less than 200 usec in the worstcase configuration. Shaded volume rendering of dynamic volumes currently takes 70 msper frame (i.e. over 14 frames per second) using 256 depth slices on an 800x600 windowwith 16 bit color and 2X zoom rendering.The overall realism of the simulation is considered sufficient for training purposes. Asrequired by the task analysis, the haptic sub system is able to provide a reasonable forcefeedback effect, bone removal and noise simulation. The visual system is able to providebone dust and water bleeding effects, blood simulation, and manipulators displayed at therequired frame rate. Figure 3: Snapshot from a video demonstration of a simulated mastoidectomy. The central imageshow the image viewed by the training surgeon with bone, water, blood and manipulators. In thetop right corner it is possible to see the corresponding external view with the training surgeonfeeling the haptic feedback through the Phantoms while observing the scene in the binocular displaySubjective input is currently being used to tune the parameters that control forcefeedback. The speed of the water flow simulation has been identified as a principal area forimprovement.Fig. 3 shows a snapshot of the current system in use. The surgeon is looking at the scenewith the binocular display and manipulates the burr and sucker through the twoPHANToM devices. Fig. 4 shows selected frames of the initial phase of a virtualmastoidectomy, where debris formation and suction effects are clearly visible.4. Conclusions and Future WorkWe have described our preliminary results in the realization of a virtual realitymastoidectomy simulator designed to work on patient-specific volumetric object modelsdirectly derived from 3D CT and MRI images. We carried out an in-depth task analysis, that highlighted key human interface features,including visual and haptic feedback requirements, as well as burring primary andsecondary effects. Our simulator prototype demonstrates the possibility of building,exploiting parallelism on a PC platform, and a VR simulator mimicking a realistic patient-specific operating environment, including key secondary effects such as debris formationand the related visual masking effects.The results obtained thus far were judged of good quality by the Otolaryngologysurgeons collaborating to the project. The prototype is currently running directly usingpatient specific data as input, opening, therefore, the road towards the use of the simulatorfor pre-operation planning and rehearsal. This would make it possible to plan surgerydirectly on a model of the individual patient, rather than by referring to a model surgicalprocedure on a standard anatomy.While subjective input from selected end users is encouraging, it has been realized thatit would be of extreme interest to have available direct forces measurements obtained bydrilling actual samples, since there are, to our knowledge, no available data on the subjectin the literature. In the near future we plan to start an activity aimed at defining anexperimental setup and measurement procedures.Our current work is also concentrating on improving the quality and speed of fluid flowsimulation.AcknowledgmentsThese results were obtained within the framework of the European Union IERAPSIproject (EU-IST-1999-12175). The authors would like to acknowledge the contributionsfrom members of the University of Pisa's Department of Otolaryngology, University ofPisa's Department of Radiology, Manchester Royal Infirmary's Department ofOtolaryngology, Head & Neck Surgery, and of the Department of Surgery and North ofEngland Wolfson Centre for Minimally Invasive Therapy.References[1] A. Jackson, N. W. John, N. A. Thacker, E. Gobbetti, G. Zanetti, R. J. Stone, A. D. Linney, G. H.Alusi, and A. Schwerdtner, Developing a virtual reality environment for petrous bone surgery: a "state-of-the-art" review.  Journal of Otology & Neurology, To appear.[2] N. W. John, N. Thacker, M. Pokric, A. Jackson, G. Zanetti, E. Gobbetti, A. Giachetti, R. J. Stone,J. Campos, A. Emmen, A. Schwerdtner, E. Neri, S. Sellari Franceschini, and F. Rubio. An integratedsimulator for surgery of the petrous bone, In J. D. Westwood, editor, Medicine Meets Virtual Reality 2001,Amsterdam, The Netherlands, January 2001. IOS.[3] ISO. Human-Centered Design Processes for Interactive Systems. International Standard ISO13407, ISO, 1999.[4] G. Wiet, J. Bryan, D. Sessanna, D. Streadney, P. Schmalbrock, and B. Welling. Virtual TemporalBone Dissection Simulation. Proc. Medicine Meets Virtual Reality, Newport Beach, CA. 2000.[5] C.V. Edmond, et al. Simulation for ENT Endoscopic Surgical Training. Proc. Medicine MeetsVirtual Reality 5, San Diego, CA, January 1997. [6] S. Weghorst, et al. Validation of the Madigan ESS Simulator. Proc. Medicine Meets VirtualReality, San Diego, CA. 1998.[7] J. Bryan, D. Stredney, G. Wiet and D. Sessanna. Virtual Temporal Bone Dissection: a Case Study,Proc. IEEE Visualization, San Diego, CA, 2001.[8] M. Pokric et al. Multi-dimensional Medical Image Segmentation with Partial Voluming.Proceedings of Medical Image Understanding and Analysis, July 2001[9] R. J. Stone, A Human-Centered Definition of Surgical Procedures, IERAPSI (IST-1999-12715)Work Package 2, Deliverable D2 (Part 1), July 2000Figure 4: From top left to bottom right, selected frames of the initial phase of a virtualmastoidectomy. Debris formation and suction effects are clearly visible.

http://www.p-arch.it/bitstream/handle/11050/746/mmvr2002-ierapsi.pdf?sequence=1&isAllowed=y

Mastoidectomy simulation with combined visual and haptic feedback

Abstract

Similar works

Full text

Available Versions

University of Birmingham Research Portal

Catalogo dei prodotti della ricerca

P-arch

University of Birmingham Research Portal