The results of research the intelligence multimodal man-machine interface and virtual reality means for
assistive medical systems including computers and mechatronic systems (robots) are discussed. The gesture
translation for disability peoples, the learning-by-showing technology and virtual operating room with 3D
visualization are presented in this report and were announced at International exhibition "Intelligent and Adaptive
Robots–2005"

Andreev, Vasily

Chernakova, Svetlana

Gulenko, Igor

Litvinov, Mikhail

Nechaev, Alexander

Timofeev, Adil

Bulgarian Digital Mathematics Library at IMI-BAS

International Journal "Information Theories & Applications" Vol.14 / 2007   133MULTIMODAL MAN-MACHINE INTERFACE AND VIRTUAL REALITY FOR ASSISTIVE MEDICAL SYSTEMS Adil Timofeev, Alexander Nechaev, Igor Gulenko, Vasily Andreev,  Svetlana Chernakova, Mikhail Litvinov  Abstract: The results of research the intelligence multimodal man-machine interface and virtual reality means for assistive medical systems including computers and mechatronic systems (robots) are discussed. The gesture translation for disability peoples, the learning-by-showing technology and virtual operating room with 3D visualization are presented in this report and were announced at International exhibition "Intelligent and Adaptive Robots–2005". Keywords: multimodal man-machine interface, virtual reality, assistive medical systems. ACM Classification Keywords: I.2. Artificial Intelligence 1. Introduction The modern medicine is actually required the development of new information technologies and the multimodal man-machine interface (MMI) for control of medical robots and mechatronic systems, for automation of surgery with virtual reality means for creating telemedical diagnostic systems, etc.  The NATO-grant № PST.CLG 975579 “The man-machine interface for assistive systems in neurosurgery”, executed in 1999-2000 by partners from the St.-Petersburg Institute for Informatics and Automation of Russian Academy of Science (SPIIRAS), the State University of Aerospace Instrumentation, University of Karlsruhe (Germany) and Harvard Medical School (USA) has been directed on development of the man-machine interface, adaptive robots and multiagent technologies intended for neurosurgery [1]. The results of researches the medical MMI, robotics and mechatronic systems, including the results submitted in this report, have been presented at the international exhibition and a symposium “Intelligent and Adaptive Robots - 2005” [2]. 2. Video Capture of Motion for Translating from Sign Language on Natural Language and Back The development of anthropomorphous robots and their animation are very important for solution of various problems. Innovative results in this area, based on models and means of virtual reality, may be applied at such human activities, as medicine, sports, learning, computer and cognitive graphics. For example, in medicine the motion analysis of person (patient) can be essentially for automatic diagnostics and treatment of orthopedic diseases.  The researches of human’s or anthropomorphous robot’s motion are very important in the computer graphics for animation of virtual actors with methods and means of video capture was offered in [3]. Currently these methods being adapted for solution the problem of translation from a natural language to a sign language and back and for development of MMI for disable peoples. The communication for people with restrictions on hearing or speech is taking place in completely different way, than it is for people without such restrictions. Peoples with such restrictions cannot watch TV, listen to radio without additional means and communicate by phone, as it done usually by people. So it is necessary to develop the approach to realization of specialized interfaces, which could be used in any telecommunication devices. The interfaces being discussed include the input-output informative means and data links.  As for data links, it is widely used a highly developed and reliable Information Telecommunication (IT) technologies. The advanced input-output informative systems for disability peoples are the novel MMI technologies.  International Journal "Information Theories & Applications" Vol.14 / 2007   134 As for gesture exchange with MMI, it is offered the output of gesture translation to be carried out by means of animated “Avatar”, the simplified 3D computer model of human. Avatar can reproduce gestures by two ways [3, 4]: - Generating gestures from corresponding text expressions in natural language using script sequences from database; - Reproducing animation of gestures commands transmitted directly under the data link. The gesture input means also can work by two following ways: - Text generating in natural language according to analysis of gestures reproduced by operator; - Transforming gestures of operator to gesture command sequences of animation for avatar. 3. Man-machine Interface for Aassistive Medical Systems Based on Video Capture of Motions The important task in the development of robotic systems is a design of the man-machine interface. The new approach in creation of such interface is based on technologies of video capture of robot’s motion and a virtual reality means [5, 6].  The general structure of man-machine interface and robot control system designed by using means of video capture and virtual reality technology includes two subsystems, see  Fig. 1.    Fig. 1 Subsystem 1 (subsystem of interaction with operator) includes the following components: - System of video capture of operator’s motions, - Virtual control devices, - Visualization procedures. Subsystem 2 (subsystem of interaction with robot) has a various structures depending on application. However it is necessary to include the following components in this subsystem: - Means of processing and distribution of robot control signals, - Means of supervision over a condition of robots.  Control System operator subsystem 1 motion capture  virtual interfaces visualization subsystem 2 motion capture object state monitoring visual monitoring   robot object control data link International Journal "Information Theories & Applications" Vol.14 / 2007   135It is also proposed to develop the following components increasing the control efficiency and reliability of MMI systems [3, 4]: - Active multi-channel system of video capture of motion; - Means of visual observation.  4. The Learning Technology Using Multimodal Man-machine Interface The intelligence learning technology with showing the natural movements (gestures) of operator [7, 8] is developed to creating natural and accessible means for people’s communication with technical and information systems (fig. 2). The offered learning technology for medical mechatronic systems (robots) is based on the intelligence MMI [9], which provide the following features for medical applications: - Simple and correct understanding of gesture and speech commands of doctor or patient [10], - The intuitive teaching the medical mechatronic system by means of natural operator’s hand movements (without traditional movement’s programming) [8], - The visualization of real and virtual 3D medical images (X-ray, endoscopic, thermography, etc), - The doctor’s automatic workplace (AWP) with 3D viewing of X-ray images in real time mode [13];  - The novel stereo-projective systems for observation X-ray, endoscopic and other medical images;  - The human interaction with computer’s models and real medical equipment by means tactile and force-torque sensors during diagnostic or planning of medical operation [6, 12], - The information and telecommunication support of medical equipment control systems using the Virtual Operating Room (VOR).              Fig.2 The development of the Multimodal Man-machine Interface (MMI) for telemedicine and medical robotics assumes the creation of highly realistic effect of doctor’s presence in remote environment of patient.  For this purpose virtual models and computer-synthesized 3D-images of virtual objects composed with images of real environment (Augmented Reality technologies, AR) are used. The basic problems of AR technology realization is a problem of exact registration of a computer-synthesized image of geometrical model with real image and a problem of 3D image visualization in a real time mode. It is suggested to use the following means to 3D visualization of medical images (fig.3): - Stereo-glasses for observation on the computer monitor; - Stereo-displays glasses for 3D viewing color images without computer monitor; - Means of augmenting real medical images by virtual images (AR). Tactile-force interaction with virtual medical objects is necessary when it is not only required to observe the environment, but it is also necessary to perform any actions in it. These manipulations in remote environment will International Journal "Information Theories & Applications" Vol.14 / 2007   136 be more successful, if it will be possible to create realistic and adequate perception of objects in environment surrounding the patient and medical robot, to give an ability of feeling a virtual object with mass, shape, elastic and friction features as a real object.           Fig. 3  These technologies have partially been developed within the Partner Project 1992p with EOARD (European Office of Aerospace Research and Development, London, United Kingdom) and based on a long-term experience of medical system development [14].  The Head Tracking System (HTS) and Hand Tracking System (HTS+) prototypes have been developed for accurate measurement of human-operator’s head and hands movements. The basic aim of this development is to create the simple (cost-effective), reliable and steady means for human interaction with telecontrol objects. For medicine the development of so-called “haptic” interface, intended for tactile and force-torque displaying on hand of doctor (surgeon) is especially actual. The developed handle with force-torque feedback sensors reflects the real tactile and force interaction of mechatronic system (robots) instrument with real objects.  The prototype of handle with force-torque sensor for controlling and learning of medical mechatronic systems is shown on Fig. 4. As an example of medical application of novel force-torque sensor is a prototype of robot-masseur with exact control of force contact to patient’s muscles and a high level safety of robot-masseur movements near the patient or personnel.   Fig. 4  The offered technologies of 3D visualization, virtual reality means and learning-by-showing technology are especially useful for navigation, programming of movements and control of medical mechatronic systems [4, 11]. Thus information technologies with creating a highly realistic effect of presence the doctor in remote patient’s environment are useful for telemedicine application too. International Journal "Information Theories & Applications" Vol.14 / 2007   137Using of medical robots in surgery or telemedicine may be dangerous to patients. Therefore it is necessary to create virtual models of robot, patient and operating room for testing, planning and safety providing of medical operation with usage of mechatronic systems (medical robots).  5. Virtual Model of Operating Room Virtual operating room (VOR) should be similar to real one as much as possible, because the doctor’s work in usual conditions would be more effective and productive. Therefore the VOR must be “filled" with familiar medical tools, equipment and assistive robots in their computer representation. In this case there is an opportunity of carrying out of trial educational operations not on the real patient body, but on his virtual model. Carrying out of operation on the virtual “electronic patient” can be considered as a planning stage of real surgical operation. The second important problem being solved using VOR is an increasing professionalism of medical personnel. The dynamic virtual model of neurosurgical VOR is presented on fig. 5. The information on real and virtual operation and clinical data of patient are united in a uniform picture of medical operation. It is possible to return to past operation for comparative analysis and finding-out the efficiency of different methods of medical operation. Due to this not only doctors, but also students and post-graduate students can to training with VOR due to improve their skill [3-6, 11]. During the virtual operation the surgeon can to observe the trajectory of medical tool, for example, under brain tomogram of patient. In a case when the virtual medical tool to reach zone of high risk for virtual patient, the doctor can request the VOR program to find other variant of virtual tool targeting to necessary point of brain on a more safety trajectory or to take a conclusion on impossibility of carrying out of operation.   Fig. 5 6. Conclusion The offered information technologies, the multimodal man-machine interface and a virtual reality technique find more and more wide applications not only in medicine, but also in other areas (service, home assistance, telecommunications, space, etc.). Thus the intelligence man-machine interface, dynamic models of a virtual reality and multi-agent technologies will play especially important role [5, 6]. At the International exhibition «Intelligent and Adaptive Robots – 2005» the authors of this report have been awarded with 4 Gold Medals and Diplomas «for the development and introduction of innovative technology of man-machine interaction for telemedicine and medical robotics». The presented work is executed at partial support of the grant RFBR № 05-01-08044_ofi and the grant № 17 “Development of Man-Machine Interface and Control Methods for Neurosurgical Robots on the Base of Virtual Reality Models” of Saint-Petersburg Scientific Center of Russian Academy of Sciences. Bibliography [1] C. Burghart, O. Schorr, S. Yigit, N. Hata, K. Chinzei, A. Timofeev, R. Kikinis, H. Wörn, U. Rembold A Multi-Agent-System Architecture for Man-Machine Interaction in Computer Aided Surgery. – Proceedings of the 16th IAR Annual Meeting (Strasburg, November 22-23, 2001), pp. 117-123. [2] Intelligent and Adaptive Robots – 2005. Official Exhibition Catalogue – M.: Open Company "ZETP", 2005, p. 45 (In Russian). [3] Gulenko I.E., Shugina V.S. Application of technologies of video capture of movement in medicine and sports. – Works of conference “New information technologies” (Sudak, Crimea, May, 22-29, 2005), pp. 318-320 (In Russian). International Journal "Information Theories & Applications" Vol.14 / 2007   138 [4] Timofeev A.V., Gulenko I.E. and Litvinov M.V. Analysis, Processing and Transfer of Dynamic Images in Virtual Reality Models. – Pattern Recognition and Image Analysis, 2004, vol. 16, No.1, pp.97-99. [5] Timofeev A.V., Andreev V.A., Gulenko I.E., Derin O.A., Litvinov M.V. Design and Implementation of Multi-Agent Man-Machine Interface on the Base of Virtual Reality Models. – Proceedings of 9-th International Conference “Speech and Computer” (20-22 September, 2004), Saint-Petersburg, Russia, pp. 670-676. [6] Timofeev A.V. Intellectualization for Man-Machine Interface and Network Control in Multi-Agent Infotelecommunication Systems of New Generation. – Proceedings of 9-th International Conference “Speech and Computer” (20-22 September, 2004), Saint-Petersburg, Russia, pp. 694-700. [7] Kulakov F.М., Nechaev А.I., Chernakova S.E. Modelling of Environment for the Teaching by Showing Process. In Proc. of SPIIRAS, Russia, St. Petersburg, 2002, Issue № 2, pp. 105–113. [8] Chernakova S.E., Kulakov F.M., Nechaev A.I. Learning of the robot by method of showing with use «sensing» gloves. – Works of the First international conference on mechatronics and a robotics (Saint Petersburg, May, 29 – June, 2, 2000), pp. 155-164 (In Russian). A. Karpov, A. Ronzhin, A. Nechaev, S. Chernakova. Assistive multimodal system based on speech recognition and head tracking. In Proceedings of 9-th International Conference SPECOM’2004, St. Petersburg, 2004, pp. 521-530. [9] А. Karpov, А. Ronzhin, А. Nechaev, S. Chernakova. Multimodal system for hands-free PC control. In Proc. of 13-th European Signal Processing Conference EUSIPCO–2005, Antalya, Turkey, 2005. [10] Litvinov M.V., Timofeev A.V., Popov A.B. Virtual model of the adaptive robot for neurosurgery. – Proceedings of the International conference “Adaptive robots and the general logic theory of systems - AR&GSLT, (Saint Petersburg, on May, 8-11 2004), pp. 119–122 (In Russian). [11] Timofeev A.V., Shibzuhov Z.M., Sheojev A.M. Designing and learning multi-agent diagnostic systems. – Proceedings of the First international conference on mechatronics and a robotics “M&R. [12] Nechaev A.I., Nazaruk V.P., Chernakova S.E. Method of registration and visualization of 3D X-ray images in real time for technical nondestructive checking and medical diagnostics. – "Information technologies", 11, 2005, Moscow, pp. 11-21, in Russian. [13] Сhernakova S.E., Kulakov F.M., Nechaev A.I. Advanced man-machine interface for telerobotic based on man-operator’s motions and gaze direction tracking – Proceedings of Sixth ISTC Scientific Advisory Committee Seminar “Science and Computing”, Moscow. 15–17 September 2003, Volume 1, part II, рр. 303-308. Authors’ Information Timofeev Adil Vasilievich – Dr. Sc., Professor, Honoured Scientist of Russian Federation, Saint-Petersburg Institute for Informatics and Automation of Russian Academy of Sciences, 199178, Russia, Saint-Petersburg, 14-th Line, 39, phone: +7-812-328-0421; fax: +7-812-328-4450, e-mail: tav@iias.spb.su Nechaev Alexander Ivanovich – Scientific Researcher, Saint-Petersburg Institute for Informatics and Automation of Russian Academy of Sciences, 199178, Russia, Saint-Petersburg, 14-th Line, 39, phone: +7-812-328-0421; fax: +7-812-328-4450, e-mail: nechaev@iias.spb.su Gulenko Igor Evgenievich – Post Graduate Student, Saint-Petersburg Institute for Informatics and Automation of Russian Academy of Sciences, 199178, Russia, Saint-Petersburg, 14-th Line, 39, phone: +7-812-328-0421; fax: +7-812-328-4450, e-mail: gig@yandex.ru Andreev Vasily Alexandrovich – Post Graduate Student, Saint-Petersburg Institute for Informatics and Automation of Russian Academy of Sciences, 199178, Russia, Saint-Petersburg, 14-th Line, 39, phone: +7-812-328-0421; fax: +7-812-328-4450, e-mail: vasiliy-a@yandex.ru Chernakova Svetlana Eduardovna – Minor Scientist, Saint-Petersburg Institute for Informatics and Automation of Russian Academy of Sciences, 199178, Russia, Saint-Petersburg, 14-th Line, 39, phone: +7-812-328-0421; fax: +7-812-328-4450, e-mail: chernakova@iias.spb.su Litvinov Mikhail Vladimirovich – Post Graduate Student, Baltic State Technical University “Voenmech”, 190005,Saint-Petersburg, Russia, 1-st Krasnoarmeyskaya, 1, e-mail: sid-4d@inbox.ru  

Multimodal Man-machine Interface and Virtual Reality for Assistive Medical Systems

http://sci-gems.math.bas.bg/jspui/bitstream/10525/669/1/ijita14-2-p04.pdf

Multimodal Man-machine Interface and Virtual Reality for Assistive Medical Systems

Abstract

Similar works

Full text

Available Versions

Bulgarian Digital Mathematics Library at IMI-BAS