A software platform based on Leap Motion Controller (LMC) movements’ detection was developed. It allows measurements of clinically proved effective hand and finger exercises. The developed software allows representation of amplitude of each different movement, time interval for each movement, frequency of different movements, asymmetry of bilateral movements, standard deviation of signal amplitude, Poincaré plots. A serious game Collect Color Cube, was developed using Unity, C# scrips, and signals from LMC related to movements of the user’s hands and fingers.info:eu-repo/semantics/publishedVersio

Antunes A.

Lourenço F.

Postolache, G.

Postolache, O.

English

Repositório Institucional do ISCTE-IUL

 Repositório ISCTE-IUL Deposited in Repositório ISCTE-IUL:2018-07-17 Deposited version:Publisher Version Peer-review status of attached file:Peer-reviewed Citation for published item:Postolache, G., Lourenço F., Antunes A. & Postolache, O. (2017). Wrist and hand rehabilitationsoftware platform based on leap motion controller. In 3rd International Conference on SensorsEngineering and Electronics Instrumentation Advances, SEIA' 2017. Moscow: International FrequencySensor Association. Further information on publisher's website:-- Publisher's copyright statement:This is the peer reviewed version of the following article: Postolache, G., Lourenço F., Antunes A. &Postolache, O. (2017). Wrist and hand rehabilitation software platform based on leap motioncontroller. In 3rd International Conference on Sensors Engineering and Electronics InstrumentationAdvances, SEIA' 2017. Moscow: International Frequency Sensor Association.. This article may beused for non-commercial purposes in accordance with the Publisher's Terms and Conditions for self-archiving.Use policyCreative Commons CC BY 4.0The full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission orcharge, for personal research or study, educational, or not-for-profit purposes provided that:• a full bibliographic reference is made to the original source• a link is made to the metadata record in the Repository• the full-text is not changed in any wayThe full-text must not be sold in any format or medium without the formal permission of the copyright holders.Serviços de Informação e Documentação, Instituto Universitário de Lisboa (ISCTE-IUL)Av. das Forças Armadas, Edifício II, 1649-026 Lisboa PortugalPhone: +(351) 217 903 024 | e-mail: administrador.repositorio@iscte-iul.pthttps://repositorio.iscte-iul.pt3rd International Conference on Sensors Engineering and Electronics Instrumentation Advances (SEIA' 2017),  20-22 September 2017, Moscow, Russia   Wrist and Hand Rehabilitation Software Platform  Based on Leap Motion Controller   G. Postolache 1, F. Lourenço 2, A. Antunes 1, and O. Postolache 1,2  1 Instituto de Telecomunições, Lisbon, Portugal 2 Instituto Universitário de Lisboa, ISCTE-IUL, Lisbon, Portugal E-mail: opostolache@lx.it.pt   Summary: A software platform based on Leap Motion Controller (LMC) movements’ detection was developed. It allows measurements of clinically proved effective hand and finger exercises. The developed software allows representation of amplitude of each different movement, time interval for each movement, frequency of different movements, asymmetry of bilateral movements, standard deviation of signal amplitude, Poincaré plots. A serious game Collect Color Cube, was developed using Unity, C# scrips, and signals from LMC related to movements of the user’s hands and fingers.   Keywords: Leap Motion Controller,  wrist and hand rehabilitation, serious games.   1. Introduction Various wearable and unobtrusive devices have been developed for assessment of neuromusculoskeletal and movement related functions (i.e., for joint mobility, muscle tone or strength) and/or activity limitation (i.e., difficulties that an individual may have in executing activities). Devices based on inertial sensors (i.e., accelerometer, gyroscope, inertial measurements unit) [1], sensors for electromiography [2] or mecanomiography [3], or unobtrusive sensors based on microwave radar [4] and infrared technology  [5] were described in the last decades as useful instruments for diagnosis as well as for in-clinic or remote monitoring of progress in treatment of neuromusculoskeletal impairments. Quantifying the overall effects of the physiotherapy is important both for health professionals (i.e., for optimization of therapy interventions and objective monitoring of recovery), as well as for patients which by receiving digital information on many aspects of movements (eg., hand velocity or joint angles) are physically and cognitively challenged, in an environment that promote therapeutic engagement. The Leap Motion Controller (LMC), was publicly presented for the first time in 2013. It is a new type of device based on infrared technology that by his small dimension, low price and millimetre accuracy promise development of myriad applications for motor rehabilitation environment. Study on LMC efficacy in characterization of essential tremor was recently presented [6]. Use of LMC in conjunction with virtual reality (VR) have shown being a useful tool for rehabilitation of children with cerebral palsy [7] and for stroke patients [8]. Also, several serious game for upper limb training (VirtualRehab) were developed by VirtualWare [9]. The goal of our work was to developed a platform for hand and finger exercise monitoring including an extended number of exercises that may be monitored during training in motor rehabilitation sessions.   2. Methods  2.1. Leap Motion Controller  Leap Motion Controller is a small, rectangular device (13mmx13mmx76mm) that weights 45g. The LMC consists of three IR (Infrared Light) emitters and two IR cameras [10]. It streams data at a variable acquisition rate, that varies between mean value of less than 40 Hz [11] to up to 120 Hz [12], under both static and dynamic conditions [11]. LMC is dual platform (Macintosh/Windows) and connects to a computer via a USB 3.0 connection. It has a full-functioning Software Developer Kit (SDK). The controller itself is accesses and programmed through Application Programming Interfaces (APIs), with support for a variety of programming langguages, ranging from C++ to Python [13]. The accuracy and robustness of the LMC were evaluated [10-12,14]. During static setup a deviation between a desired 3D position and the average measured positions below 0.2mm was obtained and of 1.2mm for dynamic setups. The repeatability had an average of less than 0.17mm. Standard deviation was below 0.7mm per axis when moving to discrete positions on a path [10].  Positive value of the vertical y-axis increase upwards. The effective range of the controller extends from approximately 25 to 600 millimeters above the device (Fig. 1).   Fig. 1. Leap Motion Controller interaction area.  3rd International Conference on Sensors Engineering and Electronics Instrumentation Advances (SEIA' 2017),  20-22 September 2017, Moscow, Russia  2.2. Metrics  When the user’s hand is detected by the LMC sensor, the discrete position and orientation of the palm and fingers are recorded and processsed. The developed software allows representation of amplitude of each different movement, time interval for each movement, frequency of different movements, asymmetry of bilateral movements, standard deviation of signal amplitude, Poincaré plots. Asymmetry of movement was estimated using ratio index (RI) and symmetry index (SI). Thus assuming that XR < XL, where XR and XL are the values of the upper limb movements’ duration for the right and left limbs, the factors were calculated as follows: 𝑅𝐼 = (1 −𝑋𝑅𝑋𝐿) 100[%]                      (1) The factor indicates which of the variables has the highest value, and as such creates asymmetries. The value of RI = 0 indicates full symmetry, while RI ≥ 100% indicates asymmetry.  𝑆𝐼 =|𝑋𝐿−𝑋𝑅|0.5∙(𝑋𝐿+𝑋𝑅)∙ 100[%]                 (2) The value of SI = 0 indicates full symmetry, while SI ≥ 100% indicates its asymmetry. Metrics were implemented for the following type of movements: Bend and Stretch the Thumb; Claw Stretch; Diadochokinesis; Finger Abduction; Finger Lift; Finger Stretch; Hand/Finger Tendon Glide; Make a fist/Release and Spread the Finger; Thumb Extension/Flexion; Thumb Touch by each four Fingertips; Trace Geometrical Forms (eg., circle, spiral, zigzag, triangle, square, or rectangle); Wrist Flexion/Extension; Wrist Supination/Pronation; Wrist Ulnar/Radial Deviation. During the experiment video recording of hands movements was realized with an action camera (Ricoh Action Cam WG-M2). Video images were analysed in order to evaluate the LMC precision detection at different movements’ frequencies. Tests were realized with 6 healthy participants, mean age and range 36 (24-49). Instructions of participants were realized for the movements be realized into LMC area of interaction, that research studies have shown better accuracy for movements detection [10-12].  2.3. Serious Game  The serious game Collect Color Cube was developed using Unity and C# scrips. The LMC captures the movements of the user’s hands and fingers, and represents them as part of the virtual game scenario for upper limb motor rehabilitation. The goal of the game is to use claw stretch and fingers abduction exercises to place the coloured cubes in the collecting box that presents the same colour as the captured cubes. Points were obtained taking into account the number of cubes correctly placed in the boxes. The game allows configuration of interval of time and type of hands that can be used. The boxes can change the colour during the game. The therapist may change the setting of the game creating new scenarios for the rehabilitation sessions by changing the size of cubes, colour and texture.  3. Discussion and Conclusions  A software platform based on LMC movements’ detection was developed. It allows measurements of clinically proved effective hand and finger exercises. In Fig. 2 a spiral form traced by movement captured by LMC are represented. Can be observed the difficulties that even the healthy subjects have on realizing the task. The standard deviation of movements amplitude can be used as indicator of severity of impairments and progress in therapy.  In Fig. 3, an image of the developed serious game that used LMC for movements detection is represented. The image may be an important visual feedback for the patient with movement impairments (eg. he/she can observe that the impaired limb have more difficulties on realizing the task – more cubs are presented on affected side).   Fig. 2. Traced spiral using Leap Motion Controller detection of movements.     Fig. 3. Image from Collect Color Cube game.  LMC did not have the features of a mouse for tracing geometrical forms. As previously shown [14-15] users took a longer time to perform simple tasks, and indicated muscle fatigue and tired arms. Fortunately, these are the most important requirements for software development for upper limbs movements’ rehabilitation. The patients cannot realize the movements with high or normal velocity and need to 450100150200250300350400-     X     +200-200 -150 -100 -50 0 50 100 1503rd International Conference on Sensors Engineering and Electronics Instrumentation Advances (SEIA' 2017),  20-22 September 2017, Moscow, Russia  strengthen the muscles involved in those movements’ training.  Future research should be realized on algorithms and low cost sensors, that may be used in conjunction with LMC to compensate the errors caused by uncertainty in sampling frequency and also on strategies that may reduce the impact on perceived constrains and discomfort related with narrow LMC interaction space. . Acknowledgements  The work was supported by Fundação para a Ciência e Tecnologia, project PTDC/DTT-DES/6776/2014 and Instituto de Telecomunicações.   References   [1]. A. Haubenberger, G. Abbruzzese, P.G. Bain, N. Bajaj, J. Benito-Leon, K.P. Bhatia, G. Deuschl, M.J. Forjaz, M. Hallett, E.D. Louis, K.E. Lyons, T.A. Mestre, J. Raethjen, M. Stamelou, E.K. Tan, C.M. Testa, R.J. Elble. Transducer-based evaluation of tremor. Movement Disorder, Vol. 319, 2016, pp. 1327-1336. [2]. E.F. Shair, S.A. Ahmad, M.H. Marhaban, S.B. Mohd Tamrin, A.R. Abdullah. EMG processing based measures of fatigues assessment during manual lifting. BioMed Research International, 2017, 3937254. [3]. M.O. Ibitoye, N.A. Hamzaid, A.K. Abdul Wahab, N. Hasnan, S.O. Olatunji, G.M. Davis. Estimation of electrically-evoked knee torque from mechanomyography using support vector regression. Sensors (Basel), Vol. 16, Issue 7, 2016, pii. E1115.  [4]. O. Postolache, P.M. Girão, J.M. Dias Pereira, G. Postolache. FM-CW radar sensors for vital signs and motor activity monitoring, ICST Transactions on Ambient Systems, Vol. 1, Issue 10-12, 2011, pp. 1 - 10. [5]. F.C. Cary, O. Postolache, P.M. Girão. Kinect based system and serious game motivating approach for physiotherapy assessment and remote session monitoring. International Conf. On Sensing Technology, ICST, Vol. 1, 2014, pp. 1-5. [6]. K.H. Chen, P.C. Lin, Y.J. Chen, B.S. Yang, C.H. Lin. Development of method for quantifying essential tremor using a small optical device. Journal of Neuroscience Method, Vol. 266, 2016, pp.78-83 [7]. J.M. de Oliveira, R.C.G. Fernandes, C.S. Pinto, P.R. Pinheiro, S. Ribeiro, V.H.C. de Albuquerque. Novel virtual environment for alternative treatment of children with cerebral palsy. Computational Intelligence and Neuroscience, 2016, ID.8984379. [8]. M. Iosa, G. Morone, A. Fusco, M. Castagnoli, F.R. Fusco, L. Pratesi, S. Paolucci. Leap motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study. Top Stroke Rehabilitation, 2015, pp. 306-316. [9]. VirtualWare. (virtualwaregroup.com). [10]. F. Weichert, D. Bachmann, B. Rudak, D. Fisseler. Analysis of the accuracy and robustness of the Leap Motion controller. Sensors, Vol. 13, 2013, pp. 6380-6393. [11]. J. Guna, G. Jakus, M. Pogacnik, S. Tomazic, J. Sodnik. Na analysis of the precision and reliability of the Leap Motion sensor and its suitability for static and dynamic tracking. Sensors, Vol. 14, 2014, pp. 3702-3720. [12]. A.H. Smeragliuolo, N.J. Hill, L. Disla, D. Putrino. Validation of the Leap Motion Controller using markered motion capture technology. Journal of Biomechanics, 2016, Vol. 49, pp.1742-1750. [13]. Leap Motion Controller.( www.leapmotion.com). [14]. D. Bachmann, F. Weichert, G. Rinkenauer. Evaluation of the Leap Motion Controller as a new contact-free pointing device. Sensors, Vol. 15, 2015, pp. 214-233. [15]. A. Falcão, A.C. Lemos, M. Soares. Evaluation of natural user interface: a usability study based on the leap motion device. Procedia Manufacturing of 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and Affiliated Conferences, Vol. 3, 2015, pp. 5490-5495.           

Wrist and hand rehabilitation software platform based on leap motion controller

https://repositorio.iscte-iul.pt/bitstream/10071/16421/1/SEIA_2017_art_PG.pdf

Wrist and hand rehabilitation software platform based on leap motion controller

Abstract

Similar works

Full text

Available Versions

Repositório Institucional do ISCTE-IUL