The assessment of hand posture and kinematicsis increasingly important in various fields. This includesthe rehabilitation of stroke survivors with restricted handfunction. This paper presents a modular, ambulatory mea-surement system for the assement of the remaining handfunction and for closed-loop controlled therapy. The de-vice is based on inertial sensors and utilizes up to fiveinterchangeable sensor strips to achieve modularity and tosimplify the sensor attachment. We introduce the modularhardware design and describe algorithms used to calculatethe joint angles. Measurements with two experimentalsetups demonstrate the feasibility and the potential of such a tracking device.BMBF, 16SV7069K, Verbundprojekt: Bewegungsfähigkeit und Mobilität wiedererlangen - BeMobil -; Teilvorhaben: Nutzerzentrierte Entwicklung technischer Methoden für eine optimale Mensch-Technik-Interaktion in der Bewegungsrehabilitatio

Laidig, Daniel

Salchow-Hömmen, Christina

Schauer, Thomas

Seel, Thomas

Valtin, Markus

Current Directions in Biomedical Engineering

DepositOnce

Current Directions in Biomedical Engineering 2017; 3(1): 19–23Open AccessMarkus Valtin*, Christina Salchow, Thomas Seel, Daniel Laidig and Thomas SchauerModular finger and hand motion capturing systembased on inertial and magnetic sensorsDOI 10.1515/cdbme-2017-0005Abstract: The assessment of hand posture and kinematicsis increasingly important in various fields. This includesthe rehabilitation of stroke survivors with restricted handfunction. This paper presents a modular, ambulatory mea-surement system for the assement of the remaining handfunction and for closed-loop controlled therapy. The de-vice is based on inertial sensors and utilizes up to fiveinterchangeable sensor strips to achieve modularity and tosimplify the sensor attachment. We introduce the modularhardware design and describe algorithms used to calculatethe joint angles. Measurements with two experimentalsetups demonstrate the feasibility and the potential of sucha tracking device.Keywords: inertial measurement units; joint angle estima-tion; hand and finger kinematics; human motion analysis.1 IntroductionThe human hand is an extremely articulated system withup to 24 degrees of freedom (DoF) even for a simplifiedmodel [1]. The accurate tracking of the hand and finger posebecomes more and more important for animation, virtualreality, sports, and also rehabilitation.Current measurement systems are primarily based oninstrumented gloves with resistive, magnetic, or opticalsensing methods [2]. Resistive sensors measure the changein electrical resistance depending on the joint angle(s).The sensors have to be mounted across the finger joints,which requires an accurate placement. Each joint requiresa separate sensor. Otherwise, the sum of all joint anglesunderneath the sensor will be measured. This is why thecorrect sensor attachment is crucial and why the sensors*Corresponding author: Markus Valtin, Control Systems Group,Technische Universität Berlin, Einsteinufer 17, D-10587 Berlin,Germany, E-mail: valtin@control.tu-berlin.deChristina Salchow, Thomas Seel, Daniel Laidig and ThomasSchauer: Control Systems Group, Technische Univer-sität Berlin, Einsteinufer 17, D-10587 Berlin, Germany,E-mail: salchow@control.tu-berlin.de, seel@control.tu-berlin.de,laidig@control.tu-berlin.de, schauer@control.tu-berlin.deare often integrated into gloves, which simplify the sen-sor attachment. A specific calibration procedure is stillnecessary, depending on the number of sensors and therequired accuracy. This might be a problem if the handfunction is impaired, prohibiting a patient to execute thecalibration movements. Optical data gloves based on fiberoptics also measure the joint angle sum and therefore havethe same drawbacks as resistive data gloves, although ahigher accuracy is achieved [2].Putting on gloves can be very difficult, time consuming,and frustrating if the hand function is impaired. In a clinicalenvironment, cleaning or disinfecting the instrumentedglove is also a major issue and may hinder the use of datagloves in this field. Data gloves must have a good fit tominimize themeasurement errors, which is why universallysized gloves are not feasible.Another class of optical systems are camera-basedtracking systems. These systems use one or more camerasto track the finger segments, which are often equippedwith optical markers to enhance accuracy and robustness.The major disadvantages are the high price, the systemcomplexity, and the dependency on a known camera align-ment, which makes this kind of system unsuitable for dailylife situations, or rehabilitation. Marker-less systems withone or two cameras are more affordable, but the accuracyand robustness e.g. against overlapping finger segments isseverely reduced.Recently, inertial measurement unit (IMU) based track-ing systems have been suggested by Kortier et al. [3].The finger segments are instrumented with IMUs whichmeasure the acceleration, rate of turn, and in some casesalso the magnetic field in three dimensions. Kortier et al.employ an extended Kalman filter to estimate the handand finger kinematics using assumptions on the kinematicchain of the finger segments.We present a similar system intended as an integralpart of a functional electrical stimulation (FES) basedclosed-loop neuroprosthesis for grasping [4]. We employa modular approach in which inertial measurements (i.e.acceleration, rate of rotation, and magnetic field) areobtained for every finger segment separately. For each seg-ment, we perform sensor fusion of these measurements tocalculate the segment orientation directly, i.e. without any© 2017 Markus Valtin et al., licensee De Gruyter.This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 License.20 | M. Valtin et al.: Modular IMU hand sensor systemFigure 1: The presented hand sensor system consisting of the baseunit, two sensor strips, and one wireless sensor.assumption on the kinematic chain of the finger segments,and then combine the orientations to calculate joint angles.2 System designThe primary use case of the presented sensor systemis the hand and finger assessment and the closed-loopapplication of FES during rehabilitation of stroke survivors.The main goals of the sensor system are therefore:– accurate tracking of the hand and finger pose,– flexibility regarding the sensor setup (number oftracked fingers),– and convenient attachment of the sensors to an im-paired hand.This translates into themodular designdepicted in Figures 1and 2. The system is not integrated into a glove but consistsof up to five sensor strips, a base unit, and one wirelesssensor for the forearm. Only the base unit, which is placedon the back of the hand, is mandatory. All other parts canbe added depending on the requirements.The base unit collects the sensor raw data and sends itvia USB to the computer. Orientation estimation algorithmson the computer use this raw data to calculate the ISB-conform joint angles [5], which can be used in open- orclosed-loop applications.2.1 Base unitThe base unit is a custom printed circuit board (PCB) designwith one 32 bit ARM Cortex-M4 microcontroller unit (MCU),power supply, high side switches and connectors for thefive sensor strips, and one 9D inertial sensor (MPU9250,InvenSense Inc., San Jose, California, USA). USB 2.0 (fullspeed; 12Mbit), which is directly supported by the MCU, isUSBMCUPOWERMPU+FlashBase IMUIndexf.Middlef.Ringf.Littlef.MPU1MPU2MPU3Flash Flash Flash FlashPOWERDataFlashWirelessIMUMPU1MPU2MPU3MPU1MPU2MPU3MPU1MPU2MPU3MPU1MPU2MPU3ThumbFigure 2: Block diagram of the hand sensor system and itscomponents.used to communicate with the computer program and tosend the IMU raw data to the computer.The inertial sensor MPU9250 was chosen because ofthe fully integrated 9D measurement capabilities, the verysmall footprint of only 3× 3mm, and the support for theSerial Peripheral Interface bus (SPI). The integrated circuit(IC) combines a 3D accelerometer (ACC), a 3D gyroscope(GYR), and a 3D magnetometer (MAG), with one commoncommunication interface. SPI was chosen because of themuch higher data transfer rates compared to the Inter-Integrated Circuit bus (I2C), which is usually used withthe this type of sensor. The MPU9250 IC is susceptible topower variations, e.g. when connecting a sensor strip to thebase unit. Therefore, high side switches are used to enableand disable the sensor strips power supply and to reset thesensors if necessary.2.2 Sensor stripEach sensor strip consists of three 9D inertial sensors(MPU9250), 512 kBit SPI flash memory, and one connectorto the base unit. All components are connected by a 19 cmlong flexible PCB depicted in Figure 3. The length of thesensor strips was chosen to support different hand sizesand to deal with the length variation of the straight andbent finger. A maximum width of only 6.5mm reduces theinterference with the finger movements to a minimum.A unique strip ID, as well as strip-specific IMU cali-bration data can be stored on the flash memory. Thestrip ID is used to detect which sensor strip is connectedto the base unit. If a sensor strip is detected, the MCUprogram initializes each of the IMU sensors with a sensorM. Valtin et al.: Modular IMU hand sensor system | 21MCPPIPDIPFigure 3: Sensor strip with 3 MPU9250 9D IMUs attached exemplaryto the index finger.configuration received by the computer program. Thisconfiguration includes, among others, the measurementfrequencies and the full scale ranges for accelerometer andgyroscope. The ACC and GYR measurements can be as fastas 1 kHz, while the MAG is limited to 100Hz. Collectingthe raw IMU data reliably from the base unit and up to 5sensor strips, which may run at different frequencies, ischallenging, which is why the internal 512 byte buffer ofthe MPU9250 is used.Each MPU9250 uses its internal 1 kHz clock to generatethe desired measurement frequencies for the ACC and GYR.The MAG is independent from ACC and GYR and operateswith fixed 100Hz. The measurements for ACC, GYR, andMAG are taken automatically by the MPU9250 controllerand added to the first in, first out (FIFO) buffer. This ensuresa stable measurement frequency, regardless of delays orjitter within the MCU program. This is important if the rawIMU data is later integrated over time.The sensor strips are attached to the finger segmentswith double-sided medical tape, as depicted in Figure 3.Other ways of attachment, like rings, or clamps, werealso investigated but were found to be not as good as themedical tape. However, silicon caps for the fingertip arealso feasible, if the subject’s sense of touch is not importantfor the application.2.3 Joint angle estimationThe orientation of the IMUs is estimated by sensor fusionof the ACC, GYR, and MAG data. Since the system is likelyto be used in indoor environments and near ferromagneticmaterials, we employ an algorithm that reduces the influ-ence of non-homogeneous magnetic fields by assuring thatthe inclination (roll and pitch) portion of the orientationremains unaffected [6].The attachment of the individual sensors to the body ispredetermined by the sensor strip layout and by the layoutof the base unit. The coordinate systems of ACC, GYR, andMAG are therefore transformed to follow the ISB recommen-dations for the hand and forearm [5]. The estimated IMU ori-entations coincide with the finger segment orientations asdefined by the ISB and are represented as quaternions Segiq.They describe the segment orientation relative to a commonreference frame defined by gravity and magnetic north.To obtain the joint angle αA between Segment 1 and 2,the relative quaternion Seg2Seg1qA is calculated from the twosegment orientations surrounding the joint A bySeg2Seg1qA =Seg1qA−1⊗ Seg2qA , (1)where Seg1qA is the orientation of the proximal segmentand Seg2qA is the orientation of the distal segment. Thisrelative quaternion can be decomposed into Euler anglesgiven a sequence of rotation axes. For the hand and wrist,intrinsic z-x’-y” Euler angles are used to obtain the jointangles specified by the ISB.Writing the quaternion Seg2Seg1q asSeg2Seg1q = qw + iqx + jqy + kqz (2)yields the following z-x’-y” Euler angles:α = atan2(2(qzqw − qyqx), q2w + q2y − q2x − q2z ) (3)β = arcsin(2(qxqw + qyqz)) (4)γ = atan2(2(qyqw − qxqz), q2w − q2y − q2x + q2z ) (5)with– α: flexion (pos.), extension (neg.),– β: adduction (pos.), abduction (neg.),– γ: pronation (pos.), supination (neg.).For the approximate 1D joints (DIP, PIP, and TIP), theangles β and γ are negligible, and γ is close to zero forthe approximate 2D joints (MCP).The accuracy of the estimated joint angles is influencedby three major factors: the calibration of ACC, GYR, ANDMAG; the accuracy of the orientation estimation algorithm;the precision with which the sensors are attached to thefinger segments.While algorithms exist that allow to reducethe latter [7], no such methods are employed at the currentstate.3 Preliminary resultsThe following experimental data was obtained withSimulink (MathWorks, Natick, USA) and post-processedwith Matlab. The developed Simulink block for the handsensor system allows for a detailed configuration of the22 | M. Valtin et al.: Modular IMU hand sensor systemmodular hardware and the MPU9250 setup. The mea-surement frequency for the following preliminary resultswas 100Hz. The full scale range was 4 g for the ACC and1000deg/sec for the GYR.Figure 4 shows the joint angles for single segment flex-ion, using a mechanical finger model that allows movingthe joints one by one.Note that the angles are close to zero for a straightfinger, i.e. the sensor axes align well with the anatomicalaxes. Note furthermore that, although rotating the middleor proximal segment implies moving the distal segmentwithin the global frame, the DIP joint angle remains closeto 90 ° for t ∈ [50, 140].Time (s)40 60 80 100 120 140 160 180Angle (°)–100102030405060708090100Flex. WristFlex. MCPFlex. PIPFlex. DIPFigure 4: Flexion angles obtained on a mechanical finger modelduring single segment movements. Starting with a straight finger,the distal segment is rotated (DIP, 90 °), then the middle segment(PIP, 90 °), then the proximal segment (MCP, 90 °), then the jointsare extended again in reverse order.Figure 5 shows the original and the reconstructedfinger posture of a pinch grip. The reconstruction usesonly the measured joint angles. The segment lengths weretaken from an average handmodel. The reconstructed posematches the original posewell, despite a small gap betweenthe fingers.4 Discussion and conclusionThe presented hand sensor system enables the assessmentof hand motor function and feedback applications, e.g.in FES rehabilitation of stroke survivors. The modularsensor strip design reduces the system complexity andsetup time for common rehabilitation tasks, where trackingof all fingers or finger segments is not necessary. Theachieved accuracy is found to be sufficient for current FESapplications but will be improved in a future refinementof the sensor calibration and algorithms. Exploiting thekinematic constraints of the joints might help to eliminateangular offsets caused by potential sensor misalignmentor environmental disturbances [7].Furthermore, we plan to reduce the size of the base unit,optimize the current layout, and improve the enclosure tomeet the clinical requirements.An in-depth evaluation using an optical referencesystem is subject of current research, as well as a usersurvey.Author’s StatementResearch funding: This work was funded within the re-search project BeMobil, which is supported by the Ger-man Federal Ministry of Education and Research (FKZ200150100Y500– 506040200– 20X– 40– 600– 5050100ZFigure 5: Pinch grip captured with the hand sensor system. Only the thumb, index finger, and middle finger were measured (red).M. Valtin et al.: Modular IMU hand sensor system | 2316SV7069K). Conflict of interest: The Authors state noconflict of interest. Informed consent has been obtainedfrom all individuals included in this study. Ethical approval:The research related to human use complies with all therelevant national regulations, institutional policies andwasperformed in accordancewith the tenets of theHelsinki Dec-laration, and has been approved by the ethics committeeof the Berlin Chamber of Physicians (number Eth-25/15).References[1] Cobos S, Ferre M, Aracil R, Ortego J, Angel M. Simplified HumanHand Models for Manipulation Tasks. In Cutting Edge Robotics2010. InTech Sept; 2010.[2] Dipietro L, Sabatini AM, Dario P. A survey of glove-basedsystems and their applications. IEEE Trans Sys Man CybernetPart C: Applic Rev. 2008;38:461–82.[3] Kortier HG, Sluiter VI, Roetenberg D, Veltink PH. Assessmentof hand kinematics using inertial and magnetic sensors.J Neuroeng Rehabil. 2014;11:70.[4] Salchow C, Valtin M, Seel T, Schauer T. Developmentof a feedback-controlled hand neuroprosthesis: FES-supported mirror training. In Automed Workshop, Wismar Sep;2016.[5] Wu G, van der Helm FC, Veeger HE, Makhsous M, Van Roy P,Anglin C, et al. ISB recommendation on definitions of jointcoordinate systems of various joints for the reporting ofhuman joint motion–Part II: shoulder, elbow, wrist and hand.J Biomech. 2005;38:981–92.[6] Seel T, Ruppin S. Eliminating the effect of magneticdisturbances on the inclination estimates of inertialsensors. Proc. of IFAC World Congress 2017 (submitted);2017.[7] Seel T, Schauer T, Raisch J. Joint axis and position estimationfrom inertial measurement data by exploiting kinematicconstraints. Proc. IEEE Multi-Conference on Systems andControl; 2012. p. 45–9.

Modular finger and hand motion capturing system based on inertial and magnetic sensors

https://depositonce.tu-berlin.de/bitstream/11303/11991/2/Valtin_etal_2017.pdf

Modular finger and hand motion capturing system based on inertial and magnetic sensors

Abstract

Similar works

Full text

Available Versions

DepositOnce