wearable sensors systems for human motion analysis: sports and rehabilitation

WEARABLE SENSORS SYSTEMS FOR HUMAN MOTION ANALYSIS SPORTS AND REHABILITATION ANA SOFIA MATOS SILVA THESIS SUBMITTED IN FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN BIOMEDICAL ENGINEERING BY THE FACULTY OF ENGINEERING OF THE UNIVERSITY OF PORTO, PORTUGAL. SUPERVISOR: MIGUEL FERNANDO PAIVA VELHOTE CORREIA CO-SUPERVISOR: ORLANDO JOSÉ DOS REIS FRAZÃO FEBRUARY, 2014 © ANA SOFIA MATOS SILVA: FEBRUARY, 2014 i ABSTRACT Wearable technologies introduced a refinement to personal signal capturing by permitting a long-term on-person approach. The need for precise training regimes, continuous monitoring and quantitative analysis of movement has taken the athletes and patients out of the laboratory to their “natural environments”, in order to improve their performance, monitor their evolution, mitigate the risks of injuries and preventing the adoption of improper strategies. The system described hereby will provide a functional instrumented wearable device that has the potential to be used in different applications, from sports to rehabilitation, allowing the collection of data capable of quantitatively describe performance and movements. To do so, the system was based on a modular approach, where several sensor modules could be assembled and used together to extract meaningful information. In parallel, a proof-of-concept intensity-modulated fiber optic sensor was designed and characterized with the purpose to measure human joint angles and that could ultimately be integrated in the wearable system. The experiences developed in swimming have demonstrated that inertial sensors can provide meaningful information about swimmers’ performance through the extraction of angular displacements and temporal parameters. Moreover, different swimming styles can be differentiated by analyzing acceleration profiles. In the field of rehabilitation, the system was used in poststroke subjects in order to establish methods to analyze upper-limb performance during the reaching task, provide quantifiers to characterize the movement and better understand specific motor mechanisms underlined the task, such as the postural control. Keywords: Inertial sensors, fiber optic sensors, swimming performance analysis, stroke rehabilitation. iii RESUMO As tecnologias denominadas wearable (vestíveis) introduziram um refinamento na monitorização pessoal, ao permitir a aquisição centrada no indivíduo e por longos períodos de tempo. A necessidade de obter regimes de treino precisos, de monitorizar de forma contínua e de obter uma análise quantitativa do movimento, conduziu ao afastamento dos atletas e pacientes de laboratórios para o seu “meio natural”, de forma a melhorar o seu desempenho, acompanhar a sua evolução, mitigar o risco de lesões e prevenir a adoção de estratégias de movimento desadequadas. O sistema apresentado nesta tese é um instrumento funcional wearable que pode ser utilizado em diversas aplicações, desde o desporto à reabilitação, e que permite a aquisição de dados quantitativos capazes de descrever movimentos. Para isso, foi utilizado um conceito modular que permite a integração e combinação de diversos módulos de sensores de forma a extrair informação relevante. Em paralelo, foi criado e construído um sensor em fibra ótica baseado no efeito de atenuação, com o objetivo de medir ângulos corporais e ser integrado no sistema desenvolvido. Os resultados da utilização do sistema na natação demonstraram que os sensores inerciais têm potencial para analisar o desempenho do nadador, através de parâmetros cinemáticos. Para além disso, permitem distinguir entre diferentes técnicas de nado. Em reabilitação, o sistema foi utilizado em indivíduos que sofreram um acidente vascular encefálico de forma a estabelecer protocolos de aquisição do movimento do membro superior durante a tarefa de alcance, fornecer quantificadores que caracterizam o movimento e compreender melhor os mecanismos motores específicos da tarefa, tal como o controlo postural. Palavras-chave: Sensores inerciais, sensores em fibra ótica, análise do desempenho na natação, reabilitação, acidente vascular encefálico. v ACKNOWLEDGMENTS The number of people directly and indirectly involved in this investigation was very wide. Therefore, to all who had contributed with technical and scientific knowledge and have collaborated in some way for my personal growth, I would like to give you my gratitude. I would like to thank the Fundação para a Ciência e Tecnologia de Portugal for the financial support through a PhD grant (SFRH/BD/60929/2009). In addition, I would like to thank INESC Porto (Instituto de Engenharia de Sistemas e Computadores do Porto) and my unit UOSE (Unidade de Optoelectrónica e Sistemas Electrónicos) not only for the administrative support as my host institution but also for presenting me with the spirit of union, friendship and cooperation. A special thank to my supervisor, Prof. Miguel, and co-supervisor, Prof. Orlando, for the guidance during the investigation. I would also like to thank Prof. João Paulo Vilas-Boas for being always available to share his knowledge and advices. I must thank to all my colleagues and friends at Lab I301 who shared my achievements and anxieties and helped me when everything seemed “stuck”. Last, but definitely not the least, I would like to thank Hugo. Thank you for being my best friend, for the patience to deal with my frustrations, for making me smile and most of all for your love. To Hugo There are no mistakes. The events we bring upon ourselves, no matter how unpleasant, are necessary in order to learn what we need to learn; whatever steps we take, they're necessary to reach the places we've chosen to go. Richard Bach “The Bridge Across Forever: A True Love Story” List of Figures xiv Figure 14: Code flowchart for the Application Point (remote station). .................. 41 Figure 15: Code flowchart for the End Device (sensor node). ................................... 43 Figure 16: Global diagram of the LabVIEW’s program for managing WIMU acquisitions. ........................................................................................................................ 44 Figure 17: LabVIEW GUI for COM Port configuration and swimmer´s personal data insertion. .................................................................................................................... 45 Figure 18: LabVIEW GUI with real-time graphs of the captured data. ................... 46 Figure 19: Walking and return with WIMU located at (a) right front lower leg and (b) lower back. .......................................................................................................... 47 Figure 20: Component view of W2M2. ................................................................................ 49 Figure 21: W2M2 architecture. .............................................................................................. 50 Figure 22: Experimental procedure for the six-parameter calibration. ................. 53 Figure 23: Arduino Fio microcontroller program flowchart. ..................................... 56 Figure 24: W2M2 GUI developed in Processing. .............................................................. 57 Figure 25: W2M2 inertial data of three independent reach-press-return movement: (a) subject without pathology and (b) subject with pathology. .................................................................................................................................................. 58 Figure 26: Paper distribution under the 22nd International Conference on Fiber Optic Sensors according to measurands of interest. ........................................... 62 Figure 27: Total internal reflection phenomenon: (a) reflection of some light portion; (b) no refraction (critical angle) and (c).total reflection (Adapted from (Krohn, 2000)). ....................................................................................................... 63 Figure 28: Fiber optic sensors. (Adapted from (Krohn, 2000)) ................................ 65 Figure 29: Schematic drawing of a microbending sensor. ........................................... 67 Figure 30: Schematic of different configurations used for intensity-modulated fiber optic sensors: (a) single loops; (b) sinusoidal, (c) U-shape and (d) figure-of-eight shape. ...................................................................................................... 68 Wearable Sensor System for Human Motion Analysis xv Figure 31: Respiratory abdominal movements recorded simultaneously by two belts, one completely elastic (Belt #1) and other semi-elastic (Belt #2), embedding a bending sensor, at (a) 1310 nm and (b) 1550 nm. (Adapted from (Grillet et al., 2008)) ..............................................................................................70 Figure 32: Example of a side-polished POF cable loop. (Adapted from (Lomer et al., 2007)) .............................................................................................................................71 Figure 33: Experimental setup to validate optical sensor for gait analysis: (a) sensor attached to knee joint and video-based system for simultaneous acquisition; (b) optical sensors and video’s knee angle of a complete gait cycle. (Adapted from (L. Bilro et al., 2008)) ...........................................................72 Figure 34: (a) Photo of the sensing glove and (b) experimental data acquired during hand waving. (Adapted from (Nishiyama & Watanabe, 2009)) .......73 Figure 35: Reflected spectrum of a standard FBG sensor. ...........................................74 Figure 36: Respiratory signal captured through a FBG sensor of (a) a normal subject in sitting position and (b) a subject turning his arms. (Adapted from (Wehrle et al., 2001)) ...........................................................................................76 Figure 37: FBG sensing glove. (a) FGB sensor positioning; (b) Real-time monitoring of hand posture. (Adapted from (da Silva et al., 2011)) ............78 Figure 38: Fiber optic interferometers: (a) Mach-Zehnder configuration; (b) Michelson interferometer; (c) Fabry-Perot scheme and (d) Sagnac interferometer. (Adapted from (Krohn, 2000)) ....................................................79 Figure 39: Interferometer sensor for monitoring breathing. Top: sensor scheme; Bottom Left: sensor integration on a medical device; Bottom Right: phase shifts due to breathing pattern. (Adapted from (Mathew et al., 2012)) ......81 Figure 40: Fiber optic sensor: (a) customized piece of garment with different sensor configurations; (b) optical fiber channel detail on the fabric. ..........84 Figure 41: Relationship between curvature radius and flexion angle. ....................85 Figure 42: Schematic of sensor configurations studied: (a) single loop, (b) two loops, (c) three loops and (d) four loops. ................................................................86 List of Figures xvi Figure 43: (a) Experimental setup; (b) sensor at rest position (0º) and (c) sensor at maximum elbow flexion (60º). ............................................................................... 87 Figure 44: Light output power variation with increasing elbow flexion angle for (a) single loop, (b) two loops, (c) three loops and (d) four loops configurations. ................................................................................................................... 88 Figure 45: Comparison of sensor response for the four configurations studied.90 Figure 46: Biomechanical relevance in swimming analysis. ....................................... 97 Figure 47: Multi-camera experimental setup used to determine to swimmer’s instantaneous velocity (adapted from (Barbosa, Fernandes, Morouco, & Vilas-boas, 2008)). ........................................................................................................... 98 Figure 48: Swimming analysis using accelerometers: (a) sensor placed on swimmer’s goggles; (b) pitch and roll angles in front crawl swimming. . 101 Figure 49: Body coordinate system. Body balance and body rotation can be estimated from the pitch and roll angles, respectively. .................................. 104 Figure 50: WIMU positioned at the upper back of the athlete. ............................... 105 Figure 51: Acceleration in all three axes (X-axis: green; Y-axis: red; Z-axis: blue) for two laps crawl technique. .................................................................................... 106 Figure 52: Elapsed time between each stroke for two laps. The 10th stroke corresponds to lap end, i.e. vertical turnaround. .............................................. 107 Figure 53: Equivalent lap section acceleration signals for each performed technique. ......................................................................................................................... 108 Figure 54: Pitch angle for one lap crawl, butterfly and breaststroke techniques. ............................................................................................................................................... 109 Figure 55: Roll angle for one lap crawl, butterfly and breaststroke techniques. ............................................................................................................................................... 110 Figure 56: Reaching sub-phases according to postural control demands. ......... 118 Figure 57: Sensor positioning under consideration. ................................................... 122 Wearable Sensor System for Human Motion Analysis xvii Figure 58: Accelerometry data for Subject A and B for locations P1, P2, P3, P4 and P5. ................................................................................................................................ 123 Figure 59: Sensor positioning. ............................................................................................. 128 Figure 60: Accelerometry data for subject without pathology in positions P1, P2 and P3. ................................................................................................................................ 129 Figure 61: Accelerometry data for subject A in positions P1, P2 and P3. ........... 129 Figure 62: Accelerometry data for subject B in positions P1, P2 and P3. ........... 130 Figure 63: Accelerometry data for subject C in positions P1, P2 and P3. ............ 130 Figure 64: Accelerometry data for subject D in positions P1, P2 and P3. ........... 131 Figure 65: Sensor location for the study of postural control. .................................. 138 Figure 66: Movement duration for the ipsilesional and contralesional limbs of post-stroke subjects and for the dominant limb of healthy subjects in the (a) shoulder plane and (b) scapula plane. ............................................................ 139 Figure 67: Pitch variation for the ipsilesional and contralesional limbs of poststroke subjects and for the dominant limb of healthy subjects in the (a) shoulder plane and (b) scapula plane. ................................................................... 139 Figure 68: Roll variation for the ipsilesional and contralesional limbs of poststroke subjects and for the dominant limb of healthy subjects in the (a) shoulder plane and (b) scapula plane. ................................................................... 140 xix LIST OF TABLES Table 1: Relevant research on wearable monitoring systems. ...................................20 Table 2: Relevant healthcare and sports applications of MEMS-based inertial sensors. .................................................................................................................................24 Table 3: Comparison between popular accelerometers, gyroscopes and inertial measurement units. .........................................................................................................25 Table 4: MMA7260QT accelerometer dynamic ranges. ................................................38 Table 5: Scaling factors and zero-offset values for each module’s gyroscope. .....52 Table 6: Acceleration values for each accelerometers’ channels for the six positions. ..............................................................................................................................53 Table 7: Gains and offsets for each accelerometer in each board calculated using the six-parameter method. ............................................................................................55 Table 8: Sensitivity, standard deviation and R-squared values for each sensor configuration. .....................................................................................................................90 Table 9: Some relevant performance parameters extracted from accelerometry data for crawl technique. ............................................................................................ 107 Table 10: Demographic data and clinical scores of post-stroke subjects. ........... 120 Table 11: Sensitivity descriptive analysis of movement components for sensor locations............................................................................................................................. 125 Table 12: Demographic data and clinical scores of post-stroke patients. ........... 127 Table 13: Summary of accelerometry profiles observations. .................................. 131 Table 14: Extracted quantifiers from accelerometry profiles. ................................ 134 List of Tables xx Table 15: Study sample characterization. ....................................................................... 136 Table 16: Post-stroke group characterization ............................................................... 137 xxi LIST OF ABBREVIATIONS AMA American Medical Association AP Application Point ADC Analog-to-Digital Converter BMI Body Mass Index BSN Body Sensor Network CNS Central Nervous System DCM Direction Cosine Matrix DNA Deoxyribo-Nucleic Acid ECG Electrocardiography ED End Device EMG Electromyography FAB Functional Assessment of Biomechanics FBG Fiber Bragg Grating FMA Fugl-Meyer Motor Assessment FOS Fiber Optic Sensors GUI General User Interface IDE Integrated Development Environment List of Abbreviations xxii INS Inertial Navigation System I2C Inter-Integrated Circuit LED Light-Emitting Diode LMCA Left Medial Cerebral Artery LPG Long-Period Gratings MCA Medial Cerebral Artery MEMS Micro-Electro-Mechanical Systems MMSE Mini-Mental State Examination MRI Magnetic Resonance Imaging OLE Optical Linear Encoder OTDR Optical Time Domain Reflectometer PASS Postural Assessment Scale for Stroke patient PCB Printed Circuit Board PCF Photonic Chrystal Fiber POF Polymer Optical Fiber PWM Pulse-Width Modulation RF Radio-Frequency RMA Rivermead Motor Assessment RMCA Right Medial Cerebral Artery RPS Reach Performance Scale RSSI Received Signal Strength Indicator sEMG Surface Electromyography SMF Single-Mode Fiber SPI Serial Peripheral Interface Wearable Sensor System for Human Motion Analysis xxiii TIA Transient Ischemic Attack UART Universal Asynchronous Receiver Transmitter USB Universal Serial Bus WIMU Wearable Inertial Measurement Unit WSN Wireless Sensor Networks WPAN Wearable Personal Area Network W2M2 Wireless Wearable Modular Monitor Chapter 1 Introduction 6  rehabilitation as an alternative tool for rehabilitation monitoring, by giving means to quantitatively assess the upper-limb movement of post-stroke patients and by providing timely feedback. 1.3. Work Description The first approach to accomplish the proposed goals was the utilization of a MEMS-based inertial unit (usually known as IMU) for linear acceleration and position and orientation estimation, and its integration with a transmitting and data-logging sections. This device was named as WIMU – Wearable Inertial Measurement Unit. With the knowledge provided during the design and utilization of the first prototype, a new wearable device based on a modular approach was developed, called W2M2 – Wireless Wearable Modular Monitor. The modular feature allows the integration of different analog and digital sensors, from touch sensors to surface electrodes. In parallel with the inertial measurement units, a new fiber optic sensor was developed and tested, as a proof-of-concept, for the measurement of body segments angular displacements. The ultimate goal will be the integration and combination of both systems, electrical and optical. The WIMU and the W2M2 were used to monitor swimming athletes during training and to quantify upper-limb movement of stroke patients. Naturally several questions arise from the implementation of the system, both in swimming and rehabilitation:  In swimming: o Evaluate swimmers performance through kinematical variables; o Characterize swimmers movement; o Differentiate between swimming techniques. Wearable Sensor System for Human Motion Analysis 7  In rehabilitation: o Monitor upper-limb movement of stroke patients; o Extraction of meaningful variables to characterize the movement; o Identify compensatory strategies while performing functional tasks. The diagram presented in Figure 1 shows this thesis’ roadmap and the main research questions that led the investigation. Chapter 1 Introduction 8 Swimming Analysis  Are inertial sensors a good alternative for video-based systems?  Can IMUs provide a viable solution for “usable-time” quantititative analysis of swimming performance?  Can swimmer’s performance be improved using feedback from IMUs? Wearable Technology Rehabilitation  Identification of compensatory strategies of stroke survivors during reaching.  Can IMUs provide quantitative indicators of stroke survivors recovery status?  Can IMUs complement current methods for stroke survivors rehabilitation? APPLICATIONSSYSTEM Sensor Integration  In-system integration of sensors.  Investigation of novel sensors for motion analysis. Figure 1: Thesis description: main questions and challenges that led the investigation. Wearable Sensor System for Human Motion Analysis 9 1.4. Main Contributions The research work completed under this thesis yielded the following outcomes:  Design and development of wearable inertial measurement units, the first named as WIMU, for swimming performance analysis and the second, based on a modular approach, designated W2M2, for human motion analysis. The modular approach introduced functionality and versatility, since it allowed different sensor modules to be connected to the device and its application in several areas from sports to rehabilitation;  Design and characterization of a novel fiber optic sensor to measure elbow flexion. The sensing principle was based on macrobending effect, and a customized piece of garment was specifically designed and built to integrate the optical fiber as the sensing element;  Method for quantification of swimming athletes’ performance parameters, such as stroke frequency, stroke duration, number of laps and lap-time, from accelerometry data, obtained by using the WIMU;  Characterization of different swimming techniques from pitch and roll angles estimation;  Method for data collection and analysis of upper-limb movement of healthy or impaired subjects during specific functional tasks using the W2M2;  Identification of upper-limb compensatory strategies adopted by post-stroke patients while performing reaching functional task; Chapter 1 Introduction 10 In addition, during the research work several outcomes were submitted to scientific evaluation with success and published:  Journal articles: Salazar, A. J., Silva, A. S., Silva, C., Borges, C. M., Correia, M. V., Santos, R. S., & Vilas-Boas, J. P. (2014). Low-cost wearable data acquisition for stroke rehabilitation: a proof of concept study on accelerometry for functional task assessment. Topics in Stroke Rehabilitation, 21(1), 12-22. Silva, A. S., Catarino, A., Correia, M. V., & Frazão, O. (2013). Design and characterization of a wearable macrobending fiber optic sensor for human joint angle determination. Optical Engineering, 52(12), 126106-126106. Silva, A. S., Salazar, A. J., Borges, C. M., & Correia, M. V. (2013) Wearable Monitoring Unit for Swimming Performance Analysis. Vol. 273. CCIS Lecture Notes in Computer Science (pp. 80-93). Heidelberg: Springer. Silva, C., Silva, A., Sousa, A., Pinheiro, R., Bourlinova, C., Silva, A. S., A. J. Salazar, C. M. Borges, C. Crasto, M. V. Correia, J. P. Vilas-Boas & Santos, R. S. (2014). Co-Activation Study of Upper Limb Muscles During Reaching in PostStroke Subjects: an Analysis of Ipsilesional vs Contralesional Limb. Journal of Electromyography and Kinesiology. (accepted)  Conference papers: Silva, A. S., Casanova, O. E., Zambrano, A., Borges, C. M., & Salazar, A. J. (2012). Experiencias en Tecnología Portable para Comunicacíon y Monitoreo Personal de Bajo Costo. Paper presented at the IV Congreso Venezolano de Bioingeniería (BIOVEN 2012), San Cristobal, Venezuela. Salazar, A. J., Silva, A. S., Silva, C., Borges, C. M., Correia, M. V., Santos, R. S., & Vilas-Boas, J. P. (2012). W2M2: Wireless wearable modular monitor. A multifunctional monitoring system for rehabilitation. Paper presented at the International Conference on Biomedical Electronics and Devices (BIODEVICES2012), Vilamoura, Portugal. Wearable Sensor System for Human Motion Analysis 11 Borges, C. M., Silva, C., Salazar, A. J., Silva, A. S., Correia, M. V., Santos, R. S., & Vilas-Boas, J. P. (2012). Compensatory movement detection through inertial sensor positioning for post-stroke rehabilitation. Paper presented at the International Conference on Bio-inspired Systems and Signal Processing (BIOSIGNALS2012), Vilamoura. Portugal. Borges, C. M., Salazar, A. J., Silva, A. S., Bravo, R. J., & Correia, M. V. (2012). Estudio de factibilidad del uso de acelerometría para Análisis de Movimientos Compensatorios del Miembro Superior en Pacientes Post ACV. Paper presented at the 8th International Seminar on Medical Information Processing and Analysis (SIPAIM 2012), San Cristobal, Venezuela. Silva, A. S., Salazar, A. J., Borges, C. M., & Correia, M. V. (2011). WIMU: Wearable Inertial Mmonitoring Unit - A MEMS-based device for swimming performance analysis. Paper presented at the International Conference on Biomedical Electronics and Devices (BIODEVICES 2011), Rome, Italy. Salazar, A. J., Silva, A. S., Borges, C. M., & Correia, M. V. (2010). An initial experience in wearable monitoring sport systems. Paper presented at the 10th IEEE International Conference on Information Technology and Applications in Biomedicine (ITAB).  Conference abstracts: Salazar, A. J., Silva, A. S., & Correia, M. V. (2011). Sensor characterization for portable and wearable applications. Paper presented at the 17th edition of the Portuguese Conference on Pattern Recognition (RecPad2011), Porto, Portugal. Silva, C., Borges, C. M., Salazar, A. J., Silva, A. S., Correia, M. V., & Santos, R. S. (2011). Post-stroke patients functional task characterization through accelerometry data for rehabilitation intervention and monitoring. Paper presented at the 17th edition of the Portuguese Conference on Pattern Recognition (RecPad2011), Porto, Portugal. Chapter 1 Introduction 12 1.5. Thesis Organisation This thesis is organised in six chapters and two main parts. After this introductory chapter, Part I is dedicated to wearable sensor systems, with chapters 2 and 3, and Part II, with chapters 4 and 5, focus on the applications of wearable systems to quantitatively assess human movements. Chapter 6 closes the thesis. Part I – Wearable Sensor System, starting in Chapter 2, presents inertial systems, its main concepts and the state-of-the-art on the application of these systems as wearable devices. In addition, the design and the main features of the wearable inertial units referred to as WIMU and W2M2 is reported. Chapter 3 will focus on fiber optic sensors. The main concepts behind fiber optic technology and its application for wearable purposes will be discussed. Additionally, the development and outcomes of a new intensity-modulated fiber optic sensor for the measurement of human joint angles is presented. Part II – Human Motion Analysis: sports and rehabilitation, initiates with a chapter dedicated to swimming performance analysis (Chapter 4). The main features and results obtained from WIMU are presented and discussed. Chapter 5 focus on the analysis of stroke survivor movements during the performance of specific functional tasks. Three different studies on the analysis of upper-limb movement during reaching, either in post-stroke and healthy subjects, are presented. The final chapter includes an overall discussion of the developed wearable systems, the challenges and difficulties experienced and the main outcomes of the application of such systems in sports and rehabilitation. Moreover, the main conclusions, final remarks and future work provide the closure for this thesis. Figure 2 shows a schematic that illustrates this thesis structure. The reader interested in technological features related with the wearable system will find all the details in chapters 2 and 3, whether its interest is in inertial sensors or fiber optic sensors, respectively. The reader most interested in sports Wearable Sensor System for Human Motion Analysis 13 analysis, namely swimming performance analysis, can consult the outcomes provided by the WIMU for the analysis of swimming techniques in Chapter 4, while the reader concerned with post-stroke rehabilitation matters can focus on Chapter 5 in which the results of using inertial systems to quantify upper-limb movement performance during reaching are described. Chapter 1 Introduction Motivation Objectives Project description Contributions Part I Wearable Sensor System Chapter 2 MEMS sensors Chapter 3 Fiber optic sensors Concepts State-of-the-art WIMU W2M2 Concepts State-of-the-art Intensity FOS Part II Human Motion Analysis Chapter 4 Swimming analysis Chapter 5 Rehabilitation State-of-the-art Performance analysis through accelerometry State-of-the-art Analysis of upperlimb movement of stroke survivors Chapter 6 Discussion Conclusions Overall discussion Conclusions Future work Figure 2: Thesis structure. PART I WEARABLE SENSOR SYSTEM Chapter 2 MEMS-based Inertial Sensors 22 2.1. Accelerometers and Gyroscopes MEMS accelerometers have become an attractive tool for use in wearable systems for detecting and measuring aspects of human movement. To extract the acceleration value, typically the accelerometer has a movable mass which is connected to a fixed frame via spring structures (see Figure 4). An external acceleration will displace the mass from its rest position. The magnitude of this displacement is proportional to the magnitude of the acceleration and inversely proportional to the stiffness of the spring structures. The principle for converting the displacement of the proof mass into an electrical signal can be fulfilled by different mechanisms, such as capacitive, piezoresistive, piezoelectric, optical or tunnelling current (Maluf, 2004). The most commonly used and versatile operation mode is the differential capacitive measurement. Measuring capacitance changes caused by displacement provides a large output signal, good steady-state response, and better sensitivity due to low noise performance. The main drawback is that capacitive sensors are susceptible to electromagnetic fields. The principle of operation is based on two parallel capacitor plates that are fixed while the third one in the middle moves. This movement will increase one capacitance and will decrease the other. Differential capacitors provide a signal that is zero at the balance point and carries a sign which indicates the direction of motion. Accelerometers are gravity sensitive. This means that their orientation affects their output. If they are oriented with the active element perpendicular to the axis of gravity, they will register the effect of gravity on the mass mounted on the beam and so give an accelerometer reading of 9.81 m/s2 or 1 g. If the accelerometer is rotated 90º, the axis of gravity will run parallel to the mass and so it will not deform the beam on which it is mounted. In this case the accelerometer will give an output reading of 0 g. The accelerometer output then Figure 4: Accelerometer model. Wearable Sensor System for Human Motion Analysis 23 represents the vector sum of the gravity (static) and kinematic (dynamic) accelerations of self-movement. Micro-machined gyroscopes usually rely on a mechanical structure that is driven into resonance and excites a secondary oscillation in either the same structure or in a second one, due to the Coriolis force. The amplitude of this secondary oscillation is directly proportional to the angular rate signal to be measured. The Coriolis force is a virtual force that depends on the inertial frame of the observer. The effect of this force is the apparent deflection of moving objects from a straight path when they are viewed from a rotating frame reference (Maluf, 2004). A model of a micro-machined gyroscope is illustrated in Figure 5. The proof mass is excited to oscillate along the xaxis with a constant amplitude and frequency. Rotation about the z-axis couples energy into an oscillation along the y-axis whose amplitude is proportional to the rotational velocity. Any motion along the sense axis is measured and a force is applied to counterbalance this sense motion. The magnitude of the required force is then a measure of the angular rate signal. Thus, the Coriolis force induces a motion in a third direction, perpendicular to both the direction of rotation and the driven motion (Lapadatu, 2009). There are some problems concerning the fabrication of those structures. For example, the amplitude of the Coriolis is very small which results in a small displacement. It is also very difficult to design structures for an exact resonant frequency, and since these structures need to be continuously excited the power consumption is usually higher than other micro-machined devices. Also, gyroscopes can present some drift problems, which means they do not return to zero-rate when rotation stops. (Baluta, 2009) Nevertheless, micro-machined gyroscopes are still one of the best commercially available solutions for angular Figure 5: Gyroscope model. Chapter 2 MEMS-based Inertial Sensors 24 rate measurements and combined with other inertial sensors present a viable solution to monitor displacement rates. These inertial sensors detect and measure motion, with minimal power and size, and are valuable to nearly any application where movement is involved. Table 2 outlines some of the basic pertinent healthcare and sports applications by motion type. Table 2: Relevant healthcare and sports applications of MEMS-based inertial sensors. Motion Type Acceleration/ Position Tilt Angular Rate/ Angle Vibration Sensor Fusion Healthcare CPR Assist Bed-patient Positioning Scanning Instruments Tremor Control Precision Surgical Navigation Activity Monitors Blood Pressure Monitors Basic Surgical Tools Remote Diagnostics Biofeedback Monitors Imaging Equipment Prosthetics Rehabilitation Assistance Gait Analysis Sports Activity Monitors Control Applications Rate Monitors Referee Assistance Biofeedback Equipment Accessories Pedometers Performance Improvement Injury Prevention Table 3 presents a comparison between some popular MEMS-based accelerometers, gyroscopes and inertial measurement units currently used as wearable monitoring sensors. Wearable Sensor System for Human Motion Analysis 25 Table 3: Comparison between popular accelerometers, gyroscopes and inertial measurement units. Manufacturer Sensing Axes Output Type Dynamic Range Sensitivity Operating Voltage Current Consumption Accelerometers MMA7361 Freescale Semiconductor 3-axis Analog ± 1.5/6g from 200 mV/g 2.2V – 3.6V 400 µA BMA180 Bosch 3-axis Digital (4-Wire, SPI, I2C) ±1/1.5/2/3/4/ 8/16 g from 512 LSB/g 1.6V – 3.6V 650 µA ADXL345 Analog Devices 3-axis Digital (SPI, I2C) ± 2/4/8/16 g from 256 LSB/g 1.7V – 2.75V 145 µA Gyroscopes IDG300 InvenSense 2-axis (x,y) Analog ± 500 º/s 2 mV/º/s 3V – 3.3V Not Specified ITG3200 InvenSense 3-axis Digital (I2C) ± 2000 º/s 14.375 LSB/º/s 2.1V – 3.6V 6.5 mA ADXRS450 Analog Devices 3-axis Digital (SPI) ± 300 º/s 80 LSB/º/s 3V – 5.25V 6 mA Sensor Fusion SCC1300 muRata 3-axis acc 3-axis gyro Digital (SPI) ± 6g ± 300 º/s 650 LSB/g 18 LSB/º/s 3V – 3.6V 10 mA MPU9150 InvenSense 3-axis acc 3-axis gyro 3-axis compass Digital (I2C) ±2/4/8/16g ±250/500/ 1000/2000 º/s ± 1200 µT from 2048 LSB/g 16.4 LSB/º/s 0.3 µT/LSB 2.4V – 3.6V 4 mA ADIS16300 Analog Devices 3-axis acc 3-axis gyro Digital (SPI) ± 3g ± 300º/s 0.6 mg/LSB 0.05 º/s/LSB 4.75V – 5.3V 42 mA Chapter 2 MEMS-based Inertial Sensors 26 2.2. Inertial Systems Concepts While simple motion detection (linear movement along one axis, for example) is valuable to a number of applications, such as detecting whether an elderly person has fallen, a majority of applications involve multiple types and multiple axes of motion. Being able to capture this complex, multi-dimensional motion can enable new benefits while maintaining accuracy in the most critical of environments. In many cases, it is necessary to combine multiple sensor types – linear and rotational, for instance – in order to precisely determine the motion an object has experienced. As an example, accelerometers are sensitive to the Earth’s gravity, so they can be used to determine inclination angle. As a MEMS accelerometer is rotated through a 1 g field (90º), it is able to translate that motion into an angle representation. However, the accelerometer cannot distinguish static acceleration (gravity) from dynamic acceleration. In the last case, an accelerometer can be combined with a gyroscope and post-processing of both devices can discern the linear acceleration from the tilt, based upon known motion dynamics models. This process of sensor fusion obviously becomes more complex as the system dynamics (number of axes of motion, types, and degrees of freedom of motion) also increases complexity. The two primary challenges found in any high-performance motion capture implementation are the conversion of raw sensor data to calibrated and stable sensor data, and the translation of precision sensor data into actual position/tracking information, as depicted in Figure 6. Overcoming the first hurdle involves motion calibration, which is based on intimate knowledge of motion dynamics. The second hurdle requires merging an understanding of motion dynamics with a deep knowledge of the peculiarities of the application at hand. Fortunately, many of the principles required for solving these challenges are based on proven approaches from classical industrial navigation problems, including sensor calibration, fusion and processing techniques. Wearable Sensor System for Human Motion Analysis 27 Figure 6: Motion capture implementation steps. Analysis of accelerometer and gyroscope data to determine position, velocity and attitude, can be derived from inertial navigation systems (INS). In an inertial navigation system the principle is to determine navigation parameters through acceleration and velocity integrations. The first integration provides velocity and the second gives displacement with respect to an initial point. To determine the navigation parameters in a certain frame, the acceleration projections on that frame must be provided. In addition, gyroscopes are needed to reference the sensitive axes of the accelerometers with a certain reference frame. Usually, the algorithm used to perform such operations is called strapdown inertial navigation system and uses both information of accelerometers and gyroscopes to estimate position and velocity (Salychev, 2004). A strapdown navigation system usually contains three accelerometers and three gyroscopes (or three-axis versions of them), which measure the projections of acceleration and angular velocity on their sensitive axes. In order to re-calculate the above projections into the navigation frame, the direction cosine matrix (DCM) between the body (b) and the navigation (n) frame is needed (Titterton, 2007): (1) Chapter 2 MEMS-based Inertial Sensors 28 where an and ab are the acceleration projections in n-frame and b-frame, respectively, and is a 3x3 matrix which defines the attitude of the body frame with respect to the n-frame. The direction cosine matrix, , may be calculated from the angular rate measurements provided by the gyroscopes using the following differential equation (Titterton, 2007): (2) where is the skew symmetric matrix (Titterton, 2007): (3) where is the vector which represents the angular rate of the body as measured by the gyroscopes. The diagram shown in Figure 7 presents the main functions to be implemented within a strapdown inertial system. The processing of the rate measurements to generate body attitude, the resolution of the accelerations into the inertial reference frame, gravity compensation and the integration of the resulting acceleration, are the steps to estimate velocity and position. Wearable Sensor System for Human Motion Analysis 29 Figure 7: Strapdown inertial navigation system. (Adapted from (Titterton, 2007)) In order to start the computation, the initial coincidence between the accelerometer sensitive axes and the reference frame is needed; this is called alignment of the INS. In a strapdown INS this procedure is done to estimate the initial value of the direction cosine matrix. DCM is usually computed through the Euler angles. These angles are three independent quantities able of defining the position of one coordinate frame with respect to another. Let’s coincide xn yn zn coordinate frame with xb yb zb coordinate frame and consider three righthanded rotations on the Euler angles (see Figure 8):  yaw (ψ), rotation about the z-axis;  roll (φ), rotation about the x-axis;  pitch (θ), rotation about the y-axis. Chapter 2 MEMS-based Inertial Sensors 30 Figure 8: Coordinate transformation from navigation frame to body frame. (a) Yaw – rotation about the z-axis; (b) Roll - rotation about the x-axis and (c) Pitch - rotation about the y-axis. The first rotation is made about the z-axis by angle ψ. A position vector in the new system can be expressed in terms of the original coordinates as: (4) Or, in matrix form: (5) Wearable Sensor System for Human Motion Analysis 31 Similarly, the transformation due to second rotation, around x-axis can be described as: (6) And finally, third rotation about the y-axis has a transformation in the form: (7) The total transformation matrix can be defined using multiplication of C3, C2 and C1 matrices. Thus the transformation due to all three rotations, from navigation frame to body frame is given by: (8) There is no standardized definition of the Euler angles, thus if the order of the rotations is interchanged, different direction cosine matrix is defined. So, it is important to define right at the beginning the rotation order. The direction cosine matrix is an orthogonal matrix, which means C-1=CT. This is an important property since it makes easy the transformation from one coordinate system to another and backwards (Titterton, 2007). Following the example above, the transformation matrix from body frame to navigation frame is given by: (9) Chapter 2 MEMS-based Inertial Sensors 38 Figure 12: WIMU components: (a) MMA7260QT accelerometer board; (b) IDG-300 gyroscope board and (c) eZ430-RF2500 debugging board (left) and target board (right). The MMA7260QT is a capacitive micromachined accelerometer with onchip signal conditioning, temperature compensation and g-select pins which allow the selection of four different dynamic ranges (Table 4) ("±1.5g - 6g Three Axis Low-g Micromachined Accelerometer ", 2008). Table 4: MMA7260QT accelerometer dynamic ranges. Dynamic Range Sensitivity ± 1.5 g 800 mV/g ± 2 g 600 mV/g ± 4 g 300 mV/g ± 6 g 200 mV/g Similarly, the IDG-300 Invensense gyroscope can be found integrated in an evaluation board (see Figure 12 (b)) along with the components necessary for application-ready functionality. The IDG-300 gyro uses two sensor elements with a vibrating dual-mass bulk silicon configuration that sense the rate of rotation about the Xand Y-axis ("IDG-300 Dual-Axis Gyroscope Evaluation Board Specification ", 2007). Wearable Sensor System for Human Motion Analysis 39 In order to integrate both the accelerometer and the gyroscope into the inertial unit, the Texas Instrument microcontroller based board, eZ430-RF2500, was selected from the many commercially available devices. The eZ430-RF2500 uses the MSP430F2274 microcontroller which combines 200 Ksps 10-bit analog-to-digital converter (ADC) with the CC2500 multi-channel radiofrequency (RF) transceiver, designed for low-power wireless applications. This board offers a combination of hardware and software appropriate for fast prototyping of wireless projects, while offering low-power consumption. The development tool includes two target boards and a USB debugging interface (see Figure 12 (c)). The eZ430-RF2500 uses the IAR Embedded Workbench Integrated Development Environment (IDE) to write, download and debug an application. The eZ430-RF2500 can readily use the SimpliciTI wireless communication protocol, a low-power RF (2.4 GHz) protocol aimed for simple and small RF networks (proprietary of Texas Instruments). The SimpliciTI protocol claims to use a minimal set of microcontroller requirements, which in theory lowers the associated system cost. In this case, the network topology was configured so that the WIMU behaved as an end device (ED), transmitting data packets to a remote or base station referred to as application point (AP). The protocol permits multiple end devices, therefore multiple WIMUs can be allocated at different body segments, in a truly body sensor network scheme. The WIMU architecture can be seen on Figure 13. Chapter 2 MEMS-based Inertial Sensors 40 Figure 13: WIMU architecture. On the WIMU side, minimal pre-processing of the gathered data occurs, just enough to prepare the data for adequate transmission. The microcontroller embedded code performs no complex algorithms: the acceleration and angular velocity signals are acquired and converted sequentially by the 10-bit ADC integrated in the MSP430F2274 micro-controller at a sampling rate of approximately 50 Ksps. The ADC is kept in a continuous loop, converting sequentially all sensor inputs. A simple broadcasting scheme was used for the data gathering: the WIMU acquires and transmits while the receiver station behaves unique as a listener. Although this strategy introduces reduced timing errors and is not applicable for faster transmission rates or for multi-location sensors synchronization, it proved to be effective during the experimental acquisitions stages, providing a modest rate of approximately 7 packets/sec. Additional data processing operations at the WIMU level were left for postprocessing analysis, in order to alleviate microcontroller resources and speed up the communication process. Wearable Sensor System for Human Motion Analysis 41 Figure 14 shows the flowchart of the application developed for the AP, i.e., the remote base station. The first thing the AP does is to initialize both the communication between the MSP430 and the CC2500 radio and the LEDs on the board that are to be used. After initialization, a random 4-byte address is created and written in flash memory for reuse on system reset. Initialize Radio Address is written on FLASH? Retrieve address from FLASH Create random address and write on FLASH No Yes Initialize MSP430 Initialize network sJoinSem, sSelfMeasureSem or sPeerFrameSem Listen for a link sNumCurrentPeers ++ Decrement sJoinSem Read Temperature and Voltage Format into a msg Transmit msg to PC For (i=0; i < sNumCurrentPeers; i++) Define input msg buffer Retrieve RSSI from sender Msg received? Transmit RSSI and msg to PC Yes sSelfMeasureSem sPeerFrameSem No i==sNumCurrentPeers sJoinSem Figure 14: Code flowchart for the Application Point (remote station). The MSP430 is then initialized: the main clock is set to run at 8 MHz and the UART (Universal Asynchronous Receiver Transmitter) is initialized to communicate with the PC COM port with Baud Rate of 9600 bits/second. Once Chapter 2 MEMS-based Inertial Sensors 42 the hardware is initialize, the AP enters in a while loop waiting for three different events to occur. The events are identified by three semaphores:  sJoinSem: this semaphore is set when an ED request a network join. This is actually a side effect of an ED initialization procedure. On a successful link creation, the sJoinSem increases the number of devices that the AP recognizes as part of the network;  sPeerFrameSem: this semaphore is incremented every time the AP receives a message from an ED. In this case, the AP first defines a message buffer to store the incoming frame and then searches for messages until it has processed all waiting frames. The messages are formatted containing the sensors data, the ED identification (its address) and the Received Signal Strength Indicator (RSSI) value, and are transmitted to the PC via COM Port;  sSelfMeasureSem: it is an AP specific semaphore that is set at a defined time interval and executes a particular routine. This routine uses the temperature sensor integrated in the ADC to measure self-temperature and transmits it to the PC. The ED (sensing node) code flowchart is depicted in Figure 15. The initialization procedure is similar to the one AP uses. Once ED’s hardware is initialized, a join is requested and the link to the AP is created. The ED enters then in a loop dedicated for converting sensors data and transmits messages to the AP. All the five sensor inputs (three accelerations, Ax, Ay and Az, and two angular velocities, Vx and Vy) are acquired and converted sequentially. After correct conversion an ADC flag is triggered indicating the end of conversion. The time-stamp is then retrieved from TIMER A and all the data is ready to be formatted into a message and to be transmitted to the AP. The complete microcontroller code either for the AP and ED is found in Appendix A. Wearable Sensor System for Human Motion Analysis 43 Initialize Radio Address is written on FLASH? Retrieve address from FLASH Create random address and write on FLASH No Yes Initialize MSP430 Initialize network Link to Access Point Timer A (Real Time Clock) Convert all analogical sensor inputs sequentially Retrieve time-stamp Format sensor data and time-stamp into a msg for transmission Transmit msg to Access Point Conversion Completed Figure 15: Code flowchart for the End Device (sensor node). A custom software application for the acquisition, processing and visualisation of the sensor packets coming from the WIMU was developed using National Instruments´ LabVIEW platform for the computer interface. The programing was based on a single state-machine architecture, which allows the program to change the way it executes based on user inputs/actions and results of the application, i.e., it responds intelligently to a stimulus. A diagram of the global software architecture is shown in Figure 16. The LabVIEW’s general user interfaces (GUI) for the configuration stage and data visualisation are shown in Figure 17 and Figure 18, respectively. Chapter 2 MEMS-based Inertial Sensors 44 START IDLE Configure COM Port Configure Personal Data Process Data Read from File CLOSE SESSION Figure 16: Global diagram of the LabVIEW’s program for managing WIMU acquisitions. The user can either start an experimental acquisition or perform an offline analysis by reading saved data from a previous measurement. The acquisition routine starts with COM port parameters configuration and swimmer’s personal information retrieval (the application allows this information to be store in a database). During the acquisition itself, the program enters a producer/consumer loop, which enhances data sharing between loops running at different rates. The data produced in one loop can be buffered for posterior processing in the consumer loop by creating queues, ensuring that no message received from the WIMU is lost during the processing stage. The consumer loop is responsible for extracting data from the sensor packet, convert it to a proper format and display it in real-time graphs. Wearable Sensor System for Human Motion Analysis 45 Figure 17: LabVIEW GUI for COM Port configuration and swimmer´s personal data insertion. Chapter 2 MEMS-based Inertial Sensors 46 Figure 18: LabVIEW GUI with real-time graphs of the captured data. Wearable Sensor System for Human Motion Analysis 47 The WIMU was firstly intended for swimming performance analysis. Nonetheless, due to its architecture, small size and weight it can be placed in different parts of the human body and used for a variety of applications. Two examples of data collected at different body segments, namely at right frontal leg and lower-back, during walking is presented in Figure 19 (a) and (b), respectively. Figure 19: Walking and return with WIMU located at (a) right front lower leg and (b) lower back. Chapter 2 MEMS-based Inertial Sensors 54 For the case of the x-channel, if is the first measurement when aligned for +1 g and is the second measurement when aligned for -1 g, then Equation 12 gives: (13) Solving the above equations, the gain and offset for x-channel can be easily obtained: (14) The y and z-channel calibration parameters are similarly given as: (15) (16) These six-parameter computed with this method can then be applied to the accelerometer output using Equation 12. This calibration method was applied for each accelerometer on each board utilized during the experimental acquisitions. The correspondent gains and offsets for each board’s accelerometers are presented in Table 7. Wearable Sensor System for Human Motion Analysis 55 Table 7: Gains and offsets for each accelerometer in each board calculated using the sixparameter method. Gain, S’ Offset, O’ (bits) x-channel y-channel z-channel x-channel y-channel z-channel MCU Board -29.037 -28.621 -1.018 -3590.203 -3664.015 -3.253 Acc Board -0.979 -0.968 -1.019 12.691 0.476 -4.887 COMBO Board -0.981 -0.976 -1.001 1.103 -0.762 -8.767 MPU6050 Board -1.000 -0.994 0.985 107.742 377.565 516.155 The microcontroller on the Arduino Fio board (MCU board) was programmed using the Arduino programming language (based on Wiring 8 ). The control unit is responsible for initializing all modules and communications. Afterwards, the microcontroller enters in an idle mode where it waits for user commands. There are four commands available:  Start Acquisition: this command initiates data sending, by reading all sensor outputs and format sensor packet to be wirelessly transmitted;  Stop Acquisition: the program interrupts data sending and returns to idle mode;  Data Format Raw: the data from sensor outputs is kept in units per bits count;  Data Format Scaled: data from sensors outputs is converted to proper units according to the measure being read. The Arduino Fio code flowchart is depicted in Figure 23. The complete microcontroller code is found in Appendix A. 8 http://wiring.org.co/ Chapter 2 MEMS-based Inertial Sensors 56 Start Initialize SPI communications Configure MCU Accelerometer Initialize I2C communications Wait for user’s command Check which sensor boards are connect and configure them Start Acquisition Data Format Raw Data Format Scaled Stop Acquisition 3 2 4 1Check touch sensors Read sensor outputs Retrieve time-stamp Transmit msg to XBee Read raw data from sensor Perform scaling operations Figure 23: Arduino Fio microcontroller program flowchart. To handle data coming from the W2M2 a simple and easy-to-use GUI was developed in Processing 9 . Processing is an open-source programming language and integrated development environment (IDE) built for the electronic arts and visual design communities. It uses simplified syntax and graphics programming model, and it has more than one hundred libraries. As mentioned before, the concept behind the W2M2 was to make a functional device where no relevant expertise in electronics and programming is necessary in order to use it and perform acquisitions. This feature is crucial given the potential users of such device, i.e., physicians, patients, athletes or coaches. The user interface developed using Processing was created in line with this principle. 9 http://processing.org Wearable Sensor System for Human Motion Analysis 57 Figure 24 shows the user interface developed for the W2M2. It allows for hardware configuration, synchronization with video system, storing patient information and data acquisition to file. Figure 24: W2M2 GUI developed in Processing. An example of accelerometry data collected with W2M2 during experiments with post-stroke patients for rehabilitation purposes can be seen in Figure 25. The inertial data illustrates the difference between three independent reach-press-return trials performed by a typical subject (Figure 25 (a)), with no neural nor musculoskeletal pathologies, and a stroke survivor (Figure 25 (b)). The 3-axis accelerometers data was captured at a frequency of approximately 100 Hz, which was then buffered and transmitted wirelessly. Chapter 2 MEMS-based Inertial Sensors 58 After package format verification, the data was processed in Matlab 10 by applying a simple moving average smoothing filter. Figure 25: W2M2 inertial data of three independent reach-press-return movement: (a) subject without pathology and (b) subject with pathology. 10 http://www.mathworks.com/ Wearable Sensor System for Human Motion Analysis 59 Looking at these figures, clear differences are observed between movement path of typical vs. post-stroke individuals, namely the smoothness and the consistency of the reach and return movements. 2.5. Summary Recent advances in MEMS-based sensors, such as miniaturization and power consumption, have allowed their presence in today’s wearable solutions. Accelerometers and gyroscopes dominate the movement/position category and are becoming each day more accessible for numerous research opportunities. The first inertial prototype developed under this thesis was called WIMU, and was primarily intended for swimming performance analysis. It comprised an accelerometer and a gyroscope as sensing elements, a microcontroller for data packaging and wireless transmission. The experimental tests made with this system revealed some drawbacks: it provided a low packet transmission rate, did not allowed for sensor expansion and integration with other acquisition systems and lacked of memory storage for measurements in which wireless transmission could not be possible. The W2M2 was based on a modular approach, where several sensor modules can be assembled and used together to extract meaningful information according to the application requirements. Instead of a complex sensor system, the key aspect of this system was to combine functionality and usability in a simple wearable solution. Along with simplicity comes small size and weight, which are crucial features when referring to wearable technology and sensor integration. The biggest improvement of W2M2 with respect to WIMU was the concept of modular device: multiple modules, connected through digital communication (although analog connection could be also possible), which could be linked to the processing unit, transforming the device into a multipurpose monitoring system. The W2M2 allows the interconnection of accelerometer, magnetometer and gyroscope boards, electrodes for surface Chapter 2 MEMS-based Inertial Sensors 60 electromyography (sEMG) and also On/Off sensor; nonetheless the use of standard communication protocols such as I2C and SPI allow interconnecting to a wide variety of commercially available sensors which may be added to address particular purposes. In addition, data transmission rate was significantly improved, from roughly 10 packets/s with the WIMU device to a maximum of 500 packets/s with the W2M2. Wearable Sensor System for Human Motion Analysis 61 Chapter 3 FIBER OPTIC SENSORS Fiber optic technology emerged in the 60s with the description of an optical waveguide fiber (Snitzer, 1961) and was later boosted by the demands on optical communications. Since then, a rapid evolution on optical devices and fibers technology has been noticed, especially concerning fiber fusion splicing devices, fiber coupling and intensity losses measuring equipments, such as the Optical Time Domain Reflectomer (OTDR) (Frazao, 2009). In parallel, an increasing interest on using optical fibers in different areas rather than communications has triggered the fiber optic sensor technology. The initial idea of using these fibers only for communication purposes was replaced by the perception that the optical fiber could also be used as a sensing element to monitor several physical and chemical quantities. Later, the concept of using fibers as both sensing element and communication channel has demonstrated the huge potential of fiber optic sensors which made them a strong competitor among sensors technology. Even so, fiber optic sensors (FOS) still remain apart from most engineers and researches, perhaps due to the more conventional use of non-optical (mostly electrical) technologies and also given the limited number of available turn-key solutions. FOS represent a technology base that can be applied to a multitude of sensing applications, since most physical properties can be sensed optically with fibers. Light intensity, displacement, temperature, pressure, strain or flow are just some of the phenomena that can be measured (Krohn, 2000). Nowadays, technology can challenge traditional sensors in a large number of innovative applications. Figure 26 shows the distribution of papers presented at the 22nd International Conference on Optical Fiber Sensors in 2012 according to Chapter 3 Fiber Optic Sensors 62 measurands of interest. This conference is a major event in the field of fiber optic sensors. The most highly reported developments are found in the field of chemical and gas sensors, followed by strain and temperature optical fiber sensors. Biosensors have suffered a substantial growth compared with past conferences (from 2.4% to 12.63%) (Lee, 2003). Figure 26: Paper distribution under the 22nd International Conference on Fiber Optic Sensors (2012) according to measurands of interest. A huge number of fiber optic sensors for biomedical applications has been reported in the last years. The most referenced optical sensing devices are for biomechanical, biological and physiological measurands. In the field of biomechanics, FOS based on different sensing principles are described to measure force and pressure (Roriz, Frazão, Lobo-Ribeiro, Santos, & Simões, 2013), as well as to monitor different joint angle (Kwang Yong et al., 2008). For biological purposes, several optical devices have been developed to target cells and proteins (Velasco-Garcia, 2009), identify DNA (Deoxyribo-Nucleic Acid) markers (Leung, Shankar, & Mutharasan, 2007) or to measure humidity/moisture (Yeo, Sun, & Grattan, 2008). In addition, physiological Wearable Sensor System for Human Motion Analysis 63 parameters such as temperature, pH (Shao, Yin, Tam, & Albert, 2012) or heart beat (Kreber, 2013) can be sensed with fiber optic devices. 3.1. Fiber Optic Sensor Concepts Light propagation inside fibers is due to the phenomenon of total internal reflection. It is common sense that refraction occurs when light passes from one homogeneous medium to another, according to the Snell’s Law: 2211 sinsin  nn  (17) where n1 and n2 are the refractive indices of the two mediums and θ1 and θ2 are the angles of the incident and refractive light rays, respectively. Therefore, when the light passes from a high-index of refraction medium to a lower-index medium, a certain portion of the incident light is reflected (Figure 27 (a)). If the incident ray hits the boundary with an angle equal to the angle between the refracted ray and the normal to the interface, no refraction will occur (Figure 27 (b)). This angle is known as the critical angle. Incident ray with angles greater than the critical angle will be entirely reflected at the interface and no refraction takes place (Figure 27 (c)). This is the total internal reflection phenomenon. Figure 27: Total internal reflection phenomenon: (a) reflection of some light portion; (b) no refraction (critical angle) and (c).total reflection (Adapted from (Krohn, 2000)). Chapter 3 Fiber Optic Sensors 70 Figure 31: Respiratory abdominal movements recorded simultaneously by two belts, one completely elastic (Belt #1) and other semi-elastic (Belt #2), embedding a bending sensor, at (a) 1310 nm and (b) 1550 nm. (Adapted from (Grillet et al., 2008)) A different application of an optical sensor that uses the microbending concept is the case of a plantar pressure monitor device, used to control and prevent the plant ulcers (that can lead to infection and subsequent amputation) associated with complications caused by diabetes disease. In this study, a fiber optic sensor array was built, which consists of an array of optical fibers lying in perpendicular rows and columns separated by elastomeric pads. Intensity attenuation is caused by the physical deformation of two adjacent perpendicular fibers (Wang, Ledoux, Sangeorzan, & Reinall, 2005). Nonetheless, this new sensor still lacks calibration optimization, especially for fibers located near the periphery of the array, and needs improvements related to repeatability. When considering silica fibers as the tool to build bend-loss sensors, one issue that immediately appears is the problem of fiber breakage. If no special attention is given to the bending limit of the fiber, the sensor functionality can be compromised. In order to overcome such limitation, polymer-based optical fibers, also known as POF, can be used. Besides the advantages of elasticity and easy-handling, this type of fibers suffer from higher attenuation and distortion compared with traditional silica fibers; where this represents a drawback for telecommunication purposes, it also means increased detection capabilities when considering macrobending sensors. In addition, since POF usually have larger diameters, they permit the use of plastic connectors with lower precision. Wearable Sensor System for Human Motion Analysis 71 POF macrobending sensors have been reported for monitoring seated spinal posture (Dunne, Walsh, Smyth, & Caulfield, 2006) and for monitoring rescuers’ heart rate and respiratory rate wearing a smart fabric suitable to be used in high-risk and complex environments (i-Protect project) (Witt, Krebber, Demuth, & Sasek, 2011). These sensors have also been utilized to detect the flexion angles of finger joints for commercial sensing gloves (Zimmerman & Lanier, 1987). Still, POF sensors alone have low sensitivity and there are some error sources in power losses measurements (Kyoobin & Dong-Soo, 2001). One method that improves the sensitivity of a bent optical fiber based sensor is sidepolishing (L. Bilro, Pinto, Oliveira, & Nogueira, 2008; Lomer, Quintela, LópezAmo, Zubia, & López-Higuera, 2007). The side-polishing is accomplished by removing a portion of the jacket, cladding and core of the fiber, bringing a polished elliptical surface out into the open. This polished region of the fiber core is in direct contact with the external medium, with a specific refraction index. If one combines this method with a loop configuration, a sensor with twice the sensitivity is obtained (see Figure 32). In this case, the input rays suffer refraction from the curved region and a subsequent refraction due to the polishing area (Lúcia Bilro, Alberto, Pinto, & Nogueira, 2012). Figure 32: Example of a side-polished POF cable loop. (Adapted from (Lomer et al., 2007)) Chapter 3 Fiber Optic Sensors 72 This sensor configuration was proposed as a liquid-level sensor, using the changes of the refraction index in the sensing area to promote variations in the optical power (Lomer et al., 2007). Also similar sensor configurations have been reported for gait analysis, by measuring human joints angles, with good results when compared with simultaneous video analysis (see Figure 33)(L. Bilro et al., 2008). Figure 33: Experimental setup to validate optical sensor for gait analysis: (a) sensor attached to knee joint and video-based system for simultaneous acquisition; (b) optical sensors and video’s knee angle of a complete gait cycle. (Adapted from (L. Bilro et al., 2008)) Another referred concept, based on macrobending losses, that has intrinsically more sensitivity than silica fibers to measure intensity variations is the use of hetero-core fiber optic sensors. Hetero-core optical fibers are fabricated by inserting a small portion of fiber with a smaller core diameter into two identical fibers with larger core diameters. The cladding diameters of the fibers should be the same. Hetero-core fiber optic sensors have large sensitivity because of the power coupling that takes place at the interface, which makes leakage easier when an external deformation occurs (Efendioglu, Sahin, Yildirim, & Fidanboylu, 2011). This way, hetero-core sensors have been found to have highly sensitive and reproducible performance in response to macrobending based on optical intensity variations and, additionally, are unaffected by temperature fluctuations (Nishiyama, Sasaki, & Watanabe, 2006). Wearable Sensor System for Human Motion Analysis 73 Some researchers have developed a wearable sensing glove with embedded hetero-core fiber-optic sensors capable of detecting finger flexion for hand motion monitoring (see Figure 34) (Nishiyama & Watanabe, 2009). However, the sensors used in the glove cannot discriminate between different joint flexion, and different sensors are needed to detect each particular joint movement. Another study reports the use of sensing clothes with embedded hetero-core fibers to measure a joint angle ranging from 0-90º during human walking (Nishijima, Sasaki, & Watanabe, 2007). Figure 34: (a) Photo of the sensing glove and (b) experimental data acquired during hand waving. (Adapted from (Nishiyama & Watanabe, 2009)) 3.1.2. Wavelength-modulated Sensors Fiber Bragg Gratings (FBG) are truly wavelength-modulated sensors. The parameter being measured is a direct function of the wavelength shift associated with the Bragg resonance condition (Krohn, 2000). Bragg grating sensors have several advantages: high sensitivity and accuracy, and because they are small, they are excellent point sensors, which means a single fiber may have several sensors written in sequence, each at a different wavelength (multiplexing capabilities), measuring different physical variables (multiparameter sensor). Bragg gratings are intrinsic elements of a fiber where the Chapter 3 Fiber Optic Sensors 74 index of refraction in the fiber core is periodically modulated after illuminating it with ultraviolet light. As light propagates through the modulated region, some with a specific wavelength (resonant Bragg grating wavelength, λ) will be reflected. The resonant Bragg grating wavelength depends on the refraction index, n, and the spacing between grating periods, Λ, as follows (Krohn, 2000): (19) At wavelengths that do not satisfy the Bragg condition, the light passes through without being affected; however, at the Bragg wavelength, the signal is reflected. An example of a reflection spectrum of a standard FBG sensor is shown in Figure 35. Figure 35: Reflected spectrum of a standard FBG sensor. The Bragg grating responds to both strain and temperature. Strain effects the elongation of the fiber, changing the grating spacing, and thus affecting the transmitted wavelength. Also, the index of refraction is affected due to Poisson’s effect (photoelasticity). Temperature causes thermal expansion, which also changes the grating spacing. The refractive index itself is temperature dependent. Usually, to discriminate temperature from strain effects, a reference grating is used in the sensing head. Wearable Sensor System for Human Motion Analysis 75 Another grating approach uses long-period gratings (LPG): when the grating spacing is in the 500 µm range (typically ten times larger than FBGs) the resonant condition couples light from the core to the cladding (Krohn, 2000). LPGs have specific properties that make them possible to differentiate between strain and temperature, twist and bending (Raman, 2010). As both FBGs and LPGs can directly sense variations in temperature and strain and, indirectly, a variety of physical properties such as pressure, rotation and curvature, they can be useful for many applications in the field of healthcare and, more specifically, human motion analysis. An embedded array of FBGs can be used for pressure mapping at different human segments and joints. A temperature independent grating with the proper configuration can be used as pressure sensor for biomechanics or rehabilitation. The EU FP7 IASiS project addresses pressure ulcer incidence and treatment and aims to develop and demonstrate an Intelligent Adaptable Surface for serving as the skin/machine interface in therapy beds and wheelchair seating systems (Pleros, Kanellos, & Papaioannou, 2009). The system integrates FBG sensor arrays in a 2D mesh structure, providing an effective way of monitoring pressure and strain along the entire surface. Another similar system was built that monitors the patient continuously, registering the amount and frequency of movement over a given time, generating a report that alerts the nursing staff if the patient stays unmoved for a long period of time (Hao et al., 2010). Chest strain can be used for monitoring patients, as breathing is difficult to monitor. A temperature independent FBG can be designed for monitoring ventilator movements and to determine the respiratory frequency spectrum. In (Wehrle, Nohama, Kalinowski, Torres, & Valente, 2001), a FBG was attached to an elastic belt, which was held in position just above the breast. As the patient breathes, the thorax cage distends and deflates rhythmically. In addition, the frequency of the signal can be used to trigger corrective action should the patient be under stress. Figure 36 shows two respiratory signals recorded with this system: the first reveals a normal inhaling and exhaling movement of a 26 Chapter 3 Fiber Optic Sensors 76 year old man, in sitting position; the second represents the respiratory movement when the subject is turning his arms. Figure 36: Respiratory signal captured through a FBG sensor of (a) a normal subject in sitting position and (b) a subject turning his arms. (Adapted from (Wehrle et al., 2001)) Using the same concept and taking advantage of the FBG linear sensitivity to longitudinal mechanical stress, a system comprising a FBG sensor was developed to investigate the capabilities of optical sensors for healthcare monitoring in magnetic resonance imaging (MRI) environments (De Jonckheere et al., 2010; De Jonckheere et al., 2009). Due to the well-known immunity of fiber optics against electromagnetic radiations, FBG sensors are suitable for MRI environments. The sensors were placed at abdominal and thoracic levels to continuously monitor ventilation motion. Detecting these movements can be useful to monitor anaesthetized patients during MRI scans, especially for infants who are susceptible to the Sudden Infant Death Syndrome (De Jonckheere et al., 2010). The developed system was able to precisely detect respiratory movements; however, due to the variability of age, size and weight of the patients, this wearable solution needs to be adapted to different groups. There are many other different examples of using FBG and LPG sensors to monitor respiration (Allsop et al., 2007; Allsop et al., 2005; Xiaobin, Chunxi, Kun Mean, Hao, & Xunming, 2008). All these systems are based in the same sensing principle described above and differ in the specific purpose for which they were built. Wearable Sensor System for Human Motion Analysis 77 For a proof-of-concept demonstration, an FBG sensor was placed in a vibrating membrane of a subwoofer. Using recordings of various heartbeat sounds, the vibrations felt by the membrane would induce stretching and/or contraction of the FBG. Some relevant features such as strength of the heartbeat or heart rate can be extracted from the wavelength signal. The researchers predicted that in real-life scenario a FBG sensor could be used as a simple stethoscope (Gurkan, Starodubov, & Xiaojing, 2005). Another example of how to use an FBG sensor to measure cardiac activity was proposed by Witt et al. (2011). The sensor was placed at the wrist of the patient and the small elongations produced by heartbeat were detected by means of wavelength shifts. FBG sensors can also be used in biomechanics to measure human joint angles. The system proposed in (da Silva, Goncalves, Mendes, & Correia, 2011) is a simple sensing glove which uses FBGs to measure the angle between the finger phalanges. The method consists in measuring the elongation of the upper side of the finger joints when stretching out or in the finger. Since the purpose is to detect the flexion/extension of the finger, sensor positioning and disposition is crucial. The sensors must be placed in order to maximize elongation and minimize wrinkles that could lead to fiber breakage. In this system, the fiber was positioned in a curvilinear layout and a sensor was placed over each phalanx joint, performing a total of 14 FBGs sensors in the glove (see Figure 37). Due to FBGs multiplexing capabilities and high sensitivity, a single fiber is enough to accurately measure the angles. The sensing glove revealed a linear response while the hand was opening and closing, and a good agreement with real angles was achieved. However, the integration of the fiber into the textiles revealed some problems and the systems lacks portability. Chapter 3 Fiber Optic Sensors 78 Figure 37: FBG sensing glove. (a) FGB sensor positioning; (b) Real-time monitoring of hand posture. (Adapted from (da Silva et al., 2011)) 3.1.3. Phase-modulated Sensors The use of interferometers in optical measurement has been well established for many decades. Because of their extreme sensitivity, phasemodulated sensors are the most publicized of all fiber optic sensors (Krohn, 2000). Typically, these sensors use a coherent laser lightsource and two singlemode fibers. The light is split and injected in each fiber. Therefore, if the environment perturbs one fiber relative to the other, a phase shift occurs that can be detected very precisely (Krohn, 2000). This phase shift is detected by an interferometer. As mentioned before, there are four interferometric configurations: the Mach-Zehnder, the Michelson, the Fabry-Perot and the Sagnac. The Mach-Zehnder interferometer configuration can be seen in Figure 38 (a). It uses a laser beam that is split using a 3 dB coupler, which means 50% of the light is injected into the sensing fiber and 50% into the reference fiber. The light beams are recombined using another 3 dB coupler and the phase shift is measured. This shift results from changes in the length and the refractive index of the sensing fiber (Krohn, 2000). Wearable Sensor System for Human Motion Analysis 79 Lightsource Detector 3 dB Lightsource Detector 3 dB Sensing Fiber Reference Fiber Transducer Mirrored Fiber End Mirrored Fiber End Lightsource Detector Signal Processor Transducer Reference Fiber Sensing Fiber 3 dB 3 dB Lightsource Detector Mirror Mirror Transducer (a) (b) (c) (d) Figure 38: Fiber optic interferometers: (a) Mach-Zehnder configuration; (b) Michelson interferometer; (c) Fabry-Perot scheme and (d) Sagnac interferometer. (Adapted from (Krohn, 2000)) The Michelson interferometer approach is depicted in Figure 38 (b). It employs a configuration very similar to the Mach-Zehnder interferometer, but it uses back reflection due to the fibers having mirrors at its ends. The initial beam is split and injected in the reference and sensing fibers. Since both fibers have end mirrors, the light is reflected and re-coupled to a shift detector. The sensitivity of such configuration is higher but it has the disadvantage of feeding light both into the detector and the laser, which causes noise (Krohn, 2000). Chapter 3 Fiber Optic Sensors 86 consequently, increases sensor lifetime. Four different configurations were tested in order to investigate the sensor behaviour in terms of sensitivity and repeatability with increasing number of turns. The schemes studied covered sensors with a single, two, three and four loops (Figure 42). Figure 42: Schematic of sensor configurations studied: (a) single loop, (b) two loops, (c) three loops and (d) four loops. The first scheme studied, shown in Figure 42 (a), has only one curvature and the light input and output are on the same side, while in the second case (Figure 42 (b)) the sensor has two curvatures and the light input and output are opposite to each other. Figure 42 (c) represents a situation similar to the single loop configuration and Figure 42 (d) is analogous to the two loops configurations; in both cases the number of loops increases relatively to the previous case. Figure 43 (a) presents the experimental setup used to measure flexion angle. Light from a broadband source centered at 1550 nm and with width 100 µm was guided into the fiber. A photodetector (Agilent Technologies 8163B Lightwave Multimeter) in the telecommunications window of 1550 nm was Wearable Sensor System for Human Motion Analysis 87 used to measure the optical power at a sample frequency of approximately 10 Hz. A Matlab script was used to acquire and visualize the output signal in real-time (the complete code can be found in Appendix C. For each sensor configuration, ten repetitions were performed at a fixed angle. The flexion angle ranged from 0º, equivalent to rest position (Figure 43 (b)), to 60º, which corresponded to the maximum elbow flexion (Figure 43 (c)), with increments of 3 degrees. As mentioned before, the range chosen for the flexion angle was based on the typical range of motion observed while performing the reach task, which is one of the most important upper-limb functional tasks that are in the basis of most daily-life routines. During the experiments, no degradation in-use of the fiber was observed. Figure 43: (a) Experimental setup; (b) sensor at rest position (0º) and (c) sensor at maximum elbow flexion (60º). Results and Discussion The results of the measurements for each sensor configuration are shown in Figure 44. Each graph shows the output light power for all ten measurements (small circles) for a given angle, the corresponding average (large circles) and standard deviation, as well as a linear fit (shown in red) between the light power and the flexion angle. Chapter 3 Fiber Optic Sensors 88 Figure 44: Light output power variation with increasing elbow flexion angle for (a) single loop, (b) two loops, (c) three loops and (d) four loops configurations. Overall, all configurations show losses in optical power as the angle increases, as expected. Concerning sensor response, the way the optical fiber behaved inside the channels on the fabric as the flexion angle increased changed between configurations and, consequently, influenced the sensitivity of the sensor device. For the single loop scheme, the sensitivity is dictated by the decrease on the curvature radius of the only loop present and, as one can see on Figure 44 (a) the light losses due to bending do not change uniformly as the flexion angle increases. In fact, it is possible to note that the sensitivity is smaller for smaller angles (until approximately 15º) and likewise the variability Wearable Sensor System for Human Motion Analysis 89 between repetitions increases. One can also observe a lack of resolution on the sensor response; however this limitation is caused by the resolution of the acquisition equipment used for this experiment. For the second case, i.e., two loops configuration, the sensor response is a consequence of a symmetric decrease of both curvature radius. Thus, the sensitivity of this configuration is approximately the double of the single loop scheme. It should be noted the high variability between samples for this sensor (see Figure 44 (b)). Concerning the three loops scheme, shown on Figure 44 (c), one should assume a higher sensitivity when compared with the two loops configuration. In fact, there is a slight increase of the sensitivity, but the difference is not as high as it should, theoretically, be expected. This behaviour can be explained by the way the curvature radius decreased as the flexion angle increased: the inner loop (referred as number 2 in Figure 42 (c)) has a different behavior from the outer loops (1 and 3), which translates into a behaviour similar to the one seen on the two loops configuration. Therefore, the sensitivity of this scheme is very close to the two loops sensor. However, the dispersion between the ten repetitions for this configuration is smaller than the previous case. Finally, during the experiments with the four loops scheme (Figure 44 (d)) it was possible to see that the decrease on the curvature radius of the outer loops (referred as numbers 1 and 4 of Figure 42 (d)) and the inner loops (2 and 4) was different from each other, which reflected in an increase of the sensitivity of approximately the triple of the sensor with a single loop. Additionally, this configuration provided the best approximation to a linear response. A comparison between the sensitivity of all configurations is shown in Figure 45. Table 8 presents the values of sensitivity, correspondent standard deviation and R-squared of the linear fit for all configurations. Chapter 3 Fiber Optic Sensors 90 Figure 45: Comparison of sensor response for the four configurations studied. Table 8: Sensitivity, standard deviation and R-squared values for each sensor configuration. Sensor Scheme Sensitivity (dBm/degree) Standard Deviation R-squared 1 Loop 0.197 0.008 0.963 2 Loops 0.42 0.01 0.985 3 Loops 0.46 0.01 0.982 4 Loops 0.64 0.01 0.989 Looking at these values it is clear that the configuration that assures a higher sensitivity, i.e. 0.64 dBm/degree, is the one with four loops, as expected. In fact, this sensor has a sensitivity 3 times higher than the single loop configuration and almost 1.5 times higher than the two and three loops schemes. Additionally, this configuration provided the best approximation to a linear response. Wearable Sensor System for Human Motion Analysis 91 3.3. Summary Fiber optic sensors are a relatively new concept for the measurement of biomechanical variables. Besides the advantages of small size and weight, electromagnetism and radio-frequency immunity, these sensors have high sensitivity and accuracy which makes them a good alternative for conventional electrical sensors. In the previous section a simple intensity-modulated fiber optic sensor was designed and characterized to determine elbow flexion. This movement is the first phase of the reach task and one of the most important features physiotherapist evaluate in order to score stroke survivors performance. A customized piece of garment was specifically designed to integrate the optical fiber sensor making it truly wearable. Different sensor configurations were studied and light loss resulting from bending was measured with an elbow angle ranging from 0º to 60º. Results showed that, when compared to other sensor configurations, the four loops scheme had a good sensor response in what concerns linearity, sensitivity and repeatability. The sensitivity for this configuration (0.64 dBm/degree) was 3 times higher than the single loop scheme and approximately 1.5 times higher than the three loops configuration. This wearable sensor has demonstrated a good response to elbow flexion and, when compared with traditional sensors, such as electrogoniometers, it can overcome some hurdles such as sensor alignment and ergonomy, and can also provide automatic means for monitoring human movement without the dependency on therapists or other end-users. Since the proposed system revealed good performance for the determination of elbow flexion, it is simple to handle and comfortable for the user, it has a great potential for rehabilitation, especially to monitor upper-limb intervention progress of stroke patients, identify relevant compensatory movement strategies and help physicians with their clinical reasoning process. For swimming analysis, the proposed sensor represents a breakthrough for determining human segments angles in real-time, although some efforts need to Chapter 3 Fiber Optic Sensors 92 be done in order to insulate the electrical part of the system (light emitter, photodetector, etc.). PART II HUMAN MOTION ANALYSIS SPORTS AND REHABILITATION Wearable Sensor System for Human Motion Analysis 95 Chapter 4 SWIMMING PERFORMANCE ANALYSIS Swimming is a technically challenging sport. The forward displacement during swimming results from the ratio between propulsive and resistive forces. From the biomechanical point of view, swimming performance depends on the mechanical interaction between water and the dynamical actions of a swimmer’s body. Swimmer’s propulsion results from angular movements of the superior and inferior limbs and their synchronization. The mechanics behind propulsion were first considered as an application of Newton’s Third Law: the hand and the forearm push dense masses of water backwards in a straight line to propel the swimmer’s body (Lauer, Figueiredo, Vilas-Boas, Fernandes, & Rouard, 2013). In fact, to generate high propulsive forces, a swimmer must perform a complex cyclic motion. However, swimmer’s body and movements do not generate exclusively propulsive forces. In truth, it is well-known that body orientation and volume contribute significantly for motion resistance. Whereas some factors are more difficult to be changed in order to reduce resistance forces, since they are intrinsic to body characteristics or swimming style, other factors such as body orientation in the water or angular adjustments of specific segments can be easily improved to reduce some of the resistance forces and the production of waves. Several studies report that the angle formed between the swimmer’s body and the horizontal plays and important role in swimmers’ forward displacement: if the legs are deep into the water than the drag coefficient increases (Bächlin, Förster, & Tröster, 2009). Also, the body and head rotations contribute to increase resistive forces. It can be said that the Chapter 4 Swimming Performance Analysis 102 transmitted to a remote station for visualisation. Nevertheless, wireless transmission is not as trivial as it seems at the beginning. The water-air interface represents a barrier which is difficult to overcome using standard wireless protocols, such as ZigBee or Bluetooth. In fact, studies have demonstrated that significant data can be lost if air transmission path overcomes 1 meter distance (Hagem, Thiel, O'Keefe, Wixted, & Fickenscher, 2011; Daniel A. James, Galehar, & Thiel, 2010). As an alternative, transmission antennas operating in different frequency bands can be used, but its size and limited data rate are prohibitive (Abbosh, James, & Thiel, 2010). Different strategies for providing feedback to swimmers can be adopted, as the one described by Hagema et al. (2013) which consists of a wrist-mounted accelerometer which calculates time difference between strokes and activates a LED which makes swimmer react according to light colour. 4.2. Swimming Analysis through WIMU In swimming analysis some of the most important parameters to be monitored can be divided in two groups:  Performance parameters:  Time and distance;  Average velocity;  Number of laps;  Number of strokes, stroke length and frequency.  Kinematic parameters:  Angular positions of body segments;  Linear velocity and acceleration profiles;  Angular velocities. The above mentioned parameters can be obtained through inertial sensors, either directly from sensors outputs or indirectly by applying the principles described in Chapter 2. The estimation of such parameters can be used to characterize swimmer’s movements and allow for the comparison of Wearable Sensor System for Human Motion Analysis 103 different experience level swimmers and to differentiate between swimming styles. When analyzing swimmer’s performance through accelerometers, some of the equations presented in Chapter 2 for angles estimation can be simplified. This is because of the nature of swimming movements. Typically, the inertial units are placed at the upper back or at the lower back, which is expected to better correspond to the center of mass of the swimmer. The upper back location is often preferable since it is more accurate when detecting strokes, turns and styles (Siirtola et al., 2011). Movements at this region are very slow and body (dynamic) acceleration is small when compared with gravity (static acceleration). Thus, swimmer’s movements can be considered as nonaccelerated movements and a quasi-static condition can be adopted. The application of this assumption has revealed very good results when compared with video-based systems (Bächlin & Tröster, 2011; Daukantas et al., 2011; Pansiot et al., 2010). In this case the extraction of some angular positions can be done using only the information given by the accelerometer. The acceleration measured by the accelerometer, ab, depends on the actual acceleration of the system, an, the orientation (given by the Direction Cosine Matrix, Equation 8 of Chapter 2) and gravity, g: (21) Considering that , Equation 21 can be simplified to estimate: (22) Therefore, the pitch and roll angles can be derived from the measured acceleration as follows: Chapter 4 Swimming Performance Analysis 104 (23) (24) The estimation of such angles can give information about alignment of the body. Vertical alignment, also referred to as body balance, is associated with the angle between the swimmer’s body and the water surface (pitch angle); Horizontal alignment, or body rotation, refers to the swimmer’s body rotation along his own longitudinal body axis (roll angle) (see Figure 49). Figure 49: Body coordinate system. Body balance and body rotation can be estimated from the pitch and roll angles, respectively. A good body balance is important for efficient swimming as drag forces are reduced. There are several possible reasons for a bad body balance, for example weak leg kicks, a weak body tension or a bad posture of the head due to looking upwards instead of aligning spine and head. Body rotation is efficient in crawl and backstroke swimming because the stroke length can be increased and the side-lying gliding position is the body position with the least water resistance. The body rotation can be initiated by the leg kicks. The upper and the lower body part should rotate together, because a synchronized rotation leads to a fluent motion (Bächlin, Förster, & Tröster, 2009). The roll angle information is also useful for breathing studies and to determine the number of strokes (Siirtola et al., 2011). Wearable Sensor System for Human Motion Analysis 105 The study described in this section presents the main outcomes accomplished with the inertial measurement unit referred to as WIMU in the first part of this thesis, to measure the performance of a swimming athlete. Experimental Procedure During the experimental tests the WIMU was located at the dorsal zone of the coronal plane of the swimmer, within the vertebral region at the inferior scapular section, as can be seen on Figure 50. This location was chosen in order to measure, in addition to the accelerations on the three axes, the longitudinal rotation of the trunk and body balance. As mentioned before, these parameters can be considered a useful factor when evaluating swimming performance, due to its correlation to the displacement velocity. Another important consideration was the comfort of the swimmer and avoiding restricting or modifying their average sequence of movements. Figure 50: WIMU positioned at the upper back of the athlete. A female athlete in her late teens served as the main tester. Before entering the pool, the swimmer was required to perform a number of flexibility related routines, in order to determine movement constraints. Once in the pool, she was asked to swim, submerge and perform various movements in order to determine if the WIMU’s presence represented an obstruction to her movements. In both cases (outside and inside the pool) the swimmer reported that the unit did not affect her movements. Finally, the swimmer was told to Chapter 4 Swimming Performance Analysis 106 complete several sets of laps using the crawl technique, then a number of laps with, butterfly and finally breaststroke. For all styles indicated, the turn at the end of the pool was perform through a vertical turnaround (i.e., stop-touch wall turnaround), reversing direction without flipping under water. Although data gathering for the crawl techniques laps did not required signal compensation strategies, the butterfly and breaststroke techniques did present some data gaps. The recorded data was later processed applying compensation techniques (interpolation). Results and Discussion The acceleration signals (in units of g’s) for two laps crawl technique are shown in Figure 51. Considering these signals, the X-axis points opposite to the direction of displacement and together with the Y-axis form the coronal plane of the subject; while the Z-axis is pointing inwards to the subject, forming the sagittal plane with the X-axis and the transversal plane with the Y-axis. Figure 51: Acceleration in all three axes (X-axis: green; Y-axis: red; Z-axis: blue) for two laps crawl technique. As mentioned before, performance parameters such as the number of strokes, number of laps or start/end of lap can be directly taken from the accelerometry profiles. As expected for crawl technique, higher acceleration amplitudes occur in the Y-axis, which relates to rotation movements around the Wearable Sensor System for Human Motion Analysis 107 X-axis, corresponding to arms strokes. A direct analysis also allows to quantify the time between each arm stroke and its evolution along each lap, as shown in Figure 52. Figure 52: Elapsed time between each stroke for two laps. The 10th stroke corresponds to lap end, i.e. vertical turnaround. Table 9 summarizes some relevant performance features extracted from the accelerometry profile. Table 9: Some relevant performance parameters extracted from accelerometry data for crawl technique. Lap 1 Lap 2 Duration (s) 24.16 23.84 Number of strokes 9 10 Stroke frequency (Hz) 2.68 2.38 Average stroke duration (s) 2.41±0.33 2.38±0.24 Equivalent lap sections acceleration signals (in units of g’s) for different swimming techniques are presented in Figure 53. Chapter 4 Swimming Performance Analysis 108 Figure 53: Equivalent lap section acceleration signals for each performed technique. Crawl, butterfly and breaststroke styles were evaluated for each movement axis. Observing the graphs concerning each swimming technique, it was possible to differentiate between styles. For example, crawl technique can be readily distinguish from breaststroke and butterfly by the signal provided by Wearable Sensor System for Human Motion Analysis 109 the accelerometer in Y-axis. This signal presents large variations for the crawl technique, while for the other styles these variations are comparatively small. Alternatively, the butterfly and breaststroke technique can be differentiated from each other from the data produced by the accelerometers in the X-axis, or the longitudinal axis. As mentioned in the beginning of this section, pitch and roll angles can be estimated from acceleration signals. These values provide a better understanding about the swimmer’s movements to progress in the water and to minimize drag. The pitch angle, which is related with the angle formed with the horizontal, for one lap crawl, butterfly and breaststroke techniques is presented in Figure 54. Figure 54: Pitch angle for one lap crawl, butterfly and breaststroke techniques. As expected, variations in the pitch angle for the butterfly technique are higher when compared with the other styles. In addition, it is clear to observe double-bell shapes of this signal, each corresponding to a single stroke. This behaviour is strongly related with this swimming technique pattern, which is characterized by two leg movements (the first when entering the water and the second at water exit) in each cycle or stroke (Soares, 2000). Chapter 4 Swimming Performance Analysis 110 The roll angle for all the evaluated techniques, i.e. crawl, butterfly and breaststroke, is shown in Figure 55. As one should deduce, the crawl technique exhibits roll angles with a higher amplitude, due to the swimmers’ rotation along the longitudinal axis. It is expected that both butterfly and breaststroke techniques have small body rotations. Figure 55: Roll angle for one lap crawl, butterfly and breaststroke techniques. 4.3. Summary The leading method for swimming analysis is based on image processing of video data. Nonetheless, there are some hurdles to overcome due to timeconsuming setup and data processing procedures and also due to water interference at the air-water interface (waves and bubbles, for instance). The use of wearable technology, such as inertial measurement units, allow the extraction of meaningful features that can be transmitted to the swimmer or coach, providing real-time feedback of swimmer’s performance. In addition, the information from these systems can be combined with other systems in order to overcome some of the obstacles mentioned before. In this chapter, results from accelerometry data acquired with the WIMU device are presented. The WIMU was placed at swimmers’ backs and Wearable Sensor System for Human Motion Analysis 111 movements were acquired for different swimming techniques. It was possible to obtain acceleration profiles for each style, allowing the identification of the number of laps swam, lap turnaround, number of strokes, stroke frequency and duration. The pitch and roll angles for each swimming technique were estimated and differences between each technique were identified from these profiles. Chapter 5 Stroke Patients Movement Analysis 118 Figure 56: Reaching sub-phases according to postural control demands. The upper-limb analysis is becoming an increasingly topic of interest among scientists and physiotherapists. Although most of the studies rely on image-based data acquisition and analysis systems (Mendonça, Santos, & LópezMoliner, 2011; Teasell, Foley, Bhogal, & Speechley, 2003; Vandenberghe, Levin, De Schutter, Swinnen, & Jonkers, 2010), several research is starting to be conducted based on wearable sensors, more specifically through inertial measurement units (Paten et al., 2010; Pérez et al., 2010; Zhou et al., 2008). Some studies report classification algorithms of upper-limb performance (Patel et al., 2010; Zhe, Qiang, & Ferry, 2011) while others are focus on estimating qualitative scales scores, such as the FMA score, from accelerometry data (Del Din, Patel, Cobelli, & Bonato, 2011). The analysis of the reaching movement in post-stroke patients and in healthy subjects was performed with the wearable system described previously as W2M2. Several studies were performed in order to establish a systematic protocol for the acquisition of inertial data and more important, to understand the mechanisms underlined in the execution of the reaching movement, both for post-stroke and healthy subjects. Therefore, the next sub-sections describe the experimental procedures and results achieved in these studies. All the participants of the studies were informed of the experimental procedures and provided written consent in accordance with policies of the institution’s Ethics Wearable Sensor System for Human Motion Analysis 119 Committee (see the Experimental Protocol approved by Ethics Committee in Appendix D). 5.2.1. Inertial Measurement Units Positioning The presence of compensatory strategies is often observed in post-stroke subjects when attempting to reach an object. Although some controversy remains regarding the functional benefits of compensatory movements as a way of accomplishing a given task, studies suggest that such maladaptive strategies may limit the plasticity of the nervous system to enhance neuro-motor recovery. The study presented in the current section intends to aid in the development of a system for compensatory movement detection in post-stroke patients through accelerometry data, by analyzing the best sensor positioning for the identification of such strategies. Participants The sample was composed by two post-stroke patients receiving physiotherapy care at a rehabilitation center. Participants had to meet the following inclusion criteria:  Confirmatory neuroimaging results of a single, unilateral stroke in the Medial Cerebral Artery (MCA) territory, sustained at least 3 months prior;  Absence of hemispatial neglect;  Absence of major visual, perceptual or cognitive deficits, confirmed by the mini-mental state examination (MMSE);  Active range of motion in the compromised arm of at least 15 in the shoulder (flexion/extension; abduction/adduction and internal/external rotation) and elbow (flexion/extension) (Sveistrup, 2004). Chapter 5 Stroke Patients Movement Analysis 120 Explicit exclusion criteria included cerebellar or brain stem lesions and pain/sub-luxation in the upper-limb. Arm motor impairment was evaluated prior to measurements, as seen on Table 10, with the arm subsection of the Fugl-Meyer scale and the Reach Performance Scale (close target). This clinical evaluation was performed by a team of three experienced physiotherapists with more than 10 years of clinical practice in neurological field. At the time of the experiment, the post-stroke patients were following the conventional rehabilitation procedures associated with their condition, based on the Bobath Concept principles (Raine, 2009). This is a problem-solving approach for the assessment and treatment of individuals with disturbances of function, movement and postural control due to a lesion of the central nervous system (Raine, 2009) Although sitting balance was not measured directly, all subjects were ambulatory without aids and had no difficulty in maintaining a stable sitting posture during data collection. Table 10: Demographic data and clinical scores of post-stroke subjects. Subjects A B Age 49 47 Gender Male Female Location of lesion LMCA RMCA Months post-stroke 66 20 RPS score 5/18 12/18 FMA (shoulder, elbow, forearm) 4/36 20/36 FMA (wrist) 0/10 2/10 FMA (hand) 2/14 12/14 FMA (coordination) 0/6 3/6 LMCA/RMCA: Left/Right Medial Cerebral Artery RPS: Reach Performance Scale FMA: Fugl-Meyer Motor Assessment Wearable Sensor System for Human Motion Analysis 121 Experimental Procedure Each subject was assessed in the sitting position, with a table placed in front of him/her, at a height corresponding to the alignment of the iliac crests. The proximal table limit was coincident with the distal border of the subject's knees, so as not to interfere with the arm trajectory. The subjects started the task with approximately 0º of flexion/extension/internal rotation at the shoulder; approximately 100º of flexion at the elbow with forearm in pronation and the palm of the hand resting on the thigh. The subjects were instructed to reach a target placed ipsilaterally to the upper-limb in study, in groups of three repetitions (as to avoid variations due to fatigue) separated by one minute rest period. The target's placement reference was the anatomical reaching distance of the hand, using the measured distance from the acromion to the metacarpophalangeal joint of the thumb (Vandenberghe et al., 2010). The individual was instructed, after verbal command, to perform reaching. Performance was video recorded for posterior visual cross-reference. In order to insure sensor placement repeatability, precise bone landmarks were required. After a physiological study of the target area and experimental trial of sensor positioning for assured subject upper-limb mobility and comfort, the following positions were considered (see Figure 57):  P1, placed under the acromion, following the line that connects the lateral epicondyle and the acromion;  P2, placed on the middle point between lateral epicondyle and the acromion;  P3, immediately above lateral epicondyle, in alignment with acromion;  P4, immediately below the lateral epicondyle, after elbow articulation;  P5 in the trunk over the T12 vertebra. Chapter 5 Stroke Patients Movement Analysis 122 Figure 57: Sensor positioning under consideration. Results and Discussion A set of accelerometry data for subjects A and P is presented in Figure 58. The different colors represent a set of three different trials performed. The inherent difference in acceleration amplitudes shown especially in Xaxis between subjects is related to the fact they present opposite compromise limbs (LMCA vs. RMCA). The discussion that follows is based on the multiple data collected from both subjects and their correspondent video records. In relation with sensor position P1, post-stroke patients present on the collected data, elevation and abduction of the shoulder, at the initial phase of the movement, corroborating the visual analysis. Position P2 exhibits an increased displacement in the anterior direction (X-axis) when compared with P1; however there is a lack of marked differences observed on the global pattern of the movement. Such could suggest that P2 offers more movement detection sensitivity when compared to P1. In reference to the Y-axis, the opposite seems to occur, i.e., there is a reduced sensitivity for such detection when compared with P1, in both cases. Wearable Sensor System for Human Motion Analysis 123 Figure 58: Accelerometry data for Subject A and B for locations P1, P2, P3, P4 and P5. Chapter 5 Stroke Patients Movement Analysis 124 Sensor position P3 shows some variability among the patients. The movement in the anterior direction (X-axis), performed by subject A is more pronounced when compared with P1; in turn, for subject B this movement is better detected when compared to both P1 and P2. A similar situation occurs in the remaining movements, i.e. superior direction (Y-axis) and lateral direction (Z-axis). Subject B presents no pronounced differences among the sensor position P1, P2 and P3 for the lateral direction. This could be explained by lack of evident movement component recruitment as compensation during the functional task. Given the localization of position P4, there exists a need for redefining the detected movement components by each of the axis. Thus, the movement in the anterior-posterior direction is now captured by the Y-axis, and the superiorinferior direction by the X-axis, remaining the Z-axis capturing the lateral movements. Subject A, does not present a significant elevation component (Xaxis), which could be related with the deficit to enlist selective flexion of the elbow. Subject B presents an increase elevation component, resulting from an improved shoulder-elbow interjoint coordination, being able to perform selective flexion of the elbow as an integrating part of the movement pattern. Some evidences exist thus, that sensor position P1 presents increased commitment between movement detection in the superior direction (identification of shoulder elevation as compensation) and an inter-patient variability; however a larger number of measurements and varied sample size would be required for such validation. Finally, as for sensor position P5, one verifies that such position offers increased reproducibility among trials, while presenting reduced acceleration variations (less than 0.1 g in most cases), translating into a reduced movement of the trunk, especially in the superior-inferior direction (Y-axis). Some anterior-posterior movement (Z-axis) and rotation (X-axis) are present, which behave as compensations, given the reduced capacity of enlisting shoulder flexion with elbow extension (extensor synergy); implying a displacement of the trunk as attempting to reach the target. Subject B presents increased anterior- Wearable Sensor System for Human Motion Analysis 125 posterior displacement of the trunk when compared to subject A. The presence of a larger compensation at this level, in a clinically less affected individual, could be related to its fully completion of the functional task. Data analysis seems to suggest that the P1 position is advantageous for compensatory movement detection at the shoulder level, being however necessary to complement with information provided by P5, in order to discriminate between shoulder or trunk elevation. The information provided by sensor locations P2 and P3 do not seem to add relevant knowledge to that provided by sensor position P1. The P4 position seems the most appropriate for detecting the abduction component of the limb; however, in relation with the superior-inferior movement, this particular sensor position is insufficient for determination of the corporal segment where the elevation occurs (shoulder/elbow/trunk), limiting its reliability for compensatory movement identification in this direction. Finally, sensor position P5 presents a good sensitivity for anterior-posterior movement and rotation detection. Table 11 summarizes the sensitivity of each position for the detection of compensatory movements: anterior-posterior (A-P), superior-inferior (S-I) and medial-lateral (M-L). A growing sensitivity scale ranging from 1 to 3 was used for the characterization by a team of physiotherapists. Table 11: Sensitivity descriptive analysis of movement components for sensor locations. Subject A Subject B A-P S-I M-L A-P S-I M-L P1 1 3 1 1 3 2 P2 2 1 1 2 2 2 P3 2 2 2 2 2 2 P4 2 2 3 2 2 3 P5 3 3 3 3 3 3 A-P: Anterior-Posterior; S-I: Superior-Inferior; M-L: Medial-Lateral Chapter 5 Stroke Patients Movement Analysis 126 5.2.2. Compensatory Movements Detection in Post-Stroke Subjects Following the initial results described above, an in-depth analysis of upper-limb movement of post-stroke patients during reaching was performed, in order to identify compensatory strategies and to extract quantitative parameters that allow describing the behaviour of such movements. Participants The sample was composed by four post-stroke patients receiving physiotherapy care at a rehabilitation center. Participants had to meet the same inclusion criteria as the ones described in the previous study. In addition, exclusion criteria included cerebellar or brain stem lesions and pain/subluxation in the upper-limb. Arm motor impairment was evaluated prior to measurements, as seen on Table 12, with the arm subsection of the Fugl-Meyer scale and the Reach Performance Scale (close target). Experimental Procedure The experimental procedure followed the protocol described in the previous study: each subject was assessed in the sitting position, with a table placed in front of them; the table limit was coincident with the distal border of the subject's thigh, so as not to interfere with the arm trajectory. The individual was instructed, after verbal command, to perform the functional task. Wearable Sensor System for Human Motion Analysis 127 Table 12: Demographic data and clinical scores of post-stroke patients. Subjects A B C D Age 64 47 53 49 Gender Male Female Female Male Location of lesion LMCA RMCA LMCA LMCA Months post-stroke 19 20 34 66 RPS score 7/18 12/18 8/18 5/18 FMA (shoulder, elbow, forearm) 8/36 20/36 17/36 4/36 FMA (wrist) 1/10 2/10 2/10 0/10 FMA (hand) 6/14 12/14 10/14 2/14 FMA (coordination) 1/6 3/6 4/6 0/6 LMCA/RMCA: Left/Right Medial Cerebral Artery RPS: Reach Performance Scale FMA: Fugl-Meyer Motor Assessment Following the results described in the previous study, the positions chosen for the present study were as follows (Figure 59):  P1, placed under the acromion, following the line that connects the lateral epicondyle and the acromion;  P2, immediately below the lateral epicondyle, after elbow articulation;  P3, in the trunk over the T12 vertebra.