Myoelectric Prosthesis - Modelization and Control of a Bionic Arm

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Myoelectric Prosthesis - Modelization and Control of a Bionic Arm Rafael de Castro Aguiar Mestrado Integrado em Engenharia Eletrotécnica e de Computadores Supervisor: Prof. Fernando Lobo Pereira April 15, 2015 c Rafael Aguiar, 2015 Abstract This project concerns the analysis, modeling and control of a prosthetic arm, by means of Electromyographic signals. The social need to fight adversity continuously allows dedicated and innovative studies that aim to achieve new and improved solutions to fight disabilities. The connection between the Biomedical, Mechanical and Electrical Engineering fields becomes more evident, with countless researchers presenting their new ideas and projects in the various fields of biomechatronics. The project begins with an initial study on the state-of-the-art technologies concerning myoelectric acquisition and processing, and respective control of prosthetic devices. Associated with this study, comes the need to analyze how a real human arm functions, the respective ranges of motion, degrees of freedom, relative weight and segment lengths and normal behaviors in specific case studies. This leads to the design of a somewhat anthropomorphic model in a simulation environment, enabling the replication of the natural limb, by resorting to the Denavit Hartenberg convention for kinematic and dynamic studies, close to those used for normal robotic manipulators. Another critical feature of this project is to provide a human like behaviour to the simulated limb. To do so, the need to study different control design systems is required. Notions of Hybrid Systems, Nonlinear Control and Model Predictive Control (MPC) are introduced, tools that can in fact provide the desired classes of motion. The selected control architecture is based on MPC, that has the capacity to not only adapt its inputs but its references itself. The designed system consists in a multilayered control architecture, taking into consideration the human and mechanical constraints, exhibiting 3 separate layers: Motion Planning, Motion Coordination and Low Level Motion Execution Control. The Motion Planning layer receives the processed EMG signal and generates a concatenated set of sub references of the intended motion to be executed. The Motion Coordination layer will include the devised specific MPC adapted controller and adjust the references generated by the higher level. Lastly, the Low Level Motion Execution Control provides all the subsystems with the pre-processed references provided by the MPC. With this architecture, the system provides an extremely "human-like" motion by allowing itself several re planning and adjusting stages. Several different approaches and redesigns of the control architecture and arm model are performed. A simulation environment is designed to test a 3 degree of freedom arm model (transhumeral type prosthetic), which receives a motion command from a simple control system. The results are compared to a specific case study of a natural, healthy limb with the same movement pattern, yielding a positive response to external disturbances. Due to the limited scope of this project, not all subsystems of the Control Architecture were able to be tested, remaining on a design stage, awaiting further developments. i ii Agradecimentos Antes de tudo mais, ao Professor Fernando Lobo Pereira, por ter ouvido uma pequena ideia e ter ajudado tão fortemente a desenvolver e orientar um fantástico projeto que apenas em sonho se encontrava. E neste longo percurso, várias foram as fontes de inspiração e ombros amigos que me suportaram, ouviram e choraram/riram a meu lado. Nunca esperei ser tão bem recebido no Laboratório I-109 (Robis) no meio de investigadores fantásticos, de técnicos experientes e de ter saído com inúmeros e incalculáveis amigos e mentores. Vários são os nomes que gostaria de mencionar ao longo destes meses, porém acho merecerem destaque alguns dos que mais me "aturaram". A ti Daniel, pela tua forte amizade e por me introduzires e encaminhares neste pequeno mundo pós curso. A ti Jorge e a si Sr. Fernando, por me guiarem e instruírem num mundo técnico para mim novo, pelas histórias fantásticas e sessões de "treino" que me facultaram. A ti Héber e a ti Filipe, por olharem para mim como um miúdo que precisava de ajuda e logo prontificarem os vossos conselhos, fornecendo também aquela motivação extra no dia-a-dia com a constante pergunta "Já está escrita?". A ti Diana, por toda a tua ajuda, por todo o teu conhecimento e pela tua amizade fantástica. E a ti minha nova irmã Graça, pelo sorriso que me conseguias dar todos os dias. Aos meus pais e irmãos, maiores fontes de inspiração da minha vida e alicerces de todo o meu sucesso. And last, but not least, to you Cécile. Your words and love guide me through the greyest days and give me that needed ember to carry on and achieve whatever dream I have. Rafael de Castro Aguiar iii iv "One man’s ’magic’ is another man’s engineering” Robert A. Heinlein v xii LIST OF TABLES Abbreviations and Acronyms ANN Artificial Neural Network AR Autoregressive CMRR Common Mode of Rejection Ratio CNS Central Nervous System DH Denavit-Hartenberg DoF Degrees of Freedom EEG Electroencephalography EMG Electromyography FSM Finite State Machine IMES Implanted Myoelectric Sensors LMS Least Mean Square MES Myoelectric Signal MLP Multilayer Perceptron Neural Network MPC Model Predictive Control NMPC Nonlinear Model Predictive Control PID Proportional, Integrative and Derivative RoM Range of Motion xiii xiv Abbreviations and Acronyms Glossary Angle of fiber pinnation The angle between the line of orientation of muscle fibres and the line of muscle action (i.e. the line of pull of the tendon) Anthropomorphic Attribution of human form or other characteristics to anything other than a human being. In this case, to the prosthetic arm model Bionic Application of biological methods and systems found in nature to the study and design of engineering systems and modern technology Circumduction Combination of abduction, adduction, extension, and flexion, allowing for a circular motion of a limb Electroencephalography (EEG) Recording of electrical activity along the scalp. EEG measures voltage fluctuations resulting from ionic current flows within the neurons of the brain Electromyography (EMG) Electrodiagnostic medicine technique for evaluating and recording the electrical activity produced by skeletal muscles. EMG is performed using an instrument called an electromyograph, to produce a record called an electromyogram Intrafascicular Whitin a bundle of skeletal muscle fibers surrounded by connective tissue (fascicles) Physiological Refers to the normal, healthy operation of your body and its organs Pronation Rotation of the hand or forearm so that the surface of the palm is facing downward or toward the back Prosthetic device Artificial device that replaces a missing body part, which may be lost through trauma, disease, or congenital conditions Subcutaneous Situated or applied under the skin Supination Rotation of the hand or forearm so that the palmar surface is facing upward xv Chapter 1 Introduction 1.1 Goals and Scope This dissertation concerns the challenges underlying modeling and control of prosthetic arms. Besides the obvious and extremely interesting challenge this project presents, it is also a theme that regards the improvement on the quality of life of patients with this type of physical disabilities. This effort is designed to fulfill the requirements of the Master degree to be awarded by the Faculty of Engineering. Moreover, it will serve as basis for future investigation by candidates in this area. The effort in this dissertation aims at providing a better understanding on the concepts behind the topics previously referred (Biomechatronics), having in mind the investigation of control systems for state-of-the-art prosthetic arms. Moreover, it examines the challenges involved in the design of advanced control systems with a hierarchic structure and with a Model Predictive Control (MPC) scheme, which, by combining adaptive and robustness properties, enables prosthetic arms - with a structure much simpler than a real arm - achieve functionalities and performances very close to those of a natural arm. Thus, a key challenge concerns the careful analysis of natural arms behaviors required to extract motion patterns and, from these, motion requirements for the prosthetic arm. These will be key ingredients to specify a control architecture to endow the prosthetic arm with the appropriate behavioral capabilities. Another key challenge concerns the proper decodification of the myoelecric signals able to discriminate the subtle range of commands required to exploit the wealth of sophisticated behaviors to be performed by prosthetic arm. Finally, another important challenge consists in development an MPC based control architecture which is, on the one hand, endowed with the required properties - position and velocity control accuracy, robustness and adaptivity - and, on the other hand, able to satisfy the real-time requirements underlying the behaviors of a natural arm. 1 2Introduction Due to the scope of this project, a tight selection of which challenges are to be addressed is performed, considering as well the background of the project participant, revealing the following areas of focus: •creation of an anatomically correct kinematic model of the arm/forearm; •design of a multi-layered control architecture, with motion planning, motion coordination and the low level motion execution control; •study of a specific movement type and respective acquisition of joint angles data sets and; •creation of a case study based on the control architecture, natural movement and joint angle data sets, with respective comparison of natural and artificial movements. 1.2 Methodology The objectives and goals of this project are organized into three major complementary scopes, which require specific approaches and methodologies. These are as follows: 1. Anatomical Study and Myoelectrical Systems •Determination and study of the body area where the prosthesis will be applied, which arm/forearm muscles might produce the most reliable EMG signals for processing, as well as of which kind of EMG sensors should be utilized; •Techniques for myolelectric signal processing and interpretation; •Anthropomorphic arm model analysis and modeling; •Techniques for identifying natural arm motions and determining the associated motion characterizations in an adequate form to the overall control system. 2. Control: Design of a Control Architecture capable of handling the distinct degrees of motion of the prosthesis by using dynamic systems control techniques. It is crucial that the Control Architecture designed handles the replication of natural-limb type of movements and with acceptable response times 3. Simulation: By using MATLAB, a simulation tool will be developed in order to implement and test the efficiency of the overall system, which includes: •Model of the prosthetic arm; •Sets of motion constraints specifying the motion requirements of a natural arm; •Control Architecture which includes several modules and their coordination systems and; 1.3 Motivation 3 •Simulation environment centered in the chosen test scenario allowing a proper assessment of the models, control system and of the overall system. To achieve this, the difficulties caused by myoelectric signals and motion perturbations should be easily dealt with in the overall simulation. 1.3 Motivation The interaction between human and machine has always been a topic that generates a great source of curiosity. Concepts of prostheses and orthoses have been around for a long time as humans tried to mitigate the loss of functionalities due to missing or injured limbs with artificial devices, mitigating the loss of functionalities. With the advancements of the technologies associated with this area, new products and new ways to create links between human and robot are being discovered and studied every day. Since the 1950’s, when Jack E. Steele coined the term “Bionics”, the study of the connection between biology and electronics has been growing exponentially. Investigations have further developed in the area of Biomechatronics, involving already concepts in the areas of biology, electronics, mechanics, robotics and neuroscience. Nowadays, one of the main areas of research within Biomechatronics is precisely the development of products to mitigate human disabilities. The invention of robotic organs, namely locomotion organs and upper limbs, has allowed many researchers to bring a new hope for those people who lost their own. With the association of a responsive element, as a robotic limb, the artificial movement feeling associated with normal prosthetic limbs is avoided, trying to provide the user the feeling that the prosthetic limb is somewhat alive and not merely an lifeless attachment. With the high level of innovation in this area, new materials are being associated with the prosthesis and orthosis and new control methods are being developed, providing a more “human” range of motions and level of interaction with this kind of equipment. All these factors and the exciting challenges that they present motivate the research in this dissertation that addresses issues in modeling and control by building on the current state-of-theart in these fields. The investigation of advanced control approaches methods and respective implementation in a simulation environment proposed in this dissertation consists in a first step towards a more ambitious goals project with countless possibilities and a chance to further develop into more sophisticated research. 4Introduction 1.4 Document Structure This dissertation is organized in 8 chapters, each one dealing with a specific part of the project. This current chapter presents a short background about this theme, serving as a basis to justify the objectives, the motivation and the project. Chapter 2 presents a general literature review on the state-of-the-art of Myoelectric Systems encompassing different approaches for prostheses control, the sensors involved in each one and several mathematical models. The acquisition, processing and interpretation of the Myoelectric Signal are briefly studied. In Chapter 3, an analysis on control systems is conducted, presenting models and technologies fit to be integrated in the project. Chapter 4 concerns an anatomical study of the human arm will be presented, to better understand what the overall system will be required to provide to the user. This specifies the constraints for the design of the overall systems, starting with the arm model and after the verification of the ranges of motion. Chapter 5, based on the previous anatomical study, concerns the kinematic model of the considered prosthesis device and extracts the elements to be controlled by the low level controllers that will be designed in the project. Chapter 6 discusses the overall problem statement and the control architecture will be formulated and discussed. Moreover, a detailed analysis of each one of the building blocks of the control architecture and of their interaction is provided. This is tightly coupled with a brief exposition of the underlying background. In Chapter 7, the results of simulations run with the model and control systems implemented are presented and analyzed. The final chapter will present the conclusions and possible follow up developments for this project. Chapter 2 Myoelectric Systems Overview To better present the technologies and concepts pertinent to this project, this chapter is organized in two separate sub-chapters, with the relevant Signal Acquisition specifications and techniques, including sources of error and Myoelectric Systems Architecture, demonstrating the pathway from acquired signal to generated motion class. 2.1 Signal Acquisition It is important to refer that EMG signals are not the only signals currently used to control robotic prostheses. This is an area that has received quite a lot of attention and innovation over the past years. Namely, the use of Electroencephalography (EEG), which, unlike EMG, detects voltage fluctuations resulting from ionic current flows within the neurons of the brain instead of electric muscle impulses, usually with sensors located along the scalp. This way, the precision and number of signals collected increases, allowing for higher complexity prostheses but increasing exponentially the complexity of the control models. One of the main areas of innovation with the use of this technology is the possibility to implement a more sophisticated feedback, enabling the user to regain some of his/her lost sense of touch. Although the use of this technology provides a higher interest, the complexity and analysis involved would deem the project undoable for the time period, which makes the EMG signals the ideal and selected ones for the context of this project In order to understand how a myoelectric signal can be analyzed, it is necessary to understand exactly how it is produced. According to [13], the signal originates from the depolarization and repolarization of the muscle fibers, during a muscular contraction, caused by an ionic current transmitted to those fibers by a nerve’s axon, thus creating measurable action potentials. These can be read by electrodes on the skin’s surface or by invasive techniques, extracting the signal from within the muscle, using implanted myoelectric sensors (IMES). The work of [13] shows that no significant difference in the classification accuracy of the signals was detected between surface myoelectric sensors or IMES, which is profitable for this project, given the fact that the implementation of IMES would present a higher degree of difficulty, 5 12 Myoelectric Systems Overview better results, resulting in a more effective classification of the signals, by being simpler to process and reducing this processing time to a minimum. Two of the strongest features of the application of an AR method reside in the fact that, small alterations on the position of the sensors will not have a great impact on the coefficients and, has mentioned before, due to the small number of coefficients, the information to be presented to the classifier will be reduced, reducing as well the processing time required. The AR model studied is presented by [2]: ˆy(n) = M ∑ m=1 am(n).y(n−m)+e(n)(2.1) Where: •ˆy(n): estimated signal in a discrete time n; •am: AR coefficients; •e(n): estimated error; •M: order of the model; The referred author also presents the steps that were taken to calculate the AR coefficients, based on Least Mean Square (LMS) method: 1. Initialize the filter coefficients with zeroes; 2. Calculate the predicted value of the input signal: ˆy(n) = M ∑ m=1 am(n).y(n−m)(2.2) 3. Estimate the prediction error: e(n) = y(n)−ˆy(n) = y(n)+a1(n−1)+a2(m−2)+...+aM(n−M)(2.3) 4. Update the AR coefficients using the constant of convergence µ: am(n+1) = am(n)−2µ.e(n).y(n−m)(2.4) Regarding the convergence constant, there isn’t an ideal value for it, although, [2], based on the experience of other researches, mentions that the use of a very small positive value is advised (approximately 0.001), providing noticeable improvement when the LMS algorithm showed very small errors when processing the whole EMG data set. Although this solution presented itself 2.2 Myoelectric System Architecture 13 valuable, the use of such a small convergence constant brought some minor errors later on, in the Classification element, generating some distortion at the beginning of the EMG activity. To correct this situation, [2] implemented a new strategy, iterating the LMS algorithm as a whole, in which, in the first iteration, the AR coefficients were still initialized with zeroes and in the following iterations, the AR coefficients were initialized with the previous AR coefficients, instead of zeroes, continuing this process until a maximum number of iterations is reached or a minimum error between the estimated and real values of the signal. 2.2.3 Classification With the desired features extracted, it is now necessary to integrate them into distinctive classes for the recognition of the desired motion patterns. According to [14], a problem that presents itself is the fact that, due to the nature of the MES, variations in the values of a particular feature are to be expected. To cope with this, the Classifier needs to be able to adapt to varying patterns and still provide a fast response, in order to meet the real time constraints. There are several methods proposed to perform this classification, using fuzzy models, neural networks, statistical models, Bayesian models and even hybrid ones, mixing, for example, fuzzy and neural networks, which are explained by [14]. One of the most successful methods presented is the use of artificial neural networks (ANN), being able to process linear and non-linear relationships and still meet the time constraints. An example of a time-delayed ANN, containing both feature extraction and classification is presented by [17] (Figure 2.8): Figure 2.8: Structure of a time-delayed ANN Although, the implementation of such a complex model raise some difficulties, despite the advantages that it might bring. With that in mind, a unique system to deal solely with the classification seems to render better results. Such a model is implemented in the work of [2], where a Multilayer Perceptron (MLP) neural network is implemented (Figure 2.9), solution which was already explored in other similar projects. The use of an ANN implies a training stage of that network (see Figure 2.10). To do so, [2] presents a backpropagation algorithm, which is a standard method for training a MLP neural network. 14 Myoelectric Systems Overview Figure 2.9: Multilayer perceptron neural network Figure 2.10: ANN controller scheme This neural network presented is composed of n neurons in the input layer (where n is the number of AR coefficients of the previous element), 80 neurons in the hidden layer and 4 neurons in the output layer, which correspond to each movement class defined by the author (elbow flexion and extension, wrist pronation and supination). There is no clear algorithm to define the number of neurons present on the hidden layer, resulting in a more trial-and-error/empirical selection. For this project, the author conducted several experiments with different configurations until an acceptable response was provided. Also regarding hidden layers, an MLP topology may contain several of those, although, as mentioned by [2], according to the Universal Approximation Theorem, demonstrated in the work of [18], only one hidden layer is enough to guarantee the convergence of the MLP training. Having established the model, the completion of the classifier integrates two phases: the training phase, in which several patterns for each class of movement will be fed into the 2.2 Myoelectric System Architecture 15 MLP; execution/test phase to analyze the correct responses of the model. To do so, it is crucial that a correct configuration of the network is met and enough successful training is provided, without overtraining the network though. For the work of [2], the author uses two alternative stopping criteria for the training stage: a total mean square error of 0.01 is achieved; the training will be stopped after 100 epochs. The remaining parameters established for the training stage were: a fixed learning rate of 0.01 and a fixed momentum of 0; a binary sigmoid activation function for all layers; random weight initialization (between 0 and 1) and random presentation of the training patterns; target vectors. The responses expected are presented in Figure 2.11: Figure 2.11: Matrix of MLP expected responses [2] Using this configuration, the authors achieved surprising mean rates of success between 95% and 96%, when using AR coefficients with an order between 4 and 10, so, once more this method becomes attractive to implement in this type of project. 2.2.4 Actuators’ Control The controller’s objective is to provide the output commands to the actuators of the prosthesis by “translating” the classifier outputs. This element provides perhaps the highest degree of freedom, enabling a different array of possible responses. Different kinds of feedback elements can be inserted, so the chance for improvement in this element is always present. In higher complexity projects there can already be seen re-innervation techniques that allow a more “human-like” feel and the regain of touch. A possible high-level controller is presented in the work of [1] (Figure 2.12), where the author implemented a finite state machine (FSM), after implementing a neural network classifier. This specific controller was created for a three joint upper-limb prosthesis, prepared to work with six movement types: elbow flexion and extension, hand pronation and supination, close and open hand; the “modes” presented in Figure 2.12 are related to the origin/channels that produce the MES and thus correspond to different elements for the classifier. Associated with the FSM, a low-level controller was also devised to efficiently handle force, speed and position variables associated with the prosthesis movements. 16 Myoelectric Systems Overview Figure 2.12: State transition diagram of an FSM controller The final essential aspect of the controller is that the feedback provided has to allow an increase in the controllability and dexterity of the users, without presenting itself as a nuisance. The research conducted brought up a couple of elements that require proper feedback (force, speed and position) and curious techniques used to provide it. Ranging from complex ones, as mentioned previously, by muscle re-innervation where lost or damaged nerves exist, to simpler ones like visual feedback for the force/pressure variable, presenting different colors according to the pressure being applied by the prosthesis in a random object. A more complex research [19] designed a sensory feedback system where the finger tips sensation was mapped into the patient’s forearm, by air-mediated pressure, restoring the patient’s individual fingers sensation. This mapping is presented in Figure 2.13 Figure 2.13: (A) Prosthetic hand with the feedback system (B) Phantom finger mapping on the amputee (C) Conceptual illustration of the whole system 2.2 Myoelectric System Architecture 17 Please note that the controller reviewed in this section will be highly different from the one proposed for the project at hand, in notion and application. In this case, the "controller" consists in the system that generates signals to the actuators directly from the results of the Classifier and the interpretation of the MES. Learning methods are implemented to obtain the desired results without the use of models, thus presenting direct feedback without taking into consideration the mechanical model of the prosthetic device itself. This is were this project aims to improve and innovate. By using the mechanical models, the proposed control architecture aims to decrease the load from these learning processes, in respect to the actuation of the device, presenting more stable and accurate responses. The segment of classification is suppressed and this task is performed by the Control Architecture itself. 18 Myoelectric Systems Overview Chapter 3 Control Background A bionic prosthetic limb is basically a robotic manipulator that approximates a humanoid limb and its type of behavior. With that in mind, using the Denavit-Hartenberg (DH) parameters, a model of an anthropomorphic limb can be devised. Although, unlike a normal robotic manipulator, the control of a prosthetic limb isn’t, by all means, linear and is required to be able to withstand brute and unpredicted changes while maintaining an acceptable response. Considering these constraints, this chapter aims to present several Control related tools and concepts capable of providing benefit to the project. 3.1 Hybrid Systems To better understand what an Hybrid System encompasses it is necessary to initially consider the following: a Discrete System is a system with a countable number of states, for example, a computer program that may even possess a large state space, will not only be countable but knowingly will be finite; as for Continuous Systems, these are systems that present a continues behavior, normally ruled by a time variable, and that have a real-valued state space, where the system’s evolution over time can be described by a continuous function or differential equations. An Hybrid Dynamic System consists in a dynamic system driven by both ordinary differential equations (time-drive component) and by discrete events. This is a class of systems whose importance has been increasing in engineering due to the fact that the complexity inherent to the modern advanced engineering systems involves not only the laws of physics but also logic. A combination of both Continuous and Discrete behaviors is, for example, a physical system being controlled by a discrete controller (see examples in Figure 3.1). 19 20 Control Background Figure 3.1: Hybrid System Model of a car with 4 gears (above) and a Computer-Controlled System (bellow) [4] Among the various types of models to represent this class of systems, Hybrid Automata have been the ones gaining the most popularity. In his work, [4] presents this modeling language that describes the evolution in time of the values of a set of discrete and continuous state variables. So, in order to properly define an hybrid automaton, the following elements must be presented: •Q→set of discrete states; •X→set of continuous states; •f(.,.):QX →vector field; •Init ⊆Q∗X→set of initial states; •Dom(.):Q→P(X)→domain (P(X) represents a powerset containing all subsets of X); •E⊆Q∗Q→set of edges; •G(.):E→P(X)→set of edges; •R(.,.):E∗X→P(X)→reset map; The hybrid system is this way defined by H= (Q,X,f,Init,D,E,G,R), in which (q,x)∈ Q∗Xis a state of H. The modeling of the system also defines possible evolution for it’s state. Considering an initial value (q0,x0)∈Init, the coninuous state variable xevolves according to the differential equation: 3.1 Hybrid Systems 21 ˙x=f(qo,x),x(0) = x0 while the discrete state variable qremains unchanged: q(t) = q0 The evolution of the continuous component will proceed as long as xremains within Dom(q0). In case the continuous state xreaches the guard G(q0,q1)⊆Rnof an edge (q0,q1)∈E, the discrete state may change it’s value to q1and, subsequently, the continues state gets reset to a value defined in R(q0,q1,x)⊆Rn. Prior to a discrete transition, the continuous evolution resumes and the process repeats. Associated with these systems are several constraints that require to be analyzed for an optimal performance. Conditions associated with continuous and discrete systems are naturally transposed to an hybrid system and require, in the modeling stage, to be taken into consideration. The presence of dead-locks associated with the discrete systems, where the operation is locked within a untransitionable state for absence of transition conditions or the Zeno phenomenon, in which and infinite number of discrete transitions occur during a finite period of time, are but a few of the concerns that need to be recognized. A few of the conditions required to be met at the modeling stage of hybrid systems are presented in [4]: •Existence of solution, precisely taking into consideration the case of dead-locks; •Uniqueness of solution which, if isn’t met, imposes to the system a decision between different alternatives. A few measures can be taken in said situations, where different decisions have different priorities; •Reachability which, in rough terms, defines if a state is reachable from any system condition within a finite time period; •Verification which is an important operation that ensures the logic correctness of the behaviors, guaranteeing that the hybrid system meet the desired specifications; 28 Control Background In Figure 3.5, the overall basic structure of a NMPC control loop is presented: Figure 3.5: Basic NMPC Control Loop [5] As it can be seen, it is similar to the one of a MPC, as presented before and follows the same scheme of action. From all these considerations and from the NMPC setup, the following key characteristics are extracted: •Allows the use of a nonlinear model for prediction; •Allows the explicit consideration of state and input constraints; •Specified performance criteria is minimized on-line; •The predicted behavior is in general different from the closed loop behavior; •The on-line solution of an open-loop optimal control problem is necessary for the application of NMPC; •To perform the prediction the system states must be measured or estimated. Chapter 4 Anatomical Model Analysis It is of the utmost importance to understand how a human arm works in order to understand what a prosthetic arm needs to cope with. With this in mind, the current chapter will provide a brief anatomical study on the human arm, trying to understand its composition and functionality, analyzing the type and purpose of movements involved, presenting data on the degrees of freedom (DOF) of each joint, a normal range of motion (ROF) and the analysis of the typical displacements in the joints according to a normal arm movement (hand to mouth motion). 4.1 Anatomy of the Arm For the purpose of building a prosthetic limb, the initial and most important elements to consider are the functionality of the given limb and the joints associated with it which, for the human arm, are: glenohumeral joint (shoulder), elbow joint, wrist joint and the joints of the hand (including gliding joints, saddle joint for the thumb and lastly the metacarpophalangeal and interphalangeal joints). The arm segments eill be considered rigid elements, that will be controlled by motors, in order to simplify the model, suppressing the individual muscles. Given the complexity and extent of the project, the hand joints will not be studied thoroughly, due to the high level of different grip patterns possible by the humam hand, considering merely the hand as an end-effector. Another point that shall not be fully considered is the fact that the upper limb presents a key role in human locomotion, providing stability. Interesting by itself, this feature would greatly increase the complexity of this study, removing focus on the basic functionality of the arm itself that is intended to be studied. In Figure 4.1 the structure and main regions of the arm are presented. 29 30 Anatomical Model Analysis Figure 4.1: Regions, bones and joints of the arm [6] 4.1.1 Average Length and Weight The feeling of inclusion of the "external" limb is one of the most important elements of a prosthetic device. The main goal is to make the user believe that the prosthesis is not an attachment but a part of his/her own system. Considering this, it becomes important to know how the arm/forearm length and weight compare to other parts of the body to properly design the prosthetic device. Although it improves the previously referred sense of inclusion, this element also adds complexity to the system given that a prosthetic arm will always be designed to one specific user only, given the user’s required arm length and weight specifications. 4.1 Anatomy of the Arm 31 The following Table 4.1 presents a comparison between average body segment lengths with the total body height of person, taken from [26]. Table 4.1: Average Body Segment Length (in percentage of Total Body Height) Segment Males Females Average Head and Neck 10.75 10.75 10.75 Whole Trunk 30 29 29.5 Thorax 12.7 12.7 12.7 Abdomen 8.1 8.1 8.1 Pelvis 9.3 9.3 9.3 Upper Arm 17.2 17.3 17.25 Forearm 15.7 16 15.85 Hand 5.75 5.75 5.75 Thigh 23.2 24.9 24.05 Leg 24.7 25.7 25.2 Foot 4.25 4.25 4.25 Biacromial 24.5 20 22.25 Bi-iliac 11.3 12 11.65 Gathering the necessary information presented in Table 4.1, the average arm length (from shoulder to fingertip) is around 38,85 % of the average individual total body height. Regarding weight, Table 4.2, like the one before, presents a comparison between average body segments weight and the individual total body weight, taken from [27]. Table 4.2: Average Body Segment Weight (in percentage of Total Body Weight) Segment Males Females Average Head and Neck 6.94 6.68 6.81 Trunk 43.46 42.58 43.02 Upper Arm 2.71 2.55 2.63 Forearm 1.62 1.38 1.5 Hand 0.61 0.56 0.585 Thigh 14.16 14.78 14.47 Shank 4.33 4.81 4.57 Foot 1.37 1.29 1.33 As can be seen from Table 4.2, the average total arm weight is around 4.715 % of the average individual total body weight. Unlike other prosthetic devices, given the kind of electronics and materials involved, a myoelectric arm prosthesis (adult arm) can weight down to a minimum of 1.010 kilograms, which is about a quarter of the weight of the arm of an average individual (about 4kilograms). This guarantees that the user’s posture will not be affected by heavy devices nor will the prosthesis be presented as a nuisance/burden to the user. 32 Anatomical Model Analysis 4.1.2 Types of Joints and their Functionality As seen on Figure 4.1, the human arm possesses 3 distinct joints (apart from the previously mentioned hand joints), each with very specific features and functions. Shoulder Joint Figure 4.2: Ball and Socket Joint/Shoulder Joint [7] Starting with the most complex one, the shoulder joint (glenohumeral joint) (Figure 4.2) is aball and socket joint and, as its name clearly states, consists of the articulation between the spherical surface of a bone and a dish-shaped depression of another bone, which in case of the shoulder, represents the articulation between the humerus and the glenoid cavity. This type of joint allows movement in every plane, involving flexion, extension, adduction and abduction (both vertical and horizontal) and medial and lateral rotation (Figure 4.3). Figure 4.3: Shoulder Joint Movements [7] This means that the shoulder joint is capable of circumduction, involving a combination of some of the previous actions, which together allow for a cone-shaped movement. 4.1 Anatomy of the Arm 33 Elbow Joint Figure 4.4: Hinge Joint/Elbow Joint [7] Next one will be the elbow joint (humeroulnar joint), which is an hinge joint, as can be seen in Figure 4.4. This type of joint, as its name clearly states, provides a hinge type of movement between two body segments, which in case of the elbow connects the distal end of the humerus in the upper arm and the proximal ends of the ulna and radius in the forearm. Unlike the glenohumeral joint, the elbow joint only allows movement in one plane due to its hinge nature. In this case, the elbow joint will allow the flexion and extension of the elbow, as can be seen in Figure 4.5. Figure 4.5: Elbow Joint Movements [7] Associated with the elbow joint and the forearm are two other movements: supination and pronation (Figure 4.6). Figure 4.6: Pronation and Supination of the Forearm [7] 34 Anatomical Model Analysis By the action of the pronator or supinator muscles, over the radial and ulnar bones, the forearm is able to be placed in a palm up or palm down position. This action although is not so simple to be recreated with a prosthetic device, hence the commom use of hand prostheses in which the wrist itself provides a rotating mechanism, allowing to replicate the supine and pronated stances. Wrist Joint Figure 4.7: Ellispoid Joint/Wrist Joint [7] The last joint that will be covered in this study is the wrist joint (radiocarpal joint), which is an ellipsoid joint, as can be seen in Figure 4.7. This joint connects the distal part of the radial bone to the carpals. As an ellipsoid joint, the wrist will allow movement in two planes, allowing flexion, extension, adduction and abduction of the joint, as can be seen in Figure 4.8. Figure 4.8: Wrist Joint Movements [7] 4.1 Anatomy of the Arm 35 4.1.3 Range of Motion An important aspect that has to be noted is that the previously presented joints are limited in motion. It is important to make sure that the recreated model follows these "restrictions" to correctly recreate a normal human arm type of motion. According to the data collected present in [8], the range of motion for each degree of freedom of the joints previously mentioned is shown in Figures 4.9 and 4.10. Figure 4.9: Range of Motion of Glenohumeral Joint [8] Note that the notation presented in the figures relative to the angles is now in global coordinates (roll, pitch and yaw, respectively, rotation in x-axis, y-axis and z-axis). Figure 4.10: Range of Motion of Elbow and Wrist Joints [8] The previously mentioned supination/pronation stances can be seen in Figure 4.10, recreated by the alteration of the wrist yaw. 36 Anatomical Model Analysis The previous data is summarized in Table 4.3 : Table 4.3: Range of Motion in human arm joints Joint Range of movement Angle (o) Shoulder Pitch upper/lower 180/50 Roll upper/lower 180/0 Yaw upper/lower 90/90 Elbow Pitch upper/lower 150/10 Wrist Pitch upper/lower 60/60 Roll upper/lower 20/30 Yaw upper/lower 90/90 Knowing then the average lenght of the each arm segment, the degrees of freedom of the arm joints and lastly, the range of motion of said joints, it becomes possible to estimate a human arm’s "functional workspace", as compared to normal robotic manipulators (see Figure 4.11). Figure 4.11: Anthropomorphic Robotic Arm Workspace [9] It is important to estimate this functional workpspace correctly in order to perform an accurate anatomical/mechanical parameters verification during the movement/action of the prosthetic device. Ideally system requires to initially estimate if the point of interest will be reachable by the prosthetic, prior to the activation of the actuators. The work of [28] is precisely directed towards determining the human arm reachable workspace. Although the measurements present in this study slighty differ from the ones mentioned previously (arm segment measurements and range of motion), the study successfully demonstrates the workspace volume of an healthy human arm, which, for the subject used wasV=0.667±0.055m3. The computed workspace for said subject’s right arm was then computed and is presented in Figure 4.12. 4.2 Movement Analysis 37 Figure 4.12: Reachable Workspace of Right Arm The similarities between Figure 4.12 and 4.11 can now be seen. Relating this information with the desired simulation of the project, any point estipulated for the end-effector of the manipulator that is outside this volume will be discarded, forcing the user to re-plan the trajectory/plan of motion. 4.2 Movement Analysis The previous sections focused on describing the characteristics and constraints of the human arm. This last section aims to present a small study on the joint displacements that occurs during a specific motion, in order to later replicate this motion in the simulation. There are several ways to study and analyze how a body joint behaves during a certain movement and this is a field of study that has been approached for many different topics and for several years. One of the most wide spread technologies to study the joint motion is using high speed cameras (with high capture rates of several hundred frames per second, e.g. Biomechanical Laboratories) and a series of anatomically placed markers in the subject’s joints being studied (an example of such placement is show in Figure 4.14). Studies like these allow for example the comprehension of how a normal or an incapacitated user reacts to specific tasks and the differences they present between each other regarding joint displacements. 44 Anthropomorphic Arm Kinematic Modelling Figure 5.2: Structural scheme of a 7 DoF anthropomorphic arm [12] Based on the previous DH parameters and the structural scheme presented in [12], shown in Figure 5.2 (considering the base frame at the shoulder), the following homogeneous matrices were constructed for each frame of the structure (see transformation and rotation matrices in the Appendix), where Lua,Lf a and Lhrepresent, respectively, the average calculated lengths of the upper arm, forearm and hand (as seen in Chapter 3): H1=Ry(θ1) =       cos(θ1)0sin(θ1)0 0 1 0 0 −sin(θ1)0cos(θ1)0 0 0 0 1       (5.5) H2=Rx(θ2) =       1 0 0 0 0cos(θ2)−sin(θ2)0 0sin(θ2)cos(θ2)0 0 0 0 1       (5.6) H3=Rz(θ3).Tz(Lua) =       cos(θ3)sin(θ3)0 0 −sin(θ3)cos(θ3)0 0 0 0 1 Lua 0 0 0 1       (5.7) 5.1 Kinematic Modeling 45 H4=Ry(θ4).Tz(Lf a) =       cos(θ4)0sin(θ4)sin(θ4).Lf a 0 1 0 0 −sin(θ4)0cos(θ4)cos(θ4).Lf a 0 0 0 1       (5.8) H5=Rz(θ5) =       cos(θ5)sin(θ5)0 0 −sin(θ5)cos(θ5)0 0 0 0 1 0 0 0 0 1       (5.9) H6=Rx(θ6) =       1 0 0 0 0cos(θ6)−sin(θ6)0 0sin(θ6)cos(θ6)0 0 0 0 1       (5.10) H7=Ry(θ7).Ty(Lh) =       cos(θ7)0sin(θ7)0 0 1 0 Lh −sin(θ7)0cos(θ7)0 0 0 0 1       (5.11) Having the homogeneous transformation matrices for each frame of the structure, the Forward Kinematics solution becomes available in the form of the following DH matrix of the system: 0T7=H1.H2.H3.H4.H5.H6.H7(5.12) Thus, expressing any point in the system, in relation to the base frame or any other, becomes possible by the respective adjustment and calculation of the expression 5.12. 5.1.2 Inverse Kinematics Opposite to the Forward Kinematics, the Inverse Kinematics allows for the inference of the joint parameters required to provide a specific position and orientation of the end-effector (x, y, z and φx,φy,φzare known). The inverse kinematics problem is not as simple nor as linear as the forward kinematics one, having several possible approaches and sometimes multiple possible solutions. Several methods are studied and presented in [36] and [37], focusing in the use of Algebraic and Geometric solutions, specific to the problem at hand, or the use of an Iterative method, like the use Jacobian inverted matrix of the system or the CCD (cyclic coordinate descent) for simple structures. Another possible method is the use of computer software that model the manipulators (like AutoCAD) and export them to software like Matlab’s Virtual Environment, allowing for the computation of the Inverse Kinematics through the analysis of the model itself. In the case at hand, a geometric approached was used, based on [34], following the next steps: 46 Anthropomorphic Arm Kinematic Modelling 1. given its nature, the robot arm has a DoF in excess of 6 possible DoF in the 3D space, so an initial condition for θ1(shoulder adduction/abduction) is imposed, considering this joint value known and given by the user (which doesn’t pose a problem, considering that the final prosthetic limb is located anteriorly of the shoulder and it’s respective joints); 2. knowing the position and orientation parameters (x, y, z, φx,φy,φz) a transfer matrix is created which characterizes the position and orientation of the end-effector in accordance with a fixed coordinate system; 3. using the previous transfer matrix, the wrist position is determined (xw,yw,zw); 4. having then the wrist position and orientation, the remaining joint parameters will be determined (θ2...θ7); Having established the procedures, the system’s transfer matrix is determined using 5.12 and the given position and orientation parameters. The matrix that then determines the wrist position and orientation is given by: Tw=0T7∗      1 0 0 0 010Lh 0 0 1 0 0 0 0 1       −1 (5.13) The elements of 0T7will henceforth be represented by: 0T7=      a11 a12 a13 a14 a21 a22 a23 a24 a31 a32 a33 a34 a41 a42 a43 a44       (5.14) Considering this notation, the following expression becomes valid: ha14 a24 a34 a44iT=Tw∗h0 0 0 1iT(5.15) The expression that defines the distance between the base frame and the end-effector position is: d=qa2 14 +a2 24 +a2 34 (5.16) Having established these initial conditions, θ4is the first joint parameter that can be inferred, by following the expression: θ4=π±L2 ua +L2 f a −d2 2LuaLfa(5.17) 5.1 Kinematic Modeling 47 The formula for θ4is determined. Expanding Equation 5.15, the next system of equations is derived: H1.H2.H3.H4.      0 0 0 1       =      a14 a24 a34 0       (5.18) Multiplying this system of equations by A−1 1to the right gives the following system of equations: H2.H3.H4.      0 0 0 1       =      b14 b24 b34 0       <=>      b14 b24 b34 0       =A−1 1.      a14 a24 a34 0       (5.19) Having the value of θ4and the parameters from the previous system of equations, the value of θ3can be obtained by the following expression: θ3=acosb14 Lf a.sin(θ4)(5.20) From Equation 5.19 and knowing both θ3and θ4, the deduction of θ2is performed by the following equation (where c(*) is cosine and s(*) is the sine): θ2=−2atanLua +Lf ac(θ4)+qL2 ua +2LuaLf ac(θ4)+(Lf ac(θ4))2+(Lf as(θ3)s(θ4))2−b2 24 b24 +Lf as(θ3)s(θ4) (5.21) To determine the last joint parameters (θ5,θ6and θ7), the following system is considered (remembering expression 5.13): H5.H6.H7=H−1 4.H−1 3.H−1 2.H−1 1.Tw(5.22) Let the above system be represent as: H−1 4.H−1 3.H−1 2.H−1 1.Tw=      m11 m12 m13 m14 m21 m22 m23 m24 m31 m32 m33 m34 m41 m42 m43 m44       (5.23) 48 Anthropomorphic Arm Kinematic Modelling Through the equality of the above matrices and knowing the remaining joint parameters, the final parameters can be obtained through the following expressions:          θ6=asin(m32) θ7=−atan2(m31,m33) θ5=−atan2(m12,m22) (5.24) As can be seen, the system requires long and complicated calculations for determining the joint parameters, which is extremely error prone if done by hand. In order to do simplify the calculations, the whole model of the Inverse Kinematics was replicated in the tool Simulink, present in Matlab, in order for the required computation to be done fairly quick and with minimum error possible. A fraction of model implemented is presented in Figure 5.3 and 5.4, with a response time of under 50ms: Figure 5.3: Fraction of Inverse Kinematics Simulink model 5.1 Kinematic Modeling 49 Figure 5.4: "VirtuArm" Inverse Kinematics 50 Anthropomorphic Arm Kinematic Modelling Chapter 6 Problem Statement and Approach From the literature review concluded, plenty of information about the acquisition and processing of the MES was studied and discovered. Although, regarding the Actuators’ Control itself for these prostheses (link between the detected classified types of motion and the actuators in the prosthetic device), the information discovered was vague, mostly presenting information on Finite State Machines that handle and coordinate the different motion, not mentioning the actuation and reference for each individual actuator (e.g. linear motor). Given the high complexity of this project, with a broad range of subsystems, there is plenty of areas for improvement, since the initial debate over which features to extract and process from the MES to "feed" the Classifier, the method of Classification itself, be it using a possible ANN, a Fuzzy or any other alternative, and, as mentioned before, the Control System itself. Interesting on its own, this Classification segment wasn’t integrated in the final project, again due to the limitations in time and resources. As this project is very limited in time, and given the area of study of the author, the Control System will be the main focus. This chapter then intends to raise the necessary project requirements, presenting an initial system concept, identifying the planned strategies to divide the overall control of the prosthetic arm into simpler subsystems and approach them with two separate levels of control (low level control with a possible Model Predictive Controller and high level control with a possible Finite State Machine). 6.1 Project Requirements Based on the reviewed literature and market survey, an initial list of requirements was established, as can be seen in Table 6.1 dividing the items between Functional and Market/User Requirements (classifying from A to C and N/A, where A is of the greater and C of lesser importance, and N is non applicable for the project at hand): 51 52 Problem Statement and Approach Table 6.1: Functional and Market/User Requirements Functional Requirements Description Priority Level Anatomical Accuracy The system must respect certain anthropomorphic constraints: average human arm length and speed; joint degrees of freedom B Accurate MES Processing In order to generate the correct references in the Control Architecture, the previous acquisition module must be able to correctly identify a certain desired natural motion N Robust and Adaptable Control Due to the nature of this system, the control system needs to be able to quickly and effectively adapt to different situations, maintaining an accurate response A Instant and corrective Feedback If a not so accurate response is obtained the system will need to perform an iterative reforcing in order to obtain a close to perfect solution. In case no command is provided to the muscle, the prosthesis maintains its current position B Real Time Response Given that the end user is a human being, the overall system must produce a response within 100 ms, rapid enough to make the user minimally aware of this delay C Market/User Requirements Description Priority Level Adaptability to different individuals A prosthetic device is something extremely personal so each product needs to have a specific size and weight for an individual user. Making a modular device which could be easily adjusted would reduce slightly the costs associated with the "personalization" of each build N Energy efficient The prosthetic device encompasses a high amount of sensors and actuators that require a local battery source, making it important to bear in mind the power involved in each subsystem to maintain an acceptable battery life for the user N Cost efficient Prosthetic devices in the market nowadays are highly priced due to the materials, product individuallity and technology involved. Reducing the cost of manufacture is an overall must for these systems N Acceptable weight The device cannot become a nuisance for the user due to overweight (usually it’s considered for a myoelectric prosthesis to have around 1/4 of the weight of the user’s arm, which usually corresponds to around 1 Kg) N 6.2 General System Diagram The following Figure 6.1 presents an early vision of the overall system architecture: Figure 6.1: System Overview 6.3 Control Architecture 53 The system is divided into four segments: •The user and the myoelectric acquisition system: the responsibility of this segment is to feed the control layers the processed initial EMG signals, taking into account the verification of the anatomical and mechanical constraints, considering a targeted motion (this segment, given the goals for the project and the limited scope, will not be simulated, considering a set of know typical inputs taken from previous studies); •The High Level Controller: composed by different layers, this segment is initially responsible by the decoding of the targeted motion into a motion plan with a set trajectory, feeding a motion coordination layer responsible for dividing this trajectory into separate segments for each joint, trying to approximate and adapt the signals towards a reference (natural limb motion) and for issuing the necessary commands to each of the low level controllers; •The Low Level Controller: receives commands from the high level controller in order to generate and adapt the joints angle/position references for the prosthesis actuators, plus receiving information from the prosthetic sensors; •The prosthetic device: with respective sensors and actuators; 6.3 Control Architecture The purpose of the control architecture consists in enabling "natural" behaviors of the prosthetic arm by adapting the references made available to the low level control subsystems as the motion is being executed. The overall idea is to be able to achieve a behavior as close as possible to that of a natural normal limb with the much simpler mechanical device that embodies the prosthesis upon receiving the appropriate commands from the brain in the form of a set of EMG signals. 60 Simulated System Results Figure 7.2: Simulated Prosthetic’s Workspace - Proximal Perspective Each dot in these graphs represents a valid target point for the end-effector, by means of a combination of plausible joint angle values for said joints. Note that the different concentrations of points in these Figures represent increased possibility of solutions, where the same point can be reached with different joint configurations. The higher concentrated areas thus represent a higher likelihood to find a reachable point. The similarities between the manipulator’s and the right human arm’s functional workspace present in Chapter 4 are evident, as was intended. After the mapping the function exports all these values into a data table, which is then used as an oracle to verify if the intended estimated positions are reachable (Figure 7.5). 7.1 Anatomical and Mechanical Constraints Verification 61 Figure 7.3: Simulated Prosthetic’s Workspace Two Dimensional Overview Figure 7.4: Excerpt of generated .csv file with end-effectors possible positions Using this procedure, the system compares if the estimated position is present or approximate to the values of the data table, confirming the reachability of said position and allowing the system to direct itself towards it. If not, it will require that the user himself/herself adjusts his/her own position. 62 Simulated System Results 7.2 Case Study Arm Model Implemented The implementation of the devised Control Architecture to the previously mentioned anthropomorphic arm model (Chapter 4) isn’t applicable for the scope of this project, due to the high complexity of such operation. In order to test the given architecture, a new simpler kinematic model was created, based on the lecture notes [38], emulating a typical forearm prosthetic device (elbow and wrist joints only): Figure 7.5: Structural scheme of a 3 DoF Forearm The respective DH representation is then presented in Table 7.1: Table 7.1: DH parameters for 3 DoF Forearm Frame aiαidiθi 1 0 qe0 0 2 0 qwr Lf a 0 3 0 0 0 qwy 4 0 0 Lh0 7.2 Case Study Arm Model Implemented 63 Having computed the Forward and Inverse Kinematics equations, the following model was constructed: Figure 7.6: Forward and Inverse Kinematics, yielding correct results with example joint angle values 7.2.1 Generation of joint angle references Having tested the kinematics model, data sets were created, replicating the joint angle deviations previously selected and demonstrated in the last section of Chapter 4. In Figure 7.7 the trajectory taken by each of the joints is shown: Figure 7.7: Elbow, Wrist Yaw and Wrist Roll trajectory during "drinking water" task 64 Simulated System Results Having the kinematic model and the 3 joint trajectory references, a simulation of the natural movement during the "drinking water" task was performed: (a) Initial forearm condition (b) Forearm at 30% of the task’s trajectory (c) Forearm at 70% of the task’s trajectory (d) Forearm at 90% of the task’s trajectory Figure 7.8: Natural trajectory of human limb during "drinking water" task In Figure 7.8, note that the elbow is located at the origin of the referential. The line segment that connects the origin to the wrist position (represented by the red circle) represents the forearm and the line segment that connects the wrist to the tip of the end-effector (represented by the red square) represents the hand segment. 7.2 Case Study Arm Model Implemented 65 This complete motion is represented in Figure 7.9: Figure 7.9: Complete "drinking water" task representation This motion is replicated with high fidelity, demonstrating a complete natural, human-like set of movements. In order to test the system’s response to possible external disturbances, a simulation was created to introduce a variable error factor into each respective joint angle trajectory, across several points within the trajectories themselves, having a simple PID controller attempting to correct these errors. The results, taking into consideration the case study previously mentioned, are presented in the following Figures 7.10(a),7.10(b) and 7.10(c), where the purple line corresponds to the natural motion, intended to be replicated, while the yellow line corresponds to the controller’s response to the given disturbances: 66 Simulated System Results (a) Elbow joint angle trajectory approximation (b) Wrist Yaw joint angle trajectory approximation (c) Wrist Roll joint angle trajectory approximation Figure 7.10: Disturbance response in "drinking water" motion task Chapter 8 Conclusions The broadness of this project revealed itself to be quite extensive, often presenting bigger and harder challenges that initially weren’t fully considered. An extensive study on Electromyography, Human Anatomy and prosthetic devices kick-started the design of a system that would be able to encompass all the human like features required to replicate an artificial limb. By analyzing State-of-the-Art technologies on EMG signal acquisition and processing, and how to use this signal to control several different devices (mostly prosthetic limbs), allowed for the inference of the project’s goals and possible areas for innovation. Conducting a medical/mechanical study on the human arm culminated in the definition of the human constraints required to be met by the artificial limb. Information on average segment lengths and weights were gathered, essential to further devise the model. Even more important, the degrees of freedom involved in each joint of the human arm (excluding hand) and respective ranges of motion were gathered. This study also allowed to understand the type of variables that would be further controlled. To model the human arm, several studies were analyzed. In the ideal case, an anthropomorphic arm, presenting the full seven degrees of freedom, would be the best fit for such a project. Based on the work [12], the author gathered the required information to build a kinematic model of the human arm, as was presented. Later on, due to the complexity of the remaining elements of the project, the complexity of this arm model itself was reduced, altering the initial seven degrees of freedom model to one with three degrees of freedom, more adapted for the required simulations and considering the most common type of arm prosthetic device, located at a transhumeral position (forearm prosthetic). Knowing then how the model should function, being able to provide natural, human-like motion, came the need to identify which control techniques would grant an artificial limb means to be able to replicate such kind of movements. A study on nonlinear systems, hybrid control and most importantly, model predictive control was conducted, understanding how these tools could help formulate the mathematical problem behind the control of the prosthetic device. 67 68 Conclusions Taking these elements into account and the respective model allowed finally for the design of a control architecture. In the authors opinion, the design of the Control Architecture presented itself to be the hardest obstacle to overcome. The definition of a high level controller based on an MPC, with the ability to adapt not only its inputs but its references as well, was the innovative factor that would approximate the prosthesis’s movements as close as possible to a human limb. The design of the control architecture produced three layers - Motion Planning, Execution and Low Level Control - in which the high level controller mentioned before (integrated into the Motion Execution Layer) generates the signals for the subsystems bellow, controlling individually each join actuator. Having both model and control, the selection of simple case study was performed, in which the simulation environment could test the controller’s design. Based on the software "Matlab" the simulation of some of these elements was performed. As mentioned, the greatest difficulties of this project were in the Control Architecture. Due to lack of time, some of the elements were designed but weren’t able to be fully implemented or tested, which is a key element for further developments. As mentioned before, this project encompasses several distinct areas, ranging from the initial acquisition and processing of the EMG, to the generation of robust, quick and adaptable control signs to the prosthesis actuators. The main focus of the project was set on the model and the Control Architecture, rather than the EMG signal acquisition and processing. In this area lies a great motivation to continue this project, where real signals would be used to feed the Motion Planning block, instead of simulated ones. The use of a more accurate and with an higher degree of anthropomorphy model is one of the author’s next goals. Indeed several subsystems of the project have, by themselves, related improvement areas which the author will try to explore in further developments. Appendix A Appendix A.1 Foward Kinematics Rotation and Translation A.1.1 Translations Translation in X, Y and Z axis, by x,y,z values respectively: Tx=      1 0 0 x 0 1 0 0 0 0 1 0 0 0 0 1       ;Ty=      1 0 0 0 0 1 0 y 0 0 1 0 0 0 0 1       ;Tz=      1 0 0 0 0 1 0 0 0 0 1 z 0 0 0 1       A.1.2 Rotations Rotation in X, Y and Z axis, by φx,φy,φzdegrees/radians respectively: Rφx=      1 0 0 0 0cos(φx)−sin(φx)0 0sin(φx)cos(φx)0 0 0 0 1       ;Rφy=      cos(φy)0sin(φy)0 0 1 0 0 −sin(φy)0cos(φy)0 0 0 0 1       Rφz=      cos(φz)−sin(φz)0 0 sin(φz)cos(φz)0 0 0 0 1 0 0 0 0 1       A.2 Functional Workspace The following algorithm was developed to present an approximation of the manipulators functional workspace and creation of an "oracle" table for the respective constraint verification: 69 76 REFERENCES [12] M Crenganis, R Breaz, G Racz, and O Bologa. The inverse kinematics solutions of a 7 DOF robotic arm using Fuzzy Logic. In Industrial Electronics and Applications (ICIEA), 2012 7th IEEE Conference on, pages 518–523, 2012. doi:10.1109/ICIEA.2012.6360783. [13] Levi J Hargrove, Kevin Englehart, and Bernard Hudgins. A comparison of surface and intramuscular myoelectric signal classification. Biomedical Engineering, IEEE Transactions on, 54(5):847 – 853, 2007. [14] Mohammadreza Asghari Oskoei and Huosheng Hu. Support Vector Machine-Based Classification Scheme for Myoelectric Control Applied to Upper Limb. Biomedical Engineering, IEEE Transactions on, 55(8):1956–1965, 2008. [15] K Englehart, B Hudgins, P.a Parker, and M Stevenson. Classification of the myoelectric signal using time-frequency based representations. Medical Engineering & Physics, 21(67):431–438, July 1999. URL: http://linkinghub.elsevier.com/retrieve/ pii/S1350453399000661,doi:10.1016/S1350-4533(99)00066-1. [16] Reza Boostani and Mohammad Hassan Moradi. Evaluation of the forearm EMG signal features for the control of a prosthetic hand. Physiological Measurement, 24(2):309–319, May 2003. URL: http://stacks.iop.org/0967-3334/24/i=2/a=307?key= crossref.c8c3361c322cda3999f29eb9ee7dd556,doi:10.1088/0967-3334/ 24/2/307. [17] Arthur T C Au and Robert F Kirsch. EMG-based prediction of shoulder and elbow kinematics in able-bodied and spinal cord injured individuals. Rehabilitation Engineering, IEEE Transactions on, 8(4):471–480, 2000. [18] S S Haykin. Neural Networks and Learning Machines. Pearson International Edition. Pearson Education, 1994. URL: http://books.google.pt/books?id=KCwWOAAACAAJ. [19] Christian Antfolk, Anders Björkman, Sven-Olof Frank, Fredrik Sebelius, Göran Lundborg, and Birgitta Rosen. Sensory feedback from a prosthetic hand based on air-mediated pressure from the hand to the forearm skin. Journal of rehabilitation medicine, 44(8):702–7, July 2012. URL: http://www.ncbi.nlm.nih.gov/pubmed/22729800,doi:10.2340/ 16501977-1001. [20] Alberto Bemporad and Manfred Morari. Robust model predictive control: A survey. In Robustness in identification and control, pages 207–226. Springer, 1999. [21] J B Rawlings and D Q Mayne. Model Predictive Control: Theory and Design. Nob Hill Pub., 2009. URL: http://books.google.pt/books?id=3_rfQQAACAAJ. [22] Mark Cannon. C21 Model Predictive Control Lecture 1. University of Oxford, 2013. [23] Book of Abstracts 21st Benelux Meeting on Systems and Control. 21st Benelux Meeting on Systems and Control, 2002. [24] S Joe Qin and Thomas A Badgwell. An overview of nonlinear model predictive control applications. In Nonlinear model predictive control, pages 369–392. Springer, 2000. [25] Venkateswarlu Ch. Model predictive control of nonlinear processes. In Tao Zheng, editor, Model Predictive Control, pages 109–141. 2010. REFERENCES 77 [26] S. Plagenhoef, F.G. Evans, and T Abdelnour. Anatomical data for analyzing human motion. Research Quarterly for Exercise and Sport, (54):169–178, 1983. [27] Paolo de Leva. Adjustments to Zatsiorsky-Seluyanov’s Segment Inertia Parameters. Journal of Biomechanics, 29(9):1223–1230, 1996. [28] Nives Klopˇ car and Jadran Lenarˇ ciˇ c. Kinematic Model for Determination of Human Arm Reachable Workspace. Meccanica, 40(2):203–219, April 2005. URL: http://link.springer.com/10.1007/s11012-005-3067-0,doi:10. 1007/s11012-005-3067-0. [29] Mohammed Z Al-faiz. Human Arm Simulation Based on Matlab with Virtual Environment. Iraqi Journal of Computers, Communication and Control & Systems, 11(1):86–96, 2011. [30] Mohammed Reyad Abuqassem. Simulation and Interfacing of 5 DOF Educational Robot Arm. Islamic University of Gaza, (June), 2010. [31] Panagiotis Artemiadis. Closed-Form Inverse Kinematic Solution for Anthropomorphic Motion in Redundant Robot Arms. Advances in Robotics & Automation, 02(03), 2013. doi:10.4172/2168-9695.1000110. [32] Valentin Grecu, Luminita Grecu, Mihai Demian, and Gabriela Demian. A Virtual System for Simulation of Human Upper Limb. In Proceedings of the World Congress on Engineerin g (WCE), London, UK, volume II, pages 1–5. 2009. [33] M. Rygaard and S. Bai. Design of Anthropomorphic Robot Arm: ARA-1. Aalborg University, 2008. [34] M Creganis, I Chera, and O Bologa. The Inverse Kinematics of an Anthropomorphic Robot Arm with Seven Degrees of Freedom. Industrial Electronics and Applications (ICIEA), 2012 7th IEEE Conference on, 8(2):20–25, 2012. [35] Lorenzo Sciavicco and Bruno Siciliano. Modelling and control of robot manipulators. Springer, 2000. [36] Samuel R Buss. Introduction to Inverse Kinematics with Jacobian Transpose , Pseudoinverse and Damped Least Squares methods. IEEE Journal of Robotics and Automation, 17:1–19, 2004. [37] Lukas Barinka and Roman Berka. Inverse Kinematics - Basic Methods. Czech Technical University, pages 1–10, 2002. [38] Paulo Costa and António Moreira. Lecture Notes - Robótica Industrial. Faculdade de Engenharia, Universidade do Porto, 2013. [39] Karim Abdel-Malek, Jingzhou Yang, Richard Brand, and Emad Tanbour. Towards understanding the workspace of human limbs. Ergonomics, 47(13):1386–1405, 2004. doi: 10.1080/00140130410001724255. [40] Prof Erika Ábrahám. Modeling and Analysis of Hybrid Systems Lecture Notes. RWTH Aachen University, 2012. [41] MZ Al-Faiz and AH Miry. Artificial Human Arm Driven by EMG Signal. INTECH Open Access Publisher, 2012. URL: http://cdn.intechopen.com/pdfs-wm/39325.pdf. 78 REFERENCES [42] Happy H. An, William I. Clement, and Benjamin Reed. Analytical inverse kinematic solution with self-motion constraint for the 7-DOF restore robot arm. 2014 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pages 1325–1330, July 2014. URL: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm? arnumber=6878266,doi:10.1109/AIM.2014.6878266. [43] Alejandro Hernandez Arieta and Hiroshi Yokoi. Study on the Effects of Electrical Stimulation on the Pattern Recognition for an EMG Prosthetic Application. (16360118):6919–6922, 2005. [44] P.K. Artemiadis and K.J. Kyriakopoulos. EMG-Based Control of a Robot Arm Using LowDimensional Embeddings. Robotics, IEEE Transactions on, 26(2):393 – 398, 2010. [45] Panagiotis K Artemiadis and Kostas J Kyriakopoulos. A switching regime model for the EMG-based control of a robot arm. IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society, 41(1):53– 63, February 2011. URL: http://www.ncbi.nlm.nih.gov/pubmed/20403787, doi:10.1109/TSMCB.2010.2045120. [46] Per Ferdinand Bach. Myoelectric signal features for upper limb prostheses. Institutt for teknisk kybernetikk, 2009. [47] Paul Bach-y Rita and Stephen W. Kercel. Sensory substitution and the human–machine interface. Trends in Cognitive Sciences, 7(12):541–546, December 2003. URL: http: //linkinghub.elsevier.com/retrieve/pii/S1364661303002900,doi:10. 1016/j.tics.2003.10.013. [48] Thomas Bak and Roozbeh Izadi-zamanabadi. Lecture Notes - Hybrid Systems. Aalborg University, 2004. [49] Amy Blank, Allison M Okamura, and Katherine J Kuchenbecker. Identifying the Role of Proprioception in Upper-Limb Prosthesis Control : Studies on Targeted Motion. ACM Transactions on Applied Perception (TAP), 7(3):15, 2010. [50] Amy Blank, Allison M Okamura, and Katherine J Kuchenbecker. Effects of Proprioceptive Motion Feedback on Sighted and Non-Sighted Control of a Virtual Hand Prosthesis. In Haptic interfaces for virtual environment and teleoperator systems, 2008. haptics 2008. symposium on, number March, pages 141 – 142. 2008. [51] Hanneke Bouwsema, Corry K van der Sluis, and Raoul M Bongers. Learning to control opening and closing a myoelectric hand. Archives of physical medicine and rehabilitation, 91(9):1442–6, September 2010. URL: http://www.ncbi.nlm.nih.gov/pubmed/ 20801265,doi:10.1016/j.apmr.2010.06.025. [52] M Chiara Carrozza. On the Shared Control of an EMG-Controlled Prosthetic Hand : Analysis of User – Prosthesis Interaction. IEEE Transactions on Robotics, 24(1):170–184, 2008. [53] Patrick E Crago, Ning Lan, Peter H Veltink, James J Abbas, and Carole Kantor. New control strategies for neuroprosthetic systems. Journal of Rehabilitation Research & Development, 33(2):158–172, 1996. REFERENCES 79 [54] Gurpreet Singh Dhillon and Kenneth W Horch. Direct neural sensory feedback and control of a prosthetic arm. IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society, 13(4):468–72, December 2005. URL: http://www.ncbi.nlm.nih.gov/pubmed/16425828,doi: 10.1109/TNSRE.2005.856072. [55] Erik J Dijkstra. Upper limb project. University of Twente, 2010. [56] D S Dorcas, V A Dunfield, R N Scott, and New Brunswick. Improved myo-electric control system. Medical and biological engineering, 8(July 1969):333–341, 1970. [57] Richard F ff. Weir, Phil R Troyk, Jack F Schorsch, and Huub Maas. Implantable Myoelectric Sensors (IMESs) for Intramuscular Electromyogram Recording. IEEE Trans Biomed Eng, 56(1):159–171, 2009. doi:10.1109/TBME.2008.2005942.Implantable. [58] C. Fleischer, C. Reinicke, and G. Hommel. Predicting the intended motion with EMG signals for an exoskeleton orthosis controller. 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2029–2034, 2005. URL: http://ieeexplore. ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1545504,doi:10.1109/ IROS.2005.1545504. [59] Anders Fougner, Oyvind Stavdahl, Peter J. Kyberd, Yves G. Losier, and Philip a. Parker. Control of upper limb prostheses: Terminology and proportional myoelectric controla review. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 20(5):663–677, 2012. [60] Amartya Ganguly. Modelling of bionic arm. Journal of Biomedical Science and Engineering, 03(03):327–329, 2010. URL: http://www.scirp.org/Journal/PaperDownload. aspx?paperID=1536&fileName=JBiSE20100300015_95591588.pdf,doi:10. 4236/jbise.2010.33045. [61] Angel Gil-agudo, Antonio Ama-espinosa, Ana De Los Reyes-guzmán, Alberto Bernalsahún, and Eduardo Rocón. Applications of Upper Limb Biomechanical Models in Spinal Cord Injury Patients. INTECH Open Access Publisher, 2011. [62] Diana Guimarães. Biped Locomotion Systems Analysis , Modeling and Control. Faculdade de Engenharia da Universidade do Porto, 2013. [63] Michael a. Henson. Nonlinear model predictive control: current status and future directions. Computers & Chemical Engineering, 23(2):187–202, December 1998. URL: http: //linkinghub.elsevier.com/retrieve/pii/S0098135498002609,doi:10. 1016/S0098-1354(98)00260-9. [64] Kenneth Horch, Sanford Meek, Tyson G Taylor, and Douglas T Hutchinson. Object Discrimination With an Artificial Hand Using Electrical Stimulation of Peripheral Tactile and Proprioceptive Pathways With Intrafascicular Electrodes. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 19(5):483–489, 2011. [65] P a Kaplanis, C S Pattichis, L J Hadjileontiadis, and V C Roberts. Surface EMG analysis on normal subjects based on isometric voluntary contraction. Journal of electromyography and kinesiology : official journal of the International Society of Electrophysiological Kinesiology, 19(1):157–71, February 2009. URL: http://www.ncbi.nlm.nih.gov/pubmed/ 17544702,doi:10.1016/j.jelekin.2007.03.010. 80 REFERENCES [66] H.I. Krebs, N. Hogan, W. Durfee, and H. Herr. Rehabilitation robotics, orthotics, and prosthetics. In Michael Selzer Gage, Stephanie Clarke, Leonardo Cohen, Pamela Duncan, and Fred, editors, Textbook of Neural Repair and Rehabilitation, pages 165–181. 2006. [67] Todd a Kuiken, Guanglin Li, Blair a Lock, Robert D Lipschutz, Laura a Miller, Kathy a Stubblefield, and Kevin B Englehart. Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. Jama, 301(6):619–28, February 2009. URL: http: //www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3036162& tool=pmcentrez&rendertype=abstract,doi:10.1001/jama.2009.116. [68] Frank L.Lewis, Darren M.Dawson, and Chaouki T.Abdallah. Robot Manipulator Control Theory and Practice. CRC Press, 2003. [69] Guey Lau. An Intelligent Prosthetic Hand using Hybrid Actuation and Myoelectric Control. The University of Leeds, 2009. [70] C M Light, P H Chappell, B Hudgins, and K Engelhart. Intelligent multifunction myoelectric control of hand prostheses. Journal of Medical Engineering & Technology, 26(4):139–146, 2002. doi:10.1080/0309190021014245. [71] Ren C. Luo, Tsung-Wei Lin, and Yun-Hsuan Tsai. Analytical inverse kinematic solution for modularized 7-DoF redundant manipulators with offsets at shoulder and wrist. 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, (Iros):516– 521, September 2014. URL: http://ieeexplore.ieee.org/lpdocs/epic03/ wrapper.htm?arnumber=6942608,doi:10.1109/IROS.2014.6942608. [72] D J Magermans, E K J Chadwick, H E J Veeger, and F C T van der Helm. Requirements for upper extremity motions during activities of daily living. Clinical biomechanics (Bristol, Avon), 20(6):591–9, July 2005. URL: http://www.ncbi.nlm.nih.gov/pubmed/ 15890439,doi:10.1016/j.clinbiomech.2005.02.006. [73] Paul D Marasco, Keehoon Kim, James Edward Colgate, Michael a Peshkin, and Todd a Kuiken. Robotic touch shifts perception of embodiment to a prosthesis in targeted reinnervation amputees. Brain : a journal of neurology, 134(Pt 3):747–58, March 2011. URL: http: //www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3044830& tool=pmcentrez&rendertype=abstract,doi:10.1093/brain/awq361. [74] Ebrahim Mattar. Advances in Robotics & Automation e _ GRASP : Robotic Hand Modeling and Simulation Environment. Adv Robot Autom, 2(2):1–11, 2013. doi:10.4172/ 2168-9695.10001. [75] Robert Peter Matthew, Gregorij Kurillo, Jay J Han, and Ruzena Bajcsy. Calculating Reachable Workspace Volume for use in Quantitative Medicine. In Carsten Agapito, Lourdes and Bronstein, Michael M. and Rother, editor, Computer Vision - ECCV 2014 Workshops, pages 570–583. 2015. [76] Nima Mohajerin. Identification and Predictive Control Using Recurrent Neural Networks. Orebro University, 2012. [77] Mohammadreza Asghari Oskoei and Huosheng Hu. Adaptive myoelectric humanmachine interface for video games. 2009 International Conference on Mechatronics and Automation, pages 1015–1020, August 2009. URL: http://ieeexplore.ieee. REFERENCES 81 org/lpdocs/epic03/wrapper.htm?arnumber=5246300,doi:10.1109/ICMA. 2009.5246300. [78] P. Parker, K. Englehart, and B. Hudgins. Myoelectric signal processing for control of powered limb prostheses. Journal of Electromyography and Kinesiology, 16(6):541–548, December 2006. URL: http://www.ncbi.nlm.nih.gov/pubmed/17045489,doi: 10.1016/j.jelekin.2006.08.006. [79] J. Norberto Pires. Robot Manipulators and Control Systems. In Industrial Robots Programming, pages 35–107. 2007. doi:10.1007/978-0-387-23326-0\_2. [80] Alvaro Rios Poveda. Myoelectric prostheses with sensorial feedback. University of New Brunswick, 2002. [81] M B I Raez, M S Hussain, and F Mohd-Yasin. Techniques of EMG signal analysis: detection, processing, classification and applications. Biological procedures online, 8(1):11–35, January 2006. URL: http://www.pubmedcentral.nih.gov/articlerender. fcgi?artid=1455479&tool=pmcentrez&rendertype=abstract,doi: 10.1251/bpo115. [82] Ranjeet Ranjan, Arbin Kumar, and Praveen Dhyani. Modeling and Simulation of Robotic Humanoid Arm, volume 4. 2012. [83] Rinku Roy, Amit Konar, D.N. Tibarewala, and R. Janarthanan. EEG driven model predictive position control of an artificial limb using neural net. In Computing Communication & Networking Technologies (ICCCNT), 2012 Third International Conference on, number July, pages 1 – 9. 2012. [84] Kathleen Talbot. Using Arduino to Design a Myoelectric Prosthetic. Saint John’s University, 2014. [85] Federico Thomas. Solved Problems in Robot Kinematics Using the Robotics Toolbox. Universitat Politècnica de Catalunya, 2012. [86] Deepak Tolani, Ambarish Goswami, and Norman I. Badler. Real-Time Inverse Kinematics Techniques for Anthropomorphic Limbs. Graphical Models, 62(5):353–388, September 2000. URL: http://linkinghub.elsevier.com/retrieve/pii/ S1524070300905289,doi:10.1006/gmod.2000.0528. [87] Quanzhao Tu, Xiafu Peng, Jiehua Zhou, and Xunyu Zhong. Kinematics Simulation and Analysis of 2-DOF Parallel Manipulator with Highly Redundant Actuation. pages 2–5, 2011. [88] Huiyu Zhou and Huosheng Hu. Human motion tracking for rehabilitation—A survey. Biomedical Signal Processing and Control, 3(1):1–18, January 2008. URL: http: //linkinghub.elsevier.com/retrieve/pii/S1746809407000778,doi:10. 1016/j.bspc.2007.09.001.