The potency of optical and augmented reality mirror boxes in amputees and people with intact limbs

Advances in Electrical and Electronic Engineering 310 THE POTENCY OF OPTICAL AND AUGMENTED REALITY MIRROR BOXES IN AMPUTEES AND PEOPLE WITH INTACT LIMBS K. O’Neill 1,2 , A. de Paor 1,2 1 School of Electrical, Electronic and Mechanical Engineering, National University of Ireland, Dublin (UCD),Belfield, Dublin 4, Ireland 2 Rehabilitation Engineering, National Rehabilitation Hospital, Dún Laoghaire, Co. Dublin e-mail:annrao[email protected]e Summary. Phantom limb pain is the distressing problem experienced by many amputees, defined as a painful sensation perceived in the area of the missing body part. Phantom limb pain can be very severe and disabling. It continues to be experienced by two thirds of amputees, eight years post-amputation. Augmented reality has the ability to change a person’s sensory experience. More applications of this technology are gradually being utilised for therapeutic purposes as augmented environments can be used both to distract the attention of patients from excruciatingly painful experiences and to promote cortical re-mapping at the site from where the pain arises. Using Augmented Reality, an environment has been created where upper limb amputees can both view and control motion of their phantom limb to help alleviate phantom limb pain. 1. INTRODUCTION The sensation of a phantom limb can be described as the feeling of the presence of an amputated extremity as still present. This phenomenon of a phantom limb has been recognised for well over a century [1,2]. Almost everyone who has a limb removed will experience a phantom limb, with most amputees having reported experiencing a phantom immediately after the loss of a limb [3]. Immediately after amputation, the phantom limb usually resembles the pre-amputation limb in shape, length and volume. The sensation can be described as very vivid and often includes feelings of posture and movement. In many cases, the phantom limb is present initially for only a few days or weeks, before it gradually fades away from consciousness. Once the phantom fades away it generally does so completely, yet some amputees can recall the phantom by rubbing the stump or by intense concentration [2]. However, the phantom limb can also persist for years and may never fade away. 30% of patients still have their phantom limb decades later [4] and case reports exist that detail phantom limbs that have persisted for 44 years [5]. Phantom limb pain occurs when a person feels pain that is attributed to the area of a limb that has been removed. Phantom limb pain can be severe and disabling and continues to be experienced by two thirds of amputees, eight years post amputation [6]. As many as 70% of phantoms still remain painful 25 years after the loss of the limb [7]. Phantom limb pain occurs in variable forms. In some amputees the pain is continuous but varying in intensity, while others experience intermittent pain of a high intensity [8]. Phantoms tend to be more vivid, and persist longer, after traumatic limb loss, or following amputation for a pre-existing painful limb pathology, than after a planned surgical amputation of a non-painful limb. It is often reported that a phantom limb will occupy a habitual posture, but spontaneous changes in position are also common. The phantom may also assume, either temporarily or permanently, an awkward and painful posture. Phantoms are more vivid and persist longer after traumatic limb loss, or following amputation for pre-existing painful limb pathology, than after a planned surgical amputation of a non-painful limb. This may be due to the greater attention paid to the mutilated or painful limb before it is lost. Memories of the limb’s posture and form prior to amputation often survive in the phantom [9]. In addition, after amputation of a deformed limb, the deformity is often carried over into the phantom [4,10]. Telescoping is a phenomenon that is common in approx. 50% of cases, especially in upper-limb amputees. This is where the distal part of the limb is gradually felt to approach the residual limb. In the case of an upper limb amputee, the phantom arm gets shorter over time until the patient is left with just a phantom hand attached to their stump [11,12]. 2. CORTICAL BRAIN REMAPPING AFTER AMPUTATION The study of phantom limbs also provides an opportunity to understand exactly how the brain constructs a body image, and how this image is continuously updated in response to changing sensory inputs. A complete somatotopic map of the body surface exists in the somatosensory cortex of primates [13,14], including humans [15] (See Figure 1). The somatosensory cortex localises the source of sensory input and perceives the level of intensity of the stimulus. It is capable of spatial discrimination, so it can discern shapes of objects being held or distinguish subtle differences in similar objects that come into contact with the skin. The potency of optical and augmented reality mirror boxes… 311 Fig. 1: Penfield Homunculus shows where the site for each limb is located on the somatosensory cortex of the brain It has been shown that cortical reorganisation occurs after amputation [16] and that there is a high positive correlation between the magnitude of phantom limb pain and the amount of cortical reorganisation [17]. Studies of arm amputation using magnetoencephalography have shown that brain areas, which ordinarily represent the hand, were activated when either the lower face or upper arm was touched [18,19]. Studies have shown that when the face of an amputee was stimulated by touch, sensory information went not only to the facial area of the cortex, but it also invaded the site that, prior to amputation, had represented the hand. The Penfield Homunculus can be reorganised over a distance of 2-3cm in the adult brain, following loss of a limb [20,21,22]. In this way, the brain circuitry can be altered. 3. MIRROR BOX ILLUSION V.S. Ramachandran developed a technique to treat phantom limb pain in upper limb amputees by using an ordinary mirror [23]. The box is made by placing a vertical mirror inside a cardboard box with the roof of the box removed. The front of the box has two holes in it, through which the subject inserts his/her good arm and the phantom arm. (See Figure 2) When viewed from slightly off-centre, the reflection of their “good” arm gave the impression that the subject had two intact arms. By sending motor commands to the intact arm to make symmetric movements, as if conducting an orchestra, it gives the impression that the phantom arm has resurrected and the subject receives positive visual feedback informing the brain that the phantom arm is moving in synchrony. This process has been dubbed the Mirror Box Illusion and is the main basis for this study. Fig. 2: Demonstrating the Mirror Box Illusion There is great variability in the experienced authenticity of the mirror box illusion and its ability to alleviate phantom limb pain. The effect of this positive visual feedback to the amputee can be very therapeutic. In some cases it was possible to reposition phantom limbs that were perceived to be held in painful or awkward positions, into nonpainful postures, giving temporary relief from phantom limb pain. Whilst this technique had quite dramatic therapeutic value for some people, it was only moderately effective, or completely ineffective for others [23]. 4. AUGMENTED REALITY MIRROR BOX The mirror box illusion had quite dramatic therapeutic value for some people, but it was only moderately effective, or completely ineffective for others. For those that found it to be beneficial, it was possibly because the illusory phantom provides feedback that is consistent with actual phantom experiences. However amputees’ perceptions of their phantom limbs often differ greatly from their original limbs [24]. The phantom limb can be shorter, or longer, vary in thickness, have gaps or be continuous, in comparison to the original limb. It is not unheard of for a phantom limb to be disconnected from the amputee’s stump (See Figure 3). This is one reason why the mirror box illusion does not have any therapeutic value in some cases, as it is not uncommon for the reflected image to bear no resemblance to the phantom limb perceived by the amputee. These irregularly shaped phantoms cannot be viewed in the mirror, as it necessarily reflects the image of the intact arm. Fig. 3: The Phantom limb of this Amputee is perceived to resemble a hand disjointed from the elbow stump [24] Advances in Electrical and Electronic Engineering 312 We have created an environment using Augmented Reality that simulates the mirror box illusion. It consists of a 3-d graphical representation of an arm on a flat screen that is controlled by a wireless data glove (5DT Data Glove 5W, Transmission Frequency 433.92MHz.). The appearance of the arm can be altered to resemble the phantom limb of the user. The data glove is worn on the intact arm, whilst the phantom appears on a flat computer screen that takes the place of the mirror. As the intact arm moves, the virtual arm on the screen representing the phantom also moves in unison. To create the virtual representation of an arm, we used the 3D Graphics animation package 3DS Max to deform a single mesh object. By a process referred to as “Skinning”, we added a bone structure to a deformable mesh. When a bone is assigned to a mesh, it is given a set of vertices. These vertices are weighted to determine how much influence each bone has on the mesh. We want our mesh to bend at the joints of the arm, but not in the middle of the forearm when there is motion in the bones (See Figure 4). Skinning is the most intricate part of the design process as it dictates how the mesh will deform under the influence of each bone. Fig. 4: We set the influence of the bones on the vertices in order to get the mesh to deform properly The main advantage of the Virtual Mirror is that all types of phantom limbs, such as Figure 3, can be graphically represented on the screen. In effect, phantom limbs can be tailored to visually resemble what the amputee perceives them to be. Another extra benefit that the Augmented Reality mirror has over the generic Mirror Box is that it can also be set up to give more general movements, for example, motion of the virtual image in the same direction as the intact arm, or having a third party control the motion of the image on the screen. 5. EXPERIMENTAL STUDY Four amputees and a control group of 22 ablebodied people participated in the study under conditions of informed consent and in accordance with ethical procedures. Amputee 1 is a 40 yr old male who underwent right side above elbow amputation necessitated by osteogenic sarcoma. After amputation, he had the sensation that his arm was intact and in its normal position. Gradually, over the period of six to seven months he experienced telescoping of his phantom. He now perceives his phantom hand to be attached to the end of the residual limb, the section of the phantom that originally joined the phantom hand to the residual limb having disappeared over time. The phantom hand is felt to be cramped in a claw-like position, which although discomforting is not painful. Amputee 2 is a 25 yr old male who sustained an amputation of the right forequarter and minor injury to his right leg when his motorbike collided with a car whose driver was intoxicated. His phantom arm is positioned across his body as though raised in a protective pose. Sensation is most vivid in the hand, which is held in a loosely clenched fist with the thumb on the outside. Although he does not feel the forearm and upper arm per se, he is aware of the position of the elbow. The phantom does not make involuntary movements and he cannot voluntarily generate movements in his phantom limb. Amputee 3 is a 64 yr old female who sustained injuries in a vehicle accident leading to the subsequent amputation of her right arm below the elbow. She experienced phantom sensation immediately after amputation surgery. She is not ordinarily aware of sensation in the missing forearm but is aware of her phantom hand. Her fingertips are felt to be touching her thumb, and although originally felt to be touching her thumb, her little finger is now felt to be raised. She does not experience her phantom fingers individually but rather as one unit. When painful, her phantom hand is felt as a clenched fist with the nails digging into the palm of the hand. Amputee 4 is a 42 yr old female who underwent right side above elbow amputation after a traffic accident. She experienced the sensation of a phantom limb immediately after amputation. Over time her phantom hand has telescoped, so that she now perceives it to reside at the end of her residual limb. When she experiences phantom pain, her phantom hand is felt as a clenched fist with the nails digging into the palm of the hand. For the control group, the subjects consisted of 8 males and 14 females, ranging in age from 20 to 29 (mean age = 24.91, standard deviation = 12.72). The participants were asked to draw a card from an unbiased deck to determine whether they would start with the standard mirror or the Augmented Reality version. This is to ensure that randomness was observed in the order tasks that were performed. (The procedure described below is based on a subject first performing the tasks with the standard mirror box, before repeating them with the Augmented Reality version. For those that selected The potency of optical and augmented reality mirror boxes… 313 to perform the tasks with the Augmented Reality mirror box, the order was simply reversed). Initially, each subject was introduced to the mirror box illusion and he/she tried to experience the sensation of a phantom limb by opening and closing their left hand in front of the mirror, as if they were conducting an orchestra and also to flex their fingers accordingly. The participants were then required to try the mirror box illusion for 3 periods of 30 seconds, during which they were instructed to repeatedly close their left hand into a fist, pause briefly before opening it out again in front of the mirror. Then they held their hand out straight and bent the third finger inwards before releasing it. This action was replicated by a randomly chosen finger (other than the third digit). This process was repeated for 3 periods of 30 seconds duration. Finally, the subjects were instructed to place their hand in front of the mirror flat on a table and to drum their fingers rhythmically against the flat surface. The participants were then fitted with the lefthanded 5DT data glove. They positioned their left arm in front of the flat screen and their right arm behind it. An image of a right arm appeared on the screen. The image used here was a realistic 3-d representation of a human arm. The subjects were instructed to keep their eyes fixed on the image on the screen right throughout the experiment. The same experimental procedures as before were carried out, with the flat screen taking the place of the mirror. Fig. 5: Subject using the Augmented Reality Mirror Box, whilst data is being recorded Finally, the subjects were asked to repeat the experiments again using the Augmented Reality mirror box. However, this time a cartoon-like image of an arm appeared on screen, different in proportion and shape to their intact arm. We measured the EEG signal from the site on the cortex corresponding to the right hand to determine whether any activity was occurring in this region. The mu rhythm was recorded over the sensorimotor cortex of the brain. The main electrode placements for this area are along the C line of the 10-20 system of electrode placement with C3 and C4 being used for recording of changes of mu rhythms associated with hand movements [25]. During initial mirror box experiments it was noted that the right hand of some of the subjects (held behind the screen/mirror) moved as the image on the screen did. We tried to track this motion by using two small light-sensitive accelerometers. These were taped to two separate thimbles, which were placed on the middle fingertip of each hand while the experiments were being performed. We also recorded an EMG signal from the right arm as in some cases the movement of the right hand was too slight to be observed by accelerometers. The EEG, EMG and accelerometer signals were only recorded from the control group, as the idea behind them only evolved after testing with the group of amputees. It is hoped to record these readings from the next group of amputees. These captured signals were recorded using a Biopac system and were fed into MATLAB via a PCI6023E data acquisition card from National Instruments. 6. RESULTS Amputee 1 found that his phantom initially became tense while doing the experiments. It produced a painful sensation in the digits that were moving, but a relaxed sensation in the parts of his phantom that were at rest. The Augmented Reality mirror had the same effect to a lesser degree and resulted in a less intensive phantom sensation than when the mirror was used. He reported that the tight sensation in his phantom began to diminish and become milder over time, but that his phantom sensation became much stronger in all of his fingers. Prior to testing Amputee 2 was unable to produce voluntary movement in the phantom. The provision of feedback, whether using a standard mirror or the augmented reality box, provided him with no change in the position of the phantom and did not alter his volitional control of his phantom limb. For Amputee 3 visual feedback both in the standard mirror and augmented mirror conditions intensified and extended phantom experience, though it was much more pronounced when using the augmented environment. When using the standard mirror, she reported that she could vividly feel the palm of her phantom hand, and that this sensation could be felt right up through her arm. Viewing the augmented reality phantom image facilitated movement of her phantom forefinger as an independent digit. Such movement had previously been impossible. Movement of the fingers of her phantom hand could be remotely generated. Amputee 4 experienced the sensation of her phantom hand moving using both mirror techniques, but found that it was more vivid when using the Augmented Reality mirror. She had never before experienced voluntary motion of her phantom limb. She was able to produce small motions of opening Advances in Electrical and Electronic Engineering 314 out her phantom hand that seemed to bring temporary relief from her phantom pain. After the members of the control group had completed the study, it was shown that all but 2 of the participants found that they could agree with the effect created by the standard mirror box illusion. They agreed that the sensation of visualising their reflection moving in unison with their controlling arm was quite realistic. When using the Augmented Reality Mirror, 19 of the 22 subjects found that the computer generated image responded and behaved in a convincing manner to that of their controlling arm wearing the glove (2 of those who were unable to experience a phantom sensation using the Augmented Mirror were the same subjects that found the standard mirror box illusion to have no effect). 16 subjects reported that they were content that the cartoon-like image behaved reasonably like a realistic one when using the Augmented Reality mirror, but many found it to be a lot less convincing than the image of a proper arm. Analysis of the accelerometer and EMG readings indicate that 12 of the subjects subconsciously moved their right arm while performing the tasks. We were able to detect an increase in the activity of the C3 site in most of the participants whenever motion occurred in the image. While we expected to observe this in the cases of people whose right arm moved behind the screen/mirror, it was interesting to note an increase in activity in some of those whose right arm remained static throughout the experiments. 7. DISCUSSION To date, we have unfortunately been unable to obtain a sufficient number of right-hand amputee volunteers for this study in order to carry out a proper quantitative analysis. At present, four amputees have tried both types of mirror box. Three of these found them to have a beneficially therapeutic effect on their Phantom Limb and assisted them in their attempt to alleviate Phantom Limb Pain. The other amputee found any attempt to incite movement in his phantom fruitless. Prior to experimentation, none of the amputees had been able to produce any large-scale voluntary movement of their phantom limbs. However, by using both mirror box techniques described here to give movement to their phantom limb, three of the amputees reported feeling a unique sensation that none of them had experienced before. Whilst these are preliminary results, they are nonetheless greatly encouraging, in that three of the four amputees tested the Augmented Reality mirror was able to provide some form of relief from phantom pain. The ultimate goal of this research group is to create an environment using AR technology that will allow amputees to visually experience their phantom limbs first hand. They should be able to see their phantom limbs as they are perceived, as well as have the ability to control their position, motion and orientation. By harnessing the potential of Augmented Reality technology, computer generated images of amputee’s experienced phantom limbs can be created in such a way that they move and look like their own phantom experience. This brings the therapeutic value of the visual illusion created by the mirror box to a wider range of amputees. It may then be possible that Augmented Reality has the potential to reduce or maybe remove phantom limb pain by the visual feedback received from the augmented environment. This may help reduce the influence of cortical remapping, reverse it, or maybe even prevent it from occurring. Acknowledgements The authors thank Dr. Deirdre Desmond & Prof. Malcolm MacLachlan Trinity College Dublin and Dr. Gary Mc Darby & his Group, formerly of Media Lab Europe for all their help. REFERENCES [1] S. Wier-Mitchell: Lippincott’s Magazine 8 (1871) 563. [2] S. Wier-Mitchell: Injuries of nerves, and their consequences. Philadelphia: J.B. Lippincott 1872 [3] V. S. Ramachandran and W. Hirstein: Brain 121 (1998) 1603 [4] S. Sunderland: Nerves and nerve injuries. 2 nd ed. Edinburgh: Churchill Livingstone 1978 [5] K. E. Livingston: Journal of Neurosurgery, 2 (1945) 25. [6]P. Gallagher and M. MacLachlan: Journal of Health Psychology 6 (2001) 85. [7] R. A. Sherman, C. J. Sherman and L.Parker: Pain 18 (1984) 83. [8] R. A. Sherman and C. J. Sherman: Am J Phys Med 62 (1983) 227. [9] J. Katz and R. Melzack: Pain 43 (1990) 319. [10] E. J. Browder and J. P. Gallagher: Ann Surg 128 (1948) 456. [11] S. A. Weiss and S. Fishman: Journal Abnormal Soc Psycol, 66 (1963) 489. [12] T. S. Jensen, B. Krebs, J. Nielson and P. Rasmussen: Pain, 17 (1983) 243. [13] J. H. Kaas, R. J. Nelson, M. Sur, C. S. Lin and M. M. Merzenich: Science 204 (1979) 521. [14]M. M. Merzenich, R. J. Nelson, M. P. Stryker, M. S. Cynader, A. Schoppmann A and J. M. Zook:J Comp Neurol 224 (1984) 591. [15] W. Penfield and T. L. Rasmussen: The potency of optical and augmented reality mirror boxes… 315 The cerebral cortex of man: a clinical study of localisation of function. New York: Macmillan; 1950 [16] H. Flor, T. Elbert, S. Knecht, C. Wienbruch, C. Pantev, N. Birbaumer, W. Larbig and E. Taub: Nature 357 (1995) 482. [17] N. Birbaumer, W. Lutzenberger, P. Montoya, W. Larbig, K. Unertl, S. Topfner, W. Grodd, E. Taub and H. Flor: Journal of Neuroscience 17 (1997) 5503. [18] V. S. Ramachandran, M. Stewart and D. Rogers-Ramachandran: Neuroreport 3 (1992) 583. [19] P. M. Halligan, L. C. Marshall, D. T. Wade, J. Davey and D. Morrison: Neuroreport 4 (1993) 233. [20] V. S. Ramachandran: Proc Natl Acad Sci USA 90 (1993) 10413. [21] T. T. Yang, C. Gallen, B. J. Schwartz, F. E. Bloom, V. S. Ramachandran and S. Cobb: Nature 368 (1994) 592. [22] T. T. Yang, C. Gallen, V. S. Ramachandran, S. Cobb, B. J. Schwartz and F. E. Bloom: Neuroreport 5 (1994) 701. [23] V. S. Ramachandran and D. RogersRamachandran: Proc R Soc Lond B Biol Sci 263 (1996) 377. [24] A. Wright: Wellcome Trust Sci Art Project (1997) [25] H. Jasper: Journal of Clinical Neurophysiology, 7(2) (1980) 191.