Sensory substitution for force feedback recovery: A perception experimental study

Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study ANGELICA I. AVILES-RIVERO, University of Cambridge, UK SAMAR M. ALSALEH, The George Washington University, USA JOHN PHILBECK, The George Washington University, USA STELLA P. RAVENTOS, Josep Trueta University Hospital, Spain NAJI YOUNES, The George Washington University, USA JAMES K. HAHN, The George Washington University, USA ALICIA CASALS, Universitat Politècnica de Catalunya, Spain Robotic Assisted Surgeries are commonly used nowadays as a more efficient alternative to traditional surgical options. Both surgeons and patients benefit from those systems as they offer many advantages, including less trauma and blood loss, fewer complications, and better ergonomics. However, a remaining limitation of currently available surgical systems is the lack of force feedback due to the teleoperation setting, which prevents direct interaction with the patient. Once the force information is obtained by either a sensing device or indirectly through vision-based force estimation, a concern arises on how to transmit this information to the surgeon. An attractive alternative is sensory substitution, which allows transcoding information from one sensory modality to present it in a different sensory modality. In the current work, we used visual feedback to convey interaction forces to the surgeon. Our overarching goal was to address the question How should interaction forces be displayed to support efficient comprehension by the surgeon, without interfering with the surgeon’s perception and workflow during surgery? Until now, the use the visual modality for force feedback has not been carefully evaluated. For this reason, we conducted an experimental study with two aims: (1) to demonstrate the potential benefits of using this modality and (2) to understand the surgeons’ perceptual preferences. The results derived from our study of 28 surgeons revealed a strong positive acceptance of the users (96%) using this modality. Moreover, we found that in order for surgeons to easily interpret the information, their mental model must be considered, meaning that, the design of the visualizations should fit the perceptual and cognitive abilities of the end user. To our knowledge, this is the first time that these principles are analyzed for exploring sensory substitution in medical robotics. Finally, we provide user-centered recommendations for the design of visual displays for robotic surgical systems. CCS Concepts: •Information Interfaces and Presentation →User Interfaces ;Evaluation/methodology; •Models and Principles →User/Machine Systems;Human Information Processing; Additional Key Words and Phrases: Robotic teleoperation, flow visualization, visualization ACM Reference Format: Angelica I. Aviles-Rivero, Samar M. Alsaleh, John Philbeck, Stella P. Raventos, Naji Younes, James K. Hahn, and Alicia Casals. 2017. Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study. ACM Transactions on Applied Perception 1, 1 (December 2017), 19 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn Authors’ addresses: A.I. Aviles-Rivero [email protected] ; S.M. Alsaleh [email protected] ; J. Philbeck [email protected] ; S.P. Raventos mpie. [email protected]; N. Younes [email protected]; J.K. Hahn [email protected]; A. Casals [email protected]. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor, or affiliate of the United States government. As such, the United States government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for government purposes only. ©2017 Association for Computing Machinery. XXXX-XXXX/2017/12-ART $15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. The final publication is available at ACM via http://dx.doi.org/10.1145/3176642 :2 •A. Aviles-Rivero et al. SLAVE Console TELEOPERATION Robotic arms Camera (endoscope) Instruments Surgeon Patient MASTER rotation Body wall Target tissue Fulcrum t1 Zoom-in view Zoom-in views Foot Pedals 3D environement Finger controls Instruments degrees Fig. 1. A typical teleoperated robotic surgical system using a master-slave configuration. At the master side, surgeon is provided with a 3D patient view and is able to perform the procedure using finger controls and foot pedals. All surgeon’s actions are reproduced by the slave which holds the surgical instruments. 1 INTRODUCTION Technological advancements are revolutionizing the field of medicine, by, for example, creating and integrating robotic devices in clinical scenarios such as diagnosis and surgery. In particular, Robotic-Assisted Surgical Systems (RASS) have emerged to tackle the deficiencies associated with traditional open and minimally invasive surgeries [ 74 ]. RASS offer distinct advantages for both patients and surgeons. For patients, RASS minimize intra-operative invasiveness, tissue trauma, blood loss, and complication rate, while also reducing post-operative infection risk, pain, scarring, and recovery time. RASS also provide better ergonomics for the surgeons by helping them maintain dexterity, and extending their surgical capabilities by offering optimal hand-eye alignment, motion scaling, and tremor filtering [33, 67]. A robotic surgical system attempts to reproduce the surgeon’s motion in a master/slave teleoperated setting. Fig. 1 shows the architecture of one such system. At the master side, the operating surgeon is immersed into a three dimensional environment in which additional useful information can be added to improve transparency in the teleoperated system [ 49 ]. Nonetheless, the physical separation between the operating surgeon and the instruments in the operating field leads to complete deprivation of force feedback during surgery. Even though surgeons are capable of performing procedures without force feedback, the medical robotic community still considers the lack of force feedback a major limitation in currently-available surgical systems. This is because it has been demonstrated that the lack of this feature causes: (1) an increase in errors during the procedure (see e.g. [ 40 , 60 , 72 ]), (2) an increase in mental workload, which can complicate the task at hand and lead to irreversible damages (see e.g. [ 36 , 70 ]), (3) an increase in task-completion time (see e.g. [ 48 , 70 , 72 ]) and (4) a decrease in surgeon-patient transparency [ 49 ]. Moreover, depending on the surgeon’s level of expertise and other situational factors, the lack of force feedback can have negative consequences for the surgeon’s mental workload and add further complication to the task in hand. This limitation is reputed to be one of the causes that restrict further spread of medical robotics [15]. As it is still an open problem in surgical robotics, researchers have attempted to acquire force information using direct sensing devices. Some of these sensing devices were designed to be placed at the outside of the patient as in [ 16 , 51 ]. However, force measurement with such a setup is not specific to the tool-tissue interaction region, but is comprised of different forces from the body wall, friction, and the instrument itself. Another placement option for the device is the instrument tip, as in [ 54 , 75 ], in which the sensor would pass through the insertion port inside the body for a more accurate force measurement. However, this placement enforces strict miniaturization ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study •:3 Master Slave Master Slave Master Slave Microsurgery Master Needle-based Procedures Surgical knot tying Palpation and Exploration Slave Workspace Slave Master Slave Master Slave Fig. 2. Surgical tasks where knowing the applied force is relevant and help to decrease the procedure completion time and to avoid injuries. constraints on the design of the device. Furthermore, any sensing instrument has to satisfy a strict list of medical regulations and restrictions including sterilization, biocompatibility, stability, and robustness [22, 65]. Due to the aforementioned issues, direct force sensing devices have not yet been integrated into current robotic surgical systems. An alternative group of force measuring solutions emerged to overcome these limitations by estimating interaction forces using visual information. The idea behind what is called Vision-Based Force Estimation (VBFE) comes from the conservation principles of continuum mechanics, which point out that the change in shape of an elastic object is directly proportional to the force applied. These kinds of solutions depend on visual information, such as the deformation of tissue under load, to estimate the applied forces. Several authors have demonstrated the benefits of VBFE, for example [3, 20, 29, 45]. Whether it is direct or estimated forces that are available, a natural question arises: how to provide this information to the surgeon? Studies show that force feedback information enables surgeons to have better control and precision when manipulating tissue [ 18 , 27 ]. Moreover, force feedback is especially relevant in the performance of many surgical tasks. For example, in many situations, surgeons perform exploration and palpation tasks in order to identify abnormal or cancerous tissue regions. Having force feedback is helpful in these situations as it enables surgeons to sense tissue mechanical properties and identify specific tissue features that are hard to identify visually. Other surgical tasks involve tissue manipulation, such as dissection and suturing, in which force feedback is important to prevent puncturing the tissue or breaking sutures due to the application of large forces. Fig.2 shows an illustration of these common surgical tasks. One solution for implementing force feedback would be to transmit the force information to the surgeon’s hands using a haptic master device. However, there are many concerns associated with this option, including cost, stability of the controller, degrees of freedom, and space limitations [ 15 , 24 , 47 ]. Moreover, when the gains size is too large, it can result in fatigue for the surgeon, which could consequently affect his/her performance [38]. An attractive alternative to direct force feedback to the surgeon’s hands is sensory substitution [ 4 ], in which one sense, the sense of touch in this case, is replaced by another sensory modality, vision or audition for example, to convey the lost information indirectly. This option is inspired by theories of perception and mechanisms of brain plasticity, which posit that the brain’s complexity allows it to restore certain functions using input from other stimuli or sensory modalities [ 44 , 69 ]. These theories formed the basis for a variety of studies and conceptual developments involving sensory substitution in teleoperation settings [5, 43, 55, 57]. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. :4 •A. Aviles-Rivero et al. 1.1 Sensory Substitution in Teleoperation The term sensory substitution refers to the ability of the central nervous system to learn a new mode of perception and it has been successfully used for many years to develop sensory aids for people with full or partial deficiency in one or more of their sensory systems [ 35 ]. In engineering, this term has come to have a much more general definition than was originally described by Bach-y-Rita [ 5 ], and now means simply transcoding information from one sensory modality to present it in a different sensory modality. This is the way we will use the term in the current paper. Since direct force feedback has not yet been integrated into current commercial surgical robotic systems, much of the existing research has investigated using the tactile sensing modality to convey to the surgeon a representation of the forces applied by the robotic tele-manipulators. This offers a significantly more practical solution in RASS settings as it can be easily integrated into existing consoles, is less expensive to implement, and is more stable and manageable/controllable than direct force feedback [ 47 ]. Furthermore, sensory substitution can be very effective in training surgeons to use RASS and can compensate for the lack of haptic feedback by the robotic system. Several studies presented and evaluated different sensory substitution options to transmit forces and tissue properties information. The most commonly used sensory modalities for feedback in this context can be classified into two groups: (i) monomodality including tactile, auditory or vision and (ii) multimodality which refers to the combination of two or more sensory modalities. 1.1.1 Single Sensory Modality (Monomodality). Starting with tactile feedback, early investigations noted that the fingertips contain sensitive sensory receptors that project to relatively large areas in the sensory cortex for information processing, making the vibrotactile modality a good option for presenting force feedback information [ 39 ]. The potential benefits of vibrotactile sensory substitution for force feedback were first explored in the work of Massimino and Sheridan, in which they tested the use of tactile and auditory senses to convey forces in teleoperation tasks. In that work, force was scaled to a vibration stimulus presented to the index finger and thumb, and the subjects were required to react as quickly as possible once they recognized the presence of a contact force. The results showed that the operators reacted faster with the vibrotactile feedback than when working with no such feedback. Researchers in a different study designed a simulated tissue probing task to measure the effect of vibrotactile feedback on surgeons’ performance. They measured the effect of this feedback on three main aspects: control of force application, tissue material differentiation, and task completion time [ 61 ]. The results showed that vibrotactile feedback allowed subjects to perform better, reducing the depth error and maximum force applied, and achieving more consistency compared to when no vibrotactile feedback was available. Similar results were reported in a more recent study in which authors tested the value of adding vibration feedback to the surgical setup during robotic surgery. The study illustrated that vibration feedback increased the level of awareness about tool contacts and demonstrated a strong user preference for this technology [ 31 ]. Despite these benefits, vibrotactile feedback is limited in the amount of information it provides as it is difficult to convey both force direction and magnitude at the same time with vibration. Other drawbacks of vibrotactile feedback start to appear when it is used for long periods of time, as the devices become uncomfortable and the skin starts to lose its sensitivity to the vibration stimuli [6, 31, 46]. Another form of feedback uses the auditory modality, which has been shown to improve task performance in many teleoperation settings. Authors in [ 30 ] studied the effect of sensory substitution on suture-manipulation forces. One feedback scenario studied by these authors involved auditory feedback, in which a single tone was provided to the operating surgeon when the applied force reached a specified ideal value. Even though the audio cues did not differentiate forces applied by the left or right-hand instrument, they still improved the consistency of the robotically applied forces. However, surgeons who participated in that study preferred having a continuous/real-time feedback over a discrete/single event information. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study •:5 This was examined in a different study in which authors presented force feedback as an auditory signal to both ears, with tone loudness being proportional to the magnitude of the force [ 39 ]. The results revealed that the reaction speed for recognizing the presence of a contact force was quickest for auditory feedback compared to vibrotactile and traditional force feedback. Even though some studies have shown that continuous frequency-modulated audio feedback is easier to interpret by surgeons, there were still concerns about continual auditory signals being disruptive and confusing in the operating room, as it is already noisy with different sounds coming from medical instruments and verbal communication [ 70 ]. Additionally, continuous sounds during long procedures can be a source of discomfort and/or annoyance to the surgeon and might distract communication between assistants and the surgeon [53]. Early investigations showed the feasibility of the visual modality, sight-to-touch, for sensory substitution during delicate surgical tasks. In the work of Bethea et al. [ 7 ], surgeons were instructed to perform a robot assisted knot tying task with and without the aid of a color bar sensory substitution. A visual color bar scale was used to convey the mean tension applied to the suture; the bar changed dynamically as the tension increased. The authors found that visual sensory substitution allowed surgeons to have more consistent, precise, and greater control over the tension applied to the fine suture material without breakage. Visual feedback was also compared against other sensory substitution alternatives in [ 30 ], in which the authors presented visual feedback in the form of two bars, one for each hand, in the upper right corner of the display, with the height and color of the bars changing according to the measured force. Out of the different sensory substitution options, visual feedback appeared to enhance most the consistency of applied forces and was superior to the other alternatives. A real-time visual force feedback graphic overlay was presented in [ 56 ] during the performance of delicate repetitious robotic manipulation of fine sutures. The graphic overlay in that experiment consisted of two semi-transparent circles superimposed over the corresponding moving instrument tips, which color changed in relation to the force magnitude. Subjects reported a preference for the use of visual feedback as it helped them avoid applying excessive forces and gave them more control over the task. In a more recent work [ 41 ], the authors studied sensory subtraction, which substitutes haptic force with cutaneous stimuli using fingertip skin deformation devices, and compared this method against several other sensory substitution modalities. They reported favorable performance of sensory subtraction and also noted the potential of visual and auditory modalities in medical robotic systems. 1.1.2 Multiple Sensory Modality (Multimodality). Apart from the use of a single sensory modality, multimodal feedback has also been reported in the literature. In [ 10 ], authors conducted a meta-analysis to compare the effects of visual-auditory and visual-tactile feedback against the use of visual feedback alone. They reported that multimodal feedback helped improve reaction time, but was not effective in decreasing error rates. The influence of multimodal feedback was also explored in [ 12 ], in which the authors suggested that using a combination of modalities can improve realism between the user and the environment, leading to better task performance. In more recent work [ 68 ], authors performed a study of all possible combinations between visual, auditory and tactile feedback. They stated that there is no significant difference among them. However, when the visual modality was combined with another modality, users expressed a preference for these combinations, and performance improved when such a combination was offered. The improved performance of multimodal feedback was further supported by Wickens’ multiple resource model [ 73 ], which states that human task performance can improve when increased sensory resources are available. However, past work has not fully taken into account the impact of long-duration tasks or the constraints of real surgical task environments. The use of multimodal feedback can be affected in real clinical environments by the attentional capacity of humans, which constrains the amount of information that can be effectively processed [ 32 , 58]. Information loss due to selective attention could lead to increase error during the procedure [8, 13]. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. :6 •A. Aviles-Rivero et al. 1.2 Aim of the Work Of all abovementioned modalities, we use visual feedback in this work for the following reasons: • Surgeons who operate the robotic systems primarily rely on their visual system to view and control the remote task via the console monitor. This makes the visual modality a particularly promising sensory substitution option for clinical adoption. Dividing attention between the visual modality and feedback in some other sensory modality could add to the burden on the operator. • The visual modality allows transmitting continuous spatiotemporal information of the environment over longer periods of time, while minimizing the interference to which the auditory modality is susceptible, and without the reduction in sensitivity that can occur when the tactile modality is used [6, 31, 46, 53]. When vision is used, visualizations representing the information should be integrated in the three dimensional environment displayed to the surgeon. Effective visualization of this information is essential to avoid increasing the surgeon’s cognitive workload, something that could otherwise cause fatigue, tissue damage, or increases in the procedure time. Although the visual modality is a feasible and promising option, there still exists the perceptual and cognitive burden of transcoding visual information into the force domain. When attempting to address this issue and develop an effective representation of the information that can be quickly interpreted, several questions naturally arise: Do all users interpret the different visual representations in the same way? How do users perceive these visualizations? Are they understood correctly? This leads us to formulate a particular question: − How should interaction forces be displayed to support efficient comprehension by the surgeon, without interfering with the surgeon’s perception and workflow during surgery? In this paper and the clinical user study it describes, we offer an extensive discussion of the aforementioned questions with the aim of reporting our findings and recommendations on the best options to display force information in an efficient way based on the surgeons’ preferences . We achieve this by testing four visualization systems that use different ways to encode force information using force-to-color mappings and are further explained in the following section. To the best of our knowledge, there are no works that address this issue or analyze how to efficiently represent the information for RASS. In the remainder of this paper, our perceptual experimental study is structured as follows: Section 2 describes all relevant details about how our clinical study was conducted. In Section 3 we report our findings using statistics and graphical methods, as well as a description of the results from a perceptual and cognitive point of view. Finally, we present the conclusions of the work in Section 4. 2 PERCEPTUAL STUDY This section describes in detail aspects that are particularly relevant for this study. 2.1 Subjects Description Twenty eight surgeons, on a voluntary basis, participated in the study. The participants came from four specialties: Obstetrics and gynecology (OB/GYN), Neurosurgery (NS), Pediatric surgery (PDS) and Cardiovascular surgery (CS). This population was divided into two main groups: experts and novices. What defines participants as experts or novices? This has been a central question in psychology, since we rely on experts to make decisions that affect our environment almost every day. Examples of works that address this question can be seen in [ 11 , 17 , 63 ]. Distinguishing experts from novices depends heavily on individual psychological differences and behavioral characteristics and varies according to the area of study [63]. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study •:7 Risk- Free Minimal Risk Potential Risk Risk No Force None No Force Minimal Risk Risk Microsurgery CIRCLE No Force Risk-Free Minimal-Risk Potential-Risk Color CodeVisualizations Samples BAR Risk- Free Minimal Risk Potential Risk Risk No Force No Force Risk-Free Minimal-Risk Potential-Risk HEAT MAP TRAFFIC LIGHT Warning! Risk- Free Minimal Risk Potential Risk Risk No Force No Force Risk-Free Potential-Risk Minimal-Risk No Force Risk-Free Minimal-Risk Risk Risk- Free Potential Risk Fig. 3. (From left to right) The four visualizations used in our experiments at different time instants. The coding-color used to indicate the level of risk to the surgeon according to the magnitude of the force applied. In the medical domain the distinction between experts and novices can be determined based on the number of hours in which surgeons can practice to improve skills such as reduction of task completion time, movement accuracy, and identifying and solving errors [23, 28]. Based on this, we defined the two groups as: • Experts: surgeons who perform more than 20 robotic-assisted surgeries, minimally invasive procedures and non-invasive procedures each per month. • Novices: surgeons who perform more than 20 minimally invasive procedures and non-invasive procedures each per month but no robotic-assisted surgeries. As mentioned before, defining experts by the number of surgeries is not trivial and also depends on the speciality. However in this work, we selected our threshold of 20 surgeries based on works such as [ 50 , 66 ]. With these criteria, we worked with 19 novices and nine experts. All the analyses of the remaining sections are taken from these two subgroups. 2.2 Visualizations Description Information is essential to understand our environment and its correct visualization determines our level of interpretation. Particularly in human-machine interaction, visual displays offer the highest bandwidth channel since our visual system is capable of acquiring more information than all other senses. Therefore, having a good representation of the information is crucial for supporting rapid interpretation of what is happening in the environment and effective decision-making. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. :8 •A. Aviles-Rivero et al. 2.2.1 Data and Task Description. Participants viewed videos taken from an in-vivo porcine dataset from the Hamlyn Center Laparoscopic / Endoscopic Video Library [ 42 ]. The video sequence was composed of stereo-pair images of size 720x288 recorded during 450 sec, in which tissue deformation was repeatedly exhibited due to the tool-tissue interaction. The dataset was acquired doing palpation on the tissue, varying factors such as illumination and position and orientation of the tool, as well as varying the force of the palpation. Palpation is clinically relevant since it is used to identify tumors, cut tissues and avoid penetration in the tissue. These videos included four different kinds of force visualizations, described below. After watching these videos, participants filled out a questionnaire (see Table II). As in any Robotic-assisted surgical system, which inherently lose all patient-surgeon interaction forces, participating surgeons were provided with this internal view of the surgical region of interest. But how to estimate how much force is applied? 2.2.2 Estimating the Interaction Forces. The conservation principles of continuum mechanics specify that any change in shape of an elastic object is directly proportional to the applied force. Using these principles, we estimated online the forces applied in the dataset based on the observable deformation. Roughly speaking, in our previous work [ 2 , 3 ] we proposed an energy functional based on L2 in which the minimization of the residual error was changed by a maximum likelihood type estimator. Also, we guarantee the anatomy preservation by proposing a topology regularizer. Once deformation was computed, we used a learning system to find either the nonlinear relationship between deformation of different tissues and force, or a mixture elements model (by maximizing a likelihood function), to assign the value to a particular color label. This estimation was used as the force feedback information displayed to the users. Because the applied forces were estimated rather than directly measured, the feedback information likely did not exactly reflect the true forces, but our method provided a good approximation for current purposes. This approximation could be refined in future work by incorporating other means of modeling the applied force. 2.2.3 Visualizations Design. For displaying the force information as previously explained, we designed four different visualizations (described below) which are labeled based on the indicator used: Circle, Bar, Heat Map and Traffic light. Each of these visualizations has been used in past works. Our goal here was to compare surgeons’ relative preferences for these relatively common visualization types and use their judgments to develop an evidence-based set of best practices for using visualizations to convey force feedback information in future work. Illustration of these visualizations can be seen in Fig. 3. Circle This visualization provides force feedback information by means of a dynamic circle that tracks the tool tip. The circle fluctuates between four different color indicators corresponding to the magnitude of force. When no force is applied, there is no circle embedded in the environment. This representation has been used previously in [1, 56]. Bar The force is represented by a dynamic bar at the top-right corner of the display. The bar fluctuates between five states representing the intensity of applied force. In contrast with the previous visualization, this option presents stacked states, that is, past states remain displayed during the current state. This representation has been used previously in [30, 62]. Heat map A heat map is shown at the top-right corner of the view. In this visualization, force is denoted by the level of deformation that the tissue undergoes. The level of risk is represented by the color intensity where darker shades correspond to larger forces (risk). This representation has been used previously in [ 26 ]. Traffic light For this option, a traffic light type visualization is displayed at the top-right corner of the environment. It fluctuates between four color indicators illustrating the magnitude of the applied force. The traffic light also shows a void state (colorless) which indicates no force. This representation has been used previously in [21]. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study •:9 Table 1. Color-coding used at each visualization to indicate the level of risk depending on the force applied Indicator Coding-color Meaning Circle (System A) Bar (System B) Traffic light (System D) Green Risk-free −represents minimal interaction forces Yellow Minimal Risk −symbolizes a safe amount of force Orange Potential Risk −references a potential damage in the tissue Red Risk −warning the physician of a tissue damage Heat map (System C) Yellow No force −There are no interaction forces Red Minimal Risk −represents a safe amount of force Black Risk −denotes tissue damage These visualizations were selected based not only on technical aspects, where we took into consideration the limited bounded domain of the display, but as well psychological ones, where we used typical color association and common cues like dynamic bars and traffic lights. The use of bar or circles is well-established in medical robotics experiments [ 1 , 21 , 30 , 56 , 62 ]. The selection of appropriate colors is important for adequate transfer of information. We followed a color-coding based on the perceptual phenomena related to colors. Based on the functional and sensory-social meaning of colors [ 25 , 64 ], we used red for example to convey warning messages and green to indicate a small magnitude of force. Details of the color-coding we used in relation to the amount of applied force can be seen in Table 1. During the experimental procedure, we assigned letters to the visualizations as described in Table I. This assignment is the one used during the user study. 2.3 Experimental Procedure After explaining the problem to the participating surgeons and giving the required instructions, they were provided with the four visualizations, those explained in subsection 2.2, each with a corresponding computerbased questionnaire. The instructions given were as follows: “We will display four different visualizations embedded in the robotic surgical system environment and labeled as System A, System B, System C and System D. Please interact with each visualization mode and respond to the corresponding questionnaire. At any time, you can return and interact again with any system and change any response in the questionnaire". The interaction was a passive one, in which users watched the prerecorded video demonstrating tool-use interaction, with one of the four visualization systems being overlaid on the video to convey information about the force applied in the video. 2.3.1 Visualizations Evaluation. Questionnaires are thought to be useful instruments to assess, for example, the usability and reliability of human-machine interfaces [ 52 , 59 ]. For our study, we designed questionnaire items using a five-point Likert rating scale in which participants were asked to indicate the level of agreement with the given statements, ranging from Strongly disagree to Strongly agree. The questionnaire was composed of twenty-four questions, shown in Table 2, that evaluated five human factors that are relevant in the context of human-machine interfaces. These factors are not derived from the data (as might be the case in a Principal Components Analysis), but instead are common conceptual groupings of factors used in assessing usability and reliability in human-machine interfaces. These factors are thought to provide important insight into the perceptional flow of the end user [34, 37]. The factors are: (1) Perceived Usefulness − Refers to the extent to which each participant believes that using each one of these systems will improve his/her surgical performance. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. :16 •A. Aviles-Rivero et al. • If the visual cue is big, such as the visualizations shown in Fig. 8 System B and D, avoid placing it on the surgical tool. This is because it could cause distraction and might not fit on the tool at all times when the tool is partially visible (i.e. only part of the tool is in the current field of view). • Use a color-coding that is compatible with the mental model of the surgeon. A simple but good example is using green-yellow-red. A relatively simple way to augment this recommendation to accommodate users having color vision deficiencies would be to provide the option of using an alterantive color-coding scheme that avoids using colors that are typically confused. •Use a simple but efficient geometric shapes (for example see Fig. 8 Systems A, B and C). • Do not overburden the display. You can include text but only in cases where it is needed, such as in a dangerous situation (e.g. see Fig. 8 Systems D). • Offer visual cues that represent the information with more than one cue, such as position and color (e.g. see Fig. 8 Systems B and D). 4 CONCLUSION The absence of force feedback in robotic surgical systems continues to be one its major limitations and is one of the reasons why surgeons need to go through extensive training to accommodate the indirect interaction. Having interaction forces information is of huge importance since it is directly related to the reduction of complexity of the surgical task in hand. This information has also been shown to increase the transparency between the operating surgeon and the patient as it gives the sensation of direct interaction. Although the current literature in medical robotics is quite large, the topic of designing a proper visual display of force feedback has not yet been sufficiently discussed. This is a very important aspect since having an effective visualization of force information has direct repercussions on the surgeons’ performance, particularly when it takes into account the perceptual and cognitive principles that are relevant for the surgeon. The main goal of this work was twofold. First, to carefully assess the use of visual cues to transmit information about interaction forces information, and second, to offer recommendations for proper design of visual displays based on the surgeons’ preference. To achieve these two goals, we conducted a clinical study to demonstrate the potential benefits of using visual feedback taking into account the opinion and preference of the end users, i.e. the operating surgeons. Out of the entire population, 96%of participating surgeons preferred having the visual feedback over none. Going back to the questions we posed in subsection 1.1, we found that in order to present the force information in a way that can be easily interpreted, we have to take into account the surgeon’s mental model. That is, the visual cues used to convey the force information should mesh well with the perceptional and cognitive abilities of the end user. In this work, we conducted an initial study focusing on the perception of the operating surgeons. Future work will include a more extensive evaluation to test the clinical potential of our findings. This future work will be based on a long-term follow-up study with the aim of evaluating a variety of other factors, such as: learning curves of performing surgery with visual feedback, adaptability time, the impact of visual feedback in terms of improving surgery procedures, and safety. A parallel study will be performed, involving combinations of the vibrotactile and auditory modalities with other sensory modalities. Taken together, the aim of these future studies is to provide an evidence-based foundation for evaluating the potentials and advantages of using visual feedback in clinical practice. ACKNOWLEDGMENTS This work was supported by a FPU national scholarship from the Spanish Ministry of Education with reference AP2012-1943. Support from the Centre for Mathematical Imaging in Healthcare (CMIH), University of Cambridge is greatly acknowledged. We would like to thank to the surgical departments of Obstetrics and gynecology, Pediatrics, Cardiology, Cardiovascular Medicine, and Abdominal and General Surgery from The Josep Trueta ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study •:17 University Hospital in Girona, Spain, and the surgical departments of Pediatrics, Obstetrics and gynecology and Pediatric Cardiology from The Vall d’Hebron University Hospital in Barcelona, Spain. Also, the surgical department of Obstetrics and gynecology from The Sant Joan de Deu Hospital in Manresa, Spain. REFERENCES [1] Takintope Akinbiyi, Carol E Reiley, Sunipa Saha, Darius Burschka, Christopher J Hasser, David D Yuh, and Allison M Okamura. 2006. Dynamic augmented reality for sensory substitution in robot-assisted surgical systems. In Engineering in Medicine and Biology Society, 2006. EMBS’06. 28th Annual International Conference of the IEEE. 567–570. [2] Angelica I. Aviles, Samar M. Alsaleh, James K. Hahn, and Alicia Casals. 2016. Towards Retrieving Force Feedback in Robotic-Assisted Surgery: A Supervised Neuro-Recurrent-Vision Approach. IEEE Transactions on Haptics (2016). [3] Angelica I. Aviles, Samar M. Alsaleh, Pilar Sobrevilla, and Alicia Casals. 2015. Force-Feedback Sensory Substitution using Supervised Recurrent Learning for Robotic-Assisted Surgery. Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (2015). [4] Paul Bach-y-Rita, Carter C. Collins, Frank Saunders, Benjamin White, and Lawrence Scadden. 1969. Vision substitution by tactile the image projection. Nature (1969), 963âĂŞ–964. [5] Paul Bach −yRita and Stephen W. Kerce. 1972. Brain Mechanisms in Sensory Substitution. Academic Press (1972). [6] Sliman BensmaÃŕa, Yuk-Yuen Leung, Steven S Hsiao, and Kenneth O Johnson. 2005. Vibratory adaptation of cutaneous mechanoreceptive afferents. Journal of neurophysiology 94, 5 (2005), 3023–âĂŤ3036. [7] Brian T Bethea, Allison M Okamura, Masaya Kitagawa, Torin P Fitton, Stephen M Cattaneo, Vincent L Gott, William A Baumgartner, and David D Yuh. 2004. Application of haptic feedback to robotic surgery. Journal of Laparoendoscopic & Advanced Surgical Techniques 14, 3 (2004), 191–195. [8] Donald Eric Broadbent. 2013. Perception and communication. Elsevier. [9] Wolfgang Broll, Irma Lindt, Jan Ohlenburg, Iris Herbst, Michael Wittkamper, and Thomas Novotny. 2005. An infrastructure for realizing custom-tailored augmented reality user interfaces. IEEE transactions on visualization and computer graphics 11, 6 (2005), 722–733. [10] Jennifer L. Burke, Matthew S. Prewett, Ashley A. Gray, Liuquin Yang, Frederick R. B. Stilson, Michael D. Coovert, Linda R. Elliot, and Elizabeth Redden. 2006. Comparing the Effects of Visual-auditory and Visual-tactile Feedback on User Performance: A Meta-analysis. Proceedings of the 8th International Conference on Multimodal Interfaces (2006), 108–117. [11] L. Cuthbert, B. Duboulay, D. Teather, B. Teather, M. Sharples, and G. Duboulay. 1999. Expert/Novice differences in Diagnostic Medical Cognition- A Review of the Literature. (1999). [12] Iñaki Díaz, Josune Hernantes, Ignacio Mansa, Alberto Lozano, Diego Borro, Jorge Juan Gil, and Emilio Sánchez. 2006. Influence of multisensory feedback on haptic accessibility tasks. Virtual Reality 10, 1 (2006), 31–40. [13] Jon Driver. 2001. A selective review of selective attention research from the past century. British Journal of Psychology 92, 1 (2001), 53–78. [14] Andreas Dünser, Raphaël Grasset, Hartmut Seichter, and Mark Billinghurst. 2007. Applying HCI principles to AR systems design. (2007). [15] Nima Enayati, Elena De Momi, and Giancarlo Ferrigno. 2016. Haptics in Robot-Assisted Surgery: Challenges and Benefits. IEEE Reviews in Biomedical Engineering 9 (2016). [16] Angela Faragasso, Joao Bimbo, Yohan Noh, Sina Sareh, Hongbin Liu, Thrishantha Nanayakkara, Helge Wurdemann, and Kaspar Althoefer. 2014. Novel Uniaxial Force Sensor based on Visual Information for Minimally Invasive Surgery. IEEE International Conference on Robotics and Automation (2014), 2934–2939. [17] Wendy Faulkner, James Fleck, and Robin Williams. 1998. Exploring Expertise: Issues and Perspectives. Palgrave Macmillan UK, 1–27. [18] David Feygin, Madeleine Keehner, and Frank Tendick. 2002. Haptic Guidance: Experimental Evaluation of a Haptic Training Method for a Perceptual Motor Skill. In Proceedings of the 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS ’02). 40–47. [19] Wendell R Garner. 1962. Uncertainty and structure as psychological concepts. (1962). [20] Michael A. Greminger and Bradley J. Nelson. 2004. Vision-based force measurement. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15, No. 4 (2004), 290–298. [21] James C Gwilliam, Mohsen Mahvash, Balazs Vagvolgyi, Alexander Vacharat, David D Yuh, and Allison M Okamura. 2009. Effects of haptic and graphical force feedback on teleoperated palpation. In Robotics and Automation, 2009. ICRA’09. IEEE International Conference on. 677–682. [22] Kristen L. Helton, Buddy D. Ratner, and Natalie A. Wisniewski. 2011. Biomechanics of the Sensor-âĂŞTissue Interface-âĂŤEffects of Motion, Pressure, and Design on Sensor Performance and the Foreign Body Response–Part I: Theoretical Framework. Journal of Diabetes Science and Technology Vol. 5 (2011), 632–646. [23] Robert R. Hoffman. 1998. How Can Expertise be Defined? Implications of Research from Cognitive Psychology. Palgrave Macmillan UK. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. :18 •A. Aviles-Rivero et al. [24] John M. Hollerbach. 2000. Some current issues in haptics research. In IEEE International Conference on Robotics and Automation ICRA, Vol. 1. 757–762. [25] John F. Hughes, Andries Van Dam, Morgan McGuire, David F. Sklar, James D. Foley, Steven K. Feiner, and Kurt Akeley. 2013. Computer graphics: principles and practice (3rd ed.). Addison-Wesley Professional. 1264 pages. [26] Archie Hughes-Hallett, Erik K Mayer, Philip Pratt, Alex Mottrie, Ara Darzi, and Justin Vale. 2015. The current and future use of imaging in urological robotic surgery: a survey of the European Association of Robotic Urological Surgeons. The International Journal of Medical Robotics and Computer Assisted Surgery 11, 1 (2015), 8–14. [27] Stephan MD Jacobs, David Holzhey, Gero MD Strauss, Oliver Burgert, and Volkmar Falk. 2007. The impact of haptic learning in telemanipulator-assisted surgery. Surg. Laparosc. Endosc. 17, 5 (2007), 402–406. [28] Timothy N. Judkins, D. Oleynikov, and Nicholas Stergiou. 2009. Objective evaluation of expert and novice performance during robotic surgical training tasks. Surgical Endoscopy and Other Interventional Techniques, no. 3 29 (2009), 590–597. [29] Fatemeh Karimirad, Sunita Chauhan, and Bijan Shirinzadeh. 2014. Vision-based force measurement using neural networks for biological cell microinjection. Journal of Biomechanics, Volume 47 (2014), 1157–1163. [30] Masaya Kitagawa, Daniell Dokko, Allison M. Okamura, Brian T. Bethea, and David D. Yuh. 2004. Effect of sensory substitution on suture manipulation forces for surgical teleoperation. Studies in Health Technology and Informatics 98 (2004), 157–163. [31] Jacqueline K. Koehn and Katherine J. Kuchenbecker. 2015. Surgeons and non-surgeons prefer haptic feedback of instrument vibrations during robotic surgery. Surgical Endoscopy 29, 10 (2015), 2970–2983. [32] Árni Kristjánsson, Alin Moldoveanu, Ómar Jóhannesson, Oana Balan, Simone Spagnol, Vigdís Vala Valgeirsdóttir, and Rúnar Unnpórsson. 2016. Designing sensory-substitution devices: Principles, pitfalls and potential. Restorative Neurology and Neuroscience Preprint (2016), 1–19. [33] Matthew Kroh and Sricharan Chalikonda. 2015. Essentials of Robotic Surgery. Springer International Publishing (2015). [34] Karin Laumann, Martin Rasmussen, and Ronald L. Boring. 2017. A Literature Study to Explore Empirically: What Is the Scientific Discipline of Human Factors and What Makes It Distinct from Other Related Fields. Springer International Publishing, 63–73. [35] Charles Lenay, Olivier Gapenne, Sylvain Hanneton, Catherine Marque, and Christelle Geouelle. 2013. Sensory Substitution: limits and perspectives. Touching for Knowing, Cognitive psychology of haptic manual perception (2013), 275–292. [36] Thomas Sean Lendvay, Blake Hannaford, and Richard M Satava. 2013. Future of robotic surgery. The Cancer Journal 19, 2 (2013), 109–119. [37] James R Lewis. 1995. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction 7, 1 (1995), 57–78. [38] Mohsen Mahvash and Allison Okamura. 2007. Friction Compensation for Enhancing Transparency of a Teleoperator With Compliant Transmission. IEEE Transactions on Robotics 23, 6 (2007), 1240–1246. [39] Michael J. Massimino and Thomas B. Sheridan. 1992. Sensory Substitution For Force Feedback In Teleoperation. Analysis, Design and Evaluation of Man-Machine Systems (1992). [40] Giuseppe Meccariello, Federico Faedi, Saleh AlGhamdi, Filippo Montevecchi, Elisabetta Firinu, Claudia Zanotti, Davide Cavaliere, Roberta Gunelli, Marco Taurchini, Andrea Amadori, and Claudio Vicini. 2016. An experimental study about haptic feedback in robotic surgery: may visual feedback substitute tactile feedback? Journal of Robotic Surgery 10, 1 (2016), 57–61. [41] Leonardo Meli, Claudio Pacchierotti, and Domenico Prattichizzo. 2014. Sensory Subtraction in Robot-Assisted Surgery: Fingertip Skin Deformation Feedback to Ensure Safety and Improve Transparency in Bbimanual Haptic Interaction. IEEE Transactions on Biomedical Engineering 61, 4 (2014), 1318–1327. [42] P. Mountney, D. Stoyanov, and G.-Z. Yang. 2010. Three-Dimensional Tissue Deformation Recovery and Tracking: Introducing techniques based on laparoscopic or endoscopic images. IEEE Signal Processing Magazine (2010), 14–24. [43] Matthew C. Murphy, Amy C. Nau, Christopher Fisher, Seong-Gi Kim, Joel S. Schuman, and Kevin C. Chan. 2016. Top-down influence on the visual cortex of the blind during sensory substitution. NeuroImage 125 (2016), 932–940. [44] Saskia K. Nagel, Christine Carl, Tobias Kringe, Robert MÃďrtin, and Peter KÃűnig. 2005. Beyond sensory substitutionâĂŤlearning the sixth sense. Journal of Neural Engineering 2, 4 (2005). [45] Ehsan Noohi, Sina Parastegari, and Milos Zefran. 2014. Using Monocular Images to Estimate Interaction Forces During Minimally Invasive Surgery. IEEE International Conference on Intelligent Robots and Systems (2014), 4297–4302. [46] Shogo Okamoto, Masashi Konyo, and Satoshi Tadokoro. 2011. Vibrotactile Stimuli Applied to Finger Pads as Biases for Perceived Inertial and Viscous Loads. IEEE Transactions on Haptics 4, 4 (2011), 307–315. [47] Allison M. Okamura, Lawton N. Verner, Tomonori Yamamoto, James C. Gwilliam, and Paul G. Griffiths. 2011. Force Feedback and Sensory Substitution for Robot-Assisted Surgery. Springer US, 419–448. [48] Claudio Pacchierotti, Leonardo Meli, Francesco Chinello, Monica Malvezzi, and Domenico Prattichizzo. 2015. Cutaneous haptic feedback to ensure the stability of robotic teleoperation systems. The International Journal of Robotics Research 34, 14 (2015), 1773–1787. [49] Claudio Pacchierotti, Asad. Tirmizi, and Domenico Prattichizzo. 2014. Improving transparency in teleoperation by means of cutaneous tactile force feedback. ACM Transactions on Applied Perception (2014). ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017. Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study •:19 [50] Elizabeth W Paxton, Robert S Namba, Gregory B Maletis, Monti Khatod, Eric J Yue, Mark Davies, Richard B Low, Ronald WB Wyatt, Maria CS Inacio, and T Ted Funahashi. 2010. A prospective study of 80,000 total joint and 5000 anterior cruciate ligament reconstruction procedures in a community-based registry in the United States. J Bone Joint Surg Am 92, Supplement 2 (2010), 117–132. [51] Christopher J. Payne, Hedyeh Rafii-Tari, Hani J. Marcus, and Guang-Zhong Yang. 2014. Hand-held microsurgical forceps with forcefeedback for micromanipulation. In 2014 IEEE International Conference on Robotics and Automation (ICRA). 284–289. [52] Gary Perlman. 1985. Electronic surveys. Behavior Research Methods, Instruments, & Computers 17, 2 (1985), 203–205. [53] Srinivas K. Prasad, Masaya Kitagawa, Gregory S. Fischer, Jason Zand, Mark A. Talamini, Russell H. Taylor, and Allison M. Okamura. 2003. A modular 2-DOF force-sensing instrument for laparoscopic surgery. Lecture Notes in Computer Science 2878 (2003), 279–286. [54] Pinyo Puangmali, Hongbin Liu, Lakmal D. Seneviratne, and Kaspar Althoefer. 2012. Miniature 3-Axis Distal Force Sensor for Minimally Invasive Surgical Palpation. IEEE Transactions on Mechatronics (2012), 646–656. [55] Josef P. Rauschecker. 1995. Compensatory plasticity and sensory substitution in the cerebral cortex. Trends in Neurosciences 18, 1 (1995), 36–43. [56] Carol E. Reiley, Takintope Akinbiyi, Darius Burschka, David C. Chang, Allison M. Okamura, and David D. Yuh. 2008. Effects of visual force feedback on robot-assisted surgical task performance. The Journal of Thoracic and Cardiovascular Surgery 135, 1 (2008), 196 – 202. [57] Laurent Renier and Anne G. De Volder. 2005. Cognitive and Brain Mechanisms in Sensory Substitution of Vision: A Contribution to the Study of Human Perception. Journal of Integrative Neuroscience 04, 04 (2005), 489–503. [58] Giulio Rognini and Olaf Blanke. 2016. Cognetics: Robotic Interfaces for the Conscious Mind. Trends in Cognitive Sciences 20, 3 (2016), 162 – 164. [59] Robert W. Root and Steve Draper. 1983. Questionnaires as a Software Evaluation Tool. In ACM Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 83–87. [60] JP Ruurda, IAMJ Broeders, B Pulles, FM Kappelhof, and C Van der Werken. 2004. Manual robot assisted endoscopic suturing: time-action analysis in an experimental model. Surgical endoscopy and other interventional techniques 18, 8 (2004), 1249–1252. [61] Ryan E. Schoonmaker and Caroline G.L. Cao. 1992. Vibrotactile Feedback Enhances Force Perception in Minimally Invasive Surgery. Proceedings of the Human Factors and Ergonomics Society Annual Meeting (1992), 1029–1033. [62] Samuel B Schorr, Zhan Fan Quek, Ilana Nisky, William R Provancher, and Allison M Okamura. 2015. Tactor-induced skin stretch as a sensory substitution method in teleoperated palpation. IEEE Transactions on Human-Machine Systems 45, 6 (2015), 714–726. [63] James Shanteau. 1992. The Psychology of Experts An Alternative View. Springer US, 11–23. [64] Ben Shneiderman. 2010. Designing the user interface: strategies for effective human-computer interaction. Pearson Education India. [65] Saeed Sokhanvar, Javad Dargahi, Siamak Najarian, and Siamak Arbatani. 2012. Clinical and Regulatory Challenges for Medical Devices Tactile Sensing and Displays. Haptic Feedback for Minimally Invasive Surgery and Robotics, Wiley Publications (2012). [66] Nathaniel J Soper, Lee L Swanström, and Steve Eubanks. 2008. Mastery of endoscopic and laparoscopic surgery. Lippincott Williams & Wilkins. [67] Giuseppe Spinoglio. 2015. Robotic Surgery: Current Applications and New Trends. Springer-Verlag Mailand (2015). [68] Minghui Sun, Xiangshi Ren, and Xiang Cao. 2011. Effects of multimodal error feedback on human performance in steering tasks. Journal of Information Processing 18 (2011), 284–292. [69] Paul Bach-y-Rita and Stephen W. Kerce. 2003. Sensory substitution and the human âĂŞ machine interface. TRENDS in Cognitive Sciences (2003), 541–546. [70] Olivier A.J. Van der Meijden and Marlies P. Schijven. 2009. The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surgical Endoscopy (2009), 1180–1190. [71] Colin Ware. 2004. Information Visualization: Perception for Design. Morgan Kaufmann Publishers Inc. [72] Bernhard Weber and Sonja Schneider. 2014. The effects of force feedback on surgical task performance: a meta-analytical integration. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, 150–157. [73] Christopher D Wickens. 2002. Multiple resources and performance prediction. Theoretical issues in ergonomics science 3, 2 (2002), 159–177. [74] Erik B. Wilson, Hossein Bagshahi, and Vicky D. Woodruff. 2014. Overview of General Advantages, Limitations, and Strategies. Book Chapter, Robotics in General Surgery, Springer (2014). [75] Michael. C Yip, Shelten G. Yuen, and Robert D. Howe. 2010. A Robust Uniaxial Force Sensor for Minimally Invasive Surgery. IEEE Transactions on Biomedical Engineering, Vol. 57, No. 5 (2010), 1008–1011. ACM Transactions on Applied Perception, Vol. 1, No. 1, Article . Publication date: December 2017.