Smarthands for prosthetic and robotic applications

Department of Information Technology and Electrical Engineering Spring Semester 2024 Smarthands for Prosthetic and Robotic Applications Master Thesis Maria Ruiz Dorado [email protected] August 2024 Supervisors: Dr. Xiaying Wang, xiayw[email protected]h Dr. Christian Vogt, christian.v[email protected]h Marco Giordano, [email protected]h Professor: PD Dr. M. Magno, mic[email protected]h Acknowledgements I would like to extend my sincere gratitude to my advisors for their invaluable guidance and support during these six months of work, and to Center for Project-Based Learning (PBL) for offering me the opportunity to conduct this thesis and for providing the necessary resources. I also appreciate my family and friends for their encouragement throughout this journey. ii Abstract The advancement of tactile sensor technology has the potential to significantly enhance the functionality of prosthetic and robotic hands by improving their ability to mimic human touch and grasping capabilities. This research presents a comprehensive study on the application of Scalable tactile glove (STAG) sensors for grasp classification and pressure distribution analysis for both human and robotic hands. The study involves a detailed examination of the sensor system’s performance across various grasps and objects, highlighting its effectiveness in capturing pressure patterns. The research includes reproducing the embedded STAG system for extracting pressure distribution data from hand grasps. A complete dataset was generated, split into different sessions, and obtained from both human and robotic grasps. This new dataset was subjected to an exhaustive analysis in three main ways. Two distinct neural network models were trained and tested using data from different recording sessions, achieving classification accuracies of 85 % for human data and 87 % for combined human and robotic hands. However, the research also reveals significant challenges, such as a substantial drop in accuracy when applying a network trained on human data to test data from robotic hands. The comparative analysis focused on five different grips, which include seven of the top ten gestures used in 80 % of daily activities. This analysis revealed performance patterns in the human hand, confirming the sensor’s qualitative capabilities for pressure mapping. Finally, the comparative analysis of human and robotic hand performance showed that while robotic hands can replicate various grasps, their pressure distribution is more localized due to limited flexibility and adaptability compared to the more distributed pressure patterns seen in human hands. This discrepancy highlights the need for further advancements in robotic hand design to achieve more natural and versatile grasping capabilities. iii Declaration of Originality I hereby confirm that I am the sole author of the written work here enclosed and that I have compiled it in my own words. Parts excepted are corrections of form and content by the supervisor. For a detailed version of the declaration of originality, please refer to Appendix B Maria Ruiz Dorado, Zurich, August 2024 iv Contents List of Acronyms xi 1. Introduction 1 1.1. Motivation.................................... 2 1.2. Objective .................................... 2 2. Related Work 4 2.1. Improvement of the STAG . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2. Design and implementation of the embedded system . . . . . . . . . . . . 5 2.3. Adaptation of the Conventional neural network (CNN) . . . . . . . . . . . 7 2.4. Integration and provision of the Robotic Prosthetic Hand (RPH) . . . . . 7 3. Background 9 3.1. Flexible resistive sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2. Signalprocessing ................................ 10 3.3. Network for object classification . . . . . . . . . . . . . . . . . . . . . . . . 12 4. Implementation 14 4.1. Embeddedsystem ............................... 14 4.1.1. Hardware ................................ 14 4.1.2. Software................................. 20 4.2. Mia Robotic Prosthetic Hand . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.3. Datasetset-up ................................. 27 5. Results 31 5.1. Materialsandcosts............................... 31 5.2. Dataset analysis and performance . . . . . . . . . . . . . . . . . . . . . . . 32 5.2.1. Method 1: Network Training and Testing Resolution . . . . . . . . 32 5.2.2. Method 2: Grasps comparative . . . . . . . . . . . . . . . . . . . . 34 5.2.3. Method 3: Robotic vs. Human Hand: Similarities and Differences . 40 v Contents 6. Discussion 47 7. Conclusion and Future Work 48 A. Task Description 49 B. Declaration of Originality 55 C. Costs and Materials 57 D. File Structure 60 E. Dataset structure 62 F. Template of the hands 67 G. Detailed Photographs of the Integrated System Components 71 vi List of Figures 2.1. Fabricated sensor before being glued to a glove from reference project [1] 5 2.2. Fully assembled system from the top, including the customized Printed Circuit Board Assembly (PCBA) on the top and the STM32 one below [1] 6 2.3. MIA Hand provided by the Institute of Neuroinformatics (INI) laboratory and developed by Prensilia, front view [2] . . . . . . . . . . . . . . . . . . 8 2.4. MIA Hand provided by the INI laboratory and developed by Prensilia, backview [2].................................. 8 3.1. Pressure-Sensitive Conductive Sheet of Velostat provided by Adafruit [1] 10 3.2. Zero-potential method applied to the array structure of the Velostat-based sensor ...................................... 11 3.3. Representation of the Residual Block in a Residual Network (ResNet) . . 13 4.1. Mechanical structure of the STAG for one point of connection in the global array....................................... 15 4.2. Photograph of the STAG Sensor Adapted to the Human Hand Used in Recorded Sessions 1 and 3: The left side of the image shows the STAG sensor, with the velostat layer and the electrodes array matrix that are used to capture pressure data. The right side of the image displays the back of the hand, where the accelerometer is securely attached to the glove. 16 4.3. Functional schematic of the readout circuit applying the zero-potential method for mitigating crosswalks. . . . . . . . . . . . . . . . . . . . . . . . 19 4.4. Basic schematic of the designed CNN network, illustrating the data behaviour and principal layer: input layer, convolution layers with 3x3 kernels, pooling layer, ResNet layer, flatten layer, and the final fully connected layer, which together formed the architecture used for classifying grasp types in the research. . . . . . . . . . . . . . . . . . . . . . . . . . . 23 vii List of Figures 4.5. Completed schematic of the designed CNN network, illustrating all the components and real disposition: input layer, convolution layers with 3x3 kernels, pooling layers, Rectified Linear Unit (ReLu) layers ResNet blocks, flatten layers, and the final fully connected layer, which together formed the architecture used for classifying grasp types in the research. . . . . . . 24 4.6. Mia Hand reproducing the six studied positions, in order: Cylindrical Grasp, Precision Grasp, Lateral Grasp, Pointing Down Position, Neutral Position, Pointing Up Position [3] . . . . . . . . . . . . . . . . . . . . . . . 26 4.7. Set of objects arranged from left to right and top to bottom: big bottle, card, clip for hair, coin, cotton swab, cream, empty hand, fork, gel, glass, key, lipstick, pen, pointing down, pointing up, scissors, screwdriver, small bottle, tape, tissue, toothbrush, tomato and tweezers . . . . . . . . . . . . 29 4.8. Mia RPH equipped with the STAG sensor shown from various angles . . . 30 5.1. Training accuracy and training loss metrics recorded during the dataset training process in session 1 in a total of 40 epochs. The training accuracy curve shows the model’s performance in correctly classifying the grasps over successive epochs, while the training loss curve indicates the error rate. 33 5.2. Training accuracy and training loss metrics recorded during the dataset training process in session 123 in a total of 40 epochs. The training accuracy curve shows the model’s performance in correctly classifying the grasps over successive epochs, while the training loss curve indicates the errorrate..................................... 33 5.3. Pressure distribution on the STAG sensor in the human hand during a cylindrical grasp on a glass. . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.4. Pressure distribution on the STAG sensor in the human hand during a precision grasp on a lipstick and pressure concentration percentage by area. 37 5.5. Relation between the average pressure in the human hand during a precision grasp across the entire set of objects and their respective weights and pressure concentration percentage by area. . . . . . . . . . . . . . . . . . 38 5.6. Pressure distribution on the STAG sensor in the human hand during a lateral grasp on a key and pressure concentration percentage by area. . . . 38 5.7. Relation between the average pressure in the human hand during a lateral grasp across the entire set of objects and their respective weights. . . . . 39 5.8. Pressure distribution on the STAG sensor in the human hand during a pointing down grasp and pressure concentration percentage by area. . . . 39 5.9. Pressure distribution on the STAG sensor in the human hand during a pointing up grasp and pressure concentration percentage by area. . . . . . 40 5.10. Pressure distribution on the STAG sensor during the neutral position in the human hand and the robotic hand, from left to right. . . . . . . . . . . 41 5.11. Pressure concentration in different regions of the human and robotic hands during cylindrical, lateral, and precision grasps. . . . . . . . . . . . . . . . 41 viii List of Figures 5.12. Pressure distribution on the STAG sensor during a cylindrical grasp of a big bottle: human hand (left) vs. robotic hand (right). . . . . . . . . . . 42 5.13. Pressure distribution on the STAG sensor during a precision grasp of a card: human hand (left) vs. robotic hand (right). . . . . . . . . . . . . . . 43 5.14. Pressure distribution on the STAG sensor during a lateral grasp of a lipstick: human hand (left) vs. robotic hand (right). . . . . . . . . . . . . . . 43 5.15. Edges strength distribution on the STAG sensor during a lateral grasp of a lipstick: human hand (left) vs. robotic hand (right). . . . . . . . . . . . 45 5.16. Edges strength distribution on the STAG sensor during a lateral grasp of a lipstick: human hand (left) vs. robotic hand (right). . . . . . . . . . . . 45 5.17. Edges strength distribution on the STAG sensor during a lateral grasp of a lipstick: human hand (left) vs. robotic hand (right). . . . . . . . . . . . 46 F.1. Robotic Hand base: This image shows the template for the Velostat base used in the robotic hand, including the arrangement and structure of the electrode matrix. Scale, 1:0.8 . . . . . . . . . . . . . . . . . . . . . . . . . 68 F.2. Human Hand Base: This image shows the template for the Velostat base used in the human hand, including the holes for the electrode matrix. Scale,1:1 .................................... 69 F.3. Hand Map with Labeled Regions: This image provides a detailed map of the hand, highlighting and labeling the different regions. Each numbered section corresponds to a specific area of the hand, aiding in the analysis and interpretation of pressure distribution across various grasps. . . . . . . 70 G.1. Customized board as readout circuit designed by previous student [1], frontview.................................... 71 G.2. Customized board as readout circuit designed by previous student [1], back view ....................................... 72 G.3. STM32F769I-DISC1 development board [4], front view . . . . . . . . . . . 72 G.4. STM32F769I-DISC1 development board [4], back view . . . . . . . . . . . 73 ix Chapter 2 Related Work This thesis relies on significant previous researches in the fields of tactile sensing and RPH. The first and most relevant reference is the project presented in [1], which was published on Arxiv and developed by a previous Master Student at the Federal Institute of Technology (ETH) in Zürich. It laid the foundation for the sensor and embedded system used in this research. The second significant collaboration is the integration of the RPH developed by Prensilia [2] and provided by the INI of the University of Zürich (UZH). This section describes the key contributions of these works, which are summarized as follows: •Reference project [1]: –Improvement of the STAG –Design and implementation of embedded system –Adaptation of the CNN •Prensilia RPH [2]: –Integration and provision of the RPH Each of these contributions is integral to the advancement of this thesis and will be discussed in more detail in the following sections. 2.1. Improvement of the STAG The STAG represents a major advancement in tactile sensor technology, utilizing a Conductive polymer composite (CPC) that is highly sensitive to pressure applications, as explained in Chapter 3.1. This CPC material is sandwiched between two orthogonal sets 4 2. Related Work of flexible electrodes, where each crossing point functions as an individual Force-sensing resistor (FSR). This configuration allows for a highly adaptable sensor array that can be manufactured in various shapes and sizes, making it suitable for a wide range of applications. One notable implementation of this technology, as discussed in reference [1], involved fabricating the sensor into a wearable glove. This design was chosen to capture comprehensive tactile data from the entire hand, facilitating the study of human grip patterns and interactions with different objects. Therefore, the glove works as a high-resolution sensor grid able to detect subtle variations in pressure and force. Figure 2.1.: Fabricated sensor before being glued to a glove from reference project [1] 2.2. Design and implementation of the embedded system The embedded system was designed to optimize low power consumption and high-speed data acquisition, capable of processing tactile data at a rate of 100 Hz. The system provides robust performance and efficiency in real-time applications, by utilizing a combination of a commercial STM32 PCBA and a customized PCBA for the readout circuit. The STM32 microcontroller serves as the core, handling data communication, power distribution and signal conditioning. Meanwhile, the customized PCBA commits sensor control, managing the interface with the STAG sensor and ensuring accurate data acquisition. 5 2. Related Work Figure 2.2.: Fully assembled system from the top, including the customized PCBA on the top and the STM32 one below [1] The system processes data from the STAG sensor and transmits real-time pressure information via Universal asynchronous receiver / transmitter (UART) communication. Additionally, it is equipped to run a CNN for object classification, enhancing its capability to identify and classify objects based on tactile data. This dual functionality allows for comprehensive tactile sensing and intelligent data analysis. The embedded system is pertinent for applications such as RPH, where detailed tactile feedback is crucial, and real-time monitoring, where continuous data acquisition is essential. By bridging the gap between advanced tactile sensors and practical applications, this system significantly contributes to the fields of robotics and prosthetics. 6 2. Related Work 2.3. Adaptation of the CNN The prior study focused on exploring neural network architectures for object classification using tactile data that were compatible with the microcontroller of STM32. Various network architectures were evaluated, including Temporal convolutional neural networks (TCN) and CNN without residual blocks, using datasets from MIT and newly collected data towards the end of the thesis project. The reference neural network model employed multiple stages of convolutions with two residual blocks to extract features efficiently, achieving an inference speed of 100 ms with a compact model size of 117 kB. The architecture achieved high accuracy and low loss on training data but exhibited variability in test data performance based on dataset splits. Confusion matrices illustrated classification issues, particularly among objects with similar shapes. Moreover, another cause for the inter-session variability was the degradation of the tactile sensor response across the different sessions. Consequently, these insights remark the importance of session-based data handling and the need for robust neural network models capable of accommodating variability in tactile datasets. 2.4. Integration and provision of the RPH The project focuses on studying the behavior of the sensor on a RPH, from analyzing its characteristics to comparing them with a human hand. When evaluating the efficiency of an upper limb prosthesis, the primary focus is on the variety and execution of grasps and gestures. For this purpose, MIA Hand was utilized, a sophisticated prosthetic device developed by Prensilia and provided by the INI laboratory of UZH. The MIA Hand is capable of performing five different grips, covering seven out of the top ten gestures used in 80 percent of daily activities. This versatility makes it an excellent candidate for assessing the practical functionality of prosthetic hands in replicating human hand movements. By focusing on these common gestures, the evaluation can closely mirror the real-world challenges faced by upper limb amputees. In chapter 4.2, a detailed examination of the MIA Hand’s functionalities is provided. 7 2. Related Work Figure 2.3.: MIA Hand provided by the INI laboratory and developed by Prensilia, front view [2] Figure 2.4.: MIA Hand provided by the INI laboratory and developed by Prensilia, back view [2] 8 Chapter 3 Background This chapter provides the theoretical foundation essential for understanding the core components of this research. It covers three main areas: sensor characteristics, signal processing and neural networks. Each section introduce into the principles and technologies that define the functionality of the tactile sensing system. 3.1. Flexible resistive sensors The pressure sensor used in this research consists of flexible resistive sensors. Compared to pressure sensors based on rigid materials, they offer not only standard applications in pressure or tactile sensing but also advanced applications in modern technology, including wearable electronics as in this case. Specifically, this project focuses on the type based on piezoresistivity, which is defined as the change in resistance of semiconductors due to applied mechanical stress. Piezoelectric sensor systems have become a dominant category in pressure sensing because of their easy fabrication, low cost, and simple signal processing circuitry for data acquisition. In recent years, various types of materials and structures based on this technology have been developed. In this research, the velostat material was used. Velostat [5] is a cheap and widely available material for constructing simple flexible pressure sensors at home. It is a CPC based on an elastic polymer impregnated with conductive particles of carbon black, which enables electrical conductivity. Given the distance between these particles, the resistivity of the material fluctuates. Consequently, it forms a flexible and thin sheet that can be easily cut or shaped for many applications. Moreover, another interesting capability of this material is its utility for assessing contact and movement patterns. 9 3. Background Figure 3.1.: Pressure-Sensitive Conductive Sheet of Velostat provided by Adafruit [1] On the opposite side, previous sensors based on this technology have demonstrated accuracy errors in the pressure data obtained. The material changes resistance with strain produced by pressure as well as by flex. Recent research has explored the feasibility of employing a velostat-based sensor on curved surfaces for measuring pressure. In the following conference paper [6], the material was subjected to different bending radii and mechanical tests to evaluate its response. The results of the investigation demonstrated that the material could be used in highly scalable matrices under rigorous calibration and for relative applications rather than precise ones. 3.2. Signal processing Resistive sensors, such as piezoresistive tactile sensors, have been widely used for measurement and instrumentation in various fields. These sensors are typically arranged in a matrix to achieve the desired measurement resolution. Due to this array structure, crosstalk currents between each sensor element in the array must be considered. Crosstalk is the phenomenon where signals from one line interfere with adjacent circuits or lines, generating electromagnetic variations. In this specific case, crosstalk currents can flow through unintended paths when the voltage of the velostat resistance of interest is measured. 10 3. Background In the field of signal processing and crosstalk problems, several solutions have been proposed. These range from inserting a diode or a transistor for each element in the sensor array to using voltage feedback methods to remove the crosstalk current. However, these different methods often suffer from measurement errors or high system complexity. The most widely used method to solve the crosstalk current problem is the zero-potential method, which was applied in the readout circuit. Figure 3.2.: Zero-potential method applied to the array structure of the Velostat-based sensor This method uses an operational amplifier at each column terminal. Since both the negative and positive input voltages of the operational amplifier must be equal, due to the virtual short characteristics of an operational amplifier, all the columns and the rows have the same voltage of VDRIV E , except for a selected row whose voltage is 0 V. Additionally, considering the velostat as a variable resistance RV EL and the resistance 11 3. Background from the feedback of the amplifier RF A, the voltage at the output of the amplifier follows the equation 3.1: VDRIV E =VDRIV E · RF A +RV EL RV EL ,(3.1) In summary, the zero-potential method applied allows currents to only flow through the resistive sensor elements in the selected row, ideally without crosstalk currents, while the system is able to read out all columns of the active row simultaneously. 3.3. Network for object classification Neural networks are computational models that mimic the complex functions of the human brain. Like neurons in the brain, neural networks consist of interconnected nodes that process and learn from data, enabling tasks such as pattern recognition and decisionmaking in machine learning. A significant part of this research involved training a neural network for grasp classification. Therefore, this section will explain the basic principles of neural networks and machine learning to provide an introduction to this field. CNNs are a type of artificial neural network primarily used for processing data with a grid-like topology, such as the developed sensor. The main advantage of CNNs over other algorithms is that they perform unsupervised learning, which doesn’t require labeled samples to learn features from data. Due to this advantage, CNNs have become one of the most popular deep learning networks. In a fully connected neural network, all hidden layers are fully connected, meaning every neuron connects to every neuron of the previous layer. This design has two major disadvantages: the coordinate information is lost when the data matrix is reshaped into a one-dimensional array, and the large number of variables makes it difficult to train and prone to overfitting. Conversely, in convolutional layers, which are the core building blocks of CNNs, each neuron does not connect to every neuron in the previous layer but only to those in a receptive field. This allows the neural network to study features in small local areas rather than the entire dataset. This characteristic makes CNNs powerful in grid classification, as they can learn local features more efficiently than traditional neural networks, reducing the number of variables during training and making CNNs easier to train and less prone to overfitting. Specifically, a simplified ResNet was applied in this research. ResNets provide significant advantages in deep learning, including improved training for very deep networks and faster convergence, which reduces model complexity. The degradation problem in neural networks refers to the phenomenon where, as the depth of a neural network increases, the performance on the training data saturates and then starts to degrade. In a classical neural network, the input is transformed by a set of convolutional layers and then passed 12 3. Background to the activation function. In a residual network, the input to the block is added to the output of the block, creating a residual connection. The output of the residual block R(x)can be represented by the equation 3.2, R(x) = F(x) + x, (3.2) where F(x)represents the residual mapping learned by the network.The presence of the identity term xallows the gradient to flow more easily. Skip connections help the residual blocks bypassing the input over the convolutional layer and adding it to the output of the residual block. This connection makes it easier for the model to learn the identity function when needed, leading to faster convergence and shorter training times. Figure 3.3.: Representation of the Residual Block in a ResNet 13 4. Implementation 4.1.2. Software This thesis involved two primary domains. The first domain was the code for the applications that ran on the microcontroller. This code was entirely written in C and developed using the STMCubeIDE application across various projects. All applications were created as bare metal applications, meaning they operated without an operating system. The second domain pertained to the evaluation and training of neural networks. Python was chosen for this domain due to its extensive support for machine learning models, tools and algorithms. The following subsections provide detailed information on the code implemented for each domain. STM32 discovery board’s system configuration This chapter delves into the intricate implementation details of the system, focusing on the critical roles played by various peripherals and configurations. In this implementation, the ADC1 peripheral played a crucial role in converting analog signals to digital data, essential for the accurate data frames reading from the STAG. ADC1 was configured with a clock prescaler of 2 and a 12-bit resolution, allowing it to handle multiple channel conversions. Specifically, channels 4 and 6 were set up for data acquisition to ensure precise measurements. For inter-device communication, the Inter-Integrated Circuit (I2C)1 peripheral was configured. The timing was initially set to 0x6000030D and used a 7-bit addressing mode. This setup facilitated efficient communication between the STM32 board and the customized one, enabling the data frame exchange from the STAG. Timing and event control were managed using the Time Input Mode (TIM)6 and TIM7 peripherals. TIM6 was set with a prescaler of 107, while TIM7 had a prescaler of 9999, both configured for up-counting mode. These timers were critical for managing timedependent operations within the system, ensuring synchronized functionality. Serial communication was handled by the USART1 peripheral. Configured with a baud rate of 921600, 8-bit word length, and no parity, USART1 enabled high-speed data transmission, vital for debugging and transferring data with the laptop connected by the micro-USB port of the PCBA. Different General-Purpose Input/Output (GPIO) ports were configured for various functions. First, the Light-Emitting Diode (LED)s control informed of the activity of the sensor: the red LED indicated that the sensor was sending data, while the green LED showed that the system was on hold and ready to turn on. Second, the multiplexer selection of the customized PCBA for operating the zero-potential method of signal control was controlled from the peripheral of the STM32 PCBA. Third, the system had two button inputs for controlling the turn-on and off of the data extraction: the black button activated the extraction and the blue button deactivated it. 20 4. Implementation The DMA controller was initialized to enhance data transfer efficiency. By allowing memory transfers without CPU intervention, DMA significantly improves the performance of data-intensive operations, facilitating smooth and efficient data handling. Additionally, the X-CUBE-AI initialization was included to leverage the STM32’s capabilities for machine learning and neural network applications. This setup enables advanced AI functionalities, which for this research was used for providing the network and network data files adapted for its processing on the PCBA from the upload of the network in onnx format, allowing the system to process real-time complex algorithms and provide classification predictions based on the network. Overall, the STM32F769I-DISC1 development board, with its comprehensive set of features and peripherals, provided a robust foundation for the designed embedded system. The careful configuration and initialization of these components ensured efficient operation and seamless integration, enabling us to meet the specific demands of the project. Projects and code files Project 1: Extraction and visualization of the data frames This project encompassed two main applications. The first application, initiated by flashing the DataLogging starting code onto the STM32F769I-DISC1 board, enabled the collection and logging of tactile data. This was particularly useful for capturing data during object interactions, with the base timer configured to 100Hz. The application operated in three main states: idle, data collection, and data transfer. In the idle state, the system remained inactive, awaiting a command to start data collection. The base timer was deactivated during this state. Data collection commenced when the black button on the discovery board was pressed, triggering data transmission via the UART protocol to the microcontroller. Given the latency involved in sending a tactile frame, which spanned several tens of milliseconds, the external SDRAM on the discovery board was utilized as intermediate storage for the tactile data. The system could be configured to count the number of frames to be collected. In such cases, once the specified number of frames was acquired, data collection ceased, the SDRAM content was immediately transferred to a connected computer, and the system reverted to the idle state. The SDRAM had the capacity to store a maximum of 4096 tactile frames, limiting a single interaction to approximately 40 seconds at a data collection rate of 100Hz, which was sufficient for the purposes of this thesis. It was found that the simplest method for logging data sent by the microcontroller was using the built-in logging feature of Tera Term. When setting up the serial port, it was important to configure the speed to 921600, with data in 1-byte format and no parity or flow control. Once all necessary data was saved in log files, the loadAndSave realData.py code was used to generate a metadata file in Matlab format for further analysis. 21 4. Implementation The second application, initiated by flashing the GUI test code onto the STM32F769IDISC1 board, enabled the collection and visualization of tactile data. The idle state functioned similarly to the first application, but during data acquisition, the base timer was activated. Thus, every 100 milliseconds, one tactile frame was collected and directly sent to the connected computer. The frame rate was kept relatively low to ensure each frame could be transferred via UART before the next one was acquired. Instead of using Tera Term, the STAG GUI PyQt.py code was executed to visualize the data frames. Project 2: Predictions and Classifications by the data frames Flashing the Demos code onto the STM32F769I-DISC1 board enables predictions on the tactile data collected by the sensor. The primary utility of this project lied in acquiring tactile data and transmitting it to a connected PC via a DMA controller. Simultaneously, the Arm Cortex-M7 made predictions on the tactile frames based on a previously loaded neural network, and these inference outcomes were transmitted to the PC. Critical to the operation of this project was the neural network, predefined as an ONNX model as detailed in the next chapter 4.1.2. Once the STM32 project was flashed onto the board, interaction with the system were facilitated through the Python script STAG Demo.py, located in the python files folder. This graphical interface allows users to visualize tactile frames along with their corresponding predictions within a unified environment. Python for neural network The network structure used in this research was a simplified version of the CNN Resnet3x3, as theoretically introduced in section 3.3. This chapter explores the structure of the final network, focusing on its various layers. As with any network, the first block was the input, which in this case was a data matrix with dimensions of 32x32. Each frame contained pressure data normalized to a value between 0 and 1. This normalization allowed the matrix to be treated as a gray-scale image, where each pixel represented a data frame. To preserve the two-dimensional information, multiple convolution layers were used as the core building blocks of the CNN. Unlike fully connected layers, each neuron in a convolution layer connected only to a subset of neurons in the previous layer, defined by a 3x3 kernel. This approach enabled the neural network to capture features in small local areas, which was particularly useful in this context. Interspersed between the convolution layers were ReLu blocks. These layers removed negative values from the filtered image, replacing them with zero. When the input was below zero, the output was zero, but when the input exceeded a certain threshold, it had a linear relationship with the dependent variable. This process prevented the accumulation of zeros and accelerated the training of the dataset. 22 4. Implementation Additionally, two pooling layers were incorporated into the network structure to reduce the dataset size by aggregating multiple input values into a single output value. These layers also used a 3x3 kernel. The pooling layers served to decrease the data size and the number of variables, thus reducing the computational load. The implemented network was a simplified ResNet, meaning that the various blocks were not arranged sequentially. As illustrated in figure 4.5, there were parallel pathways throughout the neural network. These pathways enhanced training efficiency for deep networks and promoted faster convergence, thereby reducing model complexity. After the feature maps, the flatten layer converted the 16x16x1 output of the last convolutional layer into a single one-dimensional vector, allowing progression to the final dense layer. This fully connected layer linked each element of the vector to the output block. The primary objective of this neural network was to classify the six different grasps that a human or the Mia RPH are able to execute. Consequently, the output block had a batch size of six, corresponding to each of the feasible grasps, which were explained in greater detail in the following chapter 4.2. Figure 4.4.: Basic schematic of the designed CNN network, illustrating the data behaviour and principal layer: input layer, convolution layers with 3x3 kernels, pooling layer, ResNet layer, flatten layer, and the final fully connected layer, which together formed the architecture used for classifying grasp types in the research. 23 4. Implementation Figure 4.5.: Completed schematic of the designed CNN network, illustrating all the components and real disposition: input layer, convolution layers with 3x3 kernels, pooling layers, ReLu layers ResNet blocks, flatten layers, and the final fully connected layer, which together formed the architecture used for classifying grasp types in the research. 24 4. Implementation 4.2. Mia Robotic Prosthetic Hand The Mia Hand is a five-fingered, self-contained anthropomorphic prosthetic hand designed to aid individuals with upper limb amputations. This advanced prosthesis aims to replicate the functionality and versatility of a natural hand, providing users with improved dexterity and independence. When assessing the efficiency of upper limb prostheses, the primary focus is on the variety and execution of grasps and gestures, as these determine the hand’s practical usability in daily activities. The Mia Hand is capable of performing various grasps in an automatic, pre-programmed manner, closely modeled on those of a natural hand. These pre-programmed grasps were controlled by simple commands, provided by the INI laboraty , making the device intuitive and user-friendly. This research specifically concentrated on five different grips, which encompass seven of the top ten gestures utilized in 80 percent of daily activities. The selected grips were: 1. Cylindrical Grasp (cyl) is one of the most useful everyday grips, which allows to hold heavier objects as bottles, glasses or cups, where a mayor wide is necessary. 2. Precision Grasp (pre) is used for pinching or picking up small objects. This grip allows gestures in which full control of the action is essential, such as a holding cotton swab, picking a hair clip or grabbing a coin. 3. Lateral Grasp (lat) is a very useful for all those everyday actions when holding thin objects between the thumb and the side of the index finger, such as holding a spoon, paying by card or writing with a pen. 4. Point-up Position or extended index finger is the gesture of the index finger being raised, which allows actions as ringing a doorbell or typing keys. 5. Pointing-down Position is used for pointing or pressing with the index finger downwards in activities such as typing on the computer or operating with the calculator. These specific grasps were chosen for their high relevance and frequency in everyday tasks, making them critical for evaluating the prosthetic hand’s performance and utility. Additionally, the neutral position was considered for the calibration of the sensor, ensuring that the hand’s movements are accurately captured and interpreted. 25 4. Implementation Figure 4.6.: Mia Hand reproducing the six studied positions, in order: Cylindrical Grasp, Precision Grasp, Lateral Grasp, Pointing Down Position, Neutral Position, Pointing Up Position [3] 26 4. Implementation 4.3. Dataset set-up The data extraction process was conducted in three distinct sessions involving a total of 23 objects. The selected objects were common household items used in daily activities, including a big bottle, card, coin, cotton swab, cream, empty hand, eyebrow tweezers, fork, gel, glass, hair clip, key, lipstick, pen, pointing down, pointing up, scissors, screwdriver, small bottle, tape, tissue, toothbrush, and tomato. Their technical characteristics, such as size, weight, and material, can be found in the appendix E. The recordings were split on three types of sessions: Session 1, Human Hand Dataset, involved initial testing using a human hand. In this session, three types of recordings were performed for each object. Each object was grasped in two main ways from the list of grasps, mimicking natural human grasping behavior. Additionally, a third recording involved picking up the object from a table in the most natural and easy way for a human, with the objective of recording the entire process of grabbing. Session 2, Robotic Hand Testing, involved testing with a robotic hand using the same five main grips. This session focused on grasping objects in at least two feasible ways. The robotic hand operated in 30 cycles of 10 seconds: 5 seconds in the neutral position and 5 seconds grasping the object. Assistance was required to place objects directly into the robotic hand, which is why it was not possible to reproduce the third type of recording. The hand operated at a fixed speed, with the range of movement adjusted according to the object’s size. Detailed ranges of movement, from 00 (completely open hand, neutral position) to 90 (maximum closed for the specific grasp), are also provided in the appendix E. Session 3, Humans Imitating Robotic Hand Behavior, involved using a human hand to grasp objects while imitating the behavior of the robotic hand, exactly as explained in the session 2 but with a human hand. The objective of this third session was to obtain a dataset comparable to Session 2, to extract differences and similarities under the same behavior. A common element in the three sessions was that the data was labeled with the name of the object, type of grasp, and date of the recording session. This last point is especially important due to sensor degradation over time and use. The neutral position was considered for sensor calibration in each session, extracting data frames from the hand in a static open position. Before applying any external pressure or movement to the sensor, it was already subjected to pressure from the glove on one side and ambient pressure on the other. Therefore, neutral position data was extracted to subtract this noise, ensuring that only the new pressure resulting from the grasping was analysed. 27 4. Implementation Moreover, in the first session, the data frames were extracted with the hand empty in a dynamic situation, cyclically closing and opening the hand and moving the fingers. The objective of this last recording was to study the effects of hand flexion on the sensor, due to being affected by both pressure and flexion. Each extraction session was recorded by video, resulting in 5 minutes of video capturing a type of grasp per object and session, making a total of 35 minutes per each object. This yielded 3500 lines of data frames, with each line consisting of a 32x32 matrix of data, per grasp, object, and session. The videos were recorded from a fixed position using a tripod. In the first and third sessions, the camera was placed from the executor’s perspective, while in the second session, the camera was positioned to obtain lateral and frontal views of the robotic hand to facilitate visualization. All the recordings can be found on the server of ETH. For the data extraction, Tera Term was used to obtain the .log files with the data frames for a complete extraction. The project flashed on the microcontroller was GUI test via STM32CubeIDE, as explained in section 4.1.2. For a detailed overview of the dataset structure, including the file naming conventions, session segmentation and object’s characteristics, refer to the appendix E. 28 4. Implementation Figure 4.7.: Set of objects arranged from left to right and top to bottom: big bottle, card, clip for hair, coin, cotton swab, cream, empty hand, fork, gel, glass, key, lipstick, pen, pointing down, pointing up, scissors, screwdriver, small bottle, tape, tissue, toothbrush, tomato and tweezers 29 5. Results As with the previous grasps, despite the variation in size, material, or shape of the objects, a similar relative pattern of behavior was observed in the data. When the human hand executed a lateral grasp, the index finger, middle finger and thumb were the primary actors. The index finger, being the main executor, was fully flexed to support the object in the grip, which is why the entire region from the distal to the proximal phalanx [areas 2 and 7] was particularly highlighted. The middle finger, usually extended for support, mainly engaged at the middle phalanx [area 8], with the illuminated area varying according to the object’s size and dimensions. The ring and little fingers contributed minimally, primarily in the proximal phalanx area, serving as additional support elements. On the opposite side of the grip, the thumb, especially the distal phalanx (or fingerprint area) [area 1], applied the pressure needed to secure the object. Following the relative analysis, the data was examined for precise conclusions. However, as with the first and second grasps, no direct relationship was found between the pressure data and the weight, shape, or materials of the objects. Pointing down and up grasp The fourth and fifth types of grasps analyzed were the pointing down and pointing up grasps, respectively. In these cases, only one object was used for the recordings. For the pointing down grasp, a calculator was used, with the fingers extended and the index finger pushing the buttons. For the pointing up grasp, a button was pressed with all fingers flexed except for the index finger and thumb, which were extended. In both scenarios, the index finger was the main executor of the grasp, as indicated by the illuminated distal, middle, and proximal phalanges [areas 2 and 7]. However, in the pointing up grasp, the other fingers were also slightly illuminated, indicating their flexion. This distinction between finger positions is the primary difference between the two grasps. 36 5. Results Figure 5.3.: Pressure distribution on the STAG sensor in the human hand during a cylindrical grasp on a glass. Figure 5.4.: Pressure distribution on the STAG sensor in the human hand during a precision grasp on a lipstick and pressure concentration percentage by area. 37 5. Results Figure 5.5.: Relation between the average pressure in the human hand during a precision grasp across the entire set of objects and their respective weights and pressure concentration percentage by area. Figure 5.6.: Pressure distribution on the STAG sensor in the human hand during a lateral grasp on a key and pressure concentration percentage by area. 38 5. Results Figure 5.7.: Relation between the average pressure in the human hand during a lateral grasp across the entire set of objects and their respective weights. Figure 5.8.: Pressure distribution on the STAG sensor in the human hand during a pointing down grasp and pressure concentration percentage by area. 39 5. Results Figure 5.9.: Pressure distribution on the STAG sensor in the human hand during a pointing up grasp and pressure concentration percentage by area. 5.2.3. Method 3: Robotic vs. Human Hand: Similarities and Differences In this section, datasets from sessions 2 and 3 were analyzed to compare the similarities and differences between the behaviors of the robotic and human hands. The analysis focused on comparing areas of pressure application and understanding the distribution and strength of pressure edges across the entire hand. Neutral Position As discussed in previous sections, calibrating the sensors was crucial for analyzing the grasps and eliminating extraneous noise. Both sensors were constructed using the same materials and manufacturing processes; however, one was placed on a robotic hand and the other on a human hand. In both cases, intermediary cotton gloves were used, with the sensors directly glued to them. The gloves differed in size, with the robotic hand being larger than the human hand. Consequently, the STAG on the robotic hand was especially tighter compared to the human hand. These differences resulted in observable variations in the pressure distribution of the neutral positions of both hands. The following image illustrates the variation in pressure distribution between the sensors. 40 5. Results Figure 5.10.: Pressure distribution on the STAG sensor during the neutral position in the human hand and the robotic hand, from left to right. Pressure Concentration The concentration of pressure in different regions of the human and robotic hands during cylindrical, lateral, and precision grasps was studied to identify differences and similarities. This is illustrated in image 5.10. The first grasp analyzed was the cylindrical grasp. As discussed in the previous chapter, the pressure distribution for this grasp revealed that all fingers of the hand were significant contributors, as expected for the human hand. The image 5.11 shows the pressure distribution while grasping a large plastic bottle cylindrically, comparing both human and robotic hands. In the case of the robotic hand, pressure is more localized to specific areas, particularly the upper palm and index finger, unlike the more evenly distributed pressure observed with the human hand. This Figure 5.11.: Pressure concentration in different regions of the human and robotic hands during cylindrical, lateral, and precision grasps. 41 5. Results discrepancy is likely due to the robotic fingers’ lack of flexibility, which prevents them from fully adapting their shape to the object being grasped. The second grasp examined was the precision grasp. In the human hand, pressure was primarily distributed among the index, middle, and thumb fingers, as previously described. In contrast, the robotic hand exhibited a more localized pressure distribution centered on the index and ring fingers. The limited range of motion in the robotic thumb caused it to rely on a few specific contact points. As a result, while the robotic hand had a larger support area on the thumb, it only utilized a few contact points, leading to localized pressure distribution. This could result in faster wear in these areas and potentially impact the effectiveness of the grasp. The third and final grasp analyzed was the lateral grasp. In the human hand, pressure was distributed mainly among the index, middle, and thumb fingers, as previously explained. For the robotic hand, the pressure concentration followed a similar pattern, suggesting that the lateral grasp closely resembles the human hand’s performance regarding pressure distribution. Figure 5.12.: Pressure distribution on the STAG sensor during a cylindrical grasp of a big bottle: human hand (left) vs. robotic hand (right). 42 5. Results Figure 5.13.: Pressure distribution on the STAG sensor during a precision grasp of a card: human hand (left) vs. robotic hand (right). Figure 5.14.: Pressure distribution on the STAG sensor during a lateral grasp of a lipstick: human hand (left) vs. robotic hand (right). 43 5. Results Pressure distribution The analysis of pressure distribution focused on understanding the distribution and strength of edges related to the pressure applied across the entire hand. The key variables used in this analysis were: 1. Mean Edge Strength: This variable represents the average value of the gradient magnitude, which reflects the strength of the edges at each point of pressure across the entire hand. A higher mean edge strength indicates that, on average, the hand exhibits more prominent edges or higher contrast between adjacent pressure points. 2. Standard Deviation of Edge Strength: This variable measures the variability in edge strengths. A higher standard deviation signifies a broader range of edge strengths within the hand, with some areas displaying very strong edges and others showing weaker or no edges. 3. Strong Edges Ratio: This metric represents the proportion of pressure points where the gradient magnitude exceeds a specified threshold. Analyzing these variables, as presented in Table 5.2 and the accompanying visuals, yields the following results. Mean Edge Strength Std Deviation Strong Edges Ratio Cylindrical grasp Human Hand 27.76 122.25 0.09964 Robotic Hand 23.89 112.58 0.09525 Lateral grasp Human Hand 16.83 71.62 0.09560 Robotic Hand 14.58 78.87 0.09094 Precision grasp Human Hand 14.71 76.08 0.09422 Robotic Hand 24.33 109.45 0.09818 Table 5.2.: Mean Edge Strength, Standard Deviation, and Strong Edges Ratio for pressure edges on human and robotic hands across different grasp types. The analysis of the cylindrical grasp suggests that the pressure distribution of the human hand during a cylindrical grasp has more pronounced edges with greater variability. In contrast, the robotic hand exhibits fewer edges but with sharper transitions. The most noticeable difference is attributed to the fingers: due to their lack of flexibility and softness, the pressure is localized and confined to a smaller area in the robotic hand. Similar to the cylindrical grasp, the pressure distribution of the human hand in the case of the lateral grasp shows more edges compared to the robotic hand. The visuals indicate that the pressure application area is significantly larger and more broadly spread in the human hand. The results for the precision grasp show that pressure is concentrated mainly in the thumb and index finger, resulting in a pressure distribution with fewer edges. Conversely, in the robotic hand, the palm plays a crucial role in pressure application. However, this appears to be more related to the flexion of the hand rather than the pressure data itself. 44 5. Results Figure 5.15.: Edges strength distribution on the STAG sensor during a lateral grasp of a lipstick: human hand (left) vs. robotic hand (right). Figure 5.16.: Edges strength distribution on the STAG sensor during a lateral grasp of a lipstick: human hand (left) vs. robotic hand (right). 45 2! Phase 2 (Months 4-5) 1. Designing and implementing a prototype that can acquire and process the information directly on board. Both the hardware and firmware need to be implemented. 2. The final version of the developed device(s)/subsystem (i.e., a power, processing, radar, wireless communication) 3. Proposing or optimizing the NN model increasing the number of classes and testing, measurements, and simulations for estimation of the accuracy of the algorithm. 4. In-field test, if possible identify and try to adapt the system to other application scenarios, i.e., drones, automotive, etc. Phase 3 (Month 6) 1. Finalizing the tests and optimizations. 2. Write final document and prepare presentation. Milestones By the end of Phase 1 the following should be completed: • A complete list of the components that the candidate will use. • Have a full characterization of the subsystem that must be designed to acquire data from the sensor. • Preliminary design of sensing devices and their PCB (including at least processor, sensors, PC interface), with preliminary in-field measurements. • Achieving preliminary evaluation of machine learning algorithms (i.e. CNN vs TCN). By the end of Phase 2 the following should be completed: • Evaluation of the system with in-field measurements. • The final version of the developed sensing device(s) • Testing and characterization of the developed boards/systems in the application scenario 3! By the end of Phase 3 the following should be completed: • Final design and in-field test. • Final Presentation • Final Report, including final results. 3 Project Organization During! the! thesis,! students! will! gain! experience! in! the! independent! solution! of! a! technical-scientific!problem!by!applying!the!acquired!specialist!and!social!skills.!The! grade! is! based! on! the! following:! student! effort;! thoroughness! and! learning! curve;! achieving!qualitative!and!quantitative!results!with!a!scientific!approach;!supporting! practical! findings! with! theoretical! background! and! literature! investigations;! final! presentation!and!report;!documentation!and!reproducibility.!All!theses!include!an! oral!presentation,!a!written!report!and!are!graded.!The!report!and!presentation!need! to!have!publication!grade!quality!to!achieve!a!good!grade.!Students!are!graded!based! on!the!official!ITET!grading!form1.!For!students!of!IIS!(Prof.!Benini)!a!special!grading! scheme!exists,!please!contact!your!supervisor!for!details!there.!Before!starting,!the! project! must! be! registered! in! myStudies! and! all! required! documents! need! to! be! handed!in!for!archiving!by!PBL.! 3.1 Laboratory+Rules+ The!students!agree!to!follow!the!lab!rules!set!by!PBL!staff,!for!detail!please!contact!us.! The!most!important!points!are:! • All!ETH!safety!regulations!need!to!be!followed2,!in!addition!to!ones!given!by! PBL!staff!! • No!device!in!the!lab!is!used!without!introduction!by!your!supervisor!or!PBL! staff! • No!device!leaves!the!lab!without!being!officially!borrowed,!this!is!done!by!PBL! staff!and!needs!your!Legi.! • Any!damage!to!devices!or!tools!needs!to!be!reported!immediately!to!PBL!staff.! • The!Lab-desk!is!clean!and!free!for!others!after!you!finished!your!task,!or!when! you! take! longer! breaks.! All! tools! are! correctly! sorted! into! their! drawers/cupboards!when!you!leave.! ! 1https://ethz.ch/content/dam/ethz/specialinterest/itet/department/Studies/Forms/Grading%20Form.xlsx= = 2https://ethz.ch/staffnet/en/service/safety-security-health-environment/sicherheit-inlaboren-und-werkstaetten/laborsicherheit.html= = 4! 3.2 Weekly Report There will be a weekly report/meeting held between the student and the assistants. The exact time and location of these meetings will be determined within the first week of the project in order to fit the students and the assistants schedule. These meetings will be used to evaluate the status and document the progress of the project (required to be done by the student). Beside these regular meetings, additional meetings can be organized to address urgent issues as well. The weekly report, along with all other relevant documents (source code, datasheets, papers, etc), should be uploaded to a clouding service, such as Polybox and shared with the assistants. 3.3 Project Plan Within the first month of the project, you will be asked to prepare a project plan. This plan should identify the tasks to be performed during the project and sets deadlines for those tasks. The prepared plan will be a topic of discussion of the first week’s meeting between you and your advisers. Note that the project plan should be updated constantly depending on the project’s status. 3.4 Final Report and paper PDF copies of the final report written in English are to be turned in. Basic references will be provided by the supervisors by mail and at the meetings during the whole project, but the students are expected to add a considerable amount of their own literature research to the project ("state of the art"). 3.5 Final Presentation There will be a presentation (15 min presentation and 5 min Q&A for BT) at the end of this project in order to present your results to a wider audience. The exact date will be determined towards the end of the work. References: Will be provided by the supervisors by mail and at the meetings during the whole project. Appendix B Declaration of Originality 55 Appendix C Costs and Materials This section provides a detailed bill of the materials for both STAGs, fitted on the human and robotic hands, and the readout PCBA and the discovery board STM32. The STAG BOM C.1 outlines the necessary materials required for assembling two STAGs, including various electronic components, adhesives, and other supplies. Each entry specifies the component, its quantity per assembly, the supplier, and the cost details. The PCBA BOM C.2 lists the components essential for constructing the readout circuit. It includes resistors, capacitors, connectors, and integrated circuits, detailing the quantity per assembly, provider, and associated costs. Additionally, the details for acquiring the STM32 discovery board are provided in C.3. 1Purchase link: https://www.adafruit.com/product/641 2Purchase link: https://www.digikey.ch/short/q3nvwcn7 3Purchase link: https://www.adafruit.com/product/1361 4Purchase link: https://www.digikey.ch/short/0jdf4t8r 5Purchase link: https://www.migros.ch/en/product/704504700000 6Purchase link: https://www.stoffhandschuhe.ch/Arbeitshandschuhe/Baumwollhandschuhe/ Extra-duenne-Baumwollhandschuhe-weiss-12er-Pack::23.html?MODsid 7Purchase link: https://www.digikey.ch/short/3mz02twp 8Purchase link: https://www.digikey.ch/short/vznqv1b0 9Purchase link: https://www.digikey.ch/short/rzzrrpfh 57 C. Costs and Materials Component Quantity per STAG Provider Cost per unit (CHF) Total Cost (CHF) Stainless Conductive Thread1 2 rolls Adafruit 8.80 35.20 Double-sided tape2 1 roll Digikey 26.88 53.76 Velostat31 sheet Adafruit 4.40 8.80 Double-sided tape4 1 sheet Digikey 6.34 12.68 Cling film, Tangan nº115 1/2 roll Migros 3.60 3.60 Cotton glove61 piece Stoffhandschuhe 2.00 4.00 Female crimp termination7 70 units Digikey 0.085 11.90 Connector housing8 2 units Digikey 5.38 21.52 Shrink Tube92 m Digikey 2.02 2.02 Total cost per 2 STAGs 153.48 Table C.1.: Bill of Materials for two STAGs 10Purchase link: https://www.digikey.ch/short/c8bv51v0 11Purchase link: https://www.digikey.ch/short/7nwmrtqt 12Purchase link: https://www.digikey.ch/short/wmtfvd0h 13Purchase link: https://www.digikey.ch/short/pmm7wrf3 14Purchase link: https://www.digikey.ch/short/839jm34m 15Purchase link: https://www.digikey.ch/short/02vjn8d5 16Purchase link: https://www.digikey.ch/short/zbhb7cp2 17Purchase link: https://www.digikey.ch/short/4ht21fvz 18Purchase link: https://www.digikey.ch/short/h3nptq88 19Purchase link: https://www.digikey.ch/short/c3mcvz8v 20Purchase link: https://www.digikey.ch/short/qv4htv03 21Purchase link: https://www.digikey.ch/short/3bb0vn13 22Purchase link: https://www.digikey.ch/short/rf00dwqv 23Purchase link: https://www.digikey.ch/short/rqc4fvd8 24Purchase link: https://www.digikey.ch/short/c8bv51v0 58 C. Costs and Materials Component Quantity per PCBA Provider Cost per unit (CHF) Total Cost (CHF) Resistor 35.7kΩ10 1 Digikey 0.09 0.09 Resistor 1.0 kΩ11 32 Digikey 0.093 2.976 Resistor 3.0 kΩ12 32 Digikey 0.017 0.544 Capacitor 100nF13 11 Digikey 0.023 0.253 Capacitor 1 µF14 2 Digikey 0.09 0.18 Potentiometer15 1 Digikey 4.34 4.34 Connector for accelerometer16 1 Digikey 1.14 1.14 Connector for boards17 2 Digikey 1.73 LDO18 1 Digikey 1.42 1.42 Inverter19 1 Digikey 0.26 0.26 LTC6357 OA20 8 Digikey 7.38 59.04 SPDT21 8 Digikey 4.43 35.44 Decoder22 2 Digikey 0.73 1.46 Multiplexer 16:123 2 Digikey 7.81 15.62 Total cost per 1 PCBA 122.763 Table C.2.: Bill of Materials for customized PCBA Component Quantity Provider Cost per unit (CHF) Total Cost (CHF) STM32F769IDISC124 1 Digikey 77.88 77.88 Table C.3.: Component details for PCBA 59 Appendix D File Structure This section provides a detailed description of the project’s directory structure, highlighting the main files and their purposes. It was saved on the ETH server and organized as follows: / 01_Python_files ....................... The source files of the project report. 01_Data_frames_visualization STAG_GUI_PyQt.py .........................Visualization of data frames 02_Networks .................... Project 3 for obtaining the ONNX network 03_Square_sensor .................... Initial testing of the material’s sensor 04_Demo_Classification...Visualization of the predictions and data frames STAG_Demo_real_time STAG_Demo_sensor_file 05_LoadAndSave_realData...............Transferring the log files to Matlab 06_Calibration..............................Files used for data calibration 02_STM32_projects Demos..........................Project 1 for visualization of the data frames GUI_test ...................... Project 2 for Predictions and Classifications. 03_Dataset Data_frames........................................Extracted Data frames Files_log Session1 Session2 Session3 dataset.mat Videos..............................Video-recordings of the data extraction 04_Matlab_analysis........Useful MATLAB files for analysis and comparisons 60 D. File Structure STM32 Projects For working on this projects, use the programa STM32CubeIDE version 1.14.1. The easiest way to import this project is creating a new project from the .ioc file, which will automatically add the neccessary drivers. Python files This lists the installed python packages and versions. However, it is not absolutely necessary to install it with exactly the same versions. Consult this list if there are some broken dependencies in your environment. Required packages: Python 3.8.1 numpy 1.18.1 pytorch 1.4.0 CUDA version imbalanced-learn 0.6.2 scikit-learn 0.22.1 scipy 1.4.1 Imported standard packages: argparse collections datetime os random re shutil sys time 61 F. Template of the hands Figure F.1.: Robotic Hand base: This image shows the template for the Velostat base used in the robotic hand, including the arrangement and structure of the electrode matrix. Scale, 1:0.8 68 F. Template of the hands Figure F.2.: Human Hand Base: This image shows the template for the Velostat base used in the human hand, including the holes for the electrode matrix. Scale, 1:1 69 F. Template of the hands Figure F.3.: Hand Map with Labeled Regions: This image provides a detailed map of the hand, highlighting and labeling the different regions. Each numbered section corresponds to a specific area of the hand, aiding in the analysis and interpretation of pressure distribution across various grasps. 70 Appendix G Detailed Photographs of the Integrated System Components Figure G.1.: Customized board as readout circuit designed by previous student [1], front view 71 G. Detailed Photographs of the Integrated System Components Figure G.2.: Customized board as readout circuit designed by previous student [1], back view Figure G.3.: STM32F769I-DISC1 development board [4], front view 72 G. Detailed Photographs of the Integrated System Components Figure G.4.: STM32F769I-DISC1 development board [4], back view 73 G. Detailed Photographs of the Integrated System Components 74 Bibliography [1] X. Wang, F. Geiger, V. Niculescu, M. Magno, and L. Benini, “SmartHand: Towards Embedded Smart Hands for Prosthetic and Robotic Applications,” Master’s thesis, ETH Zürich, 2021, [Online; accessed 5-Feb-2024]. [Online]. Available: https://arxiv.org/abs/2107.14598 [2] Mia Hand - Upper limb prosthesis, [accessed 20-May-2024]. [Online]. Available: https://www.mia-hand.com/en/ [3] Datasheet: Mia Hand Series, provided by Pensilia s.r.l. [4] STM32, 32F769IDISCOVERY Dataset. [5] Adafruit, “Pressure-sensitive conductive sheet of velostat,” [bought at https://www.adafruit.com/product/1361]. [6] A. Fatema, I. Kuriakose, D. Devendra, and A. M. Hussain, “Investigation of the Mechanical Reliability of a Velostat-Based Flexible Pressure Sensor,” in 2022 IEEE International Conference on Flexible and Printable Sensors and Systems (FLEPS) At Vienna, Austria, 2022. [7] T. d’Alessio, Measurement errors in the scanning of piezoresistive sensors arrays, ser. Sensors and Actuators A: Pyhsical. Elsevier, Jan. 1999. [8] K. Jong-Seok, K. Dae-Yoing, and C. Byong-Deok, “High-accuracy, compact scanning method and circuit for resistive sensor arrays,” 2016, [accessed 20-May-2024]. [Online]. Available: https://doi.org/10.3390/s16020155 [9] Z. Xuefeng and Y. Xiangdong, “Zero potential method measurement error analysis for networked resistive sensor arrays,” in IET Science, Measurement Technology, 2017. [10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” ARXIV, 2015, [accessed 20-Apr-2024]. 75 Bibliography [11] E. Stevens, L. Antiga, and T. Viehmann, Deep Learning with Pytorch. Manning Shelter Island, 2020. [12] A. Géron, Hands-On Machine Learning with Scikit-Learn, Keras and Tensor Flow. O’Reilly Media, Inc., 2019. [13] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. The MIT Press, 2017. [14] User Guide: Mia Hand Series, provided by Pensilia s.r.l. [15] Mechanical Specifications: Mia Hand Series, provided by Pensilia s.r.l. [16] L. Mooney, “Velostat for home-made pressure sensors,” 2022 Tech Notes, 2022. [17] R. Dorf and J. Svoboda, “Operational amplifiers,” in Introduction to Electric Circuits. Joh Wiley and Sons, 2010. [18] M. K. Saini, “Convolution in signals and systems,” Tutorials point, 2023. [19] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” Cornell University, Tech. Rep., 2018. [20] O. Kroemer, C. Lampert, and J. Peters, “Learning dynamic tactile sensing with robust vision-based training,” IEEE Robotics, Tech. Rep., 2011. [21] H. Liu, Y. Yu, J. Gu, and F. Sun, “Visual–tactile fusion for object recognition,” in Transactions on Automation Science and Engineering. IEEE, 2017. [22] Y. Han, Q. Xu, and F. Wu, “Design of wearable hand rehabilitation glove with bionic fiber-reinforced actuator,” IEEE Journal of Translational Engineering in Health and Medicine, Tech. Rep., 2022. 76