Compact Functional Testing for Neuromorphic Computing Circuits

1 Compact Functional Testing for Neuromorphic Computing Circuits Sarah A. El-Sayed, Theofilos Spyrou, Luis A. Camu˜ nas-Mesa, and Haralampos-G. Stratigopoulos, Member, IEEE Abstract—We address the problem of testing Artificial Intelligence (AI) hardware accelerators implementing Spiking Neural Networks (SNNs). We define a metric to quickly rank available samples for training and testing based on their fault detection capability. The metric measures the inter-class spike count difference of a sample for the fault-free design. In particular, each sample is assigned a score equal to the spike count difference between the first two top classes. The hypothesis is that samples with small scores achieve high fault coverage because they are prone to misclassification, i.e., a small perturbation in the network due to a fault will result in these samples being misclassififed with high probability. We show that the proposed metric correlates with the per-sample fault coverage and that retaining a set of high-ranked samples in the order of ten achieves near perfect fault coverage for critical faults that affect the SNN accuracy. The proposed test generation approach is demonstrated on two SNNs modelled in Python and on actual neuromorphic hardware. We discuss fault modeling and perform an analysis to reduce the fault space so as to speed up test generation time. Index Terms—Neuromorphic computing, spiking neural networks, testing, fault modeling, fault simulation. I. INTRODUCTION The design of Artificial Intelligence (AI) hardware accelerators has attracted significant attention in recent years. On one hand, AI hardware accelerators are essential for efficient processing of high-dimensional AI workloads, i.e., Deep Neural Networks (DNNs). On the other hand, technologies such as edge computing are witnessing growing interest with several new use cases. Designing lightweight AI hardware accelerators, i.e., having low energy consumption and form factor, for performing inference directly on edge devices can offer significant advantages compared to performing the computation centrally in the cloud. In particular, performing Manuscript received July 22, 2022; revised October 31, 2022; accepted November 6, 2022. This work was supported by the ANR RE-TRUSTING project under Grant ANR-21-CE24-0015-03 and by Junta de Andaluc´ ıa under contract US-1260118 (Neuro-Radio). The work of T. Spyrou was supported by the Sorbonne Center for Artificial Intelligence (SCAI) through Fellowship. L. A. Camu˜ nas-Mesa was funded by the VI PPIT through the Universidad de Sevilla. This article was recommended by Associate Editor G. Di Natale. (Corresponding author: Haralampos-G. Stratigopoulos.) Sarah A. El-Sayed was with Sorbonne Universit´ e, Centre National de la Recherche Scientifique (CNRS), LIP6, 75005 Paris, France. She is now with Minia University, 61519 Minia, Egypt (e-mail: [email protected]). T. Spyrou and Haralampos-G. Stratigopoulos are with the Sorbonne Universit´ e, CNRS, LIP6, 75005 Paris, France (e-mail: theofilos.sp[email protected]; [email protected]). Luis A. Camu˜ nas-Mesa is with the Instituto de Microelectr´ onica de Sevilla (IMSE-CNM), Consejo Superior de Investigaciones Cient´ ıficas (CSIC), Universidad de Sevilla, 41092 Sevilla, Spain (e-mail: [email protected]). Digital Object Identifier 10.1109/TCAD.2023.XXXXXXX computations closer to the user’s side allows autonomous operation, saves energy and bandwidth by reducing data transfer, circumvents cloud service latency, and assures data privacy. High-volume manufacturing of Application-Specific Integrated Circuit (ASIC) AI hardware accelerators is foreseen in the near future. On-die neural networks have been explored in the past for building an on-die “test brain” that classifies chips as functional or faulty [1]. The “inverse” problem, i.e., how to efficiently test AI hardware accelerators, however, is an emerging problem [2]–[4]. In general, existing and proven test methods for traditional computing devices can be portable to AI hardware accelerators. Nevertheless, the unique architectural features of AI hardware accelerators make these test methods less efficient and give rise to new test challenges. For instance, AI hardware accelerators usually consist of multiple identical cores, e.g., the Multiply-and-Accumulate (MAC) units (also referred to as Processing Elements (PEs)), which are too small to implement traditional Design-for-Test (DfT) techniques, e.g., scan test, with reasonable overhead. Another characteristic of AI hardware accelerators is that they are memory-hungry, with the memory storage being dominated by the synapse weights which can be in the order of millions. Testing large embedded memories with today’s Memory Built-In Self-Test (MBIST) tools can pose large Power, Performance, and Area (PPA) penalties. Traditional fault models, such as stuck-at, delay, and cellaware fault models, can be reused as well in the context of AI hardware accelerator testing, but emerging in-memory computing architectures based on memristive crossbars [5] or Spiking Neural Networks (SNNs) [6]–[8] for example, require new fault models. Moreover, fault models for AI hardware accelerators could be defined in software at a higher-abstraction behavioral-level, i.e., variations in neuron outputs and synapse weight values [9], aiming at speeding up test generation. This is because software and hardware implementations of neural networks closely match together [10]. Recent fault injection experiments in AI hardware accelerators [10]–[16] have shown that there is some inherent fault tolerance since many faults are benign, i.e., they are masked before their effect propagates to the output layer or they produce an output change that is tolerable. This is expected based on the analogy with biological neural networks which have remarkable fault tolerance capabilities. Some faults, though, are still critical and evidently impact the performance, and test efforts can focus on these critical faults to reduce test time [4]. Test methods that take into account the architectural par- 2 ticularities of AI hardware accelerators have started surfacing recently. A brief overview of the state-of-the-art is provided in Section II. In this work, we focus on SNN hardware accelerators for which the literature on fault modeling, fault injection experiments, test and reliability methods is still fairly limited, as it will also be discussed in Section II. More specifically, we propose a method for generating compact functional test sets. A functional test is an input, i.e., an image for Convolutional Neural Networks (CNNs) used in computer vision, that aims at detecting the presence of a maximum number of hardware-level faults and their combination thereof, such that a compact set of few functional tests can detect all faults. In the context of an AI hardware accelerator device, a fault is detected if the predictions of the nominal fault-free and faulty devices differ. The prior art on functional test generation methods for AI hardware accelerators will be described in detail in Section VIII. This article makes the following contributions: •We use available samples from the training and testing sets as functional tests. We propose a fault-agnostic metric that assesses the fault detection capability of each sample based on how close the sample is to being misclassified. The underlying hypothesis is that such corner samples lying close to the classification hyper-boundary will be classified differently if a fault occurs, thus producing a distinguishing output compared to the nominal fault-free case. •We confirm experimentally that the fault detection capability of a sample is directly correlated with the ranking of the sample based on the chosen metric. •Based on this metric, we cumulatively add test samples according to their ranking and demonstrate that we can generate compact functional test sets with the size of a few tens of tests that achieve 100% fault coverage of critical faults. A good percentage of benign faults can also be detected, which is beneficial since they are considered as reliability hazards. Such compact functional test sets can be used not only for fast post-manufacturing testing, but they can also be stored on-chip to be re-used for periodical online test in idle times. •We discuss fault modeling and we perform fault injection experiments for different fault models identifying critical fault types and fault locations. These findings, corroborated on various SNN models, can be used to reduce the effective fault space and, thereby, the fault simulation time for computing fault coverage. •The proposed compact functional test set generation method is demonstrated for SNNs at behavioral-level and on actual neuromorphic hardware. The rest of the article is structured as follows. In Section II, we discuss the state-of-art on testing AI hardware accelerators including SNNs. In Section III, we provide a brief overview of SNNs. In Section IV, we introduce the proposed functional test generation algorithm. In Section V, we present the SNN case studies. In Section VI, we discuss fault modeling, fault simulation frameworks, and fault space reduction. Experimental results are presented in Section VII. In Section VIII, we discuss, in more detail, the prior art on functional test generation for AI hardware accelerators and provide comparisons to our work. In Section IX, we discuss general properties of the proposed functional test generation approach and future work directions. Finally, Section X concludes this article. II. PRIOR ART ON TESTING AI HARDWARE ACCELERATORS DfT methods suited for AI hardware accelerators based on large arrays of small PEs are proposed in [2], [17]. Test generation algorithms aiming at creating a compact set of functional tests that can detect the presence of faults are proposed in [18], [19], [20]. Symptom detectors that detect some anomaly in intermediate nodes, i.e., high neuron activation, are proposed in [11], [21]–[23]. Selective Triple Modular Redundancy (TMR) applied to the most critical neural network layers is proposed in [22], [24], [25]. Algorithmic-based error detection and correction methods using checksum arithmetic are discussed in [13], [26]–[29]. On-line test methods are proposed in [30] based on Software Test Libraries (STL), in [31] based on a simplified metric of dynamic power consumption, and in [32] based on encrypting weights in the memory with an encryption algorithm that spreads single bitflips extending them to multiple bit-flips and checking if the padding bytes used for the encryption to work properly are correctly decrypted. Specifically now for SNNs, behavioral-level fault modeling is discussed in [33], [34]. Fault injection experiments at behavioral-level are described in [22], [23], [33], [35], [36], while fault injection experiments onto actual neuromorphic hardware are described in [16]. In [22], [23], symptom detectors are designed for the two main catastrophic fault mechanisms in SNNs. In particular, for synaptic faults it is proposed to perform weight bounding [23]. If the weight drifts to a value greater than a threshold, then it is replaced with a pre-defined value, i.e., zero or maximum weight value from the nominal SNN. For neuron saturation faults, if an output spike train is observed while there is no input activity, then spike generation is disabled and the neuron is silenced [22]. A similar condition is proposed in [23] where spike generation is disabled if the membrane voltage stays above the threshold for more than two clock cycles. A BIST technique for biological spiking neurons that checks for the appearance of all expected firing patterns is proposed in [37]. Inherent fault resilience of SNNs when trained with different algorithms is studied in [35], in addition to showing how to modify a training algorithm to improve fault tolerance. Fault-tolerance schemes to mitigate memory failures in SNNs are proposed in [36], [38]. And finally, a functional test generation method for SNNs is proposed in [20], which will be discussed in more detail in Section VIII. III. SNNS Unlike traditional Artificial Neural Networks (ANNs), information in SNNs flows as spike trains propagating across network layers asynchronously, which resembles the biological brain operation. SNNs form the basis of neuromorphic computing pioneered by Carver Mead in the 1980s [39]. One of the least computationally complex, and hence hardware-friendly, 3 Fig. 1: Functional test generation. The street images are from [41]. The chip image is from [1]. neuron model used to implement SNNs is the Integrate & Fire (I&F) model [40]. The I&F neuron integrates spikes from incoming synapses by increasing its membrane’s potential, and when the potential exceeds a threshold it fires a spike of its own that propagates through synapses to other neurons. It also resets the threshold so as to be able to fire again. The I&F neuron can be equipped with two additional brain-inspired functionalities, namely a refractory period, i.e., it is allowed to fire only if a certain time is elapsed since the last output spike, and a leakage behavior, i.e., the membrane potential decreases between two consecutive input spikes. The synapse operation in SNNs also resembles the biological synapse operation where a synapse receives spikes and, in turn, stimulates the membrane potential of post-synaptic neurons via a current that is proportional to the synapse weight. As for decision making methods at the output of SNNs, the most common method is based on evaluating the firing rate of output neurons, i.e., number of spikes over an observation period. Other coding schemes have been used, such as the time-to-first-spike. From a hardware perspective, there is a belief that SNNs offer faster inference and lower energy consumption compared to ANNs. However, there are still challenges facing SNNs such as the complexity of training. In general, the discussion on the relative performance between ANNs and SNNs is not trivial due to the different input type, i.e., sequence of static frames versus continuous-time event flow. For an extensive discussion on SNNs and their hardware implementation, the readers are referred to [6]–[8]. IV. PROPOSED FUNCTIONAL TEST GENERATION ALGORITHM FOR SNNS Functional test generation aims at either identifying or generating new input samples that are capable of sensitizing the fault and propagating its effect to the output, leading to a different prediction with respect to that of the nominal fault-free network. As shown in Fig. 1 using as an example an image recognition cognitive task, these samples could be original images from training and testing sets [18], [19], adversarial examples generated from original images [20], or synthetic images generated from original images [41], [42]. These related works will be described in more detail in Section VIII. The proposed algorithm for SNNs selects tests from the set of available samples in the training and testing sets. Let us consider an SNN employed for an N-class classification cognitive task. The SNN has Nneurons in the output layer each corresponding to one class. We consider that the SNN uses the firing rate as classification criterion, i.e., the winning class is the one whose corresponding neuron produces the largest number of spikes within a given duration interval. Let us consider also a set of input samples of cardinality M. For a given sample ti,i= 1,· · · , M, let nj idenote the spike count for output neuron j,j= 1,· · · N. We rank the values nj ifrom high to low, resulting in the ordered set {n(1) i, n(2) i, n(3) i,· · · }, i.e., the neuron that produces n(1) i spikes corresponds to the top-1 class, the neuron that produces n(2) ispikes corresponds to the top-2 class, and so forth. Then, we define the margin Xi=n(1) i−n(2) i,(1) i.e., Xiis the difference in spike count between neurons corresponding to the top-1 and top-2 classes. The quality metric of sample tiis defined by the score qi=1 Xi (2) The rationale is that samples with high scores (or, equivalently, small margins) are likely to be distinguishing samples, i.e., they are prone to producing different top-1 class predictions for the nominal and faulty networks. This is based on the intuition that when the first top classes are close in terms of firing rate, the network has low confidence in its decision and it is likely that a fault will alter the class ranking for this particular sample. In other words, this sample lies close to the multi-class classification hyper-boundary and is likely to be misclassified when a fault occurs. Let us consider now the ranking of samples based on their scores from high to low, resulting in the ordered set {t(1), t(2),· · · , t(i),· · · , t(M)}, where t(1) is the sample with the highest score, and so forth. A functional test set of size Tcan be generated by considering the first Thigher-score samples in this ordered set. As we will see in our experimental results, the score in Eq. (2) directly correlates with the per-sample fault coverage, i.e., the samples fault coverage is shown to increase linearly with the samples score. To this end, the proposed algorithm first performs Minferences, i.e., one inference per available sample, on the nominal network, then ranks the samples according to their score. Next, starting from the top ranked sample and sequentially adding the next top ranked samples, it evaluates the cumulative fault coverage. In each step, the detected faults are dropped from the fault list. Let Nuf (i) denote the number of undetected faults at the beginning of iteration iwhere the i-th ranked sample is examined, i.e., Nuf (1) = Kfor a fault model of size K. The algorithm stops adding samples when the fault coverage saturates. Our experimental results show that the size of the resultant test 4 set needs to be in the order of few tens of samples for reaching 100% fault coverage for critical faults. For a test set of cardinality T, the total number of inferences which dominates the test generation time is Ninf =M+ T X i=1 Nuf (i)(3) where the first term corresponds to test generation and the second term to fault coverage evaluation. Finally, for a fault model of size K, let Fkdenote fault k. We define the following indicator function for test ti Iti(Fk) = 1 : Fkis detected 0 : otherwise (4) where detection means that the responses of the nominal faultfree network and the faulty network with fault Fkinjected differ, i.e., a different class is predicted. The fault coverage of test tiindicates the percentage of faults detected by this particular test. It is defined as FC(ti) = PK k=1 Iti(Fk) K(5) Considering a test set of cardinality Tdenoted by {t1,· · · , tT}, its global fault coverage is defined as FC =PK k=1 min(1,PT i=1 Iti(Fk)) K(6) V. SNN CASE STUDIES As case studies, we use three convolutional SNNs performing three different cognitive tasks, namely an SNN trained to classify the N-MNIST dataset [43], an SNN trained to classify IBM’s DVS gesture dataset [44], and an SNN trained to classify the 4 poker card symbols [45]. Their architectures are shown in Figs. 2, 3, and 4, respectively. In each case, the winning class is declared based on the most triggered neuron at the output layer. In CNNs, a convolutional layer is composed of several feature maps each forming a channel [46]. A feature map is a plane of neurons where each neuron is connected to the outputs of spatially nearby neurons contained in a lowerdimensional plane of the prior layer, referred to as a receptive field. Each neuron has a different receptive field located at different coordinates of the prior layer. In a given feature map, all neurons are constrained to share the same synaptic weights, whereas synaptic weights change from one feature map to another. Convolutional layers may be alternated with subsampling layers which are used to down-sample the output of the preceding convolutional layer. There are different types of sub-sampling, such as max pooling and average pooling. Max pooling captures the maximum value of the receptive field and processes it to the output, whereas average pooling calculates the average value. CNNs allow synapse reuse and reduce the number of synapses compared with a Fully Connected (FC) network. The last convolutional or sub-subling layer is flattened, i.e., each feature map is essentially a single neuron. Thereafter, there is a number of FC layers to perform the 0 1 2 9 Convolution layer #1 14x14 Convolution layer #2 5x5 Convolution layer #3 1x1 Fully Connected Layer n = 50 Output n = 10 6 Channels 16 Channels 120 Channels Input 34x34 2 Channels Flattened Data: n = 120 5 5 7 7 5 5 Stride = 2 Stride = 2 SC1 SC2 SC3 SF5 SF4 Fig. 2: Architecture of the N-MNIST SNN. A Arm Roll Hand Clap Air Guitar Other Convolution layer #1 32x32 Convolution layer #2 16x16 Fully Connected Layer n = 512 Output Layer n = 11 2 Channels Flattened Data: n = 2048 Padding=2 SC1 SC2 SF4 SF3 Input 128x128 Pooling layer #1 32x32 Pooling layer #2 16x16 Pooling layer #3 8x8 4 4 5 5 3 3 16 Channels 32 Channels SP0 SP1 SP2 Padding=1 2 22 2 Fig. 3: Architecture of the IBM DVS Gesture SNN. classification. In FC networks, the neurons of a new layer are connected via synapses to the outputs of all neurons in the prior layer. The N-MNIST and IBM’s DVS gesture SNNs are designed and simulated in Python, while the poker card-symbols SNN is running on an FPGA-based neuromorphic hardware platform. A. SNN models in Python The SNN models designed in Python are based on the opensource Spike LAYer Error Reassignment (SLAYER) [47] and PyTorch [48] frameworks, and run on a Graphics Processing Unit (GPU) accelerator. They are trained using batch learning with a variation of back-propagation, incorporating the dropout technique [49]. The SLAYER framework uses the Spike Response Model (SRM) [50] to model spiking neurons, which is a generalized form of the ubiquitous I&F model. 1) N-MNIST SNN: The N-MNIST dataset [43] is a neuromorphic version of the handwritten digits MNIST dataset. The SNN achieves a 98.08% classification accuracy on the testing set, which is comparable to the performance of state-of-the-art level-based ANNs. 2) IBM DVS Gesture SNN: The IBM’s DVS gesture dataset [44] is created directly in spiking form using a 128 ×128 Dynamic Vision Sensor (DVS). It consists of 29 individuals performing 11 hand and arm gestures in front of the DVS, such as hand waving and air guitar. The SNN performs with an 82.5% accuracy on the testing set. B. SNN hardware implementation The poker card symbols dataset is created by presenting a deck of poker cards in front of a DVS sensor. The events are recorded and processed to extract a 32 ×32 pixel window 5 SC1 Convolution Layer #1 28x28 Input 32x32 6 Nodes 6 Channels 8 Nodes 8Channels 4 Nodes 4 Channels 4 Nodes 4Channels SP1 Pooling Layer #1 14x14 SC2 Convolution Layer #2 10x10 SP1 Pooling Layer #2 5x5 SC3 Convolution Layer #3 1x1 SC4 Convolution Layer #4 1x1 2 2 5 5 1 1 5 5 2 2 1 1 Fig. 4: Architecture of the poker card-symbols SNN. East Port West Port North Port South Port CONFIGURATIO N BLOCK Parameters Weights Kernel Memory Neuron Memory Rate Saturation Memory Input FIFO Output FIFO SPI Slave Input events Output events Configuration Parameters Address Calculation Block Leakage Counter Controller Block Pixel Array ROUTER ROUTER CONVOLUTIONAL UNIT Fig. 5: Configurable convolutional node. that shows only the centered symbol. The hardware implementation of the poker card-symbols SNN uses as a fundamental building block a generic event-driven configurable convolutional node that constitutes one feature map [45]. As shown in Fig. 5, each node consists of three main blocks, namely a convolutional unit that encompasses a 128 ×128 I&F neuron (pixel) array and is responsible for the convolution calculations, an internal configuration block that receives and interprets the configuration data of the convolutional node, and a router responsible for handling the transmission of the events from their origin to their destination according to a predefined routing scheme. Using this building block, the SNN of Fig. 4 is designed in VHDL and implemented on a Zynq®UltraScale+TM MPSoC ZCU104 FPGA board. VI. FAULT MODELING Fault models can be defined either at behavioral-level using an abstract representation of the neural network, e.g., a Python model, or at hardware-level. Performing test generation at behavioral-level rather than at hardware-level, is faster and the derived tests are still actual samples, i.e., images, which are directly portable to the hardware. The disadvantage is that the resultant fault coverage may deviate from the true one computed on the actual hardware implementation. However, as it will be discussed in Section VI-C, for both options the fault space quickly explodes even for small-size neural networks, making fault sampling strategies essential. In this regard, performing test generation at the hardware-level yields approximative fault coverage. Similar to testing of traditional chips, fault coverage metrics should be reported by specifying the used underlying fault model. A. Behavioral-level fault modeling For the purpose of this analysis, faults are inserted in the main operators that support the neural computational task, i.e., neurons and synapses, assuming that neurons and synapses can fail independently [51]. 1) Neurons: In [34], detailed transistor-level fault simulations were performed on a spiking neuron so as to deduce the possible output faulty behaviors. Main behavioral-level faults include: (i) dead neuron faults, i.e., the neuron halts spiking even in the presence of input activity; (ii) saturated neuron faults, i.e., the neuron fires spikes non-stop even at the absence of input activity; and (iii) timing variations, i.e., variations in the firing rate and the time-to-first-spike. Such neuron behavioral-level fault types are in agreement with recent works [20], [22], [23], [33] and encompass other fault types that have been considered in the past, such as stuck-at neurons and errors in the activation values [9]. 2) Synapses: Common synapse faults considered in the literature include disabled synapses, stuck-weight synapses or saturated synapse weights [20], [35]. However, this synapse fault model is not realistic from a hardware perspective since real-weighted synapses after model training in software are quantized and stored as digital words in an on-die memory. Herein, we use a hardware-aware synapse fault model where we quantize the weight value given the data type in hardware, i.e., 8-bit integer in the neuromorphic hardware used in this work, we perform bit-flips across bit positions and then we map back the binary value to a real value. This fault model essentially corresponds to synapse weight perturbation that could be major if one or more Most Significant Bits (MSBs) are flipped. 3) Fault injection framework: For the neuron behaviorallevel faults we use a fault injection framework described in [22], which is built on top of the SLAYER [47] and PyTorch [48] frameworks. Neuron faults are inserted by modifying the SRM model or by customizing the flow of computations in the frameworks. Synapse faults are inserted by modifying the weight values using as root-cause bit-flips as described above. Fault simulation is accelerated by creating a faulty SNN instance, then mapping it on a GPU to perform inference. B. Hardware-level fault modeling The neuromorphic hardware used in this work is reconfigurable storing all the main SNN parameters in ondie memories, including: (i) synapse weights; (ii) feature maps size and center-shift; (iii) neuron threshold, leakage, and refractory period; (iv) router parameters that configure 6 the architecture and direct spiking events; and (v) splitter parameters that define the input event copies to be sent to the first layer nodes. Each parameter has an 8-bit integer representation. We consider as fault model (i) single bit-flips for all parameters and across all bit positions and (ii) multiple bit-flips across the entire memory with different Bit Error Rate (BER) probabilities. C. Fault space reduction The common conclusion of several published fault injection and reliability experiments for ANNs [10]–[15] and SNNs [16], [20], [22], [23], [33], [35], [36] is that not all faults are equal. A large number of faults are either completely masked or they induce a negligible drop in the network classification accuracy. Such benign faults can be excluded to reduce the fault space and speed up fault simulation that is invoked several times during test generation. Including all faults can quickly make fault simulation intractable, even for small-size networks. For example, for the N-MNIST and IBM DVS Gesture SNNs the number of neurons is 1756 and 25099, respectively, and the number of synapses is 57488 and 1059616, respectively. Consequently, a prudent elimination of benign faults is required so as to avoid inadvertently excluding critical faults. In the context of this work, we define benign and critical faults in a more strict fashion. A fault is considered benign if the response of the nominal network and the network in the presence of the fault match on a sample-by-sample basis with a certain tolerance, e.g., a 0% tolerance requires an exact match between both responses. If the number of mismatched samples between the nominal network response and that of the faulty network is above the tolerance, the fault is considered critical. 1) Behavioral-level fault model: For SNNs, in [22], it is shown that training with dropout [49] offers proactive fault resilience against dead neuron faults and neuron timing variations. Dropout was originally proposed to prevent over-fitting and reduce the generalization error on unseen data. It has its roots on the observation that model combination, a.k.a. ensemble learning, nearly always improves performance. In this regard, it temporarily removes neurons and their associated synapse connections during training with some probability that could vary from one layer to another, which is equivalent to combining many “thinned” scaled-down models during one training session. Dropout achieves fault resilience because it equalizes the importance of neurons and distributes the neuron activity across the network. Thus, if a neuron becomes dead or it presents timing variations, the impact is inherently tolerated [22]. In contrast, a neuron saturation fault is not compensated because a neuron constantly firing is likely to severely perturb the propagating spike trains [22], [23]. As an example, Fig. 6 shows for the N-MNIST SNN the effect of neuron faults in the last 3layers on the network classification accuracy. Each row corresponds to a layer and each rectangle within a row corresponds to a neuron in this layer. The color of the rectangle shows the accuracy when this particular neuron is faulty based on the color map shown at (a) Dead neuron faults. (b) Saturated neuron faults. 0 10 20 30 40 50 60 70 80 90 100 Fig. 6: Effect of neuron faults on the classification accuracy of the N-MNIST SNN. the bottom of Fig. 6. As it can be seen, dead neuron faults are benign except in the last layer. Layers SC1 and SC2 are not shown due to their high neuron count. Similarly, timing variations are proven benign apart from the last layer [22]. The same findings were obtained for the IBM DVS Gesture SNN. As for synapse faults, fault space reduction becomes of utmost importance since the number of synapses can be in the order of several millions. In [20], only the last layer synapse faults were considered arbitrarily. In this work, we performed an analysis for the N-MNIST SNN to validate this hypothesis. The result is shown in Fig. 7. We assumed extreme synapse faults, namely dead synapses and positively saturated weights. Each box in Fig. 7 corresponds to one synapse connecting two neurons in two subsequent layers j−1and j, with the neuron numbers for layers j−1and jshown in the row and columns, respectively. The classification accuracy in the presence of a synapse fault is shown with the box color according to the color map at the bottom of Fig. 7. As it can be seen, only positively saturated weights appear to be critical, and this holds primarily for the synapses connecting the last two layers SF4- SF5, while few such critical synapse faults are observed for layers SC3-SF4. Synapse faults in previous layers have no impact and are excluded from Fig. 7. The reason behind this observation is that positive saturated weights could cause the post-synaptic neuron to always fire, i.e, saturate. In contrast, dead synapses just reduce the firing activity of the postsynaptic neuron, while we know that with dropout dead neuron faults are benign. These findings were corroborated on the IBM DVS Gesture SNN as well. Note that such extreme faults are not realistic from a hardware perspective as explained in Section VI-A2. Using the hardware-aware synapse fault model, i.e., bit-flips on quantized weights resulting in weight perturbation, has a smaller impact as shown in Fig. 8 for the IBM DVS Gesture SNN considering synapses connecting the last two layers SF3-SF4. We observe that only bit flips in the first two MSBs 7 and 6 can be critical, while for bit 5 the baseline accuracy is observed. Bit-flips for bit positions 0-4 are benign and are not shown in Fig. 8. From these fault injection experiments we can conclude about the criticality of fault types occurring at different locations in the network. Critical faults most likely can only be neuron saturation faults at any point in the network, dead 7 (a) Dead synapses between SC3-SF4. (b) Positively saturated synapses between SC3-SF4. (c) Dead synapses between SF4-SF5. (d) Positively saturated synapses between SF4-SF5. 0 10 20 30 40 50 60 70 80 90 100 Fig. 7: Effect of synapse faults on the classification accuracy of the N-MNIST SNN. neuron faults in the last two layers, and synapse faults in the last two layers. Still, many of these faults will end up being benign. The rest of the faults are benign with a very high probability and could be excluded from fault simulation. 2) Hardware-level fault model: A detailed reliability analysis for the poker card symbols SNN implemented on neuromorphic hardware was presented in [16]. The used fault model is the one described in Section VI-B and fault injection was performed on the actual hardware. Critical bit-flip faults were located across different network parameters and bit positions. For example, it was shown that bit-flips in the 4 Least Significant Bits (LSBs) of synapse weights are benign which can help to significantly reduce the fault space by nearly 50% since synapse weights occupy most of the memory size. VII. RESULTS Thanks to the acceleration of fault injection on hardware, i.e., using a GPU for the N-MNIST and IBM DVS Gesture SNNs and the FPGA-based SNN hardware accelerator for the (a) Bit 7 (MSB). (b) Bit 6. (c) Bit 5. 0 10 20 30 40 50 60 70 80 90 100 Fig. 8: Effect of bit-flip synapse faults between SF3-SF4 for the IBM DVS Gesture SNN. poker card symbols SNN, we considered a conservative fault space reduction only for the synaptic faults and only in the N-MNIST and IBM DVS Gesture SNNs. In particular, we considered only the synapse faults in the last two layers. The results are grouped per SNN in Figs. 9-11. 8 0 2 4 6 8 10 Margin Value 0 10 20 30 40 50 60 Sample Fault Coverage % Saturated Neuron Faults Dead Neuron Faults Synapse Faults (a) Relationship between the margin of a sample and the percentage of faults it covers. 10-1 100101 Tolerance (%) 102 103 104 105 # of Benign Faults 101 102 103 104 # of Critical Faults Saturated Neurons Dead Neurons Synapses Benign Critical (b) Change in number of critical & benign faults with the tolerance. (c) Cumulative fault coverage of benign synaptic faults. (d) Cumulative fault coverage of critical synaptic faults. 0 20 40 60 80 100 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 0.05% Tol 0.1% Tol 0.2% Tol 0.5% Tol 1% Tol (e) Cumulative fault coverage of benign dead neuron faults. 0 50 100 150 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 0.05% Tol 0.1% Tol 0.2% Tol 0.5% Tol 1% Tol (f) Cumulative fault coverage of critical dead neuron faults. 0 20 40 60 80 100 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 0.05% Tol 0.1% Tol 0.2% Tol 0.5% Tol 1% Tol (g) Cumulative fault coverage of benign saturated neuron faults. 0 20 40 60 80 100 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 0.05% Tol 0.1% Tol 0.2% Tol 0.5% Tol 1% Tol (h) Cumulative fault coverage of critical saturated neuron faults. Fig. 9: N-MNIST SNN. 0 5 10 15 20 Margin Value 0 10 20 30 40 50 60 70 Sample Fault Coverage % Saturated Neuron Faults Dead Neuron Faults Synapse Faults (a) Relationship between the margin of a sample and the percentage of faults it covers. Saturated Neurons Dead Neurons Synapses Benign Critical (b) Change in number of critical & benign faults with the tolerance. 0 10 20 30 40 50 60 70 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 0.5% Tol 1% Tol 2% Tol 4% Tol 10% Tol (c) Cumulative fault coverage of benign synaptic faults. 0 10 20 30 40 50 60 70 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 0.5% Tol 1% Tol 2% Tol 4% Tol 10% Tol (d) Cumulative fault coverage of critical synaptic faults. 0 10 20 30 40 50 60 70 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 1% Tol 2% Tol 4% Tol 10% Tol (e) Cumulative fault coverage of benign dead neuron faults. 0 10 20 30 40 50 60 70 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 1% Tol 2% Tol 4% Tol 10% Tol (f) Cumulative fault coverage of critical dead neuron faults. 0 10 20 30 40 50 60 70 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 1% Tol 2% Tol 4% Tol 10% Tol (g) Cumulative fault coverage of benign saturated neuron faults. 0 10 20 30 40 50 60 70 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 1% Tol 2% Tol 4% Tol 10% Tol (h) Cumulative fault coverage of critical saturated neuron faults. Fig. 10: IBM DVS gesture SNN. 0 5 10 15 Margin Value 0 10 20 30 40 50 60 70 Sample Fault Coverage % Single Bit-Flip Faults Multiple Bit-Flip Faults (a) Relationship between the margin of a sample and the percentage of faults it covers. 0 2 4 6 8 10 Tolerance (%) 101 102 103 104 # of Benign Faults 101 102 103 104 # of Critical Faults Single Bit-Flips Multiple Bit-Flips BER = 10-4 Benign Critical (b) Change in number of critical & benign faults with the tolerance. 0 5 10 15 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 2.5% Tol 5% Tol 10% Tol Critical Benign (c) Cumulative fault coverage of single bit-flip faults. 0 5 10 15 Number of Samples 0 20 40 60 80 100 Global Fault Coverage % 0% Tol 2.5% Tol 5% Tol 10% Tol Critical Benign (d) Cumulative fault coverage of multiple bit-flip faults at BER=10−4. Fig. 11: Poker card symbols SNN. 9 Figs. 9(a), 10(a), and 11(a) show the average per-sample fault coverage as a function of the margin value. Samples with identical margin values are grouped. For each sample, the average fault coverage is computed across all faults separately for each fault type, i.e., dead neuron, saturated neuron, and synapse faults for the N-MNIST and IBM DVS Gesture SNNs, and single bit-flip and multiple bit-flip faults with BER=10−4 for the poker card symbols SNN. Then, fault coverage values are averaged again across all samples within the same group. A clear trend is observed, i.e., samples with low margin, or equivalently with high score, tend to achieve a higher fault coverage. This proves the suitability of the chosen faultagnostic metric for ranking samples according to their fault detection capability. We also observe that the fault coverage rapidly increases when the margin decreases for small margin values, while for large margin values, fault coverage values are flattened or show a small fluctuation. This means that input samples with relatively large margin values may show small deviation in their fault detection capability, whereas for input samples with small margin values, the deviation can be significant. Note also that a flat fault coverage curve does not necessarily imply that adding more samples with larger margins to the test set is meaningless since they may be achieving the same fault coverage, but at the same time they may be detecting a different set of faults compared to the faults already detected by samples of higher ranking. Figs. 9(b), 10(b), and 11(b) show for the different fault types the number of benign and critical faults by varying the error tolerance. By increasing the tolerance from 0%, some critical faults are labelled as benign, thus the number of benign faults increases and the number of critical faults drops. The number of truly benign faults, i.e., none of the samples is misclassifed when these faults occur, is shown in Figs. 9(b), 10(b), and 11(b) for tolerance 0%. Fault type criticality for a given tolerance value can be assessed by examining the percentages of benign and critical faults. For tolerance 0% we observe that for all fault types, faults turn out to be more critical than benign, while for dead neuron faults, as expected, the critical and benign fault populations are more balanced. For all fault types, we observe that there is a tolerance value for which the critical and benign curves cross, which means that there is a certain tolerance value for which benign faults start outnumbering critical faults. In Figs. 9(c)-(h), 10(c)-(h), and 11(c)-(d) we rank the samples in an ascending order according to their scores, and we show the global cumulative fault coverage as we add samples in the test set for the different fault types. Two general conclusions can be drawn here. First, for the critical faults, the global cumulative fault coverage curves quickly reach 100%, while the convergence speed in general increases with the tolerance. This is because higher tolerance means less critical faults and, thereby, smaller test effort to reach 100% fault coverage. The number of samples required to achieve 100% coverage varies from one SNN to another and from one fault type to another. Second, for the benign faults, the global cumulative fault coverage is not expected to reach 100% since many faults are truly benign. For tolerance 0%, the benign fault coverage is by definition zero. We also observe that the % Tol Neuron Faults Synapse Faults Dead Saturated Benign Critical Benign Critical Benign Critical 0 - 150 - 33 - 1 0.2 92 36 33 33 1 1 0.5 92 36 17 9 1 1 (a) NMNIST SNN. % Tol Neuron Faults Synapse Faults Dead Saturated Benign Critical Benign Critical Benign Critical 0 - 32 - 25 - 66 2 13 28 31 19 35 35 4 13 28 31 15 32 28 (b) IBM DVS gesture SNN. % Tol Single Bit-Flips Multiple Bit-Flips Benign Critical Benign Critical 0 - 6 - 6 2.5 6 6 6 6 5 6 6 6 6 (c) Poker card symbols SNN. TABLE I: Number of samples needed to reach the maximum fault coverage for different fault types at different tolerance values. global cumulative fault coverage curves quickly saturate. The number of samples at the saturation point corresponds to the number of samples needed for detecting all benign faults that are not truly benign. Table I extracts some representative results from Figs. 9(c)-(h), 10(c)-(h), and 11(c)-(d), summarizing for each SNN and for three representative tolerance values the two most interesting quantities, i.e., the number of samples required to reach 100% fault coverage for critical faults and the number of samples required to detect all benign faults that are not truly benign. For example, for the N-MNIST SNN, we observe that the most highly ranked sample alone can detect all critical synapse faults for any tolerance value. To reach 100% fault coverage for critical dead neuron and saturated neuron faults for 0% tolerance, 150 and 33 samples are required, respectively. The required number of samples drops as the tolerance increases, i.e., for 0.2% tolerance 36 and 33 samples are required and for tolerance 0.5% 36 and 9 samples are required. Also from the cumulative benign fault coverage curves, we observe that, in general, for any tolerance value, the saturation point is reached before the maximum number of 150 samples required for detecting all critical faults at 0% tolerance, i.e., for 0.2% tolerance saturation is observed from 92, 33, and 1 samples onwards for benign dead neuron, saturated neuron, and synapse faults, respectively. For the IBM DVS gesture SNN, the number of samples required to achieve 100% critical fault coverage is 66, 35, and 28, for tolerance values 0%, 2%, and 4%, respectively. For these test sets, maximum coverage for not truly benign faults is achieved, except for tolerance 4% where an additional 31-28=3 samples from the ordered list need to be included. For the poker card symbols SNN, 6 samples suffice to detect all critical and not truly benign faults for any fault type. Overall, results show that in all cases a compact functional test set is found that is capable of detecting all critical faults for any tolerance value, and as an auxiliary benefit it detects all benign faults that result in even the smallest accuracy drop, i.e. one sample is misclassified. Taking as an example the N- MNIST SNN for which 70,000 samples are available from