A spiking neural network for real-time Spanish vowel phonemes recognition

A spiking neural network for real-time Spanish vowel phonemes recognition L. Miró-Amarante, F. Gómez-Rodríguez, A. Jiménez-Fernández, G. Jiménez-Moreno Robotics and Technology of Computers Lab, ETS Ingeniería Informática, University of Seville, Av. Reina Mercedes s/n, Seville, Spain Keywords: Neuromorphic engineering Address event representation (AER) Event-based processing FPGA Digital cochlea Speech recognition ABSTRACT This paper explores neuromorphic approach capabilities applied to real-time speech processing. A spiking recognition neural network composed of three types of neurons is proposed. These neurons are based on an integrative and fire model and are capable of recognizing auditory frequency patterns, such as vowel phonemes; words are recognized as sequences of vowel phonemes. For demonstrating real-time operation, a complete spiking recognition neural network has been described in VHDL for detecting certain Spanish words, and it has been tested in a FPGA platform. This is a stand-alone and fully hardware system that allows to embed it in a mobile system. To stimulate the network, a spiking digital-filter-based cochlea has been implemented in VHDL. In the implementation, an Address Event Representation (AER) is used for transmitting information between neurons. 1. Introduction Nature is a notable source of inspiration for researchers around the world. Neuromorphic engineering attempts to integrate biological knowledge into artificial systems [1,2]. This work combines both psychoacoustic knowledge and neuromorphic engineering principles to propose a spiking neural network for Spanish vowel phonemes recognition. This paper is organized as follows: a brief introduction to Spanish psychoacoustic and neuromorphic processing is presented in the remainder of this section; Section 2 is devoted to describing the spiking digital-filter-based cochlea; Section 3 presents the spiking recognition neural network; the experiments and results are presented in Section 4. 1.1. Spanish psychoacoustic The biological process of Spanish phoneme recognition is based on the first two formants of the vowel phoneme spectrum. Formants are the resonant frequencies of a vocal tract when pronouncing a voiced sound such as a vowel [3]. There is an intimate relationship between the acoustic properties of speech sounds and their source, the human organs of speech. The acoustic structure of speech is completely predictable (on physical principles) from the position and states of the organs of production. Because vowels are normally voiced, the source (known as the glottal source) is situated in the larynx when the sound arises from the quasiperiodic vibrations of the vocal folds. This near-periodicity of the glottal waveform gives rise to an essentially harmonic spectrum with the energy at the frequency of the vocal fold vibrations and its integer multiples. Hence, as the vocal fold vibration rate increases the frequency of the fundamental component in the spectrum and the inter-harmonic spacing also increase. As it decreases, the harmonics become more densely packed with the fundamental component decreasing in frequency. Before the sound from the glottal source can reach a listener, it must pass through the vocal tract. For non-nasalized vowels, the vocal tract has no anti-resonances; it only contains resonances. These resonances are called formants, and only the lowest four or five formants are important in speech perception. The formant resonances of the vocal tract give rise (as long energy is present from the source at that frequency region) to peaks in the output spectrum, also known as formants. Numbers increasing from 1 usually identify formants, starting from the formant at the lowest frequency. The spectrum of phonemes can consist of several formants, but the first two formants are the most important for recognition [4]. During the pronunciation of Spanish vowels, in an adult male, the frequency of the first formant (F1) can vary in range from 200 Hz to 1000 Hz. The lower value occurs when the tongue is closer to the roof of the mouth. The vowel /i/ has one of the lowest F1 values – approximately 250 Hz; in contrast, the vowel /a/ has the highest F1 value at approximately 916 Hz. The second formant (F2) can vary from 600 Hz to 2500 Hz; the F2 value is proportional to the front or back of the highest part of the tongue during the production of the vowel. Additionally, lip rounding causes a lower F2 than unrounded lips. /i/ has an F2 of 2400 Hz, the highest F2 of any vowel. In the production of this vowel, the tip of the tongue is quite far forward, and the lips are MARK unrounded. At the opposite extreme, /u/ has an F2 of 624 Hz; in this vowel, the tip of the tongue is very far back, and the lips are rounded. The experimental studies conclude that there is a wider frequency range when a listener perceives Spanish vowel phonemes. Tables 1 and 2show the lower and upper limits as well as the range corresponding to the narrow zone (where the phoneme is correctly identified by 80% of the listeners) for F1 and F2, respectively [5]. 1.2. Neuromorphic processing In 1990, Caver Mead started using the term, neuromorphic engineering [6]; neuromorphic engineering is a field of engineering devoted to neural system design and fabrication, in which architecture and design principles are based in biological neural systems. Different from artificial neural networks, neuromorphic systems attempt to imitate the function of biological neural system such as speech [7,8– 10], object recognition [11] or cognitive processes [12–15]. One of the most important neuromorphic developments is address event representation (AER) [16,17]. AER is a communication scheme that was initially proposed for neuromorphic chip communication; today, it is in the basis of several neuromorphic systems [18–20]. AER provides a solution to one of the most difficult questions that neuromorphic engineering has to solve, which is how to imitate the neural-biological connectivity, where each single neuron can connect with tens of thousands of other neurons. Fig. 1a explains the principle behind AER. The emitter chip contains an array of cells (e.g., neuromorphic cochlea [21]) where each cells shows a state that changes with a slow time constant (on the order of milliseconds). Each cell includes an oscillator that generates pulses of minimum width (a few nanoseconds). Each time a cell generates a pulse (an "event"), it communicates with the periphery, and its address is placed on the external digital bus (the AER bus). Handshaking lines (Acknowledge and Request) are used for completing the communication (Fig. 1b). An advantage of AER scheme is that computation is event driven and thus can be very fast and efficient. Both the spiking recognition neural network and the spiking digital-filter-based cochlea proposed in this work use AER for representing and transmitting information. 1.3. Proposed spiking neural network overview The proposed spiking neural network comprises two stages that model the different processes of the human auditory system, from the auditory periphery to the cells in the auditory brainstem. Fig. 2 shows an overview of the proposed spiking neural network. First, speech is passed through a digital-filter-based cochlea where the speech is split into its frequency components and translated into spikes. The biological cochlea has the capability of performing a frequency analysis of the incoming vibrations, separating them into overlapping frequency bands [22–24]. The mechanical motion of the basilar membrane leads to the displacement of the inner hair cells. The inner hair cells act as transducers; a neurotransmitter is released at the synaptic cleft, and action potentials are propagated into the auditory brainstem. The frequency-selective displacement characteristic of the basilar membrane is modelled using an array of 21 band-pass digital filters and a set of spike generators. The band-pass filters, whose cut-off frequencies range over the human speech frequencies (20 Hz to 20 kHz), split the speech signal into different frequency components. Each filter output is connected to a spike generator that models the spike generation. The pair of the band-pass filter and the spike generator is denominated by the Cochlea Band (Section 2). Secondly, a spiking recognition neural network is used for detecting phonemes and words as sequences of phonemes. This network is composed of several recognition-neurons (RNeuron). RNeurons spike when a set of their inputs spike simultaneously and when a pre-set pattern is detected in their inputs. Because some RNeurons can spike simultaneously, a winner-take-all network, composed by some winnerneurons (WTANeuron), is used for empowering the most active RNeurons and depressing the rest. A third spiking neuron, the delay neuron (DelayNeuron), is needed to detect words such as sequences of phonemes. All of these spiking neuron models are defined in this work in Section 3. 1.4. Hardware platform A library of IP cores for speech information spiking processing using VHDL were developed for this research [25]. The system is implemented on a Cyclone IV E FPGA [26]. The FPGA chip is on a board, a EP4CE115F29C7 that contains an audio codec, WM8731/ WM8731L [27]. A USB2AERmini2 board [28] is also used to communicate between the FPGA chip and the PC. This board was used as an AER monitor to test the output of each stage. The monitoring function involves reading and time-stamping events from AER bus and making them available to Table 1 The dispersion field of F1 of Spanish vowel phonemes in an adult male. F1 /i/ /e/ /a/ /o/ /u/ Lower limit (Hz) 200 400 800 325 200 Upper limit (Hz) 450 800 1000 800 400 80% field (Hz) 290/330 580/750 915 750 210/330 Table 2 The dispersion field of F2 of Spanish vowel phonemes in an adult male. F2 /i/ /e/ /a/ /o/ /u/ Lower limit (Hz) 1300 800 1000 700 600 Upper limit (Hz) 2500 2500 1800 2000 2500 80% field (Hz) 2300/2400 1600/1900 1500 950/1050 625/660 a) b) REQ ACK DATA 1 2 3 4 Fig. 1. Address event representation (AER). the PC. Fig. 3 shows the hardware platform used; a USB2AERmini2 board is on the right upper corner. 2. Spiking digital-filter-based cochlea The spiking cochlea is composed of 21 bands (the digital filter and the spike generator). The digital cochlea works with an 18-bit PCM signal obtained from the ADC of the WM8731/WM8731L audio codec. The ADC is set at a 48 kHz sample rate. 2.1. Digital filters bank Because the biological cochlea acts as a spectrum analyser, bandpass filters are used for decomposing complex sound into its frequency components. The band-pass filters show if a signal contains a frequency component within a specific frequency range. A large number of options are available for filter design and implementation. However, the selected option allows low cost and low complexity in its hardware. The digital filter bank consists of 21 infinite impulse response (IIR) Fig. 2. Spiking recognition neural network overview. Fig. 3. EP4CE115F29C7 platform and USBAERmini2 board. Table 3 Subdivision of the audible frequency range into critical bands. Bands Centre frequency (Hz) Frequency range of band-pass filter (Hz) 0 350 200–400 1 450 400–510 2 570 510–630 3 700 630–770 4 840 770–920 5 1000 920–1080 6 1170 1080–1270 7 1370 1270–1480 8 1600 1480–1720 9 1850 1720–2000 10 2150 2000–2320 11 2500 2320–2700 12 2900 2700–3150 13 3400 3150–3700 14 4000 3700–4400 15 4800 4400–5300 16 5800 5300–6400 17 7000 6400–7700 18 8500 7700–9500 19 10500 9500–12000 20 13500 12000–15500 Fig. 4. Magnitude response of the 21 band-pass filters (Bode diagram). second order filter sections using parallel form. Each of these filters is set to have a cut-offfrequency in the range of 20–20 kHz, as shown in Table 3. The digital filter bank is based on the subdivision of the audible frequency range into critical bands proposed by Zwicker [29]. These 21 band-pass filters are all IIR filters. The FIR filter provides the linear phase; however, cochlear filters do not need a linear phase. Therefore, IIR filters are preferred to FIR filters. Also IIR filters can achieve a given filtering characteristic using less memory and calculations than the similar FIR filters. Butterworth filters are used in this work. In these filters, the magnitude response is maximally flat in the pass-band and monotonic overall. The filter models are all designed in two's complement fixedpoint arithmetic that implies the need for the direct form II transpose structure. All filters are described in VHDL language using the Filter Design HDL Coder [30].Fig. 4 shows the magnitude response of the 21 band-pass filters. 2.2. Spikes generator Each digital filter is connected to a spike generator that mimics the functionality of the inner hair cells (IHCs). This spike generator is based on the accumulator spiking neuron model proposed by Gómez Rodríguez et al. [31]. This spiking neuron model (shown in Fig. 5) uses an accumulator register, initialized with a threshold level. This value is increased in the PCM-filtered value every clock cycle. Concurrently, the output from the digital filter, the PCM-filtered value, is compared with the value of the register; if the register value is higher than the PCM-filtered value, a spikes will be generated. The number of spikes and the firing rate of each cochlea band are proportional to the amplitude of the corresponding digital filter output. So, the firing rate changes as the PCM filtered signal changes. Although, each of the spike generators has its own threshold level that can be set independently, all of the threshold values are set to 127 in this implementation. This value guarantees that a sufficient number of spikes will be generated. 2.3. AER generator and arbiter As previously mentioned, AER is used for coding and transmitting spikes. So, an AER arbiter encodes the address in each cochlea band from 0 to 20 and sends the AER spike on the time-multiplexed AER bus using a hand-shaking protocol. In this case, each AER spike is identified with an address in range from 0 to 20. Thus, the output of this spiking digital cochlea contains the AER spikes train associated to each cochlea band as an answer to the PCM signal. 2.4. Spiking digital cochlea response to acoustic stimuli Matlab and the USBAERmini2 board are used to test the behaviour of the spiking digital cochlea. Fig. 6 shows the output histogram of the spiking digital cochlea in response to the swept-sine wave from 200 Hz to 1200 Hz. The maximum number of events of each cochlea band is analysed for each of the sine waves. Fig. 6 shows the most active cochlea band, which is the band whose central frequency matches with the frequency of the sine wave. The maximum number of events of each cochlea band is different because the spike generators do not make a homogeneous distribution of spikes; the number of events depends on the PCM value. To illustrate the behaviour of the spiking digital cochlea, Fig. 7 shows the cochlear response to acoustic stimuli, a 689 Hz sine wave. The spiking cochleogram (left-hand column) displays the temporal distribution of spikes in the cochlea bands domain, and the histogram (right-hand column) displays the number of spikes per cochlea band. Like a spectrum analyser, the bandpass filters separate sound into its frequency components. The spiking cochleogram display shows the output of the digital cochlea from a 689 Hz sine wave; each cochlea band fires a sequence of spike trains. The output of the third cochlea band, which ranges from 630 Hz to 770 Hz, indicates that the signal contains a frequency component within this specific frequency range. The histogram determines that the third cochlea band fires the largest number of events and is the most active cochlear band. 3. Spiking recognition neural network As previously mentioned, the spiking recognition neural network is used to recognize phonemes and words as sequences of phonemes. Three types of neurons are needed to accomplish this recognition, Fig. 5. Accumulator spiking neuron model. 0 2 4 6 8 10 12 14 16 18 20 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Histograma. volume 10%. monitorTime 0,2 Bands Num Events 300 455 570 700 845 1000 1175 1375 1600 1860 2160 2510 2925 3425 4050 4850 5850 7050 8600 10750 Fig. 6. Histogram plot. A 127-threshold level of the spike generator. The legend shows the frequency of the sine-signals. RNeurons, WTANeruons and DelayNeurons. 3.1. RNeuron The RNeuron is the main element of the recognition neural network. It has the function of identifying a pattern or category. Both phoneme recognition and word recognition are carried out by a set of this RNeuron. There is one RNeuron for each of the patterns to recognize. The behaviour of this neuron is based on artificial neural networks, particularly in the perceptron neuron and its function of pattern recognition, and the integrate-and-fire model. RNeurons are derived from well-known neuron models, the perceptron [32] and integrate-and-fire models [33,34]. In these models, incoming spikes increase the internal cell potential depending on its input weight. RNeurons fire when the internal potential reaches a threshold. In this implementation, weights take a binary value. Fig. 8 shows the proposed model. The RNeuron has nspiking inputs, x xx,,…, n12 . These spikes are integrated, ∫ Xx= i i when enough spikes are received, ⎧ ⎨ ⎩ y if X Synapse threshold X if X Synapse Threshold =0< ≥ i i ii These spikes may contribute to the internal potential, depending on its weight: ∑ p xyω()= i n ii =1 The hardware implementation of RNeuron is based on a finite states machine, shown in Fig. 9, with the parameters shown in Table 4. Fig. 9 shows the state machine diagram that models the behaviour of RNeuron. This figure indicates how the dendrites, axon and axon terminals of the neuron are implemented. The init state is the initial state after an asynchronous reset is received. In this state, the RNeuron parameters are initialized with the default values (Table 4). At the next cycle, it transitions to the standby state. In this state, if it receives a spike, then it transitions to the spike count update state; otherwise, it stays in the standby state. There is a timer that detects the absence of activity (Reset time parameter), and it goes back to the init state. In the Spike count update state, the number of spikes received from each synapse is update in accordance with the current active synapses. If the number is higher than its synapse threshold, it transitions to the pattern mask check state; otherwise, it returns to the standby state. In the pattern mask check state, if the synapse is marked with a '1' in the pattern mask, its internal potential is incremented in the increase internal potential state; otherwise, it returns to the standby state. In the increase internal potential state, if its potential threshold is reached, the pattern has been identified. In this case, it transitions to the pattern detected and fire state, and a spike is issued. Then, it returns to the init state; otherwise, it transitions to the standby state. Fig. 7. Response of the spiking digital cochlea to a 689 Hz sine wave. Fig. 8. RNeuron model. INIT Standby Spike count update Pattern mask check Synapse marked with ‘0’ in the pattern mask Spike received Synapse Threshold reached Synapse Threshold is not reached Increase Internal Potential Synapse marked with ‘1’ in the pattern mask Pattern detected and fire Internal Potential > Potential Threshold Internal Potential < Potential Threshold No spikes received or no pattern detected during long time Dendrites Axon Axon Terminal Fig. 9. The finite states machine of RNeurons. Table 4 RNeuron parameters. Name Parameter description Pattern mask Corresponds to the synapse weights. Array of npositions; position iwill be '0' if the RNeuron does not expect spikes by the synapse i; '1' indicates that RNeuron expects spikes by this synapse. For example, if the pattern mask is “000000000000000001110”, the RNeuron expects to receive spikes by the synapses 1, 2 and 3. Synapses threshold Each synapse has an associated a threshold level that indicates when a synapse is active and may contribute to the internal potential Internal potential Internal potential is increased when spikes are received by active synapses. This potential defines the behaviour of the RNeuron; while its value is lower than its potential threshold, the RNeuron receives and processes new spikes; when the potential threshold is reached, the RNeuron fires, indicating that a pattern was recognized. Then, its internal potential is reset. Potential threshold This threshold determines when RNeuron fires. Reset time If RNeuron does not receive a spike during a specified period of time (Reset time), all of internal values will be reset. Fig. 10. Finite states machine of WTANeuron. Table 5 WTANeuron parameters. Name Parameter description Synapse weights Array of npositions that corresponds to the synapse weights. Each neuron has one excitatory synapse and n−1inhibitory synapses. These weights are integers; positive values occur for the excitatory synapses and negative values for the inhibitory synapses. Synapses threshold Each synapse has an associated threshold level that indicates when a synapse is active. Internal potential When spikes are received by active synapses, the internal potential is increased or decreased depending of synapse weight. This potential defines the behaviour of the WTANeuron; when its value is lower than its potential threshold, the WTANeuron receives and processes new spikes; when the potential threshold is reached, the WTANeuron fires, indicating that it is the winner neuron. Then, its internal potential is reset. Potential threshold Each neuron has an associated level, which indicates that it is the winner neuron. Reset time If WTANeuron does not receive a spike during a certain period of time (Reset time), all of internal values will be reset. D D D D Fig. 11. DelayNeuron chain. INIT Delay Standby Spike received Send delayed spike Dendrites Axon Axon Terminal Delay time reached Fig. 12. DelayNeuron finite states machine. 3.2. WTANeuron It is possible that several RNeurons fire at the same time. For this reason, a winner-take-all network is needed. The winner-take-all spiking network has been used to enhance the recognition process. It is composed of a layer of spiking neurons, which receives spike trains as inputs. The neuron that receives spikes with the highest rate is selected as the winner after a pre-determined number of input spikes [35]. Although there are two winner-take-all modes, the hard and leaky modes, the leaky winner-take-all mode is chosen in this model, where the winning neuron is active and all other neurons are also active in less proportion. There is one WTANeuron for each of the patterns to recognize. The WTANeurons inhibit each other while simultaneously activating themselves. Therefore, only one WTANeuron will be active, the one corresponding to the strongest input. In this model, incoming spikes increase or decrease the internal neuron potential, depending on its input weight and its synapse type (excitatory or inhibitory). WTANeurons fire when the internal potential reaches a threshold. In this case, it is the winner neuron. In this implementation, weights take an integer value, positive value is taken for the excitatory synapses and a negative value for the inhibitory synapses. The hardware implementation of WTANeurons is also based in a finite states machine, shown in Fig. 10, with the parameters shown in Table 5. The following state machine diagram models the behaviour of a WTANeuron (Fig. 10). After an asynchronous reset, the WTANeuron begins in the init state with its setting parameters with default values (Table 5). At the next cycle, it transitions to the standby state. In that state, if it receives a spike, it transitions to the spike count update state; otherwise, it stays in the standby state. In the standby state, there is a timer that detects the absence of activity (the reset time parameter), and it returns to the init state. In the spike count update state, the number of spikes received from each synapse is updated in accordance with the current active synapsis. If the number is higher than its synapse threshold, it transitions to the synapse type check state; otherwise, it returns to the standby state. In the synapse type check state, it transitions to the decrease internal potential state or to the increase internal potential state according to its synapse weight. In the decrease internal potential state, its internal potential is decremented by the inhibitory weight, and it transitions to standby state; otherwise, its internal potential is incremented according to the excitatory weight. When its potential threshold is reached, it is the winner neuron. It transitions to the winner and fire state where a spike is issued, and it the transitions to the init state. 3.3. DelayNeuron To recognize a word as a sequence of phonemes, it is necessary to delay the spike train associated with the first recognized phoneme. In this case, it ensures both phonemes are closely matched when they arrive to the next spiking recognition neural network (composed by RNeuron and WTANeuron tuned to detect the word). DelayNeuron is devoted to the spike delay, and its behaviour is implemented with the finite state machine, shown in Fig. 12. DelayNeuron has only one parameter: DelayTime set to 204.8 µs. These experiments ensure that no spikes will be lost with this DelayTime value. To obtain a larger delay time, several DelayNeuron can be connected in a chain (Fig. 11). Fig. 12 shows the finite state machine diagram, which models the behaviour of a DelayNeuron. After an asynchronous reset, the DelayNeuron begins in the init state with its setting parameters having default values. At the next cycle, it transitions to the standby state. In that state, if it receives an incoming spike. Then it transitions to the delay state; otherwise, it stays in the standby state. In the delay state, there is a timer that measures the delay time, and it goes to send delayed spike state where the spike is sent by the axon terminal. 4. Experiments and results This section describes an experiment where the spiking recognition neural network is configured to categorize a Spanish two-syllable word in real-time from the identification of a sequence of two Spanish vowel phonemes. The Spanish word “letra”is used as an example. This word is considered as a sequence of two Spanish vowels phonemes, /e/ and /a/. This process is performed in two stages: the first stage identifies the Spanish vowel phonemes, and the second stage identifies two-syllable words as a sequence of two vowel phonemes. 4.1. Vowel phonemes recognition neural network First, the output of the spiking digital-filter-based cochlea, a spike train, arrives to the vowel phonemes recognition neural network. This recognition neural network is composed of five RNeurons, one for each Spanish vowel phoneme. Every RNeuron is configured to identify one vowel phoneme (Table 6). As it is described in the previous section, each vowel phoneme is characterized with the F1 and F2 formant; it has been experimentally determined which cochlea band corresponds with each formant. Then, this information is used for setting the pattern mask of the RNeuron. All RNeurons have a reset time of 10 ms, and all of these values were obtained experimentally. As a result, an AER spike train is obtained, where each AER spike Table 6 Vowel phonemes for the parameters of RNeuron. Id (AER spike address) Pattern mask Cochlea bands associated to formant frequencies (F1, F2) RNeuron A 0“000000000000001010000”Bands 4 and 6 RNeuron E 1“000000000001000000010”Bands 1 and 9 RNeuron I 2“000000010000000000001”Bands 0 and 13 RNeuron O 3“000000000000000010100”Bands 2 and 4 RNeuron U 4“000000000000000001001”Bands 0 and 3 Table 7 Vowel phonemes for the parameters of WTANeuron. WTANeuron A WTANeuron E WTANeuron I WTANeuron O WTANeuron U Id (AER spike address) 02468 We 55555 Wi 10 10 23 23 15 Potential threshold 16 16 31 16 16 has an address from 0 to 4 that is associated with each RNeuron (0 for /a/, 1 for /e/, 2 for /i/, 3 for /o/ and 4 for /u/). The recognition neural network is also composed of five WTANeurons (Table 7). Each WTANeuron has one excitatory synapse (for one of the five vowel phoneme) and four inhibitory synapses (for the other phonemes). The values of the excitatory weight (We), the inhibitory weight (Wi), the synapses threshold and the potential threshold have been obtained experimentally. All WTANeurons have a reset time of 10 ms. As a result, an AER spike train is obtained, where each AER spike has an address associated with each WTANeuron (0 for /a/, 2 for /e/, 4 for /i/, 6 for /o/ and 8 for /u/). 4.2. Word recognition neural network Once the vowel phonemes are identified, the words recognition neural network is able to recognize a two-syllable word as a sequence of two vowels phonemes. This recognition neural network consists of 11 RNeurons, which are set to simultaneously detect two vowel phonemes. It is necessary to add some delays to the vowel phonemes to ensure that the two vowels phonemes, the first phoneme and the second phoneme, arrive to the RNeurons at the same time. So, it is necessary to delay the first phoneme. Fig. 13 shows the entire two-syllable word recognition neural network. This network is comprised of three parts: the phonemes recognition neural network, the DelayNeuron chain and the words recognition neural network. Because of the AER hardware implementation, additional elements are required including an AER_Splitter & Mapper, which replicates and sends each incoming spike through the two output. One spike will go directly to the words recognition neural network, and the other will be delayed. To distinguish between the original spike and the delayed spike, it also changes the addresses of the delayed spike. So, the original spikes have odd addresses (0(a), 2(e), 4(i), 6(o), 8(u)) and the delayed spikes have even addresses (1(a delay ), 3(e delay ), 5(i delay ), 7(o delay ), 9(u delay )). In this implementation, the delay time is 200 ms (the experimental value). It is interesting to note that the recognition neural network can be adapted to the speed of the speaker with a change in this value. The recognition neural network has as many Rneurons as words that it wants to recognize. The implementation identifies 11 words, which means that it recognizes 11 sequences of vowel phonemes (Table 8). The output of this module is also an AER spike train. Each AER spike will have a value that belongs to range 0 to (number of words –1) associated with each word. All RNeurons have a reset time of 5 ms. All these values are obtained experimentally. In this recognition neural network, a set of WTANeurons is also added to improve the recognition process. Table 9 shows the setting parameters for the words WTANeurons. All WTANeurons have a reset time of 2.5 ms. Fig. 14 shows the full recognition process for the speech signal corresponding to the Spanish word “letra”. Each row shows the output of one stage: spiking digital-filter-based cochlea (row 1), phoneme recognition neural network (rows 2 and 3), DelayNeuron chain (row 4) and words recognition neural network (row 5). A row also includes a spiking cochleogram image (left-hand column) and a histogram (righthand column). First, the spiking digital-filter-based cochlea transforms the PCM values of this speech signal into spike trains. The spike address has a value associated with a cochlea band within the range of 0–20. In the left-hand column, it displays the spiking representation of the two phonemes of the Spanish word “letra”. Corresponding to the first phoneme, the cochlea bands 1 and 9 are the most active bands; they are the cochlea bands which contain the F1 and F2 frequency components of the /e/ phoneme. For the second phoneme, the most active cochlea bands are the bands 3 and 7, which contain the F1 and F2 frequency components of the /a/ phoneme (Table 6). Previous to the /a/ WORDS Recognition Nerual Network AER Splitter & Mapper D D D D DelayNeuron Chain PHONEMES Recognition Nerual Network Fig. 13. Word recognition neural network. Table 8 Words RNeurons parameters. Id (AER spike address) Pattern mask Address of the two vowel phonemes RNeuron AE 0“0000000110”1(a delay ), 2(e) RNeuron AO 1“0001000010”1(a delay ), 6(o) RNeuron EA 2“0000001001”3(e delay ), 0(a) RNeuron EO 3“0001001000”3(e delay ), 6(o) RNeuron IA 4“0000100001”5(i delay ), 0(a) RNeuron IE 5“0000100100”5(i delay ), 2(e) RNeuron IO 6“0001100000”5(i delay ), 6(o) RNeuron OA 7“0010000001”7(o delay ), 0(a) RNeuron UA 8“1000000001”9(u delay ), 0(a) RNeuron UE 9“1000000100”9(u delay ), 2(e) RNeuron UI 10 “1000010000”9(u delay ), 4(i) Table 9 Words WTANeurons parameters. WTANeuron WTANeuron WTANeuron WTANeuron WTANeuron WTANeuron AE AO, OA, UI EA, EO, IE, UE IA IO UA We 444 444 Wi 281 454 Potential threshold 2151 411 Fig. 14. Recognition of Spanish word "letra".