Time-Reduced Model for Multilayer Spiking Neural Networks

: Spiking neural networks (SNNs) is a type of biological neural network model, which is more biologically plausible and computationally powerful than traditional artificial neural networks (ANNs). SNNs can achieve the same goals as ANNs, and can build a large-scale network structure (i.e. deep spiking neural network) to accomplish complex tasks. However, training deep spiking neural network is difficult due to the non-differentiable nature of spike events, and it requires much computation time during the training period. In this paper, a time-reduced model adopting two methods is presented for reducing the computation time of a deep spiking neural network (i.e. approximating the spike response function by the piecewise linear method, and choosing the suitable number of sub-synapses). The experimental results show that the methods of piecewise linear approximation and choosing the suitable number of sub-synapses is effective. This method can not only reduce the training time but also simplify the network structure. With the piecewise linear approximation method, the half of computation time of the original model can be reduced by at least. With the rule of choosing the number of sub-synapses, the computation time of less than one-tenth of the original model can be reduced for XOR and Iris tasks.


Introduction
Recently, multilayer neural networks have been successfully applied to many fields including pattern recognition [1,2], time series forecasting [3], and bioinformatics [4], etc.The basic unit of conventional artificial neural networks (ANNs) is called a node (or non-spiking neuron).The node is merely represented by non-linear activation functions without biological interpretability, whereas a spiking neural network (SNN) [5] is composed of spiking neurons that can transmit and receive substantial amounts of information through the relative timing of spikes.The characteristics of biological neurons are the basis of spiking neuron models.Spiking neurons can simulate the full process of biological neurons from receiving stimuli to firing spikes.This property of an SNN is particularly suitable for applications where the timing of input signals carries important information.It has been shown that the SNN can be applied to problems that can be solved by non-spiking neural networks and more importantly, the SNN is more powerful than conventional neural networks [6].
As the complexity of data increases, the number of layers will increase.It causes the network to consume much time for training.Accordingly, deep neural networks (DNNs) [7] can be optimized to work effectively.For instance, a binarization scheme is used to train DNNs [8].It consists in training the DNNs with binary weights during the forward and backward propagations while retaining the precision of the stored weights in which gradients are accumulated.A fixed-point factorized network (FFN), which was proposed in [9], is used for pre-train models to reduce the computational complexity as well as the storage requirement of networks.The resulting networks have only weights of -1, 0, and 1, which significantly eliminates the most resource-consuming multiply-accumulate operations [9].The hash trick can also be used for compressing neural networks [10].It exploits inherent redundancy in neural networks to achieve drastic reductions in model size.Different from these approaches, a key goal of our work is to reduce the time consumption of the multilayer SNN.A time-reduced model is employed for this paper.
The main contributions of this work are as follows: a) Based on the approximation of piecewise linear manner, this paper proposes a set of simple spike response models.It can get reduce much time during the training period.b) Based on the simple spike response model, an SNN with multiple sub-synapses can be obtained.Yet the number of sub-synapses is not explicit.To obtain the explicit sub-synapses, a method of choosing the suitable number of sub-synapses is proposed.The manners not only simplify the SNN structure but can get satisfactory results.
The rest of this paper is organized as follows.Section 2 discusses the spike neuron model and learning algorithm of multilayer SNN briefly.The time-reduce model is derived in Section 3, it contains a set of piecewise linear models and a rule for choosing the number of sub-synapses.Simulation experiment results are provided in Section 4, followed by the conclusion and the future work in Section 5.

Multilayer Spiking Neural Network
The continuously valued activations are used for non-spiking DNNs communication.On the contrary, the SNNs transmit information between neurons depending on sending a series of action potentials (i.e.spike trains).A one-layered SNN can solve a simple non-linear problem, however, it may not be able to represent all possible mappings [11].Therefore, the multilayer SNN is worth researching.The spiking neuron model and multilayer SNN learning rule are discussed in this section.

SRM Neuron Model
The SRM neuron model [12] can give a good approximation of the synaptic response for the neuron.Therefore, it is used as the target neuron model for the investigation of the anti-noise learning rule in this work.Figure 1 shows the connection of SRM-based neurons via one synapse.The state of the post-synaptic neuron j is described by a single variable in the framework of the SRM.The neuron is at its resting value when equals to zero if there is no input spikes and it can be denoted by 0. If reaches the threshold after summing the effects of several incoming spikes, an output spike is triggered.After firing, the evolution of is described by where is the synapse weight from pre-synaptic neuron i to post-synaptic neuron j with a delay of .The and denote the spike-times of neuron j and neuron i, respectively.If 1 !" # (n denotes the total number of firing spikes), then ! .The function describes the time course of the response to an incoming spike, which is given by where + is the post synaptic membrane potential time constant.* $ is the Heavy-side step function.H(s)=0 if s≤0 and H(s)=1 if s>0.The return of the membrane potential to baseline after an action potential is described by a function , which is named refractoriness.The refractoriness is characterized experimentally by the observation immediately after a first action potential.It is impossible (absolute refractoriness) or more difficult (relative refractoriness) to excite a second spike.The function is given by where + is a slow time constant and , is a scale factor for the refractory function.The SNN is composed of neurons connecting with each other via synapses.Figure 2 shows a multilayer network architecture.It consists of a feedforward fully connected network of spiking neurons with multiple delayed synaptic terminals.The different multiple delayed synaptic terminals are used in this work.

Neural Network Architecture and Algorithm
The SNNs rely on synapses to complete learning tasks.A synapse can be either excitatory or inhibitory.Excitatory synapse increases the membrane potential of the neuron upon receiving input spikes, whereas inhibitory synapse decreases the membrane potential of the neuron.The strength of a synapse is modeled by adjustable scalar weights [13].It can change over time to express the synapse efficacy in SNNs.This plasticity of synapses represents a brain-inspired learning mechanism in neural networks, and the learning rule is aiming for adjusting the strength of interconnecting synapses.
A supervised learning algorithm, the tempotron, has been explored in [14], where neurons learn to distinguish between spatial and temporal sequences of spike patterns [15].The gradient descent approach is used to train the tempotron.However, this learning rule can only be employed in one-layered networks.The leaky integrated-and-fire neuron model [12,16] in the tempotron is trained to only solve binary classification.Because this learning rule discriminates the patterns by neurons firing spikes or not.
A backpropagation learning rule, named multi-spikeprop [17,18], is used in our work.It is also on the basis of a gradient descent algorithm, and the network can converge gradually to the minimum error via many training epochs [13].The error function is an important criterion to measure the error between desired and actual output spikes.It can be calculated by where ̂ is the desired time of output spikes.Once the error is not a desired value, the synaptic weights will be updated.The updating rule of weights is the basis of gradient descent, and it can be calculated by where 8 is the earning rate.The gradient computation of synapses between neurons in the last layer and neurons in the hidden layer should be calculated firstly.The weights updating for the kth synapse between the presynaptic neuron i and postsynaptic neuron j can be calculated by According to the chain rule, ∇1 can be calculated by The weights updating between neurons in the input (or hidden) layer and neurons in the hidden layer can be computed by error backpropagation.It can be described as According to the chain rule, 91 @ can be calculated by The detailed learning rule is researched in [17,19].Although the multi-spikeprop learning rule can effectively handle classification tasks, it consumes much time to training the SNN.

The Time-Reduced Model
To solve the problem of time-consuming in the multilayered SNN, a set of piecewise linear spiking neural models and a rule for choosing the number of sub-synapses are proposed in this section.There is usually a trade-off between model accuracy and computational complexity for the piecewise linear spiking neuron model.For example, if the neural behavior is required to understand, the Hodgkin-Huxley model [20] will be more suitable.However, it is computationally expensive and cannot be simulated in large numbers.Therefore it is essential to simplify the SNN model.

SRM Linear Approximation
The original spike response function ε(s) is an exponential function, and the computation cost can be decreased by approximation.To reduce the computation time for multilayer SNN, a piecewise linear model is used to approximate the spike response function.
Five piecewise linear models are used to approximate the spike response function in our work, i.e. two segments, three segments, four segments, and five segments and six segments respectively.The two-segment piecewise linear model is shown in Figure 3.The blue line is the curve of the original spike response function, and the black line is the approximated curve.It can be seen that this segment is not precise enough.The detailed function of the two-segment model can be calculated by The parameters of E , E 2 , G and G 2 are coefficients, and  The detailed function of the three-segment model can be calculated by The values of E R , E S , E T , G R , G S and G T are respectively 0.1367, -0.0477, -0.013, 0.2086, 1.399 and 0.5939.Figure 5 shows the four-segment piecewise linear model.The blue line shows the curve of the original spike response function, and the black line shows the curve of the approximated function.The detailed function of the four-segment model can be obtained by The values of and G Y are respectively 0.1367, -0.0477, -0.0296, -0.0068, 0.2086, 1.399, 1.0226 and 0.3376.Figure 6 shows the five-segment piecewise linear model.The blue line shows the curve of original spike response function, and the black line shows the curve of approximated function.The detailed function of the five-segment model can be obtained by The values of E , , E , E 2 , E R , E S , G , , G , G 2 , G R and G S are respectively 0.2918, 0.0594, -0.0477, -0.0296, -0.0068, 0.0149, 0.6182, 1.399, 1.0226, and 0.3376.The six-segment piecewise linear model is shown in Figure 7.The blue line shows the original curve of spike response function, and the black line shows the curve of approximated function.The detailed function of the six-segment model can be obtained by The values ofE

Method of Choosing Sub-Synapses
In conventional neural networks, there is usually one synapse between two nodes.It contains multiple synapses between presynaptic neurons and postsynaptic neurons (i.e.sub-synapses) in SNNs.A suitable number of sub-synapses is crucial to the performance of the SNNs.For instance, too many sub-synapses will cause the SNN computationally expensive and much time-consuming.On the contrary, the network with fewer sub-synapses cannot work effectively.Therefore, a method of choosing a suitable number of sub-synapses is an important task.
The number of sub-synapses is usually set to a big value by experience [11,19,21].However, it is not suitable for all tasks.Thus, a rule of choosing the number of sub-synapses is employed for our work.It can be calculated by where n is the current number of sub-synapses, # , is the final number of sub-synapses, and 1 b is the minimum acceptable error of multilayered SNN.It shows that the number of synapses increases gradually from # , .If the value of ; ` more than the given threshold (1 b ), continue to increase until it meets the conditions.

Experimental Results
The exclusive-or (XOR) [11] and Iris [15] datasets are employed to test the performance of the time-reduced model in this work.The temporal encoding scheme is chosen for encoding the data.Particularly, the iris data is encoded to integral spike firing time using a direct mapping method.

Experimental Settings
A fully connected feedforward multilayered SNN with multiple delays per connection is used in our work.The time range and time step of the network are respectively set to 50 ms and 0.1 ms.The learning rate is set to 1.The mean squared error (E) is set to 0.1 as a stop criterion, and the maximum epoch is 1000.If iterations of the network are more than 1000 and E>0.1, the learning process will be failed.
All the experimental simulations are done under the MATLAB 2018a platform on a Windows 10 operating system with Intel (R) Core (TM) i5-9400 CPU 2.90GHz processor with 16 GB of random-access memory.Table 2 shows the temporal encoding scheme of the XOR task.For the input signals, an input spike at 0 ms represents logic 0 while a spike at 6 ms represents logic 1.For the output, a spike at 16 ms represents logic 0 while a spike at 10 ms represents logic 1.

XOR Task
The time constant of membrane potential should be slightly larger than the interval of encoding.Therefore the initial value of τ is set to 7 ms.The network consists of three neurons (two coding neurons and one bias neuron) in the input layer, and a single neuron in the output layer.Table 3 shows the performance of multilayer SNN using the model of piecewise linear approximation.It can be seen that the original multilayer SNN model runs 22.64s and achieves a hundred percent successful convergence rate.The two-segment piecewise linear model runs with the least time (3.54s), however, it only achieves a thirty-four percent rate of successful convergence.The five-segment and six-segment piecewise linear models achieve the same rate of successful convergence (96%), and the five-segment piecewise linear model consumes less time than the six-segment piecewise linear model.Therefore, the five-segment piecewise linear model is suitable for the XOR task.Table 4 shows the performance of multilayer SNN with the different number of sub-synapses.It can be seen that the network with 2, 3, 4 and 5 sub-synapses cannot converge to a target value.The network with 14 sub-synapses carries out 17 epochs and consumes 2.55s to converge to the target value.Therefore, the multilayer SNN with 14 sub-synapses is the best choice for the XOR task.The relationships between epochs and mean squared error under different spike response functions are shown in Figure 8.(a) refers to the original multilayer SNN, it carries 179 epochs and consumes 22.64s.(b) refers to the multilayer SNN using the five-segment piecewise linear model and the rule of choosing the number of sub-synapses, it carries out 24 epochs and consumes 2.47s to converge to the target value.The experimental result of the XOR task can prove that the method is effectively proposed in this paper.

Iris Classification Task
The iris dataset is used for this testing.It contains three classes (i.e.Iris Setosa, Iris versicolor, Iris Virginia) of 150 samples, where each class corresponds to a type of iris plant.There are four attributes in this dataset.Each attribute corresponds to an input neuron.Thus, the network has four input neurons in the first layer and one output neuron in the last layer.Due to the value of each attribute being a decimal value, it is encoded to integral spike firing time first.The encoding interval is set to 1, and the time constant of membrane potential is set to 3 ms.120 samples are used to train the multilayer SNN, and the rest samples are used to test the performance.Table 5 shows the performance of multilayer SNN using the model of piecewise linear approximation.It can be seen that the original multilayer SNN model runs 794.78s and achieves a 96.67 percent successful convergence rate.The five-segment piecewise linear model runs with the least time (106.61s),and it achieves a higher rate of successful convergence (93.33%).The five-segment piecewise linear model consumes less time than the six-segment piecewise linear model, however, the six-segment piecewise linear model can achieve the same successful convergence rate as the original model.Therefore, considering the time consumption, the five-segment piecewise linear model is more suitable for the Iris task.epochs and consumes 38.34s to converge to the target value.Therefore, the multilayer with 14 sub-synapses is the best choice for the Iris task.Table 7 shows the performance of this work with other SNN algorithms.It can be seen that our work can get an effective accuracy on Iris dataset with less time consumption.
Figure 9 depicts the curves of learning performance in the interval of epochs and mean squared error under different spike response functions.(a) depicts the original multilayer SNN, it carries 46 epochs and consumes 794.78s.(b) depicts the multilayer SNN using the five-segment piecewise linear model and the rule of choosing the number of sub-synapses, it carries out 14 epochs and consumes 33.56s to converge to the target value.The experimental result of the Iris task is further explained that the method is effectively proposed in this paper.

Conclusion
In this paper, methods for reducing the time consumption of multilayer SNN were proposed.The methods include a piecewise linear approximation of the spike response function and a rule of choosing the suitable number of The experiment of the XOR task and Iris dataset classification task were used to verify the performance of the proposed time-reduced model.The experimental results showed that the proposed methods can reduce time consumption and have a better performance.Future work includes the theoretical proof of the proposed methods and further exploration of spiking neuron models and deep SNNs.

Figure 1 .
Figure 1.The connection of SRM-based neurons via one synapse.

Figure 2 .
Figure 2. The multilayer network architecture.The right part indicates that sub-synapses are included in each synaptic junction.
, E 2 = −0.0232, G = 0.2086 , G 2 = 0.9833 .The three-segment piecewise linear model is shown in Figure4.The blue line refers to the original curve of spike response function, and the black line refers to the curve of the approximated function.

Figure 8 .
Figure 8.The relationships between epochs and mean squared error under different spike response functions.(a).The performance of the original multilayer SNN.(b).The performance of multilayer SNN with the time-reduce model.

Figure 9 .
Figure 9.The relationships between epochs and mean squared error under different spike response functions.(a).The performance of the original multilayer SNN.(b).The performance of multilayer SNN with the time-reduce model.

Table 1 .
G 2, are respectively 0.2918, 0.0594, -0.0477,It can be seen that the more the number of segments, the smaller the error, and the less the number of segments, the less time consuming.

Table 1 .
The Time consumption of different piecewise linear models.

Table 2 .
The encoding scheme of the XOR task.

Table 3 .
The performance of different spike response models for the XOR task.

Table 4 .
The performance of multilayered SNN with the different number of sub-synapses for the XOR task.

Table 5 .
The performance of different spike response models for the Iris task.

Table 6 .
The performance of multilayered SNN with the different number of sub-synapses for the Iris task.

Table 6
shows the performance of multilayer SNN with the different number of sub-synapses.It can be seen that the network with 2, 3, and 4 sub-synapses cannot converge to a target value.The network with 14 sub-synapses carries out 9

Table 7 .
Comparison of SNN training results for Iris task.