Convergence of Online Gradient Method for Pi-sigma Neural Networks with Inner-penalty Terms

: This paper investigates an online gradient method with inner-penalty for a novel feed forward network it is called pi-sigma network. This network utilizes product cells as the output units to indirectly incorporate the capabilities of higher-order networks while using a fewer number of weights and processing units. Penalty term methods have been widely used to improve the generalization performance of feed forward neural networks and to control the magnitude of the network weights. The monotonicity of the error function and weight boundedness with inner-penalty term and both weak and strong convergence theorems in the training iteration are proved.


Introduction
A novel higher order feedforward polynomial neural network is known to provide inherently more powerful mapping abilities than traditional feed forward neural network called the pi-sigma network (PSN) [2]. This network utilizes product cells as the output units to indirectly incorporate the capabilities of higher-order networks while using a fewer number of weights and processing units. The neural networks consisting of the PSN modules has been used effectively in pattern classification and approximation problems [1,4,10,11].There are two ways of training to updating weight: The first approach, batch (offline) training [18],the weights are modified after each training pattern is presented to the network. Second approach, online training, the weights updating immediately after each training sample is fed see [13]. The penalty term is often introduced into the network training algorithms has been widely used so as to control the magnitude of the weights and to improve the generalization performance of the network [6,8], here the generalization performance refers to the capacity of a neural network to give correct outputs for untrained data. Specially cause, in the second approach the training weights updating become very large and over-fitting tends to occur, by adding the penalty term in into the cost function, when use second approach has been successfully application see [3,7,12,14], which acts as a brute-force to drive unnecessary weights to zero and to prevent the weights from taking too large in the training process. In the work area of penalty term at the same of the inner-penalty term (IP), which have worked to reduce the magnitude of the network weights with efficiency improve the generalization performance of the network [5,9,17]. In this paper, we prove the (strong and weak) convergence of the online gradient with inner penalty and the monotonicity of the error function and the weight sequence are uniformly bounded during the training procedure with inner-penalty.
The rest of this paper is organized as follows. The neural network structure and the online gradient method with innerpenalty are described in Section 2. The preliminary lemmas are disruption in Section 3. The convergence results are presented and the rigorous proofs of the main results are provided in Section 4. Finally, in Section 5 we conclusions this study.

PSN-Algorithm
PSN is a higher order feed forward polynomial neural network consisting of a single hidden layer. The hidden layer has summing units where as the output layer has product units. PSN, which has a three-layer network consisting of input units, summation units, and 1 product layers. Let , … , 1 the weight vectors connecting the input and summing units, and write , , … , . We have included a special input unit , corresponding to the biases , with fixed value-1. The structure of PSN is shown in Figure 1. Where , … , , and 1. Assume g: ! is a given activation function. For an input , the output of the network is The network supplied with a given set of training samples , . The error function with a inner penalty given by - Where 3 7 0 is a inner penalty coefficient and g 5 9 The gradient function is given by Given an initial weight C , the online method with inner penalty updates them iteratively by the form DE D 2 ∆ D , G 0,1, …. (4) Where H 7 0 is the learning rate in the G9I training cycle.We denote byJ·J the usual Euclidean norm and the corresponding derived matrix norm and the following Assumption is imposed throughout this paper. Assumption 1.
The learning rateH and penalty parameter 3are chosen to satisfy the condition: 0 H 1/ 3L T 2 L Assumption 4. ' D ) D&C, ,U are contained in bounded closed region Θ + , and there are exist points in set Θ C ' Θ|-: 0).

Preliminary Lemmas
The next lemma present the montonicity of the sequence '-). It is essential for the proof of weakly convergence of PSN with penalty, presented in the following Theorems. For sake of description, we denote X , To begin with, first we present a few lemmas as preparation to prove Theorems Proof .By Assumption 2and Cauchy-Schwartz inequality, we have Here, L T = L . This is completes the proof.
Proof. Applying Taylor's formula to extend g 5 Y DE at Y D , we have where 9 , 9 ∈ ℝ are on the line segment between Y DE and Y D . After dealing with (14) by accumulation g 5 Y DE for 1 ≤ N ≤ O, we obtain from (2), (4), (5) and Taylor's formula we obtain where where L a = L L + 3L . It follows from Assumption 1~2, (2) and Taylor's formula, we obtain Proof. This result is basically the same as Theorem 14.1.5 in [16], and the detailed proof is thus omitted.
Furthermore, if Assumption 4 is also valid, we have the strong convergence: There exists * ∈ Θ C such that Next we prove the strong convergence it follows from (4)~(5) and (42) that leads lim D→v J∆ D J = 0, 0 ≤ ≤ Note that the error function -D defined in (2) is continuously differentiable. By (43), Assumptions 4 and Lemma 3, immediately get the desired result. This completes the proof.

Conclusion
Through our study of this paper, the monotoncity of the error function -D in formula (2) and the weight sequence boundedness ' D ) D&C, ,⋯ via formula (4) ~ (5) for the online gradient method with inner-penalty are presented, under those condition both weakly and strongly convergence theorems are proved.