The Most General Intelligent Architectures of the Hybrid Neuro-Fuzzy Models

: Hybrid systems of the fuzzy logic and neural networks, are widely spread in real world problems with high effectiveness and versatility for different kinds of applications. The state description of unknown plant by using mathematical models, sometimes, is difficult to obtain. The fuzzy logic systems with their ability of tackling imprecise knowledges, and neural networks with their advantages of establishing a relationship between the inputs and the outputs of the system, are represented as qualified tools for systems of unknown plant. Furthermore, the hybrid systems which utilize the features of the fuzzy logic and Neural networks has been employed for better characteristics. Whilst, there are several different architectures of the neuro-fuzzy system proposed in literature, this article come out to highlight the common known architectures of how these techniques fuse together to build an enhanced system that can complement the lack of each method individually and improve the system performance over all.


Introduction
The capability of computers, and the innovation in soft computing boosted the usability and applications of modern techniques, almost they have been emerged in all science with more great impact in the engineering field. Neuro-fuzzy systems represent a type of hybrid intelligent systems which combines the main features of artificial neural networks and fuzzy logic systems to process the described associated problem.
Fuzzy logic is a technique, was initiated by Lotfi Zadeh in 1965 [1]. The basic of the fuzzy logic is to deal with imprecise variables to represent a crisp set. The fuzzy logic system mimics the human sort of describing and representing of things. Crisp values along the fuzzy system go through different stages, these stages are represented in: Firstly, assigning a function to the crisp values which indicates the strength of the association in the distributed set, this function is called membership function, and it takes values between 0 to 1. Mathematically, this function can be described as, ∈ [0,1], it considers as the membership function of element in the set of . Secondly, according to the rules base and rules inference, a suitable mapping between the inputs and the outputs data will be set up. The rules base act as the prediction of the system when the specified inputs are applied. This prediction is formulated as fuzzy antecedentconsequent according to the designer knowledges of the current problem. The fuzzy rules can be written as: IF <antecedent> THEN <consequent>. The fuzzy inference system works to make the set of the selected rule executed and decide the associated values which belong to them. From literature there are different methods for fuzzy inference, Mamdani inference and Sugeno inference, are the most popular ones [2]. Lastly, the crisp value of the fuzzy system can be obtained by the process of the defuzzification. Neural networks or more specifically artificial neuron networks(ANN), are mathematical models which have the capability of learning like biological neurons, were originated by McCulloch and Pitts (1943). Neural networks are useful to build a relationship between the inputs and the outputs of the system. Neural networks model simulates the performance of the biological systems. The nodes in neural networks emulate the action of a neuron, links to these nodes like synapses, and weights between nodes simulate the action of the neurotransmitters. Different algorithm can be used under the learning process to develop such kind of mapping between the input and the output samples. The research has been conducting for learning algorithms since the last fifties. The most brighten algorithm is the backpropagation algorithm, was introduced by Rumelhart et al. in 1986.
Fuzzy logic and neural networks, are the types of the artificial intelligent with same ground base, both of them are inspiration of the human computational, and they don't require mathematical models. Thus, in order to enhance the ability of both method and to increase the system performance at all, the researchers have proposed new structures which combining the two methods to generate hybrid neuro-fuzzy systems. These architectures, are proposed to overcome the weakness of the fuzzy logic and neural networks to solve problems which involving at the same time both linguistic and numerical knowledges. Table 1. Shows the features of neural networks and fuzzy logic -based system.

Architectures of the Neuro-Fuzzy System
The complementary features of the fuzzy logic and neural networks make them ideal solution in individual use for some kinds of artificial intelligent but not all. For example, system modeling technique through the neural networks, is obviously well satisfied for intelligent control but less satisfy for decisions making. Apart of that, fuzzy logic systems can easily to be handling imprecise data and explain their decisions, but they aren't capable to acquire these decisions automatically. Such capabilities and restrictions of individual intelligent technologies urged the researchers to produce such kinds of hybrid system (neuro-fuzzy) to deal with more complicated problems. Different architectures of hybrid neuro-fuzzy systems can be found in literature. Lee S. C. and Lee E. T. [3,4,5], they were first introduced the concepts of the fuzzy set into neural networks, since then the neuro-fuzzy system initially devolved [6,7,8]. Through all the developments, different kinds of architectures have been proposed. According to the obtained network topology and architecture [9,10], the hybrid neuro-fuzzy systems can be divided into three main categories:

Cooperative Neuro-Fuzzy Systems
The cooperative neuro-fuzzy systems work to adjust and tune the fuzzy inference system by using the learning algorithms of the neural networks. Both of neural network and fuzzy logic in this kind of system, they work independently. Hence, during the network adjusting at a certain level the fuzzy system remains without any change. The tuning process can be carried out by adjusting a specific fuzzy system parameter. With the accordance of which parameters could be adjusted, the cooperative neuro fuzzy systems can be classified into four types: 1-The cooperative neuro-fuzzy systems for adjusting the membership function. In this type, the output of the neural networks is connected to the fuzzy system, the neural networks are trained to represent a number of fuzzy sets. 2-Cooperative neuro-fuzzy systems for extracting fuzzy rules from training data. In this scheme, firstly, the neural network learns the rules, and then implements them in the fuzzy system. Predetermined information is given to the fuzzy system as well. 3-The cooperative neuro-fuzzy systems for regularly update the fuzzy system structure. For this kind, the tuning of the determined parameters is applied during the online use of the fuzzy system. A feedback from the fuzzy system is required in this structure. 4-The cooperative neuro-fuzzy systems for identifying the weighted fuzzy rules. In this class, one of the learning method is used to fire the importance of each rule in the fuzzy system. Weights are set to each rule using similar procedures to that provided in [11]. In all types of cooperative neuro-fuzzy systems, neural networks are just used to tune a specific parameter of the fuzzy system and then, are removed, and only the fuzzy system is executed. Figure 1. Depicts the general architecture of the cooperative neuro-fuzzy system.

Neural Networks -Driven Fuzzy Reasoning
This architecture of the neuro-fuzzy system is proposed to solve the problem when the number of the linguistic variables of the fuzzy system is increased, it may not be easy to construct the fuzzy rules. Numerous of researches induced to solve this issue [12,13,14]. The most known method is the neural network-driven fuzzy reasoning (NNDF) introduced by Takagi and Hayashi [15]. The principle of the Takagi-Hayashi method is to implement the membership function of the antecedent in the inference function of the consequent. Thus, a suitable neural network is used to implement the principle. The Takagi-Hayashi method can be divided into three major parts. Figure 2. shows the principle of the NNDF: 1. Partitioning the inference rules.
2. The identification of IF parts (the determination of a membership function). 3. The identification of THEN parts (the determination of the amount of control for each rule). In the first part, the fuzzy inference rules and the combination of the data belong to it, are determined. In the second part, any arbitrary input of each rule is related to the corresponding rule. By the last part, the conclusion of (THEN part) will be derived.

Hybrid Neural Networks -Based Systems
Hybrid neural networks -based systems, are based on an architecture which integrates the neural networks and the fuzzy logic based system in the form of parallel structure. Unlike cooperative neuro-fuzzy architectures, the neural networks and fuzzy logic system work synchronously. And they exploit the same learning algorithm of the neural networks itself. The synchronization of hybrid neural-fuzzy systems can be viewed as a result of the parallelism matching of the fuzzy system into a neural network structure. Therefore, each of the fuzzy logic sub-blocks can be represented by a particular layer in the neural networks. Numerous of researches have been conducting for hybrid neuro-fuzzy architectures by different number of layers, each researcher has defined its own particular model. In [16], an on line self-constructing neuro-fuzzy inference system was proposed using six layers. L. Maguire et al. presented a neuro-fuzzy system to approximate fuzzy reasoning using only three layers [17]. The most common recent structures of the hybrid neuro-fuzzy systems are: The Adaptive-Network-based Fuzzy Inference System (ANFIS), and the Fuzzy Wavelet Neural Networks (FWNN).

Adaptive-Network-Based Fuzzy Inference System
(ANFIS) Adaptive-Network-based Fuzzy Inference System (ANFIS), was invented by J-S Jang [18]. The ANFIS executes a Sugeno -type fuzzy inference system, where the fuzzy rules can be described by the following form: : If is and is ... and is , then Where is -input linguistic variable in the antecedent part of the -rule with = (1,..., n), and is the linguistic label associated with it. has its associated membership function given by . is the consequent output of the -rule, and α ,..., α , are the Sugeno parameters.
The ANFIS embodies the input data by using a membership function with different parameters, then through the output membership function, it concludes the result. The initial membership function and rules of the fuzzy inference system, can be driven according to the acquired knowledges of the target system. The ANFIS uses totally five layers to implement different node functions of learning and adjusting of the parameters. The hybrid learning algorithm works to tune the parameters of the consequent and antecedent parts of the fuzzy rule based system, it merges between gradient decent method and least mean square method to achieve the system purposes. Figure  3. Shows the ANFIS five layers architecture.
The least mean square method is used first to identify the optimal values of the consequent parameters while the premise parameters are fixed. Then, the gradient decent method will be applied to update the system parameters at all. The five layers ANFIS can be defined sequentially as follows: Fuzzy layer, it's the first layer which has function to produce the membership grade of linguistic variable, every node in this layer is characterized by its corresponding output function. The output function can be stated as:

= (2)
Where is the node input, and is the linguistic label associated with the node function. For bell-shaped membership function, each node will be stored three parameters , , to represent the associated linguistic variables. The bell-shaped membership function has the following equation: Product layer, this layer stores the rules, each rule has one node, all nodes in this layer are connected to those nodes in the previous layer that forms the antecedent of the rule. The output is calculating via product of all incoming signals. The output equation has shape as follows: . = ( ) ⊗ ( ), ! 1, 2; 2 ! 1, 2; ! 1, . . . ,4.
Normalized layer, the nodes in this layer indicates normalization to the firing strength from previous layer. The normalization is to calculate the ratio of the 5 6 rule's firing strength to the sum of all rules firing strengths. The output of this layer can be expressed as: where . is the firing strength of the 6 rule. De-fuzzy layer, the nodes in this layer are connected to all input nodes and with exactly one node in layer three. The nodes in this layer compute the consequent of the rules, ( => 5 ? ). The output of this layer is obtained by the multiplication process between the normalized firing rule strength and first order polynomial. the output equation can be written as: Where . 7 represents the layer 3 output, and (B , B , B ) are the parameters set.
Output layer, is the last layer where the crisp output value is deduced. This layer computes the overall output as the summation of the incoming signals. The crisp output value can be derived by the following equation: Therefore, when the values of the premise parameters in the F => ?GH= are fixed, the final output of the adaptive neuro-fuzzy system can be expressed as linear combination of the consequent parameters. The system overall has the same function of Sugeno type fuzzy inference system. This type of architecture has more wide spread in science and engineering.

Wavelet Fuzzy Neural Networks (FWNN)
Based on wavelet transform and its ability to represent the signals in both time and frequency domain, the neural networks which combine neurons use activation function drawn from an orthonormal wavelet family, are formerly proposed in literature [19,20]. And, by the aim of increase the approximation capability of fuzzy neural networks and educational properties of neural networks, such kind of structures which utilize wavelet function are introduced. Different syntheses of fuzzy logic and wavelet neural networks have been proposed by researchers [21,22]. The fuzzy wavelet neural network (FWNN), is similar to the traditional ANFIS except that, the outputs of the fuzzy rules in the normalized layer feed a layer of wavelet's neurons instead of Takagi-Sugeno polynomial models. Figure 4. Views the six layers fuzzy wavelet networks. The layers from 1 to 6 execute the same tasks as ANFIS model except the fifth layer where the Takagi-Sugeno model has been replaced by wavelet model. The nodes in this layer hold wavelet activation function with adjustable parameters, these parameters will be tuned during the learning process. Therefore, the wavelet function must be differentiable function. The output of this layer can be described according to the Mexican Hat family of wavelets as follows: F ! I̅ Ψ 5 ! 1,2, … . , M Where I̅ and Ψ , are the output of the fourth layer, and wavelet function for 2 6th neuron respectively. N O , is the synaptic weight and Z O , O , are the wavelet dilations and translation parameters, while , is the input vector. An appropriate learning method will be used for parameters adjustment. This class of the neuro-fuzzy systems are given more intention in the recent research, and its proved that the FWNN has fast variation over ANFIS in nonlinear system identification [23].

Conclusion
In this article, the most general architectures of the neuro-fuzzy systems were presented for ill-defined system. Due to the vast framework of these architectures, it's difficult to compare conceptually between them and to evaluate their performance individually, each one, it may reflect the satisfied solution for a specified case. Through the proposed architectures the systems would be able to deal with both numerical and linguistic knowledges. The hybrid neuro-fuzzy architectures, are reflected a strong structure when the prediction of the input-output is obtained. Meanwhile, they don't require analytical description of the systems, so the designer can just start with an intuitive realistic membership function and by the use of learning methods the system will approximate the desired output. The ANFIS and the FWNN are shown to be the robust architectures type of the system modeling. For more accurate and fast convergence of the architectures, it would be more effective to execute learning methods like extended Kalman filter or genetic algorithm instead of gradient decent method.