B-spline Neural Network using Artificial Immune Network Applied to
Identification of a Ball-and-Tube Prototype
Leandro dos Santos Coelho and Rodrigo Assunção
Production and Systems Engineering Graduate Program, PPGEPS
Pontifical Catholic University of Parana, PUCPR
Imaculada Conceição, 1155, Zip code 80215-901, Curitiba, Parana, Brazil
Abstract B-spline neural network (BSNN), a type of basis function neural
network, is trained by gradient-based methods that may fall into local minima
during the learning procedure. When using feed-forward BSNNs, the quality of
approximation depends on the control points (knots) placement of spline
functions. This paper describes the application of an artificial immune network
inspired optimization method − the opt-aiNet − to provide a stochastic search to
adjust the control points of a BSNN. The numerical results presented here indicate
that a artificial immune network optimization methods useful for building a good
BSNN model for the nonlinear identification of an experimental nonlinear balland-tube system.
Keywords: B-spline neural network, artificial immune system, nonlinear
identification.
Introduction
The use of neural networks to model chaotic systems and nonlinear
identification problems has attracted considerable attention in recent years [1]-[4].
A relevant approach is to find the best approximation with respect to a certain
class of basis functions for the representation neural of networks. In this case,
there are many possible choices of basis functions, such as radial basis function
[5], associated memory networks [6], wavelets [7], and B-spline functions [8].
The main advantage of the B-spline functions over other radial functions are
the local controls of the curve shape, since the curve only changes in the vicinity
of a few control points that have been changed [9]. A B-spline neural network
(BSNN) consists of the piecewise polynomials with a set of local basis functions
to model an unknown function for which a finite set of input-output samples are
available. The identification performance depends largely on an optimization
algorithm for the training procedure of the BSNN in order to avoid any possible
local minima.
In this context, the development of training methods and improvements for
BSNN is an emerging research area. Several heuristics have been developed in
recent years to improve the performance and set up the parameters of the BSSN
design and the fuzzy system approaches [10]-[17]. Recently, as an alternative to
the conventional mathematical approaches based on gradient information [18],
modern heuristic optimization techniques such as evolutionary algorithms [19]
2
Leandro dos Santos Coelho and Rodrigo Assunção
and swarm intelligence [20] have received much attention by many researchers
due to their ability to find an almost global optimal solution.
Artificial immune systems (AIS) are learning and optimization methods that
can be used for the solution of many different types of optimization problems [21],
[22], [23]. A meta-heuristic optimization approach employing artificial immune
networks called opt-aiNET algorithm to solve the knots of BSNN is proposed in
this paper. The aiNET algorithm is a discrete immune network algorithm based on
the artificial immune systems paradigm that was developed for data compression
and clustering [24], and was also extended slightly and applied to optimization to
create the algorithm opt-aiNET [25]. Opt-aiNET, proposed in [25], evolves a
population, which consists of a network of antibodies (considered as candidate
solutions to the function being optimized). These undergo a process of evaluation
against the objective function, clonal expansion, mutation, selection and
interaction between themselves.
In this paper, we propose modified opt-aiNET approach to train a BSNN.
Numerical results for identification of the nonlinear dynamics of an experimental
ball-and-tube system confirm the feasibility and effectiveness of the proposed
approach.
B-spline neural network
BSNN is introduced as a class of one-hidden-layer feedforward neural
networks composed of B-spline functions. BSNN is an example of associate
memory networks. The input space is defined over an n-dimensional lattice with
basis functions defined for each cell. Each basis function is composed of q
polynomial segments. A simple, stable recursive relationship exists to evaluate the
membership of a B-spline basis function of order k,
 x − λ j − q  j −1
 λj − x  j
 N ( x) + 
 N ( x)
N qj ( x) = 
 λ j −1 − λ j − q  q −1
 λ j − λ j − q +1  q −1




(1)
1 if ( x ∈ I j )
N1j ( x) = 
0 otherwise
(2)
where N qj (⋅) is defined as the j-th univariate basis function of order q and
λj
the
j-th knot and Ij is the j-th interval. The output of neural network is
p
oˆ k = f ( xk ) = ∑ w j N qj ( xk )
j =1
(3)
where xk and ôk are, respectively, the inputs and output of the network, wj is the
weight attached to the j-th basis function, and N qj (⋅) is given by the recursive
form (2). The index j is associative with the region of local support
B-spline Neural Network using Artificial Immune Network Applied to
Identification of a Ball-and-Tube Prototype 3
λ( j −q ) ≤ x ≤ λ( j ) , whereas the index q indicates the order of the basis functions
[20].
The main advantage of B-spline functions over other radial functions e. g., the
Bezier curve, is the local control of the shape of the curve, as the curve only
changes in the vicinity of a few control points that been changed [9].
The quality of approximation depends on the placement of knots of B-spline
functions. The purpose of optimizing a BSNN by opt-aiNET is to determine the
knots of each B-spline basis function. In particular, the number of basis functions
here depends on the user’s choice.
Opt-aiNET
Opt-aiNET is capable of performing local and global search, as well as to
adjust dynamically the size of population [27]. Opt-aiNET creates a memory set of
antibodies (points in the search space) that represent (over time) the best candidate
solutions to the objective function. Opt-aiNET is capable of either unimodal or
multimodal optimization and can be characterized by five main features [26]: (i)
the population size is dynamically adjustable; (ii) it demonstrates exploitation and
exploration of the search space; (iii) it determines the locations of multiple optima;
(iv) it has the capability of maintaining many optima solutions; and (v) it has
defined stopping criteria. The steps of opt-aiNET are summarized as follows:
A. Initialization of the parameter setup
The user must choose the key parameters that control the opt-aiNET, i.e.,
population size (M), suppression threshold (σs), number of clones generated for
each cell (Nc), percentage of random new cells each iteration (d), scale of affinity
proportion selection (β), and maximum number of iterations allowed (stop
criterion), Ngen.
B. Initialization of cell populations
Set iteration t=0. Initialize a population of i=1,..,M cells (real-valued ndimensional solution vectors) with random values generated according to a
uniform probability distribution in the n dimensional problem space. Initialize the
entire solution vector population in the given upper and lower limits of the search
space.
C. Evaluation of each network cell
Evaluate the fitness value of each cell (in this work, the objective of the fitness
function is to maximize the cost function).
D. Generation of clones
Generate a number Nc of clones for each network cell. The clones are offspring
cells that are identical copies of their parent cell [25].
E. Mutation operation
4
Leandro dos Santos Coelho and Rodrigo Assunção
Mutation is an operation that changes each clone proportionally to the fitness
of the parent cells, but keeps the parent cell. Clones of each cell are mutated
according to the affinity (Euclidean distance between two cells) of the parent cell.
The affinity proportional mutation is performed according to equations (4) and (5),
given by:
c' = c + α ⋅ N ( 0 ,1 )
(4)
α = β −1e − f *
(5)
where c' is a mutated cell c, N(0,1) is a Gaussian random variable of zero mean
and unitary standard deviation, β is a parameter that controls the decay of the
inverse exponential function, and f * is the fitness of an individual normalized in
the interval [0,1].
F. Evaluation the fitness of all network cells
Evaluate the fitness value of all network cells of the population including new
clones and mutated clones.
G. Selection of fittest clones
For each clone select the most fit and remove the others.
H. Determination of affinity of all network cells
Determine the affinity network cells and perform network suppression.
I. Generate randomly d network cells
Introduce a percentage d of randomly generated cells. Set the generation
number for t = t + 1. Proceed to step C until a stopping criterion is met, usually a
maximum number of iterations, tmax. The stopping criterion depends on the type of
problem.
Case study: Identification of ball-and-tube prototype
The prototype of the ball-and-tube process consists in a plastic tube, a DC
motor and a polystyrene sphere. The aim of this project is to control the height of
the sphere (ball) by applying a flux of air through the tube base, considering that
this air flux is generated by the application of tension in the DC motor.
A photograph and a general view of the ball-and-tube prototype design are
shown in figures 1 and 2, respectively. The project contains three main modules:
(i) ball-and-tube structure, (ii) hardware, and (iii) software.
B-spline Neural Network using Artificial Immune Network Applied to
Identification of a Ball-and-Tube Prototype 5
Figure 1. General view of ball-and-tube prototype.
Figure 2. Photograph of ball-and-tube system, a low cost prototype.
The hardware module of ball-and-tube prototype involves the distance sensor,
DC motor, interface circuit, and firmware. The interface circuit is composed by
the serial communication circuit, the distance sensor multiplexer circuit, and the
PWM (Pulse Width Modulator) circuit, as shown in figure 3. The Circuit Maker
Student Version software was chosen to build up the circuit diagrams.
6
Leandro dos Santos Coelho and Rodrigo Assunção
Figure 3. Hardware module of ball-and-tube prototype.
A personal computer with an acquisition data board to transmit the control sign
and to register the value of the height of the sphere in the tube (by the use of a
group of sensors based on phototransistors) is utilized. Through the acquisition
data board, a tension is converted into an output tension to be acquired by the
acquisition data board. In figure 4 it is presented the ball-and-tube prototype with
55 cm of height and 4.5 cm of diameter. The chosen DC motor (fan generation)
allows change the input tension from 0 to 20 V, those converted to values between
0 and 5 V (TTL level) to facilitate the construction of the power electronics
circuit.
The conception of this low cost ball-and-tube prototype aims at validating
nonlinear identification methods and the configuration of adaptive and fuzzy
control algorithms.
Identification results using BSNN with opt-aiNET
System identification is a procedure to identify a model of an unknown process,
for purposes of forecasting and/or understanding the dynamic behavior of a
dynamic system. In practice, system identification is an iterative procedure. The
lack of a priori information regarding the process model will require that each
step initially be examined superficially. The mathematical model employed in this
work to represent the yo-yo motion system is a NARX (Nonlinear AutoRegressive
with eXogenous inputs). In this case, the NARX model with series-parallel
conception is used for one-step ahead forecasting of the BSNN model (see figure
4).
A computer with a data acquisition board for generating the input signal was
used to obtain system measurements. In the identification procedure based on the
BSNN model, 330 samples of input (tension applied to the DC motor) and output
(ball height) were collected with a time sampling of 200ms. The tension value
corresponds to the maximum value configuration of the driver in PWM control of
a DC motor. The operation range used in experiments was between 9 cm (0.094
V) and 53 cm (4.707 V) of full height of tube (see details of data in figure 5).
B-spline Neural Network using Artificial Immune Network Applied to
Identification of a Ball-and-Tube Prototype 7
Figure 4. Structure of NARX model used in this work.
Figure 5. Input and output signals applied to ball-and-tube prototype.
Experiments for the estimation phase of the mathematical model of the balland-tube system are carried out using samples 1 to 170. For the validation phase,
the BSNN model uses the input and output signals of samples 171 to 330. The
system identification by BSNN model based on opt-aiNET is appropriate if a
performance index is in values permissible for the user’s needs. In this case, the
fitness function for maximization proposes using opt-aiNET and is given by the
harmonic mean of multiple correlation indices of estimation and validation phases.
2
The fitness function (to be maximized) is calculated using the expression of Rest
given by:
8
Leandro dos Santos Coelho and Rodrigo Assunção
170
2
Rest
2
∑ [ y (t ) − yˆ (t )]
= 1−
t =1
170
∑ [ y (t ) − y ]
2
(6)
t =1
2
where Rest
is the multiple correlation index of the estimation phase, y(t) is the
output of the real system, ŷ( t ) is the output estimated by the BSNN, and y is the
mean value of the system’s output. For the validation phase (verification of
2
generalization capability) of optimized BSNN, we have employed the Rval
index
give by
330
2
Rval
∑ [ y (t ) − yˆ (t )]
= 1 − t =171
330
2
∑ [ y (t ) − y ]
2
(7)
t =171
2
where Rval
is the multiple correlation index of the validation phase. When the
2
value R =1.0 (estimation or validation phases), it indicates the model’s accurate
approach to the system’s measured data. A R 2 value between 0.9 and 1.0 is
considered sufficient for applications in identification and model-based controller
designs [28].
All the computational programs were run on a 3.2 GHz Pentium IV processor
with 3 MB of RAM. In each case study, 30 independent runs were made for each
of the optimization methods involving 30 different initial trial solutions for each
optimization method. The setup of opt-aiNET algorithm used was: suppression
threshold = 5, percentage of newcomers: d=40%, scale of the affinity proportional
selection using a linear reduction of β with initial and final values of 10 and 100,
respectively, and the number of clones generated for each cell is Nc=10.
In the above case studies, the population size N was 20 and the stopping
criterion, tmax, was 200 generations for the opt-aiNET algorithm. The three chosen
vectors of the BSNN’s input were [ u(t-1); y(t-2); and y(t-1) ]. The space searches
for the knots of each B-spline basis function are [-1.0; 1.0]. Simulation tests were
conducted using 3 to 5 knots in each input of the BSNN.
Table 1 presents the simulation results (best of 30 experiments) using optaiNET for optimization of the BSNN using 4 knots. As indicated in Table 1, the
results of the optimized BSNN are precise, providing an appropriate experimental
mathematical model for the ball-and-tube prototype. The best result shown in
Figure 5 represents the BSNN using opt-aiNET with 4 knots for each network
input.
B-spline Neural Network using Artificial Immune Network Applied to
Identification of a Ball-and-Tube Prototype 9
2
Table 1. Results obtained by the maximization of Rest
using opt-aiNET
2
after completing 30 runs)
(adopted here is the solution with the best Rest
knots for each
BSNN’s input
3
4
5
2
Rest
2
Rval
maximum
(best)
mean
minimum
(worst)
standard
deviation
0.9331
0.9472
0.9470
0.9246
0.9411
0.9361
0.9178
0.9359
0.9143
0.0078
0.0056
0.0189
2
(using Rest
best result)
0.8109
0.8398
0.8211
2
Figure 5. Best result of Rest
for BSNN using opt-aiNET (see Table 1).
Conclusion and future research
There is a natural parallel between the immune system and optimization. In
this context, one well-known immune inspired algorithm for function optimization
is the opt-aiNET. The op-aiNET is inspired by the idiotypic network theory for
explaining the immune system dynamics.
This paper describes the application of opt-aiNET algorithm to adjust the
control points of a BSNN. Our simulation results confirmed the potential of optaiNET algorithm for BSNN optimization for the nonlinear identification of an
experimental nonlinear ball-and-tube system. Further studies are needed to test the
opt-aiNET algorithm on benchmark optimization problems in system
identification and power systems.
References
[1] Narendra, K. S.; Parthasarathy, K. (1990). Identification and control of dynamical systems using
neural networks, IEEE Transactions on Neural Networks, Vol. 1, No. 1, pp. 4-27.
[2] Karayiannis, N. B.; Randolph-Gips, M. M. (2003). On the construction and training of reformulated
radial basis function neural networks, IEEE Transactions on Neural Networks, Vol. 14, No. 14,
pp. 835-846.
[3] Mcloone, S.; Brown, M. D.; Irwin, G.; Lightbody, A. (1998). A hybrid linear/nonlinear training
algorithm for feedforward neural networks, IEEE Transactions on Neural Networks, Vol. 9, No.
4, pp. 669-684.
[4] Sierakowski, C. A..; Guerra, F. A.; Coelho, L. S. (2005). Particle swarm optimization approach for
multi-step-ahead prediction using radial basis function neural network, Proceedings of 16th IFAC
World Congress, Prague, Czech Republic.
10
Leandro dos Santos Coelho and Rodrigo Assunção
[5] Yingwei, L.; Sundararajsn, N.;. Saratchandran, P. (1997). Identification of time-varying nonlinear systems
using minimal radial basis function neural networks, IEE Proc.-Control Theory Applications, Vol. 144,
No. 2, pp. 202-208.
[6] Harris, C. J.; Moore, C. G.; Brown, M. (1993). Intelligent control: aspects of fuzzy logic and neural
nets, Word Scientific.
[7] Zhang, Q.; Benveniste, A. (1992). Wavelet networks, IEEE Transactions on Neural Networks, Vol.
3, No. 6, pp. 889-898.
[8] Lightbody, G.; O’Reilly, P.; Irwin, G. W.; Kelly, K.; McCormick, J. (1997). Neural modelling of
chemical plant using MLP and B-spline networks, Control Engineering Practice, Vol. 5, No. 11,
pp. 1501-1515.
[9] Newmann, W. M.; Sproull, R. F. (1979). Principles of interactive computer graphics, McGrawHill, Inc.
[10] Wang, K; Lei, B. (2001). Using B-splines neural network to extract fuzzy rules for a centrifugal
pump monitoring, Journal of Intelligent Manufacturing, Vol. 12, No. 1, pp. 5-11.
[11] Wang, H.; Oyebande, B. (1995). On the application of neural network modeling to a wet end
chemical process in paper making, Proceedings of IEEE Conference on Control Applications,
Albany, NY, pp. 657-662.
[12] Starrenburg, G.; van Luenen, W. T. C.; Oelen, W.; van Amerongen, J. (1996). Learning
feedforward controller for a mobile robot vehicle, Control Engineering Practice, Vol. 4, No. 9, pp.
1221-1230.
[13] Zhang, J.; Knoll, A. (1999). Designing fuzzy controllers by rapid learning, Fuzzy Sets and
Systems, Vol. 101, pp. 287-301.
[14] Yiu, K. F. C.; Wang, S.; Teo, K. L.; Tsoi, A. C. (2001). Nonlinear system modeling via knotoptimizing B-spline networks, IEEE Trans. on Neural Networks, Vol. 12, No. 4, pp. 1013-1022.
[15] Saranli, A.; Baykal, B. (1998). Complexity reduction in radial basis function (RBF) networks by
using radial B-spline functions, Neurocomputing, Vol. 18, No. 1-3, pp. 183-194.
[16] Shimojima, K.; Fukuda, T.; Arai, F. (1994).Self-tuning fuzzy inference based on spline function,
Proceedings of IEEE International Conference on Fuzzy Systems, Orlando, FL, USA, pp. 690-695.
[17] Chua, V. K.; Tomizuka, M. (1995). Rule generation for fuzzy systems based on B-splines, Proc.
IEEE Int. Conference on Neural Networks, Perth, Australia, pp. 6098-611.
[18] Chan, C. W.; Jin, H.; Cheung, K. C. ; Zhang, H. Y. (1998). An adaptive learning rate for the
training of B-spline networks, Proc. UKACC Int. Conf. Control, Swansea, UK, pp. 342-347.
[19] Wang, W. -Y.; Lee, T. -T.; Hsu, C. -C.; Li, Y. -H. (2002). GA-based learning of BMF fuzzyneural network, Proc. Int. Conference on Fuzzy Systems, Honolulu, HI, USA, pp. 1234-1238.
[20] Coelho, L. S.; Krohling, R. A. (2006). Nonlinear system identification based on B-spline neural
network and modified particle swarm optimization, Proc.IEEE World Conference on
Computational Intelligence (WCCI), Int. Joint-Conf. on Neural Networks, Vancouver, Canada.
[21] Ishida, Y.; Hirayama, H.; Fujita, H.; Ishiguro, Mori, K. (eds.) (1998). Immunity-based systems –
intelligent systems by artificial immune systems, Corona Pub. Co. Japan (in Japonese).
[22] Dasrupta, D. (ed.) (1999). Artificial immune systems and their applications, Springer-Verlag.
[23] De Castro, L. N.; Timmis, J. I. (2003). Artificial immune systems as a novel soft computing
paradigm, Soft Computing Journal, Vol. 7, No. 7, pp. 526-544
[24] De Castro, L. N.; Von Zuben, F. (2001). AINET: an artificial immune network for data analysis,
Data Mining: a heuristic approach, Abbas, H.; Sarker, R.; Newton, C. (eds.), Idea Group
Publishing.
[25] De Castro, L. N.; Timmis, J. (2002) An artificial immune network for multimodal function
optimization, Proc.IEEE Congress on Evolutionary Computation, Hawaii, USA, pp. 699-674.
[26] Timmis, J.; Edmonds, C. (2004). A comment on opt-AiNET: an immune network algorithm for
optimisation, D. Kalyanmoy et al. (eds.), Genetic and Evolutionary Computation, vol. 3102,
Lecture Notes in Computer Science, Springer, pp. 308-317.
[27] Campelo, F.; Guimarães, F. G.; Igarashi, H.; Ramírez, J. A.; Noguchi, S. (2006). A modified
immune network algorithm for multimodal electromagnetic problems, IEEE Transactions on
Magnetics, Vol. 42, No. 4, pp. 1111-1114.
[28] Schaible, B.; Xie, H.; Lee, Y. C. (1997). Fuzzy logic models for ranking process effects, IEEE
Transactions on Fuzzy Systems, Vol. 5, No. 4, pp. 545-556.