A Novel PSO Based Back PropagationLearning-MLP (PSO-BP-MLP)for Classification

Himansu Das, Ajay Kumar Jena, Janmenjoy Nayak, Bighnaraj Naikand H.S. Behera

Abstract Particle swarm optimization (PSO) is a powerful globally acceptedevolutionary swarm intelligence method for solving both linear and non-linearproblems. In this paper, a PSO based evolutionary multilayer perceptron is pro-posed which is intended for classification task in data mining. The network istrained by using the back propagation algorithm. An extensive experimental anal-ysis has been performed by comparing the performance of the proposed methodwith MLP, GA-MLP. Comparison result shows that, PSO-MLP gives promisingresults in majority of test case problems.

Keywords Data mining � Classification � Particle swarm optimization � Geneticalgorithm � Multilayer perceptron

1 Introduction

PSO is a meta-heuristic evolutionary optimization technique which can be directlyapplied in a continuous global space environment and was proposed by Kennedyand Eberhart [1, 2]. Due to its simplest algorithmic structure, less parameter use and

H. Das (&) � A.K. JenaSchool of Computer Engineering, KIIT University, Bhubaneswar, Odisha, Indiae-mail: das.himansu2007@gmail.com

A.K. Jenae-mail: ajay.bbs.in@gmail.com

J. Nayak � B. Naik � H.S. BeheraDepartment of Computer Science Engineering and Information Technology, Veer SurendraSai University of Technology, Burla, Sambalpur 768018, Odisha, Indiae-mail: mailforjnayak@gmail.com

B. Naike-mail: mailtobnaik@gmail.com

H.S. Beherae-mail: mailtohsbehera@gmail.com

© Springer India 2015L.C. Jain et al. (eds.), Computational Intelligence in Data Mining - Volume 2,Smart Innovation, Systems and Technologies 32, DOI 10.1007/978-81-322-2208-8_42

461

free from gradient use of an objective function, it is quite popular in the swarmintelligence community. The conceptual development of PSO is based upon thebehavior of the swarms like bird, fish etc. Strong convergence and global optimi-zation solution makes this algorithm more popular and attracts the attention of theresearchers for solving various wide range of diversified problems [3–9].

A hybrid PSO algorithm (DNPSO) with the diversity enhancement and neigh-borhood search has been proposed by Wang et al. [10]. Neri et al. [11] addressed anovel optimization method called Compact PSO which employs the probabilisticrepresentation and search logic of PSO but does not use either the position orvelocity. A fuzzy based hybridized PSO is proposed by Valdez et al. [12] toimprove the performance of the modular neural network. A general method forobject detection in images based on deformable models and swarm intelligence/evolutionary optimization algorithms is proposed by Ugolottia et al. [13]. SHIN andKITA [14] used the second global best and second personal best particles toimprove the performance of the original PSO. Akay [15] proposed two swarm-intelligence-based global optimization algorithms, namely particle swarm optimi-zation (PSO) and artificial bee colony (ABC), those have been applied to find theoptimal multilevel thresholds. A Particle swarm optimization with grey evolu-tionary analysis for performing a global search over the search space with fasterconvergence speed has been introduced by Leu et al. [16]. Sun et al. [17] describeda novel PSO technique called FEPSO, for reducing the number of fitness evalua-tions as well as computational cost. Imran et al. [18] gave an brief overview onvarious variants of PSO and analyzed on the parameters performance of PSO. PANet al. [19] analyzed the performance of a standard PSO based on Markov chain bydefining the state sequence of a single particle or swarm.

In this study, a PSO based back propagation trained multilayer perceptron isproposed for data classification. The proposed method has been tested with variousbenchmark datasets considered from UCI machine learning repository. The rest ofthe paper is organized in the following manner. Section 2 introduces some basicconcepts like PSO and MLP. Section 3 describes the method of proposed work.Section 4 presents Experimental Setup and Result Analysis. Section 5 concludesour work with future scope.

2 Preliminaries

2.1 Particle Swarm Optimization

Particle swarm optimization (PSO) [1, 2] is a widely used stochastic based algo-rithm and it is able to search global optimized solution. Like other population basedoptimization methods, the particle swarm optimization starts with randomly ini-tialized population for individuals and it works on the social behavior of particle to

462 H. Das et al.

get the global best solution by adjusting each individual’s positions with respect toglobal best position of particle of the whole population (Society). Each individual isadjusting by changing the velocity according to its own experience and byobserving the position of the other particles in search space by use of Eqs. 1 and 2.Equation 1 is for social and cognition behavior of particles respectively where c1and c2 are the constants in between 0 and 2 and rand (1) is random function whichproduces random number between 0 and 1.

Vi t þ 1ð Þ ¼ Vi t þ 1ð Þ þ c1 � rand 1ð Þ � lbesti � Xið Þ þ c2 � rand 1ð Þ � gbesti � Xið Þð1Þ

Xi t þ 1ð Þ ¼ Xi tð Þ þ Vi t þ 1ð Þ ð2Þ

Basic steps of PSO can be visualized as:

Initialize the position of particles Vi tð Þ (population of particles) and velocity ofeach particle Xi tð Þ.Do

Compute fitness of each particle in the population.Generate local best particles (LBest) by comparing fitness of particles inprevious population with new population.Choose particle with higher fitness from local best population as global bestparticle (GBest).Compute new velocity Vi t þ 1ð Þ by using Eq. 1.Generate new position Xi t þ 1ð Þ of the particles by using Eq. 2.

While (iteration <= maximum iteration OR velocity exceeds predefined velocityrange);

2.2 Multilayer Perceptron

MLP (Fig. 1) is the simplest neural network model which is consists of neuronscalled perceptron (Rosenblatt 1958). From multiple real valued inputs, the per-ceptron compute a single output according to its weights and non-linear activationfunctions. Basically MLP network is consists of input layer, one or more hiddenlayer and output layer of computation perceptron.

MLP is a model for supervised learning which uses back propagation algorithm.This consists of two phases. In the 1st phase, error (Eq. 4) based on the predictedoutputs (Eq. 3) corresponding to the given input is computed (forward phase) and inthe 2nd phase, the resultant error is propagated back to the network based on thatweight of the network are adjusted to minimize the error (Back Propagation phase).

A Novel PSO Based Back Propagation … 463

y ¼ fXn

i¼1

wixi þ b

!ð3Þ

where w is the weight vector, x is the input vector, b is the bias and f :ð Þ is the non-linear activation function.

dk ¼ tk � ykð Þf yinkð Þ ð4Þ

where tk and yk is the given target value and predicted output value of input kthpattern and dk is the error term for kth input pattern.

The popularity of MLP increases among the neural network research communitydue to its properties like nonlinearity, robustness, adaptability and ease of use. Alsoit has been applied successfully in many applications [20–28]. It can also be appliedto some advanced computing applications [29–32] of grid computing.

3 Proposed Method

In this section, we have proposed a PSO based back propagation learning-MLP(PSO-BP-MLP) for classification. Here basic concepts and problem solving strat-egy of PSO evolutionary algorithm is used to enhance performance of MLPclassifier.

Fig. 1 MLP with input layer, single hidden layer and output layer

464 H. Das et al.

Algorithm PSO based Back Propagation Learning- MLP (PSO-BP-MLP) forclassification

A Novel PSO Based Back Propagation … 465

466 H. Das et al.

A Novel PSO Based Back Propagation … 467

4 Experimental Setup and Result Analysis

In this section, the comparative study on the efficiency of our proposed method hasbeen presented. Benchmark datasets (Table 1) from UCI machine learning repos-itory [33] and KEEL dataset repository [34] have been used for classification andthe result of proposed PSO-MLP model is compared with MLP, GA-MLP based onGenetic Algorithm. Datasets information is presented in Table 1. Datasets havebeen normalized and scaled in the interval −1 to +1 using Min-Max normalizationbefore training and testing is made. Classification accuracy (Eq. 5) of models hasbeen calculated in terms of number of classified patterns are listed in Table 2.

If cm is confusion matrix of order m × n then, accuracy of classification iscomputed as:

Clasification Accuracy ¼

Pni¼1

Pmj ¼ 1;i ¼¼ j

cmi;j

Pni¼1

Pmj ¼ 1 cmi;j

� 100 ð5Þ

Table 1 Data set information

Dataset Numberof pattern

Number offeatures (excludingclass label)

Numberof classes

Numberof patternin class-1

Numberof patternin class-2

Numberof patternin class-3

Monk 2 256 06 02 121 135 –

Hayesroth 160 04 03 65 64 31

Heart 256 13 02 142 114 –

Newthyroid

215 05 03 150 35 30

Iris 150 04 03 50 50 50

Pima 768 08 02 500 268 –

Wine 178 13 03 71 59 48

Bupa 345 06 02 145 200 –

Table 2 Performance comparison in terms of accuracy

Dataset Accuracy of classification in average

MLP GA-MLP PSO-MLP

Train Test Train Test Train Test

Monk 2 86.94648 85.27453 87.23734 87.85732 90.19375 92.44732

Hayesroth 83.48576 82.38657 85.43675 81.04653 88.38271 81.97365

Heart 82.84653 74.77453 85.44937 75.03645 86.23755 75.84651

New thyroid 92.03782 73.26876 92.74834 73.92756 93.02785 75.92784

Iris 90.87365 92. 15368 92. 56873 95. 93158 92. 51736 93. 36158

Pima 73.73645 73.88647 76.82642 77.38275 78.17464 78.28746

Wine 80.69731 80.87294 88.93645 77.37565 90.75389 91.77319

Bupa 68.66251 69.82637 70.97485 71.27459 70.27841 71.21465

468 H. Das et al.

4.1 Parameter Setting

During simulation, c1 and c2 constants of PSO has been set to 2 and usedthroughout the experiment. In MLP, one input layer, one hidden layer and oneoutput layer for the neural network has been set during training and testing.

5 Conclusion

PSO is a popular interesting swarm intelligence technique which is able to find boththe global minima and maxima for complex problems. This paper describes thePSO based back propagation neural network for classification of various benchmarkdatasets. The comparison of performance analysis of the results indicates that theproposed method gas better classification accuracy than the other defined methods.But many researchers have found out some of the major limitations of PSO like:slow convergence, convergence at local minima and large search space etc. In eachiteration, the time complexity is more due to the search of the weakest performedparticle. In future, our work may extend in this interest by better adjustment of allthe parameters along with hybridization of some higher order neural network.

References

1. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the 1995 IEEEInternational Conference on Neural Networks, vol. 4, pp. 1942–1948 (1995)

2. Kennedy, J., Eberhart, R.: Swarm intelligence morgan kaufmann, 3rd edn. Academic Press,New Delhi (2001)

3. Cai, J., Pan, W.D.: On fast and accurate block-based motion estimation algorithms usingparticle swarm optimization. Inf. Sci. 197(15), 53–64 (2012)

4. Du, W., Li, B.: Multi-strategy ensemble particle swarm optimization for dynamicoptimization. Inf. Sci. 178(15), 3096–3109 (2008)

5. Zhang, Y., et al.: A bare-bones multi-objective particle swarm optimization algorithm forenvironmental/economic dispatch. Inf. Sci. 192(1), 213–227 (2012)

6. Chuanwen, J., Bompardb, E.: A hybrid method of chaotic particle swarm optimization andlinear interior for reactive power optimization. Math. Comput. Simul. 68(1), 57–65 (2005)

7. Li, Y., et al.: Dynamic optimal reactive power dispatch based on parallel particle swarmoptimization algorithm. Comput. Math Appl. 57(11–12), 1835–1842 (2009)

8. Li, Y., et al.: Optimal reactive power dispatch using particle swarms optimization algorithmbased pareto optimal set. Lect. Notes Comput. Sci. 5553, 152–161 (2009)

9. Naik, B., Nayak, J., Behera, H.S.: A Novel FLANN with a hybrid PSO and GA based gradientdescent learning for classification. In: Proceedings of the 3rd International Conference onFrontiers of Intelligenct Computing (FICTA). Advances in Intelligent Systems and Computing327, vol. 1, 745–754 (2015). doi:10.1007/978-3-319-11933-5_84

A Novel PSO Based Back Propagation … 469

10. Wang, H., et al.: Diversity enhanced particle swarm optimization with neighborhood search.Inf. Sci. 223, 119–135 (2013)

11. Neri, F., et al.: Compact particle swarm optimization. Inf. Sci. 239, 96–121 (2013)12. Valdez, F., et al.: Modular neural networks architecture optimization with a new nature

inspired method using a fuzzy combination of particle swarm optimization and geneticalgorithms. Inf. Sci. 270, 143–153 (2014)

13. Ugolottia, R., et al.: Particle swarm optimization and differential evolution for model-basedobject detection. Appl. Soft Comput. 13, 3092–3105 (2013)

14. Shin, Y., Kita, E.: Effect of second best particle information for particle swarm optimization.Procedia Comput. Sci. 24, 76–83 (2013)

15. Akay, B.: A study on particle swarm optimization and artificial bee colony algorithms formultilevel thresholding. Appl. Soft Comput. 13, 3066–3091 (2013)

16. Leu, M.-S., et al.: Particle swarm optimization with grey evolutionary analysis. Appl. SoftComput. 13, 4047–4062 (2013)

17. Sun, C., et al.: A new fitness estimation strategy for particle swarm optimization. Inf. Sci. 221,355–370 (2013)

18. Imran, M., et al.: An overview of particle swarm optimization variants. Procedia Eng. 53,491–496 (2013)

19. Pan, F., et.al. : Analysis of standard particle swarm optimization algorithm based on markovchain. Acta Automatica Sinica. vol. 39, no. 4. April (2013)

20. Mahyar, H., et al. : Comparison of multilayer perceptron and radial basis function neuralnetworks for EMG-based facial gesture recognition. In: Proceedings of the 8th internationalconference on robotic, vision, signal processing and power applications. Springer, Singapore(2014)

21. Ndiaye, A., et al.: Development of a multilayer perceptron (MLP) based neural networkcontroller for grid connected photovoltaic system. Int. J. Phys. Sci. 9(3), 41–47 (2014)

22. Roy, M., et al.: Ensemble of multilayer perceptrons for change detection in remotely sensedimages. Geoscience and remote sensing letters. IEEE11.1, pp. 49–53 (2014)

23. Hassanien, A., et al.: MRI breast cancer diagnosis hybrid approach using adaptive ant-basedsegmentation and multilayer perceptron neural networks classifier. Appl. Soft Comput. 14,62–71 (2014)

24. Aydin, K., Kisi, O.: Damage detection in Timoshenko beam structures by multilayer perceptronand radial basis function networks. Neural Comput. Appl. 24(3–4), 583–597 (2014)

25. Velo, R., et al.: Wind speed estimation using multilayer perceptron. Energ. Convers. Manag.81, 1–9 (2014)

26. Lee, S.: Choeh. J. Y. : Predicting the helpfulness of online reviews using multilayer perceptronneural networks. Expert Syst. Appl. 41(6), 3041–3046 (2014)

27. Azim, S., Aggarwal, S. : Hybrid model for data imputation: Using fuzzy c means and multi layerperceptron. 2014 IEEE International Advance computing conference (IACC). IEEE (2014)

28. Chaudhuri, S., et al.: Medium-range forecast of cyclogenesis over North Indian Ocean withmultilayer perceptron model using satellite data. Nat. Hazards 70(1), 173–193 (2014)

29. Das, H., Mishra, S.K., Roy, D.S.: The topological structure of the Odisha power grid: acomplex network analysis. IJMCA 1(1), 012–016 (2013)

30. Das, H., Roy, D. S. : A grid computing service for power system monitoring. Int. J. Comput.Appl. 62 (2013)

31. Das, H., et.al. : The complex network analysis of power grid: a case study of the West Bengalpower network. In: Intelligent Computing, Networking and Informatics, pp. 17–29. SpringerIndia (2014)

32. Das, H., et.al. : Grid computing-based performance analysis of power system: a graphtheoretic approach. Intell. Comput. Commun. Dev. 259–266 (2015)

470 H. Das et al.

33. Bache, K., Lichman, M.: UCI machine learning repository. University of California, Irvine.CA. School of Information and Computer Science. (2013) [http://archive.ics.uci.edu/ml]

34. Alcalá-Fdez, J., et al.: KEEL data-mining software tool: data set repository. integration ofalgorithms and experimental analysis framework. J. Multiple-Valued Logic Soft Comput. 17(2–3), 255–287 (2011)

A Novel PSO Based Back Propagation … 471