C O M P U T A T I O N A L I N T E L L I G E N G E

P A R A D I G M S ТНЕОRV AND

A P P L I C A T I O N S USING M A I L A B

S. S U M Л Т H I S U RE К Н Л Р.

Ctf$) CRC Press \ V * J Taylor & Francis Grt Taylor &. Francis Group

Boca Raton London New York

CRC Press is an imprint of the Taylor Si Francis Group, an informa business

A CHAPMAN & HALL BOOK

Contents

Preface xvii

1 Computational Intelligence 1 1.1 Introduction 1 1.2 Primary Classes of Problems for CI Techniques . . . . 5

1.2.1 Optimization 5 1.2.2 NP-Complete Problems 6

1.3 Neural Networks 7 1.3.1 Feed Forward Neural Networks 8

1.4 Fuzzy Systems 9 1.4.1 Fuzzy Sets 9 1.4.2 Fuzzy Controllers 11

1.5 Evolutionary Computing 12 1.5.1 Genetic Algorithms 13 1.5.2 Genetic Programming 14 1.5.3 Evolutionary Programming 15 1.5.4 Evolutionary Strategies 15

1.6 Swarm Intelligence 16 1.7 Other Paradigms 17

1.7.1 Granular Computing 18 1.7.2 Chaos Theory 20 1.7.3 Artificial Immune Systems 21

1.8 Hybrid Approaches 22 1.9 Relationship with Other Paradigms 23 1.10 Challenges To Computational Intelligence 25 Summary 26 Review Questions 27

2 Artificial Neural Networks with MATLAB 29 2.1 Introduction 29 2.2 A Brief History of Neural Networks 29 2.3 Artificial Neural Networks 32

2.3.1 Comparison of Neural Network to the Brain . 33 2.3.2 Artificial Neurons 33

v

2.3.3 Implementation of Artificial Neuron Electroni­cally 35

2.3.4 Operations of Artificial Neural Network . . . . 37 2.3.5 Training an Artificial Neural Network 40 2.3.6 Comparison between Neural Networks, Tradi­

tional Computing, and Expert Systems . . . . 44 2.4 Neural Network Components 46

2.4.1 Teaching an Artificial Neural Network 52 2.4.2 Learning Rates 54 2.4.3 Learning Laws 55 2.4.4 MATLAB Implementation of Learning Rules . 61

Summary 67 Review Questions 67

Artificial Neural Networks - Architectures and Algo­rithms 69 3.1 Introduction 69 3.2 Layered Architectures 71

3.2.1 Single-Layer Networks 71 3.2.2 Multilayer Networks 73

3.3 Prediction Networks 75 3.3.1 The Perceptron 75 3.3.2 MATLAB Implementation of a Perceptron Net­

work 79 3.3.3 Feedforward Back-Propagation Network . . . . 83 3.3.4 Implementation of BPN Using MATLAB . . . 91 3.3.5 Delta Bar Delta Network 95 3.3.6 Extended Delta Bar Delta 97 3.3.7 Directed Random Search Network 100 3.3.8 Functional Link Artificial Neural Network

(FLANN) or Higher-Order Neural Network . . 102 Summary 106 Review Questions 106

Classification and Association Neural Networks 109 4.1 Introduction 109 4.2 Neural Networks Based on Classification 109

4.2.1 Learning Vector Quantization 110 4.2.2 Implementation of LVQ in MATLAB 115 4.2.3 Counter-Propagation Network 116 4.2.4 Probabilistic Neural Network 123 4.2.5 Implementation of the Probabilistic Neural Net

Using MATLAB 128

Vll

4.3 Data Association Networks 129 4.3.1 Hopfield Network 130 4.3.2 Implementation of Hopfield Network in MAT­

LAB 133 4.3.3 Boltzmann Machine 134 4.3.4 Hamming Network 137 4.3.5 Bi-Directional Associative Memory 141

4.4 Data Conceptualization Networks 146 4.4.1 Adaptive Resonance Network 146 4.4.2 Implementation of ART Algorithm in MATLAB 151 4.4.3 Self-Organizing Map 153

4.5 Applications Areas of ANN 158 4.5.1 Language Processing 159 4.5.2 Character Recognition 160 4.5.3 Data Compression 160 4.5.4 Pattern Recognition 160 4.5.5 Signal Processing 161 4.5.6 Financial 161

Summary 162 Review Questions 162

5 MATLAB Programs to Implement Neural Networks 165 5.1 Illustration 1: Coin Detection Using Euclidean Distance

(Hamming Net) 165 5.2 Illustration 2: Learning Vector Quantization - Clustering

Data Drawn from Different Regions 171 5.3 Illustration 3: Character Recognition Using Kohonen

Som Network 174 5.4 Illustration 4: The Hopfield Network as an Associative

Memory 182 5.5 Illustration 5: Generalized Delta Learning Rule and

Back-Propagation of Errors for a Multilayer Network . 187 5.6 Illustration 6: Classification of Heart Disease Using

Learning Vector Quantization 189 5.7 Illustration 7: Neural Network Using MATLAB Simulink 198 Summary 200 Review Questions 200

6 MATLAB-Based Fuzzy Systems 203 6.1 Introduction 203 6.2 Imprecision and Uncertainty 205 6.3 Crisp and Fuzzy Logic 205 6.4 Fuzzy Sets 207

viii

6.5 Universe 209 6.6 Membership Functions 210

6.6.1 Types of Membership Functions 212 6.6.2 Membership Functions in the MATLAB Fuzzy

Logic Toolbox 214 6.6.3 MATLAB Code to Simulate Membership Func­

tions 217 6.6.4 Translation of Parameters between Membership

Functions Using MATLAB 223 6.7 Singletons 224 6.8 Linguistic Variables 225 6.9 Operations on Fuzzy Sets 225

6.9.1 Fuzzy Complements 227 6.9.2 Fuzzy Intersections: t-norms 230 6.9.3 Fuzzy Unions: t-conorms 233 6.9.4 Combinations of Operations 235 6.9.5 MATLAB Codes for Implementation of Fuzzy

Operations 236 6.9.6 Aggregation Operations 239

6.10 Fuzzy Arithmetic 242 6.10.1 Arithmetic Operations on Intervals 243 6.10.2 Arithmetic Operations on Fuzzy Numbers . . 244 6.10.3 Fuzzy Arithmetic Using MATLAB Fuzzy Logic

Toolbox 246 6.11 Fuzzy Relations 247 6.12 Fuzzy Composition 251

6.12.1 MATLAB Code to Implement Fuzzy Composi­tion 254

Summary 260 Review Questions 260

7 Fuzzy Inference and Expert Systems 261 7.1 Introduction 261 7.2 Fuzzy Rules 261

7.2.1 Generation of Fuzzy Rules 262 7.2.2 Disintegration of Rules 262 7.2.3 Aggregation of Rules 263

7.3 Fuzzy Expert System Model 264 7.3.1 Fuzzification 264 7.3.2 Fuzzy Rule Base and Fuzzy IF-THEN Rules . 265 7.3.3 Fuzzy Inference Machine 266 7.3.4 Denazification 267

IX

7.3.5 Implementation of Defuzzification using MAT-LAB Fuzzy Logic Toolbox 273

7.4 Fuzzy Inference Methods 276 7.4.1 Mamdani's Fuzzy Inference Method 278 7.4.2 Takagi-Sugeno Fuzzy Inference Method . . . . 281 7.4.3 Tsukamoto Fuzzy Inference Method 282

7.5 Fuzzy Inference Systems in MATLAB 283 7.5.1 Mamdani-Type Fuzzy Inference 286 7.5.2 Sugeno-Type Fuzzy Inference 292 7.5.3 Conversion of Mamdani to Sugeno System . . 295

7.6 Fuzzy Automata and Languages 297 7.7 Fuzzy Control 298

7.7.1 Fuzzy Controllers 300 7.7.2 A Fuzzy Controller in MATLAB 302

Summary 305 Review Questions 305

MATLAB Illustrations on Fuzzy Systems 307 8.1 Illustration 1: Application of Fuzzy Controller Using

MATLAB — Fuzzy Washing Machine 307 8.2 Illustration 2 - Fuzzy Control System for a Tanker Ship 317 8.3 Illustration 3 - Approximation of Any Function Using

Fuzzy Logic 336 8.4 Illustration 4 - Building Fuzzy Simulink Models . . . . 343 Summary 348 Review Questions 348

Neuro-Fuzzy Modeling Using MATLAB 351 9.1 Introduction 351 9.2 Cooperative and Concurrent Neuro-Fuzzy Systems . . 352 9.3 Fused Neuro-Fuzzy Systems 352

9.3.1 Fuzzy Adaptive Learning Control Network (FALCON) 353

9.3.2 Adaptive Neuro-Fuzzy Inference System (AN-FIS) 353

9.3.3 Generalized Approximate Reasoning-Based In­telligent Control (GARIC) 355

9.3.4 Neuro-Fuzzy Control (NEFCON) 356 9.3.5 Fuzzy Inference and Neural Network in Fuzzy

Inference Software (FINEST) 360 9.3.6 Fuzzy Net (FUN) 362 9.3.7 Evolving Fuzzy Neural Network (EFuNN) . . . 363

X

9.3.8 Self-Constructing Neural Fuzzy Inference Net­work (SONFIN) 364

9.3.9 Evolutionary Design of Neuro-Fuzzy Systems . 364 9.4 Hybrid Neuro-Fuzzy Model — ANFIS 367

9.4.1 Architecture of Adaptive Neuro-Fuzzy Inference System 367

9.4.2 Hybrid Learning Algorithm 370 9.5 Classification and Regression Trees 372

9.5.1 CART — Introduction 372 9.5.2 Node Splitting Criteria 373 9.5.3 Classification Trees 374 9.5.4 Regression Trees 375 9.5.5 Computational Issues of CART 375 9.5.6 Computational Steps 376 9.5.7 Accuracy Estimation in CART 379 9.5.8 Advantages of Classification and Regression

Trees 380 9.6 Data Clustering Algorithms 382

9.6.1 System Identification Using Fuzzy Clustering . 382 9.6.2 Hard C-Means Clustering 383 9.6.3 Fuzzy C-Means (FCM) Clustering 386 9.6.4 Subtractive Clustering 388 9.6.5 Experiments 389

Summary 393 Review Questions 393

10 Neuro-Fuzzy Modeling Using MATLAB 395 10.1 Illustration 1 - Fuzzy Art Map 395 10.2 Illustration 2: Fuzzy C-Means Clustering — Compara­

tive Case Study 402 10.3 Illustration 3 - Kmeans Clustering 403 10.4 Illustration 4 - Neuro-Fuzzy System Using Simulink . 411 10.5 Illustration 5 - Neuro-Fuzzy System Using Takagi-

Sugeno and ANFIS GUI of MATLAB 414 Summary 417 Review Questions 418

11 Evolutionary Computation Paradigms 419 11.1 Introduction 419 11.2 Evolutionary Computation 420 11.3 Brief History of Evolutionary Computation 422 11.4 Biological and Artificial Evolution 423

11.4.1 Expressions Used in Evolutionary Computation 423

XI

11.4.2 Biological Evolution Inspired by Nature . . . . 423 11.4.3 Evolutionary Biology 425

11.5 Flow Diagram of a Typical Evolutionary Algorithm . 428 11.6 Models of Evolutionary Computation 430

11.6.1 Genetic Algorithms (GA) 430 11.6.2 Genetic Programming (GP) 431 11.6.3 Evolutionary Programming (EP) 435 11.6.4 Evolutionary Strategies (ESs) 436

11.7 Evolutionary Algorithms 439 11.7.1 Evolutionary Algorithms Parameters 442 11.7.2 Solution Representation 442 11.7.3 Fitness Function 443 11.7.4 Initialization of Population Size 444 11.7.5 Selection Mechanisms 444 11.7.6 Crossover Technique 451 11.7.7 Mutation Operator 455 11.7.8 Reproduction Operator 458

11.8 Evolutionary Programming 459 11.8.1 History 459 11.8.2 Procedure of Evolutionary Programming . . . 460 11.8.3 EPs and GAs 461 11.8.4 Algorithm of EP 461 11.8.5 Flowchart 461

11.9 Evolutionary Strategies 478 11.9.1 Solution Representation 480 11.9.2 Mutation 480 11.9.3 Recombination 481 11.9.4 Population Assessment 482 11.9.5 Convergence Criteria 483 11.9.6 Computational Considerations 483 11.9.7 Algorithm Performance 484

11.10 Advantages and Disadvantages of Evolutionary Compu­tation 485

Summary 487 Review Questions 488

12 Evolutionary Algorithms Implemented Using MATLAB 491 12.1 Illustration 1: Differential Evolution Optimizer . . . . 491 12.2 Illustration 2: Design of a Proportional-Derivative Con­

troller Using Evolutionary Algorithm for Tanker Ship Heading Regulation 502

X l l

12.3 Illustration 3: Maximizing the Given One-Dimensional Function with the Boundaries Using Evolutionary Algo­rithm 523

12.4 Illustration 4: Multiobjective Optimization Using Evo­lutionary Algorithm (MOEA) 526

12.5 Illustration 5: Evolutionary Strategy for Nonlinear Func­tion Minimization 541

Summary 545 Review Questions 545

13 MATLAB-Based Genetic Algorithm 547 13.1 Introduction 547 13.2 Encoding and Optimization Problems 548 13.3 Historical Overview of Genetic Algorithm 549 13.4 Genetic Algorithm Description 550 13.5 Role of Genetic Algorithms 552 13.6 Solution Representation of Genetic Algorithms . . . . 553 13.7 Parameters of Genetic Algorithm 554

13.7.1 Standard Parameter Settings 554 13.8 Schema Theorem and Theoretical Background . . . . 555

13.8.1 Building Block Hypothesis 556 13.8.2 The Dynamics of a Schema 557 13.8.3 Illustrations Based on Schema Theorem . . . . 559

13.9 Crossover Operators and Schemata 563 13.9.1 1-Point Crossover 563 13.9.2 2-Point Crossover 563 13.9.3 Linkage and Defining Length 564 13.9.4 Linkage and Inversion 565

13.10 Genotype and Fitness 566 13.11 Advanced Operators in GA 567

13.11.1 Inversion and Reordering 567 13.11.2 Epistasis 567 13.11.3 Deception 568 13.11.4 Mutation and Naive Evolution 569 13.11.5 Niche and Speciation 569 13.11.6 Restricted Mating 569 13.11.7 Diploidy and Dominance 569

13.12 GA Versus Traditional Search and Optimization Meth­ods 570 13.12.1 Neural Nets 570 13.12.2 Random Search 571 13.12.3 Gradient Methods 571 13.12.4 Iterated Search 571

Xlll

13.12.5 Simulated Annealing 572 13.13 Benefits of GA 572 13.14 MATLAB Programs on Genetic Algorithm 573

13.14.1 Illustration 1: Maximizing the Given One-Dimensional Function within Given Boundaries 573

13.14.2 Illustration 2: Solving Economic Dispatch Prob­lem Using Genetic Algorithm 576

13.14.3 Illustration 3: Traveling Salesman Problem . . 580 Summary 587 Review Questions 588

14 Genetic Programming 591 14.1 Introduction 591 14.2 Growth of Genetic Programming 594 14.3 The Lisp Programming Language 595 14.4 Functionality of Genetic Programming 596

14.4.1 Generation of an Individual and Population . . 596 14.4.2 Creating a Random Population 597 14.4.3 Fitness Test 599 14.4.4 Functions and Terminals 599 14.4.5 The Genetic Operations 599 14.4.6 Selection Functions 600 14.4.7 MATLAB Routine for Selection 601 14.4.8 Crossover Operation 604 14.4.9 MATLAB Routine for Crossover 607 14.4.10 Mutation Operation 608 14.4.11 MATLAB Routine for Mutation 608

14.5 Genetic Programming in Machine Learning 610 14.6 Elementary Steps of Genetic Programming 612

14.6.1 The Terminal Set 613 14.6.2 The Function Set 613 14.6.3 The Fitness Function 613 14.6.4 The Algorithm Control Parameters 614 14.6.5 The Termination Criterion 614

14.7 Flowchart of Genetic Programming 615 14.8 Benefits of Genetic Programming 618 14.9 MATLAB Examples Using Genetic Programming . . . 618

14.9.1 Illustration 1: Static Function Identification . . 618 14.9.2 Illustration 2: Dynamical Input-Output Model

Identification 629 14.9.3 Illustration 3 - Symbolic Regression Problem

Using Genetic Programming Toolbox 632 Summary 645

xiv

Review Questions 646

15 MATLAB-Based Swarm Intelligence 649 15.1 Introduction to Swarms 649 15.2 Biological Background 650 15.3 Swarm Robots 652 15.4 Stability of Swarms 653 15.5 Swarm Intelligence 654

15.5.1 Properties of a Swarm Intelligence System . . 655 15.6 Particle Swarm Optimization (PSO) 656

15.6.1 Mathematical Model of PSO 657 15.6.2 Algorithm of PSO 658 15.6.3 Parameters and Tuning of Parameters in PSO 658 15.6.4 Neighborhood Topologies 659

15.7 Extended Models of PSO 661 15.7.1 PSO-I 661 15.7.2 PSO-II 661 15.7.3 PSO-III 662 15.7.4 PSO-IV 663

15.8 Ant Colony Optimization 663 15.8.1 Mathematical Model of ACO 664 15.8.2 Ant Colony Optimization Algorithm 664

15.9 Studies and Applications of Swarm Intelligence . . . . 665 15.9.1 Ant-based Routing 665 15.9.2 Clustering Behavior of Ants 665 15.9.3 Graph Coloring 666 15.9.4 Machine Scheduling 666 15.9.5 Quadratic Knapsack Problem 667 15.9.6 Traveling Salesman Problem 668

15.10 MATLAB Examples of Swarm Intelligence 668 15.10.1 Illustration 1: Simulation of the Movement of

Swarm to Minimize the Objective Function . 669 15.10.2 Illustration 2: Behavior of Particle Swarm Op­

timization 671 15.10.3 Illustration 3: Ant Colony Optimization to De­

termine the Shortest Path 684 15.10.4 Illustration 4: Ant Algorithm for the Quadratic

Assignment Problem (QAP) 687 Summary 691 Review Questions 691

A Glossary of Terms 693

XV

В List of Abbreviations 737

С MATLAB Toolboxes Based on CI 741 C.l Genetic Algorithm Toolbox for MATLAB 741 C.2 Fuzzy Logic Toolbox 2.2.7 744 C.3 Neural Network Toolbox 6.0 746 C.4 Genetic Algorithm and Direct Search Toolbox 2.2 . . 752 C.5 GPLAB - A Genetic Programming Toolbox for MAT­

LAB 754

D Emerging Software Packages 763 D.l BUGS 763 D.2 ComputerAnts 763 D.3 DGenesis 764 D.4 Ease 764 D.5 Evolution Machine 764 D.6 Evolutionary Objects 765 D.7 GAC, GAL 765 D.8 GAGA 766 D.9 GAGS 766 D.10 GAlib 767 D.l l GALOPPS 767 D.12 GAMusic 768 D.13 GANNET 769 D.14 GA Workbench 769 D.15 GECO 769 D.16 Genesis 770 D.17 GENEsYs 770 D.18 GenET 771 D.19 Genie 771 D.20 Genitor 771 D.21 GENlib 772 D.22 GENOCOP 772 D.23 GPEIST 772 D.24 Imogene 773 D.25 JAG 773 D.26 LibGA 774 D.27 mGA 774 D.28 PGA 774 D.29 PGAPack 775 D.30 SGA-C, SGA-Cube 776 D.31 Splicer 776 D.32 Trans-Dimensional Learning 777

XVI

D.33 WOLF 777 D.34 XGenetic 778 D.35 XFUZZY: A Simulation Environment for Fuzzy Logic

Control Systems 778 D.36 ART*Enterprise 779 D.37 COMDALE/C, COMDALE/X, and ProcessVision . . 779

E Research Projects 781 E.l An Evolutionary Algorithm for Global Optimization

Based on Level-Set Evolution and Latin Squares . . . 781 E.2 Evolving Problems to Learn about Particle Swarm Op­

timizers and Other Search Algorithms 782 E.3 Analog Genetic Encoding for the Evolution of Circuits

and Networks 782 E.4 A Runtime Analysis of Evolutionary Algorithms for Con­

strained Optimization Problems 783 E.5 Solving the Register Allocation Problem for Embedded

Systems Using a Hybrid Evolutionary Algorithm . . . 783 E.6 Semantic Understanding of General Linguistic Items by

Means of Fuzzy Set Theory 784 E.7 Fuzzy Evaluation of Heart Rate Signals for Mental Stress

Assessment 785 E.8 An Ant Colony Optimization Approach to the Proba­

bilistic Traveling Salesman Problem 785 E.9 Neural Processing of Symbolic Data 786

References 787

Index 821