1

Volodymyr STEPASHKO

Prof., Dr. Sci., Head of Department for Information Technologies of Inductive Modeling

Kyiv, March 2013

Inductive Modelingfrom historical perspective

International Research and Training Centre

for Information Technologies and Systemsof the National Academy of Sciences of

Ukraine

2

Terminology evolution:

heuristic self-organization of models (1970s) inductive method of model building (1980s) inductive learning algorithms for modeling

(1992) inductive modeling (1998)

MGUA: Method of Group Using of Arguments

(direct translation of original name of the method)

GMDH: Group Method of Data Handling(standard international name starting from the very first

translation in USA)

3

T h e o r y o f I n d u c t i v e M o d e l i n g

T h e o r y o f N o i s e - I m m u n i t y M o d e l i n g

G r o u p M e t h o d o f D a t a H a n d l i n g ( G M D H ) P o l y n o m i a l N e u r a l N e t w o r k ( P N N )

D a t a - B a s e d M o d e l i n g o f C o m p l e x S y s t e m s

S t r u c t u r a l I d e n t i f i c a t i o n o f C o m p l e x P l a n t s

G M D H - B a s e d I n f o r m a t i o n T e c h n o l o g y A S T R I D

E c o l o g i c a l P r o c e s s e s

E c o n o m i c a l S y s t e m s

T e c h n o l o g i c a l P l a n t s

Structure of Knowledge in the

Inductive Modeling Area

II nn ff oo rr mm aa tt ii oo nn TT ee cc hh nn oo ll oo gg yy

AA SS TT RR II DD

GG MM DD HH

DD aa tt aa BB aa ss ee

PP rr aa cc tt ii cc aa ll EE xx pp ee rr ii ee nn cc ee

MM oo dd ee ll

PP rr ee dd ii cc tt ii oo nn

GMDH-Based Information Technology A S T R I D

for modeling complex processes from data

5

Attempt to define IM: what is it?IM generally is a process of inductive transition from data to models under uncertainty conditions: limited size of data set: small sample of noisy data limited volume of a priori information: - unknown character and level of noise - inexact composition of relevant arguments (inputs) - unknown structure of relationships in an object

IM is the MGUA/GMDH based technique for model

self-organization IM is a technology for noise-immunity modeling MODELTHEORY DATA

Two opposite (but supplemental) approaches to model building

Deduction

Induction

6

IM destination: what is this for?

There is a wide experience of IM using for the following problems to be solved:

Forecasting of complex processes Structural and parametric identification Data compression (via optimal approximation) Classification and pattern recognition (supervised

learning) Data clustering (unsupervised learning) Machine learning Data Mining Knowledge Discovery

7

IM explanation: main tasksGeneral problem definition

Given: data set of n observations after m input

x1,x2,…xm and one output y

variables

Find: model y=f(x1,x2,…xm ,θ) with minimum variance of prediction

error, θ is unknown vector of model

parameters

GMDH Task: f *= arg minΦ C (f )

C (f ) is a model quality criterion Φ is a set of models, f Φ

Illustration: choose an optimal subset of monomials

out of the member set of Kolmogorov-Gabor polynomial:

8

Two interdependent tasks

Φ – set of model structuresС – criterion of a model qualityStructure of a model:

)(minarg ff QmRf

Q – quality criterion for parameters estimation

),( ff Xfy

Estimation of parameters:

)(minarg* fCff

Discrete optimization task

Continuous optimization task

9

DD AA TT AA (( ss aa mm pp ll ee ,, aa pp rr ii oo rr yy ii nn ff oo rr mm aa tt ii oo nn ))

CC hh oo ii cc ee oo ff aa mm oo dd ee ll cc ll aa ss ss SS tt rr uu cc tt uu rr ee gg ee nn ee rr aa tt ii oo nn

PP aa rr aa mm ee tt ee rr ee ss tt ii mm aa tt ii oo nn

CC rr ii tt ee rr ii oo nn mm ii nn ii mm ii zz aa tt ii oo nn AA dd ee qq uu aa cc yy aa nn aa ll yy ss ii ss FF ii nn ii ss hh ii nn gg tt hh ee pp rr oo cc ee ss ss

Main stages of the modeling process

Main components of a method of inductive modeling

Method of inductive

modeling

Generator of model structures

Method of parameter estimation

Criterion of model selection

Class of models

11

Basic Principles of GMDH as an Inductive Method

1. Generation of variants of the gradually complicated structures of models

2. Successive selection of the best variants using the freedom of choice of decisions

3. External criteria (based on the sample division) for the best model selection

Training subset А

Generation a set of models being complicated f Ф

Model quality evaluation -calculation of criterion С (f )

C min DATA SET

Checking subset В

f *

Data-driven inductive modelling process

12

External Selection Criteria

Given data set: W = (X y), X [n x m], y [n x 1]Division into two subsets A and B :

nnny

yy

X

XXyXW BA

B

AW

B

AWWW

;;;

,,,,)( 1 WBAGyXXX GTGG

TGG

Parameter estimation by LSM for a model y =X :

Regularity criterion: 2

BWAW XXCB

Unbiasedness criterion:

2

ABBB XyAR

W=AB

13

GMDH algorithms

Sorting-out Iterative

1. Combinatorial

(Exhaustive Search) COMBI

2. Multistage(Directed

Search) MULTI

3. Multilayered

Iterative MIA

4. Relaxation

Iterative RIA

General classification of GMDH algorithms

14

Main types of model structure generators

1. COMBI GMDH = COMBInatorial algorithm

(sorting-out type)

All possible combinations are considered:

yν = Xν θν , ν =1,2,3,…,2m

Structural binary vector: d =(d1,d2,…,dm), dj

={0;1}

ExampleLevel 1: yi = αi xi i = 1,2,…,m

Level 2: yk = αkixi +αkj xj , i,j =1,2,…,m , k=1…Сm2

Level 3: Сm3 model structures of 3 arguments etc.

Total number of models: pm = s Сms = 2m

15

Main generator types

2. MULTI GMDH = MULTIstage (combinatorial- selective) algorithm (sorting-out type)

It selects only the most significant models and retrieves the exhaustive search result

In any stage s , the argument subset of a best model of previous stage is supplemented by anyone argument absent in this model by turn:

ysk = (X i

s-1 | xj )θs ,

s,j =1,2,…,m ; i=1,2,…,Fs-1 ; k = 1,2,…,(m – s)Fs-1

Fs is the freedom of choice in a stage s , usually Fs

≤ m

The whole number of generated models ~ m 3

(polynomial rise) vs. 2m (exponential rise) in COMBI

16

3. MIA GMDH = Multilayered Iterative Algorithm (basic, classical, iterative type) Layer 1: yh = φh (xi ,xj ) , h = 1,2,…Cm

2

Layer 2: fk = φk (yi ,yj ) , k = 1,2,…CF2

Layer 3: gk = φk (fi , fj ) , k = 1,2,…CF2 etc.

Typical variants of the partial description φ : (a) yk

r+1=α0 +α1kyir +α2kyj

r , yi0 =xi , yj

0 =xj

(b) ykr+1=α1kyi

r +α2kyjr +α3kyi

ryjr

(c) ykr+1=α1kyi

r +α2kyjr +α3kyi

ryjr

+α4k(yir)2+α5k(yj

r)2

r = 1,2,…..; i,j =1,2…F ; k = 1,2,…CF2

F is the freedom of choice, usually F =m

Main generator types

17

Classical Multilayered Iterative Algorithm

MIA GMDH

1st layer 2nd layer

)),(,( min arg*

*f

fXfyCRf

18

4. RIA GMDH = Relaxational Iterative Algorithm

(non-classical, iterative type)

Any iteration (layer): fk = φk (yi ,xj ) , k = 1,2,…CF2

It considers pairs (yi ,xj) instead of (yi ,yj) in MIA

Typical variants of the partial description φ : (a) yk

r+1= α0+α1kyir+α2kxj , yi

0 =xi

(b) ykr+1= α1kyi

r +α2kxj +α3kyirxj

(c) ykr+1= α1kyi

r +α2kxj +α3kyirxj

+α4k(yir)2+α5k(xj)2

r= 1,2,…; i=1,2,…,F ; j=1,2,…,m ; k=1,2,…,mF

F is the freedom of choice, usually F =m

Main generator types

19

Main concept: Self-organizing evolution of the model of

optimal complexity under uncertainty conditions

Main result: Complexity of the optimum forecasting model depends on the level of uncertainty in the data: the higher it is, the simpler (more robust) there must be the optimum model

Main conclusion: GMDH is the method for construction of

models with minimum variance of forecasting error

Basic Theoretical Results

20

Reduction of optimal complexity s o when σ 2

growsHere true model contains: 3 relevant + 2 redundant arguments

Illustration to GMDH theory

Coordinates:complexity scriterion C (s)

Parameter:noise variance σ 2

21

MIA GMDH as Polynomial Neural Network (PNN)

Illustration for inductive (forward) process of model construction

IM compared to ANN

22

Optimal structure of the tuned GMDH net

Trained GMDH network (after backward tuning)Argument x2 appears to be redundant

23

IM from CI perspectiveGMDH-based algorithms have typical features of Computational Intelligence tools: evolutionary-type computations network-like structures data-driven learning nature-inspired procedures – BICA !

Main advantages of IM algorithms: automatic evolving of model structure and parameters self-organizing net structures (node and layer numbers) fast learning (locally optimized nodes)

Equally, IM may be attributed to Soft Computing means: inductive inference procedures precedence-based reasoning fuzzy GMDH realizations

24

Some real-world applications of IM Modelling of economical processes: - prediction of tax revenues and inflation - system prediction of power indicators Modelling of ecological processes: - activity of microorganisms in soil and green

algae in water under influence of heavy metals - irrigation of trees by processed wastewaters Simulation in medicine: - self-monitoring of diabetes - prediction of drugs effectiveness Integral evaluation of the state of complex

multidimensional systems - economic safety - investment activity - ecological state of water reservoirs Technology of informational support of

operative management decisions

25

Illustration example 1: Comparison of prediction quality of the real inflation process (USA, 1999) using regr. analysis (LSM) and GMDH

0

0,02

0,04

0,06

0,08

0,1

0,12

0,14

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

INF

LA

TIO

N

MONTHS

Data LSM GMDH

Note: learning interval = 15 months (training+checking)

prediction points (3 months) indicated by arrows

Y – inflation rateX1 – personal savings; X4 – personal

consumption;X2 – total unemployed number; X5 – personal

incomes;X3 – interest rates; X6 – GDP

LSM model structure (static model):

Y = a1X1 + a2X2 + a3X3 + a4X4 + a5X5 + a6X6

GMDH model structure (static model):

Y = a1X1 + a3X3 + a6X6

Illustration example 2: Ukraine budget revenues

Y – budget revenuesX1 – cash income of populationX2 – cash disbursements and savings of populationX3 – consumer price index (%)X4 – price index in light industryX5 – GDP indexX6 – total retail turnoverX7 – total employmentX8 – light industry employmentX9 – wage (nominal) X10 – wage index (real, %) X11 – money supply X12 – US$ official course X13 – account payable betveen enterprises

X14 – expenditure of the consolidated budget

Dependence of prediction efficiency for tax revenues on the statistical sample volume is investigated.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11121 2 3 4 5 6 7 8 91011121 2 3 4 5 6 7 8 91011121 2 3 4 5 6 7 8 91011121 2 3 4 5 6 7 8 9

Data NA=14

 

  Training Prediction

1995 1996 1997 1998 1999

 Fig.1. Prediction quality (14 training points)

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11121 2 3 4 5 6 7 8 91011121 2 3 4 5 6 7 8 91011121 2 3 4 5 6 7 8 91011121 2 3 4 5 6 7 8 9

Data NA=18

 

 Training Prediction

1995 1996 1997 1998 1999

 

Fig.2. Prediction quality (18 training points)

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

1112 1 2 3 4 5 6 7 8 9 101112 1 2 3 4 5 6 7 8 9 101112 1 2 3 4 5 6 7 8 9 101112 1 2 3 4 5 6 7 8 9

Data NA=29

 

 Training Prediction

1995 1996 1997 1998 1999

  Fig.3. Prediction quality (29 training points)

30

Main developed IM tools

Information Technology ASTRID (IRTC, Kyiv)www.mgua.irtc.org.ua

KnowledgeMiner (Frank Lemke, Berlin)www.knowledgeminer.net

FAKE GAME (Pavel Kordik et al., Prague)http://ncg.felk.cvut.cz

GMDH_Shell (Oleksiy Koshulko, Kyiv)http://www.gmdhshell.com

31

Main centers of IM research

IRTC ITS NANU, Kyiv, Ukraine NTUU “KPI”, Kyiv, Ukraine KNURE, Kharkiv, Ukraine KnowledgeMiner, Berlin, Germany CTU in Prague, Czech Sichuan University, Chengdu, China

32

IM development prospects

The most promising directions:

1. Theoretical investigations Study of GMDH performance for cases of different noise

properties and model classes, new external criteria, etc.

2. Integration of IM, NN and CI best developments Algorithms with heterogenic and fuzzy neurons,

immune networks, GA and nature-inspired estimators etc.

3. Paralleling Constructing computational schemes oriented to multi-core and cluster computer architectures

33

The most promising directions :

4. Preprocessing Constructing the optimal combination of preprocessing procedures

5. Ensembling Modelling and forecasting based on weighted averaging

of a group of the best models

6. Intellectual interface Knowledge-based interactive modes with strong support

and control of user’s activity, default modes, etc.

7. Case studies New applications to real-world tasks of different nature

34

Main Web sources on GMDH

Basic home page: info, books, articles, reviews, software

http://www.gmdh.net Professional research group site: ITIM

Departmentdevelopments, staff, info, Proceedings of ICIM/IWIMs

http://www.mgua.irtc.org.uaResearch&training group at CTU in Prague: Developments, staff, ICIM/IWIM home pages

http://cig.felk.cvut.czBusiness site: info, software, applications

http://www.knowledgeminer.netOpen-source site: info, parallel algorithms

http://opengmdh.org

35

THANK YOU !

Volodymyr STEPASHKO

Address: Prof. Volodymyr Stepashko, International Centre for ITS of NANU Akademik Glushkov Prospekt 40 Kyiv, MSP, 03680, Ukraine

Phone: +38 (044) 526-30-28 Fax: +38 (044) 526-15-70 E-mail: stepashko@irtc.org.uaWebWeb: : www.mgua.irtc.org.ua