2007NEURAL NETWORKS, E. Tulunay, Y. Tulunay 1 COST 724 WHY NEURAL NETWORKS? E. Tulunay 1, Y. Tulunay 2 ODTU / METU 1.Dept. of Electrical and Electronics

2007 NEURAL NETWORKS, E. Tulunay, Y. Tulunay

1

COST 724

WHY NEURAL NETWORKS?

E. Tulunay1, Y. Tulunay2

ODTU / METU1. Dept. of Electrical and Electronics Eng.

2. Dept. of Aerospace Eng.06531, Ankara, TURKEY


2

• OUTLINE– Neural Networks– METU-NN

• Training, validation in training, validation in test

– NN Applications– Space Weather-borne processes– A case study example:

• Forecasting Total Electron Content Maps by Neural Network Technique

– Conclusions– References


3

• Neural Networks (NN)NN: system of interconnected computational elementsoperating in parallel, arranged in patterns similar to biological NNs andmodeled after the human brain [Rumelhart et al., 1986].

Since 1990’s, interest in NNs has increased mainly because of

the developments in very large scale integrated circuit technology, optical devices and

new learning paradigms which make rapid and inexpensive implementation possible.


4

Neuron : information processing unit consisting of connecting links, adder and activation function.

Adder : for summing bias and the input signals weighted in neuron’s connecting links.

It follows an activation function for limiting the amplitude of the neuron’s output [Haykin, 1999].

ANN is a system of inter-connected computational elements, the neurons, operating in parallel, arranged in patterns similar to biological neural networks and modeled after the human brain [Tulunay, 1991].

[Y. Tulunay et al., 2004-a]


5

Biological to Artificial Neurons

Tulunay, 2004

METU-NN

The multilayer perceptron - feed forward NNs - neurons arranged in layers.


6

Training, validation in training, validation in test

In training, weights - initially set to arbitrary values. The inputs - applied and then NN produces an output. The difference between the NN output and the desired output : the error.

During training, weights - adapted to minimize the error by using various algorithms.

Memorization is avoided.

After the original software was developed in order to implement this algorithm, the training process - completed and

these trained NNs - ready to perform the forecasting function [Altinay et al., 1997].


7

NN applications in various fields including:adaptive pattern recognition, adaptive signal processing,adaptive dynamic modeling,adaptive control,optimization,expert systems andEarth System Science applications.

Some other specific applications include control of robot arm,diagnosis and numeric to symbolic conversion [Tulunay, 1991].


8

NN based models are generic in that they have been applied to variety of several processes.

The only requirement for NN application is the availability of representative data

Relative Significances of Inputs


9

Unpredictable variability of the ionospheric parameters due to Space Weather-borne processes

significant effects on both space and Earth based systems such as communication, radar, navigation etc.

Space Weather has significant effects on Earth climate, weather and on biological systems including human health.

Therefore, forecasting Near-Earth Space parameters, especially during the disturbed Space Weather conditions, is crucial.


10

It is most desirable to drive mathematical forecasting and mapping models based on physics.

However, this is very complex and prohibitively difficult task since Space Weather processes are non-linear and time-varying.

It has been demonstrated that the data driven approaches such as the use of NN based methods are promising in modeling of ionospheric processes.


11

In particular, the NN models are promising in forecasting applications under disturbed conditions,

eg. [Williscroft and Poole., 1996; Tulunay et al., 1997; Altinay et al., 1997; Cander et al., 1998; Wintoft and Cander, 1999; Francis et al, 2000; Y. Tulunay et al., 2001; Vernon and Cander, 2002; E. Tulunay et al., 2004-a; Y. Tulunay et al., 2004-a; Y. Tulunay et al., 2004-b; Radicella and Tulunay, 2004; Stamper et al., 2004; McKinnell and Poole, 2004; E. Tulunay et al., 2006-a].

Space weather centers provide forecasts of solar and geophysical parameters. As an example, the Lund Space Weather Center uses artificial intelligence (AI) to forecast Kp parameters [Boberg et al., 2000].


12

Since the 1990’s a small group at the METU in Ankara has been working on data-driven generic models of Earth System processes.

The METU-NN model has been applied to Near-Earth Space processes for a variety of tasks including

the forecasting and mapping of the foF2 and TEC [Tulunay et al., 2000; Tulunay et al., 2001, E. Tulunay et al., 2006-a].

Some recent applications include forecasting of solar flux bursts [Y. Tulunay et al., 2005-a]

and Schumann resonances [E. Tulunay et al., 2006-b].


13

METU Models on NES processes:• 1. Temporal and Spatial Forecasting of the ionospheric critical frequencies,

foF2• [Tulunay et al., 2000]• 2. Forecasting Total Electron Content Maps by Neural Network Technique• [Tulunay et al., 2006-a; Ciraolo, 2004]• 3. Forecasting GPS TEC Using the Neural Network Technique “A Further

Demonstration” [E. Tulunay et al., 2004]• 4. An Attempt to Model the Influence of the Trough on HF Communication

by Using Neural Network [Tulunay et al., 2001]• 5. Forecasting Magnetopause Crossing Locations by Using Neural

Networks• [Tulunay et al., 2005-b]• 6. The ELF Characterization of the Earth-Ionosphere Cavity: Forecasting

the Schumann Resonances (SR) [Tulunay et al., 2006-b]• 7. Timed EUV Flux Data and the METU-NN Model• 8. Neural Network Modeling in Forecasting the Near Earth Space

Parameters: Forecasting of the Solar Radio Fluxes [Tulunay et al., 2005-a]


14

A Case Study Example:Forecasting Total Electron Content Maps by Neural

Network Technique

[Tulunay et al., 2006-a; Ciraolo, 2004]

In order to understand more about the complex response of the magnetosphere and ionosphere to extreme solar events, the METU-NN model was used in connection with the series of space weather events in November 2003. TEC values of the ionosphere were forecast during these space weather events.

In order to facilitate an easier interpretation of the forecast TEC values, maps of TEC are produced by using the Bezier surface-fitting technique


15

Observed GPS TEC data [Ciraolo, 2004] :- Ten minute vertical TEC databetween 1 Nov. 2003 and 11 Dec. 2003- 104 grid locations centered over Italybetween latitudes of (35.5º N; 47.5º N) and longitudes of (5.5º E; 19.5º E)


16

METU-NN technique

- to forecast TEC grid values

- surfaces in mapping the forecast grid values


17

- Figure 1. Observed (dotted) and 1 hour ahead forecast (solid) TEC during 16 Nov. 2003 03:10 UT - 29 Nov. 2003 24:00 UT for single grid point

(13.5˚ E; 41.5˚ N)

Results


18

- Figure 2. Scatter Diagram (dots) with best-fit line (solid) for the 1 hour ahead Forecast mapping and Observed TEC values for single grid point (13.5˚ E; 41.5˚ N) for 20 November 2003


19

METU-NN technique

- to forecast TEC grid values

- surfaces in mapping the forecast grid values


20

- Video 1. Observed and 1 hour ahead forecast TEC Maps by METU-NN during the afternoon of 20 Nov. 2003


21

- Figure 3. Absolute Error Map of observed and 1 hour ahead forecast TEC during 16-29 Nov. 2003


22

Table 1. Error Table for 1 h in advance forecasts by METU-NN for the validation time period (16-29 Nov. 2003)

1 TECu = 1016 el./m2

The forecast mapping error values - within operational tolerance[Radicella, 2004].

Location11.5˚E38.5˚N

13.5˚E41.5˚N

15.5˚E44.5˚N

OverallTEC Map

Absolute Error (TECu) 1.58 1.49 1.52 1.65

Normalized Error (%) 14.29 15.22 16.30 15.63

Cross Correlation Coeff. (x10-2) 97.8 97.5 97.0 97.0


23

Conclusions

- The system reached correct operating point during training

- METU-NN learned the inherent nonlinearities of the process

- Representable data in inputs +

Proper construction of METUNN =

complex nonlinear processes are modeled

- LAN access for the METU models on NES processes except TEC Mapping are available. Web access will also be available when COST 724 web is operative [Ozkok (supervisor: E. Tulunay; co-supervisor: Y. Tulunay), 2005].


24

• References

• Altinay O., E.Tulunay, and Y. Tulunay (1997), Forecasting of ionospheric critical frequency using neural networks, Geophysical Research Letter, 24(12), 1467-1470, and COST251 TD(96)016.

• Boberg F., P. Wintoft, and H. Lundstedt (2000), Real time Kp predictions from solar wind data using neural networks, Phys. Chem. Earth, Pt C, 25(4), 275-280.

• Cander Lj.R., M.M. Milosavlijevic, S.S. Stankovic, and S. Tomasevic (1998), Ionospheric Forecasting Technique by Artificial Neural Network, Electronics Letters, 34(16), Online No: 19981113, 1573-1574.

• Ciraolo G. (2004), Private communication.

• T. Dudok de Wit, J. Lilensten, J. Aboudarham, P.-O. Amblard, and M. Kretzschmar (2005), Retrieving the solar EUV spectrum from a reduced set of spectral lines, Annales Geophysicae, 23, 3055–3069.

• Francis, N.M., P.S. Cannon, A.G. Brown, and D.S. Broomhead (2000), Nonlinear prediction of the ionospheric parameter foF2 on hourly, daily, and monthly timescales, J. Geophys. Res., 105(A6), 12839-12849.

• Hagan M.T., and M.B. Menhaj (1994), Training Feedforward Networks with the Marquard Algorithm, IEEE Transactions on Neural Networks, 5(6), 989-993

• Haykin S. (1999), Neural Networks: A Comprehensive Foundation, 2nd ed., pp. 2, 10, 21-22, 83-84, 169, 215, Prentice-Hall, Inc., New Jersey, USA.

• Jackson, J.H. (1958), On localization, in Selected writings, 2, Basic Books, New York, 1958. Original work 1869.

• Kalnay et al. (1996), The NCEP/NCAR 40-year reanalysis project, Bull. Amer. Meteor. Soc., 437-470

• Kaya, A. (1998), The effects of interplanetary magnetic field, IMF on the Earth space radio systems M.S. Thesis, supervisor: Y. Tulunay, Aeronaut. Eng. Dep., Middle East Tech. Univ., Ankara, Turkey, Dec., 1998.

• LASCO web site, (2002), Large Angle and Spectrometric Coronagraph Experiment (LASCO) and Solar and Heliospheric Observatory (SOHO (ESA & NASA)) Extreme ultraviolet Imaging Telescope (EIT) instruments, Coronal Mass Ejections, the SOHO/LASCO data produced by a consortium of the Naval Research Laboratory (USA), Max-Planck-Institut fuer Aeronomie (Germany), Laboratoire d'Astronomie (France), and the University of Birmingham (UK)., http://lasco-www.nrl.navy.mil/.

• Luira A.R. (1966), Higher cortical functions in man, Basic Books, New York.

• McKinnell L.A., and A.W.V. Poole (2004), Predicting the ionospheric F layer using neural networks, J. Geophys. Res., 109(A8), A08308, doi:10.1029/2004JA010445.

• Ozkok Y. (2005), Web Based Ionospheric Forecasting Using Neural Network and Neurofuzzy Models, (Supervisor: E. Tulunay, Co-supervisor: Y. Tulunay), MS Thesis, EEE Dept., Middle East Technical University, Ankara, Turkey, July 2005


25

• Radicella S.M., and E. Tulunay (2004), Space plasma effects on Earth-space and satellite-to-satellite communications: Working Group 4 overview, Annals of Geophysics, 47(2/3), 1279-1283.

• Rumelhart D.E., J.L. McClelland and the PDP Research Group (1986), Parallel distributed processing, explorations in the microstructure of cognition, Vol. 1: Foundations, A Bradford Book, The MIT Press, Cambridge, Mass, and London, 1986, 1988.

• Stamper R., A. Belehaki, D. Buresova, Lj.R. Cander, I. Kutiev, M. Pietrella, I. Stanislawska, S. Stankov, I. Tsagouri, Y.K. Tulunay, and B. Zolesi (2004), Nowcasting, forecasting and warning for ionospheric propagation: tools and methods, Annals of Geophysics, 47(2/3), 957-983.

• Tulunay E. (1991), Introduction to Neural Networks and their Application to Process Control, in Neural Networks Advances and Applications, edited by E. Gelenbe, 241-273, Elsevier Science Publishers B.V., North-Holland.

• Tulunay E., C. Ozkaptan, Y. Tulunay (2000), Temporal and Spatial Forecasting of the foF2 Values up to Twenty four Hour in Advance, Phys. Chem. Earth, 25(4), 281-285.

• Tulunay E., E.T. Senalp, Lj.R. Cander, Y.K. Tulunay, A.H. Bilge, E. Mizrahi, S.S. Kouris, N. Jakowski (2004-a), Development of algorithms and software for forecasting, nowcasting and variability of TEC, Annals of Geophysics, 47(2/3), 1201-1214.

• Tulunay E., Y. Tulunay, E.T. Senalp, Lj.R. Cander (2004-b), Forecasting GPS TEC Using the Neural Network Technique “A Further Demonstration”, Bulgarian Geophysical Journal, 30(1-4), 53-61.

• Tulunay E., E.T.Senalp, S.M.Radicella, Y.Tulunay (2006-a), Forecasting Total Electron Content Maps by Neural Network Technique, Radio Science, Vol. 41, No. 4, RS4016, American Geophysical Union, Washington, USA.

• Tulunay E., Y. Tulunay, E. Altuntas, E.T Senalp, Y. Bahadirlar (2006-b), Neural Network Forecasting Of Schumann Resonances In The Near Earth Space, Third European Space Weather Week (ESWW3), 13-17 November 2006, Royal Library of Belgium, Brussels, Belgium.

• Tulunay, Y., and J.M. Grebowsky (1978), The noon and midnight mid-latitude trough as seen by Ariel 4, J. Atmos. Terr.Phys., 40, 845-855.

• Tulunay, Y., A. Kaya, and G. Oke (1997), Further possible effect of the IMF turnings on the Slough critical frequencies and the signature of the electron density trough on the COST251 area, paper presented at Fifth Management Committee Meeting of COST251 and Joint COST/IRI Workshop, Inst. of Atmos. Phys., Kuehlungsborn, Germany, 26-31 May 1997.

• Tulunay Y., E. Tulunay, and E.T. Senalp (2001), An Attempt to Model the Influence of the Trough on HF Communication by Using Neural Network, Radio Science, 36(5), 1027-1041, September - October 2001.

• Tulunay Y., E. Tulunay, and E.T. Senalp (2004-a), The Neural Network Technique-1: A General Exposition, Adv. Space Res., 33(6), 983-987.


26

• Tulunay Y., E. Tulunay, and E.T. Senalp (2004-b), The Neural Network Technique-2: An Ionospheric Example Illustrating its Application, Adv. Space Res., 33(6), 988-992.

• Tulunay Y., M. Messerotti, E.T. Senalp, E. Tulunay, M. Molinaro, Y.I. Ozkok, T. Yapici, E. Altuntas, N. Cavus (2005-a), Neural Network Modeling in Forecasting the Near Earth Space Parameters: Forecasting of the Solar Radio Fluxes, COST 724: "Developing the scientific basis for monitoring, modeling and predicting Space Weather" Scientific Workshop, Proceedings CD, 10-14 October 2005, Athens, Greece.

• Tulunay Y., D.G. Sibeck, E.T. Senalp, E. Tulunay (2005-b), Forecasting magnetopause crossing locations by using Neural Networks, Adv. Space Res., 36(12) 2378-2383.

• Vernon A., and Lj.R. Cander (2002), Regional GPS receiver networks for monitoring local mid-latitude total electron content, Annals of Geophysics, 45(5), 649-656.

• Wang, J.S., Chen, Y.P. (2006), A Fully Automated Recurrent Neural Network for Unknown Dynamic System Identification and Control, IEEE T. Circuits Syst.- I: Regular Papers, 53-6, pp. 1363-1372

• Williscroft L.A., and A.W.V. Poole (1996), Neural Networks, foF2, sunspot number and magnetic activity, Geophys. Res. Lett., 23(24), 3659-3662.

• Wintoft P, Lj.R. Cander, (1999). Short term prediction of foF2 using timedelay neural networks, Phys. Chem. Earth, Pt C, 24(4), 343-347.

Documents

2007NEURAL NETWORKS, E. Tulunay, Y. Tulunay 1 COST 724 WHY NEURAL NETWORKS? E. Tulunay 1, Y. Tulunay 2 ODTU / METU 1.Dept. of Electrical and Electronics