Upload
allan-hampton
View
220
Download
0
Embed Size (px)
Citation preview
EE749Neural Networks
How to design a good performance NN?
Kasin PrakobwaitayakitDepartment of Electrical Engineering
Chiangmai University
Glossary• Pattern: a complete set of data inputs and
outputs that provide a snapshot of the system being modelled
– also called examples, cases
• Feature: an identifying characteristic in the data that the model will ideally capture
• Domain: a set of boundaries that define the range of expected / observed data for a particular model or problem
Backpropagation Modelling Heuristics• Selection of model inputs and outputs
• Defining the model domain
• Data pre-processing
• Selection of training/testing cases
• Data scaling
• Number of hidden layers
• Number of neurons
• Activation function selection
• Initial weight values
• Learning rate and momentum
• Presentation of patterns
• Stopping criteria for training
• Improving model performance
Selection of Model Inputs and Outputs• Start with a set of inputs that are KNOWN
to affect the process – then add other inputs that are suspected of
having a relationship in the process one at a time
• Eliminate input variables that are redundant (high covariance)
• Eliminate data patterns that do not contribute to training (no new information)
• Identify / eliminate data patterns with the same inputs that have different outputs
Defining the Model Domain• ANNs should be confined to a limited
domain– develop separate models for contradictory areas
of the domain
• Build models that predict a single output– link multiple models together, if required
Data Pre-processing• Any time the dynamic range of an input is
over a few orders of magnitude, a logarithmic transformation should be applied
• Transformations may also be useful in “reconditioning” input data when an ANN has trouble converging
• Non-numerical inputs need to be codified
Selection of Training/Testing Cases• Training cases should be representative of
the problem domain
• For good generalization capacity, the training set must be complete
– important variable must be measured
• Data set is randomly divided into training and testing sets
– in a 70:30 ratio, for example
• Rules of thumb– The number of training patterns should be at
least 5 times the number of nodes in the network – The number of training cases should be roughly
= the number of weights*the inverse of the accuracy parameter (e, e=0.9 means 90% accuracy of prediction is required)
Data Scaling• Output variables should be scaled in the 0.1
to 0.9 range – avoids operating in the saturation range of the
sigmoid function
Number of Hidden Layers• Increasing the number of hidden layers
increases both the time and number of patterns (examples) required for training
• For most problems, one hidden layer will suffice
– problems can always be solved with two
• Multiple slabs (Ward Nets) supposedly increase processing power
– each slab (group of neurons) acts as a detector for one or more input features
Number of Neurons• One neuron in the input layer for each input
• One neuron in the output layer for each output
• Proper number of hidden neurons is often determined experimentally
– too few = poor ability to capture features– too many = poor ability to generalize (ANN
simply memorizes training data)
• Various rules of thumb have been reported– 0.75*N– 2*N +1
N = number of inputs
• In general, more weights introduce more local minima to the error surface
• Flat regions in the error surface can mislead gradient-based error minimization methods (backpropagation)
• Start with a small network and then add connections as needed
– avoids convergence problems as the networks get too large
• The optimum ratio of hidden neurons in the first to second hidden layer is 3:1
Activation Function Selection• Sigmoidal (logistic) activation functions are
the most widely used
• Thresholding functions are only useful for categorical outputs
Initial Weight Values• Weights need to be randomized initially
– if all weights are set to the same number, the GDR would never be able to leave the starting point
• The backpropagation algorithm may also have difficulty if the connection weights are prematurely saturated (>0.9)
Learning Rate and Momentum• Learning rate () affects the speed of
convergence. – If it is large (>0.5), the weights will be changed
more drastically, but this may cause the optimum to be overshot
– If it is small (<0.2) the weights will be changed in smaller increments, thus causing the system to converge more slowly with little oscillation
• The best training occurs when the learning rate is as large as possible without leading to oscillation
• The learning rate can be increased as learning progresses, or a momentum term added to improve network learning speed
• The momentum factor () has the ability to dampen out oscillations caused by increasing the learning rate
– a momentum of 0.9 allows higher learning rates
Presenting Patterns to the ANN• Present patterns in a random fashion
• If input patterns can be easily classified, do not train the ANN on all patterns in a class in succession
– the ANN will forget information as it moves from class to class
• Shaping can be used to improve network training
– involves starting off on a very small cohesive data set and then adding more patterns that have greater deviations from variable means as training progresses
Stopping Criteria for Training• Training should be stopped when one of the
following conditions is met– the testing error is sufficiently small– the testing error begins to increase– a set number of iterations has passed
Improving Model Performance• Re-initialize network weights to a new set of
random values– re-train the model
• Adjust learning rate and momentum
• Modify stopping criteria
• Prune network weights
• Use genetic algorithms to adjust network topology
• Add noise to training cases to decrease the chance of memorization
ANN Modelling ApproachASS ES S NEE DS AND
SUITABIL ITY
CO LLEC T AND ANALY ZE DATA
APP LY M O DEL DEV ELO PM ENT
PRO TO CO L
EVALUATE PE RFO RM ANCE
SE LE CT AN D O RG ANIZE DATA
PATTE RNS
SE T ARCHITECTURE
CHARA CTE RISTIC S
EVALUATE M O DEL STABIL ITY
FIN E-TUNE M O DE L
SE LE CT INP UTS AND O UTPUTS
Needs and Suitability Assessment
• What are the modelling needs ?
• Is the ANN technique suitable to meet these needs ?
• Can the following be met ?– data requirements– software requirements– hardware requirements– personnel requirements
Data Collection and Analysis
• Successful models require careful attention to data details
• Recall : relevant historical data is a key requirement
• For data collection, investigate:– data availability
• parameters, time-frame, frequency, format
– QA/QC protocols • data reliability
– process changes
• Data requirements and guidelines:– data for each of the parameters must be
available– at least one full cycle of data must be available– appropriate QA/QC protocols must be in place – data collected prior to major process changes
should generally not be used
• Data analysis involves:– data characterization – a complete statistical analysis
• Data characterization for each parameter– qualitative assessment of hourly, daily, seasonal
trends (graphical examination of data)– time-series analyses may be warranted
• Statistical analysis for each parameter– measures of central tendency
• mean, median, mode
– measures of variability• standard deviation, variance
– percentile analyses– identification of outliers, erroneous entries, non-
entries
Application of a Model-Building Protocol
• There is no accepted “best method” of developing ANN models
• An infinite number of distinct architectures are possible
• A protocol is to reduce the number of architectures that are evaluated
• A sample five-step protocol:
ASS ES S NEE DS AND SUITABIL ITY
CO LLEC T AND ANALY ZE DATA
APP LY M O DEL DEV ELO PM ENT
PRO TO CO L
EVALUATE PE RFO RM ANCE
SE LE CT AN D O RG ANIZE DATA
PATTE RNS
SE T ARCHITECTURE
CHARA CTE RISTIC S
EVALUATE M O DEL STABIL ITY
FIN E-TUNE M O DE L
SE LE CT INP UTS AND O UTPUTS
• Selection of model inputs and outputsWhy it’s important: – ANN models are based on process inputs and
outputs
How it’s done:– first, select the model output
• best models only have one output parameter
– next, select model inputs from available input parameters
– selection is based on data availability, literature, expert knowledge
• Selection and organization of data patternsWhy it’s important: – ANN models are only as good as the data used– separate independent data sets are required to test
and validate the model
How it’s done: – examine each data pattern for erroneous entries,
outliers, blank entries– delete questionable data patterns– sort and divide data into training, testing, and
production sets– perform a statistical analysis on each of the three
data sets
• Determination of architecture characteristicsWhy it’s important: – each modeling scenario will have an optimal
architecture
How it’s done: – initially, hold many factors at software defaults
or at pre-determined values• use modelling heuristics from literature along with
expert knowledge
– determine the number of hidden layer neurons• compare results for different runs
• Evaluation of model stabilityWhy it’s important: – ensure that model results are independent of the
method of data sorting
How it’s done: – build new training, testing, and production sets
from the original database – re-train models on the “new” data sets – compare results with initial runs
• Model fine tuningWhy it’s important:– some models require minor improvements in
order to meet process operating criteria
How it’s done: – modeling parameters previously held constant
can be varied to improve model results– the model fine-tuning methodology is typically
researcher-specific
• In many situations, more than one good model can be developed
• The best model is the one that both– meets the modelling needs initially identified– offers the smallest prediction errors
• Therefore need to be able to evaluate model performance
Evaluating Model Performance
• Prediction errors can be assessed– graphically
• visual representation of missed predictions
0
1
2
3
4
5
6
71 11 21 31 41 51 61 71 81 91 101
111
121
131
141
151
161
171
181
Pattern #
Cla
rifi
er E
fflu
ent
Col
our
(TC
U)
Actual Network
– using statistics• absolute measures of error
– mean absolute error (MAE)
– maximum absolute error
• relative measures of error
– mean absolute percent error (MAPE)
• coefficients of correlation
– coefficient of correlation (r)
– coefficient of multiple correlation (R)
• coefficients of determination
– coefficient of determination (r2)
– coefficient of multiple determination (R2)
• Model residuals should also be studied– residual = prediction error– residuals should be
• Normally distributed
– plot a histogram of the residuals
• have a mean = 0
• independent
– plot the residuals (in time order if applicable)
– plot should be free of obvious trends
• have constant variance
– plot the residuals against the predicted values
– plot should not show spreading, converging, or other trends
Model Evaluation Using Real-time Data
• Need to consider:– changes in the frequency of data collection– changes in the methodology of data collection– existence of QA/QC protocols to detect
erroneous data
• The evaluation can take the form of– simulated real-time testing on a stand-alone PC– online testing in real-time
• Methodology– select the time frame of the test– the data is ported to the developed models– process each data pattern and record the results– the model predicted values are compared to the
actual values– prediction errors are determined