Support Vector Machine Applications Electrical Load Forecasting ICONS Presentation Spring 2007 N. Sapankevych 20 April 2007

Support Vector Machine Applications Electrical Load Forecasting

ICONS Presentation

Spring 2007

N. Sapankevych20 April 2007

N. Sapankevych - 20 April 20072

SVMs and Electrical Load Forecasting

Research Status SVMs, Time Series Prediction, and Electrical

Load Forecasting Relevance Vector Machines Way forward Q&A

AGENDA


SVM time series prediction survey complete– Major research focuses on financial market forecasting and

electrical load prediction– Paper to be submitted to IEEE Neural Network Journal

Research focus on Support Vector Machines (SVMs) and their applications for time series prediction

– Expand to Relevance Vector Machines

Thesis topic definition– Apply SVM time series prediction research

Expand research to use relevance vector machines

– Potential collaboration w/ Prof. Domijan and Prof. Islam of USF EE Department

Electrical energy consumption prediction and resource management allocation wrt severe weather

RESEARCH OVERVIEWSVMs and Electrical Load Forecasting


Summary of SVM Time Series Prediction Survey

SVM Time Series Prediction Survey Summary


Application Number of Published Papers Summarized in this Survey

Financial Market Prediction 21

Electric Utility Forecasting 17

Control Systems and Signal Processing

8

Miscellaneous Applications 8

General Business Applications 5

Environmental Parameter Estimation 4

Machine Reliability Forecasting 3

Comprehensive survey:– 66 specific application papers + 17 general

references/books + 11 SVM specific websites


Advantages and challenges of SVM Time Series Prediction

SVM Time Series Prediction Survey Summary (con’t)


Time Series Prediction Method

Advantages Challenges

Autoregressive Filter - Can be computationally efficient for low order models

- Convergence guaranteed- Minimizes mean square error by design

- Assumes linear, stationary processes- Can be computationally expensive for higher

order models

Kalman Filter - Computationally efficient by design- Convergence guaranteed- Minimizes mean square error by design

- Assumes linear, stationary processes- Assumes process model is known

Multi-layer Perceptron - Not model dependent - Not dependent on linear, stationary

processes - Can be computationally efficient (feed

forward process)

- Number of free parameters large- Selection of free parameters usually calculated

empirically- Not guaranteed to converge to optimal solution- Can be computationally expensive (training

process)

SVM/SVR - Not model dependent - Not dependent on linear, stationary

processes - Guaranteed to converge to optimal

solution- Small number of free paramters- Can be computationally efficient

- Selection of free parameters usually calculated empirically

- Can be computationally expensive (training process)


Seventeen papers reviewed and documented for electrical load prediction using SVMs

– Should note more than half of them use same general approach but different data set

Majority of papers use same basic approach to predict (short term) electrical load demand

– Some models use weather trends as inputs– Free parameters determined empirically

A few papers propose use of other algorithms, such as genetic algorithms and simulated annealing, to assist in parameter selection

– The accurate and optimal selection of the free parameters remains the challenge for SVM Time Series Prediction

Note for the same kernel functions selected, different numerical values found empirically for different applications

No set metrics for measures of effectiveness– RMSE, MSE, MAPE

Prediction horizons short– Typically less than seven samples

No error estimate in prediction (for all applications)

SVM Time Series PredictionElectrical Load Forecasting Summary



Developed by Tipping in 1999– Michael E. Tipping, The Relevance Vector Machine, patent

number 6633857 (Microsoft Research) Work focused on addressing SVM disadvantages

– No probabilistic output Uncertainty of prediction (regression) or classification (binary

selection) not given– Although, distance from maximum hyperplane (as example) could

infer accuracy?

– In general, need to empirically determine errors relative to free parameters

– Specific rules related to generating kernel functions (Mercer’s conditions)

Relevance Vector MachinesSVMs and Electrical Load Forecasting


Formulation uses same form as SVM, but a Bayesian framework is adopted

Goal, from a regression (and ultimately a time series prediction application) point of view, is to provide the predicted “state” as well as an estimate of the uncertainty

Relevance Vector Machines (con’t)SVMs and Electrical Load Forecasting

SVM Architecture There is an underlying assumption in this formulation is that Gaussian statistics are assumed for prediction

– Can this assumption hold for highly non-linear, non-stationary systems?


The formulation outline below is summarized from: M. E. Tipping, “The Relevance Vector Machine”, Advances in Neural Information Processing Systems 12, pages 652-658. MIT Press, 2000.

Recall formulation of regression function using SVM

Goal of SVM to determine optimal weights w– Define input vectors xn and “targets” tn

– The targets t are the training data (output) based on the input vector x

Now assume Gaussian distribution for targets– P(t|x) is Gaussian: N(t|y(x),2)


N

nnn wxxKwxy

10),()(


The likelihood of the data set can be expressed as:– p(t|w,2) maximize this likelihood

By introducing hyperparameters n (necessary to avoid over-fitting) and applying Bayes rule, the posterior probability is generated

– p(w|t,,2) The integration of this equation (over the weights)

produces the marginal likelihood (evidence)– p(t|,2)– Note that Tipping states that integrating out the remaining two

priors can not be done in closed form, and proposes another procedure (see ref)

Goal now is to optimize the hyperparameters and estimate the noise variance



Tipping showed results of adding Gaussian noise to sinc function (regression)



Tipping showed results of classification problem



Why use RVMs?– More sparse representation of the data– Can be more accurate– Kernel function selection not as stringent (Mercer’s Theorem –

Kernel matrix positive semidefinite, etc.)– Uncertainty of estimate now provided

Drawbacks– Training more complex with Bayesian framework

Questions– Is Gaussian assumption on training data representative of

applications where SVM is used?



Propose integrated predictive model, now based on RVM approach, to predict power loads and potential infrastructure failure probabilities (resource management) in inclement weather

Way Forward: Electrical Load ForecastingSVMs and Electrical Load Forecasting

NWS ProductsTrajectorySizeIntensityDurationAccuracy

Infrastructure DataGeospatial DataPopulation Density DataPhysical Infrastructure LaydownPower Infrastructure MTBFs (as function of Wx and maintenance)Spatial-Temporal Power Loading

Cost DataPersonnel allocationEquipment maintenance and replacementkW-Hr ratesPriorities (residential vs. commercial vs. “critical”)

Integrated Predictive Model

PREPROCESS

PREPROCESS

PREPROCESS

Power Load PredictionDecision Aids to Resource Management Directors

DRAFT ARCHITECTURE


What needs to be done– Publish survey paper (final review w/ Prof. Sankar today – per

request)– Work w/ Prof. Sankar, Prof. Domijan and Prof. Islam to:

Clearly define thesis topic Define and obtain necessary data (at least on electrical load side

of problem)– Develop RVM approach to problem

Compare to SVM? Compare to other models (MLP, ARMA, other?) Implement simulation

– Review thesis topic w/ committee members Get buy-in

Way Forward: Electrical Load ForecastingSVMs and Electrical Load Forecasting


Q&ASVMs and Electrical Load Forecasting

Alexander– 19 months and getting bigger!

Documents

Support Vector Machine Applications Electrical Load Forecasting ICONS Presentation Spring 2007 N. Sapankevych 20 April 2007