Real-time Synchrophasor Data

Quality Analysis and Improvement

PSERC IAB Meeting

December 2-4, 2015

Research Team

Le Xie, Texas A&M University

P. R. Kumar, Texas A&M University

Mani Venkatasubramanian, Washington State University

Industry Advisors

• Aftab Alam, California ISO

• Frankie Zhang, ISO-New

England

• Chaitanya A. Baone, GE

Global Research

• Santosh Veda, GE Global

Research

• Paul T. Myrda, EPRI

• Liang Min, LLNL

• Mahendra Patel, EPRI

• Giuseppe Stanciulescu,

BC Hydro

• Vijay Sukhavasi, Alstom

• Gurudatha Pai, Alstom

• Jay Ramamurthy, Entergy

• Yingchen Zhang, NREL

• Prashant Kansal, AEP

• Harvey Scribner, SPP

• Jay Caspary, SPP

• Xiaoming Feng, ABB

• Floyd Galvan, Entergy

• Naim Logic, SRP

• Alan Engelmann, ComEd

2

Summary

3

This project aims at developing strategies for real-time data quality

management of streaming PMU data.

In this project, we will build a systematic online framework for

identifying and handling typical data quality issues such as clock

errors, transducer errors and network delays.

Based on the synchrophasor data’s spatio-temporal correlations,

the proposed approach is capable of identifying bad data during

both normal and fault-on conditions.

Real-world synchrophasor data as well as synthetic dynamic grid

models will be used to differentiate the root causes of data quality

issues and to validate the proposed strategies.

Project Description - Motivation

4

Current Practice Critical Needs

There is an urgent need to

develop scalable, real-time

methods to monitor and

improve synchrophasor data

quality.

Conventional bad data

detection algorithms are

rendered ineffective, novel

algorithms are needed.

Utilities and vendors are developing more and more synchrophasor-based decision making tools.

Synchrophasor data has much higher sampling rate and accuracy requirement compared with traditional SCADA data.

Typical bad data ratio of synchrophasors in California ISO ranges from 10% to 17% (in 2011) [6].

Project Description - Overview

5

Proposed Approach for Synchrophasor Data Quality Issues

Online Bad Data

Detection Algorithm

Root Cause of Data Quality Issues &

Bad Data Mitigation Strategies

Previous Experience

Recent success of dimensionality reduction of synchrophasor data.

Vast experience in handling real-time data quality.

Technical Approach - Overview

6

Thrust I: Real-time Data Quality

Monitoring via Spatio-temporal

Correlations (Xie, Kumar)

• Quantification of PMU Data Spatio-

Temporal Correlation.

• Development of Computationally

Efficient Metrics of PMU Data

Quality.

• Test of the Algorithm During

Normal and Fault-on Conditions.

Thrust II: First principle-based

Classification and Correction of

PMU Data Quality Issues (Mani, Xie)

• Root-cause

Detection/Classification of Bad

Data.

• Correction of Bad Data.

Data Accessibility

The team has access to more than 5TB of PMU data from the field.

A strong effort will be made to test the proposed algorithms based

on realistic data.

1 PMU Measurement Curve

7

Phase Angle Measured by A Western System PMU for A Recent Brake Test Event

Event Bad DataBad Data

Weak

Temporal

Correlation

Bad DataBad Data

Weak

Temporal

Correlation

Weak

Temporal

Correlation

Weak

Temporal

Correlation

Weak

Temporal

Correlation

2 PMU Measurement Curves

8

Event Bad DataBad Data Bad DataBad Data

Weak

Spatial

Correlation

Weak

Spatial

Correlation

Weak

Spatial

Correlation

Weak

Spatial

Correlation

Strong

Spatial

Correlation

Technical Approach – Thrust I

9

Criteria:

Normal Data VS Bad/Eventful Data

Criteria:

Bad Data VS Eventful Data

For a particular PMU measurement, its

low-quality data segment and fault-on

data segment have weak temporal

correlation with its normal data

segment.

For a particular PMU measurement, its

low-quality data segment has weak

spatial correlation with corresponding

data segments of its neighborhoods.

Its fault-on data segment has strong

spatial correlation with corresponding

data segments of its neighborhoods.

Proposed

Approach

Based on the above criteria, this task will develop an

appropriate algorithm in order to differentiate low-quality

PMU measurements from normal and fault-on PMU

measurements.

Technical Approach – Thrust I

10

Quantify the strength of

spatio-temporal correlations.

Develop an appropriate

distance metric.

Investigate a variance-based

distance metric.

Apply the distance metric to

calculate the local outlier

factor (LOF), which is a

density-based indicator of

data quality.

Use LOF to detect bad data.

Technical Approach – Thrust I

11

Test the algorithm using different types of bad data:

• Gaussian random noises.• Bad data caused by physical instrumentation errors.• Bad data caused by malicious attacks.

Test the algorithm under normal/fault-on system conditions.

Test of the Bad Data Algorithm using Synthetic PMU Data

Verify algorithm performance under practical conditions.

Test the algorithm under normal/fault-on system conditions.

Test of the Bad Data Algorithm using Real-world PMU Data

Technical Approach - Thrust II

12

Contributing Factors to

PMU Data Quality Degradation

• GPS time-lock issues

• Internal clock problems

• Communication network delays

• Incorrect settings at PMU

• Incorrect settings at PDC

• Wiring issues

• Bad transducers

• Others…

II.1: Detection of Data Quality

Issues.

II.2: Correction of Data Quality

Issues.

First principle-based Classification and

Correction of PMU Data Quality Issues

Voltage Phase Angle Example

13

Phase Angle Measured by A Western System PMU for A Recent Brake Test Event

No error flag from the PMU. Data supposedly good.

Spikes real or spurious? Spikes disrupt ringdown/ambient data analysis.

Magnitude comparable to values during event.

Event

? ? ??

Voltage Angle versus Voltage Magnitude

14

No spikes in voltage magnitude data.

Spikes in angle data likely spurious.

Event

? ? ? ?

Vo

lta

ge

Ma

gn

itu

de

Vo

lta

ge

An

gle

Voltage Phase Angle after Data correction

15

Spikes fixed using a smoothing filter

Corrected angle - excellent signal for analysis.

Event

? ? ? ?

Co

rre

cte

d A

ng

leR

aw

An

gle

Technical Approach – Thrust II

16

Task II.1: Data Quality Issues - Detection

Compare local measurements available from the same PMU

(example discussed) or from neighboring PMUs.

Study signatures and develop rules.

GPS clock issues and internal clock issues studied earlier at

ASU and WSU.

Study other data quality contributing mechanisms.

Data “holes” seen in PDCs from communication network

issues or PMU device issues.

Technical Approach – Thrust II

17

Task II. 2: Data Quality Issues - Correction

Once data is deemed to be spurious or unreliable, how to “fix” it?

Smoothing filters from signal processing

Dynamic state estimation using neighbor data

Kalman filter approach

Data interpolation techniques

Study different approaches

Recommend strategies for different applications

Field data from actual PMUs to be used for testing and tuning.

Technical Approach – Summary

18

Online Bad Data

Detection

Bad Data Classification

& Correction

Quantification of PMU

data spatio-temporal

correlation.

Development of

computationally efficient

metrics of PMU data

quality.

Test of the algorithm

during normal and fault-

on conditions.

Root-cause detection &

classification of bad data

Correction of bad data

- Signal processing tools.

- Dynamic state

estimation.

Filling missing data

- Interpolation strategies.

- Dynamic state

estimation.

Implementation Plan

• Formalize the

strategy into a

general formulation.

• Test the methodology

various root-cause

mechanisms.

Systematic comparison of the data under

consideration with other local measurements as

well as neighboring measurements.

Key

Idea

Potential Benefits

19

This project will provide vendors, utilities, and system operators

with the analytical framework and computational algorithms for

real-time cleaning up of synchrophasor data.

It will boost up the level of confidence for PMU-based applications.

A proper handling of these data quality issues is also essential for

designing synchrophasor based closed loop control solutions for

the power industry.

The root-cause analysis will benefit system operators to better

identify the sources and mechanisms in the data acquisition and

communication stages so that they can be better handled in the

future.

Expected Outcomes

20

A technical report that describes the theory, algorithm,

and case studies of the proposed data quality

monitoring and correction methods.

Software modules that are available to industry

members on spatio-temporal correlation-based bad

data detection algorithm and root cause based

detection and mitigation algorithms.

Potential Applications

21

Once successful, this research will contribute to data

quality modules in all PMU-based software.

The application will include computational methods,

root-cause analysis, and the potential visualization of

streaming synchrophasor data quality.

Work Plan

22

Number Task Year 1 Year 2

1 Real-time Data Quality Monitoring via Spatio-temporal Correlations

1.1 Quantification of PMU Data Spatio-Temporal Correlation

1.2 Development of Computationally Efficient Metrics of

PMU Data Quality

1.3 Test of the Algorithm During Normal and Fault-on

Conditions

2 First principle-based Classification and Correction of PMU Data Quality

Issues

2.1 Root-cause Detection/Classification of Bad Data

2.2 Correction of Bad Data

Related Work

23

PSERC project T-43 investigates the interoperability and application performance of PMUs and

PMU-enabled IEDs at the device and system level.

Electric Power Group proposes a three-level architecture to solve PMU data quality problems.

Pre-defined thresholds are suggested for bad data detection [1].

A PMU-based state estimator with the capability of identifying phasor angle bias, current

magnitude scaling, and line parameter errors is presented [2].

Researchers from Virginia Tech proposed a Kalman filter-based approach to cleaning PMU

data. System parameter and topology information is required [3].

Synergistic ideas of using singular value decomposition to detect system physical anomaly

are developed in the context of voltage stability monitoring [4].

Automatic PMU testing tool and PMU bad data detection tools based on state estimation and

data mining techniques are developed [5].

Our approach can effectively identify bad data issues during both normal and fault-on

conditions.

The proposed algorithm involves only simple matrix operations (no matrix inversion or

decomposition needed), making it very suitable for online applications.

The root-cause analysis and correction combines first principles into the data conditioning

process, which provides key insights to utilities and system operators.

Key References

24

[1] K. Martin, “Synchrophasor data diagnostics: detection & resolution of data problems for

operations and analysis”, in Electric Power Group Webinar Series, Jan 2014.

[2] S. Ghiocel, J. Chow, et al. "Phasor-measurement-based state estimation for

synchrophasor data quality improvement and power transfer interface monitoring," IEEE

Tran. Power Systems, 2014.

[3] K. D. Jones, A. Pal, and J. S. Thorp, “Methodology for performing synchrophasor data

conditioning and validation,” IEEE Tran. Power Systems, May 2015.

[4] J. Lim; C. L. DeMarco, "Model-free voltage stability assessments via singular value

analysis of PMU data," in Bulk Power System Dynamics and Control - IX Optimization,

Security and Control of the Emerging Power Grid (IREP), 2013.

[5] Anurag Srivastava, “Meeting PMU data quality requirements for mission critical

applications”, in PSERC Public Webinar Series, Nov 2015.

[6] California ISO, “Five year synchrophasor plan,” California ISO, Tech. Rep., Nov 2011.

[7] L. Xie, Y. Chen, and P. R. Kumar, “Dimensionality reduction of synchrophasor data for

early anomaly detection: linearized analysis,” IEEE Tran. Power Systems, 2014.

[8] M. Wu and L. Xie, “Online identification of bad synchrophasor measurements via

spatio-temporal correlations,” submitted to 19th Power Systems Computation Conference,

Genoa, Italy, 2016.

[9] Q. Zhang, and V. Venkatasubramanian. "Synchrophasor time skew: formulation,

detection and correction," North American Power Symposium (NAPS), 2014.