1

A Deep Dive Advanced Synchrophasor Protocol – DE-OE-859

RitchieCarrollNERCSMSMee1ng

May16,2017

2

Objec1veBuildingonthesuccessfulelementsofGPA’sGatewayExchangeProtocol,documentanddevelopthetoolstosupportamul1-plaMorm,vendor-neutral,publish-subscribestreaming1me-seriesdataprotocolthatmeetsthedemandingrequirementsforreliabledeliveryofsynchrophasordata

2-YearProjectScheduleMay2017–April2019

§  ReleaseSpecforComment–12/31/2017§  PublishSpec–4/30/2018§  ReleaseAlphaAPIToolKit–4/30/2018§  BeginDemowithWSUandEPGTools–10/1/2018§  Version1.0ToolKitReleased–4/30/2019

Project Status §  ProjectAwardedMay4,2017§  Projectpartneragreementsbeingdeveloped

ASP Project Summary Animprovedprotocolforstreaming1me-seriesdata.

FOA1492DE-OE-859

NERCSMS-May2017

3

Background – The Problem Statement

  Protocoldataformatsarefixed  Largedataframesrequireasizablenumberofnetworkpackets§ IncreasingopportunityforUDPloss§ IncreasingTCPlatency  Dataframewillinclude“placekeepers”fordatathatdidnotarrivewithinthelag-1me  Volumeofdataperframehasafixedupperlimit--typically64K  Limitedbuilt-insecurityop1ons,withmostprotocolsofferingnone

NERC SMS - May 2017

4

Problems with IEEE C37.118 are a function of scale.

NERCSMS-May2017

Increaseddatalossandlatency.Purpose-built/allocatednetworkstypicallyrequired.

Issues,ifany,areeasytoresolve.

Datalossandlatencyissuesbegintoappear.Networktuningmayberequired.

IEEEC37.118V1&V2configura1onframesizemaxout.Asecondstreammustbecreated.

5

ASP Project Participants

NERC SMS - May 2017

6

Large-scale ASP Project Demonstrations

 WSUDemos§ TVA§ SPP§ OG&E§ SDG&E EPGDemos§ Dominion§ PJM

NERC SMS - May 2017

7

Project Plan   Forthedemandingrequirementsofsynchrophasordata:§  Createaprotocolspecifica1onbasedonGPA’sGEP§ Developtestandvalida1ontools§  Createmul1plereferenceimplementa1ons§  Reportonprotocolefficacyandperformanceasit’sdemonstratedatscale

§  Presenttheprotocoltostandardsbodies  Candidateprotocolname:StreamingTelemetryTransportProtocol

NERC SMS - May 2017

8

Project Schedule

NERCSMS-May2017

9

Background Current Phasor Protocols

NERC SMS - May 2017

10

Recognized Phasor Protocols

  IEEEC37.118-2005(mostwidelyused)  IEEEC37.118.2-2011[v2]  IEEEC37.118.3[v3]  IEC61850-90-5  IEEE1344-1995  BPAPDCstream  SELFastMessageMacrodyneStreamingDataProtocol  UTKF-NETStreamingDataProtocol

NERC SMS - May 2017

11

Commonalities of Phasor Protocols

  Eachprotocolisframe-based,i.e.:§ A1me-stamp,followedby§ Ablockofdataforoneormoredevices

  Electricindustryspecificcontentincludes:§ VoltageandCurrentphasors(complextype)§  Frequency§  RateofChangeofFrequency(dF/dt)§ Analogvalues§ Digitalvalues  Returningdatafrommeasurementdevicesisapriorityofprotocoldesign

NERC SMS - May 2017

12

Network Layer Fragmentation

 Networksareinherentlyunreliable Messagepayloadsareencodedatabit-levelasapulse§ 8-pulsesbecomeabyte§ Asmallpor1onofthebytesfromthemessagepayload,a“packet”typicallysizedaround1,500bytes,issentasthemaximumtransmissionunit(MTU)

  Datatransmissionstoanetworkrecipientarehandledinasemaphorefashion,i.e.,onlyonepacketata1me§ Whenapacketsisaoemptedtobesentwhenanotherisalreadysending,thisiscalleda“collision”–itisupthesendertoguaranteeretransmission

NERCSMS-May2017

13

Common Network Protocols

  UserDatagramProtocol(UDP)§ UDPisa“fire-and-forget”transmissionprotocol–inthismodeofopera1on,whenanyindividualpacketfromamessagepayloadisdroppedduetoacollision,theen1repayloadisdropped

§  Issue:Largermessagepayloadscauseincreasednumbersofpacketsincreasinglikelihoodofdataloss

  TransmissionControlProtocol(TCP)§  TCPmonitorsthesuccessfuldeliveryofindividualpacketstoades1na1on.Whenapacketfromamessagepayloadisdroppedduetoacollision,thepacketisretransmioed

§  Issue:Onsloworbusynetworkswithlargepayloads,packetretransmissionscanmanifestas1medelays

NERCSMS-May2017

14

Why a new protocol? Existing phasor protocols are challenged at scale

NERC SMS - May 2017

15

Why not use one method for data exchange everywhere? FieldReali1es§  Changecostlyandslow

•  Connec1onsbetweensubsta1ondevicesrarelychangeaperini1alturn-up•  Fieldhardwareandsopwareisnotupgradedopen•  Complianceassuranceamajorfactorinchangecostandapproach

§  Availabilityandsecurityassuranceisbakedin“bydesign”

ControlCenterReali1es§  High-changeenvironment

•  Growingsetofapplica1ons•  Overalldatavolumeandtheamountofdataexchangedisincreasing•  Costsassociatedwithrou1nechangeareincreasing

§  Availabilityandsecurityassurancerequiresclosemonitoringandcon1nuouschange§  HardwareandsopwaremethodscloselytrackITstate-of-the-art

Thefunc)onsandrequirementsaredifferent.Thesolu)onisdifferent.

NERCSMS-May2017

16

STTP Design Objectives   Performathighvolume/largescale§ Minimizelosses§  Lowerbandwidthrequirements  Op1mizedfortheperformantdeliveryofindividualdatatypesviaTCPandUDP  Automatedexchangeofmetadata  Detectandexposecommunica1onissues  Securityandavailabilityfeaturesthatenableuseoncri1calsystemstosupportcri1calopera1ons  Pub/Sub–MeasurementBased

…andfortheAPIso9wareincludedintheproject,tobeimplementableinmul)plelanguages,onmul)plepla@ormswiththeabilitytoeffec)vely

u)lizethehardwarepresentedtoit

NERC SMS - May 2017

17

Contextual Data Flow Diagram

NERCSMS-May2017

18

Why not use an existing protocol?

 Otherstandardprotocolswereevaluatedforsuitability Mostwereeliminatedforoneofthefollowingreasons:§ Request/Reply(i.e.,non-streaming)nature§ Insufficientspecifiedlimitsondatathroughput§ Restric1vepayloadformasng,e.g.,inabilitytosendbinarydata§ Forcedtransportspecifica1ons,e.g.,HTTP§ Limitedopportunitytoop1mizebandwidthrequirements

NERC SMS - May 2017

19

  Openandnon-proprietary  Truepub/sub,measurement-basedprotocol  Automatedexchangeofauthorizedmetadata  Tightly-compressed,binaryserializa1onof1me-seriesvalues  Adaptersprovidedin.NET,C/C++andJavaforconvenientna1veintegra1oninothersystems  Efficient--includeslosslesscompression  AvailabletransportsincludeTLS,TCP,TCPwithUDP,TLSwithAESkey-rotatedUDP,andZeroMQ  GEPisembeddedinallGPAproducts

Foundational Experience for Design of STTP is the Gateway Exchange Protocol

NERC SMS - May 2017

20

GEP is Small Without Being Frame-based

Notethatlosslesscompressiontechniquesareappliedtoserializedmeasurementgroupstofurtherreducepacketsize.

NERCSMS-May2017

21

GEP is a Signal Centric Protocol

 Unlikeexis1ngprotocols,GEPisfocusedon1melydeliveryofindividualmeasuredvalues–thisleaves1me-alignmentac1vi1estoend-userapplica1onsandeliminatescascadinglag-1medelays  Asignalfocus,insteadofafixedcontentframe,allowsdatatobereportedbyexcep1onandsendingsignalswithdifferentframeratesoverthesamecommunica1onschannel

NERC SMS - May 2017

22

GEP Fixes Issues of Scale

NERCSMS-May2017

GEPhandlesfrom3to5millionpointsper

secondperconnec1on

onpedestrianhardware.

C37.118islimitedto

~200Kpts/secperstream

23

What About DDS and ZeroMQ?

  Thesepopularmessagingexchangemiddlewaresystemsareallabouthowtoreliablydistributemessagesandaregenerallyindifferentaboutthepayload  ASPisfocusedmoreonpayloadandhowitaffectsthetransport*  ASPcouldbedeployedoverDDSand/orZeroMQtransports–buttheseop1onswouldnotberequirednordependedon

NERC SMS - May 2017

24

GEP vs. IEEE C37.118 Tests Conducted by PeakRC Vancouver

Loveland

NERC SMS - May 2017

25

As Expected, Much Less Data Loss with GEP

DataLoss

2.12

0.14

IEEEC37.118 GEP

UDP UDP UDPTCPTCPLarge Large Medium Medium Small

0.22

0.0

0.31

0.04 0.03 0.00.12

0.02

NERC SMS - May 2017

26

GEP is Less Demanding on Networks 60%to70%ofthebandwidthforlargeandmediumcases

Bandwidth(MBITS/S)

IEEEC37.118 GEP

UDP UDP UDPTCPTCPLarge Large Medium Medium Small

2.71

4.95

2.77

1.58

0.89 0.89

0.150.08

1.72

0.60

NERC SMS - May 2017

27

STTP will improve on GEP

  Documenta1ontoenableunderstandingandinteroperabilityandtopromoteuse  Stand-aloneAPI-stylecodethatcanbeintegratedintoanydevelopmentplaMorm/project  Expandingandextendingmetadatafields§ Minimumrequiredsetofmetadatafields§  Capabilityformetadataversioning

  Security–communica1onsestablishedfromthehighersecurityzone  Refinement§  Simplifythroughout§ Dropanyobsoleteorrela1velyunusedGEPfunc1onality

NERC SMS - May 2017

28

STTP will Support Multiple Data Types

  Byte  Int16  Int32  Int64 UInt16 UInt32 UInt64  Single  Decimal

  Double   Complex   Tuple TimeSpan   Char   Bool   GUID   String   Byte Array & more ...

NERC SMS - May 2017

29

Technical details The protocol internals

NERC SMS - May 2017

30

Flexible, Free-form Protocol

  STTPisnotaprescrip1vepackageofdata§ Notframebased,butmessagebased

  STTPincludesbothcontextanddata§ Metadataanddataareexchanged

  Theclosestanalogousu1lityprotocolmightbeDNP3§ UnlikeDNP3,STTPwillbepurposebuiltforhigh-volumestreamingdatawithrichdatatypes,highresolu1on1mestampsanddetailedassociatedmetadata

NERC SMS - May 2017

31

Multi-layer Security

  Communica1onswillbesecuredusingstandardTransportLayerSecurity(TLS)implementa1ons§ ThisincludesusingX-509cer1ficatesforiden1tyvalida1on  Datawillbesecuredusingauthoriza1onscontrolledatanindividualsignallevel§ Permissionswillbegrantedusingexplicitpoints,groupsofsignalsandimplicitexpressionsagainstassociatedmetadata,e.g.:

ALLOWMetadataWHERESignalType='FREQ'

NERC SMS - May 2017

32

Bidirectional Communication Establishment

  Accommodatecrossingsecurityzonesinthedesireddirec1on(typicallyhigher-to-lower)§ Regardlessofthedefinedroleofapublisherorsubscriber

§ Withoutaffec1ngsecurityimplementa1ons,authen1ca1onofsubscriberbypublisher,andsubsequentauthoriza1onofalloweddata

NERC SMS - May 2017

33

STTP Core Requirements   Reliablyexchangehigh-sampleratemeasurementswithminimum1medelayalongwithassociatedmetadata

  Allowsimpleselec1on,e.g.,byusingexpressionsofavailablemetadata,ofmeasurementstobemadeavailabletosubscribers

  Detectandexposecommunica1onsissuesforloggingbyimplemen1ngapplica1on

  Beimplementableaspartofahigh-availabilitysolu1onthatcanmeetNERCCIPcompliancerequirements

  Accommodateexecu1ononstandardcommunica1ons,networkingandserverhardware

NERCSMS-May2017

externallibrariesanddependenciesforreferenceimplementa1onsMinimize

portabilityinmindwithallprotocoldesignworkKeep

SmallestpossibleAPIfunc1onality–specializedusecaseswillbehandledbyexampleTarget

designmantratobe“keepitsimple”aspossibleSet

34

STTP Differences from GEP   Obsoleteorrela1velyunusedGEPfunc1onality,e.g.,Gatewaystyleauthen1ca1on,bufferblocks(inlieuofabyte[]datatype),.NETserializa1onop1ons,commandchannelsynchroniza1onbytes,etc.willbedropped  Protocolwillbeasetofstand-alonesetofAPIstylecodethatcanbeintegratedintoanyproject,i.e.,notalibrarywithintheGridSolu1onsFramework  Willhaveacommonabsoluteminimumsetofmeta-datafieldsformeasurementdataexchange,withasimpleserializa1onformat,thatcanbeeasilyextendeduponrequestperindustrydeployment  Alldesignwillhaveacross-plaMormmindset,thatis,anycodebeingwrioenin.NETmustbeeasilyportedtoCorC++

NERC SMS - May 2017

35

Candidate New Features

  EnhancedPer-channelDeliveryCharacteris1cs:§  Priority§  Reliability§ Verifica1on§  Excep1on§  Resolu1on  Improvedmeta-dataexchangehandlingthatincludesversioningand/or1mestampssothatonlydevia)onsinmeta-datacanbeexchangedinsubsequentupdates

NERC SMS - May 2017

36

Example Commands and Responses

PreliminaryLists

NERCSMS-May2017

37

Next Steps

Business/Project

  Developprojectagreements  ProjectKick-offwithpartnersJune16  Startworkontheprotocolspecifica1ondocument

Technical

  Defineasetofmeasurementdatatypesfortransport  Defineminimalrequiredmeta-data  SetupindependentSTTPwebsitefordocumenta1on,languagespecificreferenceimplementa1ons,andexampleusecases

NERCSMS-May2017