Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
Using Cellular Machine-to-Machine Solutions for Collection System SCADA:
Is a Small Data Plan Big Enough?
Dan Cote, PE, and Tim Baldwin, PE McKim & Creed
Outline Virginia Beach SCADA SYSTEM Telemetry Communications Modernization Project Proof of Concept Test Machine-to-Machine Data Plans DNP3 Protocol Cellular Ethernet Communications IT Issues Lessons Learned Conclusions
Virginia Beach SCADA System SCADA has been in operation in Virginia Beach for many years Water and Sewer SCADA systems were combined into one SCADA
application in 2008
SCADA Servers CitectSCADA
Water System Front-End
Water System SCADA Serial BSAP Communications on Licensed UHF and Spread Spectrum Radios
Bristol 3300 RTUs no longer available.
Toshiba EX40s are now obsolete.
Sewer System SCADA Serial Modbus Communications on Licensed UHF and Spread Spectrum Radios
Modicon Micro PLC is now obsolete
Serial Radios Approaching Limit of Data Capacity
Telemetry Communications Modernization Project • System and Goals Assessment Highlights
• Flow Monitoring to meet Regional Standards and Consent Order. • PLC/RTU Standards for Equipment Replacement • Communications Infrastructure to meet Future Needs.
• Preliminary Engineering Report Highlights • Proposed Protocol: DNP3 • Proposed PLC Hardware: Top 3 PLCs Selected • Proposed Communications Infrastructure: Cellular M2M Ethernet
• Proof of Concept Testing • Evaluate Protocol: DNP3 • Evaluate Top 3 PLCs: Hands-on-Experience • Evaluate Cellular M2M Ethernet: Actual Data Usage Testing
Telemetry Communications Modernization Project • System and Goals Assessment Highlights
• Flow Monitoring to meet Regional Standards and Consent Order. • PLC/RTU Standards for Equipment Replacement • Communications Infrastructure to meet Future Needs.
• Preliminary Engineering Report Highlights • Proposed Protocol: DNP3 • Proposed PLC Hardware: Top 3 PLCs Selected • Proposed Communications Infrastructure: Cellular M2M Ethernet
• Proof of Concept Testing • Evaluate Protocol: DNP3 • Evaluate Top 3 PLCs: Hands-on-Experience • Evaluate Cellular M2M Ethernet: Actual Data Usage Testing
Proof of Concept Test Install 3 PLCs at the same pump station to maintain identical
conditions Each PLC has its own cellular data plan for data usage comparison
SCADAPack 334E
Schneider M340
Kingfisher G30
Machine-to-Machine Machine-to-Machine Data Plans are widely used
• Plans are available as low as $1, $5, or $10 per month
Machine-to-Machine One objection to Cellular Data is the recurring cost: A 5 GB / Unlimited plan typically costs $45/month. Unattractive for
SCADA. M2M plans in the range of $10/month are attractive, compared to
building and maintaining a private SCADA radio system. The M2M plans are very small… too small?
5 GB
Conventional Tier 2 M2M
250 MB
Tier 1 M2M
25 MB
Serial UHF
12.8 MB X 2 = X 10 =
X 20 =
Data Available Per Month at Each Station
Machine-to-Machine One objection to Cellular Data is the recurring cost:
• A 5 GB / Unlimited plan typically costs $45/month. Unattractive for SCADA.
• M2M plans in the range of $10/month are attractive, compared to building and maintaining a private SCADA radio system.
The M2M plans are very small… too small?
5 GB
Conventional Tier 2 M2M
250 MB
Tier 1 M2M
25 MB
Serial UHF
12.8 MB X 2 = X 10 =
X 20 =
Data Available Per Month at Each Station
Why DNP3 Protocol? Open Industry Standard
• Not proprietary to one equipment supplier Versatility
• Highly configurable to meet your needs Efficient use of Band Width
• M2M data plans Automatic Backfill of Data
• Never lose your data Compatibility with HMI
• No third-party software
Why DNP3 Protocol? Open Industry Standard
• Not proprietary to one equipment supplier Versatility
• Highly configurable to meet your needs Efficient use of Band Width
• M2M data plans Automatic Backfill of Data
• Never lose your data Compatibility with HMI
• No third-party software
DNP3 Terminology Terminology:
EVENT: A single measurement, reading, calculation or event which is ‘Time-Tagged’ and sent to the BUFFER and later to the MASTER
Pressure 24.5 PSI 8:15 AM
Pump 1 ON
3:21 AM
Starts/Hr 6
4:00 PM
BUFFER
Holds up to 20,000 Events
PLC SCADA
DNP3 Protocol Terminology:
REQUEST EVENT DATA: Ask PLC to send event data. Called an “Event Poll”
Pressure 24.5 PSI 8:15 AM
Pump 1 ON
3:21 AM
Starts/Hr 6
4:00 PM
BUFFER
Holds up to 20,000 Events
PLC SCADA Request Event Data
Every 2 minutes
Or
Once/Hour?
DNP3 Protocol Terminology:
RESPONSE: PLC responds by sending the stored data.
Pressure 24.5 PSI 8:15 AM
Pump 1 ON
3:21 AM
Starts/Hr 6
4:00 PM
BUFFER
Holds up to 20,000 Events
PLC SCADA Request Event Data
Response
Event Trends
DNP3 Protocol Terminology:
CONFIRM: Once data is correctly received, a confirmation is sent.
Pressure 24.5 PSI 8:15 AM
Pump 1 ON
3:21 AM
Starts/Hr 6
4:00 PM
BUFFER
Holds up to 20,000 Events
PLC SCADA Request Event Data
Response
Event Trends Confirm
DNP3 Protocol Terminology:
Clear Buffer: Data has been sent. Does not need to be sent again.
Pressure 24.5 PSI 8:15 AM
Pump 1 ON
3:21 AM
Starts/Hr 6
4:00 PM
BUFFER
Holds up to 20,000 Events
PLC SCADA Request Event Data
Response
Event Trends Confirm
Event Trend: Automatic Backfill
Event Poll
Event Poll
40 seconds later
40 seconds later
40 seconds later
Figure 2
Figure 3
Figure 4
Figure 5
Figure 1
Status changes of Pumps or Valves:
150 events per day
Event Logging: Pump Running
TIME BASED RECORDING
Actual Time and Date recorded
Pump #1 Pump #2
Store an Event once every 30 seconds: 2,880 events per day
Event Logging: Time vs Dead Band
EXAMPLE SIGNAL
TIME BASED RECORDING
1,100 Events Per Day
450 Events Per Day
Event Logging: Log by Dead Band
1 Foot Deadband
2 Foot Deadband
260 Events Per Day
Event Logging: Smart Dead Band
“SMART” Dead Band Algorithm
Create Time-Stamped Generated Events for the Past
Using DNP3 in Ethernet Ethernet Sends Data in Packets. Each packet contains ‘Overhead’ for addressing, routing, security etc.
67 Bytes - Overhead Request Events 6 Bytes - Message
Response
Confirm 6 Bytes - Message
PLC/RTU SCADA
Using DNP3 in Ethernet Ethernet Sends Data in Packets. Each packet contains ‘Overhead’, for addressing, routing, security etc.
67 Bytes - Overhead Request Events 6 Bytes - Message
Response 13 Bytes – Each Event 71 Bytes - Overhead
Confirm 6 Bytes - Message
67 Bytes - Overhead
PLC/RTU SCADA
Using DNP3 in Ethernet Ethernet Sends Data in Packets. Each packet contains ‘Overhead’, for addressing, routing, security etc.
Response
PLC/RTU SCADA
1 event / 2 minutes: 5 MB/Month 11 % Effic
10 event / 2 minutes: 9 MB/Month 38 % Effic
100 event / 2 minutes: 37 MB/Month 79% Effic
60 event / 2 minutes: 22 MB/Month 77 % Effic
67 Bytes - Overhead Request Events 6 Bytes - Message
13 Bytes – Each Event 71 Bytes - Overhead
Confirm 6 Bytes - Message
67 Bytes - Overhead
Using DNP3 in Ethernet Ethernet Sends Data in Packets. Each packet contains ‘Overhead’, for addressing, routing, security etc.
Response
PLC/RTU SCADA
1 event / 2 minutes: 5 MB/Month 11 % Effic
10 event / 2 minutes: 9 MB/Month 38 % Effic
100 event / 2 minutes: 37 MB/Month 79% Effic
60 event / 2 minutes: 22 MB/Month 77 % Effic
67 Bytes - Overhead Request Events 6 Bytes - Message
13 Bytes – Each Event 71 Bytes - Overhead
Confirm 6 Bytes - Message 67 Bytes - Overhead
How Many Events Do I Need? Description ModBus (2 minute poll) DNP3 (adjustable) Wetwell Level 720 720 Pressure 720 2880 (30 seconds) Pump 1 Running 720 75 (actual start/stop time tagged) Pump 2 Running 720 75 Pump 1 Hourly RT 720 24 Pump 2 Hourly RT 720 24 Pump 1 Hourly Starts 720 24 Pump 2 Hourly Starts 720 24 Hourly InFlow 720 24 Hourly OutFlow 720 24 Intrusion 720 5 Other Possible Alarms 720 x 20 3 (only when they occur) Battery Voltage 720 20 (on deadband) Discharge Pressure 720 1,200 (on deadband) Rainfall 720 13 (only when changing) Subtotal 24,480 events 5,135 events/day FUTURE TAGS Pump Amperage 3,700 (every 10 seconds when running) Pump Voltage 600 (on deadband) Power Usage per Hour 24 Max Voltage per hour 24 Min Voltage per hour 24 Water System Pressure Monitor 8,640 (every 10 seconds) Total: 18,147 events/day
How Many Events Do I Need? Description ModBus (2 minute poll) DNP3 (adjustable) Wetwell Level 720 720 Pressure 720 2880 (30 seconds) Pump 1 Running 720 75 (actual start/stop time tagged) Pump 2 Running 720 75 Pump 1 Hourly RT 720 24 Pump 2 Hourly RT 720 24 Pump 1 Hourly Starts 720 24 Pump 2 Hourly Starts 720 24 Hourly InFlow 720 24 Hourly OutFlow 720 24 Intrusion 720 5 Other Possible Alarms 720 x 20 3 (only when they occur) Battery Voltage 720 20 (on deadband) Discharge Pressure 720 1,200 (on deadband) Rainfall 720 13 (only when changing) Subtotal 24,480 events 5,135 events/day FUTURE TAGS Pump Amperage 3,700 (every 10 seconds when running) Pump Voltage 600 (on deadband) Power Usage per Hour 24 Max Voltage per hour 24 Min Voltage per hour 24 Water System Pressure Monitor 8,640 (every 10 seconds) Total: 18,147 events/day
1 2 3 4 SECONDS
SECO
NDS
1
2
3
This is how TCP guarantees packet delivery:
Lessons Learned: Ethernet TCP
RTO = Response Timeout
WHAT IS CAUSING THESE TCP RETRANSMISSIONS??
RTO Doubled TCP Retransmission
TCP: Transmission Control Protocol
1 2 3 4 121 125 241 245 SECONDS
SECO
NDS
1
2
3
REQUEST:RESPONSE Note: First Request after delay has 2.5 seconds latency, because of cellular network.
Lessons Learned: Ethernet TCP
Round-Trip Timeout (RTO) controls TCP Retransmissions. RTO adjusts automatically:
Ethernet TCP automatic Return Trip Time control: Lessons Learned: Ethernet TCP
1 2 3 4 121 125 241 245
SECO
NDS
1
2
3
SECONDS
SECONDS
When latency exceeds RTO: • TCP Retransmission is sent. • RTO is doubled.
Lessons Learned: Ethernet TCP
1 2 3 4 121 125 241 245
SECO
NDS
1
2
3
SECONDS
Green Line is Round-Trip Timeout (RTO) RTO adjusts automatically again.
Lessons Learned: Ethernet TCP
1 2 3 4 121 125 241 245
SECO
NDS
1
2
3
SECONDS
TCP Retransmissions will be sent with every event poll.
Lessons Learned: Ethernet TCP
1 2 3 4 121 125 241 245
SECO
NDS
1
2
3
Lessons Learned: Ethernet TCP
TCP does not work well with Cellular M2M in SCADA applications.
Answer: Use UDP instead. Since UDP does not guarantee packet delivery, the DNP3 protocol must be able to control communications quality.
Conclusions
M2M Data plans can be cost effective, but efficient operation requires careful attention to details.
A versatile protocol such as DNP3 is needed for best results.
TCP does not work well with M2M SCADA. PLC equipment and protocol must be able to guarantee data delivery using UDP.