MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 1
Minutes of the Special Meeting on URTDSM / SCADA / Communication held on 13.03.2018
1. Introduction
A Special Meeting on URTDSM / SCADA / Communication was held on 13.03.2018 at SR-II,
PGCIL, Bengaluru. The list of participants is enclosed at Annexure-I.
In the 141st OCC meeting held on 09.03.2018, it was noted that there were frequent outage of
ICCP links and the ICCP data was not stable. APTRANSCO and TSTRANSCO had expressed
concern on the operational and commercial implications due to this. There were other
SCADA/Communication issues along-with URTDSM Project Phase-I was nearing
completion and it was agreed to take stock of the situation in a special meeting. The agenda
was taken up for deliberation.
2. URTDSM
2.1 Status of URTDSM project as on 12.03.2018
Region Approved Installed Integrated
SR-I SS 27 25 24
SR-I PMU 113 104 98
SR-II SS 41 40 38
SR-II PMU 142 140 135
Total PMU SR 255 244 233
SR-I - 9 PMU –Chittor (5) and Srikakulam (4) installation works under progress.
SR-I - 5 PMU -Srisailam (4) and Kurnool (1) data validation is not completed.
SR-I - 1 PMU -Simhadri (1) - SRLDC requested SR-1 to co-ordinate with NTPC Simhadri
to complete the pending installation works.
SR-II -2 PMU -Kaiga (2) installation works under progress.
SR-II -5 PMU -Arasur (2) and Kalpakkam (3) data validation is not completed.
2.2 Supply and installation of APS- Auxiliary power supply system as part of
URTDSM project
SRLDC vide letter dated 14-02-2018 (Annexure-II) had requested to SR-II to expedite
the supply Auxiliary Power Supply System.
SR-II had intimated that the Auxiliary Power Supply System is under manufacturing and
is expected to reach site by 20thApril 2018 and commissioning will be completed by 15th
May 2018
AP and TS also expressed concern regarding delay in supply of Auxiliary power supply
system. SR-I confirmed that the APS System is expected to reach site in April 2018 and
commissioning was expected by May 2018.
MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 2
2.3 Status of SAT and SAVT of URTDSM project in SR
SRLDC - SAT completed and SAVT is yet to commence.
KSEBL - SAT and SAVT completed.
KPTCL - SAT and SAVT completed.
TNEB - SAT completed and SAVT is yet to commence
TSTRANSCO - SAT completed and SAVT has commenced from 1st March 2018
APTRANSCO - Commissioning works under progress expected to be completed by
31st March 2018.
2.4 Powergrid Analytical Application installed as part of URTDSM
SRLDC vide letter dated 02-03-18 (Annexure-III) had informed Smart Grid –
PGCIL the need to improve the performance of installed Analytical Application to
make it useful for real time grid operations
SRLDC requested the forum to consider a procedure on the lines of SAT for verifying
the working of the Analytical Applications.
It was noted that proper feedback could be given for improvement of the Analytical
Application Software. All the control centers were requested to use the software and
report the deficiencies. PGCIL agreed to make a Web group of all the Control
Centers, LD&C, IIT Mumbai, SR-I/SR-II etc for exchanging the information/usage
and for deficiencies / rectification etc.
It was noted that Chief Engineer, NPC vide letter dated 12.02.2018 (Annexure-IV)
had sought certain information on the Analytics and implementation of PMUs from
CTU.
3. SCADA and Communication Issues
3.1 ICCP links failure / not stable
APTRANSCO vide letter dated 26.02.2018 (Annexure-V) had expressed concern on
the ICCP data not being received for long durations due to apparent failure of PGCIL
communication network and redundancy failure at various Network/Server
equipment at SRLDC. Concern was expressed on maintaining the overdrawal /under
drawal without the ICCP data and the huge commercial implications. Besides that
many 765 kV Powergrid’s SS SCADA data is not good and affecting the state CGS
drawals. In view of the above, the following was suggested:
To discuss SCADA/Communication issues of Powergrid SR1 & SR2, states and also
SRLDC’s redundancy readiness of ICCP servers/Routers.
To discuss the issue that the Payments may be linked to availability and healthiness
of Powergrid’s communication network. SRLDC’s ICCP equipment and percentage
availability of ICCP data to the State.
TSTRANSCO vide letter dated 27.02.2018 (Annexure-VI) had expressed concern on
the ICCP data not being stable for most of the period during the month. On 7th & 8th
MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 3
February 2018, ICCP data failed due to expiry of SSL certificates of secure ICCP.
Again on 23.02.2018, the ICCP data failed from 22:00 hrs and it was restored only at
08:30 hrs of 24.02.2018. The economical load dispatch operation solely lies on ICCP
data in real time including safety and security of the grid. It was pointed out that the
loss of real time data for long time periods affects the CGS drawal severely and
attracts penalties. To avoid such problems in future there was a necessity to discuss
these situations in detail and make a resolution.
SRLDC observation on 07.02.2018 ICCP failure are given below:
Secure ICCP certificates expired throughout SCADA systems maintained by M/s
GE (T & D) in India.
ICCP data exchange unaffected for Karnataka & TN SLDCs (connected on non
secure ICCP)
SRLDC SCADA team tried to resolve issue with known methods
Upon non restoration of ICCP data exchange to/from affected SLDCs, issue was
reported to GE Noida and ticket open for problem resolution by SRLDC.
ICCP data exchange restored in morning hours starting with AP, Kerala,
Telangana & Puducherry.
Data restored in non secure mode.
ICCP certificates were restored by GE on 15th Feb 2018.
SRLDC suggested that affected SLDCs need to raise issue with M/s GE (T&D)
directly for timely resolution and coordinate for early resolution. Further all were
requested to analyse the problem before arriving at any conclusion.
SRLDC observation on 23rd Feb 2018 (22:30 hrs onwards) ICCP failure is given below:
ICCP communication with AP & TS went down.
Multiple fibre cuts in CTU system were reported.
PGCIL was alerted in real time.
Links were restored in morning hours (around 08:30 Hrs), followed by ICCP
restoration.
Subsequently following was taken up jointly by SRLDC & APSLDC/TSSLDC :
AP & TS SCADA team informed SRLDC that in certain situations (ICCP server
2 being main at SRLDC, and ICCP server 1 being main at AP/TS end), ICCP
data is not reporting even when the ICCP links are up.
AP & TS SCADA teams claimed problem due to misconfiguration at SRLDC
servers/routers/firewalls.
FPRs raised by SRLDC for resolution of TS & AP ICCP issues.
Issue resolved for both AP & TS ICCP by attending erroneous entries in AP
firewall & TS routers.
Subsequently following was taken up jointly by SRLDC & APSLDC/TSSLDC for
ICCP Resolution :
AP ICCP Resolution steps:
AP internal firewall mis-configuration rectified.
AP ICCP application error rectified.
MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 4
AP SISCO security outdated certificate configured with new certificates.
TS ICCP Resolution steps:
Encapsulation PPP enable in TS WR1 router for serial link interface.
Deleted all the static routing from both TS WR routers and TS ICCP servers
which was enforced for SRLDC ICCP LAN.
APTRANSCO expressed concern on frequent ICCP failures recently. This failure was
likely due to communication link and other issues. Due to high CGS schedule, ICCP
data is very important for state. Due to failure of ICCP, AP SLDC control room is not
aware of its OD/UD. After secure license updation one of ICCP server it was not able
to communicate on ICCP protocol. SCADA personnel should be available in shifts to
solve any issue related to SCADA. APTRANSCO requested to waive off penalty for
UD/OD, which happened due to ICCP failure. It was pointed out that many
communication channel has been configured for ICCP data communication between
AP SLDC and SRLDC. M/s Tejas Engineers have configured 6 No of communication
channel with same ID, which creates problem in troubleshooting after link outage.
Many changes have been done in communication network by Powergrid (M/s Tejas),
without information of AP Telecom. Expiry date of all licenses to be taken from M/s
GE. APTRANSCO requested to check the healthiness of server and link redundancy
for ICCP data transfer. Recently complete redundancy test was done for server,
Router, link and found to be OK.
TSTRANSCO expressed similar concern as stated by APTRANSCO. ICCP failure
outage duration on 23.02.2018 was longer, which causes penalty for states due to
OD/UD.
KPTCL informed that ICCP with SRLDC failed on 01.02.2018. On prima facie analysis
it was analyzed that patch updation was the issue. Subsequently on further analysis it
was found that number of points being handled in ICCP database was huge and it had
created problem. Matter was resolved on priority basis with the help of OEM, M/s
ABB Engineers. During above outage real time scada data available on web server
provided by SRLDC helped KPTCL in grid operation.
TANTRANSCO requested to waive off penalty for UD/OD, which happened due to
ICCP failure.
APTRANSCO / TSTRANSCO stressed on 24 hrs shift at SRLDC /SR-I / SR-II for
SCADA and communication issues in similar lines to the SLDCs. SRLDC pointed out
that 24 hrs support was being extended from office and through remote login.
It was confirmed that there were some mis-configuration at AP int-firewall. It was
rectified by M/s GE team upon ticket raised for rectification.
It was confirmed that there were some mis-configuration at TS ICCP router. The same
was attended by M/s GE Engineers on FPR raised by SRLDC.
It was noted that ICCP failure on 12.03.2018 was due to power supply interruption of
equipment at Hoody substation of KPTCL.
SRLDC stated that data is extremely important for reliable and secure grid operation
and all the concerned entities needed to resolve the issue at their end and also co-
ordinate with other entities.
MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 5
SRLDC suggested AP & TS to actively raise all SCADA/communication related
complain to GE/CTU/STU and not leaving the matter only to SRLDC for resolution.
SR-I & SR-II pointed out that all ICCP link failures may not be due to communication.
ICCP problem occurred on 23.02.2018 was not due to fibre cut in Munirabad/Raichur
area.
After discussion the following was agreed:
Expiry date of all licenses would be furnished by M/s GE to the respective Control
Centers. It was noted that respective Control Center would ensure that M/s GE take
pro-active action to renew the licenses before the expiry. As pointed out by KPTCL
that there should not be any license renewal issues during the project cycle of the
project and it needed to be ensured by Control Centers (through M/s GE).
License for external firewall is going to expire in three months. M/s GE has to
update the certificates in ext firewall of AP. M/s GE to take necessary action.
M/s GE site engineer at respective Control Centers to escalate the matter to BO-
Noida, if they are unable the resolve the issue at their level. Further, inter-action
between M/s GE site engineers at SLDCs and M/s GE site engineers at SRLDC
needs to be strengthened.
Changeover to ICCP data communication through Backup SRLDC could be
explored during such conditions which would reduce data outage duration.
All the ICCP links (Main/Back up CC of SLDCs to Main/Back up CC of SRLDC) to
be checked daily to ensure healthiness.
All SLDCs to carry out HA test once in every month to a check
SCADA/Communication system healthiness.
All states to actively raise all SCADA/communication related complain to M/s
GE/CTU (M/s Tejas) /STU and not leaving the matter only to SRLDC for
resolution.
It was noted there was no provision for DSM waiver during ICCP failure as per
extant Regulations.
Updated communication network diagram to be submitted by SR-I and SR-II to all
SLDCs & SRLDC. Further any changes done in the communication network needed
to be informed to all.
3.2 Frequent data outage (Central Sector & State Sector)
SRLDC presented the detail of data outage from 01.01.2018 to 12.03.2018
(Annexure-VII). It was pointed out that most of the outage was due to
communication failure.
SRLDC requested Powergrid to provide all fibre cut/equipment outage information in
real time.
TSTRANSCO expressed concern that due to frequent failure of Powergrid station
(Nizamabad, Warangal etc.), TS drawl gets affected and real time CGS drawl becomes
erratic.
SR-I & SR-II stated that all data outages may not be due to communication issues.
MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 6
SR-I & SR-II stated that DCPC healthiness also needed to be checked. For this SRLDC
replied that all DCPC patches etc. has been done and at present one of DCPC at
Narendra is having the problem, which is being looked by M/s GE for faster resolution
After deliberations the following was agreed:
PGCIL to share their observations for communication failure on the instances of
data outage beyond 8 hrs (refer annexure) put up by SRLDC.
PGCIL to share all fibre cut/equipment outage information in real time.
It was suggested to take all measure to minimize down time of data
3.3 KKNPP analog data not reporting
It was noted that presently Analog data issue has been resolved after Analog Card (as
supplied by PGCIL SR2) replacement and is under observation. The forum opined
that if the problem persists, it would be prudent for KKNPP to procure a new RTU.
3.4 Data outage after commissioning
In earlier meeting it was agreed to cross check with site daily values for newly
commissioned elements for a period of two week duration from commissioning.
3.5 Data discrepancy
The latest data discrepancy list compiled by SRLDC was presented in the forum
(Annexure-VIII).
The forum opined to delete the points which are not feasible to be telemetered.
SRLDC agreed to look into the issue.
3.5 RTU/SAS Maintenance/Upgradation
All states/Powergrid/CGS to explore option of replacement of old RTUs and AMC for
SAS, as getting spare is difficult for old RTU. The issue has already been discussed
earlier and AMC need to be provided for all RTU/SAS gateways by CTU/STU.
A life/functional assessment report and action plan by PGCIL/ISGS on old RTUs
would be prepared and submitted to SRLDC/SRPC.
3.6 Backup Control Centre
Backup SCADA functionality testing frequency needs to be done once in 6
months to ensure high availability of data/preparedness to operate from Back up CC
in event of any contingency. Compliance report may be furnished to SRLDC.
Periodical testing was being carried out by SRLDC and KSEBL. TN stated that
communication network is being strengthened to Backup CC. Once same is
completed, TN will do the testing.
APTRANSCO informed that communication link has to be established from AP CC at
Vijaywada to Back Up CC of SRLDC at Delhi. Powergrid & AP confirmed to establish
the same by 16.03.2018.
MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 7
PGCIL would be floating new combined Open Tender for Backup SLDCs for AP
(Tirupati), TS (Warangal) along-with few other states.
After discussions, it was noted that backup control centre establishment for AP & TS
need to be expedited.
3.7 ICCP with NTAMC
PGCIL confirmed that as pointed out in 32nd TCC/33rd SRPC meeting, due to cyber
security concerns data would not be extended to SRLDC from NTAMC ICCP.
It was noted that functional requirement of the requisite data by SRLDC would
remain, which needed to be ensured by PGCIL. SRLDC would furnish the list of
telemetry points that have to be extended to PGCIL within 10 days, and within 1
month PGCIL would come out with a detailed plan to be completed within 6 months
(October 2018)
3.8 Modification in EMS package – TSTRANSCO
Earlier TSTRANSCO informed that they had received estimate of Rs 47 lakhs and the
contract was being firmed up.
TSTRANSCO requested PGCIL to assist in finalizing the EMS contract through M/s
GE.
SR-I stated that TSTRANSCO to interact directly with M/s GE without PGCIL
involvement.
SRPC requested that PGCIL to render assistance to TSTRANSCO if possible.
3.9 Status review of compliance of approved protocol of handling scada & communication issues (21.03.2016 communication meeting)
In the earlier meeting, it had been noted that RTU not reporting/data failure would be
communicated to the concerned entity/user with a copy to PGCIL. User would take
lead role in getting the issue resolved. User/SRLDC would ensure necessary
verification at their ends as explained in the training. Parallely, PGCIL would look into
communication issues. User/SRLDC would appraise PGCIL about the verification at
their end for PGCIL to sort out the communication issues. Official email ID for
SCADA issues to be created for each entity/User involved for easier communication
and proper delivery of information.
SRLDC pointed out that there were instances of delay in attending the faults wrt the
time schedule and requested all the constituents to attend to the faults within the
agreed time schedules.
It was noted that if the issue is not getting resolved within 8 hrs (3 hrs for ICCP) the
following action would be taken:
Shri A P Gangadharan, GM(AM), SR-II and Shri S Ravi, GM(AM), SR-I would be
informed for necessary follow up.
The issue would be escalated with M/s G E (both by SLDCs and SRLDC) and with
M/s Tejas (by SR-I & SR-II).
MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 8
In case the issue was as not resolved within 24 hrs physical involvement of OEM
(M/s GE and M/s Tejas) would be ensured through effective coordination by user
/ PGCIL/ SRLDC.
Sustained efforts/involvement of all the concerned entities to be ensured till the
issue is resolved
In case of hardware replacement / major rectification it would be completed by
72 hrs.
3.10 Sparing of Fibre in Central Sector Project
TANTRANSCO again requested for sparing of fibre in Central Sector Project.
It was noted that the issue was already deliberated in the 30th TCC/31st SRPC
meetings held on 24/25.02.2017 wherein, the following was noted:
PGCIL had informed that in most of the FO, there were no spare fibres available. Six
fibres are available for grid management purposes. The balance 18 fibres in the state
or central sector are left to the entities on how to utilize. As for Central sector, 18
fibres are being leased out to M/s Powertel as per CERC formula which covered
issues like sparing of fibres, cost sharing (50% being shared by M/s Powertel), ROW
compensation (Rs.3000/km) etc. For grid operation purpose, bandwidth on 6 fibres
could be given to the states. M/s Powertel may not take all the 18 fibres in some of
the lines but those fibres are being leased out anywhere.
Keeping in view of the non availability of spare fibres in most of FO links and the
approved MoU with M/s Powertel the scheme of allocation the fibres was dropped.
SR-II stated that request for sparing of fibre in new FO links could be communicated
for consideration.
3.11 VC issues
TSTRANSCO pointed out that Video Conference issue at TSSLDC. There was
disturbance in Video Conference at TS end when connected to SRLDC.
SRLDC confirmed as per AMC Engineer no issue was observed at SRLDC end and
software upgradation is pending at TS end.
After deliberations, TSSLDC and SRLDC were requested to coordinate and sort out
the issue.
3.12 Talcher RTU
It was noted that earlier proposed meeting was not held due to overlapping holidays.
It was noted that a meeting with NTPC Talcher, SR2 ULDC and SRLDC for the same
in is proposed 3rd/4th week of March 2018. SR-II was kindly requested to address the
communication issue between Talcher-II & HVDC. NTPC as a User needed to be
proactive in this.
4. SE (O), SRPC thanked all for the interactive deliberations and requested the Entities for
concerted/sustained action to close the issues highlighted in the Meeting.