Upload
hoangdung
View
214
Download
0
Embed Size (px)
Citation preview
Operation & Maintenance of Telecom Networks and Associated Information Systems
in the Electrical Power Utility
Mehrdad MESBAH Tutorial, Cigre D2 Colloquium, Lima, PERU
October 2015
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 1
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
2 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
1 - Introduction
• How do Electrical Power Utilities operate and maintain their telecom infrastructures and services today? Which issues are they facing?
• How are the associated resources, processes and tools evolving to assure adequacy in the evolving communication context?
• Cigre WGD2.33 : “Operation & Maintenance of Telecom Networks and Associated Information Systems in the Electrical Power Utility”
• launched in June 2012 (18 members, 12 Utilities)
• Survey over 10 T&D Utilities worldwide
• Technical Brochure published as TB588 in July 2014
– Current practices, issues of concern, evolution plans
– Mainstream trends, guidelines, best practices, standards
– Extending from previous Cigre WGs and Reports
3 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
O&M – Process, Organization and Tools
• Major adjustments are required for responding to:
• A sharp increase in the size and scope of the telecom network • Significant increase in the number of services or of service users • Major change in Utility organization (e.g. separation of telecom entity) • Major change in service commitments, in liability, or 3rd party involvement • Sharp change in technical staff capability (new technos, retirements, …) • Deployment of process automation tools and change of roles and tasks
4 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Management
Organization
Management
Process
Management
Tools
Management
Scope
Management
Organization
Management
Process
Management
Tools
Management
Scope
Management
Organization
Management
Process
Management
Tools
Management
Scope
O&M is often the forgotten component of network transformation projects
Reasons for Modelling the O&M Process
O&M Process
Modeling
Assessment of
Service Provisioning
New Rules and New
Actors
O&M Cost Estimation& Justification
New Tools & Process
Automation
New Ambitions
& Goals
Worforce Constraints
Increased Scope &
Change of Scale
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 5
Reference framework for O&M processes and related KPIs
New applications, Wider coverage Larger impact of network faults, Multiplicity of technologies, Multiplicity of skills & tools
Regulatory obligations on (Grid reliability, security and efficiency) Contracted 3rd party interventions
Smaller staff, higher turnover and new technical skills
New liabilities and commitments (SLA/KPI) in revenue generating telecom services
Growing weight of O&M in comparison to the cost of technology imply accurate
Justify tool investments Define the extent of process automation in the management of telecoms
Elaborating a Model through EPU Survey
• Statistical Data Analysis – Collect Extensive Data, Correlate, Elaborate Rules on Behavior for each aspect
– In our subject data sample is not large enough
– Special cases and contexts are numerous – Not modeling a “single box”
– Conclusions could be either Trivial or Erroneous!
• Case Studies – Select some Utilities, Observe and Elaborate Rules on Behavior in each Utility
– Would not show which rules are related to which specificities
• Transverse Assessment over selected issues (e.g. Fault Management)
– Allows the comparison of approach, work volume, staffing, automated versus manual process, tools employed, etc. investigating the diversity with respect to Utility specificities
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 8
EPU-A EPU-B EPU-C
Criterion 1
Criterion N
.
.
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
9 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
2 - Which Electrical Power Utility (and Which Requirements?)
Network Perimeter and O&M Scope • EPU’s role & footprint? Spread of the communication network?
• Wide Area? LAN? Substation /Control Platform / Tech Office?, Customer?, Enterprise Network? Control Room IT, Enterprise IT, …
• Electronics, Cable infrastructure, Towers and Antennas, Intra-site cabling?
Provisioning/Delivery Mode • Telecom as part of the Grid Operation
• Telecom as separate service providing entity
• Telecom as Contracted Activity or as Procured Service
Size & Scale of O&M Organization
• Number of Sites, Users and Services
• Number of technical staff in Technical Office/Control Platform
• Number of Field Workers (dedicated or shared asset maintenance?)
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 10
• No unique/typical model for all Electrical Power Utility Telecom nets • Need to set different modelling parameters for this investigation
Service User
Service Provider
P&C Scada
Metering Asset Mngt Surveillance
T&D Grid WAN/LAN
Telecom/Platform OT/IT
Communication domains in the EPU defining telecom O&M perimeter
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Intra-plant Local Net
Intra-plant Local Net
Platform Local Net
Field Plants Control Platforms
Platform Local Net
Core Telecom Network
Customer Local Net
Customer Premises
Private or Public IP Net (wired or wireless)
Customer Backhaul
Wide Area Cables & Telecom System
Control Room IT
Substation & Plant IED
Meters & Appliances
BD : Boundary Device (Server, Concentrator or Gateway device)
Modem & Comms device
Concentrator & Comms device
Control Centre LANs
Substation & Power Plant LAN
switches & cabling
BD BD BD BD
BD BD
Office
Local Net Network Device
& Cabling
WAN Access
Enterprise IT
Private or Public IP Net
(VPNs)
Corporate Network
BD BD
BD
Customer Access
(Last Mile) BD
U-Office & Data Centre
BD
Inter-Platform Network
Grid Inter-plant
Net
11
12 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Corporate Activities
Grid Operations
EPU
Telecom
A
Telecom is part of the operational activity. Corporate entity provisions telecom services separately
Corporate Activities
Grid Operations
EPU
Telecom Services
B
Common Telecom (& IT) Services for both Corporate and Operational Applications.
Corporate Activities
Grid Operations
Power Corporation
Telecom Service
Provider
C
EPU
TSP is a sister company to the EPU, providing services exclusively (or in priority) for the Power System
Corporate Activities
Grid Operations
EPU
Telecom Service
Contractor
D
Telecom Assets
EPU procures its telecom assets but operates them using an external Service Contractor
Corporate Activities
Grid Operations
EPU
Telecom Service
Provider
E
Telecom services are procured under SLA by a Telecom Service Provider delivering services to many customers.
EPU Service Provisioning/Delivery Modes
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process - Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
13 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
3- Evolution of O&M Scope, Process and Capabilities
• Utility Survey = How the telecom network is operated today (static vision)
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 14
How is the O&M scope changing? How can this new scope be delivered?
New Processes
New Tools
Organizational Adjustments
New Capabilities
New Services
New Users
New Sites
New Assets
Commitments
Evolution of O&M Scope and Process
15
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Technologies / infrastructure
• Broadband media & Packet-based technologies (Ethernet / IP) • Acceleration of technological changes & shorter life-cycles • Network maintenance through equipment replacement versus repair • Permanent planning and continuous transformation • Training cycles on new technologies • Deployment of tools driving process change
Services & Site Coverage
• Dedicated inter-utility extranets • Operation-support/enterprise intranets • Communication sites (e.g. distribution network) • “Smart” applications (Remote access/monitoring, “office-to-field” ...)
Human Resources & their localization
• Ageing workforce & massive retirements • continued maintenance of older technos • Re-allocation of skills & expertise • Remote support for field intervention
Service Users & Providers
• Migration of service delivery model • SLA-based relationships
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Maturity Model for Telecom Service Delivery
Maturity Scale 1 - Service Delivery Scope
Service level defined implicitly
by applications
QoS specified explicitly but
No SLA
Liable to SLA Must produce formal proof
Formal SLA No systematic measurement
Supplier Helpdesks &
Warranties only
Support Contracts for sub-systems
Full Service Delivery
Contracts
Field Maintenance
Contracts
Procure Telecom Services
Maturity Scale 2 – Service Provisioning Scope
Overall telecom budget without
repartition
Contributive repartition of
telecom budget
Service price established as
Service Catalog
Service costs estimated but not recovered
Maturity Scale 3 – Business Model and Cost Recovery
Maturity Scale 4 – Service Liability
Service costs recovered at
no profit
Operational Services Only
Internal Multi-user
Services
Int. Services & Bandwidth
Wholesale
Internal Services &
Ext. Leasing
Int. Services & Ext. User Services
Formal SLA & monitored QoS
16
Building New Capabilities – Migration Planning
• How far into O&M evolution and Maturity Scale ?
• Governed by service requirements, company size and policy, cost, …
• O&M migration plan with intermediate steps
• Prevent loss of control and Quality of Service in the “learning process”
• Allow gradual and incremental investment effort
• Must accompany network/service migration
17
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Present State
Target Phase 1
Maturity X
Maturity Y Target
Phase 2
Capability must be planned and adjusted to match new network /service delivery scope
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
18 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
4- Modelling the Telecom O&M Process & Organization
• ITIL : Framework for internal processes necessary for running an IT infrastructure (employed in many EPU IT organizations)
– Classifies tasks and processes into groups and specifies interactions between them
– Relevant for IT management in any enterprise context including that of telecom service delivery organization
• TMF NGOSS (New Gen. Operation Support System) – Specifies a Telecom Operation model called e-TOM for Business processes of a Telecom
Service Provider (used by Cigre WGD2.26 to define a Utility sub-set called u-TOM)
• ITU-T M.3010 Logical Layered Architecture (LLA) – Extends the visibility of the management system through different layers of abstraction
(Element, Network, Service, Business)
• ITU-T M.3400 FCAPS model – Fault, Configuration, Accounting, Performance, Security
– Functional partitioning mainly related to domains of telecom infrastructure management (but can be applied to some extent to processes)
19 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Modelling the Telecom Management Process
• 1st order model : Upstream Management (Planning, Deployment, Transformation, Policy) as opposed to Day-to-day current operations
• O&M is basically the Current Operations, although capability building for O&M, and definition of its scope is in the Upstream Management.
• Feedback from Current Operation allows adjustments and upgrades
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 20
Current Operations
Upstream Management
Organization, Process,
Network Assets, Service Contracts,
Service Catalog
Operational Data,
Assessments,
Audit Reports,
Asset Requests
Policy Definition
Business Planning
Deployment
Upgrade & Migration
Business Development
Customer Relations
Service Management
Infrastructure Management
Provider Management
Cigre D2.26 Utility Telecom Operations Map
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Bu
ild S
trat
egy
Bu
ild C
apab
ility
Bu
ild S
erv
ice
Off
er
Customer / User Relationship
Customer / User
Provider / Contractor
Provider / Contractor Relationship
Service Management
Resource Management
Security, BCP, Safety, Skill Mgt
Strategy & Planning
Business Development
Service Portfolio Evolution
Service Migration Planning Deploy &
Adjust
Operations
Upstream Management
Enterprise Processes
21
Operation Management Models and Segments
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
User Relations Management
Communication Service Management
Service Provider & Contractor Management
Network Mgt.
Element Mgt.
Bu
sin
ess
Mgt
.
Service Mgt.
Fault Mgt.
Config. Mgt.
Account Mgt.
Perf. Mgt.
Security Mgt.
TMN ENSB Hierarchical Management Layers
CIGRE D2.26 uTOM Telecom Operation Map
(TMF eTOM)
ITU-T FCAPS model
User Service Desk User Notification User Dashboards
User SLA Monitoring
Incident Management Fault Management
Config. Mgt. & Inventory
Performance Monitoring
Contractor SLA Monitoring
O&M Communications
Op. Site & Facility Access
Bandwidth Policing (CIR/PIR) Infrastructure Management
22
D2.33 Assessment Structure
“Current Operations” Process for Telecom Service Delivery and Provision
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 23
Fulfillment (Setting up)
Assurance (Delivery) Running
Support (Enquiry, Supervision
& Maintenance)
Accounting (Metering, Cost
Repartition, Invoicing & Settlement)
User/Customer User Order Handling
User Change Handling
User Problem Handling User SLA Management Customer Relationship
Management
Service Enquiry Desk (User Technical Support)
Customer/User Entity Invoicing & Settlement
Communication Service
Service Configuration & Activation
Incident Management
Service Quality Mgt
Service Inventory Mgt
Service Change Management
Service Policing & Usage Metering
Network Infrastructure &
Resources
Bandwidth & Capacity
Provisioning
Network Problem Mgt
Network Perf. Management
Disaster Recovery
Net. Configuration & Change Fault Management
Network Maintenance Asset Lifecycle & Spare Mgt Management Tools Support
Estimate running cost of network infrastructure
(+ tools)
External Providers & Contractors
EP&C SLA Setting
& Adjustment
EP&C Performance Mgt (SLA Monitoring)
EP&C Problem Reporting (EP&C Relationship Mgt.)
EP&C Support Management (Site Access Permits,
Safety & Certification)
EP&C Cost Assessment & Invoice Settlements
O&M Main Process Interactions
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
24
Fault Management
Incident Management
Performance Monitoring
Service Mngt. & User Relations
Config & Change Mngt
Detect & Localize Anomaly
Fault Impact & Service Restore
Notification
Detect Performance & SLA Anomalies
Network Data
Network Data Performance
Dashboard
User-reported Anomaly
Service User
Support Service
Providers
Update Network
Vision
Network Change Report
Frequency of Recurrent O&M Actions in utility telecom
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
25
1/Hour 1/Day 1/Week 1/Month 1/Year
Creation of New Connections
Receive & Acknowledge Alarms
Open Incidents & Assign Interventions
Performance Checks
Preventive Maintenance (critical or old equipment, ventilation filters, power supply batteries)
Remote Configuration
Continuously Polled
Automatic Scheduled
Routine Maintenance
Transformation Works
Generic Management Organization & Process for a typical Operational Telecom Network
Service User
Operational Depts.
Telecom Network Centre Reg. Maintenance A
Back-up Net. Centre Reg. Maintenance B
Service User Operational Depts.
Service Notification Quality Reporting
Service Request Service Anomalies
Recurrent network anomalies &
Intervention Reports
Network outages with user service impact
User identified service anomalies
Recurrent service anomalies
Network Configuration Service Outage Planning
EMS/SCADA Facility Management Protection & Control Asset Management, …
Network Infrastructure (Region A & B)
Network Planning & Transformation
Service Management & User Relations
Use
r N
etw
ork
N
etw
ork
Man
agem
ent
Network
Supervisor
Maintenance
Engineers &
Support
Service Managers
Telecom Transformation
Engineers
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
26
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
27 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
5 – Fault & Anomalies Management
• Fault : failure (broken part or weakness) in a machine or system
• Anomaly : different from what is usual, or not in agreement with something else and therefore not satisfactory
Cambridge English Dictionary
Fault is signaled as a device alarm (either the faulty device, or one of its peers, or indirectly by a device overlaying the service delivered by the faulty one)
Anomaly is detected through monitoring of equipment or system performances or reported by a user
28 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Fault & Anomalies Management Platform
Performance Monitoring
Telecom Device
User Service Desk
Performance threshold detection
Alarm Monitoring
Anomaly Reporting
• Fault Detection • External Alarm Collection/Monitoring System • Vendor-specific NMS • Unified Multi-purpose Management Platform (e.g. SNMP) • Manager-of-managers (e.g. standard connections to Vendor-specific NMS)
Fault & Anomalies Management
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 29
• Fault Notification & Reporting • Updating of network and asset inventories • Documenting of the incident, performed and proposed corrective action
• Fault Localization • Associate Network configuration and Event data • Determine location & cause (RCA) over multiple layers of fault indications
• Fault Recovery & Restoration Reporting • Field intervention in a great majority of cases (need remote support) • May be made less urgent through remote action
• Fault Diagnostics • In-depth examination of the anomalous device or connection • Performed through vendor-specific NMS through specific skills
Incid
en
t Man
ageme
nt P
rocess (in
parallel)
Fault Detection Systems
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Read values and detect threshold passing
Detect external alarm conditions
Field Communication or
Management Device
Detect device faults
Fault and Event Management
Platform
Management Data
Communication
Network
Network Element
Network Element
Network Element
External Monitoring
Device
Remote Management
Device
Vendor-specific
Management
Multi-purpose Management Platform
Network Element
SCADA Remote Terminal
Unit
Power System SCADA
Management Data Communication Network
Northbound Interface SNMP
SNMP or Proprietary
IEC101 or 104
Proprietary
Alarm contact
SNMP SNMP or
Proprietary
Value measurements
30
General Scheme
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
31 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
6 – Incident Management, Work Assignment and Escalation
• Register faults and anomalies (open incident),
• Allocate adequate resources to their resolution (assign to a person/dept.),
• Coordinate and keep track of the performed actions (escalate / document),
• Decide upon satisfactory resolution and assure reporting (close incident)
• Generate statistics on frequency of occurrence and resolution time for improving the network or the process (weekly, monthly, …) – mainly used for continuous improvement
32 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
• Majority of EPUs have some tools for managing incidents – often applied only to external suppliers, contractors, …
• Full scale enterprise tools (e.g. SAP) are cumbersome and time-consuming while a fully informal non-automated process is insufficient
• Dead time due to responsibility borders is reported as a major issue (up to 12 hours)
• Need precise roles partitioning – e.g. between user /core telecom networks
dedicated infrastructure
in-house O&M workforce
Merged treatment for Service and Network incidents
External network or workforce
Need in-house service incident management
6 – Incident Management – Interaction with O&M Contractor
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
33
Customer / User Relationship
Customer / User
Provider / Contractor Relationship
Service Management
Customer / User Relationship
Service Management
Resource Management
External O&M Contractor
In-house Service Management
(Open incident & Assign to Contractor)
Service Contract (SLA)
Operational Services (SLA) (Service Anomalies Reporting)
No
tify Service Resto
re
Close ticket
No
tify Users
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
34 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
7 – Service Management & User Relations
• Service Desk – interface point for User Relations (telephone, e-mail, or web-based automated desk for larger companies with merged telecom & IT activities)
• Creation, Modification and Deletion Requests (discussed in Change Management)
• User Detected Anomalies (widely variable across surveyed Utilities)
• Incident tickets opened by O&M staff (not by the user)
• Comprehend User issue, follow and notify its resolution, generate report (automated ?)
• Quality & SLA Monitoring – continuous monitoring of SLA metrics • Monitoring through User applications (customer feedback on perceived quality), e.g. SCADA
• Monitoring through Impact Analysis – “Backward Root Cause Analysis” (e.g. Loss of Back-up)
• User Notification – alert a service user of potential impact on his services of a network fault detected by the provider
• Can be manual or automated (upon service impact detection)
• In some cases integrated into reporting and performed only upon recovery
• User Dashboard – provide permanently an account of quality of service • Service availability, outage distribution, throughput, time latency, problem resolution upon
incident reporting, restoration upon fault, … for the services delivered to the user
• Usage Measurement and Invoicing (e.g. Service Metering) • Cost partitioning between user entities – not yet common but potentially necessary (IT/OT)
35 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
7 – Example of a User Service Dashboard
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 36
Distribution of Interruptions
Not Resolved at Site
Short Spurious Faults
Long Spurious Faults
Resolved at Site
Remotely Resolved
Statistical Availability for Services of a certain category delivered to a same User
7 – Service Management & User Relations
• CIGRE Utility Survey suggests that Service Reports, Statistics and Trends are at present mostly generated manually using data and reports from various management systems and using office tools (e.g. MS Excel)
• These are, in the majority of cases, used for internal purpose (service provider) unless external contractors are involved
• At present only monthly reports are published mainly to demonstrate conformity “in the wide sense”
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 37
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
38 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
8 – Configuration & Change Management
Process of documenting, adding/deleting, and modifying network assets, connections, and management tools in the communication system
39 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Network and Service Inventory
User Order Handling Service Activation
Configuring Network Assets
Capacity Management
Change Management
O&M IT/IS
Platform Management
Asset Life-cycle & Spares Management
Co
nfigu
ration
Man
agemen
t D
atabase (C
MD
B)
8 – Configuration Management Database (CMDB)
• Network and Service Inventory – Fundamental element for operating and maintaining the telecom system
– Accurate, structured, coherent, unique and up-to-date data on equipment and connections
– Information must be rapidly accessible to all O&M players
– Inventory information is required for
• Problem management – need knowledge of equipment types, relationship between alarm generating assets for Root Cause Analysis
• Determining Service Impacts of network faults – need knowledge of service-to-network mappings and dependencies
• Service restoration and circuit provisioning (i.e. configuration changes) – need knowledge of existing configurations
40 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
8 – Configuration Management Database (CMDB)
• Composed of a number of different data inventories:
– End-user entity inventory – contact coordinates for notification and reporting
– SLA inventory – service catalog containing specifications (availability, time/error performance, interface, throughput, security, outage and restoration time …)
– Service inventory – user-referenced connections (e.g. SCADA circuits) with corresponding mapping to logical resources, SLA elements, configuration, test records, etc. Provide service-to-network mapping for service impact analysis
– Logical Resource inventory – static and dynamic logical connections. Provide network-facing view of services (VLAN numbers, …)
– Physical Resource inventory – network infrastructure, cables, equipment, …
– Software/Firmware inventory – tools and platforms, licenses, …
– Spare Parts inventory – modules/devices in warehouse, ordered or in reparation
– Supplier/Partner Contract inventory – procured services, provider SLAs, …
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 41
8 – Configuration Management Database (CMDB) Some important considerations
• The data can be integrated into a single information system or maintained as distinct data structures (tables, lists, drawings, …) with or without interconnections
• Some can be integrated into an integrated management platform
– Assets, Connections, Notification Coordinates, SLA elements …
• Data discovery can be automated when assets are “configuration-aware”
• Operator-grade tools are over-dimensioned and expensive (highly automated)
• Interconnected or integrated data structures are used for automating highly recurrent processes or manipulating very high volumes of data
• But managing interconnected data and automated processes can increase considerably the complexity resulting in substantial cost and effort
– Trade-off between laborious (manual) and complex (automated) process
• Inventory Information Model – How much information needs to be interconnected? Which further data can reside in separate silos of information (e.g. vendor-specific management systems and tools)
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 42
8 – Configuration & Change Management
• User Order Handling and Service Activation
– Initializing a new connection of a pre-existing type or modifying an existing connection (e.g. increase bandwidth) upon a user request
– Assure data coherence and coordinate provider & user on service commissioning
• Configuring Network Assets & Managing Resource Capacities
– Not a highly recurrent activity in the power utility (except during transformations)
– For quasi-totality of survey cases, performed through vendor-specific tools (static allocation) and off-the-shelf enterprise network tools (dynamic allocation)
– Commonly relies upon upload of asset settings (however this cannot check what the setting should be from a network design perspective – does the equipment setting correspond to intended network design?)
– Remote secure access for configuring network equipment
– Back-up configuration files generation and storage in the Asset Inventory
– Keep track of resource usage and capacity (bandwidth, ports, boards, fibers …)
• Change Management
– Document asset , service and network changes in coordination with Field Intervention and Operation Support staff.
43 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
8 – Configuration & Change Management Tools, Platforms, Spares and Life-cycle
• O&M IT/IS Platform Management (ITIL Service Management applied to telecom management platforms)
– Delivery of IT services required for the fulfillment of Telecom Provider’s Business Processes
– Software release & License Management – Service Support Contracts for constituent Applications and Firmware – Security Patch Management – Software Documentation & IT Configuration Management – Telecom O&M or IT governance?
– Keeping track and maintaining operational tools, crafts and instrumentation
• Asset Life-cycle and Spares Management
– Obsolescence cycle of technology is increasingly short
– Keep continuous awareness for network assets of commercial availability for new projects, for replacements and upgrades, for spares & support …
– Keep inventory on spare modules, non-allocated equipment, …
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 44
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
45 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
9 – Quality of Service & Performance Monitoring
• Monitoring of end-to-end network performance against pre-set SLA values
– by user application (e.g. SCADA)
– by external measuring system connected at Service Access Points
– measured through agents in telecom device and stored in a MIB for retrieval by a remote monitoring platform
• Error measurements in TDM systems – incorporated into standard equipment
• Delay, loss and delay variation in Packet Switched networks
– SDH-like multi-layer performance management (i.e. Regeneration Section Overhead, Multiplex Section Overhead, etc.)
– Exchange OAM packets at each level between Maintenance Edge Points (MEP)
– Carrier-grade Ethernet and MPLS-TP
– IEEE 802.3ah at link-level (i.e. between consecutive switches)
– ITU-T Y.1731/IEEE 802.1ag at service level (i.e. end-to-end Ethernet service)
• Special cases: Packet-over-TDM and TDM-over-Packet
• Performance monitoring is to feed User Service Dashboards, SLA monitoring and fault supervision/notification
46 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Operation & Maintenance of Telecom Networks in the EPU
1. Introduction
2. Which Electrical Power Utility (and Which Requirements?)
3. Evolution of Scope and Process – Adjusting Capabilities
4. Modelling the O&M Process and Organization
5. Managing Faults and Anomalies
6. Incident Management, Work Assignment and Escalation
7. Service Management & User Relations
8. Configuration & Change Management
9. Quality of Service & Performance Monitoring
10. Telecom O&M Communications – Field Worker Support
47 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
10 – Telecom O&M Communications & Field Worker Support
• Telecom O&M actors require information exchange to carry out the described O&M tasks and processes. Some network capacity is allocated to these “own applications” as a specific IP network (i.e. one of the network’s multiple users)
• Device-to-Platform – Telecom Asset Monitoring and Fault Detection, Security Barriers reporting to a Security Operation Centre (SOC)
• Platform-to-platform – Connect Vendor-specific management systems and asset inventories to integrated management platforms, User Authentication through remote security server (e.g. RADIUS) …
• Device-to-human – Remote access to assets for diagnostics, configuration and setting (tele-maintenance), Video control of assets and facilities
• Human-to-platform – Remote access to telecom management platforms, Automated notification (to users and O&M workforce) of faults and service impacts
• Human-to-human - Work assignments, Field worker support, O&M reporting, Interaction with Service Users (e.g. Service Desk)
48 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Move from many distinct links and connections to a multi-service IP network for O&M
Telecom O&M Actors Interactions
Surveillance & Monitoring Platforms
Grid Stations Maintenance Support
On-duty Staff
Field Worker at Site
Event Notification
Alarm Notification
Combine Alarms, Durations, Correlations, Time of day, etc.
Equipment Access
Monitoring Access
Monitoring Data
Indications, Measurements, Survey, Images
Database Access, Instructions,
Documentation
Work Assignment & Scheduling
In-house or Contractor
49 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
O&M Communications around the Network Device
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Management Communication
Network Network Device O&M
Platform
O&M Workstation
Device-specific Tool
Device-specific Local Craft
Authentication & Access Control
Password Protected Local HMI Access
Role-Based Logged Authenticated Access
Vendor-specific and diversely protected
Encrypted Access to Authentication Server
SNMPv3
Vendor-specific and diversely protected
Remote Client & SNMPv3
50
O&M Communications between human users
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
Maintenance Management
& Support Field Worker
Service User
Network Supervisor
Service Manager
O&M Platform Manager
Transformation & Planning
O&M Network
51
Maintenance Management
& Support Field Worker
Service User
Network Supervisor
Service Manager
O&M Server and Admin
Transformation & Planning
IP Network
O&M Network Access beyond the Operational Telecom Network
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU
External Maintenance
Support
Field Worker
External Supervisor or Maintenance
O&M Server and Admin
Utility’s Private O&M Network
Public Network VPN
Field Worker
Public Mobile, VPN
O&M Server and Admin
Utility’s Private O&M Network
In-house Maintenance
Support
O&M Server and Admin
Utility’s Private O&M Network
Network Device
Public Network VPN
(a)
(b)
(c)
52
Conclusion
• The requirement for communication service is growing rapidly in the power system and extending to all its components
– Distributed intelligence in the power system,
– Increased requirement for situational awareness
– New IT/OT interactions (e.g. office access to substation data, remote maintenance)
• The communication network is consequently becoming larger and more complex, with more users and more actors
• A more comprehensive approach to Operation & Maintenance is required
– New process models
– New tools providing network and service situational awareness
– A reasonable level of process automation allowing a moderate-sized O&M organization to deliver the required level of service
Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 53