52
Operation & Maintenance of Telecom Networks and Associated Information Systems in the Electrical Power Utility Mehrdad MESBAH Tutorial, Cigre D2 Colloquium, Lima, PERU October 2015 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 1

Operation & Maintenance of Telecom Networks and …d2.cigre.org/content/download/71597/3277017/version/1/file/1_Cigre... · Operation & Maintenance of Telecom Networks and Associated

Embed Size (px)

Citation preview

Operation & Maintenance of Telecom Networks and Associated Information Systems

in the Electrical Power Utility

Mehrdad MESBAH Tutorial, Cigre D2 Colloquium, Lima, PERU

October 2015

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 1

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

2 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

1 - Introduction

• How do Electrical Power Utilities operate and maintain their telecom infrastructures and services today? Which issues are they facing?

• How are the associated resources, processes and tools evolving to assure adequacy in the evolving communication context?

• Cigre WGD2.33 : “Operation & Maintenance of Telecom Networks and Associated Information Systems in the Electrical Power Utility”

• launched in June 2012 (18 members, 12 Utilities)

• Survey over 10 T&D Utilities worldwide

• Technical Brochure published as TB588 in July 2014

– Current practices, issues of concern, evolution plans

– Mainstream trends, guidelines, best practices, standards

– Extending from previous Cigre WGs and Reports

3 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

O&M – Process, Organization and Tools

• Major adjustments are required for responding to:

• A sharp increase in the size and scope of the telecom network • Significant increase in the number of services or of service users • Major change in Utility organization (e.g. separation of telecom entity) • Major change in service commitments, in liability, or 3rd party involvement • Sharp change in technical staff capability (new technos, retirements, …) • Deployment of process automation tools and change of roles and tasks

4 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Management

Organization

Management

Process

Management

Tools

Management

Scope

Management

Organization

Management

Process

Management

Tools

Management

Scope

Management

Organization

Management

Process

Management

Tools

Management

Scope

O&M is often the forgotten component of network transformation projects

Reasons for Modelling the O&M Process

O&M Process

Modeling

Assessment of

Service Provisioning

New Rules and New

Actors

O&M Cost Estimation& Justification

New Tools & Process

Automation

New Ambitions

& Goals

Worforce Constraints

Increased Scope &

Change of Scale

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 5

Reference framework for O&M processes and related KPIs

New applications, Wider coverage Larger impact of network faults, Multiplicity of technologies, Multiplicity of skills & tools

Regulatory obligations on (Grid reliability, security and efficiency) Contracted 3rd party interventions

Smaller staff, higher turnover and new technical skills

New liabilities and commitments (SLA/KPI) in revenue generating telecom services

Growing weight of O&M in comparison to the cost of technology imply accurate

Justify tool investments Define the extent of process automation in the management of telecoms

Elaborating a Model through EPU Survey

• Statistical Data Analysis – Collect Extensive Data, Correlate, Elaborate Rules on Behavior for each aspect

– In our subject data sample is not large enough

– Special cases and contexts are numerous – Not modeling a “single box”

– Conclusions could be either Trivial or Erroneous!

• Case Studies – Select some Utilities, Observe and Elaborate Rules on Behavior in each Utility

– Would not show which rules are related to which specificities

• Transverse Assessment over selected issues (e.g. Fault Management)

– Allows the comparison of approach, work volume, staffing, automated versus manual process, tools employed, etc. investigating the diversity with respect to Utility specificities

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 8

EPU-A EPU-B EPU-C

Criterion 1

Criterion N

.

.

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

9 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

2 - Which Electrical Power Utility (and Which Requirements?)

Network Perimeter and O&M Scope • EPU’s role & footprint? Spread of the communication network?

• Wide Area? LAN? Substation /Control Platform / Tech Office?, Customer?, Enterprise Network? Control Room IT, Enterprise IT, …

• Electronics, Cable infrastructure, Towers and Antennas, Intra-site cabling?

Provisioning/Delivery Mode • Telecom as part of the Grid Operation

• Telecom as separate service providing entity

• Telecom as Contracted Activity or as Procured Service

Size & Scale of O&M Organization

• Number of Sites, Users and Services

• Number of technical staff in Technical Office/Control Platform

• Number of Field Workers (dedicated or shared asset maintenance?)

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 10

• No unique/typical model for all Electrical Power Utility Telecom nets • Need to set different modelling parameters for this investigation

Service User

Service Provider

P&C Scada

Metering Asset Mngt Surveillance

T&D Grid WAN/LAN

Telecom/Platform OT/IT

Communication domains in the EPU defining telecom O&M perimeter

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Intra-plant Local Net

Intra-plant Local Net

Platform Local Net

Field Plants Control Platforms

Platform Local Net

Core Telecom Network

Customer Local Net

Customer Premises

Private or Public IP Net (wired or wireless)

Customer Backhaul

Wide Area Cables & Telecom System

Control Room IT

Substation & Plant IED

Meters & Appliances

BD : Boundary Device (Server, Concentrator or Gateway device)

Modem & Comms device

Concentrator & Comms device

Control Centre LANs

Substation & Power Plant LAN

switches & cabling

BD BD BD BD

BD BD

Office

Local Net Network Device

& Cabling

WAN Access

Enterprise IT

Private or Public IP Net

(VPNs)

Corporate Network

BD BD

BD

Customer Access

(Last Mile) BD

U-Office & Data Centre

BD

Inter-Platform Network

Grid Inter-plant

Net

11

12 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Corporate Activities

Grid Operations

EPU

Telecom

A

Telecom is part of the operational activity. Corporate entity provisions telecom services separately

Corporate Activities

Grid Operations

EPU

Telecom Services

B

Common Telecom (& IT) Services for both Corporate and Operational Applications.

Corporate Activities

Grid Operations

Power Corporation

Telecom Service

Provider

C

EPU

TSP is a sister company to the EPU, providing services exclusively (or in priority) for the Power System

Corporate Activities

Grid Operations

EPU

Telecom Service

Contractor

D

Telecom Assets

EPU procures its telecom assets but operates them using an external Service Contractor

Corporate Activities

Grid Operations

EPU

Telecom Service

Provider

E

Telecom services are procured under SLA by a Telecom Service Provider delivering services to many customers.

EPU Service Provisioning/Delivery Modes

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process - Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

13 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

3- Evolution of O&M Scope, Process and Capabilities

• Utility Survey = How the telecom network is operated today (static vision)

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 14

How is the O&M scope changing? How can this new scope be delivered?

New Processes

New Tools

Organizational Adjustments

New Capabilities

New Services

New Users

New Sites

New Assets

Commitments

Evolution of O&M Scope and Process

15

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Technologies / infrastructure

• Broadband media & Packet-based technologies (Ethernet / IP) • Acceleration of technological changes & shorter life-cycles • Network maintenance through equipment replacement versus repair • Permanent planning and continuous transformation • Training cycles on new technologies • Deployment of tools driving process change

Services & Site Coverage

• Dedicated inter-utility extranets • Operation-support/enterprise intranets • Communication sites (e.g. distribution network) • “Smart” applications (Remote access/monitoring, “office-to-field” ...)

Human Resources & their localization

• Ageing workforce & massive retirements • continued maintenance of older technos • Re-allocation of skills & expertise • Remote support for field intervention

Service Users & Providers

• Migration of service delivery model • SLA-based relationships

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Maturity Model for Telecom Service Delivery

Maturity Scale 1 - Service Delivery Scope

Service level defined implicitly

by applications

QoS specified explicitly but

No SLA

Liable to SLA Must produce formal proof

Formal SLA No systematic measurement

Supplier Helpdesks &

Warranties only

Support Contracts for sub-systems

Full Service Delivery

Contracts

Field Maintenance

Contracts

Procure Telecom Services

Maturity Scale 2 – Service Provisioning Scope

Overall telecom budget without

repartition

Contributive repartition of

telecom budget

Service price established as

Service Catalog

Service costs estimated but not recovered

Maturity Scale 3 – Business Model and Cost Recovery

Maturity Scale 4 – Service Liability

Service costs recovered at

no profit

Operational Services Only

Internal Multi-user

Services

Int. Services & Bandwidth

Wholesale

Internal Services &

Ext. Leasing

Int. Services & Ext. User Services

Formal SLA & monitored QoS

16

Building New Capabilities – Migration Planning

• How far into O&M evolution and Maturity Scale ?

• Governed by service requirements, company size and policy, cost, …

• O&M migration plan with intermediate steps

• Prevent loss of control and Quality of Service in the “learning process”

• Allow gradual and incremental investment effort

• Must accompany network/service migration

17

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Present State

Target Phase 1

Maturity X

Maturity Y Target

Phase 2

Capability must be planned and adjusted to match new network /service delivery scope

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

18 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

4- Modelling the Telecom O&M Process & Organization

• ITIL : Framework for internal processes necessary for running an IT infrastructure (employed in many EPU IT organizations)

– Classifies tasks and processes into groups and specifies interactions between them

– Relevant for IT management in any enterprise context including that of telecom service delivery organization

• TMF NGOSS (New Gen. Operation Support System) – Specifies a Telecom Operation model called e-TOM for Business processes of a Telecom

Service Provider (used by Cigre WGD2.26 to define a Utility sub-set called u-TOM)

• ITU-T M.3010 Logical Layered Architecture (LLA) – Extends the visibility of the management system through different layers of abstraction

(Element, Network, Service, Business)

• ITU-T M.3400 FCAPS model – Fault, Configuration, Accounting, Performance, Security

– Functional partitioning mainly related to domains of telecom infrastructure management (but can be applied to some extent to processes)

19 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Modelling the Telecom Management Process

• 1st order model : Upstream Management (Planning, Deployment, Transformation, Policy) as opposed to Day-to-day current operations

• O&M is basically the Current Operations, although capability building for O&M, and definition of its scope is in the Upstream Management.

• Feedback from Current Operation allows adjustments and upgrades

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 20

Current Operations

Upstream Management

Organization, Process,

Network Assets, Service Contracts,

Service Catalog

Operational Data,

Assessments,

Audit Reports,

Asset Requests

Policy Definition

Business Planning

Deployment

Upgrade & Migration

Business Development

Customer Relations

Service Management

Infrastructure Management

Provider Management

Cigre D2.26 Utility Telecom Operations Map

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Bu

ild S

trat

egy

Bu

ild C

apab

ility

Bu

ild S

erv

ice

Off

er

Customer / User Relationship

Customer / User

Provider / Contractor

Provider / Contractor Relationship

Service Management

Resource Management

Security, BCP, Safety, Skill Mgt

Strategy & Planning

Business Development

Service Portfolio Evolution

Service Migration Planning Deploy &

Adjust

Operations

Upstream Management

Enterprise Processes

21

Operation Management Models and Segments

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

User Relations Management

Communication Service Management

Service Provider & Contractor Management

Network Mgt.

Element Mgt.

Bu

sin

ess

Mgt

.

Service Mgt.

Fault Mgt.

Config. Mgt.

Account Mgt.

Perf. Mgt.

Security Mgt.

TMN ENSB Hierarchical Management Layers

CIGRE D2.26 uTOM Telecom Operation Map

(TMF eTOM)

ITU-T FCAPS model

User Service Desk User Notification User Dashboards

User SLA Monitoring

Incident Management Fault Management

Config. Mgt. & Inventory

Performance Monitoring

Contractor SLA Monitoring

O&M Communications

Op. Site & Facility Access

Bandwidth Policing (CIR/PIR) Infrastructure Management

22

D2.33 Assessment Structure

“Current Operations” Process for Telecom Service Delivery and Provision

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 23

Fulfillment (Setting up)

Assurance (Delivery) Running

Support (Enquiry, Supervision

& Maintenance)

Accounting (Metering, Cost

Repartition, Invoicing & Settlement)

User/Customer User Order Handling

User Change Handling

User Problem Handling User SLA Management Customer Relationship

Management

Service Enquiry Desk (User Technical Support)

Customer/User Entity Invoicing & Settlement

Communication Service

Service Configuration & Activation

Incident Management

Service Quality Mgt

Service Inventory Mgt

Service Change Management

Service Policing & Usage Metering

Network Infrastructure &

Resources

Bandwidth & Capacity

Provisioning

Network Problem Mgt

Network Perf. Management

Disaster Recovery

Net. Configuration & Change Fault Management

Network Maintenance Asset Lifecycle & Spare Mgt Management Tools Support

Estimate running cost of network infrastructure

(+ tools)

External Providers & Contractors

EP&C SLA Setting

& Adjustment

EP&C Performance Mgt (SLA Monitoring)

EP&C Problem Reporting (EP&C Relationship Mgt.)

EP&C Support Management (Site Access Permits,

Safety & Certification)

EP&C Cost Assessment & Invoice Settlements

O&M Main Process Interactions

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

24

Fault Management

Incident Management

Performance Monitoring

Service Mngt. & User Relations

Config & Change Mngt

Detect & Localize Anomaly

Fault Impact & Service Restore

Notification

Detect Performance & SLA Anomalies

Network Data

Network Data Performance

Dashboard

User-reported Anomaly

Service User

Support Service

Providers

Update Network

Vision

Network Change Report

Frequency of Recurrent O&M Actions in utility telecom

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

25

1/Hour 1/Day 1/Week 1/Month 1/Year

Creation of New Connections

Receive & Acknowledge Alarms

Open Incidents & Assign Interventions

Performance Checks

Preventive Maintenance (critical or old equipment, ventilation filters, power supply batteries)

Remote Configuration

Continuously Polled

Automatic Scheduled

Routine Maintenance

Transformation Works

Generic Management Organization & Process for a typical Operational Telecom Network

Service User

Operational Depts.

Telecom Network Centre Reg. Maintenance A

Back-up Net. Centre Reg. Maintenance B

Service User Operational Depts.

Service Notification Quality Reporting

Service Request Service Anomalies

Recurrent network anomalies &

Intervention Reports

Network outages with user service impact

User identified service anomalies

Recurrent service anomalies

Network Configuration Service Outage Planning

EMS/SCADA Facility Management Protection & Control Asset Management, …

Network Infrastructure (Region A & B)

Network Planning & Transformation

Service Management & User Relations

Use

r N

etw

ork

N

etw

ork

Man

agem

ent

Network

Supervisor

Maintenance

Engineers &

Support

Service Managers

Telecom Transformation

Engineers

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

26

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

27 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

5 – Fault & Anomalies Management

• Fault : failure (broken part or weakness) in a machine or system

• Anomaly : different from what is usual, or not in agreement with something else and therefore not satisfactory

Cambridge English Dictionary

Fault is signaled as a device alarm (either the faulty device, or one of its peers, or indirectly by a device overlaying the service delivered by the faulty one)

Anomaly is detected through monitoring of equipment or system performances or reported by a user

28 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Fault & Anomalies Management Platform

Performance Monitoring

Telecom Device

User Service Desk

Performance threshold detection

Alarm Monitoring

Anomaly Reporting

• Fault Detection • External Alarm Collection/Monitoring System • Vendor-specific NMS • Unified Multi-purpose Management Platform (e.g. SNMP) • Manager-of-managers (e.g. standard connections to Vendor-specific NMS)

Fault & Anomalies Management

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 29

• Fault Notification & Reporting • Updating of network and asset inventories • Documenting of the incident, performed and proposed corrective action

• Fault Localization • Associate Network configuration and Event data • Determine location & cause (RCA) over multiple layers of fault indications

• Fault Recovery & Restoration Reporting • Field intervention in a great majority of cases (need remote support) • May be made less urgent through remote action

• Fault Diagnostics • In-depth examination of the anomalous device or connection • Performed through vendor-specific NMS through specific skills

Incid

en

t Man

ageme

nt P

rocess (in

parallel)

Fault Detection Systems

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Read values and detect threshold passing

Detect external alarm conditions

Field Communication or

Management Device

Detect device faults

Fault and Event Management

Platform

Management Data

Communication

Network

Network Element

Network Element

Network Element

External Monitoring

Device

Remote Management

Device

Vendor-specific

Management

Multi-purpose Management Platform

Network Element

SCADA Remote Terminal

Unit

Power System SCADA

Management Data Communication Network

Northbound Interface SNMP

SNMP or Proprietary

IEC101 or 104

Proprietary

Alarm contact

SNMP SNMP or

Proprietary

Value measurements

30

General Scheme

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

31 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

6 – Incident Management, Work Assignment and Escalation

• Register faults and anomalies (open incident),

• Allocate adequate resources to their resolution (assign to a person/dept.),

• Coordinate and keep track of the performed actions (escalate / document),

• Decide upon satisfactory resolution and assure reporting (close incident)

• Generate statistics on frequency of occurrence and resolution time for improving the network or the process (weekly, monthly, …) – mainly used for continuous improvement

32 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

• Majority of EPUs have some tools for managing incidents – often applied only to external suppliers, contractors, …

• Full scale enterprise tools (e.g. SAP) are cumbersome and time-consuming while a fully informal non-automated process is insufficient

• Dead time due to responsibility borders is reported as a major issue (up to 12 hours)

• Need precise roles partitioning – e.g. between user /core telecom networks

dedicated infrastructure

in-house O&M workforce

Merged treatment for Service and Network incidents

External network or workforce

Need in-house service incident management

6 – Incident Management – Interaction with O&M Contractor

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

33

Customer / User Relationship

Customer / User

Provider / Contractor Relationship

Service Management

Customer / User Relationship

Service Management

Resource Management

External O&M Contractor

In-house Service Management

(Open incident & Assign to Contractor)

Service Contract (SLA)

Operational Services (SLA) (Service Anomalies Reporting)

No

tify Service Resto

re

Close ticket

No

tify Users

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

34 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

7 – Service Management & User Relations

• Service Desk – interface point for User Relations (telephone, e-mail, or web-based automated desk for larger companies with merged telecom & IT activities)

• Creation, Modification and Deletion Requests (discussed in Change Management)

• User Detected Anomalies (widely variable across surveyed Utilities)

• Incident tickets opened by O&M staff (not by the user)

• Comprehend User issue, follow and notify its resolution, generate report (automated ?)

• Quality & SLA Monitoring – continuous monitoring of SLA metrics • Monitoring through User applications (customer feedback on perceived quality), e.g. SCADA

• Monitoring through Impact Analysis – “Backward Root Cause Analysis” (e.g. Loss of Back-up)

• User Notification – alert a service user of potential impact on his services of a network fault detected by the provider

• Can be manual or automated (upon service impact detection)

• In some cases integrated into reporting and performed only upon recovery

• User Dashboard – provide permanently an account of quality of service • Service availability, outage distribution, throughput, time latency, problem resolution upon

incident reporting, restoration upon fault, … for the services delivered to the user

• Usage Measurement and Invoicing (e.g. Service Metering) • Cost partitioning between user entities – not yet common but potentially necessary (IT/OT)

35 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

7 – Example of a User Service Dashboard

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 36

Distribution of Interruptions

Not Resolved at Site

Short Spurious Faults

Long Spurious Faults

Resolved at Site

Remotely Resolved

Statistical Availability for Services of a certain category delivered to a same User

7 – Service Management & User Relations

• CIGRE Utility Survey suggests that Service Reports, Statistics and Trends are at present mostly generated manually using data and reports from various management systems and using office tools (e.g. MS Excel)

• These are, in the majority of cases, used for internal purpose (service provider) unless external contractors are involved

• At present only monthly reports are published mainly to demonstrate conformity “in the wide sense”

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 37

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

38 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

8 – Configuration & Change Management

Process of documenting, adding/deleting, and modifying network assets, connections, and management tools in the communication system

39 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Network and Service Inventory

User Order Handling Service Activation

Configuring Network Assets

Capacity Management

Change Management

O&M IT/IS

Platform Management

Asset Life-cycle & Spares Management

Co

nfigu

ration

Man

agemen

t D

atabase (C

MD

B)

8 – Configuration Management Database (CMDB)

• Network and Service Inventory – Fundamental element for operating and maintaining the telecom system

– Accurate, structured, coherent, unique and up-to-date data on equipment and connections

– Information must be rapidly accessible to all O&M players

– Inventory information is required for

• Problem management – need knowledge of equipment types, relationship between alarm generating assets for Root Cause Analysis

• Determining Service Impacts of network faults – need knowledge of service-to-network mappings and dependencies

• Service restoration and circuit provisioning (i.e. configuration changes) – need knowledge of existing configurations

40 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

8 – Configuration Management Database (CMDB)

• Composed of a number of different data inventories:

– End-user entity inventory – contact coordinates for notification and reporting

– SLA inventory – service catalog containing specifications (availability, time/error performance, interface, throughput, security, outage and restoration time …)

– Service inventory – user-referenced connections (e.g. SCADA circuits) with corresponding mapping to logical resources, SLA elements, configuration, test records, etc. Provide service-to-network mapping for service impact analysis

– Logical Resource inventory – static and dynamic logical connections. Provide network-facing view of services (VLAN numbers, …)

– Physical Resource inventory – network infrastructure, cables, equipment, …

– Software/Firmware inventory – tools and platforms, licenses, …

– Spare Parts inventory – modules/devices in warehouse, ordered or in reparation

– Supplier/Partner Contract inventory – procured services, provider SLAs, …

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 41

8 – Configuration Management Database (CMDB) Some important considerations

• The data can be integrated into a single information system or maintained as distinct data structures (tables, lists, drawings, …) with or without interconnections

• Some can be integrated into an integrated management platform

– Assets, Connections, Notification Coordinates, SLA elements …

• Data discovery can be automated when assets are “configuration-aware”

• Operator-grade tools are over-dimensioned and expensive (highly automated)

• Interconnected or integrated data structures are used for automating highly recurrent processes or manipulating very high volumes of data

• But managing interconnected data and automated processes can increase considerably the complexity resulting in substantial cost and effort

– Trade-off between laborious (manual) and complex (automated) process

• Inventory Information Model – How much information needs to be interconnected? Which further data can reside in separate silos of information (e.g. vendor-specific management systems and tools)

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 42

8 – Configuration & Change Management

• User Order Handling and Service Activation

– Initializing a new connection of a pre-existing type or modifying an existing connection (e.g. increase bandwidth) upon a user request

– Assure data coherence and coordinate provider & user on service commissioning

• Configuring Network Assets & Managing Resource Capacities

– Not a highly recurrent activity in the power utility (except during transformations)

– For quasi-totality of survey cases, performed through vendor-specific tools (static allocation) and off-the-shelf enterprise network tools (dynamic allocation)

– Commonly relies upon upload of asset settings (however this cannot check what the setting should be from a network design perspective – does the equipment setting correspond to intended network design?)

– Remote secure access for configuring network equipment

– Back-up configuration files generation and storage in the Asset Inventory

– Keep track of resource usage and capacity (bandwidth, ports, boards, fibers …)

• Change Management

– Document asset , service and network changes in coordination with Field Intervention and Operation Support staff.

43 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

8 – Configuration & Change Management Tools, Platforms, Spares and Life-cycle

• O&M IT/IS Platform Management (ITIL Service Management applied to telecom management platforms)

– Delivery of IT services required for the fulfillment of Telecom Provider’s Business Processes

– Software release & License Management – Service Support Contracts for constituent Applications and Firmware – Security Patch Management – Software Documentation & IT Configuration Management – Telecom O&M or IT governance?

– Keeping track and maintaining operational tools, crafts and instrumentation

• Asset Life-cycle and Spares Management

– Obsolescence cycle of technology is increasingly short

– Keep continuous awareness for network assets of commercial availability for new projects, for replacements and upgrades, for spares & support …

– Keep inventory on spare modules, non-allocated equipment, …

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 44

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

45 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

9 – Quality of Service & Performance Monitoring

• Monitoring of end-to-end network performance against pre-set SLA values

– by user application (e.g. SCADA)

– by external measuring system connected at Service Access Points

– measured through agents in telecom device and stored in a MIB for retrieval by a remote monitoring platform

• Error measurements in TDM systems – incorporated into standard equipment

• Delay, loss and delay variation in Packet Switched networks

– SDH-like multi-layer performance management (i.e. Regeneration Section Overhead, Multiplex Section Overhead, etc.)

– Exchange OAM packets at each level between Maintenance Edge Points (MEP)

– Carrier-grade Ethernet and MPLS-TP

– IEEE 802.3ah at link-level (i.e. between consecutive switches)

– ITU-T Y.1731/IEEE 802.1ag at service level (i.e. end-to-end Ethernet service)

• Special cases: Packet-over-TDM and TDM-over-Packet

• Performance monitoring is to feed User Service Dashboards, SLA monitoring and fault supervision/notification

46 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Operation & Maintenance of Telecom Networks in the EPU

1. Introduction

2. Which Electrical Power Utility (and Which Requirements?)

3. Evolution of Scope and Process – Adjusting Capabilities

4. Modelling the O&M Process and Organization

5. Managing Faults and Anomalies

6. Incident Management, Work Assignment and Escalation

7. Service Management & User Relations

8. Configuration & Change Management

9. Quality of Service & Performance Monitoring

10. Telecom O&M Communications – Field Worker Support

47 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

10 – Telecom O&M Communications & Field Worker Support

• Telecom O&M actors require information exchange to carry out the described O&M tasks and processes. Some network capacity is allocated to these “own applications” as a specific IP network (i.e. one of the network’s multiple users)

• Device-to-Platform – Telecom Asset Monitoring and Fault Detection, Security Barriers reporting to a Security Operation Centre (SOC)

• Platform-to-platform – Connect Vendor-specific management systems and asset inventories to integrated management platforms, User Authentication through remote security server (e.g. RADIUS) …

• Device-to-human – Remote access to assets for diagnostics, configuration and setting (tele-maintenance), Video control of assets and facilities

• Human-to-platform – Remote access to telecom management platforms, Automated notification (to users and O&M workforce) of faults and service impacts

• Human-to-human - Work assignments, Field worker support, O&M reporting, Interaction with Service Users (e.g. Service Desk)

48 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Move from many distinct links and connections to a multi-service IP network for O&M

Telecom O&M Actors Interactions

Surveillance & Monitoring Platforms

Grid Stations Maintenance Support

On-duty Staff

Field Worker at Site

Event Notification

Alarm Notification

Combine Alarms, Durations, Correlations, Time of day, etc.

Equipment Access

Monitoring Access

Monitoring Data

Indications, Measurements, Survey, Images

Database Access, Instructions,

Documentation

Work Assignment & Scheduling

In-house or Contractor

49 Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

O&M Communications around the Network Device

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Management Communication

Network Network Device O&M

Platform

O&M Workstation

Device-specific Tool

Device-specific Local Craft

Authentication & Access Control

Password Protected Local HMI Access

Role-Based Logged Authenticated Access

Vendor-specific and diversely protected

Encrypted Access to Authentication Server

SNMPv3

Vendor-specific and diversely protected

Remote Client & SNMPv3

50

O&M Communications between human users

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

Maintenance Management

& Support Field Worker

Service User

Network Supervisor

Service Manager

O&M Platform Manager

Transformation & Planning

O&M Network

51

Maintenance Management

& Support Field Worker

Service User

Network Supervisor

Service Manager

O&M Server and Admin

Transformation & Planning

IP Network

O&M Network Access beyond the Operational Telecom Network

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

External Maintenance

Support

Field Worker

External Supervisor or Maintenance

O&M Server and Admin

Utility’s Private O&M Network

Public Network VPN

Field Worker

Public Mobile, VPN

O&M Server and Admin

Utility’s Private O&M Network

In-house Maintenance

Support

O&M Server and Admin

Utility’s Private O&M Network

Network Device

Public Network VPN

(a)

(b)

(c)

52

Conclusion

• The requirement for communication service is growing rapidly in the power system and extending to all its components

– Distributed intelligence in the power system,

– Increased requirement for situational awareness

– New IT/OT interactions (e.g. office access to substation data, remote maintenance)

• The communication network is consequently becoming larger and more complex, with more users and more actors

• A more comprehensive approach to Operation & Maintenance is required

– New process models

– New tools providing network and service situational awareness

– A reasonable level of process automation allowing a moderate-sized O&M organization to deliver the required level of service

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU 53

Thank You for Your Attention

[email protected]

Cigre D2 Colloquium 2015, Tutorial, Lima, PERU

54