Embed Size (px)
Citation preview
Hafslund AMS
Drinking from the fire hose at a large IOT project
Jon Andreas Pretorius, Hafslund Nett
Axel Borge,Sesam Simen Sommerfeldt, Bouvet
to NDC 2015
Hafslund s.2
s.3
Operational structure
Heat Network Markets Power
Hydro&power& District&hea1ng&produc1on&and&distribu1on&Bio7energy&
Network&System&control¢re&
Power&sales&Billing&Customer&services&
• Hafslund Nett owns and operates Norway's largest electricity grid and has long had one of the lowest net rents
• Hafslund Nett owns and operates the regional grid in Oslo, Akershus county and Østfold county
• Hafslund Nett owns and operates the distribution network in Oslo and most of Akershus and Østfold counties
• Number of distribution network customers are 675,000 • The network consists of 37,000 km overhead lines and
underground cables • Hafslund Driftssentral is one of Europe's most advanced
operating centers, that controls, monitors and optimizes power to 1.4 million people, Hafslund Varme's district heating plants in the Oslo area and Hafslund Produksjon's power plants in Glomma
Business Area Network
s.4
Hafslund was founded more than 100 years ago - in 1898
s.5
There will be more changes to the power grid operation the next five years than the 100 last years
1899 1911 2011 2020
?&
Changes in regulations will increase complexity significant and increase demand for automation
s.7
35 000 enkle fjernavleste målere
Årlig driftskost pr måler: ca 950
700 000 komplekse fjernavleste målere
Har i dag buffer for feilretting. Stort sett bare ifm måler-avlesning at programmet er tett
Årlig driftskost pr måler: 150
Alt online og ingen buffer eller servicevindu. Alt må alltid være tilgjengelig. Kan illustreres ved å tenke at det var måleravlesning hver dag hele døgnet
IT er i liten grad en trussel for omdømme
Hacking, virus mm vil utgjøre en mye større trussel generelt og målere vil kunne hackes
Kompleksitet høy, men vi reddes av rolige perioder
Kompleksitet vil være betydelig høyere og konstant
I dag 2020
AMS elHUB
Kundens forventning som i 1990 Hvordan vil kundens forventning endres?
AMS:&
• The&new&smart&meters&detect&power&consump1on&on&an&hourly&basis&and&automa1cally&sends&informa1on&about&consump1on&to&the&grid&company&
• The&new&smart&meters&records&also&includes&data&related&to&voltage&quality,&ground&faults&etc&
• The&new&meters&have&two7way&communica1on&between&the&meter&and&grid&companies,&and&will&give&customers¤t&informa1on&about&their&own&consump1on&and&instant&prices&for&electricity&and&grid&transport.&
• Grid&companies&are&responsible&for&the&installa1on&and&opera1on&of&the&AMS&system,&but&they&can¬&deny&other&service&providers&to&offer&customers&management&and&informa1on&services.&
&
Advanced&Metering&and&Control&Systems&(AMS)&are&introduced&within&2019.01.01&
AMS$%$the$founda/on$of$the$future$of$digital$power$grid
s.9
Elhub and the supplier centric model
ElHub Statnett has been commissioned by NVE to establish Elhub. Elhub shall collect all metering values for Norway and makie these values available for power suppliers and their end customers. Furthermore Elhub will support processes for customers moving or switching suppliers, and compile data for clearing between participants in the electricity market For Hafslund Nett this means that collected and verified hour ly values from all AMS meters shall be transferred to ElHub once a day When vthe supplier centric model is established , customers will only deal with the electricity company (example service and infrastructure provider of mobile telephony) The supplier centric model creates major changes in business processes and data exchange in the industry
Drawing from elhub.no
s.10
System&D& System&…&
System&C&System&B&System&A&
System&N&
Hafslund investigated two alternative solutions for integration architecture that will support the demands of new AMS solution; - ServiceBus - Data hub(Semantic/RDF) Hafslund has experience with both solutions, but the project consider a Data Hub based solution most appropriate in this context; - Increased stability
(asynchronous data exchange)
- Fewer integration points - Similarly architecture chosen
for central El Hub
DataNAV&
Choice$of$integra/on$solu/on
s.11
IFS$ERP$
Warehousing&&&Logis1cs& Project&module& WO7module& 360º&Scheduling&
New&field&system&
Economy&installa1on®istry&Documenta1on&
HR/resource&
Rollout$AMS$
Opera/ons$AMS$
Stage&Planning&and&monitoring& Assign& Start&& Perform& Report&Project,)Opera-ons,)
Maintanance)excis-ng)Recep1on&7&
withdrawals&goods&
Data&Hub&
GeoNIS&#installa1on&
Quant&#AMS&&
Generis&#old&meters&
CAB&#Customer&&
Datawarehouse&/&archive&
Consolidated&customer&and&installa1on&data&from&Data&hub&
Data&sources&for&rollout&
Data&Recipients&rollout&
Historical&data&archive&and&analysis&
Integration engine
All masterdata is consolidated in Data Hub Data Hub is the only source for all business applications In the semantic data base all data are connected Data Hub provides great potential for management of the information model and analysis
Established$applica/on$solu/on$design
www.hafslund.no www.hafslund.no
Advisor and CTO, Bouvet Oslo Lær Kidsa Koding! Oslo IoT meetup
@sisomm
Master data: The Elephant
in the room
System System
System
System
System
System
System
System System
System
System
I have given many talks and written several articles about IoT
Hypothesis: Internet of Things projects always involve integration
Archive
HR
CMS
Finance
Payroll
Classic Point-to-
Point integration
Archive
HR
CMS
Finance
Payroll
ESB ESB
Classic integration with ESB
The systems keep changing their need for information from other systems
Archive
HR
CMS
Finance
Payroll
Common data model?
Traditional SOA with a canonical datamodel
PREVENTS flexibility
DOA – Data Oriented Architecture.
Enter
27
Archive
HR
CMS
Finance
Payroll
Keep the models, keep the data
We copy the domain
context in the systems into
SESAM
Convert data to triplets - RDF ID Name Position Born E-mail Manager
101 Tim Berners-Lee
Programmer 08061955 [email protected] 958
958 Vint Cerf Inventor 23061940 [email protected] 999
765 Pål Spilling Professor 04111940 [email protected] 765
Subject Predicate Object
101
101
101
101
Type Person
Name
Position
Born
Manager 101
08061955 958 Programmer Tim Berners-Lee 101
Universally unique identifiers
Subject Predicate Object
www.org.no/data/system/person/1 Type Person
www.org.no/data/system/person/1 Name Tim Berners-Lee
www.org.no/data/system/person/1 Position Programmer
www.org.no/data/system/person/1 Manager www.org.no/data/system/person/2
www.org.no/data/system/person/2 Name Vint Cerf
www.org.no = Unique organisation on the internet www.org.no/data/system/person/1 = unique id of the information element
Archive
HR
CMS
Finance
Payroll
Link data from different GRAPHS
Duplicates Contact=Contact
Location= Depnr Empnr=ID1
Org=Depnr
With DUKE we can link elements with no common identifiers – using statistical algorithms and training on data sets
Archive
HR
CMS
Finance
Payroll
Data sources can enrich eachother
HUB
Doc
Meta1 Meta2
Doc
Archive
HR
Turnus
Finance
Payroll
Keep data from ”dead” systems
Nav
HR
Payroll
Search
Archive
HR
CMS
Finance
Payroll
Hub
How things are connected
HR Dest
SDShare
Source
HUB
• Based on Atom: Pull data, don’t push • Asynchronous • Subscribers ask for data that has changed
since the last time • Update frequencies are adjustable • You can ask for changes or the whole dataset • Data formats changed in transfer.
www.sdshare.org
Sesam uses a Hashtracker to keep track of the latest changes, or an explicit “latest changed” field
Read using SQL/File acces, Write using API’s
A push receiver is a http server receiving RDF fragments. It calls the API of the receiving system. If the call fails, Sesam can try again
DOA
Database'
Content'Mngmt'
File'System'
Enterprise'Search'
Reporting'
Analytics'
Data'Hub'
System'X'
Public'Open'Data'
Content'Mngmt'
Enhance'and'
Connect'
COLLECT
CONNECT
SHARE
Tran
sform'
Data objects flow NOT messages
Kafka for extra throughput
SDShare'Server'KaDa'
Provider'
RDF'Store'
KaDa'Queue'
The Kafka Provider Pulls Information off from the Queue and can add extra data from the RDF store before exposing it out via SDShare. It can also apply filters based on data in the hub or the item on the queue.
• SQL Databases via jdbc • CSV files • RDF triple stores • Sharepoint • Kafka • XML files • LDAP providers • Excel files • MS Exchange server (mail and calendar) • SDSHARE – anything! (MongoDB, etc)
Data sources and sinks
45
Data Analytics & Enhancement Existing Systems
Processes Search and Reporting
All Data Indexed
Contribute data
Drive process through state change
Models in data, Constraints in data
Act on all data Analytics results are
just more data
Complete views of all
systems and processes
Use Data
All people can ask all questions
Uniformly Structured data
from heterogeneous
sources
System Improved
Other systems can keep running even if one is down. And you can upgrade a system or install a new with fewer impacts
The customer controls the information model and
becomes more independent from vendors
Customers can prototype and combine data for new
solutions rapidly
1. Thou shall only get data from other domains through Sesam
2. SOA is dead, long live DOA. Processes advance through state changes
3. There can never be a common data model in the company
4. Thou shall never query Sesam directly, but through SDSHARE
5. Thou shall be comfortable with eventual consistency
6. Thou will always get the same answer when you ask Sesam the same
question. And Sesam can say the same things many times
7. The world is asynchronus, as is Sesam. Don’t try to shoehorn synchronicity
8. Thou shall embrace that data can have different sources/master and values
9. The systems need not know about Sesam
10. Sesam is not a backup.
The Commandments of Sesam
• Runs in Docker containers • Github and Saltstack are used to keep all
installations up-to-date • At the core: Virtuoso Triple Store • Includes a data browser • Indexed with SOLR to provide universal search • All communication happens with SDSHARE • Configuration over coding
Sesam tech
A paradigm shift for developers • Eventual consistency • “Pilfering” of data • RDF and SDShare • Sparql is not SQL • Idempotence: Sesam
can send duplicates • No RPC calls or
message passing • You need an information
architect in the project • Don´t add more queues.
Pedal to the metal! Now implement AMS with SESAM!
A recap of the requirements
• Massive amounts of data • Many systems must be coordinated • Many stages in the deployment, with changing
needs • Systems will be upgraded and changed • The systems were not designed to cooperate with
each other • Bugs and errors happen – in systems and human
actions.
670.000 x 4 x 24 =
64.320.000 daily readings
YAY! Need for american-scale technology
Yay! Need for
AMERICAN scale
technology!
The first challenge is to sync information across systems before the measurements can begin
Construct the SQL for the output feeds, implement push receivers for input feeds.
All the arrows are
feeds in/out of Sesam
Slide source: Ståle Heitmann, Computas
Slide source: Ståle Heitmann, Computas
When to use Sesam
• When all else is tried – you are f***ked • If you have many domains in the company • If your integration work involves a lot of
data transformation, lookup and conversion
• If the logic in the ESB rivals that of the systems
• For Internet of things projects • As a collector for big data projects
Micro services need to get information from several places
Want to know more?
• contact us at [email protected] and we will help you get started
• www.sesam.no • www.sdshare.org
• Anders Volle • Ståle Heitmann • Steinar Rudsar • Axel Borge • Øystein Isaksen • Graham Moore • Lars Marius Garshol • Steinar Rune Eriksen
Thanks to...
Questions?
