BE-ICS Group Meeting December 2019

2019 - Long Shutdown 2

• Dominates our work schedule this year and next

• Many high priority projects in parallel

• Critical services for a successful start-up

2

So far so good. We are roughly on schedule, but there’s a lot of work ahead

2019 – Service Support

• Large effort from all

• Incident handling

• New Requests

• Request handling

process converging

• More in next talks

3

• 300 to 400 distinct users per day

• ~700 support tickets in 2019

• ~80 000 access requests created in 2019

Many newcomers in 2019 – Welcome!

4Andrzej,

PJAS in APPinelopi,

GAO

Alexandros

TI TECH

Krithi

FD TECH

Gustavo

AP Fellow

Daniel

TI Fellow

Manuel

FD Fellow

Gabrielle,

ADMINFilip,

FD Fellow

Luis,

CSE Staff

Anthony,

FD Fellow

Two Indefinite Contracts awarded

• Engineer in TI section

• Ivan Prieto

• Technical Engineer in FD

• Marc Quilichini

5

Some of the very many Highlights

from 2019

6/12/2019 EDMS 2280283 6

Disclaimer: This is my selection of a few of the many slides contributed

by the sections.

A picture is worth a thousand words

Full slide deck on INDICO

TooIs & Infrastructure - Services

7

55+ Tools & Services

Managed by 11 people

https://readthedocs.web.cern.ch/display/ICKB/BE-ICS-TI+Project+inventory

TI - Operational Support charge

8

MODESTI Support

TIM Support

TI Alarm Support

ADaMS Support (TI Section)

562 SNOW tickets in 12 month

TI - Operational Support

0

10

20

30

40

50

60

70

80

DEC'18 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV

CREATED VS CLOSED SNOW TICKETS

Created Closed

9~47 tickets per month, 562 in total

6/12/2019 EDMS 2280283 10

• 300 to 400 distinct users per day

• ~80 000 access requests created in 2019

5 versions of ADaMS released during 2019

New features:

• Urgent IMPACT entries (piquet) handled immediately

• Technical services roles can be associated to permissions No need for individual access requests

• New Concept: Permits (e.g. Driving authorization) may now be required for access, just like trainings

• Display in CERN GIS the locations accessible to each permission

• ADaMS replaced Evolynx for CERN badge production and the «source» photo repository at CERN

Proposal for the simplification of the access requestsapproval process to the DSOC was received favourably.

11

New Safety System Monitoring Stack

12

1 Master + 2 Mirrors

+ 10 Zabbix data proxies

+ 2 Gateway proxies

for PS and SPS

Monitoring 600+ “Intelligent hosts”:

Windows, Linux, RaspberryPI,

Touch Screens, Routers, …

400+ checks for Small Devices

and Services:

ADaMS, VIVOTEK, MOBOTIX,

…

Regular

Application

Screenshots

• SAM

• LHC

• SPS

• Kiosk

C2MON Evolution in 2019

• Cache Layer renovation handed from Szymon to Alex

• Moving to open-source cache engine (Apache Ignite) to replace aging product

• Refactoring the code to be more legible, easier to maintain and more modern Java 8 syntax

• Replacement of the ISOLDE Fast Beam Investigation (FBI) tool back-end by C2MON

• Further collaboration with potential KT startups, CERN CBI programme

Preview of the ISOLDE FBI Web user interface

13

ALIS – Alarm Information Service

Purpose: One Web page to find all information about TI alarms

Operational since Dec. 2018

• More than 170k+ alarms included

• Live alarm updates on integrated GIS map

• Powerful search interface

• Integrates nicely with InforEAM

• 800+ unique visits this year!

Will replace Phoenix Alarm Console in 2020!

14Please visit https://helpalarm.cern.ch6/12/2019 EDMS 2280283

CV controls: Cooling & Ventilation• Team : Joanna Staniszewska, Rafael Santos, Gustavo Martin, Jesus Cortes, Marc Quilichini, Will Booth

• And thanks to Jesus Sanchez & Christina Tsiplaki

• 40 PLCs deployed in production• Bat 863 (Tours SPS)

• Cool BB3/LSS3

• Detector Cooling:

• ATLAS Muons A+C

• 5 LHCb installations :

- RICH, OT, SciFi, Common Chiller, Air Dryer

• Point 18 : HVAC SM18 Ext, Cool+HVAC SFA181

• !!! Preparation for 2020 !!!• PSB Cooling, PSB Chilled Water, Cool PS/TT2, East Area renovation (cooling + HVAC),

Siemens S5 replacement (Leir, AD Hall, B201, B2004, B2011), SPS LSS Dump, CT2 cooling towers, UJ56 pumping station

15

EDMS 2280283 In collaboration with EN-CV

• HVAC ATLAS & CMS (with CV)• North Area Eau Demi (BA80/81/82)• HVAC BA Surface Building Ventilation (x5)• HVAC BA3 Faraday Cage

ATLAS Muons A+C

Tours SPS

LHC Cryogenics controlTeam : Philippe Durand, Jean-Michel Beckers, Borja Fernandez, Bernhard Gruber

Refrigerators• Renewal of old QSCAs LEP compressor room.

• New control system architecture implemented.

• Finalization of M580 migration, new profibus module deployed. 70 control systems migrated since 2017.

• Consolidation of numerous Cryo applications:

• 1.8 K Cold box Linde & Air Liquide

• 4.5K Warm Compressor room QSCB

• Cryogenic Interconnection Box

Tunnel• Huge reengineering to improve code quality and maintainability (code size cut in half)

• 234 automatic unit tests run by Jenkins

• Several new protective features

• Version upgrades (ready for Run3)

• Full roll-out in SM18 and LHC completed

16

End of EYETS-16/17, 40% replaced. To be completed by LS2

EDMS 2280283

LIC External Conditions Consolidation• Widespread

migration of

conditions to

FESA, reducing

costs of

refurbishment

• Many changes to

conditions during

LS2, particularly

PSB and PS

EDMS 2280283 17

LIC CT RT

PROFINET

RT

~ 1ms

874

MEYRIN PREVESSIN

CCC Push

buttons

SILECS, ~10msFiber OpticCopper

G. Kruk MPP workshop INDICO 803870

6/12/2019

PSEN (Electrical distribution Supervision )• Two major releases

• February 2019• Test/Production Alarm Application

• August 2019• Consolidation

• 4 minor releases

• 126 tickets treated in total

• 283,686 signals monitored • Status as of end of November 2019

• Original specification required support for 100,000 signals

• Release of psen2CPCMapper• Used by TE/EPC

• Import and created PSEN devices on top of CPC devices

• Knowledge shared within BE-ICS-AP members

EDMS 2280283

18

QPS (Quench Protection System)

EDMS 2280283 19

• New FEC-WinCC OA Communications architecture using a simplified FESA interface

• Benefits:• Direct logging of data from FECs to NXCALS, without an

intermediate WinCC OA project and an RDB

• Possible cost reductions by reducing the number of needed QPS Servers, and eliminating the QPS-specific RDB (QPSr)

• Design:• Eliminated the custom APM logic in WinCC OA by creating one

simple CMW interface per QPS device type.

• The code for the device-specific classes is largely automatically generated.

• Allows direct subscription from NXCALS

• Added some unit tests (based on FesaTestFwk)

• Refactoring and fixed a critical issue

• Code reviews with colleagues from MPE (collaboration)

• Initial deployment: Dec 2019 in FAIR

• To be deployed in LHC in spring 2020 (FESA part)

WCC OA

unCMWClient + APMs

FEC

Supervision*/Survey Properties

RDB/CALS

RDB API

CMW (RDA2)

CMW QPSOK

PIC WCC OA

Current

WCC OA

unCMWClient

FEC

New Properties

CALS

CMW + Filters (RDA3)

CMW + Filters (RDA3)

New Interface

PIC WCCOARDB

RDB API

CMW QPSOK

LS2

6/12/2019

FAIR magnet test bench

• The facility is fully in operation after

full commissioning of the control

system

• FAIR magnets has been delivered to

CERN for testing.

• WinCC OA software stack to be updated

to latest version (including RDA3 CMW

Client/Server) on Dec 2019.

• QPS FECs to be updated to the new

QPS FESA classes on Dec 2019.

EDMS 2280283 206/12/2019

SM18Safety Instrumented Systems and control system for operation: (UNICOS + Safety concept)

● Cluster A and C have been upgraded for High Lumi - 11T magnet test capability

● Cluster G has been fully upgraded with Fail-Safe PLC and new operator interface.

● Cluster A2 will be soon modified in order to power SC-Link (in couple with Cluster G)

EDMS 2280283 216/12/2019

MOON (BE-ICS comp & apps monitoring)

• Re-evaluation of WinCC OA application monitoring system

• Close integration with COSMOS ongoing, this will simplify

maintenance of the application

• Use of industry standard monitoring software where possible

(e.g. dataserver monitoring)

• Substitute UI with web based (Grafana) for easier access

• MONARC – application to monitor and administer Oracle

databases for WinCC OA archives

22

EDMS 2280283

External clients supportMajor effort with TE-MSC in 2019

EDMS 2280283

23

Large Magnet Factory

• The new controls developed in-house

using UNICOS-CPC with TIA Portal will

provide CERN full control of the

process for maintenance and updates

purposes.

• The press control system was no longer

supported and there was not at CERN the

capability to change the controls..

External clients supportMajor effort with TE-MSC in 2019

EDMS 2280283

24

Magnet Workshops B927 and B163

maintain and revamping the control system

of existing machines, used for production

and test of future magnet technology that will

be used in LHC.

Currently performing the status of the

existing plants and proposing the upgrade

actions in function of priorities and budget.

Virtual commissioning• Strong collaborations with EN-CV,

TE-CRG and EP-DT to couple process simulations with UNICOS applications

• EN-CV: LHC Point 8 Waste Heat Recovery Project

• EP-DT: CORA CO2 cooling process simulation

• TE-CRG: Cryogenics simulator migration to OPC UA

• BE-ICS-AP responsible for technology development, and contact with EA (supplier of EcosimPro simulation software)

EDMS 2280283 256/12/2019

Testing framework• BE-ICS-AP have developed a fully automated testing pipeline to generate,

deploy and test PLC-based applications

EDMS 2280283

26

• OPC UA communication layer allows common tests to be written for all supported PLC types

• Aim is to enable Continuous Integration for developers of PLC applications and frameworks

• Collaboration with BE-ICS-CSE for SPS Access

Industrial Control Frameworks 8.4.0 release of JCOP and UNICOS Frameworks for LS2 upgrades

Compatibility with latest WinCC OA release (3.16)

Binaries, UTF compatibility, XML panel format, Qt-based help system

Aligns with changes in CERN infrastructure, enable necessary migration

OPC-DA→UA , CMW Server RDA2→RDA3, CALS→NXCALS

Validation of Oracle 19c and DB schema upgrades

CERN-wide upgrades started in November’2019

Consolidation & Redesign

Build/Test/Release infrastructure solidified and homogenized

User participation improved to shape product roadmap(UNICOS Technical/Advisory Committees)

Preparations for open-source licensing (foreseen in 2020).

Numerous structural cleanup

LS2: unique opportunity to proceed with significant changes impacting Fw users and developers

Frameworks: Innovation (SCADA) New Alarm Screen

For experiments (JCOP) and ACC/TI(UNICOS)

Optimized for performance and features

Developed in collaboration with I

New UNICOS Main Window (HMI)

Ergonomics improvement required by renovationof the operators screens in control rooms

Demonstration and discussion with users in CCC, November 2019

OPC-UA Middleware Migration

Hardware family from commercial vendor

OPC-UA Linux users OPC-UA Windows users

Server ready, many production systems already migrated.

Server ready, successful production system end-user testing done.

Server ready, end-users testing with production systems.

Server ready, end-users testing with production systems.

Server ready, many production systems already migrated.

Server not ready yet. Expected end 2019

OPC-UA Middleware

SCADA

Commercial support for legacy servers (OPC-DA) stops end 2019.

Production grade OPC-UA replacement servers for start 2020.

WinCC OA CMW Middleware Migration (RDA3)

Extended collaboration with IHEP institute

for 2019/2020 (working on multiple projects).

BE-ICS, IHEP and BE-CO collaboration

on re-writing WinCCOA CMW server for RDA3.

First users of BE-CO Device Server library.

CMW RDA3 server ready for deployment

in UNICOS framework release v8.4.0Accelerator chain software

SCADA

WinCC OA CMW

Middleware

PLC

Permanent support to the PLC users community

1500+ PLCs in production and 150+ users

Obsolescence plan and strategy for technology evolution presented

Fruitful partnership with Siemens and Schneider:

Renovation of BE-CO infrastructure with Siemens Industrial PCs.

CERN Beta version of Siemens Safety modules to fix critical issue.

New Schneider ordering online service in preparation.

Participation in testing and evolution of new Schneider Profibus module

Collaboration agreement in progress with external company

PROCON for machine learning and data analytics.

openlab: WinCC OA Next Generation Archiver

Collaboration of CERN and Siemens/ETM

New, modular, extensible archiving system

3 developers in Austria, 2 at CERN

Milestone: first production deployment starts

Critical part of the ALICE O2 online system upgrade

Backward compatibility with Oracle archiving used so far

Significant maturity reached after 3 years of development and testing

Milestone: pilot deployments with InfluxDB backend

ATLAS TileCal test systems, FASER

CERN-wide migration at the timescale of LS3

Availability for early-adopters at the end of LS2

openlab: Data Analytics for Industrial Controls

Summer 2019: creation of the

BE department Machine Learning Forum

Mean spectrum of oxygen deficiency

sensors in the LHC tunnel

Room occupancy detection

by IoT infrared imaging

Machine learning optimization

of the LINAC3 ion source

SPS Personnel Protection System

Access control: SPS• Adapted the software lifecycle for PLC programs, affecting the specification

and validation activities

• Updated the Safety Instrumented Functions (SIF) document (in progress)

• New Detailed Design Document to describe precisely the PLC program logic

based on Logic diagrams

• Application of model checking (PLCverif tool) for module verification

EDMS 2280283

35

SIF

Document

Detailed Design

Document

Specification Module validation System validation

Formal Verification

PLCverif

Testing platform +

Generated test cases

Automatic and/or

Manual Tests

TIM – SPS PPS Web application

Purpose:

Web application showing information about the SPS PPS events and persons on site

• Available at every SPS Access Point

• Special version for SPS Operators with historical log

Technology stack:

• Frontend: VueJS + TypeScript

• Backend: Java + Database

36

37

6/12/2019 EDMS 2280283 38

6/12/2019 EDMS 2280283 39

6/12/2019 EDMS 2280283 40

6/12/2019 EDMS 2280283 41

Performant systems keeping you safeBB3- March 2019Electrical rack

B182- September 2019Liquid argon leak gaz

6/12/2019EDMS 2280283

B904-February 2019Electrical rack

B112-June 2019Electric bike charger

SCX5- Mai 2019UPS rack

GIF-Mai 2019Flammable gas leak

B107-March 2019HCN gas leak

42

Fire Detection & Evacuation

• Projects realisation: ME9, FLEX Storage (B947), LHCb and ALICE Data Center,

LINAC 3, SX5 Extension, BM1-2-4-6,

• Projects preparation: SXA5, AD target, UPR HL-LHC,

• Renewal realization: First phase of PS ZS28 and ZS23 to welcome the new

kindergarten and Science Gateway, dismantling of East Area, replaced old NOVA

VEGA DEF central in North Area,

• Maintenance ensured with no main operational issue. Required urgent repair or

TDC2/TCC2 detection due to radiation degradation of PVC tubes.

6/12/2019EDMS 2280283

CMS Evacuation exercice

ALICE Data CenterTCC2/TDC2Muon Stasis SX5 New Kindergarten

43

Gaz & ODH Detection & SNIFFER

• Projects realisation: 107 Surface Treatment completed, ENH1 Saleve infrastructure operational,

manufacturing spare SNIFFER electronic cards to maintain the system until LS3,

• Projects preparation: prototype connection box for HL-LHC UPR detectors,

• Renewals realisation: Pre-design of EHN2 and ECN3 detection systems,

• Innovation: Applying Artificial Intelligence to ODH data sensors

• Maintenance was ensured with no main operational issue.

107 Surface Treatment Prototype

Maintenance in

Experiment racks

6/12/2019EDMS 2280283

ENH1 Saleve

Applying AI to ODH

44

Alarm Transmission - CSAM

6/12/2019 EDMS 2280283

SCR 2.0SPS Fire SafetyNew local synoptic

Alarm updates: 2889

Alarm inserts: 2782

Alarm deletions: 2398• Projects realisation: SCR 2.0 – B65 SCR refurbishing for the Fire Brigade, High

Resolution screens deployed, SPS Fire Safety OPC communication with new

DEF centrals and detectors, new Central Alarm Display display

• Projects preparation: Extension of CSAM HL-LHC racks

• Consolidation: Replacement of local synoptic, upgrade of smoke bar graph

analogue values,

• Maintenance ensured with no main operational issue.

High Resolution

Smoke bar graph

45

Contribution to Big Projects

6/12/2019

EDMS 2280283

• Preliminary Discussions:

• Science Gateway

• Multiservice Building Point 1 & Point 5

• PCC B775 – New Prevessin Computer Center

• Projects Realization:

• SPS Fire Safety WP2, WP4 – Accompanying project and passage to maintenance

• HL-LHC – Red Telephones production, UPR installation preparation, new demands for

surface, preliminary underground design discussion

• Cost estimates:

• HostLab

Multiservice Building Science Gateway

SPS Fire Safety

HL-LHC Red Telephones

46

Access Control & Proton Analogy (for fun)

1. SUSI protects and sees all

2. ZORAs are the main

components

3. ADaMS glues it all together

ADaMS

6/12/2019 EDMS 2280283 47

SUSIAccess & Video systems continue to increase in numbers

Completed worksCERN Open Days surveillance

SM18 new site entry

Video protection alarm analytics in Prévessin cable park and BAs

METRUM – RP Truck Monitoring

New/Ongoing projects513 and SM18 consolidation

Science Gateway

Prévessin Computer Centre

Meyrin Gate E

6/12/2019 EDMS 2280283 48

ZORA PSOperational Facilities during LS2

MEDICIS

SM18 RF test bench

GBAR

Completed worksLinac4 Phase 2

SWY new sector

LN2 -> SWY Access Point

Large SAS installation

LASER InterlocksCTF, B169, etc…

OngoingLN4 Phase 3

CLEAR modification

AD Target modification

6/12/2019 EDMS 2280283 49

ZORA SPSOperational Facilities

GIF++

LIGHT (in LHC2)

SPS Access System revamping, whilst running old one

AWAKE summer run

11 of 16 Access Points installed

Other works completedNeutrino Platform Access

GIF++ enlargement

Interface with RP CROME

Video surveillance supervision of NA62

6/12/2019 EDMS 2280283 50

ZORA LHCLHC Access System revamping whilst running the old one

20 of 34 access points renewed

11 ECRs, including

• upgrade of critical components SW/HW

• MilliQan

• Enlargement of MADs because of lift change

• oh, and also HL-LHC

Improved monitoring of critical power supplies

6/12/2019 EDMS 2280283 51

ICALEPCS

• Large contribution to ICALEPCS 19

• Orals, mini orals and posters

• Organization of the PLC based control systems

workshop

• Large group participation

• BASICS meeting planned

EDMS 2280283

52

2019, almost done

• Thank you all!

• Nice teamwork

• Take some rest

• There’s more…

53

What’s coming up for 2020?

6/12/2019 EDMS 2280283 54

A couple of words of introduction to the presentations to

come…

2020 - Long Shutdown 2 continues• Big safety projects

• SPS Access Control and Fire Safety• High priority for ICS (and CERN)

• Good team work across sections

• Industrial Controls components and applications• Deliver on time, Frameworks,

components, control systems

• Many other groups depend on our work here

• All the while, keeping our services running…

55

Additional request and projects• Primary gas for experimental areas

• PCC B775 – New PrevessinComputer Center

• Fire detection renovation 5MCHF project over 5 years

• Science Gateway

• CO3 Task Force on Web development

• FAIR Collaboration on Access Control

• Machine Learning Forum

• New contracts for access control works

• LHC Fire Safety study starts

• …

56

Plan with Priorities and Constraints

• The organisation sets the priorities

• Our capacity is limited

• Manpower

• Budget!

• We will do everything we can, with the

highest quality

57

Safety first

Quality second

Schedule thirdF. Bordry