IDE DCS developments

Ewa StaneckaINP Kraków

ID Week, 19.05.2014

2

Plan

• Overview of the IDE DCS• IDE DCS migration – progress, manpower,

timescale• New TEH control system commissioning• New systems integration- Sonar and

thermosiphon• Summary

3

IDE DCS projects

ATLIDEEVCOOL cooling plant SCT & Pixel cooling

ATLIDEEVENV ID environment monitoring

ATLIDETEH thermal enclosure heater pads

ATLIDEMAG magnetic field monitoring

ATLIDERAD radiation monitoringATLIDEBLM• beam conditions

ATLAS network

IDE LCS: overall FSM beam monitoring IDE racks monitoring safety scripts

ATLIDEBCM beam conditions luminosity

ATLIDESNR- degassing- leak check- mass flowmeter

NEW

4

DCS Migration• IDE DCS follows ATLAS DCS LS1 upgrade plan, that includes

important changes both in hardware and in software:– New computers DELL PowerEdge R610

• installed 6 new machines in P1, old machines kept until new system commissioning

– OS system upgrade: Windows NT to Linux– PVSS-> WinCC 3.11– CAN bus interfaces: KVASER->Systec– OPC DA server -> OPC UA server

• Standardised procedures delivered by central ATLAS DCS • Migration exercised on two projects

– IDEENV migration on the test system (Koichi)– IDEMAG migration on the production system (Koichi)

5

DCS Migration• More attention needed in the systems using dedicated solutions• EVCOOL

– In some ELMB framework was not used , we need to write script for readdressing data points

– Test migration pending – prepare OPC config file, scripts for re-addressing ELMB data points in DCS project, etc.

• TEH – custom ELMB firmware– scripts for migrations prepared (Alex Madsen, Ewa)– Test system using new standards commissioned in SR1 (Alex, Ewa, Jacek)

• BCM– DCS and FPGA(ROD) communication via UDP protocol and dedicated dll

extension – low voltage power supply similar to SCT LVPS – can profit from SCT DCS

experience• BLM, RadMon – standard

6

Personnel considerations

• IDE DCS core (EVCOOL, ENV, TEH, IDSCS, MAG) – Koichi, Ewa, Alex Madsen (TEH)

• BCM/BLM/Radmon – Bostian Macek starting from June (?), more manpower needed

• Technical support - Jacek Pieron is leaving very soon, he provides support in many ID areas, it will be very hard to replace him !

• Timescale for migration:– IDE DCS core – June/July– BCM/BLM/RadMon/Sonar – by the end of 2014

7

TEH recap Two Functions

To maintain the thermal interface between the various components of the Inner Detector (this includes TRT, SCT, Pixels, IDEP) at a constant temperature of 20oC.

To prevent condensation on exposed cold parts in the event that we need to open the detector and keep the cooling operational.

Construction and Layout The Pads are Copper on Kapton technology (largest pads 0.64m2), glued to the

detectors and are installed with them. The power ranges from 10s of Watts to 120 Watts. (typical density (excl PST)

100W/m2) There are 280 pads in total with a total power capacity of 23.2kW -> Pixel support

tube (PST), TRT barrel, SCT barrel, SCT endcap, ID endplate Most important pads between the SCT and TRT (-25oC one side and 20oC the other).

There are 8 Pads between SCT and TRT in Barrel Region (total power 512W) There are 24 sets of pads between SCT and TRT in each End-Cap (48 total)

(2.8KW).

8

TEH control system• New Control system

Several deficiencies of the switching system were identified in the old control system. The deficiencies were both electrical and mechanical in nature.

The new control system was designed and produced to address these deficiencies .

USA15 US15

CANBUS1 CANBUS2Crate 1 Crate 4 HW 1 2 3 4 5 6 7 8 HW 1 2 3 4 5 6 7 8PVSS 1 2 3 4 5 6 7 8 PVSS 24 25 26 27 28 29 30 31slot 1 2 3 4 5 6 7 8 slot 1 2 3 4 5 6 7 8

Crate 2 Crate 5 18 HW 9 a b c d e f 10 HW 9 a PVSS 9 10 11 12 13 14 15 16 PVSS 35 32 33 slot 1 2 3 4 5 6 7 8 slot 1 2 3 4 5 6 7 8

Crate 3 18 HW 11 12 13 14 15 16 17PVSS 34 17 18 19 20 21 22 23slot 1 2 3 4 5 6 7 8

9

Control of the TEH temperatureHeater – resistance in range ( 32 Ω – 65 Ω )

Thermistor

• Hardware: custom switch cards. The supplies are rated at 48V (typically 40V at pads)

• Software : Specialized ELMB firmware run on ELMB to monitor and control of the switches . The power is delivered in a switched ON/OFF mode (duty cycle controlled by PID)

48 V

ELMB

U [V]

T [s]0

48

Sampling period: 100ms – 1s

+

-

Switching card

HW INTERLOCK

10

TEH control system – switching card• Interlock mechanism at the level of single

channel• Full monitoring of the states of all switches,

interlocks, switches configuration, DSS_RQ and Maser Enable control signals, power inputs

• Improvements in single channel switching circuit, grounding, power distribution

• The newly-designed reset system allows for issuing hardware reset commands to ELMB and hardware interlocks

11

TEH control system – control card

• The ELMB reset function (single and group ELMB reset)

• The reset of hardware interlocks (all hardware interlocks can now be reset by any of the Control Cards).

• The Power Monitoring allows to remotely control and monitor power in the entire system. The Control Card is able to turn on/off each part of the power system (48V, +/-5V) and to monitor it

12

TEH control system installation• NEW TEH control system installed during TS1:• May/June 2013 -> EC C and EC A commissioned and tested

– each „heater pad group” was switched separately to check mapping– All left for ~ 30min in „auto” mode– Very smooth start for heater pads that were working during normal detector

operation– Software/hardware mapping verified and corrected for pads that were not in use

so far• August 2013 –> Consolidation of TEH pads/sensor connections on PP1 side

C, recovered 15 out of 17 bad pad/sensor connections.• April 2014 ->

– recovered channels implemented into TEH DCS configuration– Consolidation of PST pads and whole system re-commissioning.– new control panels commissioned

• Tuning PID parameters for operation – to be done with cooling operational• End Plate pads are disconnected

13

TEH plans

• New TEH control system for PST, TRT barrel, SCT barrel and SCT end caps is installed and ready to go.

• A few small fixes to be done by the end of June. • ID endplate pads to be commissioned• Prepare documentation and procedures for ID

experts • We need a person to take over hardware

maintenance from Jacek.

14

New systems integration - Thermosiphon

– Prepare panels for monitoring thermosiphon• detector parts remain unchanged, the same control on single

line level• monitoring of the plant changes

– Thermosiphon will be monitored from the local server based on UNICOS CPC 6 framework of EN/ICE

– These local data servers will be located in CCC– ID DCS and Thermosiphon data exchange via DIP (?)– A few crucial parameters will be monitored directly from

PLC• Work to be done in IDE DCS

– Exercise communication with CCC and PLC– define list of parameters to be exchanged and operation procedures– replace “compressor module” by “thermosiphon module”, panels etc.

15

New systems integration - Sonar • The Sonar DCS project is installed in the pit and operational• Example of sound velocity, concentration, tube temperature panel in

WINCC Graphical User Interface

16

Sonar DCS integration with IDE DCS

• Work towards integration of Sonar DCS into IDE DCS has started, discussions with ID experts on – Design and implementation of Sonar FSM– Definitions of Alarms– Operational model, sifter actions etc.

• Personnel - Cecile Deterre, A.A. Hasib, Koichi Nagai

• Sonar system should be integrated by the end of 2014

17

Summary

• IDE DCS migration to new standards is pending • TEH, IDEENV, MAG migration done on the test

systems• New TEH control system is commissioned.• Ongoing work towards integration of Sonar

DCS into IDE DCS

18

Additional Material

19ATLAS ID day, CERN, May 19th , 2014

Example of sound velocity, concentration, tube temperature panel in WINCC Graphical User Interface

Disclaimer: Material provided by Greg Hallwell

20

Future integration for sonar into IDE panel (slide of Koichi Nagai)

ATLAS ID day, CERN, May 19th , 2014Disclaimer: Material provided by Greg Hallwell

21ATLAS ID day, CERN, May 19th , 2014

Future integration for sonar into IDG panel (slide of Koichi Nagai)

Disclaimer: Material provided by Greg Hallwell

22

First ideas for sonar alarm & type reportingPixel SCT IBL? Flowmeter Degassing

C3F8 conc. > 10-4

“WARNING”C3F8 conc. > 10-4

“WARNING”CO2 conc. > 10-4

“WARNING”Flow>±1% from

norm“WARNING”

Air conc. > 1%..?“WARNING”

C3F8 conc .> 10-3

“ERROR”

C3F8 conc .> 10-3

“ERROR”

CO2 conc. > 10-3

“ERROR”

Flow >±3% from norm

“ERROR”

Air conc. > 3%..?

“ERROR”

No comms. >10s?“WARNING”

No comms. >10s?“WARNING”

No comms. >10s? “WARNING”

No comms. >10s?“WARNING”

No comms. >10s?“WARNING”

Valve selection problem/ valve

module no comm.“WARNING”

Valve selection problem / valve module

no comm.“WARNING”

Valve selection problem / valve

module no comm. “WARNING”

Sound vel >±2% from C3F8 norm

“ERROR”

Valve selection problem

“ERROR”

Vac pump problem:pressure/ pump un-start / pump module

no comm.“WARNING”

Vac pump problem: pressure, pump

un-start/ pump module no comm.

“WARNING”

Vac pump problem: pressure, pump un-start/ pump module

no comm.“WARNING”

Vac problem:Pressure

“ERROR”

Temp sensor prob.

“WARNING”

Temp sensor prob.

“WARNING”

Temp sensor prob.

“WARNING”

Temp sensor prob.“WARNING”

Temp sensor prob.“WARNING”

Shifter alarm:Shifter alarm: + call SNR & ID cooling expert (or GSM automatic?)Shifter alarm: + call SNR & EN-CV cooling expert (or GSM automatic?)

Disclaimer: Material provided by Greg Hallwell

23ATLAS ID day, CERN, May 19th , 2014

IDG and IDE panels will be modified before end LS1 by ID experts

SNR experts to propose parameters to be shown in IDE panel (space restrictions!)

SNR experts to prepare “SNR panels” & FSM tree below “ATLIDESNR” branch in “IDE panel ” following standard ATLAS DCS rules

Sonar alarms will appear on the IDE alarm screen! Shifters & IDE experts must take care of “Errors” immediately (sonar generates no “Fatals”)Alarm help to be displayed on alarm screen: how to treat error, which expert to call, etc. Also document/twiki for experts with detailed description of causes of problem & related phenomena, with expert actions needed to solve problem

SNR experts making a risk analysis document including: - list of all possible incidents - categorized into “Error” or “Warning”- description of causes and corresponding shifter or expert actions

Need to review the list of alarms with ID/Central DCS.

Disclaimer: Material provided by Greg Hallwell

24

Timescales & personnel

(1) Develop integration plan into overall DCS architecture based on examples of other subsystems: shifter level/expert panels, alarm

reporting etc. together with supporting documentation. Cecile Deterre, A.A. Hasib, Koichi Nagai,

with help from DCS inc. Ewa Stanecka, Stefan Schlenker

(2) Alarm handling, routing to shifters and experts, together with supporting documentation.

Cecile Deterre, Greg Hallewell, A.A. Hasib, Koichi Nagai, with help from DCS inc. Ewa Stanecka, Stefan Schlenker

Integration of tasks in time for late summer cooling runs with pixel, SCT, IBL

92ndSonar Meeting, CERN, May 7th , 2014Disclaimer: Material provided by Greg Hallwell