Chamonix XIV January Controls The roadmap towards LHC controls Rdiger Schmidt and Robin Lauckner Chamonix 2005 On behalf of the Controls Group and collaborators Where to go LHC beam operation LHC controls Hardware commissioning and efficient controls Milestones past and future Conclusions Effects of the compressed schedule Chamonix XIV January Rdiger Schmidt and Robin Lauckner Chamonix 2005 On behalf of the Controls Group and collaborators Perspective of a user of the controls NOT as controls expert Controls The roadmap towards LHC controls Effects of the compressed schedule Chamonix XIV January AB-CO meeting (AB-CO-Day) in December Considering the compressed LHC schedule the concurrent requirements from LEIR, CNGS, the CCC other developments prompt availability of the Control System becomes even more critical The meeting had the following aims: to ensure that the Controls Systems are ready on time to establish the goals and milestones for 2005 and 2006 to prepare for 2007 and 2008 The approach is a progressive deployment of solutions to meet successive milestones across CERN The workshop is intended to give statuses, to expose problems if any and to help providing written documents for controls clients link: Chamonix XIV January This presentation is based on the material presented during the AB-CO meeting (CO-Day) It does NOT address controls issues in other groups (only a few remarks) ! B.Frammery, P.Charrue, M.Vanden Eynden, F.Di Maio, C.H.Sicard, E.Hatziangeli, R.Billen, R.Lauckner, P.Gayet, M.Gourber-Pace, V.Baggiolini, K.Kostro, J.L.Nougaret, J.Lewis, M.Peryt, C.Roderick, K.Sigerud, L.Mestre, H.Milcent, M.Zerlauth, M.Zaera Sanz, R.Lauckner, E.Hatziangeli R.Denz, J.Casas-Cubillos, R.Gavaggio, P.Gomes, I.Laugier, M.Lamont, S.Page.....and many others inside and outside AB-CO that contributed to the work This presentation and the AB-CO-Day Chamonix XIV January (TT40) TI8 SPS new timing Prototype Real Time orbit feedback TT40 Prototype collimator control Summary of AB/CO main milestones LHC sector test with beam LHC ready for beam TI2 commissioning Large elec. circuit commissioning (HWC proper) Quench Protection Equip tests QRL start commissioning DFB test Test LSS8L (String 3) CNGS beam commissioning Start SPS with fast SC change CCC ready CNGS HW commissioning LEIR beam commissioning Chamonix XIV January LHC controls: LHC ready for beam Beam control Transfer lines Injection and Extraction (beam dumping system) Beam optics controls Beam instrumentation RF Beam interlocks Collimation Real Time orbit feedback Radiation monitors Hardware control Power converters Quench protection system Cryogenics system Powering interlocks Vacuum systems Uninterruptible Power Supplies (UPS), Arret Urgence Generale (AUG) Safety systems and general services Chamonix XIV January These are the main systems for Hardware Commissioning LHC controls: prototyping Beam control Transfer lines -> TT40 and TI8 Injection and Extraction (beam dumping system) -> SPS to TT40 Beam optics controls -> TT40 and TI8 Beam instrumentation -> TT40 and TI8 RF Beam interlocks -> TT40 and TI8 Collimation -> SPS and TT40, prototyping but with stand-alone controls Real Time orbit feedback -> SPS Radiation monitors Hardware control Power converters Quench protection system Cryogenics system -> LHC cryoplants (not for ring cryogenics) Powering interlocks-> TT40 and TI8 (for normal conducting magnets) Vacuum systems -> SPS, TT40 and TI8 Uninterruptible Power Supplies (UPS), Arret Urgence Generale (AUG) Safety systems and general services (display of TI8 tunnel temperatures by normal conducting magnet supervision using PVSS) Chamonix XIV January Hardware for LHC controls provided by AB-CO Frontends and Gateways: Computers (PLCs, PCs, VME systems) Networks WorldFIP network and Profibus Ethernet (provided by IT, but needing coordination and testing) Field Control Rooms and Servers: Consoles, displays, Specific electronics (some of it being developed) Timing modules Beam and Powering interlock electronics (VME, PLC and custom) Modules for generation and distribution of Safe LHC Parameters (VME) Cables, cables and cables HW installation planning being addressed, planning for sectors 7-8 and 8-1 will be confirmed by end January Installation body in place before end January define detailed tasks and responsibilites (cable verifications, HW procurement, installation, functional testing, etc) weekly follow-ups linked to the planning forthmentionned Chamonix XIV January HARDWARE INSTALLATION controls group ~300 VME SYSTEMS ~150 PC Rackable Gateways and WorldFIP Infrastructure 200 km cabling, ~40000 passive and ~1100 active elements Timing System Infrastructure Timing distribution via Copper cables and optical transmission VME MTGs in CernControlCentre Remote Reboot Service for PC Gateways, PLCs and Field Interfaces ~30 Schneider PLCs for the reset of ~300 systems and Local Cabling Terminal Service For PC Gateways and VME Crates with Terminal server boxes in SRx + Local cabling Local Consoles (SRx and Underground based on the PC Gateway platform) Well under way, but this is most urgent and no time to lose ! Chamonix XIV January PC Gateways, VME CPUs and Remote Reboot in stock M.Vanden Eynden for the AB/CO/HT section Chamonix XIV January Logging Post Mortem Timing Transient recording (OASIS) Alarms (LASER) Software services and deployment Chamonix XIV January Logging TT40/TI8 extractions SPS vacuum Post Mortemsome experience from TT40 SM18 quench analysis reuse? Timing PS and SPS Transient recording (OASIS)TT40 and TI8 Alarms (LASER) Software services and deployment Chamonix XIV January Logging TT40/TI8 extractions SPS vacuum Post Mortemsome experience from TT40 SM18 quench analysis reuse? Timing PS and SPS Transient recording (OASIS)TT40 and TI8 Alarms (LASER) LSA Project Developing generic application software collaboration of AB-CO and AB-OP (LSA Team) FESA Collaboration of AB-CO and BDI (FESA Team) CMW Controls Middleware AB-CO PVSS / UNICOS Supervisory Controls and Data Acquisition (PVSS as commercial product, UNICOS (home made) framework for PVSS Software services and deployment Chamonix XIV January LSA Project Magnet control, beam instruments, orbit control, driving fixed displays, FESA Access equipment: Beam Interlocks, OASIS, CMW Communication Glue between everything PVSS / UNICOS Vacuum, cryogenics, magnet interlocks, display of data from cooling and ventilation Software services and deployment Logging TT40/TI8 extractions SPS vacuum Post Mortemsome experience from TT40 SM18 quench analysis reuse? Timing PS and SPS Transient recording (OASIS)TT40 and TI8 Alarms (LASER) Chamonix XIV January LHC complexity and Data management inspired by R.Billen AB-CO and TS LHC Layout Mechanical Optical DC powering Physical equipment equipment catalogue serial number Operational Data setting measurements logging post mortem Controls configuration Hardware topology Software topology RackWizard ABCAM Portal to capture assets and installation data Chamonix XIV January Everything should fall in place: Consistent data management is a great challenge LHC Layout Mechanical Optical DC powering Physical equipment equipment catalogue serial number Operational Data setting measurements logging post mortem Controls configuration Hardware topology Software topology RackWizard ABCAM Portal to capture assets and installation data Several critical milestones upcoming (in January and February) Naming of LHC entities and signals is an important issue => see conventions (e.g. EDMS ) Chamonix XIV January Hardware commissioning Before starting: QRL commissioning Test of a section of the QRL Vacuum, cryogenics The first part all main systems together: type test of magnet feedbox (DFBA) commissioning Test with 4 electrical circuits in SM18 Vacuum, cryogenics, (power converters), quench protection, powering interlocks The fun part: first magnets in LHC (LSS8L - String 3) commissioning Commissioning of 2 powering subsectors with 29 electrical circuits Vacuum, cryogenics, power converters, quench protection, powering interlocks, UPS and AUG The tough part (HEP community impatiently waiting...): All the rest Commissioning of 26 powering subsectors with 1585 electrical circuits Chamonix XIV January Hardware Commissioning controls perspective Commissioning of the (superconducting) magnet powering system Objectives Safe magnet powering operational Ramping of magnets works Magnets can operate at high field reliably for several hours Qualify electrical circuits, and capture information Commission and validate controls for operational powering of LHC magnets (everything what can be done without beam) To be performed in two phases Power Converters not connected to Magnets Power Converters connected to Magnets Driven by HardwareCommissioningWG, software discussed in SACEC team: Software Application for Commissioning of Electrical Circuits HW and SW experts (F.Rodriguez-Mateos / R.Lauckner et al) Chamonix XIV January first current ramp for all circuits with I > 120 A current time minimum current (about 2% of nominal) minimum quench current nominal current observe voltage drop across entire circuit - should be constant when I=constant (close to zero) discharge from nominal current energy extraction ok ramp to nominal current: global protection ok Scaling from String 2: commissioning of the circuits one by one for LHC would take 15 years: substantial increase of efficiency is essential Efficient controls allows for commissioning of several circuits in parallel Chamonix XIV January Key to efficiency: Automated Testing QPS HardwarePIC HardwarePower Converter Automated Test Procedures Quench Protection (PVSS Supervision) Powering Interlocks ( PVSS Supervision) Power Converters (Gateway) CMW Gateway Ethernet WorldFIP PVSS-CMW Interface PVSS-CMW Interface CMW Detailed specs being written by SACEC ( M.Zerlauth, B.Puccio, R.Denz, H.Thiesen, S.Page, M.Zaera Sanz, H.Milcent, F.Chevrier) Implementation by AB-CO based on existing tools Chamonix XIV January Major controls contributions from other groups to LHC Hardware commissioning Vacuum system (controls by AT-VAC, Alarms and Logging from AB-CO) Quench protection system (controls shared by AT-MEL and AB-CO) Power converters (controls shared by AB-PO and AB-CO) Cryogenics system (controls shared by AT-ACR and AB-CO) a worry is the manpower for writing software specifications and then translating the specs into PLC and PVSS code and controls commissioning. This is clearly aggravated with several activities in parallel. many other tools developed by CO are being used, clarifications between AT-ACR and AB-CO (remote reset, rack wizard, ) underway Chamonix XIV January Milestones and main requirements This table is not complete, it shows the most relevant tools and milestones ! Time HC Chamonix XIV January Logging, Shot-by-Shot Logging, Alarms, Post Mortem recording All these systems store data for certain variables Post Mortem analysis is using data from all these systems and other data (e.g. from LSA, databases,..) ! 24 Post Mortem: Beam incident during TT40 tests J.Wenninger (reconstructed) from Logging System Magnetic septum current change time within SPS super-cycle Data acquired from Shot-by-Shot & WIC Logging clients proved essential for the post-mortem analysis of the TT40 high intensity beam test accident PC simulation (PC off) by AB/PO J.Wenninger AB-OP Chamonix XIV January Logging during TT40 /TI8 tests: Transient recording (horizontal zoom) used as post mortem T = 1 ms 26 Example: SPS and LHC Transfer Lines Vacuum Control System AT-VAC/IN Section Synoptic of the SPS Complex Pressure profile in LHC TI8 Vacuum sectorisation of LHC TI8 Vacuum layout of LHC TI8 I.Laugier AT-VAC 27 AB-CO-IS AT-ACR Example: Applications Deployed 2004 for the Cryogenics system LN Sector 1-2Sector 2-3Sector 3-4Sector 4-5Sector 5-6Sector 6-7Sector 7-8Sector 8-1 QSRB QURA QUI Point 4 Point 2Point 1.8Point 6Point 8 Surface (buildings+areas) Cavern QURC QSC QSRA QUI QURC QUI Shaft QRL Tunnel HP storage LN 2 2 HP storage HP storage HP storage HP storage QSCB Central HP storage QSC QSCC QURA QSCA QSCC QSRB QSRA QSCCQSCA QSCB QSC QSCC QSC QSCAQSCC QSC QSCCQSCB QSC QSCCQSCAQSCB QSC QSCC To be Deployed Deployed Concept/Unity V1/ PVSS 2.12 Deployed Unity V1 Deployed Concept/Unity V1/ PCVue Deployed ABB Ph.Gayet, AB-CO 28 FGC FGC Gateway 30 Function Generator Controllers, one per power converter WorldFIP bus (50Hz cycle) LynxOS PC gateway Controls Ethernet 50Hz Status Commands Operator console Client application (JAPC) CMW Commands Power Converter Controls Architecture S.Page AB-PO 2 ppm = 14 MeV of full current 29 PVSS Temperature Supervision Warm magnet interlocks for TI8 M.Zaera-Sanz AB-CO Chamonix XIV January Comments Controls is for LHC and for all other accelerators at CERN Similar to hardware systems, controls differs between accelerators There were THREE, and now ONLY ONE controls group A limited variety of solutions emerged / have been selected standardisation as far as reasonable The strategy: implementing these solutions progressively started 2003 (TT40) and 2004 with (SPS and TT40/TI8) with many intermediate milestones to LHC beam commissioning For hardware commissioning, some issues I consider to be more critical than others getting the controls hardware installed and tested post mortem recording and analysis automatic test procedure integration of controls efforts with other groups Chamonix XIV January Conclusions Hardware commissioning best validation of controls is via QRL tests QPS Surface tests DFB tests String 3 the rest Beam commissioning, best validation of beam controls via TI8 / TT40 LEIR SPS Sector test TI2 LHC this gives us the chance to start most critical phase of hardware commissioning with well-tested and efficient controls Develop / Implement / Commission, if possible well ahead of the date when controls is needed with very high efficiency ! Dry runs for all critical systems ! 32 Reserve slides 33 Risks I do not see any showstoppers, but Unprecedented complexity of LHC and its controls limit controls complexity whenever possible Suppression of intermediate milestones Discussing principle controls philosophy instead of going ahead with what can be done with what we have Wrong priorities diverting from the essential Squeezing out the last 5% (forgetting other issues) - diverting from the essential 34 abstract Controls for beam operation are similar to other CERN accelerators. From the experience gained in the TI8 tests, with milestones such as the commissioning of LEIR, TI2, the controls for LHC will be realised with the same building blocks as used today. Hardware commissioning will provide a different challenge with an large scale use of industrial solutions for accelerator control. Already long before first injection into the LHC, controls for vacuum, cryogenics, quench protection, powering interlocks and power converters must be fully operational. Current plans for the installation and commissioning of the controls infrastructure will be explained and the main aspects highlighted. The generic controls facilities required for the commissioning of the hardware and for beam operation will be discussed with emphasis on how the work advances in the different domains (post-mortem, logging, alarms, timing, specific applications, etc.). The requirements, in particular for Hardware Commissioning, and how they translate into existing controls and / or technical specifications will be addressed. The presentation is based on the summary of a AB-CO meeting (CO-Day) that has being organised in December. 35 BIC console application (JAVA) JAPC RDA CORBA JVM i th Client PC running on console - GET( ) - SET( ) - Monitor[On, Off] Reset, CMW Server VME BIC driver BIC HW RDA CORBA Eq. SWI/O Lib RT-Task IO Lib 1 Hz Data + Time Shared memory segment On-line Buffer (Inputs + Time) Message queue Subscribers List Data Ready Supervision of the Beam Interlock Controller based on JAVA (BIC) 36 Issues deserving special attention Hardware installation Timing hardware Data management VDSL for QRL commissioning PVSS SIEMENS PLC driver Logging, post-mortem, shot-by-shot logging, SDSS logging Naming conventions and application FESA for QPS Users of FESA need support Controls system and machine protection / interlocks Software interlocks and Integritiy of safety critical information Network security Real time feedback Collimators controls 37 The Consoles PCs commercial of the shelf with 1 to 3 screens installed in local or central control rooms used to run operational software Operating System Customised version of NICE XP SLC3 Linux distribution Basic Installed Packages Common Console Manager JAVA runtime environment Exceed 38 The Application Server HP ProLiant 2x2.8 Ghz XEON, 3Gb RAM, 2x16Gb system disks, Dual PowerSupply Redundancy, hot swap, hardware Raid0, , allows for continuity of service and tolerance to faults SLC3 LINUX distribution or RedHat RHEL-3 for Oracle 9i AS transfer.ref mechanism to launch system and user services and applications System resources and user applications monitored by clic and reported to the Alarm system 39 Status of magnet interlocks Normal conducting magnets (WIC) operational during several month of running, without single failure during TI8 and TT40 tests several magets saved from overheating interfaced to other system (cooling and ventilation) WIC is the small brother of PIC partial validation of PVSS and PLC architecture for magnet interlocks next milestone is LEIR Superconducting magnets (PIC) PIC well advanced Milestone for DFB tests 40 Issues Software maintenance Unlike the previous years, patches and upgrades have to be installed during operation Therefore in collaboration with software providers and OP we will have to test and deploy the patches throughout the year System Configuration Curtain policy not yet finalised (who can do what and where between development and operational zones) Homogenisation between (old)PS and (old)SL not yet finalised Third Party Support Java products (OC4J, SonicMQ, ) or PVSS package configuration needs manual intervention Need resources to setup an automatic recovery procedure Third Party Software can prevent system upgrades 41 Hardware Commissioning in 2 phases Power Converters not connected to Magnets Test of electrical circuits one by one Initial continuity of circuits, integrity of instrumentation Insulation tests, interlocks on water, switches 8h operation at ultimate current 24h run of all PC Power Converters, QPS and PIC tested together before connection to magnets Power Converters connected to Magnets Commission electrical circuits one by one at low current Commission electrical circuits one by one or in groups to nominal current Commission of electrical circuits powered in unison to nominal current 42 Logging, SbS Logging, Alarms, Post Mortem recording Logging System Stores logging data for certain variables (logging entities) Collects data, typically at 1 Hz or slower, in regular intervals or on change Alarm System Handles alarms in case of fault conditions and stores alarm data Defines & processes alarm data for certain variables (fault members) Shot-by-Shot Logging System (designed to monitor LHC filling) Stores logging data for certain variables (logging entities) for each extraction from SPS to TT40 / TI8 (after an event the shot) Variables can be stored as one data point, or as transient data with many data points Post Mortem transient recording Stores transient data for certain variables after a post-mortem event (post mortem entities) or after an internal fault of a system Name and time stamp for entities required Post Mortem analysis is using data from all these systems 43 Operator console 2 running programs to interface to operators LHC Controls Option 1 (simplified) Operator console 1 running programs to interface to operators Front End Controllers PLC / VME / custom electronics CCC or Field Control Room Ethernet Around the LHC Underground or Surface Buildings Example 44 Operator console 2 running programs to interface to operators LHC Controls Option 2 (simplified) Operator console 1 running programs to interface to operators Server running application programs / alarms / logging Front End Controllers PLC / VME / custom electronics CCC or Field Control Room Back of the CCC or elsewhere Ethernet Around the LHC Underground or Surface Buildings Front End Controllers PLC / VME / custom electronics Ethernet 45 Operator console 2 running programs to interface to operators LHC Controls Option 3 (simplified)) Operator console 1 running programs to interface to operators Server running application programs / alarms / logging Front End Controllers eg. QPS and PC CCC or Field Control Room Back of the CCC or elsewhere Around the LHC Underground or Surface Buildings Front End Controllers eg. QPS and PC Gateway (Front End Computer) Around the LHC Surface Buildings Ethernet ProfiBus / WorldFIP Gateway (Front End Computer) 46 Data management LHC machine description (mechanical, optics, electrical) LHC layout DC magnet powering Accelerator Controls Configuration Model for PS complex extended to LHC ABCAM controls equipment installation database (first version for end January 2005) Rack Wizard (to describe racks and what is inside) Other projects depend on data management (e.g FESA, installation, ) Calibration information (e.g. for thermometers) Asset management CERN wide with MTF Several critical milestones upcoming (in January and February) Naming is an important issue => see naming conventions (e.g. EDMS ) 47 Logging during TT40 /TI8 tests: Vertical zoom I = 29 A Data acquired from Shot-by-Shot & WIC Logging clients proved invaluable for the post-mortem analysis of the TT40 high intensity beam test accident ( ) T = 28.8 s