Global Collaboration on Accelerator Operations and … · The European XFEL X-Ray Laser Project X-Ray Free-Electron Laser Matthias Clausen, DESY ICFA Seminar, 29.October 2008 Global

XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser

Matthias Clausen, DESYICFA Seminar, 29.October 2008

Global Collaboration on Accelerator Operations and Experiments

Matthias Clausen, DESYICFA Seminar, 29.October 2008

Matthias Clausen, DESYICFA Seminar, 29.October 2008 2


Globalization …

… in the Financial World– Has a bad taste.

• Socializing risk?• Privatizing win?

… in the HEP Community– Is key to build the next big machines

… in the Controls Community– Within the existing (homogeneous) collaborations working and thinking

globally is by default no problem.– Collaborating globally between heterogeneous partners can be a

difficult task.• We must make sure that we reach a Win/Win situation for all

participating parties and that nobody is left behind.



Overview

Globalization issuesTechnical Realization

– Existing Implementations– Future Scenario

Collaborations– Control System Collaborations

Outlook



Existing Implementations

Technical Realizations in the Past• SNS• TTF/ FLASH• GAN/MVL

Work in Progress• LHC/CMS at Fermilab



MessagesAlarms

Control Systems – general Layout

FEC FEC

BPM Magnets

Master

TimingLLRF

Cavities

Process Controller

PLC PLC PLC

Utilities

Process Controller

PLCPLC

Process Controller

PLC PLC

Cryo ExperimentSlow Control

P TT

Process Controller

PLC PLC PLC

VacuumCamera

ApplicationDisplay

NetworkLayerDAQ

Rem

ote

Loca

l

Gateway GatewayNetworkLayer



MessagesAlarms

Examples for remote Operation: TTF

FEC FEC

BPM Magnets

Master

TimingLLRF

Cavities

Process Controller

PLC PLC PLC

VacuumCamera

DAQ

Rem

ote

Loca

l

Router

X-WindowProtocol

APP

?



Examples for remote Operation: TTF

Implementation:– X-Window Protocol

Advantage:– Existing Technology

Disadvantage:– Security Risk (Protocol)– Access only to local applications– Remote diagnostics with custom application not possible

Lessons learned:– Remote Operation is feasible– Experts can participate remotely



Examples for remote Operation: GAN-MVL

Camera

Rem

ote

Loca

l

Instrument

MVLServer

Router Access Control



Examples for remote Operation: GAN-MVL

Implementation:– GAN-MVL Projekt Group

Advantage:– Full Access to Local Instrument– Interactive Video and Audio Support– (Limited) Access to local control system

Disadvantage:– No transparent access to local control system

Lessons learned:– Setting up a project group dealing with control system issues



MessagesAlarms

Examples for remote Operation: SNS

Magnets

Master

Timing

CavitiesPLCPLC

Cryo

PPLC PLC PLC

Vacuum

DAQ

Rem

ote

Loca

l

Gateway

Router

FEC LLRF Process Controller

Process Controller

Commissioning

Commissioning remotely

Read access

Copying Applications

Remote Read Access


Process Controller



Examples for remote Operation: SNS

Implementation:– EPICS toolkit

Advantage:– Full Access to local Input Output Controllers (IOCs)– Running the identical applications local and remote for commissioning

Disadvantage:– Remote access using the native control system protocol opens a

security whole which requires additional security mechanisms in addition to access control locally on the IOC

– Only applicable in homogeneous control system environmentsLessons learned:

– Remote access to the local control system protocol eases commissioning and remote diagnostics



MessagesAlarms

Examples for remote Operation: LHC/CMS at Fermilab

FEC FEC

BPM Magnets

Master

TimingLLRF

Cavities

Process Controller

PLC PLC PLC

Utilities

Process Controller

PLCPLC

Process Controller

PLC PLC


P TT

Process Controller

PLC PLC PLC

VacuumCamera

DAQ

Rem

ote

Loca

l

RouterAuthentication

Server

User-ID/ RoleDatabase

Equipment-Access/Role- Database

•Introducing Common Authentication and Authorization•Based on Common Protocol

AuthenticationServer





LHC@FNAL Web Page



Unfortunate: CERN Cryo Status



If the contribution to the project from remote collaborators is exceeding a certain level, the commitment of the collaborating institutions beyond the constructionphase in commissioning, and operation is mandatory, because of the host laboratory will not be able to handle the whole facility with its own staff.

On the other hand, this commitment cannot be made by relocating the technical staff on the site of the accelerator

Far remote operating ( operating in the widest sense, that is including running the accelerator, performing maintenance, trouble shooting and repairs, tuning-up the hardware systems, maintaining and managing spare inventory, pushing performance, ) is required

The need for Far Remote Operating © F. Willeke

(underscore by M.Clausen)



Collaboration of Equal Partners (no “host” laboratory but “near-by” laboratory)Facility far away from most Collaborating institutionsEach collaborator responsible for major section of the machine incl. subsystems

design, construction, commissioning, maintenance, trouble shooting, developmentCollaborators remain responsible for the part they contributed after construction

Experts remain based at the home institutionMost of the activities via remote operating and remote access

Central Management responsible for the over-all issues, performance goals, design, interface, schedule, quality control, standards, infra structure, safety

Operation centrally organized: planning & coordination, commissioning, operation, maintenance, machine development

Operation performed by decentralized operations crews

Model for a Remote Facility © F. Willeke



MessagesAlarms

Proposed Layout of a Global Collaborative Control System

FEC FEC

BPM Magnets

Master

TimingLLRF

Cavities

Process Controller

PLC PLC PLC

Utilities

Process Controller

PLCPLC

Process Controller

PLC PLC


P TT

Process Controller

PLC PLC PLC

VacuumCamera

DAQ

Rem

ote

Loca

l

Gateway Gateway

Replica•No local consoles•No direct internet access•Authentication and Authorization Databases•Remote Replica•Heterogeneous Front End Controller•Local Infrastructure•Extendable ConsoleApplications•Common Protocol Router

AuthenticationServer




Process ControllerFEC Process

ControllerProcess

Controller

MessagesAlarms

Master

DAQ Gateway Gateway

RouterAuthentication

Server



C



The Proposed Layout provides:

• Transparent access from local and remote sites– No ‘primus interparis’

• Access control down to the equipment– Access roles and policies will be defined by all participating

partners

• Heterogeneous contributions are (maybe not desirable) but possible

• In-kind contributions are fully supported



Basic requirements for individual components in the GCCS

Client Applications– Running on all major operating systems -> Java– Reference Implementations for Authentication/ Authorization– Pluggable Applications– Support for –at least- the common control system protocol– Multi language support

Authentication/ Authorization– Role based Authorization Scheme throughout the whole system down to the front

end controller– Clean Interfaces to these services allow implementations on all platforms

Network Protocol– Serving as the common ‘glue’

Alarm-/ Message Systems– Using well define Message Based Implementations

Archiving/ DAQ– Supporting the common protocol

Front End Systems– (Common) Generic configuration Tools– Support for –at least- the common control system protocol



The Control System Protocol (TCSP)

How realistic is the development of an new common Control System Protocol?Status and Plans for Control System Protocols:System/ Institution Protocol Plans CommentEPICS Channel Access New protocol for

version 4.0Depends on available funding

TANGO Corba Interested in new protocol

TINE/ DOOCS Merged their protocols (TINE’)

TCSP candidate?

CERN Corba No known plans to change

ITER To be defined Partner for a new development?



Global Collaborative Control System Components

Component Status Candidate

Client Application(s) Requirements are already fulfilled by current developments

Starting from CSS(DESY) with an existing TINE’ plug

Authentication/ Authorization Examples for role based authorization are available from several sites (CERN, DESY…)

Check existing implementations. Define common interfaces and implementation for TCSP.

Network Protocol Collaborations are open for a change. A candidate is available.

Develop TCSP by starting from TINE’.

Alarm-/ Message Systems Many sites are already using Java Message Systems (JMS). CERN, DESY(2x), SNS …

JMS is a stable foundation. Agree on message structure

Archiving/ DAQ Many custom solutions are available…

Pick one (new?)

Front End Systems The ‘heart’ of each control system architecture.

Preserve investment by adapting to the new protocol.



Setting up a Collaboration on The Control System Protocol (TCSP)

The HEP community has a long tradition to set up collaborations.

(e.g. HEP experiments or the next large machine)

But

How to set up a collaboration on control system developments?



Brief History of Control System Collaborations and Sharing Control System Software

In 1993 Berend Kuiper presented his ideas to share control system software.– The ever lasting question how a magnet is controlled and how to model

a power supply is symptomatic for the fundamental questions – how control system software can be shared.

The discussion on a ‘software bus’ lead to an implementation by Chip Watson (cdev) in 1999.– The acceptance in the community was not high enough for a break

through. It was just too easy to use straight channel access.

Since then the EPICS collaboration was ever growing and became this way the biggest community sharing control system software.– The endless list of available drivers and the list of generic and

specialized applications are two of the success factors for EPICS.



Factors to form Control System Collaborations

Specialized control objects require device specific class libraries. This perfectly works in a single instance but does not match to many other installations.

Record or device oriented implementations leave enough freedom to the software engineers to adapt it to their individual installation.

Object orientation

Col

labo

rativ

e

lg100

1



Collaborative Development Process

Once a mutual agreement (between the HEP partners) on the basic functionality of common Control System Components has been reached…

How can this be promoted in the collaborations?– There’s nobody to write a check to– There’s no way to implant a new functionalities without taking

care of the interest of the whole collaboration.(Nobody is left behind)

– It is a long process with one final goal: A Win/ Win situation for the promoting bodies as well as the collaboration members

How would this work in the EPICS collaboration? (next page)



Collaborative Development Process –Example of a success storyDevelopment of a software package at DESY to support redundancy on front end controllers• Preparation phase ( middle of 2006):

– Define required redundancy functionality– New functionalities my not break existing code

• Start implementation based on existing EPICS version– Testing of first prototype– Parallel discussion with EPICS core developers how to add new features to the core code base

• Final tests of new version– Final discussion with core developers on integration

• Integration of redundancy features into EPICS core during an EPICS codeathon (EPICS core developers) in spring 2008

• New EPICS version includes redundancy supportClearly a Win/ Win situation:

EPICS collaboration: free access to redundancy supportDESY: can stay in synch with current and any new EPICS version

• Last EPICS meeting (Oct. 2008): – Presentation by KEK on a redundancy support for ATCA based on the DESY redundancy

implementationAdditional payoff for the collaboration and for DESY

The ideal case



How to start the collaborative development?

The probability to agree on a common protocol has never been as good as these days

(See discussion at last ICALEPCS)

It is very likely to agree on common Control System Components since many labs already share the same technologies

Client applications supporting multiple protocols are already available

Get the key players/ collaborations involved

Provide funding from major labs



It is a good time for an initiative

To form a Collaborationfor a common Control System Protocol

and other basic control system functions …

… funded by current Projects(And Projects in the near Future)

Outlook

Documents

Global Collaboration on Accelerator Operations and … · The European XFEL X-Ray Laser Project X-Ray Free-Electron Laser Matthias Clausen, DESY ICFA Seminar, 29.October 2008 Global