Upload
melinda-hill
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
The GRIDCC Instrument Element: from prototypes to production environments
Roberto Pugliese On Behalf of the GRIDCC Collaboration
EU FP6 Project
www.gridcc.org (EU FP6 Contract 511382)
1. The GRIDCC - Grid Enabled Remote Instrumentation with Distributed Control and Computation – project has the main aim to bring Instruments to the GRID
2. It is a 3-years EU FP6 project started in September 2004
3. Web site: www.gridcc.org
4. More in particular the project goals are:1. Definition of a “Instrument Element” allowing a standard remote
access to any type of instrumentation
2. Tight integration between instrument grid and classical computational grid
3. Human interaction with Grids via Virtual Control Room (collaborative environment)
4. Enactment of complex workflows
The GRIDCC project: Goals & Objectives
www.gridcc.org (EU FP6 Contract 511382)
The GRIDCC partners
Participant name Country
Istituto Nazionale di Fisica Nucleare Italy
Institute Of Accelerating Systems and Applications Greece
Brunel University UK
Consorzio Interuniversitario per Telecomunicazioni Italy
Sincrotrone Trieste S.C.P.A Italy
IBM (Haifa Research Lab) Israel
Imperial College of Science, Technology & Medicine UK
Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale delle Ricerche
Italy
Universita degli Studi di Udine Italy
Greek Research and Technology Network S.A. Greece
www.gridcc.org (EU FP6 Contract 511382)
Execution ServicesCollaborative Environment
GRIDCC: Architecture (I)Instruments Grid Computational Grid
IE
IE
CE
CE
SE
Instrument Element
Problem Solver
InstrumentManager Information &
Monitoring System
DATA
ES
VCR
VCR
VCR
VCR
Workflow
Service Oriented Architecture – SOA
The GRIDCC services are Web Services compliant, according to the WS-I convention.
Any type of client WS-I compliant, can access the GRIDCC services without any specific software library.
X509 based security is used When performances are
required , a kerberos based mechanism is used to access IEs.
www.gridcc.org (EU FP6 Contract 511382)
GRIDCC Architecture
Web ServiceInterface
Execution
Service
WfMS
WMS
AS
Instrument elements
(IE)StorageElement
(SE)
Instrument elements
(IE)StorageElement
(SE)
Instrument Element
(IE)StorageElement
(SE)
Compute element
(CE)
Compute element
(CE)
Computing Element
(CE)
StorageElement
(SE)
StorageElement
(SE)
StorageElement
(SE)
Global ProblemSolver
Information and Monitoring
Services(IMS)
Information System(BDII)
Security Services
Virtual Control
Room
Virtual Control
Room
CollaborativeServices
(CS)
WMS Work Management System
WfMS Work Flow Mng System
AS Agreement Service
Broker
www.gridcc.org (EU FP6 Contract 511382)
GridCC PermanentTest Bed
VOMSVOMS
AS, KrbServerAS, KrbServer
VCR, SE, IE, BDIIVCR, SE, IE, BDII
IE, CE/CREAM,SE, LFC IE, CE/CREAM,SE, LFC
WMProxyWMProxy
WfMS WfMS
VCR, IE VCR, IE
www.gridcc.org (EU FP6 Contract 511382)
GRIDCC main target areas(Large-scale) scientific
experimentsHigh energy particle physics
(Radio-) Telescopes
Remote process control Accelerator control (Tele-) Biomedicine
Robotics Automotive
Electronic microscopes
Widely Sparse Instrumentation Power Grids
Monitoring of the territory Monitoring of the seaGeo-hazard prediction
Distributed laboratories Transportation monitoring
Sensor network
GRIDCCMiddleware
www.gridcc.org (EU FP6 Contract 511382)
GRIDCC pilot applications
Power Grid
Power Grid
High Energy Physics
CMS
High Energy Physics
CMS
Particle Accelerator
Particle Accelerator
Device Farm
Device Farm
Geohazard MonitoringGeohazard Monitoring
MeteorologyMeteorology
www.gridcc.org (EU FP6 Contract 511382)
Accessing GRID Enabled Instrumentation: Requirements
1. Interactive access to allow the control and the monitor of the instrumentation. Uniform access to the physical devices.
2. Fast Data Publishing to allow the publishing acquired by the instrumentation
3. Information (logs, errors, etc.) Publishing to track the behaviour of the instrumentation and possibly to correct the malfunctioning
4. GRID integration. The acquired data should be visible by the GRID protocols both to :
1. Move them to large GRID data storage (Storage Element)
2. Process them from the GRID computing power (Computing Element)
www.gridcc.org (EU FP6 Contract 511382)
A new 4 channels element to fit the requirements: The Instrument Element (IE)
Inst
rum
enta
tion
Con
torl
GR
ID A
cces
s
Instruments Access
Data/Info Pubblishing
InstrumentElement
Web
Ser
vice
SR
M/G
RID
FT
P
Custom (JMS, ..... )
Custom
I’m here
Discovery
QoSAdvance Reservation
www.gridcc.org (EU FP6 Contract 511382)
The IE permits SOA accessible operations to control and monitor the instruments (via VIGS), such as:
• execute a command• get / set parameters
different data outputs:• data mover to/from a grid Storage Element (via SRM) • high bandwidth channel for data publishing (via IMS).• low bandwidth channel for logs, states etc. (via IMS).
Instrument Element: Versatile I/O multichannel
VIG
SCommands
Status
Data Subscribers
Grid Interaction
SR
M
VCR
StorageElement
(SE)
StorageElement
(SE)
StorageElement
(SE)
Logs, Errors,
States, Monitors
Instruments
IMS
ParametersES
VIGS Virtual Instrument Grid ServiceIMS Information and Monitor ServiceSRM Storage Resource Management
IE
www.gridcc.org (EU FP6 Contract 511382)
VIG
S
IMSProxy
ControlManager
DataCollector
Real Instruments
Data Flow
Control Flow
State FlowError FlowMonitor Flow
The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments.
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control Manager
IMSInstrument Element Architecture
ResourceService
IMS
ProblemSolver
InstrumentManager
Instrument Element
Data Publisher
Acc
ess
Con
trol
Man
ager
SRM/SEGridFTP
SE
IMS
Instrument Protocols
www.gridcc.org (EU FP6 Contract 511382)
Configuration Examples
Top IM
ResourceService
IMS
ProblemSolver
InstrumentManager
Instrument Element
Data Publisher
Acc
ess
Con
trol
Man
ager
SRM/SEGridFTP
SE
IMS
Top IM
Top IM
IMIM IM IM IM IM
Resource Service Instruments Pool
A B C
Configuration A, B, C can run concurrentlysharing services like RS, PS, IMS
www.gridcc.org (EU FP6 Contract 511382)
Instrument Element Framework Characteristics
The IE framework is modular, fully customizable and can be adapted to different environments and different type of instruments to be controlled.
Customizable plug-in modules are used to interface the framework with the communication protocol (both hardware and software) used by the instruments.
the dynamic behaviour of the control that has to be performed is programmable via the built in Finite State Machine (FSM).
relatively fast control feedback loops can be provided using the customizable built in Event Processor (EP).
Error repairing actions due to malfunctioning of the instruments can even be handled by EP or by the Local Problem Solver (LPS).
IE
ResourceService
IMS
LocalProblem
Solver
InstrumentManagerA
cces
s C
ontr
ol M
anag
er
DataMover
InstrumentManager
VIG
S
InstrumentInstrumentInstrumentInstrument
IMSProxy
ControlManager
DataCollector
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control Manager
www.gridcc.org (EU FP6 Contract 511382)
VIG
S
IMS
Instrument Manager
ResourceService
IMS
ProblemSolver
InstrumentManager
Instrument Element
Data Publisher
Acc
ess
Con
trol
Man
ager
SRM/SEGridFTP
SE
IMS
InstrumentManager
IMSProxy
ControlManager
DataCollector
www.gridcc.org (EU FP6 Contract 511382)
16
Instrument Manager
IM is composed by 3 main components:- Control Manager:
- Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs,errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability
- FSM. A finite state machine is implemented - Resource Proxy. It handles all the outgoing connections with the resources.
- Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen.- IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS
IMSProxy
DataCollector
Instrument Manager
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control Manager
Instruments
Data Flow
State Flow
Error Flow
Monitor Flow
Control Flow
Customizable Plug-in modules to interface to the instruments
www.gridcc.org (EU FP6 Contract 511382)
Status• data collection • data publishing on RMM-JMS • data on file in progress• data on DB to be started• sync commands (enable/disable done trough the “set param” of the IE façade (VIGS)
Data Collector
ResourceService
IMS
LocalProblem
Solver
InstrumentManager
Instrument Element
IMSProxy
ControlManager
DataCollector
Acc
ess
Con
trol
Man
ager
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control Manager
DataMover
VIG
SIE File
System
RMM-JMSDB persistency
enable/disabledata stream
InstrumentInstrumentInstrument
Instrument
data stream
Data Subscribers
(Monitor, d
ata analysis, etc.)
www.gridcc.org (EU FP6 Contract 511382)
VIG
S
IMSProxy
ControlManager
DataCollector
IMS
Resource Service
ResourceService
IMS
ProblemSolver
InstrumentManager
Instrument Element
Data Publisher
Acc
ess
Con
trol
Man
ager
SRM/SEGridFTP
SE
IMS
ResourceService
www.gridcc.org (EU FP6 Contract 511382)
Resource Service Architecture
The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). A resource can be any hardware or software component involved in the IE (instruments, Instrument
Managers, IMS components) RS stores the configuration data of the resources and download them to resource target when
necessary Resources can be discovered, allocated and queried. It is the responsibility of the RS to check resource availability and contention with other active
partitions when a resource is allocated for use. A periodic scan of the registered resources keeps the configuration database up to date.
DiscoveryManager
SubscribeManager
Partition&LockManager
ConfigurationManager
Available Resources
PartitionDefinitions
ConfigurationDefinitions
RS
Dat
a B
ases
Partition/Configurationretrieve methods
Partition and Locksetting methods
Configurationsetting methods
Discoverymethods
www.gridcc.org (EU FP6 Contract 511382)
VIG
S
ControlManager
DataCollector
IMS
Information and Monitor Service
ResourceService
IMS
ProblemSolver
InstrumentManager
Instrument Element
Data Publisher
Acc
ess
Con
trol
Man
ager
SRM/SEGridFTP
SE
IMS
IMS
IMSProxy
www.gridcc.org (EU FP6 Contract 511382)
21
Information and Monitor System (IMS)
The Information and Monitor Service (IMS) collects messages and monitor data coming from the IMS Proxy of the IMs.
The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms.
The IMS collects and organizes the incoming information either in a database or a pub/sub system or both. It publishes the data to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level.
IMSProxy
IMSProxy
IMSProxy
PE
RS
IST
EN
CY
Pub
lish
/ S
ubsc
rib
SU
BS
CR
IBE
RS
SU
BS
CR
IBE
RS
www.gridcc.org (EU FP6 Contract 511382)
IMS Fast Data Publishing: RMM-JMS
IE
IE D
ata
Pro
duc
er
Dat
a S
ubsc
ribe
rs Same data are sent to several subscribers.Multicast protocols can have a benefic impact on performances
• JMS provides a standard set of APIs for the communication• Many commercial and academic JMS implementations both in C/C++ and Java (NaradaBrokering, Sun, IBM)• GRIDCC (IBM Haifa lab) has implemented a Reliable Multicast protocol (RMM) JMS compliant• RMM-JMS works within a LAN but an efficient bridge technology has been developed to allow inter-LAN multicast communication
www.gridcc.org (EU FP6 Contract 511382)
Status: installed and running• SRM 2.2 • STORM implemetation (INFN)• Bestman implementation (Berkeley))• XFS IE file system
Embedded SRM compliant Grid Storage Element (SE)
ResourceService
IMS
LocalProblem
Solver
InstrumentManager
Instrument Element
Acc
ess
Con
trol
Man
ager
SEBackend
VIG
SIE File
System
SR M GRID
accessibile data
www.gridcc.org (EU FP6 Contract 511382)
Querring Client
Querring Client
Discovery Instruments (I) The discovery of the instruments or of the IEs is an issue
when the number of elements is high We can have two cases:
Quasi static cases The number of IEs are well defined and the single IE is quite complex with
a good hardware support In this case a register based discovery mechanism can be used. GRIDCC tesbed is using the LCG BDII (Berkely Database Information
Index ) based on LDAP The information collected in the BDII follow a GLUE schema and can be
used for match making querries
IE
IE
IE
BDIILocal BDII
Querring Client
www.gridcc.org (EU FP6 Contract 511382)
Discovering Instruments (II)
Dynamic cases The number of IEs can change very quickly, they are very simple devices,
often with poor hardware support The discovery is just use to know which are the online IEs A new approach has been developed based on Peer to Peer (P2P)
protocols See demo on this workshop
IE
IE
IEIE
M1
M2
Querring Client
Querring Client
Querring Client
www.gridcc.org (EU FP6 Contract 511382)
The GridCC middleware has been deployed to control the run of the CMS (Compact Muon Solenoid), one of the four high energy experiments in LHC (Large Hadron Collider) at CERN laboratory.
CMS Magnet Test and Cosmic Challenge (MTCC) I and II, a milestone in the CMS construction, positively carried out.
Pilot applications: CMS - I
Top IE
CSCIE
TrackerIE
HCALIE
DAQIE
RPC IE
TriggerIE
DTIE
ECAL IE
DAQIM
FB RB FF
xdaq
xdaqxdaq
DAQRS
DAQIMS
UserInterface
CMS Control Structure
CMS Detector
www.gridcc.org (EU FP6 Contract 511382)
CMS MTCC phase I and phase II carried out in 2006
Scale MTCC versus CMSFEDs: 20 out of 600 3%EVB RUs: 6 out of 600 1%Filter Farm: 14 out of 2000 0.3%Trigger rate: 100 Hz out of 100 kHz 0.1%Event size: 200 kB out of 1 MB 20%IEs 15 out of 150 10 %
Scale MTCC versus CMSFEDs: 20 out of 600 3%EVB RUs: 6 out of 600 1%Filter Farm: 14 out of 2000 0.3%Trigger rate: 100 Hz out of 100 kHz 0.1%Event size: 200 kB out of 1 MB 20%IEs 15 out of 150 10 %
www.gridcc.org (EU FP6 Contract 511382)
Pilot applications: Power grid
Instrument Manager
Instrument Element
...
Virtual Control Room
Virtual Control Room
Gas
Solar Panel
Power Grid V.O.
GRIDCC deployed to monitor:• A 50kW generator• A 1 kw Photo-Voltaic array
Power Grid. GRIDCC deployed to monitor a 50kW generator and a 1 kW Photo-Voltaic array
See D
EMO o
n this
work
shop
www.gridcc.org (EU FP6 Contract 511382)
Pilot applications: Remote Operation of an Accelerator
Elettra Synchrotron
See D
EMO o
n this
work
shop
www.gridcc.org (EU FP6 Contract 511382)
Meteorology (Ensemble Limited Area Forecasting) Weather forecasting system to detect hazardous weather
Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories The Device Farm consists of a pool of Measurement
Instruments for Telecommunication Experiments
Geo-hazards: Remote Operation of Geophysical Monitoring Network The monitoring net will be characterized by different levels of
activity: stand-by, pre-alert, alert, plus a control modality An event worth to be monitored is for example the evolution
of a “landslip”
GRIDCC: other applications
www.gridcc.org (EU FP6 Contract 511382)
Instrument Element: Scalable on embedded systems
FPGAPPC405
Xilink Virtex IV
Custom Board
Cus
tom
Ele
ctro
nics
1 Gbps Ethernet
Web
Ser
vice
Grid
CC
IM
Java
VM
Linu
x
USER INTERFACE
Custom Logic
Montavista www.montavista.com
JamVM http://jamvm.sourceforge.net
Standalone Axis
IEInstrument Manager
www.gridcc.org (EU FP6 Contract 511382)
IE technologies Web Service compliant (WS-I) Tomcat + Axis (and Java) and Axis standalone are the main
technologies of the IE All the services are deployed on a single or multiple instances
of Tomcat, according to the needs of the application Message oriented middleware (Pub/Sub) is based on the
Java Messaging System (JMS). The following implementations are used in the project Sun Narada Brokering RMM - JMS (GridCC IBM)
MySQL and Oracle are used as Data Base for the RS SRM interface version 2.2 used
www.gridcc.org (EU FP6 Contract 511382)
Conclusion - I The GridCC project is integrating instruments into the
“classic” computational/storage Grids.
One of the novel concepts introduced by GridCC are: The Instrument Element, allowing a virtualisation of the
instruments to be controlled and their insertion in a Grid hard (reservation of IE) and soft (statistical prediction)
guarantees of the IE’s methods execution times Fast Data Publication via a Message Oriented middleware
(RMM-JMS) to distribute data and information from an IE to the world wide Grid.
The IE is highly customizable and can be adapted in different environments.
www.gridcc.org (EU FP6 Contract 511382)
Conclusion - II 3 pilot applications deployed, one of them in a real
production environment The IE can be shrunk down into a chip allowing grid
enabled embedded control of the instrumentation Several heterogeneous pilot applications are
deploying and running the IE middleware We support and encourage the adoption of our
middleware in other projects/experiments.