Canadian Light Source

cience Studio

Canadian Light Source

Elder Matias (CLS)EPICS April 09 Collaboration Meeting

Index

Where is the CLS? CLS – Current projects ScienceStudio ANISE – Data streaming expansion to

ScienceStudio

Where is Saskatoon?

• 1964 Saskatchewan Accelerator Lab (SAL) was established for chemistry and nuclear physics research.

1964 Saskatchewan Accelerator Lab (SAL) was established for chemistry and nuclear physics research.

1999 SAL operations were discontinued, and the accelerator was refurbished as an injector for a new storage ring.

SR1 Machine Parameters 2.9 GeV DBA lattice with 12-fold period170.88 m

Phase 1 (operation): Total of 7 scientific

and 2 diagnostics beamlines.

Phase 2 (commissioning): Total of 7 additional

beamlines and building expansion for

medical imaging. Phase 3 (pre-design):

Total of 8 additional beamlines and

building expansion.

Major Projects

Common Control Server virtualization

Accelerator Linac upgrade - Replace/upgrade Linac Modulators, RF

Sections, Power Supplies Transverse feedback and bunch cleaning Upgrade to existing power supplies Injection with shutters open

Beamlines Phase 2 Beamlines – Final commissioning Phase 3 Beamline Projects including a building expansion ScienceStudio

vmWare ESX Virtualization

Network

Redundant Network Connection

VM Physical Node VM Physical Node

SAN

EMC Storage Array

ISCSI ISCSI

VM Virtual CentreEnvironmet

Redhat

ScientificLinux

Windows

VM Virtual CentreEnvironmet

ScientificLinux

Windows

cience Studio

ScienceStudio

Elder Matias (CLS)EPICS April 09 Collaboration Meeting

What Is Web 2.0?

In plain English …. Automating tedious tasks using web technology Tools to help people and software collaborate

Scientific American May 2008Science 2.0 – The Risk and Reward of Web-Based Research

---------------------------------

“Our real mission isn’t to publish journals but to facilitate scientific communication” Timo Hannay – Head of Web Publishing at Nature Publishing Group

Traditional Beamline Controls

EPICS Tools Configuration Tool User GUI & ROOT

User Access to Synchrotrons

What middleware was needed to resolve the limitations? Workflow management Engine for the User Office Web Portal for remote data access (during and post

experiment) Enterprise Service Bus and SOA to integrate internal and

external data analysis services How do your plans meet the needs

Users will have frequent remote access to the VESPERS beamline at the Canadian Light Source under conditions where many collaborators can participate in the experiment.

13

Science Studio serves three purposes:

Management of all aspects of a scientific experiment including data storage, collaboration with others, processing of data;

Control of, or interaction with, remote experiments on the CLSI VESPERS Beamline and UWO Nanofabrication Laboratory and

User Services (sample management, scheduling, peer review, user training)

14

15

Team: People and Orgs

Remote Control

User Services

System Deployment

Integration

System Architecture

System Requirements

Testing

Data Analysis/Grid Computing

User Office Software

Scientific Workflow Engines

16

Team: People and Orgs

Dionisio Medrano

Dylan Maxwell

Daron Chabot

Elder Matias

Chris Armstrong

John Haley

Mike Bauer

Stewart McIntyre

Marina Suominen Fuller

Jinhui Qin

Nathaniel Sherry

Yuhong Yan

Zahid Anwar

Ludeng (Eric) Zhao

Dan Ni

Yaofeng Xu

System Architecture

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

1. VESPERS Beamline2. EPICS control system3. Beamline Control Module (BCM)4. Web Application5. Database6. File Storage7. Web Interface

VESPERS Beamline

VESPERS — Very Sensitive Elemental and Structural Probe Employing Radiation from a Synchrotron

A bending magnet beamline on sector 6 at the Canadian Light Source synchrotron in Saskatoon, Saskatchewan.

A hard x-ray microprobe with an energy range of 6 to 30keV.

Techniques: X-Ray Fluorescence (XRF) & X-Ray Diffraction (XRD)

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

VESPERS Endstation

CCD Detector (XRD)

Microscope

MCA Detector (XRF)Sample

EPICS Low-level Control System

EPICS — Experimental Physics and Industrial Control System

The standard control system at the CLS.

EPICS consists of a network of Input-Output Controls (IOCs) which are connected to directly to devices.

An IOC provides many Process Variables (PVs) which relate to either an input or output from a device and have a unique name.

Channel Access (CA) is used to read or write to any PV without knowing which IOC provides the PV.

More than 50,000 PVs in the CLS control system.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

EPICS at the CLS

ProfibusTCP/IP

Siemens S7/300 PLC

ModbusTCP/IP

GPIB

RS-232

Channel Access Protocol

IOC

IOC

IOC

IOC

State MachineEngine

CA

CA

CA

CA

CA

CA

Operator WorkstationUser Applications

TouchPanels

CA

CA

Telemecanique Momentum PLC

VME

IOC

CA

Beamline Control Module (BCM)

The BCM provides a high-level interface to the low-level control system (EPICS).

Logical and physical separation of business logic and control logic.

Virtual device abstraction that provides independence from low-level control system.

Virtual devices can be logically organized into a device hierarchy.

Basic devices can be combined to build more functional devices.

Communication with external applications using two message queues (ActiveMQ).

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

Web Application

A J2EE Servlet application that provides a web-based interface Science Studio.

Tools: Spring (MVC), iBATIS (ORM), JSecurity (Apache Ki), Apache Tomcat

Divided into two parts: the Core application and the VESPERS beamline application.

Core application is responsible for providing access to the business objects.

VESPERS application is responsible for remote control of the VESPERS beamline.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

Database

Metadata associated with the operation of a remote controlled beamline and the organization of experimental data collected on that beamline.

A project is the top level organizational unit and is associated with a project team.

A session defines a period of time allocated to a project team to conduct experiments.

An experiment relates a sample and the technique being applied to that sample.

A scan records the location of the acquired experimental data.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

Database Schema

person

project_person

project_role

project

session

laboratory

sample experiment scan

techniqueinstrument

Instrument_technque

facility

Experimental Data Storage

Experimental data is stored at the CLS.

Common directory structure shared with other beamlines. A large data storage facility is now operational at the

University of Saskatchewan as part of WestGrid.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

VESPERS Web Interface

Rich web interface to Science Studio and the VESPERS beamline.

Designed to be used over commodity broadband internet.

Developed for the Firefox web browser without any additional plugins or extensions.

Known to work with other browsers, but requires the Canvas HTML tag.

AJAX is used for the VESPERS interface to provide device values in pseudo real time.

ExtJS, a JavaScript framework, provides many advanced GUI elements.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

Beamline Setup

Experiment Setup

XRF (X-Ray Fluorescence)

Beamline Hutch Cameras

Experimental Data Viewer

User Office Workflow

Goal: Many tasks in proposal & sample management at CLS To develop a workflow management system that

manages ordering of tasks e.g. (training before shipping) Tracks manual as well as SS task progression

Mar6-month cycle

CLS call for proposals

Proposal submissionTo CLS

CLS gathers proposals

CLS reviews proposals

CLS grants scientistBeamline time

cientist packs sample

I wonder if CLS

received my sample

yet?

Scientist must complete Online SS training

CLS health & safetyinspection

Many other tasks•Perform Experiment•Return Sample•Take Survey•…

User office Workflow StatusWorkflow Management Engine

Beamline User

User Office

Task :Training

Completed

Notify

ApprovedNotify

PersistentStore

Record Progress

Features Open source Petri-nets based Direct support for workflow

control flow patterns Ability to interact with web

services declared in WSDL Relies on XML standards e.g.

XPath and XQuery for data & doesn’t use proprietary languages

Architecture System Core: YAWL engine. Engine instantiates specifications

designed using YAWL designer. managed by the YAWL repository

Environment composed of YAWL services inspired by “web services”

paradigm, end-users, applications, and

organizations are all services in YAWL.

Screenshot: User Training Test Creation

Screenshot: User Survey Taking Page

Screenshot: User Survey Edit Page

Screenshot: Workflow Sample Management

Screenshot: Workflow Call for Proposals

User Office Workflow Example

Prototype Implementation1. CLS issues a call for proposals and gives deadline2. Beamline users submit proposals3. User Office administrator ends registration or extends deadline4. User Office administrator assigns proposals to user office reviewers5. Reviewers look at proposals and rank them6. User Office looks at ranking and chooses the proposals to accept7. Accepted proposals contact persons are notified8. Beamline User completes training (web service)9. After training is completed (simulated by a delay) the CLS is notified

Scheduling Module

Goal: To automate the review process and the method by which beam time is allocated and scheduled to users depending on the access mechanism chosen by the user and the stage of operation (construction, commissioning or

operation) of the beamline. Side effects:

Facilitate the management of cycles, runs and modes of operation

Use automatic scheduling to handle more scheduling conditions and constraints than human beings are able to handle manually and identify optimal solutions.

Scheduling Module Features

Users Submit proposals

Integer Programming and

Heuristic Algorithm

Schedule

INPUT:

SEARCH ANDCONSTRAINTSATISFIABILITY:

OUTPUT:

Beamlines 2

Experiments 3

Release Times [1,1,2]

Deadlines [8,15,5]

Weights [4,5,1]

Processing Times [10,4,3]

Eligibility [[0,1,0],[1,0,1]]

CONSTRAINTS1. One beamline per experiment2. Start time after release time3. Only eligible beamlines can be selected..7. No overlap of experiment per beamline

X-Ray Fluorescence (XRF): Reveals Elemental CompositionCharacteristic Element Lines Selected and Mapped Over a 2D Scan Area

S: Kα Cr: Kα & Cr: Kβ Fe: Kα & Fe: Kβ Ni: Kα & Ni: Kβ

2D Maps Generated for Selected Elemental Lines

X-Ray Diffraction (XRD): Reveals Structural InformationPeak Fitting and Indexing of Image Set to Create a Grain Orientation Map

Peak Search

Old IDL Programme – Matched Peak

Old IDL Programme – Matched Peak

New C Programme – Matched Peak

New C Programme – Expected Peak

The XRD Indexing programme examines the locations of peaks in an image in order to determine the kind of lattice structure the samples constituent atoms are arranged in. Shown here are the results of an older indexing programme written in IDL, and the new indexing programme, written in C. The new indexing programme is proving to be more versatile, and more reliable than the old programme, often indexing sets of data that the old programme failed with.

Grain Orientations

Indexing Process

How do I get access to a HPC Machine?

Compute Canada Responsible for High Performance Computing in Canada Each regional grid is a member of Compute Canada

ACEnet – Atlantic Canada CLUMEQ - Quebec SCINET - UofT HPCVL – Queens, Royal Military Collage

St. Lawrence, Carlson, Ottawa, … RQCHP - Quebec SHARCNET - Ontario WESTGRID – Western Canada

Grid Data Storage?

UofS is the host for the new WestGrid data storagefacility

Cost: $3.2 M Includes on-line and

archival storage Two sites on campus

Photo: tape backup unitholding 6,000 tape(each @1TB)

IBM Cell Processor (3.2 GHz)

cience Studio

ANISE

Elder MatiasCLS – 09-04-28

50

ANISE: Active Network for Information from Synchrotron Experiments

“Active” means near-instantaneous stream processing of complex data during transfer to the user or to storage.

Cell processing using Infosphere Streams software from IBM and lightpath provided by CANARIE network.

Distributed processing on facilities provided by SHARCNET and WESTGRID.

Objective: Develop such a network to provide processed results from experiments such as Laue diffraction at APS (34-ID) and VESPERS at CLS

The network would assist the integration of diffraction data from multiple and large area detectors.

The network would facilitate faster resolution of research problems and free up time for more users.

The network would encouage common data formats and protocols leding to closer collaboration.

51

ANISE: Active Network for Information from Synchrotron Experiments

Some project outcomes:

1 Accessibility of Laue diffraction methods to a greater number and variety of users could be achieved by reducing the time required to accumulate meaningful data.

2 The results of complex diffraction measurements involving a wider segment of angles could be assessed rapidly.

3. Data and experiment management processes of Science Studio could enable very brief follow-up experiments to answer crucial questions sometime later.

4. Distant collaborators could participate in, and learn from experiments on samples of critical importance to a project.

5. User support software could result in more rapid publications.

6. Expansion to include APS and NSLS beamlines.

X-Ray Fluorescence (XRF): Reveals Elemental CompositionCharacteristic Element Lines Selected and Mapped Over a 2D Scan Area

S: Kα Cr: Kα & Cr: Kβ Fe: Kα & Fe: Kβ Ni: Kα & Ni: Kβ

X-Ray Diffraction (XRD): Reveals Structural InformationPeak Fitting and Indexing of Image Set to Create a Grain Orientation Map

The XRD Indexing programme examines the locations of peaks in an image in order to determine the kind of lattice structure the samples constituent atoms are arranged in. Shown here are the results of an older indexing programme written in IDL, and the new indexing programme, written in C. The new indexing programme is proving to be more versatile, and more reliable than the old programme, often indexing sets of data that the old programme failed with.

Peak Search Indexing Process

Grain Orientations

Apply to Entire Data Set

2D Maps Generated for Selected Elemental Lines

VESPERS Beamline Experimental Setup

Sample

Beam

XRD Area Detector

XRF OutputXRD Output