View
25
Download
0
Category
Preview:
Citation preview
Deliverable 7.3
Workflow Manager
Poznao Supercomputing and Networking Center
D7.3 Workflow Manager Page2
Table of contents 1 Introduction ............................................................................................................ 3 2 E-VLBI Correlation Platform ................................................................................. 3
2.1 Liferay introduction......................................................................................... 3
2.2 NEXPReS model ............................................................................................. 5
2.2.1 User Roles and Permission ...................................................................... 5
2.2.2 Experiments and Workflow portlets ........................................................ 5
2.2.3 Telescope, Correlator, external tools Portlets .......................................... 6
2.3 Visualization of the experiment ...................................................................... 6
3 Correlation testbed ................................................................................................. 8 4 How to conduct an experiment – short user manual .............................................. 9
4.1 System log-in................................................................................................... 9
4.2 Experiments page (and portal navigating) .................................................... 10
4.3 Experiment creation ...................................................................................... 11
4.4 Experiment’s workflow ................................................................................. 12
4.5 Experiment submission ................................................................................. 12
5 Software ............................................................................................................... 13
Table of figures
Figure 1. Fabric Workflow Manager Application ......................................................... 3
Figure 2. NEXPReS module architecture ...................................................................... 5 Figure 3. Example of correlation's graph ....................................................................... 7 Figure 4. WP7 testbed based on partner's resources ...................................................... 8 Figure 5. Sign in portlet ................................................................................................. 9
Figure 6. Welcome screen.............................................................................................. 9 Figure 7. Experiments page ......................................................................................... 10 Figure 8. Creating new experiment .............................................................................. 11 Figure 9. Workflows portlet ......................................................................................... 12
D7.3 Workflow Manager Page3
1 Introduction This document describes e-VLBI Platform – software developed within WP7 –
Correlation in a shared infrastructure. It contains a short overview of previous
approach prepared in EXPReS project, presents a new web based perspective for data
correlation, provides description of the current testbed and finally describes the
content of the software components.
2 E-VLBI Correlation Platform
The Workflow Manager Application (WFM) has been constructed in EXPReS project
within Fabric activity. The idea was to create a prototype system for software
correlation. The WFM was a main interface between users and the system and has
been constructed as a standalone Java window-based application.
Figure 1. Fabric Workflow Manager Application
User can create so called observation workflows based on an observation schedule.
The application allows also defining correlation parameters which controls the
behaviour of the software correlator. The completed observation workflow is then
submitted for further processing.
The role of WFM application within NEXPReS project has been significantly
extended. The WFM has been transformed from a single user application to a
multiuser and multipurpose application for all VLBI actors. The application became a
central place not only to manage and control a distributed correlation platform, but
also to monitor the entire experiment process.
PSNC has created an e-VLBI Correlation Platform which will be used within VLBI
community to manage and monitor experiments. The WFM application has been
rewritten using web based technologies and nested in the Correlation Platform as a
portlet.
2.1 Liferay introduction
Experiences from the EXPReS Fabric project lead to a new approach for handling e-
VLBI experiments. Instead of a single point of entry for Principal Investigator,
allowing only submission of the correlation using specified resources, focus was put
to facilitate whole process of setting up the observation, execution, monitoring and
validating correlation process. A need for a multi-user program arose. It should
D7.3 Workflow Manager Page4
provide various, pluggable tools and allow easy development, integration and
deployment of new applications. A research was done about available solutions (WP7
State of the Art on Web Portals and Web Technologies:
http://www.jive.nl/nexpres/lib/exe/fetch.php?media=nexpres:wp7_state-of-the-
art_on_web_portals_and_web_technologies-final.pdf) and Liferay Portal
(http://www.liferay.com) was chosen as a platform for NEXPReS project.
Liferay Portal is an enterprise web platform for building business solutions that
delivers immediate results and long-term value (from Liferay website). It contains
out-of-the-box set of portlets providing basic needs for a web platform – user
management, login functionality, alerts and announcements, collaboration tools such
as chats, documents and so on. It is also a typical portlet container, which allows
writing own applications and deploying them inside the portal. Deployed applications
can be added to portal pages as portlets – modular, pluggable user interface
components. Administrators can prepare and set up which sets of portlets will be
visible on each page of the portal.
A great deal of influence on deciding on portal solution was collaboration and multi-
user approach. Using provided user management tools we can model each type of
actor as a specific user role, create communities, organizations and user groups. For
each such unit a specific page or view can be created, containing only needed
applications. This allows to create a dedicated page layouts for different user types.
Cooperation between scientists during observation is fulfilled by adaptation of the
Liferay portlets, which provide a chat functionality, mail (also connected with
external mail services) and message boards. System generated information is also
displayed to interested parties using alerts and announcements portlet.
Crucial requirement of the Workflow Manager in Liferay portal was intuitive and
responsive user interface. From technologies supported by the portal, JSF (Java
Server Faces - http://javaserverfaces.java.net/) and ICEFaces
(http://www.icefaces.org/) were chosen. JSF is a Web application framework and
simplifies building user interfaces. It supports Model-View-Controller architecture
pattern and is written in Java. ICEFaces on the other hand extends JSF framework
with Ajax support, which allows development of rich internet applications.
Asynchronous updates from server to client is provided by Automatic Ajax and Ajax
Push functionalities. This allows to update data in the user’s browser without
reloading whole page, depending on the state of the model on the server. Objects
during e-VLBI observation will be constantly changing, updating the statuses of the
components, executing steps of the correlation process. This changes will be
immediately displayed to all interested users in their Liferay portal instance.
D7.3 Workflow Manager Page5
2.2 NEXPReS model
Figure 2. NEXPReS module architecture
2.2.1 User Roles and Permission
During design of the e-VLBI Platform we had to take into account the structure and
functionalities of the Liferay Portal. One of such feature is user support. After
evaluating NEXPReS use cases we have decided to create 5 NEXPReS roles:
PI – Principal Investigator
SS – Support Scientist
TO – Telescope Operator
CO – Correlator Operator
Admin – e-VLBI Platform Administrator
Each of the four user types (beside Admin) has an unique perspective in the portal –
providing default view, customized for this particular role, containing all portlets
required by the user. User roles also allow to enforce permissions on whole e-VLBI
Platform –e.g. general information about NEXPReS displayed for not logged or
unregistered visitors, displaying monitoring information of the correlation process to
logged, non-correlator operators and finally providing interactive controls for
correlator operators.
2.2.2 Experiments and Workflow portlets
In e-VLBI Platform information about experiments (VLBI observations) will be
displayed using two portlets: Experiments and Workflow. Experiments portlet displays
all existing observations in the system. It also allows to create new experiment, by
uploading VEX file describing e-VLBI observation and specifying additional
D7.3 Workflow Manager Page6
information. General information of the experiment (Status, time to start) is being
displayed to the user.
By selecting the experiment more information about the experiment is available. Most
important is graphical representation of the experiment in Workflow portlet (see
Figure 9. Workflows portlet). Interaction with the experiment is possible using this
view: before experiment starts it allows modifications of the whole process and during
the observation it provides monitoring information about experiment.
2.2.3 Telescope, Correlator, external tools Portlets
Pluggable architecture of e-VLBI Platform allows adding new modules, providing
new functionalities to whole application. Some of the distinguished portlets, which
will be needed during VLBI observation are:
Telescope Portlet - allowing Telescope Operators to operate and communicate
with their telescope module, others can monitor progress and status of
telescope operations during observation.
Correlator Portlet - allowing Correlator Operators set up of the correlation
process, specifying parameters of the correlation. It also allows monitoring of
the correlation process to all interested parties.
Fringe Plots – graphical module used to display fringe plots required to update
and adjust telescopes and correlation parameters
2.3 Visualization of the experiment
Initial experiment upload to Workflow Manager contains only a VEX file and
author’s information. Basing on this file the correlation representation is created by
Workflow Manager – both on server side and in user tools, to visualise the flow of
data and resources taking part in e-VLBI experiment.
On client side the convenient Workflows portlet was developed. It comprises
both Java portlet and Adobe Flash technologies, thereby addressing standards support
(Java portlet can be deployed in any portlet’s container) as well as platform-
interoperation (embedding binary Flash content).
Portlet embedded in Liferay portal uses JSF (Java Server Pages) GUI libraries
– “Java-based Web application framework intended to simplify development
integration of web-based user interfaces” (source: Wikipedia). JSF facilitates mapping
from/to web application to/from Java model classes on server and extended with
AJAX libraries (e.g. IceFaces) results with responsive and intuitive user interfaces.
On user selection in web browser, the VLBI experiment’s representation is
rendered inside Workflows portlet. This portlet is intended as a tool for graphical
design of e-VLBI workflow – a graph representing data flow within telescopes,
translation nodes and correlators during correlation process. e-VLBI workflow
visualises resources taking part in experiment (see Figure 3. Example of correlation's
graph) and - by connecting them with each other – user defines destination/sources of
data for services carrying out the correlation. Changes made in correlation’s workflow
are reflected in model classes and saved on server side. Designed graph can be edited
and parameters of experiment can be changed. When correlation time is met –
D7.3 Workflow Manager Page7
workflow’s modifications are no longer possible, and the experiment is started on
infrastructure.
Figure 3. Example of correlation's graph
Representation of experiment’s workflow is rendered in Workflows portlet
using Flex/ActionScript technologies, a web-enabled toolkit supporting visualisation
of graph-like data. To enable data synchronisation between server-side model and
Flash client application an Adobe’s BalzeDS messaging technology was used.
BlazeDS allows for RPC-based communication from ActionScript clients to server as
well as for PUSH-like transfer of data from server to clients. The data sent is in AMF
format – a binary protocol enhancing light and fast communication.
On the server side experiment is passed to Experiment Manager Unit (EMU) which
ensures coherence between correlation definition and infrastructure resources. EMU
schedules experiments to start automatically – the proper model objects of workflow
are created, passed to workflow management module. The correlation is started by
invoking web services of Fabric system.
D7.3 Workflow Manager Page8
3 Correlation testbed Resources assigned by the project partners to the NEXPReS project have been set up
to form a testbed for e-VLBI Platform developed by PSNC within work package. The
testbed consists of (see also Figure 4. WP7 testbed based on partner's resourcesFigure
4):
computational resources which will be used during correlation process,
translation. These have been provided by PSNC, Jive and Ventspils.
Translation services which serve the telescope data
Correlation services which handle correlation process
VLBI Broker deployed at PSNC to handle experiments
E-VLBI Platform – replacement of Fabric Workflow Manager
Figure 4. WP7 testbed based on partner's resources
A current version of e-VLBI Platform has been deployed on local installation at
PSNC. This local installation is available to all project partners. The idea is to keep
this software operable till the end of the project duration. This site will be updated
whenever a new version of the e-VLBI Platform is available. This will allow us to
conduct experiments at any time and also use this installation for demonstration
purposes.
D7.3 Workflow Manager Page9
4 How to conduct an experiment – short user manual This section describes actions needed to be taken by user in order to create
experiment and submit it for correlation. The process is taken from user point of view
and therefore presented in form of short manual illustrated with live e-VLBI Platform
system screenshots.
The general scenario when running a correlation with e-VLBI Platform is to log
into portal, navigate to page containing experiment related-portlets (management and
workflows) where new experiment can be created and its workflow designed. The
required user interaction ends at this point, as Experiment Manager Unit (EMU)
running on server side takes responsibility for running and monitor correlation
process. Steps needed to conduct experiment are described in more detail below.
4.1 System log-in
In web browser (preferably Mozilla Firefox 9.0+ or Google Chrome 14.0+)
navigate to e-VLBI Platform main page
Click “Log in” link in upper right corner
Figure 5. Sign in portlet
Provide your user credentials in the form that will be displayed (to obtain user
and password contact your e-VLBI Platform administrator)
On successful log in you will be redirected to portal’s welcome site –
displaying general information.
Figure 6. Welcome screen
D7.3 Workflow Manager Page10
4.2 Experiments page (and portal navigating)
Figure 7. Experiments page
After successful log in you are located on e-VLBI Portal’s main page. To get
back there (in case of any navigation problems) – click NEXPReS logo
located in upper left page corner.
Page on which experiment-related portlets are placed is called “Experiments”.
It is listed among other sites on top horizontal bar. To navigate to this page
just click on the page name. Other sites can be accessed the same way.
The Experiments site contains 2 portlets:
o Experiments – main entry point to manage all your experiments –
allows for creation and selection or experiment to be edited in other
portlets. For user’s convenience view is divided into 2 tabs, grouping
experiments by their status:
Active – the experiments that are to be started in the future
(status SCHEDULED) and currently running (status
RUNNING). Some of them may still require users’ interaction
(e.g. approving telescopes configuration or correlation
workflow customization)
Archive – on this tab user can access all experiment that are no
longer running
o Workflows – this portal is intended to present correlation in graphical,
user-friendly way in form of workflow’s graph. The important thing to
note is that workflows portal is not independent – it is connected with
Experiments portal. Content rendered in Workflows comes from
experiment selected in Experiments window.
D7.3 Workflow Manager Page11
4.3 Experiment creation
Figure 8. Creating new experiment
On Experiments site, in Experiments portlet press “Create new” button
An experiment wizard form is displayed containing:
o Name – provide name under which experiment will be accessible in e-
VLBI Platform,
o Description – provide a short description of the experiment,
o VEX file – upload of file containing observation definition,
o Start date – specify the time at which correlation process should be
started.
When all information in wizard are provided – press “OK”.
On successful experiment creation it will be displayed among other
experiments in “Active” tab of Experiments portlet, with status set to
SCHEDULED.
D7.3 Workflow Manager Page12
4.4 Experiment’s workflow
Figure 9. Workflows portlet
Select one of the experiments from list in Experiments portlet
In Workflows portlet a visualization of correlation graph will be presented
Detailed information on resources taking part in correlation process
(telescopes, translation nodes and correlation nodes) can be obtained by
selecting elements of correlation graph. Information will be presented in right
panel on portlet, listing resource’s name, abbreviation and description.
4.5 Experiment submission
When creating an experiment (see 4.3) a start date of correlation has to be
specified.
Experiments due to start in future are listed on “Active” tab of Experiments
portlet (for more details see 4.2), with status set to SCHEDULED.
No user interaction is needed to start correlation process at time specified –
Experiment Manager Unit (EMU) monitors start times of experiments and
takes responsibility for running it at given time.
When experiment has been started by EMU, its status changes from SCHEDULED to
RUNNING.
D7.3 Workflow Manager Page13
5 Software The table below presents the content of the attached CD-ROM. This includes all client
and server components of new e-VLBI Platform .
LP COMPONENT / MODULE /OTHER SHORT DESCRIPTION
Actionscript
1.
nexpresWorkflowFlex
ActionScript/Flex project that builds
graphical (Flash technology) part of
Workflows management system in e-VLBI
Platform. Displays infrastructure of the
experiment (radio telescopes, translation
nodes, correlation nodes) and properties of
each node that takes a part in work.
Common
1. kiwiCommon
KIWI common library – set of general
purpose utility classes
2. misc
Main project containing all related libraries
and Ant build script for e-VLBI Platform
3.
nexpresModel
Model for e-VLBI Platform i.e. to describe
VLBI experiment and to serve internal
persistence for infrastructure resources
description
Fabric
1.
Correlation Node
Manages all computations on the underlying
high performance computers. It provides an
abstraction layer for correlation
computations, hence the WfM will be able
to work with both different hardware
infrastructures and different software
correlator.
2. expresRemote
Remote client stubs for communication
purpose between modules
3. misc
Project containing all related libraries and
Ant build script for Fabric components
4. vlabCommon
VLab common library – set of general
purpose utility classes
5.
VLBI Broker
Handles correlation process on the server
side: “talks” to Translation Nodes and
Correlation Nodes decides when to submit
new portion of data for the correlation.
Portlets
1.
nexpres-portlet
Contains GUI portal parts – all portlets
composing NEXPReS e-VLBI Platform –
JSF resources and web toolkit libraries. This
module covers also Flex resources and
configuration files.
D7.3 Workflow Manager Page14
Workflows
1.
experimentManager (EMU)
Module responsible for management of
experiments. All newly created experiment
should be registered with EMU. It monitors
for experiment-related events and notifies all
interested parts. EMU is responsible for
starting experiment at time given.
2.
Workflow Manager (WfM)
WfM handles all actions related to
correlation workflow – translates data from
experiment description into workflow
format and – after experiment start –
monitors workflow execution.
The current versions of the software components is always available at workpackage
SVN repository hosted by PSNC. The repository location is as follows:
kiwi.man.poznan.pl/svn/nexpres.
Recommended