Upload
duncan-dangerfield
View
216
Download
1
Tags:
Embed Size (px)
Citation preview
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481
Science Gateways for improving the uptake of Grid and Cloud Computing Infrastructures
Tamas Kiss, Stephen Winter, Gabor Terstyanszky, Thierry Delaitre University of Westminster
Wibke Sudholt, CloudBroker GmbH
Peter Kacsuk, Zoltan Farkas, SZTAKI
Jens Krüger, Sandra Gesing, University of Tübingen
Using Distributed Computing Infrastructures is not so easy
gsissh -p 2222 grid-compute.oesc.ox.ac.uk
globus-job-submit grid-compute.oesc.ox.ac.uk/jobmanager-pbs -q RXXXXX /bin/hostname -f
globus-job-status <uid>
globus-job-get-output <uid>
globus-job-clean <uid>
?!
or is it?
• SCI-BUS is a 3-year EU FP7 project• Started on 1st October 2011• Build around 30 science gateways to various user
communities in Europe• 10 academic and 5 commercial partners from 10
European countries.
SCIentific Gateway Based User Support
Main objectives of SCI-BUS
• Create a generic-purpose gateway technology • Elaborate an application-specific gateway building technology
and customisation methodology• Establish production gateway services both for NGIs and
various user communities• Provide seamless access to major computing, data and
networking DCIs and services including supercomputers, clusters, grids and clouds
• Create and maintain a Liferay portlet repository that enables the quick creation of user specific customised gateways
• Provide gateway development and application development support
• Develop business models to guarantee the sustainability of the gateway services and to enable the commercial exploitation
Value brought by the project
SCI-BUSGeneric-Purposegateway
Helioport EPOS MoSGrid BioGrid
Blender ETICS e-Group SimSoft
AMChospital
PireGrid
SHIWAWF
repository
EGI/EDGIAppl.
repositories
SCI-BUSAppl.
repositoryHU
UK
D
TU
B
CH
BG
IR
SP
NGI GatewaysLiferayPortlet
repository
SimSoft
Customized gateways
SCI-BUS will serve large number of user communities by providing generic-purpose and customized scientific gateways for them.
SCI-BUS will also create and provide access to many different portlet and application repositories to help user communities in sharing and running these applications via the SCI-BUS gateways
Project partners:• International seismology community • Helio-physics community• Swiss systems biology community of the SystemsX.ch project• German MoSGrid computational chemistry and bioinformatics community • Biomedical researchers community of the Academic Medical Centre of the
University of Amsterdam• Astrophysics community• PireGrid SMEs community• Business process modelling community involving a wide range of areas such as
finance, healthcare, government, production, robotics and emergency• Blender rendering community• Citizen web-2 community• Public application developer community
Subcontractors:• 6 sub-contractors to be selected representing additional user communities• Hugely competitive – over 3 times as much applications
Additional user communities supported by the project
The SCI-BUS user communities
The gateway services will enable unified access to all the major DCI infrastructures used in Europe:• Local and remote clusters accessed by queuing systems (PBS,
LSF, Condor, SGE)• Local and remote supercomputers• Local and volunteer Desktop Grids (BOINC, XtremWeb,
OurGrid)• Grids (ARC, gLite, Globus, UNICORE)• Academic Public and Private Clouds (OpenNebula,
Eucalyptus, etc.)• Public Commercial Clouds (Amazon EC2, IBM Cloud, etc.)
The SCI-BUS architecture
8
The SCI-BUS Infrastructure
CloudBroker Platform
SCI-BUSScienceGateways
API
App Storefor Cloud
Platformto accessapps inClouds
CloudProvider
AraGrid (Spain), Armenian Grid, Baltic Grid, Belgian GridBIFI Desktop Grid, Bulgarian Grid, ClGrid (Chile), COMPCHEM VO of EGEE
Croatian Grid, EGRID (Economy Grid, Italy)GILDA training grid,Grid Ireland, HunGrid (Hungarian National Grid)
IberGrid (Portugal and Spain), MathGrid (Spain)MoSGrid (Molecule Simulation Grid of D-Grid), KnowledgeGrid Malaysia
PireGrid (Spain and France), See-Grid (South-East European Grid)SwissGrid, Turkish Grid, UK NGS, UK White Rose Grid
VOCE (Central European Grid), Westminster Desktop Grid
Clusters: PBS, LSF, Condor, SGE
...
Liferay and P-Grade
DesktopGrids:
BOINC, Xtrem-Web, OurGrid
Super-computers
WebService / API
Eucalyptus OpenNebula
gLiteMiddleware
UNICOREMiddleware
ARCMiddleware
App 1 App 2 App 3
SGIAmazo
nEC2
IBM
...
...Private Clouds
API API API API
Commercial Components
SCI-BUSgenericgateway
Ger
man
Mos
Grid
Co
mm
unity
Stati
stica
l Sei
smol
ogy
Com
mun
ity
Blen
der R
ende
ring
Com
mun
ity
Amst
erda
m M
edic
al
Cent
er C
omm
unity
Swiss
Pro
teom
ics
Com
mun
ity
Astr
ophy
sics
Com
mun
ity
Hel
io p
hysic
s Co
mm
unity
Busin
ess
Proc
ess
Com
mun
ity
Soft
war
e bu
ild a
nd
test
Com
mun
ity
Citiz
en W
eb
Com
mun
ity
Pire
Grid
Com
mer
cial
Co
mm
unity
The baseline technologyP-GRADE Portals around the globe
– National Grid portals (UK, Ireland, Croatia, Turkey, Spain, Belgium, Malaysia , Kazakhstan, Switzerland, Australia, ClGrid, etc.)
– HP-SEE/SEE-GRID-SCI, BalticGrid– US Open Science Grid, TeraGrid– IDGF (International Desktop Grid Federation)– SHIWA Simulation Platform– Economy-Grid, Swiss BioGrid, Bio és
Biomed Virtual Organisations, MathGrid, etc.
HP-SEE
P-GRADE Portal software family
P-GRADE Portal2.4
NGS P-GRADE portal
P-GRADE Portal2.5
P-GRADE Portal2.8
P-GRADE Portal2.9.1
WS-PGRADE Portal 3.1
WS-PGRADE Portal 3.2
GEMLCAGrid Legacy Code Arch.
GEMLCA, storage concepts
Basic concepts
2008
2009
2010WS-PGRADE Portal
3.3
P-GRADE Portal2.10
Open source since 2008
Open source since 2011
2011 WS-PGRADE Portal 3.4
Motivations of creating WS- PGRADE/gUSE
• To overcome (most of) the limitations of P-GRADE portal:• To provide better modularity to replace any service• To improve scalability to millions of jobs• To enable advanced dataflow patterns• To interface with wider range of resources• To separate Application Developer view from Application
User viewWS-PGRADE (Web Services Parallel Grid Runtime and Developer
Environment) and
gUSE (Grid User Support Environment) architecture
WS P-GRADE in a nutshell• General purpose, workflow-oriented portal. Supports the
development and execution of workflow-based applications• Based on Liferay• Services supported by the portal:• New functionalities
– Web services– DB connectors– Embedded workflows– Job level PS– Conditional jobs– Recursive graph– Multi-generator– Multi-collector– CROSS product PS– DOT product PS
Basic Services EGEE grids
(LCG2,GLite)
Globus grids
(GT2,GT4)
Desktop
grids
clusters
Job execution
File storage
Certificate management
Information system
Brokering
Job monitoring
Workflow & job visualization
Inside of the System gUSE tires
WFGraph editor
WEB-UI(HTML)
Liferay
WS-PGRADE portal
Information System
WF Storage
File Storage
Application Repository
WF Interpreter
Backend
local submitter
GT4 submitter
Glite submitter
LCG submitter
GT2 submitter
GEMLCA submitter
WS (Axis) submitter
Glite GridGlite
Grid
GMLCA GridGMLCA
Grid
GT2 GridGT2 Grid
GT4 GridGT4 Grid
LCG GridLCG
Grid
WSWS
DCI
-Brid
ge
WS P-GRADE workflow concept
Concrete Workflow
Algorithms,Resource references,Inputs
Graph
Jobs,Edges,Ports
Template
Constraints,Comments,Form Generators
Workflow Instance
Running state,Outputs
Repository Item
Application ORProject OR,Workflow part(G,T,CW)
Legend:a b a must reference ba b a may reference b
ASM Application Specific Module
• A new module called “Application Specific Module” has been developed for gUSE that ease the creation of Application Specified Portlets – The module provides programming API to control applications developed
using gUSE– The module provides programming API to access inner gUSE services
• Any kind of Graphical interface can be developed on top of this module – GUI written in JSP, JSTL (Java Servlet Tag Library ) and Java– GUI can be customized for the grid application– GUI can be customized for end users
ASM - Application Specific Module
DCI systems
Grid Application Developer creates own Grid Application via ws-PGrade and export it to the local repository
gUSE
Repository
ws-PGradeGrid Application
Submitter
Workflow Interpreter
Workflow Storage
Storage
Information User spaceGrid Application
Then the Grid Portal Developer creates a new web-application (mainly an web-interface) which fits to the user community's needs, and connects it to the gUSE using ASM API
Developed web-interface
ASM
Finally the end-users create own application from the exported one, parameterize and execute it using the developed portlet
<< import >>
Grid Application
CloudBroker Platform• Easy, scalable, secure, integrable and pay-per-use access to scientific
and technical applications in the cloud• High performance computing application store and marketplace
with accounting and billing• Deployment and execution of applications in the cloud with the
same parameters and files as for local execution• Accessible through the internet via a web browser UI• Integration possibilities into third party tools via web service-based
APIs• Public version available under https://platform.cloudbroker.com
• Using infrastructure as a service (IaaS) from cloud providers• Offering platform as a service (PaaS) for software vendors• Providing software as a service (SaaS) to end users
CloudBroker GmbH
Platform Architecture
CloudBroker GmbH
CloudBroker PlatformCloudBroker Platform
AmazonCloud
IBMCloud
…Cloud
ChemistryAppli-
cations
BiologyAppli-
cations
MedicineAppli-
cations
WebBrowser
UI
Engineering Appli-
cations
Web Service API
GenericWorkbenches CloudBroker Integration
CloudBroker Integration
Domain-Specific Gateways
R&D End Users and Software Vendors
CLI
…Appli-
cations
CloudBroker GmbH
CloudBroker Business Model• Free access to the platform
after registration• Each organization manages
its own users• Prepayment or monthly
billing
• Platform usage costs =resource usage costs +software usage costs + CloudBroker usage surcharges
CloudBroker
EndUsers
SoftwareVendors
CloudProviders
Resources Applications
Usage $
$ $
CloudBroker GmbH
Clouds and Applications
• Currently supported cloud infrastructures:– Amazon Web Services– IBM SmartCloud
Enterprise
• Adapters to further clouds planned
• Open source-based private cloud tools will be interfaced within SCI-BUS
• Current applications:– Chemistry (GAMESS,
Gromacs)– Biology (BLAST, X!
Tandem, Rosetta)– Medicine (AutoDock)– Engineering (OpenFOAM)
• Further applications in preparation
• Own applications can be easily added by users
CloudBroker Platform Integration in Liferay/WS-PGRADE/gUSE
Gamess GromacsAutodock Open FoamBlast RosettaXtandem ...
API
gUSE
WS-PGRADE Job Configuration Portlet CloudBroker UI Portlet
UI
DCI-BRIDGE
SaaS (Softwares as a Service)
CloudBroker plugin
Expose the CloudBroker UI through WS-PGRADE interface by proxy.
User Mapping Portlet
CB Java API
• Integrate the SaaS capabilities of CB into the gUSE/WS-PGRADE system.• Enable WS-PGRADE users to run softwares (SaaS) exposed by the CB platform.
MosGrid Science gateway for molecular simulations
• Requirement: Workflow enabled grid portal for UNICORE based on Liferay
• Solution:– Adaptation of WS-PGRADE from
GridSphere to Liferay– Development of UNICORE
Submitter– Support for SAML– Development of ASM
(Application Specific Module) support
MosGrid Science gateway - architecture
MosGrid Science gateway - architecture
MosGrid Science gateway – application domains
• Quantum Chemical Calculations– based on approximated solutions of the Schrödinger equation– Average scalability– Gaussian, NWChem, Turbomole, …
• Molecular Dynamics– based on forcefields describing molecular interactions– Good scalability– Gromacs, NWChem, Amber, …
• Docking– Based on simplified forcefields– Excellent scalability– CADDSuite, FlexX, …
MosGrid Science gateway – application domains – molecular dynamics
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481
Thank you for your attention!
Any questions?
http://www.sci-bus.eu/