Innovative digital methods transforming research
A collaborative research hub for
Digital Oxford
A full department in a leading science division
HPC Engine and Storage
Next Generation Infrastructure
The Smart Grid
High Speed Communications System
Service Restorati
on
Voltage
Control
Condition Monitoring/Data Mining
Distribution
System State
Estimation
SCADA & Distribution Management System
14
UK NGS Cloud Activities (2010 – 2012)
• NGS Agile Deployment Environments
EPSRC funded, 2 years
• Staff:
– David Wallom (OeRC, Oxford);
– David Fergusson (NeSC, Edinburgh);
– Steve Thorn (NeSC, Edinburgh);
– Matteo Turilli (OeRC, Oxford).
• Goals:
– EC2 compatible, open source solution;
– Small physical systems physically distributed
– development of a dedicated pool of images, supporting both end user and NGS requirements such as training;
– collecting data about feasibility, costs, stability;
– identify use cases and gather further requirements.
15
NGS Cloud Usage
• >120 registered users: uptake has been very fast and users stayed engaged throughout the whole testing period;
• 26 institutions: 23 HEI both universities and colleges, 3 companies;
• 30 projects;
• 10 research areas.
Life sciences
Teaching
Mathematics
Cloud R&D
PhysicsEcology
GeographyMedicine
Social Science
Engineering
16
Exemplar Case Studies
• Evolutionary Genomics: “analysis and Information management of Next Generation Sequencing (NGS) of Genomic data poses many challenges in terms of time and size. We are exploring the translation of high quality NGS scientific analysis pipelines to make best use of Cloud infrastructure”;
• Geospatial Science: “geospatial data is a mix of raster and vector data. As rasterizing is CPU-hungry process, and all maps displayed on the screen of the final user are rasters, it is more efficient to do the process on the server side. I am investigating how this process can be dispersed across many, if not unlimited instances in a cloud”;
• Agent-based modelling of crime: “at the moment I have a tomcat server that hosts some web services used to run social simulation model, it needs access to the file system to run fortran scripts, create files etc. There are loads of problems with running our own server at uni and I think a virtual machine that I could have control over would be much better”.
17
Flexible Services for the Support of Research (FleSSR)
6 Partners
• Academic and industrial;
• 3 cloud infrastructures.
Goals
Building federated cloud infrastructure, extending the use of UK NGS central services with cloud brokering and accounting.
Use cases
•Multi Platform Software Development;
•On demand Research data storage.
19
FleSSR Infrastructure
• Local/Global: services depends either on local or global access. Cloud brokering is not mandatory for AWS-like service access;
• Multiple identities: every user may have multiple identities, both local and global;
• Only personal identities: group identities are not implemented. The management of every single identity is left to the legally responsible user;
• Multiple AA technologies: AA may differ depending on local and global policies/technologies;
• Multiple accounting: every single identity is accounted for its usage. Every individual may get multiple invoices.
20
FleSSR Use Case: Multi Platform Software Development
Zeel/i BrokerInstance configuration
manager
FleSSR cloud
Build manager
CVS / SVN repository
Build instance 1
Build instance 2
Build instance 3
Build instance 4
Build instance 5
21
FleSSR Use Case: On demand Research data storage
Zeel/i Broker Volume Manager
FleSSR cloud
VM EBS Interface
EBS Volume
22
FleSSR Output
Code
• Instance configuration and build manager: Perl command line utility + Java client utilising the Zeel/I API;
• Personal EBS volume manager: web-based, Java client for EBS volumes handling + tailored VM image with multiple data interfaces (SFTP, WebDAV, GlusterFS, rsync, ssh);
• Eucalyptus open-source accounting system: Perl aggregators and parsers for standard eucalyptus open-source log files + MySQL accounting database + PHP accounting client.
Use cases
• SKA community testing of Use case;
• Institutional ICT team testing WEB-DAV, GridFTP & GlusterFS solution as Use case 2.
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323
The EGI Federated Cloud solution
2011 - date
www.egi.euEGI-InSPIRE RI-261323
RationaleGrowth of Providers
• High Throughput Platform– Academic resource providers
• Federated Cloud Platform– Diversity of resource providers
Tens of 1000’s MillionsFew related use cases
Single application modelMany diverse use cases & application models
Growth of Research Communities
24
www.egi.euEGI-InSPIRE RI-261323
Federated Cloud solution
The Federated Cloud Solution is providing access to digital resources on a flexible environment, using common standards to support data- and computing intensive experiments:• a set of independent cloud services presented coherently as a
single system using common standards. • allows the user to choose freely among a broader range of service
providers• allow use of applications already developed by people within their
own community whom they trust, and from other communities who have an independent badge of quality.
Target groups:• individual researchers • larger research communities or groups
25
www.egi.euEGI-InSPIRE RI-261323
Principles of Federation
• Standards and validation: Recommended and common open standards for the interfaces and images – OCCI, CDMI, OVF, GLUE2, AAI
• Resource integration: Cloud Computing to be integrated into the existing production infrastructure.
• Heterogeneous implementation: no mandate on the cloud technology.
• Provider agnosticism: the only condition to federate resources is to expose the chosen interfaces and services.
26
www.egi.euEGI-InSPIRE RI-261323
EGI Cloud Infrastructure
27
EGI Core Platform
Federated AAIServiceRegistry Monitoring Accounting
EGI Cloud Infrastructure Platform
Instance Mgmt
Information Discovery
Storage Management
Help and Support
Security Co-ordination
Training and Outreach
EGI C
olla
bora
tion
Tool
s
EGI A
pplic
ation
D
BIm
age
Repo
sito
ryEG
I Clo
ud S
ervi
ce M
arke
tpla
ce
Sustainable Business Models
User Community
Monitoring and control of utilisation
Technical Consultancy and Support
Uniform interfaces to Cloud Compute and Storage
Cloud Management Stacks(OpenStack, OpenNebula, Synnefo, …)
Secu
re e
ndor
sed
Appl
icati
on a
nd
Serv
ice
Dep
loym
ent
GSIGLUE2
Cloudinit CDMI
SAM UR
OVF
OCCI
www.egi.euEGI-InSPIRE RI-261323
PartnershipResources
– 13 NGIs provide 22 certified resources– 4 NGIs currently integrating resources– 5 NGIs with interested resource providers– Worldwide interest & integration
• Australia* (NeCTAR)• South Africa* (SAGrid)• South Korea* (KISTI)• United States* (NIST, NSF A.C. Centres)
* Not shown on map
Usage since launch- 562k VMs- 42M CPU hours
29
www.egi.euEGI-InSPIRE RI-261323
User Model
• The offer to our users:• Total control over deployed applications • Elastic resource consumption based on real needs • Workloads processed on-demand• Endorsed and accredited applications available from
multiple different communities shared• Single sign-on at multiple, independent providers • Centralised access to service information across
multiple providers
30
www.egi.euEGI-InSPIRE RI-261323
Virtual Appliance Catalogue
• Registered Virtual Appliances: 30, ref• Supporting Sites: 21, ref• Supported Virtual Organizations: 9, ref
• atlas, • biomed, • cms, • demo.fedcloud.egi.eu, • drihm.eu, • fedcloud.egi.eu, • highthroughputseq.egi.eu, • lhcb,
• vo.chain-project.eu
[Operation of the AppDB Cloud MP, officially started on June/2014]
31
www.egi.euEGI-InSPIRE RI-261323
EGI FedCloud Launch Communities (May 2014)
• Ecology – BioVeL: Biodiversity Virtual e-Laboratory
• Structural biology – WeNMR: a worldwide e-Infrastructure for NMR and structural biology
• Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb)
• Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for
volcano and earthquakes monitoring (Collaboration with Helix Nebula)
• Software Engineering – SCI-BUS: simulated environments for portal testing
• Software Engineering – DIRAC: deploying ready-to-use distributed computing systems
• Software Engineering – Catania Science Gateway Framework
• Musicology – Peachnote: dynamic analysis of musical scores
• Earth Observation – ENVRI: Common Operations of Environmental Research
infrastructures (collaboration with EISCAT3D)
• Geology – VERCE: Virtual Earthquake and seismology Research
• Ecology – LifeWatch: E-Science European Infrastructure for Biodiversity and Ecosystem
Research
• High Energy Physics – CERN ATLAS: ATLAS processing cluster via HelixNebula
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
34
www.egi.euEGI-InSPIRE RI-261323
Strengthening the underpinning platform
Continuing a Technology Evolution
• Broader support for open standards in Cloud management frameworks– Utilisation of rOCCI for interfaces to commercial cloud
frameworks– Completion of high quality reference implementation for
CDMI
• New feature additions to foundational tools depending on requests– Accounting, monitoring, service discovery, Image
Management
37
Valu
e ad
ded
serv
ices
for U
ser
Com
mun
ities
www.egi.euEGI-InSPIRE RI-261323
Federated Cloud Services
Federated IaaS Cloud
38
Tier 1:Reliable
Infrastructure Cloud
Tier 4:Zero ICT
Infrastructures
Tier 3:Platform as a Service
Tier 2:General-purpose platform services
PaaS
PaaS
DB
aaS
Had
oop
aaS
VRE
Secure storage
Key
Mgm
t
Encr
yptio
n
ACL
mgm
t
Virtual eLaboratory
39
Conclusions
• Utilisation of virtual infrastructure is the only scalable method to support large number of disparate user communities with widely differing application models,
• Federation as robust and scalable model of national/European cloud infrastructure for research, though part of an ecosystem of e-infrastructure not e-infrastructure alone.
• Federation is only possible by the availability of open standards,• Successful pilot tests of multiple prototypes of cloud infrastructure allowed a quicker
development of the final model for EGI,• EGI Federated Cloud is attracting new communities belonging to various scientific
domains– 26 communities and 59 use cases currently supported, 5 from commercial organisations
• Paving the way for a global federated cloud marketplace led through European Innovation
– Established best practice
– Illustrating leadership
– Open standards, open technology
– Open membership, open processes
– Open competition
• Oxford and OeRC has led this activity from its inception in 2010 through to now