Upload
mario-jose-villamizar-cano
View
117
Download
1
Tags:
Embed Size (px)
Citation preview
e-Clouds: A Platform and Marketplace to Access and Publish
Scientific Applications as a Service
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País - 2014
Mario José Villamizar Cano [email protected]
Project Assistant – Cloud Computing Advisor Universidad de los Andes
Bogotá, Colombia
THE PROBLEM
RELATED WORK
SOLUTION REQUIREMENTS
SOLUTION ARCHITECTURE
IMPLEMENTATION
FUTURE WORK
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
THE PROBLEM
When Small and Medium research groups have projects that requires large processing capabilities they regularly have to use dedicated
cluster and/or grid computing infrastructures. However …
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
THE PROBLEMS WITH CLUSTER AND GRID
1) The total cost of ownership
(TCO).
2) Complex processes related to IT
management.
3) There are different models to
parallelize applications.
4) Researchers require large
computing capabilities during
peak periods.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
THE CLOUD COMPUTING MODEL - SaaS
Users pay a subscription to access the service (not to buy the license or to buy and maintain hardware)
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
THE CLOUD COMPUTING MODEL
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
THE CLOUD COMPUTING MODEL - IaaS
VM Types vCPUs ECUs Memory Storage Price/Hour USD
m1.small 1 1 1,70 1 x 160 0,044
m1.medium 1 2 3,75 1 x 410 0,087
m1.large 2 4 7,50 2 x 420 0,175
m1.xlarge 4 8 15,00 4 x 420 0,350
General Purpose Instances
VM Types vCPUs ECUs Memory Storage Price/Hour USD
c3.large 2 7 3,75 2 x 16 SSD 0,105
c3.xlarge 4 14 7,50 2 x 40 SSD 0,210
c3.2xlarge 8 28 15,00 2 x 80 SSD 0,420
c3.4xlarge 16 55 30,00 2 x 160 SSD 0,840
c3.8xlarge 32 108 60,00 2 x 320 SSD 1,680
Compute Optimized Instances
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
THE CLOUD COMPUTING MODEL
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
THE PROBLEM
Research groups of different sizes are adopting IaaS solutions. The adoption of cloud computing is a complex process, where non-IT researchers have to face similar or more complex processes as those found in cluster or grid infrastructures.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
THE PROBLEM
At enterprise level, the Software as a Service (SaaS) model allows that small businesses can easily adopt complex and ready-to-use applications. Problems associated to the management and the configuration of the underlying (PaaS, IaaS) infrastructure are abstracted.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
RELATED WORK – e-Clouds GOAL
e-Clouds: A SaaS marketplace where researches can access to on-demand and ready-to-use scientific
applications via a Web Browser using a pay-per-use model.
We believe that cloud computing may provide more opportunities for a large number of small and medium research groups.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
RELATED WORK
Some commercial scientific SaaS marketplaces such as Cyclone and Cloud Numbers are offering scientific SaaS application on public IaaS.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
RELATED WORK
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
SOLUTION REQUIREMENTS
In most in-house HPC/HTC environments, clusters are the core element used to execute scientific applications.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
SOLUTION REQUIREMENTS
A multi-tenant SaaS solution for scientific computing should allow that
researchers ca access to different ready-to-use-applications, execute
and monitor jobs, and the movement of data from and to the cloud; we
call these three elements the workspace.
In-house Cluster
End-user interfaces
Batch Applications
Distributed File Systems
External Libraries
Cluster middleware
Operating Systems
Virtualization
Servers
Storage
Networking
Public Scientific SaaS (e-Clouds)
End-user interfaces
Batch Applications
Cloud Storage Systems
External Libraries
Cluster middleware
Operating Systems
Virtualization
Servers
Storage
Networking
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País M
an
ag
ed
by
res
ea
rch
ers
/ g
rou
ps
Ma
na
ged
by
res
ea
rch
ers
/ g
rou
ps
SOLUTION REQUIREMENTS
Blob Storage Scalable storage systems that can
grow to any size
App Management On-demand installation of hundreds
of scientific applications
Scheduling Servers can be turned on or
turned off as needed.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
SOLUTION ARCHITECTURE
e-Clouds is based on a new architecture that abstract the above problems, allowing researchers to focus their efforts in result analysis and experimental designs, and not on technical IT issues.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
Application management
Data management
Job execution and
monitoring
Pricing Security Workflow
based execution
Private/Public IaaS
SOLUTION ARCHITECTURE
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
e-Clouds provides mechanisms that allow users to act as application configurators, facilitate the installation and configuration of new applications: cluster creation on demand.
Queue based communications and different components of e-Clouds scale automatically and allows e-Clouds to support the execution of large workloads.
Costs Efficiency.
Traditional directory structure based on scalable blob systems.
SOLUTION ARCHITECTURE
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
Researchers/users:
1. Choose the application.
2. Configure the execution.
3. Monitor the jobs/executions.
4. Download the results.
5. Pay only for what they use.
e-Clouds is in charge of
interacting with the public/private IaaS in all of the steps.
IMPLEMENTATION
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
Amazon Web Services (AWS) Ruby on Rails / Heroku.
PostgreSQL. EC2, S3. e-Clouds Agent (Ruby).
Script application in R of Von Humboldt Institute.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
IMPLEMENTATION
IMPLEMENTATION
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
IMPLEMENTATION
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
IMPLEMENTATION
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
IMPLEMENTATION
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
WHEN TO USE CLOUD SOLUTIONS SUCH AS e-Clouds
Cases where Private and Commercial Clouds may be Cost Effective (Magellan R.): Unknown Demand. In the case of a new project or a new application where the potential demand is still poorly understood. Sporadic Demand. One of the more common cases for using commercial cloud offerings is when the demand is highly variable, especially if there are also time sensitive requirements for the service. Facility Constrained. Some sites are severely infrastructure limited.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
FUTURE WORK Usability tests with more researchers. Large scale performance tests.
Auto scaling and queues mechanisms. More Bag of Tasks applications. e-Clouds as an Open Source project. Shared workspaces. Different public IaaS. Types of applications (MPI, GPUs). Stronger security mechanisms.
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
WE ARE LOOKING FOR BETA USERS/RESEARCHERS
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País
Más información en: eclouds.uniandes.edu.co
THANKS FOR YOUR ATTENTION!
Mario Villamizar - Universidad de los Andes
[email protected] - @mariocloud
1er Foro Internacional de Bioinformática
Ciencia y Tecnología para el Futuro del País – 2014