View
214
Download
1
Category
Tags:
Preview:
Citation preview
Advanced Computing and Information Systems laboratory
Virtual Appliances and Education using Clouds
Dr. Renato FigueiredoACIS Lab - University of Florida
Advanced Computing and Information Systems laboratory
Background
Traditional ways of delivering hands-on training and education in parallel/distributed computing have non-trivial dependences on the environment
• Difficult to replicate same environment on different resources (e.g. HPC clusters, desktops)
• Difficult to cope with changes in the environment (e.g. software upgrades)
Virtualization technologies• Remove key software dependences
• Allow packaging and replication of hands-on, executable educational environments
• Can be deployed, managed with cloud technologies
2
Advanced Computing and Information Systems laboratory
Overview
Virtual appliances Virtual networking
• Grid appliances and FutureGrid
Support for classes in FutureGrid Demo
3
Advanced Computing and Information Systems laboratory
Guiding principles
Fidelity: activities should use full-fledged, executable software: education/training modules• Learn using the proper tools
Reproducibility: Creators of content should be able to install, configure, and test their modules once, and be assured of the same functional behavior regardless of where the module is deployed• Incentive to invest effort in developing, testing and
documenting new modules
4
Advanced Computing and Information Systems laboratory
Guiding principles
Deployability: Students and users should be able to deploy modules in a simple manner, and in a variety of resources• Reduce barriers to entry; avoid dependences
upon a particular infrastructure
• Interactive use
Community-oriented: Modules should be simple to share, discover, reuse, and expand• Create conditions for “viral” growth
5
Advanced Computing and Information Systems laboratory
Role of clouds and portal
Executable modules – virtual appliances• Deployable on FutureGrid resources
• Deployable on other cloud platforms, as well as virtualized desktops
Community sharing – Web 2.0 portal, appliance image repositories• An aggregation hub for executable modules
and documentation
6
Advanced Computing and Information Systems laboratory 7
What is an appliance?
Hardware/software appliances• TV receiver + computer + hard disk + Linux +
user interface
• Computer + network interfaces + FreeBSD + user interface
Advanced Computing and Information Systems laboratory 8
Virtual appliance example
Linux + Apache + MySQL + PHP
copy
instantiate
LAMPimage
A web server Another Web server
Repeat…
VirtualizationLayer
Advanced Computing and Information Systems laboratory
Educational appliances
A flexible, extensible platform for hands-on, lab-oriented education on FutureGrid
Support clustering of resources• Virtual machines + virtual networking to create
sandboxed modules• Virtual “Grid” appliances: self-contained, pre-packaged
execution environments
• Group VPNs: simple management of virtual clusters by students and educators
9
Advanced Computing and Information Systems laboratory
Virtual Networking
A single appliance encapsulates software and configuration
Cluster/Grid/Cloud computing• Middleware expects a collection of machines,
typically on a LAN (Local Area Network)
• Appliances need to communicate and coordinate with each other
• Each worker needs an IP address, uses TCP/IP sockets
10
Advanced Computing and Information Systems laboratory 11
Virtual cluster appliances
Virtual appliance + virtual network
copy
instantiate
Hadoop+
VirtualNetwork A Hadoop worker Another Hadoop worker
Repeat…
Virtual machine
Virtual network
Advanced Computing and Information Systems laboratory 12
Grid appliance in a nutshell
Plug-and-play clusters with a pre-configured software environment• Linux + (Hadoop, Condor, MPI, …)
• Scripts for zero-configuration
Hands-on examples, bootstrap infrastructure, and zero-configuration software – you’re off to a quick start
Advanced Computing and Information Systems laboratory 13
Virtual Network - GroupVPN
Setting up and managing typical VPNs can be daunting• VPN server(s), key distribution, NAT traversal
GroupVPN makes it simple for users to create and manage virtual cluster VPNs
Key insights:• Web 2.0 interface: create/manage user groups
• All the complexity of setting up and managing VPN links is automated
Advanced Computing and Information Systems laboratory 14
Deploying virtual clusters
Same image, different VPNs
copy
instantiate
Hadoop+
VirtualNetwork A Hadoop worker Another Hadoop worker
Repeat…
Virtual machine
GroupVPN
GroupVPNCredentials
(fromWeb site)
Virtual IP - DHCP10.10.1.1
Virtual IP - DHCP10.10.1.2
Advanced Computing and Information Systems laboratory
Support for classes on FutureGrid
Classes are setup and managed using the FutureGrid portal
Project proposal: can be a class, workshop, short course, tutorial• Needs to be approved by FutureGrid project
to become active Users can be added to a project
• Users create accounts using the portal
• Project leaders can authorize them to gain access to resources
• Students can then interactively use FG resources (e.g. to start VMs)
15
Advanced Computing and Information Systems laboratory
Use of FutureGrid in classes
Cloud computing/distributed systems classes• U.of Florida, U. Central Florida, U. of Puerto
Rico, Univ. of Piemonte Orientale (Italy), Univ. of Mostar (Croatia)
Distributed scientific computing• Louisiana State University
Tutorials, workshops:• Big Data for Science summer school
• A cloudy view on computing
• SC’11 tutorial – Clouds for science
• Science Cloud Summer School (this month)
16
Advanced Computing and Information Systems laboratory
Cloud computing classes
Massimo Canonico, U. Piemonte Orientale Difficulties to overcome:
• Hardware issues: find enough free physical machines able to host virtual machines
• Software issues: time to install/configure as many as possible different cloud platforms
• University was not able to provide me the necessary hardware and software support
Students started to play with FutureGrid• After attending few lessons, they were able to
start/stop virtual instance with several Cloud Computing platforms
17
Advanced Computing and Information Systems laboratory
Cloud computing classes
Students used Eucalyptus, OpenStack and Nimbus• Half were not computer scientists.
As FutureGrid freely shares their physical machines and their cloud platforms, decided to freely share all materials of my class. • Hands-out, configuration files and link to
useful documentation are available
• https://portal.futuregrid.org/contrib/cloud-computing-class
18
Advanced Computing and Information Systems laboratory
Cloud computing classes
Graduate-level “Cloud computing for Data-Intensive Sciences” (Judy Qiu, Fall 2010)• Virtualization technologies and tools
• Infrastructure as a service
• Parallel programming (MPI, Hadoop)
• FutureGrid provided a set of software options that made it possible for students to work on different projects along the system stack.
19
Advanced Computing and Information Systems laboratory
Cloud Storage#8 Cloud Storage Survey (Xiaoming, Nixiaogang)
Cloud Storage#8 Cloud Storage Survey (Xiaoming, Nixiaogang)
Iterative MapReduce#3 LDA (Changsi, Yang)
#4 MemCache (Saliya, Yiming ,Jerome)#5 Avro (Yuduo, Yuan, patanachai)#6 PageRank (Shuo-Huan,Parag)
Iterative MapReduce#3 LDA (Changsi, Yang)
#4 MemCache (Saliya, Yiming ,Jerome)#5 Avro (Yuduo, Yuan, patanachai)#6 PageRank (Shuo-Huan,Parag)
Virtualization#9 Hypervisor Performance Analysis Project (James ,
Andrew)
Virtualization#9 Hypervisor Performance Analysis Project (James ,
Andrew)
Cloud Platfor
m
CloudInfrastruc
ture
Cloud Infrastructure #7 Nimbus, Eucalyptus (Stephen, Sonali, Shakeela)
Cloud Infrastructure #7 Nimbus, Eucalyptus (Stephen, Sonali, Shakeela)
Hypervisor/
Virtualization
Dryad/DryadLINQ#1 Matrix Multiplication (Swapnil,Amit,Pradnay)
#2 PhyloD (Ratul,Adrija,Chengming)
Dryad/DryadLINQ#1 Matrix Multiplication (Swapnil,Amit,Pradnay)
#2 PhyloD (Ratul,Adrija,Chengming)
Higher Level
Languages
Term Projects
(Slide courtesy of Judy Qiu)
Advanced Computing and Information Systems laboratory
University ofArkansas
Indiana University
University ofCalifornia atLos Angeles
Penn State
IowaState
Univ.Illinois at Chicago
University ofMinnesota Michigan
State
NotreDame
University of Texas at El Paso
IBM AlmadenResearch Center
WashingtonUniversity
San DiegoSupercomputer
Center
Universityof Florida
Johns Hopkins
July 26-30, 2010 NCSA Summer School Workshophttp://salsahpc.indiana.edu/tutorial
300+ Students (200 on sites from 10 institutes; 100 online)IU MapReduce and UF Virtual Appliance technologies are supported by FutureGrid.
(Slide courtesy of Judy Qiu)
Big Data for Science
Advanced Computing and Information Systems laboratory 22
Demonstration
Deploying virtual appliance node on FutureGrid (Nimbus @ Alamo)
Connecting to virtual machine Virtual networking Running sample job
Advanced Computing and Information Systems laboratory 23
Demonstration
Pre-instantiated VM to save us time:• cloud-client.sh --conf alamo.conf --run --name
grid-appliance-2.05.03.gz --hours 24
Connect to VM• ssh root@VMip
Check virtual network interface• ifconfig
Ping other VMs in the virtual cluster Submit Condor job
Advanced Computing and Information Systems laboratory 24
Uploading and sharing images
APIs available to upload images, customize, save, and share images
Community education pages are available• FutureGrid Web portal allows users to publish
their own content
• Tutorials, presentations on Web portal; VMs on image repositories
Advanced Computing and Information Systems laboratory 25
Where to go from here?
Tutorials on FutureGrid and Grid appliance Web sites for various middleware stacks• Condor, MPI, Hadoop
A community resource for educational virtual appliances• Success hinges on users effectively getting
involved
• If you are happy with the system, let others know!
• Contribute with your own content – virtual appliance images, tutorials, etc
Advanced Computing and Information Systems laboratory 26
Questions?
More information:• BOF: tomorrow 4:45pm, Burnham (8th floor)
• http://www.futuregrid.org
• http://grid-appliance.org
This document was developed with support from the National Science Foundation (NSF) under Grant No. 0910812 to Indiana University for "FutureGrid: An Experimental, High-Performance Grid Test-bed." Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF
Recommended