Hybrid Cloud Tutorial:From Planning to Classroom

David Rilett and Charlie WisemanDepartment of Computer Science and Networking

Wentworth Institute of Technology

Heading into the Cloud

• Cloud computing means many different things today

• This talk will do two things:– Provide an overview of the cloud world – Go through the procedure to build a private or

hybrid cloud platform using Eucalyptus, an open source cloud system that is compatible with Amazon Web Services (EC2/S3)

What was old…

• Cloud computing is the evolution of many other computing paradigms– Grid, Utility, Mainframes, …

• Virtualization is the key enabler– Better utilization of physical servers and storage

with fast backup recovery– Simpler migration of virtual instances– Easier provisioning on demand

Big Picture

Cloud Provider

Cloud ConsumerApplication

End Users

Example

• Many R&D businesses rely on cloud environments for both long -term and short-term services

• For example: – Deployed 50,000 cores of computing power for 3

hours– Cost $15,000

Types of Cloud Models

• Cloud computing systems are characterized in two ways: deployment model and service model

• The deployment model describes who the cloud consumers are

• The service model describes what types of resources are provided to the cloud consumers

Deployment Models• Public cloud

– Resources are shared by many consumers and are provided to anyone (for a fee)

• Private cloud– Resources are utilized by a single organization– Can still be operated by a third party

• Hybrid cloud– Mix of public and private, allowing the consumer to deploy

applications using either public or private resources

Service Models

• Infrastructure as a Service (IaaS)– Virtual machines and storage are provided– Consumers install their own OS from the ground up

• Platform as a Service (PaaS)– Preconfigured computing platforms are provided– For example, pre-built web systems– Consumers only write the application

• Software as a Service (SaaS)– Everything through the application is provided– Consumers can only customize and tweak

IaaS or PaaS?

• Depending on the requirements, you should be using an IaaS or PaaS model– Give students a preconfigured image with

everything they need on it (PaaS)– Allow them to install their own OS and do all the

configuration themselves (IaaS)• The former is more likely in a software

development course, the later more likely in a networking or administration course

Public, Private, or Hybrid?• Benefits of public clouds like Amazon Web

Services (AWS):– Requires no physical resources at your institution– No limit on how many instances you can run

• Drawback of using a public cloud:– Pay per usage (typically per server hour and per

GB of data transfer)– For example: http://aws.amazon.com/ec2/pricing/

Public, Private, or Hybrid?

• Benefits of building your own private cloud:– Full control over the resources– No per use fees

• Drawbacks of a private cloud:– Have to buy all the equipment up front– Can't exceed the resources you buy (locally)

Public, Private, or Hybrid? (2)

• A common solution, therefore, is to use a hybrid environment

• Build your own private cloud with a reasonable number of resources

• Do all development and initial testing in the private cloud

• For scale-out testing or demos, move the applications to the public cloud

Enter Eucalyptus

• Eucalyptus is an open source IaaS cloud platform

• Compatible with AWS (EC2/S3)• Deploy your own private Eucalyptus cloud• Move the Eucalyptus images to AWS if you

need more resources• RightScale, euca2ools and HybridFox help with

moving and management

Building a Eucalyptus Cloud

• Requirements:– At least two servers (more depending on how many instances

you need to support)– Cooperation of your friendly local network administrator for

public IP addresses and firewall configuration

• Recommendations:– External storage array for storing images– Switch for building a separate private network for the cloud

Building a Eucalyptus Cloud

• Eucalyptus website provides a lot of information but may lack adequate explanation– http://open.eucalyptus.com/wiki/EucalyptusInstallation_v2.0

• There is some terminology to get out of the way• You need one server to act as the front-end, which will run

all of the cloud-level services and store S3 snapshots in Walrus

• Other servers are nodes for deploying instances of virtual machine images and EBS volumes

Building a Eucalyptus Cloud

• The most important decision you have to make is which networking mode you are going to use with Eucalyptus

• There are four networking modes to choose from

• We recommend a setup with the MANAGED-NOVLAN mode as it provides good flexibility without too much extra hassle

Networking Modes

• MANAGED-NOVLAN mode– Require a switch to interconnect the private cloud network ( or

crossover cable to get one node up)– The nodes will not be connected to your regular network directly

but use private network

• MANAGED mode is similar– Requires VLAN-configured switches (not too difficult)– The only benefit is layer 2 isolation of instances which is less

important for a course

Networking Modes

• The other two modes are STATIC and SYSTEM• STATIC allows you to configure IP addressing by hand in the

cloud– Convenient but very hard to manage in even a small environment

• SYSTEM mode should only be used if can't get a switch for MANAGED-NOVLAN mode– Requires more cooperation with your local network admin since

not managed by Eucalyptus

Network Configuration

• From now on, we will be using MANAGED-NOVLAN mode

• One very tricky part that isn't explained well in the documentation is that there are really three different sets of IP addresses you must work with– Public, Private, and Virtual Instance IPs

Network Configuration

front-end (cloud controller)

LAN/Intern

etpublic address: 5.5.5.5 (eth0)

private address: 192.168.0.1 (eth1)

public network

private networkprivate addresses: 192.168.0.X (eth0)

nodes (node controllers)

VM’s

internal networkaddresses: 192.168.0.X, optional external addresses 5.5.5.X

Virtual Instances IPs• When you configure Eucalyptus on the front-end, you have

to tell it the public and private IP addresses of the front-end

• You also have to choose a third set of IP addresses used for the virtual network of instances running on the nodes

• This must be separate from the other IP ranges• Assuming you use 192.168.0.0/24 for the private ones, use

something like 10.0.0.0/16 for the virtual IPs

Additional Public IPs

• Lastly, you need an additional set of public IP addresses that can be assigned to instances that want to be accessible from outside of the private network

• Ask your network admin for these– For example, might get 5.5.5.10 – 5.5.5.50–Must ensure that regular network doesn't use

these addresses already and that firewalls/routers (DNS?) know about them

Example Configuration File• Example virtual network configuration on the

cloud controller (eucalyptus.conf):

VNET_PUBINTERFACE="eth0"VNET_PRIVINTERFACE="eth1"VNET_DHCPDAEMON="/usr/sbin/dhcpd"VNET_MODE="MANAGED-NOVLAN"VNET_SUBNET="10.10.0.0"VNET_NETMASK="255.255.0.0"VNET_DNS="5.5.1.1"VNET_ADDRSPERNET="32"VNET_PUBLICIPS="5.5.5.10-5.5.5.50"

Other Gotchas *

• Installing the front-end (we used CentOS 5.8)– Be sure to disable SELinux– Enable NTP– Install a DHCP server, but DON'T enable it or

configure it– Configure both public and private NICs statically* Faststart manages this for you

Other Gotchas*• Installing the nodes (we used CentOS 5.8)

– They won't have Internet access in the given configuration, so you could temporarily connect them to your regular network until the install is done

– Disable SELinux– Disable the local firewall (iptables)– Enable NTP– Set the IP address statically– After you install xen, you might have to edit the GRUB configuration

to tell it to boot the xen kernel– Start with a single node until things are working to simplify

debugging * Faststart manages this

Faststart Install CENTOS 5.7 +EucalyptusFastInstallGuide-CentOS.pdf

• Need: CentOS 5.7 install CD #1. You can find the image on one of the mirrors here: http://isoredirect.centos.org/centos/5/isos/x86_64/. The file is labeled CentOS-5.7-x86_64-bin-1of8.iso.

• Need: Eucalyptus Fast Start media (1GB USB) Video: http://vimeo.com/25163094• On install, do network interface configuration, select “Edit” and manually configure IP, netmask,

hostname, gateway and DNS. You will need to check the DNS in resolv.conf, the default route in /etc/sysconfig/network and you interface setup in /etc/sysconfig/network-scripts/ifcfg-eth0 etc.

DEVICE=eth0BOOTPROTO=noneHWADDR=78:2B:CB:3A:A9:3AONBOOT=yesIPADDR=69.43.72.190NETMASK=255.255.255.240GATEWAY=69.43.72.177

• http://www.eucalyptus.com/sites/all/files/EucalyptusFastInstallGuide-CentOS.pdf

EucalyptusConfiguration(/etc/eucalyptus/eucalyptus.conf)

• There are two of these, one on each machine: Frontend (cc walrus) and Node Controller (nc)

• Restart /etc/init.d/eucalyptus-cc cleanrestart for cc when edit on frontend

• Restart /etc/init.d/eucalyptus-nc restart for nc when edit node(s)• Be careful with /etc/init.d/eucalyptus-cloud restart on Frontend --

May lose external addresses on VM instances or may forget to take snapshots of data

eucarc run-time environment. /root/.euac/eucarc (“source” it for euca2ools)

EUCA_KEY_DIR=$(dirname $(readlink -f ${BASH_SOURCE}))export S3_URL=http://69.43.72.190:8773/services/Walrusexport EC2_URL=http://69.43.72.190:8773/services/Eucalyptusexport EC2_PRIVATE_KEY=${EUCA_KEY_DIR}/euca2-admin-f5052272-pk.pemexport EC2_CERT=${EUCA_KEY_DIR}/euca2-admin-f5052272-cert.pemexport EC2_JVM_ARGS=-Djavax.net.ssl.trustStore=${EUCA_KEY_DIR}/jssecacertsexport EUCALYPTUS_CERT=${EUCA_KEY_DIR}/cloud-cert.pemexport EC2_ACCESS_KEY='WKy3rMzOWPouVOxK1p3Ar1C2uRBwa2FBXnCw'export EC2_SECRET_KEY='GbxmI31soVHbDr8o6dQpOwTAmlrrrisP1HSQ'# This is a bogus value; Eucalyptus does not need this but client tools do.export EC2_USER_ID='339620359308903565207410324383178424330'alias ec2-bundle-image="ec2-bundle-image --cert ${EC2_CERT} --privatekey ${EC2_PRIVATE_KEY} --user 339620359308903565207410324383178424330 --ec2cert ${EUCALYPTUS_CERT}"alias ec2-upload-bundle="ec2-upload-bundle -a ${EC2_ACCESS_KEY} -s ${EC2_SECRET_KEY} --url ${S3_URL} --ec2cert ${EUCALYPTUS_CERT}“* Need lines in red to register with HyperFox etc.

euca2ools• Initialization variables:

$ . ~/.euca/eucarc # This sources the environment, check with env

• Informational $ euca-describe-availability-zones verbose $ euca-describe-images $ euca-describe-keypairs $ euca-describe-instances $ euca-describe-volumes $ euca-describe-snapshots $ euca-describe-addresses $ euca-describe-groups

euca2ools 2• Keypairs:

$ euca-add-keypair mykey > mykey.priv $ euca-delete-keypair mykey

• Registering kernel image: $ euca-bundle-image -i vmlinuz-2.6.28-11-server --kernel true $ euca-upload-bundle -b mybucket -m /tmp/vmlinuz-2.6.28-11-server.manifest.xml $ euca-register mybucket/vmlinuz-2.6.28-11-server.manifest.xml

• Registering ramdisk image: $ euca-bundle-image -i initrd.img-2.6.28-11-server $ euca-upload-bundle -b mybucket -m /tmp/initrd.img-2.6.28-11-server.manifest.xml $ euca-register mybucket/initrd.img-2.6.28-11-server.manifest.xml

euca2ools 3• Registering disk image:

$ euca-bundle-image -i image.img --kernel eki-XXXXXXXX --ramdisk eri-XXXXXXXX $ euca-upload-bundle -b mybucket -m /tmp/image.img.manifest.xml $ euca-register mybucket/image.img.manifest.xml

• Launching and manage instances: $ euca-run-instances emi-XXXXXXXX -k mykey -t c1.medium $ euca-get-console-output i-XXXXXXXX

• Launching an instance associated with the security group “webservers” : $ euca-run-instances emi-XXXXXXXX -k mykey -g webservers

euca2ools 4• Connecting to the instance:

$ ssh -i mykey.priv user@ip $ euca-reboot-instances i-XXXXXXXX $ euca-terminate-instances i-XXXXXXXX

• Create new 10GB volume: $ euca-create-volume -s 10 -z mycloud $ euca-create-volume -s 10 --snapshot snap-ZZZZZZZZ --zone mycloud

• Attach a volume to a runing instance: $ euca-attach-volume -i i-XXXXXXX -d /dev/sdb vol-YYYYYYYY

euca2ools 5• Detach a volume:

$ euca-detach-volume vol-YYYYYYYY $ euca-delete-volume vol-YYYYYYYY ### for windows only

• Snapshots: $ euca-create-snapshot vol-YYYYYYYY $ euca-delete-snapshot snap-ZZZZZZZZ

• Allocating IP address:• $ euca-allocate-address

ADDRESS IP• Releasing a public ip address from a specify user:

$ euca-release-address IP

euca2ools 6MORE ADDRESS IP

• Associating a public ip address to a running instance $ euca-associate-address -i i-XXXXXXXX ip

• Disassociating a public ip address from a running instance $ euca-disassociate-address ip

• Creating a security group named “webservers”: $ euca-add-group -d "Web Servers" webservers

• Adding a rule to the security group “webservers” allowing icmp and tcp traffic from a.b.c.d: $ euca-authorize -P tcp -s a.b.c.d webservers $ euca-authorize -P icmp -s a.b.c.d webservers

euca2ools 7• Removing the rule for icmp traffic from the source ip a.b.c.d from the security• group “webservers”:

$ euca-revoke -P icmp -s a.b.c.d webservers

• Deleting the security group “webservers”: $ euca-delete-group webserversSOURCE:http://cssoss.wordpress.com/2010/05/10/eucalyptus-beginner%E2%80%99s-guide-%E2%8093-uec-edition-chapter-10-%E2%80%93-euca%C2%A0commands

Set Up Images: Create Kernelinitrd-2.6.27.21-0.1-xen vmlinuz-2.6.27.21-0.1-xen[root@css1 xen-kernel]# euca-bundle-image -i vmlinuz-2.6.27.21-0.1-xen -kernel = trueInvalid private key[root@css1 xen-kernel]# euca-bundle-image -i vmlinuz-2.6.27.21-0.1-xen --kernel=trueChecking imageTarring imageEncrypting imageSplitting image...Part: vmlinuz-2.6.27.21-0.1-xen.part.0Generating manifest /tmp/vmlinuz-2.6.27.21-0.1-xen.manifest.xml

Upload and Register[root@css1 xen-kernel]# euca-upload-bundle -b mybucket -m /tmp/vmlinuz-2.6.27.21-0.1-xen.manifest.xmlChecking bucket: mybucketUploading manifest fileUploading part: vmlinuz-2.6.27.21-0.1-xen.part.0Uploaded image as mybucket/vmlinuz-2.6.27.21-0.1-xen.manifest.xml[root@css1 xen-kernel]# euca-register mybucket/vmlinuz-2.6.27.21-0.1-xen.manifest.xmlIMAGE eki-5FB9126E

Set Up Images: Create Ramdisk

euca-bundle-image -i initrd-2.6.27.21-0.1-xenChecking imageTarring imageEncrypting imageSplitting image...Part: initrd-2.6.27.21-0.1-xen.part.0Generating manifest /tmp/initrd-2.6.27.21-0.1-xen.manifest.xml

Upload and Register

euca-upload-bundle -b mybucket -m /tmp/initrd-2.6.27.21-0.1-xen.manifest.xmlChecking bucket: mybucketUploading manifest fileUploading part: initrd-2.6.27.21-0.1-xen.part.0Uploaded image as mybucket/initrd-2.6.27.21-0.1-xen.manifest.xml

[root@css1 xen-kernel]# euca-register mybucket/initrd-2.6.27.21-0.1-xen.manifest.xmlIMAGE eri-410611E1

Set Up Images: Create Image

[root@css1 euca-ubuntu-9.04-x86_64]# euca-bundle-image -i ubuntu.9-04.x86-64.img --kernel eki-5FB9126E --ramdisk eri-410611E1

Checking imageTarring imageEncrypting imageSplitting image...Part: ubuntu.9-04.x86-64.img.part.0Part: ubuntu.9-04.x86-64.img.part.1Part: ubuntu.9-04.x86-64.img.part.2Part: ubuntu.9-04.x86-64.img.part.3Part: ubuntu.9-04.x86-64.img.part.4Part: ubuntu.9-04.x86-64.img.part.5Part: ubuntu.9-04.x86-64.img.part.6Part: ubuntu.9-04.x86-64.img.part.7Part: ubuntu.9-04.x86-64.img.part.8Part: ubuntu.9-04.x86-64.img.part.9Part: ubuntu.9-04.x86-64.img.part.10Part: ubuntu.9-04.x86-64.img.part.11Part: ubuntu.9-04.x86-64.img.part.12Part: ubuntu.9-04.x86-64.img.part.13Generating manifest /tmp/ubuntu.9-04.x86-64.img.manifest.xml

Upload[root@css1 euca-ubuntu-9.04-x86_64]# euca-upload-bundle -b mybucket -m /tmp/ubuntu.9-04.x86-64.img.manifest.xmlChecking bucket: mybucketUploading manifest fileUploading part: ubuntu.9-04.x86-64.img.part.0Uploading part: ubuntu.9-04.x86-64.img.part.1Uploading part: ubuntu.9-04.x86-64.img.part.2………………………………………………………………………….Uploading part: ubuntu.9-04.x86-64.img.part.10Uploading part: ubuntu.9-04.x86-64.img.part.11Uploading part: ubuntu.9-04.x86-64.img.part.12Uploading part: ubuntu.9-04.x86-64.img.part.13Uploaded image as mybucket/ubuntu.9-04.x86-64.img.manifest.xml

Register[root@css1 euca-ubuntu-9.04-x86_64]# euca-register mybucket/ubuntu.9-04.x86-64.img.manifest.xmlIMAGE emi-345611F6

Wentworth Eucalyptus Front End

• Use to view images/instances/credentials/ and manage users

hybridfox• Use to manage instances/AWS/Security Groups/Volumes/Snapshots/etc.• http://code.google.com/p/hybridfox/

SUSE Studiohttp://susestudio.com/home

Troubleshooting

• There are many moving parts in Eucalyptus and it can frustrating to fix

• The most common failure case is that you try to run a new instance and it stays "pending" for a long time before terminating

• When this happens, there are a few places to check

Troubleshooting: DHCP• If the instance does not get an IP address (public or private) then

mostly likely the DHCP assignments are not working• When you actually run an instance, the cloud controller on the front-

end will start a dhcpd process to assign IP addresses to the new instance

• If you do a 'ps –ef | grep dhcpd' and don't see anything, then DHCP is not running during setup

• This almost certainly means that the VNET configuration is wrong (VNET overlap with private network, etc)

• You can also look in /var/run/eucalyptus/net/ for the DHCP configuration and lease files

Troubleshooting: Log Files

• The best place to look for clues is in the log files• As with most log files, it can be hard to pull the

relevant information out of the noise• On the front-end, look in

/var/log/eucalyptus/cc.log• On the nodes, look in /var/log/eucalyptus/nc.log

Troubleshooting Example: cc.log

• Look for the above "resource response summary" messages

• They will tell you how many resources are available for each instance size

• In the above, you see that it says there are 0 available and 0 total for each size which means that the front-end can't talk to the nodes, probably because of a firewall problem

[DAY MON DD HH:MM:SS YYYY][004513][EUCAINFO ] DescribeResources(): resource response summary (name{avail/max}): m1.small{0/0} c1.medium{0/0} m1.large{0/0} m1.xlarge{0/0} c1.xlarge{0/0}

Troubleshooting Example: cc.log

• When the nodes and front-end can talk, you should messages like this:

• And like this from the individual nodes:

[DAY MON DD HH:MM:SS YYYY][004513][EUCAINFO ] DescribeResources(): resource response summary (name{avail/max}): m1.small{16/16} c1.medium{16/16} m1.large{8/8} m1.xlarge{8/8} c1.xlarge{4/4}

[DAY MON DD HH:MM:SS YYYY][011297][EUCADEBUG ] refresh_resources(): received data from node=192.168.0.2 mem=16083/16083 disk=856137/856137 cores=16/16

Troubleshooting Example: nc.log

• In general, look for the [EUCAERROR lines to find errors that you need to address

• In the above example, it tells you that the node can't contact the walrus component of front end (for S3 storage) (i.e. snapshots and images)

• Again, this is probably a firewall issue, so be sure the firewall is off on the node

[DAY MON DD HH:MM:SS YYYY][006411][EUCAERROR ] walrus_request(): couldn't connect to host (7)[DAY MON DD HH:MM:SS YYYY][006411][EUCAERROR ] download retry 10 of 10 will commence in 4 seconds

Troubleshooting: Images• Another issue is broken or incompatible images• If Eucalyptus tells you that an instance is running but you can't log in,

try pinging it• If you can't ping it, the instance didn't boot correctly• You can test it with xen

– Log in to the node and run "xm list"– You can then log in directly to the instance with "xm console INSTANCE"– If it's waiting at a maintenance prompt then something is wrong with the image

– try another image– Note, do a CTRL-] to exit the xen console

How Many Servers Do I Need?• For most purposes, you should think about 1 core, 1GB of

memory, and 10GB of storage for each student• So, if you have a course of 30 students, you need roughly

32 cores and 32 GB of memory (HDD’s are not costly)• You can get a single server with dual hyper-threaded quad

core processors and 16GB of memory for ~$4k• Add another $2k for the front-end and it's about $10k total

(but don’t forget switches etc.)

Cloud in the Classroom

• There are many ways to bring the cloud into a course• In software courses, every student could be assigned

their own virtual machine for development and testing or run high performance or parallel applications

• In networking courses, students could create entire virtual networks to study protocols and network architectures

Conclusion• If you are interested in setting up a private or hybrid cloud for a course, Xen

is reasonable option• Eucalyptus/AWS are an infrastructure and web services platform in the

cloud• Cloud networking is an important new tool for education, scientific

discovery and business logic services• The benefit is that you can validate and verify locally before for-fee AWS

cloud with a larger implementation• EBS coupled with S3 gives a fast and safe way to protect data.• Freddy Fender “Wasted Days” is good Koala theme music

AWS Info

http://aws.amazon.com/govcloud-us/ http://aws.amazon.com/resources/webinars/ http://aws.typepad.com/http://www.youtube.com/user/AmazonWebServices/videos?view=pl http://calculator.s3.amazonaws.com/calc5.html http://aws.amazon.com/ec2/instance-types/

CONTACTS

David RilettCharlie Wiseman