3/9/15  

1  

GridKa School 2015

GridKa School 2015 Public Clouds

GridKa School 2015

Outline

•  Introduction to public clouds •  Amazon Web Services

2

GridKa School 2015

What public clouds are

•  Providers have commercial interests •  Users have no costs concerning purchase, operation

and maintenance of own hardware •  Critical situation concerning data privacy and security

of sensible information •  Fear for a Lock-in situation •  Simple Web Interface •  Raw Infrastructure Resources •  Pay-as-you-go (On-demand access) •  Elastic & “infinite” Capacity

3 GridKa School 2015

Amazon Web Services

4

GridKa School 2015

AWS regions and zones

5 GridKa School 2015

What we will do

6

•  Simple public cloud machine management with AWS Console

•  Use EC2 and EBS

3/9/15  

2  

GridKa School 2015

EC2

7

•  EC2: Elastic Compute Cloud –  There are different (virtual) hardware configurations

available •  Micro, Small, High-CPU, HPC ,etc. •  Based on AMIs (Amazon Machine Image)

–  Virtual machines can be assigned to security groups (Firewall)

GridKa School 2015

EC2

8

•  Log in at https://gks2015.signin.aws.amazon.com/console/ •  Choose EC2 in the Amazon AWS Console •  Go to Instances and then Launch Instance

GridKa School 2015

EC2 •  Instance types and families

–  Micro: small amount of consistent CPU resources with possibility to burst CPU capacity

–  Standard: memory-to-CPU ratios for most general-purpose applications

–  High-CPU: proportionally more CPU resources –  High-Memory: proportionally more memory –  High I/O: provides an increased number of low-latency,

random IOPS –  Cluster Compute: lots of CPU power + increased network

performance –  ClusterGPU: Provides general-purpose graphics

processing units + high CPU + increased network performance

9 GridKa School 2015

EC2 •  Some instance types and details:

–  Micro - ECU: up to 2; virtual cores: 1; memory: 615MiB –  Small - ECU: 1; virtual cores: 1; memory: 1.7GiB –  Large - ECU: 4; virtual cores: 2 (2 ECUs each); memory:

7.5GiB –  Cluster Compute8 XL - ECU: 88; virtual cores: 16 (2x

Xeon E5-2670); memory: 60.5GiB –  Cluster GPU 4 XL - ECU: 33.5; virtual cores: 8 (2x Xeon

X5570) + 2 Nvidia Tesla M2050 Fermi GPUs; memory: 22.5 GiB

10

One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor

GridKa School 2015

EC2

•  On-demand instances pricing

11

Cost(perhour) USEast Europe

Linux Windows Linux Windows

Micro 0.02$ 0.020$ 0.020$ 0.020$

Small 0.08$ 0.115$ 0.085$ 0.115$

Large 0.32$ 0.460$ 0.340$ 0.460$

ClusterCompute8XL 2.40$ 2.970$ 2.700$ 2.970$

ClusterGPU4XL 2.10$ 2.600$ N/A N/A

Pricing is region dependent

GridKa School 2015

EC2

•  Reserved instances pricing – Examples for 1 year:

12

Cost Upfront Hourly

Micro 23$ 0.012$

Small 69$ 0.039$

Large 276$ 0.156$

Pricing is region dependent

3/9/15  

3  

GridKa School 2015

EC2 •  Choose the latest version of Ubuntu Server LTS

13 GridKa School 2015

EC2 •  Choose Micro instances and click on Next:

Configure Instance Details •  Choose 1 instance and click on Next: Add

Storage •  Note on Kernel ID and RAM Disk ID

(Advanced Details): – User selectable kernels are useful for keeping the

instances up to date with security fixes and updates, for using the functionality provided by new distributions as well as for using the specialty applications that have a unique timing requirement

14

GridKa School 2015

EC2

•  Use the default configuration for storage •  Give a name to the new instance •  Create a new Key Pair

– Use your name – Download the new key pair file

•  Choose a default Security Group – These are firewall rules

•  Review and Launch

15 GridKa School 2015

EC2 •  Use the default configuration for storage

– You can add more storage capacity or even add extra volumes

•  Click on Next: Tag Instance •  Provide a Value for the Name key (i.e. your

name) and click on Next: Configure Security Group

•  Click on Select an existing security group and choose “All open” – Security groups are AWS EC2’s firewall

16

GridKa School 2015

EC2 •  Click on Review and Launch and review all

the options •  Click and Launch •  Choose to create a new key pair and give it

a name (use your name) – Key pairs are our way to authenticate at the

launched instance •  Click on Download Key Pair

– Store it in your $HOME and restrict permissions only for owner (chmod 400 FILE.pem)

17 GridKa School 2015

EC2

•  Click on Launch Instances •  Click on the instance ID (i-XXXXXXXX) •  Wait for the running state •  Instances can stopped, rebooted or

terminated from the AWS Console – Also, more copies of a given instance can be

launched •  Get the instance address and connect to it

$ ssh -i PATH_TO_KEYFILE ubuntu@ADDRESS

18

3/9/15  

4  

GridKa School 2015

EC2

•  Install a simple service (DidiWiki) – Small and simple Wiki implementation –  Intended for personal use for notes, To-dos,… – http://swik.net/DidiWiki $ sudo apt-get install didiwiki $ sudo didiwiki -p 8080

•  Visit ADDRESS:8080 and start using it

19 GridKa School 2015

EBS •  Amazon Elastic Block Store (EBS) provides block level

storage volumes for use with Amazon EC2 instances •  They are network-attached, and persist independently

from the life of an instance –  Highly available and reliable

20

GridKa School 2015

EBS

•  In AWS Console choose Volumes (Elastic Block Store)

•  Click on Create Volume – Volume type is standard – Size is 1 GB – Availability Zone is the same as the instance

•  Check it at Instances •  Example: us-east-1c

•  Click on Create

21 GridKa School 2015

EBS •  Click on Tags (being the volume selected

from the list) and give it a name – Click on Create Tag – Key will be Name – Value will be your name – Click on Save

•  Choose to attach volume (in Actions or right-click) – Select the desired instance from the list and click

on Attach

22

GridKa School 2015

EBS

•  Log in the instance •  Format and mount the new volume

– sudo mkfs.ext4 /dev/xvdf – sudo mkdir /media/gks2015 – sudo mount /dev/xvdf /media/gks2015

23 GridKa School 2015

EBS

•  Create a backup of the GridKa School 2005 site $ sudo wget -r http://gridmon-kit.gridka.de/gks/gks05/ -P /media/gks2015/

•  Unmount and detach the volume $ sudo umount /media/gks2015 – At AWS Console choose to detach volume

(in Actions or right-click)

24

3/9/15  

5  

GridKa School 2015

EBS

•  Create a snapshot of the volume – Choose the volume – Give it a name and a description

•  Go to Snapshots (in Elastic Block Store) •  Select the new snapshot and create a

new volume – Volume type is standard – Availability Zone is the same as the instance

25 GridKa School 2015

EBS

•  Return to Volumes (in Elastic Block Store) •  Select the new volume and give it a name

– HINT: use the Snapshot ID to identify it •  Re-attach the first volume you created to

the instance

26

GridKa School 2015

EBS

•  Attach the snapshotted volume to the instance – From AWS Console it should be in /dev/sdg

•  Mount the volume in the instance $ sudo mkdir /media/gks2015b $ sudo mount /dev/xvdg /media/gks2015b

27 GridKa School 2015

EBS

•  Create a backup of the GridKa School 2006 site $ sudo wget -r http://gridmon-kit.gridka.de/gks/gks06/ -P /media/gks2015b/

•  Unmount and detach the volumes – From the instance and AWS Console

28

GridKa School 2015

Exercise

•  Terminate the instance •  Launch 2 copies of it

– Use the Launch More like this •  Attach a volume on each instance

29 HPC School 2013

HPC School 2013