Hands On: Multimedia Methods for Large Scale Video Analysis(Project Meeting)

Dr. Gerald Friedland, fractor@icsi.berkeley.edu

1

Today

Today

• Amazon EC2

Today

• Amazon EC2– What is it?

Today

• Amazon EC2– What is it?– Concepts to understand before using

it

Today

• Amazon EC2– What is it?– Concepts to understand before using

it– Some tutorials

Today

• Amazon EC2– What is it?– Concepts to understand before using

it– Some tutorials– Issues: Data, cost, and other

considerations

Today

• Amazon EC2– What is it?– Concepts to understand before using

it– Some tutorials– Issues: Data, cost, and other

considerations• More on Project Ideas

3

Amazon EC2

4

Amazon EC2

• EC2 = Elastic Compute Cluster

4

Amazon EC2

• EC2 = Elastic Compute Cluster– Configurable set of Virtual Machines

(=Instance) running on real machines

4

Amazon EC2

• EC2 = Elastic Compute Cluster– Configurable set of Virtual Machines

(=Instance) running on real machines– Storage is virtualized as well

4

Amazon EC2

• EC2 = Elastic Compute Cluster– Configurable set of Virtual Machines

(=Instance) running on real machines– Storage is virtualized as well

• Originally designed for scalable web shops (like Amazon.com) - Software as a Service (SaaS)

4

Amazon EC2

• EC2 = Elastic Compute Cluster– Configurable set of Virtual Machines

(=Instance) running on real machines– Storage is virtualized as well

• Originally designed for scalable web shops (like Amazon.com) - Software as a Service (SaaS)

• Now: IaaS (Infrastructure as a Service)4

Infrastructure on Demand

5

Infrastructure on Demand

5

• Hardware On Demand

Infrastructure on Demand

5

• Hardware On Demand • Pay for what you use

Infrastructure on Demand

5

• Hardware On Demand • Pay for what you use • Full root access – you control the

OS and Software Stack

Infrastructure on Demand

5

• Hardware On Demand • Pay for what you use • Full root access – you control the

OS and Software Stack • Ability to scale computing

resources up and down

Infrastructure on Demand

5

• Hardware On Demand • Pay for what you use • Full root access – you control the

OS and Software Stack • Ability to scale computing

resources up and down • No dealing with racks, networks,

power, cooling, housing, etc.

Amazon EC2

6

Amazon EC2• Resizable Compute is controlled

via Instances either through

6

Amazon EC2• Resizable Compute is controlled

via Instances either through – Web Interface (Amazon Web Services)

6

Amazon EC2• Resizable Compute is controlled

via Instances either through – Web Interface (Amazon Web Services) – API

6

Amazon EC2• Resizable Compute is controlled

via Instances either through – Web Interface (Amazon Web Services) – API

• Variety of Instance Sizes: CPU Power, Cores, RAM, Disk.

6

Amazon EC2• Resizable Compute is controlled

via Instances either through – Web Interface (Amazon Web Services) – API

• Variety of Instance Sizes: CPU Power, Cores, RAM, Disk.

• Wide Variety of Pre-built AMIs (Amazon Machine Images)

6

Amazon EC2• Resizable Compute is controlled

via Instances either through – Web Interface (Amazon Web Services) – API

• Variety of Instance Sizes: CPU Power, Cores, RAM, Disk.

• Wide Variety of Pre-built AMIs (Amazon Machine Images)

• Access for each running instance using keypair required for SSH access. 6

7

EC2: Instance Examples

EC2: Special Instances

8

EC2: Special Instances• “Spot” Instances

8

EC2: Special Instances• “Spot” Instances

– Bid for unused AWS capacity

8

EC2: Special Instances• “Spot” Instances

– Bid for unused AWS capacity – Prices controlled by AWS based on

supply and demand

8

EC2: Special Instances• “Spot” Instances

– Bid for unused AWS capacity – Prices controlled by AWS based on

supply and demand • AWS can terminate Spot Instances

without notice

8

EC2: Special Instances• “Spot” Instances

– Bid for unused AWS capacity – Prices controlled by AWS based on

supply and demand • AWS can terminate Spot Instances

without notice • Best approach to temporary

requests for large numbers of servers

8

EC2: Special Instances• “Spot” Instances

– Bid for unused AWS capacity – Prices controlled by AWS based on

supply and demand • AWS can terminate Spot Instances

without notice • Best approach to temporary

requests for large numbers of servers

• Default maximum = 100 servers (instead of 20 on-demand) 8

EC2: More Concepts

9

EC2: More Concepts

9

• Regions: A region is a geographical area that contains one or more Availability ZonesData transfer: Cheap(er)

EC2: More Concepts

9

• Regions: A region is a geographical area that contains one or more Availability ZonesData transfer: Cheap(er)

• Availability Zone: Some services only available in the same AZ.Data transfer: Free

EC2: Even More Concepts

10

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)• Contains an entire operating system and software stack that can be

loaded onto one or more virtual machines

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)• Contains an entire operating system and software stack that can be

loaded onto one or more virtual machines

• Amazon Elastic Block Storage (EBS)

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)• Contains an entire operating system and software stack that can be

loaded onto one or more virtual machines

• Amazon Elastic Block Storage (EBS)• Persistent storage: Volume lifetime is independent of any particular

EC2 instance.

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)• Contains an entire operating system and software stack that can be

loaded onto one or more virtual machines

• Amazon Elastic Block Storage (EBS)• Persistent storage: Volume lifetime is independent of any particular

EC2 instance. • Raw, unformatted, block device.

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)• Contains an entire operating system and software stack that can be

loaded onto one or more virtual machines

• Amazon Elastic Block Storage (EBS)• Persistent storage: Volume lifetime is independent of any particular

EC2 instance. • Raw, unformatted, block device.• Performance equal to or better than local EC2 drive.

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)• Contains an entire operating system and software stack that can be

loaded onto one or more virtual machines

• Amazon Elastic Block Storage (EBS)• Persistent storage: Volume lifetime is independent of any particular

EC2 instance. • Raw, unformatted, block device.• Performance equal to or better than local EC2 drive. • Built-in redundancy within availability zone. AFR (Annual Failure

Rate) between 0.1% and 1%.

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)• Contains an entire operating system and software stack that can be

loaded onto one or more virtual machines

• Amazon Elastic Block Storage (EBS)• Persistent storage: Volume lifetime is independent of any particular

EC2 instance. • Raw, unformatted, block device.• Performance equal to or better than local EC2 drive. • Built-in redundancy within availability zone. AFR (Annual Failure

Rate) between 0.1% and 1%. • Sizes range from 1 GB to 1 TB.

EC2: Even More Concepts

10

• Amazon Machine Image (AMI)• Contains an entire operating system and software stack that can be

loaded onto one or more virtual machines

• Amazon Elastic Block Storage (EBS)• Persistent storage: Volume lifetime is independent of any particular

EC2 instance. • Raw, unformatted, block device.• Performance equal to or better than local EC2 drive. • Built-in redundancy within availability zone. AFR (Annual Failure

Rate) between 0.1% and 1%. • Sizes range from 1 GB to 1 TB.

• Easy to create, attach, back up, restore, and delete volumes.

EC2: More on Storage

11

EC2: More on Storage

11

• Amazon Simple Storage Solution (S3)

EC2: More on Storage

11

• Amazon Simple Storage Solution (S3)• “... a simple web service interface that can be used

to store and retrieve any amount of data, at any time, from anywhere on the web”

EC2: More on Storage

11

• Amazon Simple Storage Solution (S3)• “... a simple web service interface that can be used

to store and retrieve any amount of data, at any time, from anywhere on the web”

• Read, write, and delete binary obejcts containing from 1 byte to 5 TB of data each using API.

EC2: More on Storage

11

• Amazon Simple Storage Solution (S3)• “... a simple web service interface that can be used

to store and retrieve any amount of data, at any time, from anywhere on the web”

• Read, write, and delete binary obejcts containing from 1 byte to 5 TB of data each using API.

• Number of objects you can store is unlimited.

EC2: More on Storage

11

• Amazon Simple Storage Solution (S3)• “... a simple web service interface that can be used

to store and retrieve any amount of data, at any time, from anywhere on the web”

• Read, write, and delete binary obejcts containing from 1 byte to 5 TB of data each using API.

• Number of objects you can store is unlimited. • Each object stored in a 'bucket'and retrieved via a

unique, user-assigned key

EC2: More on Storage

11

• Amazon Simple Storage Solution (S3)• “... a simple web service interface that can be used

to store and retrieve any amount of data, at any time, from anywhere on the web”

• Read, write, and delete binary obejcts containing from 1 byte to 5 TB of data each using API.

• Number of objects you can store is unlimited. • Each object stored in a 'bucket'and retrieved via a

unique, user-assigned key• Different levels of reliability.

EC2: More on Storage

11

• Amazon Simple Storage Solution (S3)• “... a simple web service interface that can be used

to store and retrieve any amount of data, at any time, from anywhere on the web”

• Read, write, and delete binary obejcts containing from 1 byte to 5 TB of data each using API.

• Number of objects you can store is unlimited. • Each object stored in a 'bucket'and retrieved via a

unique, user-assigned key• Different levels of reliability. • Generally cheaper than EBS

EBS vs S3

12

EBS vs S3

12

• EBS only mountable to one instance

EBS vs S3

12

• EBS only mountable to one instance

• EBS can only be used with instances in same AZ

EBS vs S3

12

• EBS only mountable to one instance

• EBS can only be used with instances in same AZ

• EBS <-> S3 can be converted,but $$$

Available Datasets

Available Datasets

• Amazon makes Public Datasets available as EBS

Available Datasets

• Amazon makes Public Datasets available as EBS

• 1M Song Dataset and 10k subset is available.

Available Datasets

• Amazon makes Public Datasets available as EBS

• 1M Song Dataset and 10k subset is available.

• More info on: http://aws.amazon.com/publicdatasets/

Using EC2 for HPC: Intro

http://www.youtube.com/embed/YfCgK1bmCjw

Using EC2 for HPC: Intro

http://www.youtube.com/embed/YfCgK1bmCjwWatch video at:

Problem

Problem

EC2 provides raw compute power. There’s work to be done to create a usable cluster:

Problem

EC2 provides raw compute power. There’s work to be done to create a usable cluster:

•Software installation

Problem

EC2 provides raw compute power. There’s work to be done to create a usable cluster:

•Software installation •AMI creation

Problem

EC2 provides raw compute power. There’s work to be done to create a usable cluster:

•Software installation •AMI creation •AWS / SSH key management and distribution

Problem

EC2 provides raw compute power. There’s work to be done to create a usable cluster:

•Software installation •AMI creation •AWS / SSH key management and distribution •Persistent Disk Storage and File Sharing

Problem

EC2 provides raw compute power. There’s work to be done to create a usable cluster:

•Software installation •AMI creation •AWS / SSH key management and distribution •Persistent Disk Storage and File Sharing •Configuration management

Problem

EC2 provides raw compute power. There’s work to be done to create a usable cluster:

•Software installation •AMI creation •AWS / SSH key management and distribution •Persistent Disk Storage and File Sharing •Configuration management •Higher-level management (cluster vs. instance)

Alternative: Elastic MapReduce

Alternative: Elastic MapReduce

• Launch MapReduce jobs on EC2 using Hadoop

Alternative: Elastic MapReduce

• Launch MapReduce jobs on EC2 using Hadoop

• Workflows defined either on console or using Webinterface

Alternative: Elastic MapReduce

• Launch MapReduce jobs on EC2 using Hadoop

• Workflows defined either on console or using Webinterface

• Map/Reduce code in S3

Alternative: Elastic MapReduce

• Launch MapReduce jobs on EC2 using Hadoop

• Workflows defined either on console or using Webinterface

• Map/Reduce code in S3• Input/Output data stored in S3

Alternative: Elastic MapReduce

• Launch MapReduce jobs on EC2 using Hadoop

• Workflows defined either on console or using Webinterface

• Map/Reduce code in S3• Input/Output data stored in S3• MapReduce covered in future lecture

Alternative: Elastic MapReduce

• Launch MapReduce jobs on EC2 using Hadoop

• Workflows defined either on console or using Webinterface

• Map/Reduce code in S3• Input/Output data stored in S3• MapReduce covered in future lecture• More info on Amazon:

http://aws.amazon.com/elasticmapreduce/

Alternative: MIT Starcluster

Alternative: MIT Starcluster

Watch video at:

Alternative: MIT Starcluster

Watch video at: http://www.youtube.com/watch?v=vC3lJcPq1FY

EC2 Cost: Free Services

• 750 hours of EC2 running Micro instance usage• 30 GB of Amazon EBS Standard volume storage

plus 2 million IOs and 1 GB snapshot storage• 15 GB of bandwidth out aggregated across all AWS

services• 1 GB of Regional Data Transfer

Data Transfer Cost: Between Instances

Data Transfer Cost: Between Instances

• Inside AZ: Free

Data Transfer Cost: Between Instances

• Inside AZ: Free• Inside Region: $0.01/GB

Data Transfer Cost: Between Instances

• Inside AZ: Free• Inside Region: $0.01/GB• Public IP (even in AZ): $0.01/GB

Data Storage Cost

CPU Cost

The Tradeoffs

The Tradeoffs

• More CPUs vs more power per CPU

The Tradeoffs

• More CPUs vs more power per CPU• What data to transfer, what data to

process locally

The Tradeoffs

• More CPUs vs more power per CPU• What data to transfer, what data to

process locally• Self-configured vs automatic

(Starcluster) vs predefined (MapReduce)

The Tradeoffs

• More CPUs vs more power per CPU• What data to transfer, what data to

process locally• Self-configured vs automatic

(Starcluster) vs predefined (MapReduce)

• Console vs Web Interface vs API

The Tradeoffs

• More CPUs vs more power per CPU• What data to transfer, what data to

process locally• Self-configured vs automatic

(Starcluster) vs predefined (MapReduce)

• Console vs Web Interface vs API• S3 vs EBS

The Tradeoffs

• More CPUs vs more power per CPU• What data to transfer, what data to

process locally• Self-configured vs automatic

(Starcluster) vs predefined (MapReduce)

• Console vs Web Interface vs API• S3 vs EBS• Amazon vs. stay local (ICSI)

More on Project Ideas

More on Project Ideas

• ACM Multimedia Grand Challengehttp://www.acmmm12.org/call-for-multimedia-grand-challenge-solutions/

More on Project Ideas

• ACM Multimedia Grand Challengehttp://www.acmmm12.org/call-for-multimedia-grand-challenge-solutions/

• IEEE AASP Challenge:http://www.elec.qmul.ac.uk/digitalmusic/sceneseventschallenge/

More on Project Ideas

• ACM Multimedia Grand Challengehttp://www.acmmm12.org/call-for-multimedia-grand-challenge-solutions/

• IEEE AASP Challenge:http://www.elec.qmul.ac.uk/digitalmusic/sceneseventschallenge/

• ACM MM 2011 Proceedingshttp://dl.acm.org/citation.cfm?id=2072298

More on Project Ideas

• ACM Multimedia Grand Challengehttp://www.acmmm12.org/call-for-multimedia-grand-challenge-solutions/

• IEEE AASP Challenge:http://www.elec.qmul.ac.uk/digitalmusic/sceneseventschallenge/

• ACM MM 2011 Proceedingshttp://dl.acm.org/citation.cfm?id=2072298

• IEEE ICME 2012 Proceedingshttp://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=5997811

This Week (Lecture)

24

• More on Audio

Next Week (Project Meeting)

25

• Project Teams