[AWS Black Belt Online Seminar] Amazon EC2 · 2019-06-13 · © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS Webinar

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

AWS ��Webinarhttps://amzn.to/JPWebinar

�� https://amzn.to/JPArchive

��EC2� ��2019-03-06

Amazon EC2��

[AWS Black Belt Online Seminar]

https://amzn.to/JPWebinar

https://amzn.to/JPArchive

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

��

�� (��)�� / EC2��&( "��EC2��#'$%

�� EC2��+��*�!)��

��AWS��• Amazon EC2 Auto Scaling• AWS Support


AWS Black Belt Online Seminar ��

��( =��#0-��.1=��4:=�� %�*�<��*$1��)��( �,'13628�95��1/�1"+&��0�!��

7>�;��••

S

Twitter #awsblackbelt


��

• [=a��2019T366R>N�#�'%Q^��./��J\�� 9F�EZ�AWS7?!"(# &(http://aws.amazon.com/)��0S��

• =a;G��BXM,�� =aQ�./�AWS7?!"(# &1:�./�L-��D8�AWS7?!"(# &�./�]K��

• ./�IUW1�� R[4A@��2_�P5*�$)+�<` �D8�YOCVI�H3��

• AWS does not offer binding price quotes. AWS pricing is publicly available and is subject to change in accordance with the AWS Customer Agreement available at http://aws.amazon.com/agreement/. Any pricing information included in this document is provided only as an estimate of usage charges for AWS services based on certain information that you have provided. Monthly charges will be based on your actual use of AWS services, and may vary from the estimates provided.


��

• �)!$�1��1��;F( )72�( )�OD��• �1 &��1

• +!( )-1�."��/• �"( )$'/

• IGJH��1��1�,�#��CloudWatch ( )�AL• >��

• �"( )$0� : CE�@� ��• ( )54N• ( )9:• Q=�P( )��1��1��(�8R• (M<��)�B( )36��1#*1%36�?K


��


�� - Amazon EC2��

A 10 (

2 //

/ nEC

)%

/ /

(/

o 3 z

/ /

m a1 1


E 2 C

C

CC

��


Amazon EC2��

��• 7<930• >=6�,��+��7A -"��-� �HPC/�(��*-�(-�

��• /C�(��

1,000;.9�4D2E• &!�'-��-��)• GB1��API��@

5

� ��• /C�(��-�&-��(��#��%�� 8?2E

• F:��-��-��$�8?2E


��


�� - 110��

• z ig(s lw)• vn z

: cu• cu AWS

tm•

• 2017 8h26 ravCPU 110( y)

• ebp

• o ( )22/1 1 .- . . 1 1 - 2 - / . 11 -2 1.- - 1 2 .- / 1 1/.2 -12 - 1

https://aws.amazon.com/blogs/aws/natural-language-processing-at-clemson-university-1-1-million-vcpus-ec2-spot-instances/


�� - 110��

• i(u mny)• z p l

ew• ew AWSb vo

•• 2017 8 26 tc

vCPU 110(a )

• : : : : gr h

• ( : ): s

22/1 1 .- . . 1 1 - 2 - / . 11 -2 1.- - 1 2 .- / 1 1/.2 -12 - 1



��

H L I NR 5 / H FGT

0 D) 2

F 2 I P

2

7 D2 C DC DcA ad

C A

8 3D

( 2% I PDD /

50877 D) 2(

( E 2Y a % da &


��

• 1/42�0��$��'.�)� �*�-��,��• ��!��(�.(�.�(.��'.)

• ��"�.��3��*�-��,��• #*� +�&��%��


��AWS��OSS, ��

AWSCloudFormation

AWSOpsWorks

Amazon EMR

Amazon ECS

AWS Data Pipeline

AWS BatchAuto Scaling

AWS ThinkboxDeadline


��


�� - 201711

��


�� (1)

1 ) 41- 1 ) 1

. )

�. ) ( . )

1 ) 4 (

. )

�. ) ( . )


�� (2)

1 ) 41- 1 ) 1

. )

�( . )

1 ) 4 (

. )

�( . )

��

��

��

. )

��

. )

��


�� (3)

4 .,(4 4 ,- 4

1 -,

�1 -,

4 .,

1 -,

�1 -,

� �

�

�

1 -,

�

1 -,

�

�� T R T ) A


�� (4)

1 ) 41- 1 ) 1

. ) ( . )

1 ) 4 (

. ) ( . )

10 10

10

10

. )

10

. )

10

$

.R

$


��(1)

6 4 - 1 )-.0 4

2 .- 2 .-

143 143

2 .-

143

$

20. o m ks i Race

g lce r y

• h2 .-

• T p ce nt ks $ ($

o

.


��(2)

- 67 1 - - 73 7 0276

( 4- 0 .( 4- 0

021 021

( 4- 0

021

$

R T R T c keR T R R gr

• n - .( 4- 0 ao• hm R T

$ $ $ )$p

• R T c

R l isao

.


��(1)

4 -1 )-. 4

2 .-

$2 .-2 .-

4 ( C e T T ET T Ce A

• a RT2 .-C

c C


��(2)

- 7 1 - - 73 7 027

5( 4- 0 .( 4- 0

042 042

( 4- 0

042

)(

hg R i T

• c - .( 4- 0R

• R $ $ $

• R $ $$ $ )$

1. 3$

e a


��

• "($&�*"#*"�+-�@/10�7E�.��8��?:��

• "($&�*"#*"� AWS ��><�� 4�� 2 C;��A=��• Amazon EC2�2� )'!%��5D��• 6B��+-(�93+-�)�"($&�*"#*"+-�9,��


��


��

• ��2!�� • ��

• ��• CloudWatch Events


�� -��

• ��• �)� ��"/�%$�+�• http://169.254.169.254/latest/meta-data/ ��,��• https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/ec2-

instance-metadata.html• )&*(

• http://169.254.169.254/latest/meta-data/spot/instance-action ��,��

• .'�• 5-��#!• https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-

interruptions.html

https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-interruptions.html


��

{

"version": "0",

"id": "12345678-1234-1234-1234-123456789012",

"detail-type": "EC2 Spot Instance Interruption Warning",

"source": "aws.ec2",

"account": "123456789012",

"time": "yyyy-mm-ddThh:mm:ssZ",

"region": "us-east-2",

"resources": ["arn:aws:ec2:us-east-2:123456789012:instance/i-1234567890abcdef0"],

"detail": {

"instance-id": "i-1234567890abcdef0",

"instance-action": "action"

}

}


�� - CloudWatch Events��

. / . - . / / . / / / / 22 / /

https://aws.amazon.com/blogs/compute/taking-advantage-of-amazon-ec2-spot-instance-interruption-notices/


�� - CloudWatch Events��


��


��

�� AWS CLI

(aws ec2 ...)

��

��

�!��

��

(EC2)

run-instances(request-spot-

instances)

one-time (sync) /persistent (sync)

request-spot-fleet /

vCPU ( )request (async) / maintain (async)

EC2 create-fleet / vCPU ( )

instant (sync) / request (async) / maintain (async)

Auto Scaling(New!)

- ,-


m4.large Spot ASG Min: 1 Max: 10

m5.large Spot ASG Min: 1 Max: 10

c4.xlarge O-D ASG Min: 1 Max: 10

Availability Zone 1

Availability Zone 2

Availability Zone 3

- C ACASG

ASG 2E

- CC2ASG E

ASG: Auto Scaling GroupO-D: On Demand Instance


2 / - E DE S C C G R

m4.large Spot Instances

m5.large Spot Instances

c4.xlarge On-Demand instances

Availability Zone 1

Availability Zone 2

Availability Zone3

ASG SOGAI S

C CG R

Single ASG

ASG: Auto Scaling GroupO-D: On Demand InstanceRI: Reserved Instance


��


��

�� AWS CLI

(aws ec2 ...)

��

��

�!��

��

(EC2)

run-instances(request-spot-

instances)

one-time (sync) /persistent (sync)

request-spot-fleet /

vCPU ( )request (async) / maintain (async)

EC2 create-fleet / vCPU ( )

instant (sync) / request (async) / maintain (async)

Auto Scaling(New!)

- ,-


��


��(1)


��(2)

��' ��%*��!��!��&+�)"��• Load balancing: $��!��!��'(• Flexible workloads: ��#��!��!��'(


��(3)

�!��"��# � ��• ��#��vCPU�� • ��!��$ �


��(4)

• �% "$'�"�-;��% "�(��(��>?��75��vCPU 0:�

• �% "$'�"�*=��(!&(#�(��(� .6 ��

• �� ">?�)1�� <��98�� 0:>?�+,��3/��% "$'�"�2; 4��(��(��-;�


��(5)

• *2�1�� EC2�"�� $��$��!�02�'%�4,��

• ��'%��#��.)��/-+(�35��&


��(6)

• ��$��$��'��%�• �� !��#�&��• (" ��


EC2 Auto Scaling Group��(1)

• EC2��EC2 Auto Scaling�� !��





• �"+�� '�#��&(�.!�#��-%&(�

• *��$)�� ,



• 2/�,�7• 1. 01/��&�!��• 2. ��!�%'��'�$' "��2/��'�$' �*4

• 3. .��"��3��'�$' )-�� 65

• 2/�,�7�8• ��&�!��12• ��!�%'��'�$' "��2

• �"��3��'�$' )-��30:70

• +(• �'�$' 22+32• �#��72


<��> EC2 Auto Scaling�Spot Fleet�� (1)

(: EC2 Auto Scaling Spot Fleet

'>?@�&/ � �09��$%� � �

<4AZ�.8 � �<4�%��%�� .8 � ��%�"%��%��%��*+ � [1]

1. 564�Total target capacity2. ��"Optional On-demand portion�74��%�"%��)93. -��!��)9

� [2]1. 564�Desired Capacity2. ��"On-Demand base"�74��%�"%��)93. -��OD:Spot Ratio ��=;

ELB��#��13 � �Launch Template$latest, $default��,2

� �

[1] https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-fleet.html#on-demand-in-spot[2] https://docs.aws.amazon.com/ja_jp/autoscaling/ec2/userguide/AutoScalingGroup.html#asg-purchase-options


• ,9�� (Termination Policy)• �� 8& (Termination

Protection)• ��• !*4� �� (-

(Entering Standby)• � �� 10

(Suspending Processes)• ��'7) (Rebalancing

activities)

<��> EC2 Auto Scaling�Spot Fleet�� (2) -��

• �� #;• 6/ �� 3+

• .2:$(��*)��(%"(5+*)�3+


��

�� –��1 ��

�� –��1 �

��

60!)7� =%5-� 60!)7� 60!)7�*#0!)7� =%5-� *#0!)7��

(&�1*#8�=%5-�

*#0!)7�

• $4�� =%�./��"31��=%�2<��

• $4&�1*#0!(�� *#�) =%94�;:��94��+'�� *#��,�=%�2<��


�� (� ��)

2 ==

$ $

( (=

=

) )) )

= 43 2 1 =


�� (� ��)

1$ $

4

( (( (

) 3= 2 1


�� (� ��)

() 1 =31= =

1=

=

1=

=

$

2= =

(


�� (�� )

=

3 5= 3

21

$ $

1 =

3

= 3

( (( (

4

) =3


�� (�� )

3 53

) 21

$ $

1 3

3

4 =

(3


�� (�� )

() 1 =31= =

1=

=

1=

=

$

2= =

(


��

• � - Amazon EC2 �� - Amazon EC2 | AWS https://aws.amazon.com/jp/ec2/spot/pricing/


�� (1)EC2� ��


�� (2)AWS CLI: describe-spot-price-history

$ aws ec2 describe-spot-price-history --instance-types m3.medium{

"SpotPriceHistory": [{

"Timestamp": "2019-03-04T21:36:38.000Z","AvailabilityZone": "ap-northeast-1a","SpotPrice": "0.109600","ProductDescription": "SUSE Linux","InstanceType": "m3.medium"

},{

"Timestamp": "2019-03-04T21:36:38.000Z","AvailabilityZone": "ap-northeast-1c","SpotPrice": "0.109600","ProductDescription": "SUSE Linux","InstanceType": "m3.medium"

},:



�� (��)

On Demand Auto-Scaling group

Output S3 bucket

Availability Zone A

Availability Zone BSpot Fleet

Upload object into input S3

bucket

Job SQS Queue

Auto Scaling group and Spot fleet scale up based on queue depth and scale down based on CPU utilization

CW metrics

Workers will check for jobs in the queue

Workers will update Job status (start time, SLA end time, etc) in

DynamoDB

Uploads to S3 will trigger a Lamda function to put

jobs in SQS and DynamoDB

EFSEC2 instance worker fleet

Cost-effective Batch Processing with Amazon EC2 Spot | AWS Compute Blog https://aws.amazon.com/blogs/compute/cost-effective-batch-processing-with-amazon-ec2-spot/


AWS Batch��(�)

JobQueuesubmit-joblist-jobsdescribe-jobscancel-jobsterminate-jobs

Compute Environment

ECR/DockerRegistry

S3/EFS/NFS

Autoscaling��ECS��

��

��

Docker Image

JobDefinition

��

��

job ��AWS Batch


��

Master Node (OD

or RI)

S3

EMRFS

Master Fleet

Task Node (Spot)

Task Node (Spot)

Task Node (Spot)

Task Fleet

Instance Store

EBS Volume

Instance Store

EBS Volume

Instance Store

EBS Volume

Core Fleet

HDFS

Instance Fleet

Core Node (OD or RI)



Amazon EMR


��

Stateless Web Server (Spot)

Stateless Web Server (Spot)

Session State Data

Stateless Web Server (On-Demand)

Elastic LoadBalancing

Amazon EC2 Auto Scaling

ECS container 1

Stateless Web Server (On-Demand)

ECS container 2

ECS container 1

ECS container 2

https://github.com/awslabs/ec2-spot-labs/tree/master/workshops/ec2-spot-fleet-web-app


<��> ECS� ��

• c j ECS ga• ECSS_ _ eh i d E A

(/etc/ecs/ecs.config)• ECS AMI C

. . / / - - // /

. . / / - : / - /

https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/ecs-agent-install.html

https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/example_user_data_scripts.html


<��> ECS�� (��)�• ie do cha

• _ _ _ Draining E• ( S _)SNS lj

•→ ilj gC mnA

: . . : / / - - - - - /

https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/container-instance-draining.html


<��> ECS��

• CloudFormation

•

•• :

Draining• SNS E

. - - - - .2 .- / -2/ - 2 / - .-

https://aws.amazon.com/blogs/compute/powering-your-amazon-ecs-cluster-with-amazon-ec2-spot-instances/


��


�� 4+1��

�� 7; 3��

1)(<"��!��$91)

>2�=8��#�=8�#��#��

��+@

:.0�4?%,�-5&

�6/'*��#��#��#��#�'*


�� 4+1�� -��

• z ig(s lw)• vn z

: cu• cu AWS

tm•

• 2017 8h26 ravCPU 110( y)

• ebp

• o ( )22/1 1 .- . . 1 1 - 2 - / . 11 -2 1.- - 1 2 .- / 1 1/.2 -12 - 1



�� 4+1�� -��

4 .,(4 4 ,- 4

1 -,

�1 -,

4 .,

1 -,

�1 -,

� �

�

�

1 -,

�

1 -,

�

�� T R T ) A


��


�� AWS Auto Scaling2019��- ��

�� AWS Auto Scaling 2019 � | �� http://blog.father.gedow.net/2019/02/05/aws-auto-scaling-2019/

�� 2019��


EC2 Spot Instances Workshops

https://ec2spotworkshops.com/


��(��)

• �)!$�1��1��;F( )72�( )�OD��• �1 &��1

• +!( )-1�."��/• �"( )$'/

• IGJH��1��1�,�#��CloudWatch ( )�AL• >��

• �"( )$0� : CE�@� ��• ( )54N• ( )9:• Q=�P( )��1��1��(�8R• (M<��)�B( )36��1#*1%36�?K


��(��)

• �)"%�1� 1��=J972�MK�SG��• �1!' �1

• +"�-1�.#��/• �#�%(/

• MKNL��1� 1�, $� ��CloudWatch Events�CP• @��

• �#�%0� : EI�B� ��• <54R• H9;• U?�TF��1� 1��8V• (Q>��)�D:36��1$*1&36�AO


Q&A

��"��AWS Japan Blog �https://aws.amazon.com/jp/blogs/news/��#��! ��

https://aws.amazon.com/jp/blogs/news/


https://amzn.to/JPWebinar3 �� Black Belt Online Seminar ��

3/5 ( ) 12:00-13:00 Amazon EC2

3/6 (�) 18:00-19:00 Amazon EC2��3/12 ( ) 12:00-13:00 AWS Well-Architected Framework ��3/13 (�) 18:00-19:00 Amazon VPC

3/19 ( ) 12:00-13:00 Amazon FSx for Windows File Server/Lustre

3/20 (�) 18:00-19:00 Amazon EBS

3/26 ( ) 12:00-13:00 Amazon CloudWatch



AWS �� AWS ��




[ ��] AWS Innovate ��

https://amzn.to/AWSInnovateJP

https://amzn.to/AWSInnovateJP


. / . .- a i m

rh a i m o• [ W

ld m A ]p

• jc WbnS W/ : . .

( e)W

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

AWS ��Webinarhttps://amzn.to/JPWebinar

��https://amzn.to/JPArchive

��



Documents

[AWS Black Belt Online Seminar] Amazon EC2 · 2019-06-13 · © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS Webinar