Upload
dangthuan
View
225
Download
1
Embed Size (px)
Citation preview
© 2014 HGST, INC.
Hiroshi Kobayashi, Dev./Lab. IT Systems
HGST Japan, Ltd.
Jul 17, 2014
HPC on AWS
for HDD Development
© 2014 HGST, INC. 2
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 3
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 4
HGST, a Western Digital company
• Founded in 2003 through the combination of the
hard drive businesses of IBM, the inventor of the
hard drive, and
Hitachi, Ltd (“Hitachi”)
• Acquired by Western Digital in 2012
• More than 4,200 active worldwide patents
• Headquartered in San Jose, California
• Approximately 41,000 employees worldwide
• Develops innovative, advanced hard disk drives,
enterprise-class solid state drives, external storage
solutions and services
• Delivers intelligent storage devices that tightly
integrate hardware and software to maximize
solution performance
Capacity Enterprise
Performance Enterprise
Cloud & Datacenter
Enterprise SSD (+3 acquisitions in 2013)
7200 RPM &
CoolSpin
HDDs
Ultrastar®
Ultrastar® &
MegaScale DC™
10K & 15K
HDDs
PCIe
SAS
© 2014 HGST, INC. 5
Zero to Cloud << 12 Months
By Mid 2014:
Cloud eMail – Microsoft Office365
Cloud eMail archiving/eDiscovery
External Single SignOn (off VPN)
Cloud File/Collaboration – BOX
Salesforce.com Integrated to save files in BOX
Cloud–High Performance Computing
(HPC) on Amazon’s AWS
Cloud – Big Data Platform on AWS Extract insights from manufacturing data
Cloud - Data mart and provisioning
service using AWS Red Shift
Cloud – HR Transformation via Workday Deployment
April 2013
© 2014 HGST, INC. 6
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 7
Why choose AWS for HPC?
• Background
- 18 months ago, HPC implementation project was started. Project
team investigated several cloud HPC services except for AWS. But
did not match HGST’s requirement.
- CIO Steve Phillpott recommended AWS for HPC. He had much experience
of HPC on AWS at life-science industry.
- Through several Proof of Concept projects, began to understand
Pros/Cons of On-premise and Cloud HPC.
• Key factors are…
- Scalability, Data transfer, Remote Visualization
- Security, Commercial Application, Cost
- and more….
What kind of benefit can HPC on AWS provide?
© 2014 HGST, INC. 8
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 9
Molecular Dynamics
• CC2 & C3 cluster have equivalent scalability of HGST local
cluster with 10GbE around 64cores
• C3 provide significant improvement to the scalability
• C3 is 1.88x faster than CC2
1.88x faster
※1 EN = Enhanced Networking
※2 placement group enable
※3 evaluated by elapse time
© 2014 HGST, INC. 10
CFD “Star-CCM+”
• CD-adapco Japan provided the scalability data on their cluster.
• C3 provide significant improvement to the scalability
• C3 is 1.81x faster than CR1
• Still behind to physical cluster with InfiniBand
1.81x faster
1.70x slower
※1 EN = Enhanced Networking
※2 placement group enable
※3 evaluated by elapse time
※4 only 200steps
© 2014 HGST, INC. 11
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 12
Remote Visualization
• Result data is too huge to download
• Transferring huge data is NOT a option
• Require Remote Visualization for huge result data
Server – Client Mode (Star-CCM+ special)
Remote Desktop Console
AWS graphic
server
Client
Users
Consume server side
GPU resource and license
Remote access
via RDC/VNC
AWS file
server
Client
Users
Consume server
side license
Consume client side
GPU resource
Not good performance… Slower response
Slower rendering
Great performance!!! Almost same performance
as local workstation with
high-end graphic card
G2
© 2014 HGST, INC. 13
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 14
Data collaboration
• Transferring huge data is NOT a option
• Even 48TB of hs1.8xlarge may not be sufficient
for long term / huge data repository
• High cost for re-computing of large scale model
Client Users
Cluster
Master
Computing
Nodes
Shared storage
S3 bucket
AWS S3
job submission
small data back to client
© 2014 HGST, INC. 15
Performance Summary
• Computing Performance
- C3.8xlarge improved scalability dramatically
- Higher scalability is better
• Remote Visualization
- Star-CCM+ is ready
- Other application are NOT ready
• Data Collaboration
- No need to struggle with the storage capacity and durability
• AWS can support whole process of simulation works!!!
© 2014 HGST, INC. 16
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 17
HGST
Hybrid HPC Architecture
Virtual Private Cloud
AWS
Cluster
Master Computing
Nodes
Client Users
Shared
Storage
data I/O attached
• Local + AWS = Hybrid HPC environment
• AWS + Cycle Computing
http://www.cyclecomputing.com/
• AWS computing nodes will be launched
automatically based on the job requests
Local Cluster
S3 bucket
© 2014 HGST, INC. 18
Shape Compute To Match Work To Be Done
Time
Before:
Shared Cluster Computer
512 core 512core 512core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
Today:
AWS EC2 CC2 Cluster
(Max Total 512 core)
512core waiting
256 core 256 core
128 core 128 core
waiting
waiting
All Jobs Run In Parallel on AWS 1.67x Throughput Improvement
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
© 2014 HGST, INC. 19
Shape Storage To Match Work To Be Done
Client Users
Cluster
Master
Computing
Nodes
Shared storage
S3 bucket
job submission
small data back to client
∞
No need to struggle with the storage capacity and durability!!!
© 2014 HGST, INC. 20
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 21
What’s next for AWS HPC…
• Computing Performance
- More scalability, like InfiniBand
- More resource capacity, especially c3.8xlarge
• Remote Visualization
- Higher performance than RDC-TCP/IP
- AppStream? PC over IP®?
• Commercial Application License
- End User License Agreement (EULA)
- Hybrid License Server
- Consumption Based License?
Local
License
Server
© 2014 HGST, INC. 22
Index
• HGST
• Why choose AWS for HPC?
• Computing Performance
• Remote Visualization
• Data Collaboration
• Flexibility
• What’s Next…
• Summary
© 2014 HGST, INC. 23
Summary
• At this moment, HPC on AWS is NOT perfect
- Scalability, Resource Capacity, Remote Visualization
• HPC on AWS has extremely high flexibility
- Hybrid HPC, Shape Compute/Storage To Match Work To Be Done
• Flexibility will drive to responding to the changing
business model
• Benefit of HPC on AWS should be verified with each
applications based on its characteristic
• Required collaboration with application venders
© 2013 HGST, INC. 24
Tech Deep Dive
TC-04
Thank You