Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
The Impact of High Performance Workloads on the Evolution of the Modern Data Center
Bert Lim
Solutions Consultant, Malaysia, Dell EMC
3
More Data Means More Complex Relationships• To analyze in real-time at a large scale
Amount of Data
4.4 Zb
2013 2020Source: IDC 2014
16+ Zb
Hot Data
44 Zb
The Data Multiplier Effect
Business
Human
Machine
1X 10X 100X
Database Data
VOLUME
VARIETY
VOLUME
VARIETY
VOLUME
VELOCITY
Enterprise/
External Data
Sensor/
External Data
Satellite
Imaging
Sensors
VideoRecording
M2M Log
Files
Bio-
Informatics
Email Documents
Web LogsSocial
• More Data Needs To Be Captured Faster
• Real-time Analytics For Business Insights
• Existing Applications Are Taxed
• Evolving New Applications & Architectures
4
Time is Money
How can faster performance pay off for you?
Source: www.acfe.com
FRAUD
5% of Worldwide GDP
($3.5 Trillion) is lost to
Fraud Annually –
ACFE
E-COMMERCE
More than $3 billion in
lost sales in 2014 (USA ecommerce sites) due to poor performance
Source: www.radware.com Source: www.thinkwithgoogle.com
SEARCH/ADVERTISING
If search results were 0.4 secs slower, Google could
lose over 15 million searches/day resulting in
millions less online ads and revenue
RESEARCH
Biology workloads that were
taking four to five days to run,
and in some cases crashing, are
now finishing in 4 to 5 hours
Source: www.nextplatform.com
5
Future Proof Architecture Needed
LEGACY APPLICATIONS NEW APPLICATIONS
RDBMS
STORAGE SYSTEM
OL
AP
NoSQLIn-Memory DB
Data Pipeline
• Data is now both structured and unstructured
• Larger, undefined working data sets
• Move from batch processing to real-time
6
Vertical Applications Drive Innovation
Innovation Requires
Faster processing of data More cycles
No compromise More complex analytics
Concurrent/Collaborative access Promote experimentation
7
Applications Turn to Parallel File Systems (PFS)
NAS Parallel File system on SAN
File I/O
Storage
File System
Application
Storage
Parallel File System
Application
Compromised Architectural Choices
• Low performance density
• Limited by network bandwidth & latency
• I/O stack bottlenecks
• Can have a complex architecture
Block I/O
Network
StorageNetwork
8
High Performance Parallel File Systems on DSSDDeliver high performance innovation
High
Performance
Flexible
Architecture
Reduce TCO
9
High PerformancePowers next generation analytics applications
10 MILLIONIOPS
100 GB/SBandwidth
144 TBCapacity
100μSLatency
Bring Innovation to Market Faster!
10
Hosts Single or Multiple Apps
AnalyticsData Lake
Flexible Architecture
Minimize Time to Insight
• Simple, agile, & flexible data pipeline served from a single platform
• Move data thru the analytic pipeline anytime without impacting SLAs
• Faster Insights integrating transactional data, big data, and advanced analytics, with high performance parallel file system on DSSD D5
FILE SYSTEM
Flexible & Native Application Interface
Dual Connect up to 48 clients
Client #1 Client #48
Parallel File
System
Oracle Warehouse
11
Lower TCOIndustry leading bandwidth density
DSSD D5’s 5RU footprint is just 1/25th the size of leading solutions
5RU
Simplifies the Environment
Easy to Manage
Lowers TCO
Simplifies the Environment
Architectural Solutions and Benefits
13
Extreme file system performanceAccelerate Your Parallel File SYSTEM (PFS)
10.2 MILLION 4k
IOPS
64.3 GB/S Read
Bandwidth
100 μS Latency
PFS
Up to 48 clients
14
PFS on DSSD– Direct Attached
Simple & scalable architecture up to 48 systems
Server 1 Server 3Server 2 Client 1 Client 45
• 3 EXPRESS Servers
• Up to 48 directly attached systems
• Redundant Connections
• Up to 100TB usable
• Options to scale beyond if needed
Parallel File System
15
PFS on DSSD – Hybrid ArchitectureSimple & scalable architecture beyond 48 systems
40Gbps Network
Server 1 Server 8 Fast Client 1 Fast Client 32
• 8 EXPRESS Server Licenses
• Plus N Client Licenses
• Servers dedicated to PFS
• Fast clients with direct access
2x40Gb 10Gb
10GbClient 1 Client N
Parallel File System
16
PFS PerformanceRead Throughput
Up to 64.3 GB/s
Read
17
PFS PerformanceWrite Throughput
Up to
17.5 GB/s Write
Up to
17.5 GB/s Write
18
PFS PerformanceIOPS & Latency
Up to
10.2M 4k IOPS
Up to
10.2M 4k IOPS
@ 600 μs
19
Built for Analytics and Designed for Speed
Simple and Flexible Architecture
Lowest TCO & Highest Bandwidth Density
Accelerates Insights, Enables Better Decisions
PFS on DSSDBrings innovation to Market Faster
FILE SYSTEM
Up to 48 clients
20
Use case: Life Sciences
• Application: DNA Sequencing
• Workload: Second Phase in Bioinformatics Analysis where the data is transferred to Genomic analysis software for analysis, mapping and pairing. Variant discovery and genotyping happens in this stage. Working data set is usually in ~10s of TBs (ex: 74 Files of totaling 140GB)
• Challenge: Analysis takes weeks or days to complete
Background
• With DSSD Parallel File System (PFS) solutions, Bioinformatics analytics software solutions leverage a shared-disk high performance and low latency file management solution that provides fast, reliable access to Next Generation Sequencing (NGS) data for optimizing performance
• Analysis that used to take weeks or days it can be done in hours and many more sequences can be run in parallel
• Faster variant discovery & creating personalized medicine.
Business Challenges Customer Plans to Solve with DSSD
21
Use case: Industrial Manufacturing
• Application: Nastran & Abaqus – Finite element analysis (FEA) simulations
• Workload: Simulation currently running on 10 servers * 100 jobs per Node (1000 copies at once) - coping 2GB files (2TB total)
• Challenge: Performance problems - Current customer infrastructure takes 4 hours to write 2TB
Background
• Using DSSD PFS solution customer can decrease writing wait time from 4 hours to 270 seconds
• DSSD PFS is 53x better compared to their current architecture performance
• Faster analysis and faster time to market
Business Challenges Customer Plans to Solve with DSSD
22
Use case: Financial Industry
• Application: Potential Future Exposure (PFE) Simulation, Expected Positive Exposure (EPE) Simulation, and Value At Risk (VAR) Simulation
• Workload: Long Running Parallel Calculations/Simulations. Datasets are 10TBs, low latency and performance is highly important
• Challenge: Need faster analysis and parallel calculations to predict the outcome of an asset
Background
• DSSD PFS solution provides a scalable, low-latency, high performance and reliable file system to support these simulations. Also, it will accelerate parallel calculations such as PFE, EPE, and VAR on very large data sets.
• Larger data set can be analyzed in the same period
• Predict the outcome of an asset faster
Business Challenges Customer Plans to Solve with DSSD
23
Background
Use case: Oil & Gas
• DSSD PFS solution provides a scalable, low-latency, high performance and reliable file system to support these simulations.
• This architecture speeds up many HPC simulations including significantly reducing the time it take to run parallel simulation. It also supports high levels on concurrent reads and writes with extreme performance and low latency.
• Faster & more accurate identification of reservoir since larger working data set can be analyzed quicker
Business Challenges Customer Plans to Solve with DSSD
• Application: Seismic Processing & Hydrocarbon Reserve Modeling
• Workload: Customers with smaller scale processing and velocity models (e.g. Hydrocarbon subsurface reservoir modeling and simulation). These models and simulations are highly complex and IOPS intensive. These smaller jobs consist of high levels of concurrent read and write IO (Data set size: 1 TBs to tens of TBs)
• Challenge: Compromised working data set (smaller set) and analysis take a long time
24
Use case: High Tech Manufacturing
• Application: Electronic Design Automation (EDA)
• Workload: Analysis and verification is the most time consuming process in Electrical Design Automation flow. Diverse set of software algorithms and complex simulations run for weeks at a time.
• Challenge: Slow and complex NAS architecture. The slowness of network NAS translates to longer simulation runs, longer time to manufacturing, and eventually longer time to market.
Background
• SSD PFS solution provides up to 4-6x more performance than current NAS solutions in this market. DSSD doesn’t require complex network architectures and is not bound by network latencies and performance loss
• With DSSD, EDA customer can finish analysis much faster, run more analysis in the same time, and have a much smaller storage footprint
• Customer benefits include faster time to market and higher quality product with better resource utilization
Business Challenges Customer Plans to Solve with DSSD
Welcome to WranglerUsing DSSD Storage to Accelerate Discovery
26
Ranger: 62,976 Processor Cores,
123TB RAM, 579 TeraFlops in 2008
Fastest Open Science Cluster 2008
2 of the 6 Magnum Core Switches
Stampede: #10 in the world today,
Originally #6 in 2013
~500,000 cores 6400 nodes : >10 Petaflops.
First Xeon Phi system in production
27
28
TIME FOR HUMAN DNA SEQUENCING DECREDED FROM YEARS IN 2000 TO 3 HOURS TODAY (AND
WITHIN A OFFICE VISIT IN THE NEAR FUTURE)
29
For Data Moore’s Law is Not Enough
• Growth in Data production all around
exceeds the pace of expansion of in core
memory
• Not just for genome data
– All sensor derived data is exploding
– Internet of Things…
• Not limited by the CPU
30
DOZENS OF NEW DISRUPTIVE NEW MEASUREMENT DEVICES ARE EMERGING
THAT MAY BE CONNECTED
31
HOW DO WE TACKLE THE PROBLEM FOR DATA
KNOWING WHAT WE DID FOR SIMULATIONS
32
Wrangler: HPC meets Big Data
DSSD D5 bridging performance gap between
data analytics and storage
600 TB Usable DSSD D5 Storage
Software Defined Flash Storage supporting
both HPC and Big Data environments
– 240 TB GPFS file system
– On Demand HDFS for projects
– High Transaction Rate Databases
33
Why DSSD D5
Software Defined Storage = Flexible
Environment
– Supporting GPFS, HDFS, Individual Node
Block Store, Object Store
– Support each workflow’s needs on each node
in real time
Unparalleled Throughput & IOPS across
orders of magnitude changes in block size
34
Genomics/Protein GroupingHoffmann Lab (UT Austin)
Comparing DNA sequences to find
common sources of biological
characteristic – MySQL DB stored on GPFS
distributed over 4 D5s
– Wrangler did in 4 hours what our
other systems could not complete in
multiple days
35
Electrical EngineeringAli Yilmaz/Guneet Kanur, UT Austin
Electromagnetic Backscatter Simulation (FFT
computations)
Stampede limited by I/O not core count
Wrangler GPFS accelerates application 5X (40
Nodes vs 256 on Stampede)
Smaller workflow done in RAM only 2x faster
than out or core on DSSD D5
– Same scale would take ALL of Stampede
36
Exceeding Expectations
Chris Mattmann (JPL) exploring data
capture for The Square Kilometer Array
– To be deployed in the 2020s
– Data Capture Rate of 10 GB/s needed by
2023 using HDFS and Apache Kafka
– Wrangler data capture rate with early DSSD software within a factor of 2 of goal
(6.4 GB/s)
› Limited by Java I/O bottlenecks not D5
“it was amazing to be operating in what felt like a limitless environment”
Dr. Jane Wyngaard (NASA JPL)