Embed Size (px)
Citation preview
Xiaoyi Lu, Md. Wasi-‐ur-‐Rahman, Nusrat Islam, and Dhabaleswar K. (DK) Panda
Network-‐Based Compu2ng Laboratory
Department of Computer Science and Engineering The Ohio State University, Columbus, OH, USA
A Micro-benchmark Suite for Evaluating Hadoop RPC on High-Performance
Networks
WBDB 2013
Outline
• IntroducAon and MoAvaAon
• Problem Statement
• Design ConsideraAons • Micro-‐benchmark Suite
• Performance EvaluaAon
• Conclusion & Future work
2
WBDB 2013
Big Data Technology • Apache Hadoop is one of the most popular
Big Data technology – Provides framework for large-scale,
distributed data storage and processing • An open-source implementation of
MapReduce programming model • Hadoop Distributed File System (HDFS) is
the underlying file system of Hadoop MapReduce and Hadoop DataBase, HBase
• Hadoop Core – Common functionalities, e.g. Remote Procedure Call (RPC)
HDFS
MapReduce HBase
Hadoop Framework
3
Core (RPC, ..)
WBDB 2013
Adoption of Hadoop RPC • Hadoop RPC is increasingly being used with data-center
middlewares such as MapReduce, HDFS, and HBase because of its simplicity, productivity, and high performance. – Metadata exchange – Manage compute nodes and track system status – Efficient data management operations: get block info, create blocks etc. – Database operations: put, get, etc.
4
High Performance
Networks
(HDD/SSD)
(HDD/SSD)
(HDD/SSD)
...
...
(HDFS Data Nodes)(HDFS Clients)
...
...
(HBase Clients) (HRegion Servers) (Data Nodes)
(HDD/SSD)
(HDD/SSD)
(HDD/SSD)
...
... ...
... ...
...
High Performance
Networks
High Performance
Networks
MapReduce & HDFS HBase
Map/Reduce (HDFS Name Node)
WBDB 2013
Common Protocols using Open Fabrics
5
Applica-on
Verbs Sockets
ApplicaAon Interface
SDP
RDMA
SDP
InfiniBand Adapter
InfiniBand Switch
RDMA
IB Verbs
InfiniBand Adapter
InfiniBand Switch
User space
RDMA
RoCE
RoCE Adapter
User space
Ethernet Switch
TCP/IP
Ethernet Driver
Kernel Space
Protocol
InfiniBand Adapter
InfiniBand Switch
IPoIB
Ethernet Adapter
Ethernet Switch
Adapter
Switch
1/10/40 GigE
iWARP
Ethernet Switch
iWARP
iWARP Adapter
User space IPoIB
TCP/IP
Ethernet Adapter
Ethernet Switch
10/40 GigE-‐TOE
Hardware Offload
RSockets
InfiniBand Adapter
InfiniBand Switch
User space
RSockets
WBDB 2013
Can Big Data Processing Systems be Designed with High-‐Performance Networks and Protocols?
Enhanced Designs
Applica-on
Accelerated Sockets
10 GigE or InfiniBand
Verbs / Hardware Offload
Current Design
Applica-on
Sockets
1/10 GigE Network
• Sockets not designed for high-‐performance – Stream semanAcs oSen mismatch for upper layers (Memcached, HBase, Hadoop) – Zero-‐copy not available for non-‐blocking sockets
Our Approach
Applica-on
OSU Design
10 GigE or InfiniBand
Verbs Interface
6
WBDB 2013
Hadoop RPC over InfiniBand
Hadoop RPC
IB Verbs
InfiniBand
Applica-ons
1/10 GigE, IPoIB Network
Java Socket Interface
Java Na-ve Interface (JNI)
Our Design
Default
OSU Design
Enables high performance RDMA communicaAon, while supporAng tradiAonal socket interface
Xiaoyi Lu, Nusrat Islam, Md. Wasi-‐ur-‐Rahman, Jithin Jose, Hari Subramoni, Hao Wang, Dhabaleswar K. (DK) Panda. “High-‐Performance Design of Hadoop RPC with RDMA over InfiniBand.” To be presented in the 42nd Interna-onal Conference on Parallel Processing (ICPP 2013), Lyon, France, October, 2013. 7
rpc.ib.enabled
WBDB 2013
8
Hadoop RPC over IB: Gain in Latency and Throughput
• Hadoop RPC over IB PingPong Latency
– 1 byte: 39 us; 4 KB: 52 us – 42%-‐49% and 46%-‐50% improvements compared with the performance of default
Hadoop RPC on 10 GigE and IPoIB (32Gbps) respec-vely
• Hadoop RPC over IB Throughput
– 512 bytes & 48 clients: 135.22 Kops/sec – 82% and 64% improvements compared with the peak performance of default
Hadoop RPC on 10 GigE and IPoIB (32Gbps) respec-vely
0
20
40
60
80
100
120
1 2 4 8 16 32 64 128 256 512 1024 2048 4096
Latency (us)
Payload Size (Byte)
RPC-‐10GigE RPC-‐IPoIB(32Gbps) RPCoIB(32Gbps)
0
20
40
60
80
100
120
140
160
8 16 24 32 40 48 56 64
Throughp
ut (K
ops/Sec)
Number of Clients
RPC-‐10GigE
RPC-‐IPoIB(32Gbps)
RPCoIB(32Gbps)
WBDB 2013
• High-Performance Design of Hadoop over RDMA-enabled Interconnects
– High performance design with native InfiniBand support at the verbs-level for HDFS, MapReduce, and RPC components
– Easily configurable for both native InfiniBand and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)
– Current release: 0.9.0 • Based on Apache Hadoop 0.20.2
• Compliant with Apache Hadoop 0.20.2 APIs and applications
• Tested with
– Mellanox InfiniBand adapters (DDR, QDR and FDR)
– Various multi-core platforms
– Different file systems with disks and SSDs
– http://hadoop-rdma.cse.ohio-state.edu
Available in Hadoop-RDMA SoSware
9
WBDB 2013
Requirements of Hadoop RPC Benchmarks • To achieve optimal performance, Hadoop RPC needs
to be tuned based on cluster and workload characteristics
• A micro-benchmark tool suite to evaluate Hadoop RPC performance metrics in different configurations is important for tuning and understanding
• For Hadoop developers, this kind of micro-benchmark suite is helpful to evaluate and optimize the performance of new designs
10
WBDB 2013
Outline
• IntroducAon and MoAvaAon
• Problem Statement
• Design ConsideraAons • Micro-‐benchmark Suite
• Performance EvaluaAon
• Conclusion & Future work
11
WBDB 2013
Problem Statement • Can we design and implement a simple and standardized
benchmark suite to let all users and developers in the Big Data community evaluate, understand, and optimize the Hadoop RPC performance over a range of networks/protocols?
• What will be the performance of Hadoop RPC when evaluated
using this benchmark suite on high-performance networks?
12
WBDB 2013
Outline
• IntroducAon and MoAvaAon
• Problem Statement
• Design ConsideraAons • Micro-‐benchmark Suite
• Performance EvaluaAon
• Conclusion & Future work
13
WBDB 2013
Design Considerations
• The performance of RPC systems is usually measured by the metrics of latency and throughput
• Performance of Hadoop RPC is determined by: – Factors related to network configurations; Faster
interconnects and/or protocols can enhance Hadoop RPC performance
– Controllable parameters in RPC engine-level and benchmark-level: handler/client number, etc.
– Data types: serialization and deserialization issues of different data types in RPC system; BytesWritable, Text, etc.
– CPU Utilization: tradeoff between RPC subsystem performance and the whole system performance
14
WBDB 2013
Outline
• IntroducAon and MoAvaAon
• Problem Statement
• Design ConsideraAons • Micro-‐benchmark Suite
• Performance EvaluaAon
• Conclusion & Future work
15
WBDB 2013
Micro-benchmark Suite • Two different micro-benchmarks:
– Latency: Single Server, Single Client – Throughput: Single Server, Multiple Clients
• A script framework for job launching and resource monitoring
• Calculates statistics like Min, Max, Average
16
Component Network Address
Port Data Type
Min Msg Size
Max Msg Size
No. of Iterations
Handlers Verbose
lat_client √ √ √ √ √ √ √
lat_server √ √ √ √
Component Network Address
Port Data Type
Min Msg Size
Max Msg Size
No. of Iterations
No. of Clients
Handlers Verbose
thr_client √ √ √ √ √ √ √
thr_server √ √ √ √ √ √
WBDB 2013
Outline
• IntroducAon and MoAvaAon
• Problem Statement
• Design ConsideraAons • Micro-‐benchmark Suite
• Performance EvaluaAon
• Conclusion & Future work
17
WBDB 2013
Experimental Setup • Hardware
– Intel Westmere Cluster • 8 nodes • Each node has 8 processor cores on 2 Intel Xeon 2.67 GHz Quad-‐core CPUs, 24 GB main memory
• Network: 1GigE, 10GigE, and IPoIB (32Gbps) • SoSware
– Enterprise Linux Server release 6.1 (Santiago) at kernel version 2.6.32-131 with OpenFabrics version 1.5.3
– Hadoop 0.20.2 and Sun Java SDK 1.7. 18
WBDB 2013
RPC Latency for BytesWritable
Small Messages Large Messages
• Latency for RPC decreases if the underlying interconnect is changed to IPoIB or 10 GigE from 1 GigE.
• With 10 GigE interconnect, we observe beher latency than IPoIB for small payload sizes. For large payload sizes, IPoIB performs beher than 10 GigE. – IPoIB achieves 27% gain over 10 GigE for a 64 MB payload size, whereas it performs worse by
0.66% over 10 GigE for a 4 KB payload size. 19
0
50
100
150
200
250
Late
ncy
(us)
Payload Size (Byte)
1GigE
10GigE
IPoIB(32Gbps)
0
100
200
300
400
500
600
700
800
128K 256K 512K 1M 2M 4M 8M 16M 32M 64M
Late
ncy
(ms)
Payload Size (Byte)
1GigE
10GigE
IPoIB(32Gbps)
WBDB 2013
RPC Latency for Text
Small Messages Large Messages
• Similar performance characterisAc for RPC latency with the data type of Text.
20
0
20
40
60
80
100
120
140
160
180
200
1 2 4 8 16 32 64 128 256 512 1024 2048 4096
Late
ncy
(us)
Payload Size (Byte)
1GigE
10GigE
IPoIB(32Gbps)
0
100
200
300
400
500
600
700
800
128K 256K 512K 1M 2M 4M 8M 16M 32M 64M
Late
ncy
(us)
Payload Size (Byte)
1GigE
10GigE
IPoIB(32Gbps)
WBDB 2013
RPC Throughput for BytesWritable
7 RPC Server Handlers 16 RPC Server Handlers
• IPoIB performs beher than 10 GigE as payload size is increased.
• At 4 KB, the improvement goes upto 26% for seven handler threads. For small payload sizes, 10 GigE performs beher than IPoIB by an average margin of 5-‐6%.
21
0
5
10
15
20
25
30
35
40
45
1 2 4 8 16 32 64 128 256 512 1024 2048 4096
Thro
ughp
ut (K
ops/
Sec)
Payload Size (byte)
1GigE
10GigE
IPoIB(32Gbps) 0
5
10
15
20
25
30
35
40
45
1 2 4 8 16 32 64 128 256 512 1024 2048 4096
Thro
ughp
ut (K
ops/
Sec)
Payload Size (byte)
1GigE
10GigE
IPoIB(32Gbps)
WBDB 2013
RPC Throughput for BytesWritable
CPU utilization for the experiment with 4 handlers Throughput Comparison for 4 KB payload size
• Keep the payload size fixed to 4 KB and observe the trend with different handler numbers and different networks – IPoIB performs beher than 10 GigE as 48%, 5%, 45%, and 47% for 1, 4, 16, and 32 handlers
respecAvely.
• Easily used to monitor resource uAlizaAon. Enable a parameter in the script framework. 22
0 10 20 30 40 50 60 70 80 90
100
1 4 16 32
Thro
ughp
ut (K
ops/
Sec)
Handler Number
1GigE 10GigE IPoIB(32Gbps)
0
5
10
15
20
25
30
35
40
45
0 9 18
27
36
45
54
63
72
81
90
99
108
117
126
135
144
153
162
171
180
189
198
207
216
CPU
Util
izat
ion
(%)
Sampling Point
1GigE 10GigE IPoIB(32Gbps)
WBDB 2013
Outline
• IntroducAon and MoAvaAon
• Problem Statement
• Design ConsideraAons • Micro-‐benchmark Suite
• Performance EvaluaAon
• Conclusion & Future work
23
WBDB 2013
Conclusion and Future Works
• Design and implement a micro-benchmark suite to evaluate the performance of standalone Hadoop RPC.
• Provide standard micro-benchmarks to measure the latency and throughput of Hadoop RPC with different data types.
• Illustrate the performance results of Hadoop RPC using our benchmarks over different networks/protocols (1GigE/10GigE/IPoIB).
• Will extend our benchmark suite to help users to make the performance comparisons among Hadoop Writable RPC, Avro, Thrift, and Protocol buffers
• Will be made available to the big data community via an open-source release 24
WBDB 2013
Thank You!
{luxi, rahmanmd, islamn, panda}@cse.ohio-‐state.edu
Network-‐Based CompuAng Laboratory hhp://nowlab.cse.ohio-‐state.edu/
MVAPICH Web Page hhp://mvapich.cse.ohio-‐state.edu/
25 Hadoop-‐RDMA Web Page
hhp://hadoop-‐rdma.cse.ohio-‐state.edu/
