Embed Size (px)
Citation preview
Ankit SinglaData center architecture
2
A server rack
A rack of servers
A top-of-rack switch
3
Lots of racks
How to network the racks?
”
“Big switch” approach
4
…
“Big switch” approach
5
…
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
512 ports x 1G
40 servers x 1G
4 ports x 1G
610:1 “oversubscription”
“Big switch” approach
7
$$$$$$$$$$ $$$$$$$$$$
…
$$$ $$$
Alternative: tree network
8
9
Connect many cheap, identical switches?
Goals: high capacity, low latency
If you know your application …
10
… design for it
11
NERSC
But, other apps may not work well …
12
13
What’s so hard about this?6-port switches
4-port switches
�
���
�����
�����
�����
������
������
� � � �� �� �� �� �� ��
������������������������
������ �� ��������
14
What’s so hard about this?
15
Clos networks
Use small, cheap elements to build large networks!!
Fat-tree
17
Fat-tree network
Pod 0
10.0.2.1
10.0.1.1
Pod 1 Pod 3Pod 210.2.0.2 10.2.0.3
10.2.0.1
10.4.1.1 10.4.1.2 10.4.2.1 10.4.2.2
10.2.2.1
10.0.1.2
ACM SIGCOMM, 2008
A Scalable, Commodity Data Center Network Architecture
Mohammad Al-Fares Alexander Loukissas Amin Vahdat
Servers
Core
18
Fat-tree network
19
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
[Image: Robert Harker]
[Image: Robert Harker]
22
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
23
Cabling[Robert Harker
24
DC architecture
How to network the racks?
But can we do better?
25
Other topologies besides fat-trees?
6-port switches
4-port switches
�
���
�����
����������������������
� � � �� �� �� �� �� ��������������������������
������ �� ��������
Surprise: picking randomly from this space works better than fat-trees!
Deterministic expander constructions can also achieve these gains
But do we need full bandwidth everywhere at all times?
25% is the high mark for overall utilization (?) ”
2012.
27
hotspots
Skew in network traffic
29
How does this observation help?
“Unfortunately, given current data center network architectures, the only way to provision required bandwidth bandwidth between dynamically changing sets of nodes is to build a non-blocking switch fabric at the scale of an entire data center”
.
Set up network connections on the fly?
Servers
ToR
switch
Aggregate
switch
Core
switch
Electrical
Network
Optical
Network
Reconfigurable
optical paths
Array of Micromirrors
Diffracted beam Towards destinationReceived beam
Input beam LasersDMDs
Photodetectors
Mirror assembly Reflected beam
TX
RX
Rack%1%Rack%N%Rack%r%
Steerable%%FSOs%
Ceiling%mirror%
ToR%switch%
FireFly%Controller%
Traffic%Pa=erns%
Rule%change%
FSO%reconf%
Can statically wired networks compete with this?
31
DC architecture @ SIGCOMM 2017RotorNet: A Scalable, Low-complexity,
Optical Datacenter Network
William M. Mellette, Rob McGuinness, Arjun Roy, Alex Forencich,
George Papen, Alex C.Snoeren, and George Porter
University of California, San Diego
A
T
a
l
e
o
f
T
w
o
T
o
p
o
l
o
g
i
e
s
:
E
x
p
l
o
r
i
n
g
C
o
n
v
e
r
t
i
b
l
e
D
a
t
a
C
e
n
t
e
r
N
e
t
w
o
r
k
A
r
c
h
i
t
e
c
t
u
r
e
s
w
i
t
h
F
l
a
t
-
t
r
e
e
Yiting Xia, Xiaoye Steven Sun, Simbarashe Dzinamarira,
Dingming Wu, Xin Sunny Huang, T. S. Eugene Ng
Rice University
Beyond fat-trees without antennae, mirrors, and disco-balls
Simon KassingETH Zürich
Asaf ValadarskyHebrew University of Jerusalem
Gal ShahafHebrew University of Jerusalem
Michael SchapiraHebrew University of Jerusalem
Ankit SinglaETH Zürich
