View
36
Download
0
Category
Tags:
Preview:
DESCRIPTION
Brief Report on Tests Related to the e-VLBI Project. Richard Hughes-Jones The University of Manchester. DataGrid WP7 – Dante Tests on the G ÉANT Core End-2-End Measurements from the 4 th Year VLBI Project at Manchester DiskPack-2-Memory Throughput and PCI Activity in a Mark5 PC - PowerPoint PPT Presentation
Citation preview
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Brief Report on Tests Related to the e-VLBI Project
Richard Hughes-JonesThe University of Manchester
DataGrid WP7 – Dante Tests on the GÉANT CoreEnd-2-End Measurements from the 4th Year VLBI Project at ManchesterDiskPack-2-Memory Throughput and PCI Activity in a Mark5 PCUpdate on 1 and 10 Gigabit Ethernet NICs in the PC
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
DataGrid WP7 – Dante Tests on the GÉANT Core Set-up
Supermicro PC in: London GEANT PoP Amsterdam GEANT PoP
Smartbits in: London GEANT PoP Frankfurt GEANT PoP
Long link UK-SE-DE2-IT-CH-FR-BE-NL
Short Link UK-FR-BE-NL
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Tests GÉANT Core: UDP throughput UDP Throughput London-Amsterdam Available BW to packet on wire Then 1/t Wire rate 998 Mbit/s
for packets > 1400 bytes
Packet Loss None for large packets
Dips in BW lined to packet loss SysKonnect NIC int. per packet CPU load important
uk-nl_20tg4-hs-w100_01Oct03
0
100
200
300
400
500
600
700
800
900
1000
0 5 10 15 20 25 30 35 40Spacing between frames us
Rec
v W
ire r
ate
Mbi
ts/s
50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes
0102030405060708090
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% P
acke
t los
s
50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Tests GÉANT Core: Packet re-ordering Effect of Packet size London-Amsterdam Packets at 10 µs – line speed 10,000 sent Packet Loss ~ 0.1%
Re-order Distribution
Packet re-order uk-nl 10,000 BE sent wait 10 us 01 Oct 03
0
5
10
15
20
25
30
0 500 1000 1500Packet size bytes
Out
of o
rder
% 0
10
20
30
1400 1401 1402 1403 1404Packet size bytes
Ou
t o
f o
rde
r %
Packet re-order uk-nl 10,000 sent wait 10 us
0
100
200
300
400
500
0 1 2 3 4 5 6 7 8 9Length out-of-order
No.
Pac
kets 1400 bytes
1401 bytes
1402 bytes
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Tests GÉANT Core: Packet re-ordering Effect of LBE background Amsterdam-London BE Test flow Packets at 10 µs – line speed 10,000 sent Packet Loss ~ 0.1%
Re-order Distributions:
UDP 1472 bytes NL-UK-lbexxx_7nov03
02468
101214161820
2 2.2 2.4 2.6 2.8 3 3.2Total Offered Rate Gbit/s
% O
ut o
f ord
er
hstcpStandard TCP line speed90% line speed
Packet re-order 1472 bytes uk-nl 21 Oct 03 10,000 sent wait 10 us
020000400006000080000
100000120000140000160000180000200000
1 2 3 4 5 6 7 8 9Length out-of-order
No.
Pac
kets
0 % lbe
10 % lbe
20 % lbe
30 % lbe
40 % lbe
50 % lbe
60 % lbe
70 % lbe
80 % lbe
Packet re-order 1400 bytes uk-nl 21 Oct 03 10,000 sent wait 10 us
0500
100015002000250030003500400045005000
1 2 3 4 5 6 7 8 9Length out-of-order
No.
Pac
kets
0 % lbe
10 % lbe
20 % lbe
30 % lbe
40 % lbe
50 % lbe
60 % lbe
70 % lbe
80 % lbe
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
VLBI Project: Test Topology
SuperJANET4
Jodrell
Manchester
SURFnet
JIVEDwingaloo
Adam MathewsSteve O’TooleUniv of Manchester
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Gnt5-DwMk5 11Nov03-1472 bytes
0
2
4
6
8
10
12
0 5 10 15 20 25 30 35 40Spacing between frames us
% P
acket
loss
Gnt5-DwMk5
DwMk5-Gnt5
Manchester to Dwingeloo 2.0G Hz Xeon 1.2 GHz PIII
Re-ordering vs Offered Load
VLBI Project: Throughput
Gnt5-DwMk5 11Nov03/DwMk5-Gnt5 13Nov03-1472bytes
0
200
400
600
800
1000
1200
0 5 10 15 20 25 30 35 40Spacing between frames us
Recv W
ire r
ate
Mbits/s
Gnt5-DwMk5
DwMk5-Gnt5
Gnt5-DwMk5 11Nov03 1472 bytes
020406080
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% K
erne
l S
ende
r
Gnt5-DwMk5 11Nov03 1472 bytes
020406080
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% K
erne
l R
ecei
ver
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
1472 byte Packets man -> JIVE FWHM 22 µs (B2B 3 µs )
VLBI Project: Jitter & 1-way Delay
1472 bytes w=50 jitter Gnt5-DwMk5 28Oct03
0
2000
4000
6000
8000
10000
0 20 40 60 80 100 120 140
Jitter us
N(t
)
1472 bytes w=50 jitter Gnt5-DwMk5 28Oct03
1
10
100
1000
10000
0 20 40 60 80 100 120 140
Jitter us
N(t
)
1472 bytes w12 Gnt5-DwMk5 21Oct03
0
2000
4000
6000
8000
10000
12000
2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000Packet No.
1-w
ay
de
lay
us
1472 bytes w12 Gnt5-DwMk5 21Oct03
0
2000
4000
6000
8000
10000
12000
0 1000 2000 3000 4000 5000Packet No.
1-w
ay d
elay
us
1-way Delay – note the packet loss (points with 0 –way delay)
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Measure the time between lost packets in the time series of packets sent.
Lost 1410 in 0.6s Is it a Poisson process? Assume Poisson is
stationary λ(t) = λ
Use Prob. Density Function:
P(t) = λ e-λt
Mean λ = 2360 / s[426 µs]
Plot log: slope -0.0028expect -0.0024
Could be additional process involved
VLBI Project: Packet Loss Distributionpacket loss distribution 12b bin=12us
0
10
20
30
40
50
60
70
80
12 72 132
192
252
312
372
432
492
552
612
672
732
792
852
912
972
Time between lost frames (us)
Num
ber
in B
in
Measured
Poisson
packet loss distribution 12b
y = 41.832e-0.0028x
y = 39.762e-0.0024x
1
10
100
0 500 1000 1500 2000
Time between frames (us)
Num
ber
in B
in
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Divide time series of packets into 1000 slices of 50 packets
Total lost packets 1410 Average number / slice = 1.4
Calc Poisson Probability P(n, µ) = µ n e -µ
n!
Curves close but not exact Could be more than 1 process
VLBI Project: Packet Loss – Is it Poisson?
0
50
100
150
200
250
300
350
400
0 5 10 15n num lost in sub-sample
N(n
)
run12b
1
1.3
1.4
1.8
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Aggregated Variance Method Divide time series length N into
blocks of size m Calc mean of each section Xm(k)
k= 1 … N/m Calc variance VXm of these Xm(k) Vary m size of the blocks
Plot on log-log & fit slope β Hurst parameter H
β = 2H -2 Measure:
β = -0.355 which gives H 0.822 H =1 no long range dependence
VLBI Project: Packet Loss – Long Range Effects?
y = -0.355x + 2.8826
0
0.5
1
1.5
2
2.5
3
3.5
4
0 0.5 1 1.5 2 2.5 3sub-sample size Log10( m )
Ag
gri
ga
te-v
ari
an
ce L
og
10
( X
(m)
)
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Traffic Flows Manchester – NetNorthWest - SuperJANET Access links
Two 1 Gbit/s
Access links:SJ4 to GÉANT GÉANT to SurfNet
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Throughput & PCI transactions on the Mark5 PC:
Read /Write
n bytes
Wait timetime
Mark5 uses Supermicro P3TDLE 1.2 GHz PIII Mem bus 133/100 MHz 2 *64bit 66 MHz PCI 4 32bit 33 MHz PCI
SuperStor
NIC
Input Card
IDEDiscPack
Ethernet
Logic AnalyserDisplay
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
PCI Activity: Read 1 data block 0 wait time
Data block 131,072 bytes Read consists of:
Setup CSRs Data movement Update CSRs
Data block contains PCI bursts 4096 bytes long
For 0 wait between reads: Data block 608µs long Then 655µs gap
PCI transfer rate 830 Mbit/s Read_sstor rate 778 Mbit/s
(97 Mbyte/s)
CSR Access
PCI Burst
Data Block
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
PCI Activity: Read Multiple data blocks 0 wait Read 999424 bytes Each Data block:
Setup CSRs Data movement Update CSRs
For 0 wait between reads: Data blocks ~600µs long
take ~6 ms Then 744µs gap
PCI transfer rate 1188Mbit/s(148.5 Mbytes/s)
Read_sstor rate 778 Mbit/s (97 Mbyte/s)
PCI bus occupancy: 68.44% Concern about Ethernet Traffic
64 bit 33 MHz PCI needs ~ 82% for 930 Mbit/s Expect ~360 Mbit/s
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
PCI Activity: Read Throughput
Flat then 1/t dependance ~ 860 Mbit/s for Read blocks
>= 262144 bytes
CPU load ~20% Concern about CPU load
needed to drive Gigabit link
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
10 GigEthernet: Throughput 1500 byte MTU gives ~ 2 Gbit/s Used 16144 byte MTU max user length 16080 DataTAG Supermicro PCs Dual 2.2 GHz Xenon CPU FSB 400 MHz PCI-X mmrbc 512 bytes wire rate throughput of 2.9 Gbit/s
CERN OpenLab HP Itanium PCs Dual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz PCI-X mmrbc 512 bytes wire rate of 5.7 Gbit/s
SLAC Dell PCs giving a Dual 3.0 GHz Xenon CPU FSB 533 MHz PCI-X mmrbc 4096 bytes wire rate of 5.4 Gbit/s
an-al 10GE Xsum 512kbuf MTU16114 27Oct03
0
1000
2000
3000
4000
5000
6000
0 5 10 15 20 25 30 35 40Spacing between frames us
Rec
v W
ire
rate
Mb
its/
s
16080 bytes 14000 bytes 12000 bytes 10000 bytes 9000 bytes 8000 bytes 7000 bytes 6000 bytes 5000 bytes 4000 bytes 3000 bytes 2000 bytes 1472 bytes
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
10 GigEthernet: PCI-X Activity
Supermicro P4DP8-2G motherboard
Packet transmissionMemory to NIC
Packet receptionNIC to Memory
CSR Access
Interrupt
PCI-X segment 512 bytes
Transfer of 16114 bytes
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
10 GigEthernet: Tuning PCI-X
mmrbc 1024 bytes
mmrbc 2048 bytes
mmrbc 4096 bytes
Transfer of 16114 bytes
mmrbc 512 bytes
PCI-X segment 2048 bytes
CSR Access
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
10 GigEthernet at SC2003 BW Challenge (Phoenix) Three Server systems with 10 GigEthernet NICs Used the DataTAG altAIMD stack 9000 byte MTU Streams From SLAC/FNAL booth in Phoenix to:
Pal Alto PAIX
Chicago Starlight Amsterdam SARA
10 Gbits/s throughput from SC2003 to PAIX
0
1
2
3
4
5
6
7
8
9
10
11/19/0315:59
11/19/0316:13
11/19/0316:27
11/19/0316:42
11/19/0316:56
11/19/0317:11
11/19/0317:25 Date & Time
Throughput
Gbits/s
Router to LA/PAIXPhoenix-PAIX HS-TCPPhoenix-PAIX Scalable-TCPPhoenix-PAIX Scalable-TCP #2
10 Gbits/s throughput from SC2003 to Chicago & Amsterdam
0
1
2
3
4
5
6
7
8
9
10
11/19/0315:59
11/19/0316:13
11/19/0316:27
11/19/0316:42
11/19/0316:56
11/19/0317:11
11/19/0317:25 Date & Time
Throughput
Gbits/s
Router traffic to Abilele
Phoenix-Chicago
Phoenix-Amsterdam
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Traffic QoS Classes on GÉANT Backbone
Normal Traffic
Normal Traffic +
Less Than Best Effort 2.0 Gbit/s
Normal Traffic +
Radio Astronomy Data 500 Mbit/s
Normal Traffic +
Radio Astronomy Data +
Less Than Best Effort 2.0 Gbit/s
Max Throughput on 2.5 G PoS
JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester
Some Measurements made during ER2002
No LBE
0
2
4
6
8
10
12
14
16
18
20
0 20 40 60 80 100 120 140 160 180 200Transfer number
No.
Out
of
ord
er
0
5000
10000
15000
20000
25000
No
. L
ost
num_badorder
num_lost
With 1.8Gbit LBE
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
0 20 40 60 80 100 120 140 160Transfer number
No.
Out
of
ord
er
0
5000
10000
15000
20000
25000
No
. L
ost
num_badorder
num_lost
Recommended