6/25/2003

1

e-VLBI Development at Haystack Observatory

Dan Smythe

MIT Haystack Observatory

30 June 2003

6/25/2003 2

e-VLBI Development at Haystack Observatory

• Phase 1: Develop eVLBI-compatible data system– Mark 5 system

• Phase 2: Demonstrate 1 Gbps e-VLBI using Bossnet(w/ DARPA and NASA support)

– ~700km link between Haystack Observatory and NASA/GSFC

– First e-VLBI experiment achieved ~788Mbps transfer rate

• Phase 3: Develop adaptive network protocol(newly awarded NSF grant to Haystack Observatory; collaboration with MIT Lab for Computer Science and MIT Lincoln Laboratory);

– New IP-based protocol tailored to operate in shared-network ‘background’ to efficiently using available bandwidth

– Demonstrate on national and international networks

• Phase 4: Extend e-VLBI to national and global VLBI community– ‘Last mile’ problem remains a serious challenge

6/25/2003 3

Mark 5 e-VLBI Connectivity

• Mark 5 supports a triangle of connectivity for e-VLBI requirements

Data Port

Disk array

PCI bus/Network (e-VLBI)(64bit/66MHz)

Mark 5 can support several possible e-VLBI modes:• e-VLBI data buffer (first to Disc Array, then to Network); vice versa• Direct e-VLBI (Data Port directly to Network); vice versa• Dual Gigabit Ethernet connections will be necessary to support for

1024 Mbps; new motherboards are being examined

More information at www.haystack.edu6/25/2003 4

e-VLBI• 1 Gbpse-VLBI demonstration experiment between Westford

and GGAO; data transfer to Haystack correlator in near-real-time (www.haystack.edu/e-vlbi)

• Several Westford/Kashima e-VLBI experiments

• Establishing e-VLBI connections to KokeePark and U. of Regensburg (near Wettzell) for near-real-time Intensive UT1 demonstrations

• e-VLBI is very high visibility in networking community

• Much is going ono Haystack has 3-yr grant from NSF for development of new protocols for e-

VLBI

o Haystack is co-PI on two major proposals in U.S.

6/25/2003 5

Haystack

Westford

NASA/GSFCUSNO

(correlator)

(correlator)

Bossnet 1 Gbps e-VLBI demonstration exper iment

Initial experiment

Future

6/25/2003 6

Performance test results – Haystack/GGAO

Average sustained rate >900 Mbps over 10 hours

6/25/2003

2

6/25/2003 7

Westford-GGAO e-VLBI resultsFirst near-real-time e-VLBI experiment conducted on 6 Oct 02

– Recorded data at 1152 Mbps on Westford-GGAO baseline– GGAO disk-to-disk transfer at average 788 Mbps transfer rate– Immediate correlation on Haystack Mark 4 correlator– Nominal fringes observed

• Direct data transfer experiment conducted on 24 Oct 02– Direct transfer of GGAO data to disk at Haystack at 256 Mbps– Immediate correlation with Westford data– Nominal fringes observed

• Next step – full real-time e-VLBI– Mark 5 system is capable of transmitting in real-time– But, still need additional work on correlator software to synchronize

correlator operation to real-time

• Conclusion– e-VLBI at near Gbps speeds over ordinary shared networks is possible

but still difficult

• Full report atftp://web.haystack.edu/pub/e-vlbi/demo_report.pdf

6/25/2003 8

International e-VLBI experiments

• Westford, MA to Kashima, Japan - experiments in Oct 02 and Mar 03– Record Mark 5 at Westford - K5 at Kashima• Files exchanged over Abilene/GEMnet networks

• Nominal speed expected to be ~20 Mbps; best achieved so far ~11 Mbps

– Software format transformation– Correlation on Mark 4 correlator at Haystack and PC Software correlator

at Kashima– Normal fringes obtained– Further experiments are scheduled; network tuning is in progress

• Kauai, Hawaii and Wettzell, Germany (in progress)– Daily experiments of ~100GB are ideal candidate for early e-VLBI– Data will be transferred to Haystack Observatory for processing

(OC-3 speeds are possible)– Network links are now being brought up

6/25/2003 9

Abiline Network Backbone – 25 June 2003

http://loadrunner.uits.iu.edu/weathermaps/abilene/

Line Utilization

6/25/2003 10

Typical bit-rate statistics on Abilene network

Conclusion: Average network usage is only a few % of capacity

Usage >20Mbps less than 1% of the time

100 500 Mbps

0.1

1.0

0.01

0.001

6/25/2003 11

Typical distribution of heavy traffic on Abilene

Conclusion: Heavy usage of network tends to occur in bursts of <2 minutes

secs200 400 1000

1.0

0.9

0.8

0.7

<10% of ‘bulk’ transfers exceed ~100 secs

6/25/2003 12

New IP Protocols for e-VLBI

• Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity

• New protocols are being developed to utilize networks in ‘background’ mode for applications such as e-VLBI

– Take advantage of special characteristics of e-VLBI data

– Will ‘scavenge’ and use ‘secondary’ bandwidth

– Will give priority to ‘normal’ users

– Requires a new ‘end-point adaptive strategy’

• Work being carried out by MIT Haystack Observatory in collaboration with MIT Laboratory for Computer Science and MIT Lincoln Laboratory

– 3-year program; will demonstrate e-VLBI connections both nationally and internationally

– Dr. David Lapsley has joined Haystack staff to lead this effort

6/25/2003

3

6/25/2003 13

But there is a significant problem – ‘Last-mile’ costs

• Most of the world’s telescopes are not well connected• Electronic and electro-optic costs are dropping rapidly

– GigE switch: 2001 - $15K 2003 - $1.2K– GigE transceivers 2001 - $750 2003 - $180– CWDM transceivers $400-800 for 50-100km reach!

• Direct fiber cost is relatively low– $60/fiber-km in 80-fiber bundle• If you can buy or lease existing fiber, there is no better time!• But – fiber installation cost is still tall pole

– Europe: >$20/m (or any populous wide-area)– U.S.: >$10/m (in simplest desert environment)

• The upside: there is developing a lot of momentum and support from the greater networking community to get the job done!

Also desperately needed:– Modern digital filter banks to replace aging and obsolete analog BBC’s!

6/25/2003 14

….and a longer term e-VLBI issue as well

• Can e-VLBI survive the long-term networking costs?

…and some hope!• There is momentum gathering in the networking community to provide

national and international ultra-high-speed networking as a critical ‘enabling infrastructure’ for U.S. and international science. The NSF ‘Atkin’s Report’ recommends spending $1B/yr for the next 5 years to improve the U.S. ‘cyberinfrastructure’.

• The astronomy community needs to makes its voice heard loud and clear!

6/25/2003 15

Conclusions

• e-VLBI is riding an unprecedented wave of global network connectivity and networking community enthusiasm.

• There is no better time to lease or buy installed fiber than now!

• Gradual transition from disks to disk/e-VLBI to all e-VLBI is likely.

• 10-100 Gbps/antenna is technically possible with e-VLBI. (Can VLBI correlators keep up?!)

• Haystack is moving aggressively to exploit these new technologies

6/25/2003 16

WS/Mark 5

Summit1i

Summit5i

Westford Antenna Millstone RPE Bldg (Lorraine)

Haystack Correlator

GigEGigE

GigE GigESummit

5i

GigE

Congress StBoston

GlowNet

Lincoln Lab (Lorraine/Steve)

A1

A2

B5

B1

D2

Cisco1500

A4

C3B2

B4

C1

B3

D3

Cisco1500

Summit5i

Summit5i

Amp

E1

E3

Bossnet(Steve/Jim) ISI-E

E2 E6

A3

Figure 1: e-VLBI Path - Haystack to ISI-E

C2

GigEB6

WDM

WDM

GigE GigE

F2

Amp Amp

GlowNetD1

E8

F1

GigE

WDM

evlbi011.drw20 Dec 01

WS/Mark 5

WS/Mark 5

1 8

14

13

16

15

28-1A

28-1B

LL

(Shared with LL onscheduled basis)

To be upgraded from2 to 3λ − ϑαν 2002

To be replaced byJuniper M20 Jan 02 with OC-48 to Bossnet

1310

TestWS

To be providedby Haystack

MITCampus

1510

Plan install by 1 Jan 02

E4

E7

E5

GigE

Other users(no control)

GigE

To be updatedto OC48 Jan02

Scheduling onBossnet calendar

Schedule to becoordinated with LL

Schedule to becoordinated with LL

Dedicated -no scheduling

Dedicated -no scheduling

Details of path from Haystack to GGAO: Part 1

6/25/2003 17

Trans-

ponder

ISI-E (Tom)

G1

Figure 2: e-VLBI Path - ISI-E to GSFC/GGAO

JuniperM40

JuniperM160

JuniperM160

Cisco3508

Switch

Summit5i

WS/Mark 5

UMCP (Jerry or Dan)

GSFC/Bldg 28 (Pat) GGAO/Bldg 202 (Pat) GGAO Antenna Trailer (Bill)

WDM

WDM

WDM GigEGigEGigEGigE

OC-48Bossnet

G3

G5

G10 H3

H7

J3

J5

K1

K2

K3

K4

K7 L1

L2

L3

L4

evlbi011.drw20 Dec 01

G2

G4 G8

H8

H9

J4

H7

OC-48

GigEGigE

GigEGigE

Max @ ISI-E (Jerry)

Borrowed OC48interface - may notbe able to keep

Switch

GigEGigEBackup plan

Preferred e-VLBI path,but may not be in place.Pat working with Jerry and Dan.

Other users ( no control)

No jumbo frames

Max @ ISI-E (Jerry)

LuxN

Summit5i

TunerGigE GigE

OC-48

Plan to update toOC-48 ~Jan 02

Summit5i

On orderMAC on orderdue Jan-Feb 02

Dell Linux currently at Haystack willalso be available here

1600 ft of fiberto be installedJan 02

Other users(no control)

1310

G9

G11

G12 H1TestWS?

G6

G7

TestWSMac with

Yellowdog Linux;need details

K5

K6

J1

GigE

Plan mid Jan 02

J2

Need details

Eventually replacedwith Mark 5 unit

Need details

H6

TestWS?

H5 Need details

TestWS?

H4

Scheduling onBossnet calendar

No scheduling;sufficient capacityfor e-VLBI

Scheduling necessaryif this path used

Will try to scheduleeither K2 or K3exclusively fore-VLBI experiment

Dedicated - noscheduling necessary

H2

J6

Details of path from Haystack to GGAO: Part 2

CommATM

NASACommATM

CommMux,

ATM SW

HICRouterATM

HICRouterATM

UHRouterATM

Kokee Park

OahuPMRFBarkingSands

Makaha Ridge

DRENATM

AbileneRouter

Seattle

USNO

DC Area

HaystackObs

BOSSNET/GLOWNET

Abilene

DREN

~2.4 kmPROPOSED DARK FIBER LEASE

VLBI Correlators

MAXFiberRingOC48

2.5 Gb/s

DCNEQwest

DCGW

CLPK

DARPAISI-E

OC48 2.5 Gb/s

ARLG

155 Mb/s

OC3UMd

GWU

OC3

OC48

Pacific Wave

Pacific NWGiga pop

Westin Bldg

155 Mb/s

2.5 Gb/s

Antenna

MMLink

GigETE?

Gig ETE ?

MMLink

Microwave Link

UofH Manoa Keller Hall SPAWAR Bldg 992

STM-1155Mb/s

HIC

OC3155 Mb/s

DREN

OC3155 Mb/s

Exp

erim

enta

l Lin

k

Gb/

s@

80

GH

z

Terminal Equipment (TE)Requirements?

GigE?

~8 km

FO cable12 pairs?

FO FO

3xDS3 135 Mb/s 3xDS3 135 Mb/s

OC48

SiteA

TMATM

6/25/2003

4

6/25/2003 19

UltraLight(An Ultra-scale Optical Network Laboratory for Next Generation Science)

• Packet-switched and circuit-switched hybrid experiment research network with transcontinental 10Gbps wavelengths on National Light Rail

• Collaboration of Caltech, MIT Haystack, U. of Florida, Florida IU, U. Mich, SLAC, Fermilab, CERN and others; commercial partner Cisco Systems

• Partner projects TransLight, Netherlight, ULlight, AMPATH, CA*Net4• Flagship applications

– High-energy physics– e-VLBI– High-resolution near-real-time medical imaging

• New techniques to be explored– End-to-end monitoring agents to determine how to best manage network data flows– Dynamic traffic routing– Dynamic scheduling of additional wavelengths– ‘Tunneling’ protocols to set up sub-paths with guaranteed BW

• Key part of proposal is to develop plan for connecting U.S. antennas• Proposed 5-year project, ~$10M, with several M$ contribution from industry

6/25/2003 21

DRAGON(Dynamic Resource Allocation via GMPLS Optical Networks)

• Collaboration with Univ. of Maryland, Mid-Atlantic Crossroads (MAX), Information Sciences Institute (USC) and NASA/GSFC

• Develop a Generalized MultiProtocol Label Switching (GMPLS) network to provide deterministic network resources at the packet, wavelength, and fiber cross-connect levels

• Will develop a set of API’s for application-level use of GMPLS

• Industry partner Movaz Networks will provide pre-production GMPLS-enabled wavelength-selective switches (MEM’s based switching fabric)

• Proposed 4-year project

6/25/2003 22

VSI-E• VSI-H defines input and output interfaces of a VLBI data system, but

specifically excludes details of media or data format

• However, e-VLBI intercompatibility among heterogeneous VLBI data systems is highly desirable

• Consequently, the VSI technical development committee is now turning its attention to this problem

• Goal is define a common e-VLBI data protocol and format, called ‘VSI-E’

• First rough draft has been created and distributed to form a basis for further discussion

• It now appears that some variation of RTP protocol may be most suitable for VSI-E

• Goal: Complete VSI-E specification by end 2003!

6/25/2003 23

622 Mbps +10 Gbps λ

Transoceanic donations to IEEAF (in red)6/25/2003 24

AMPATH: Research and Education Network and International Exchange Point for the Americas

• Launched in March 2000 as a project led by Florida International University (FIU), with industry support from Global Crossing (GX), Cisco Systems, Lucent Technologies, Juniper Networks and Terremark Worldwide

• Enables wide-bandwidth digital communications between the Abilene network and 10 National Research and Education Networks (NRNs) in South and Central America, the Caribbean and Mexico

• Provides connectivity to US research programs in the region

• AMPATH is a project of FIU and the National Science Foundation’s Advanced Networking Infrastructure & Research (ANIR) Division

Note: VLBI telescopes currently in Chile and Brazil