e-VLBI Development Program at MIT Haystack Observatory

Alan R. WhitneyChester A. Ruszczyk

MIT Haystack Observatory

13 July 2005e-VLBI Workshop

Australia

Current Projects at Haystack Observatory

• Standardization– VSI-E Draft VSI-E standard distributed in January 2004

– Reference implementation released in October 2004

• Network interfacing equipment for e-VLBI– Mark 5 VLBI data system

• Network Monitoring– Evaluation, development and deployment of monitoring systems

• Intelligent Applications– Automation of e-VLBI transfers an ongoing process

– Development of optimization-based algorithms for intelligent applications ongoing (EGAE)

– Intelligent optically-switched networks (DRAGON)

• e-VLBI Experiments– Goal to put e-VLBI into routine use

VSI-E Architecture

VSI-E

• Purpose:– To specify standardized e-VLBI data formats and transmission protocols that

allow data exchange between heterogeneous VLBI data systems

• Characteristics:– Based on standard RTP/RTCP high-level protocols– Allows choice of IP transport protocols (TCP-IP, UDP, FAST, etc.)– Scalable Implementation; supports up to 100Gbps– Ability to transport individual data-channel streams as individual packet

streams; potentially useful for distributed correlators– Ability to make use of multicasting to transport data and/or control information

in an efficient manner

• Status– Draft VSI-E specification completed January 2004– Prototype VSI-E prototype implementation Nov 2004– Practical implementation for K5 and Mark 5 now is progress– Plan to use VSI-E in real-time demo at SC05, Nov 05

Reaching 1024 Mbps with Mark 5

• Achieving 1024Mbps with Mark 5 is challenging

• Can move ~1.2 Gbps between StreamStor card memory via PCI bus, but– If GigE NIC is on same PCI bus, bus contention slows aggregate transfers

to ~400-550Mbps, depending on motherboard

– Single GigE connections tops out at ~980Mbps (theoretically and experimentally)

– Typical GigE drivers require interrupt service every Ethernet frame; can generate up to ~100,000 interrupts/sec

• Elements of Solution– Capable motherboard with multiple independent PCI buses

– Dual ‘channel-bonded’ GigE links

– Driver or hardware interrupt mitigation; use of ‘jumbo frames’

– Careful software structure

Mark 5 e-VLBI Connectivity

• Mark 5 supports a triangle of connectivity for e-VLBI requirements

Data Port/FPDP

Disc array

PCI bus/Network(64bit/66MHz)

Mark 5 can support several possible e-VLBI modes:• e-VLBI data buffer (first to Disc Array, then to Network); vice versa• Direct e-VLBI (Data Port directly to Network); vice versa• Data Port simultaneously to Disc Array and Network at ~800 Mbps

Anatomy of a (fairly) modern motherboard(Tyan Thunder i7501 Pro)

PCIBridge

StreamStor

MemoryControl

Hub

1GBmemory

2.8GHzXeon

2.8GHzXeon

DualGigE

I/OCntlr

Mark 5AI/O

Board

HubLink 2.01.6GB/s

HubLink 2.01.6GB/s 64-bit

133MHzDual-edge

PC2100266MHz

1GBmemoryPCI-X

64-bit133MHz

PCI-X64-bit

133MHz

Intel 82546EBJumbo frames

to 16kB;Interrupt mitigation;channel-bonding

PCIBridge

Intel P64H2

Intel P64H2

FSB 64-bit 533MHz (4.2GB/sec)

PCI32-bit33MHz

64-bit 66MHz

HubLink 1.0

FPDPBus

Best transfer rates to date

• Memory-to-memory transfers between Tyan motherboards – ~1900Mbps• Uses dual channel-bonded GigE connection

• Mark 5A-to-memory transfer – ~1200 Mbps• Required major re-working of Mark 5A software to improve efficiency of

data-transfer to/from NIC, minimize number of internal buffer-to-buffer transfers, and support multiple threads

• More work still to be done to achieve routine 1024 Mbps Mark5-to-Mark5 transfers

• We plan to concentrate our efforts on implementing and optimizing with VSI-E to achieve 1024 Mbps• There should be no performance difference between Mark 5A

and Mark 5B

e-VLBI Network Monitoring

• Use of centralized/integrated network monitoring helped to enable identification of bottleneck (hardware fault)

• Automated monitoring allows view of network throughput variation over time– Highlights route changes, network outages

• Automated monitoring also helps to highlight any throughput issues at end points:– E.g. Network Inteface Card failures, Untuned TCP Stacks

• Integrated monitoring provides overall view of network behavior at a glance

• Also examining performance-monitoring packages such as MonaLisa, which would provide better standardization

Network State DataBase (NSDB)

• Tool to keep track of state of e-VLBI state:– Network performance– Configuration of end systems– State of end systems

• Integrates and builds on standard monitoring tools to provide a single, coherent view of e-VLBI network state:

– Maintain continuous state monitoring of entire e-VLBI system– Essential for being able to identify issues with network/end system

configuration– Diagnose at-a-glance (cf. current practice)

NSDB Architecture

e-VLBI Weather Map Web Page(Haystack to Kashima)

http://web.haystack.mit.edu/e-vlbi/evlbi.html

Network Layer Statistics

New Application-Layer Protocols for e-VLBI

• Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity

• New protocols are being developed which are tailored to e-VLBI characteristics; for example:– Can tolerate some loss of data (perhaps 1% or so) in many cases

– Can tolerate delay in transmission of data in many cases

• ‘Experiment-Guided Adaptive Endpoint’ (EGAE) strategy being developed at Haystack Observatory under 3-year NSF grant:– Will ‘scavenge’ and use ‘secondary’ bandwidth

– ‘Less than best effort’ service will not interfere with high-priority users

– Translates science-user criteria into network constraints

Automation of e-VLBI transfers

• Based on EGAE, major effort is now underway to fully automateroutine e-VLBI file transfers

• Algorithms are being built around use of standardized e-VLBI file-naming conventions (as agreed by Himwich, Koyama, Reynolds, Whitney, Nov 2004); see memo #49 at ftp://web.haystack.edu/pub/e-vlbi/memoindex.html

– We urge universal adoption of standardized e-VLBI file naming for ease of data interchange

Experimental and Production e-VLBI

• August 2004:– Haystack link link upgraded to 2.5 Gbps– Real-time fringes at 128 Mbps, Westford and GGAO antennas,

Haystack Correlator• September 2004:

– Real-time fringes at 512 Mbps, Westford and GGAO antennas, Haystack Correlator

• November 2004– Real-time e-VLBI demonstration at SC2004 at 512 Mbps– Use DRAGON optically-switched light paths

• February 2005– Real-time fringes Westford-Onsala at 256Mbps– Used optically-switched light paths over part of route

• October 2004 – present– Regular transfers from Kashima (~300GB per experiment; ~200 Mbps)

• Starting April 2005– Routine weekly transfers from Tsukuba (~1.2TB/transfer)– Preparing for CONT05 (15 days continuously; ~1TB/day)

Real-time e-VLBI SC2004 Demo

Bossnet

DRAGON

Haystack

Westford

Goddard

GGAO

Pittsburgh Convention Center

512 Mbps

512 Mbps

DRAGON Project(Dynamic Resource Allocation for FMPLS Optical Networks)

• Dynamically-provisionally optically-switched network research project

– U. of Maryland, ISI – PI’s• 10GBPS DRAGON network is being installed around Washington,

D.C. area, with connections to Abilene, HOPI and NLR• e-VLBI is primary demonstration application, using 2.4Gbps

dedicated connection to Haystack– Programmatic interfaces to EGAE are under development– Hope to upgrade Haystack connection to 10 Gbps in near future

• DRAGON will play a prominent role in e-VLBI demos scheduled for iGRID (Sep 05) and SC05 (Nov 05)

DRAGON Network

RE1RE1

RE3RE3

OSPF control plane adjacencies

WXC2

RE1RE2

WXC1

WXC1

RE1RE4

HOPI

ATDnet/Bossnet

WXC2

EXC2

EXC1

UMCP

GSFC

ISIE NCSA

MCLNARLG

CLPK

M10

WXC

HAYS

Abilene

Movaz NetworksiWSS Optical Switch

• MEMS-based switching fabric

• 400 x 400 wavelength switching, scalable to 1000s x 1000s

• 9.23"x7.47"x3.28" in size

• Integrated multiplexing and demultiplexing, eliminating the cost and challenge of complex fiber management

Dynamic power equalization (<1 dB uniformity), eliminating the need for expensive external equalizers

Ingress and egress fiber channel monitoring outputs to provide sub-microsecond monitoring of channel performance using the OPM

Switch times < 5ms

In summary - Some lessons learned

• High-performance e-VLBI is still hard to do– Cannot count on consistent performance

– Varying traffic loads

– Network configuration changes

– Equipment failures

– Continuous network monitoring is critical to success of on-demand RT e-VLBI

• Jumbo-frame support is important at rates >~256Mbps on GigE– Jumbo-frame support is spotty, but improving

Some Challenges

• Network bottlenecks well below advertised rates• Performance of transport protocols

– untuned TCP stacks, fundamental limits of regular TCP• Throughput limitations of COTS hardware

– Disk-I/O - Network• Complexity of e-VLBI experiments

– e-VLBI experiments currently require significant network expertise to conduct

• Time-varying nature of network• Define standard formats for transfer of data and control information

between different VLBI systems• ‘Last-mile’ connectivity to telescopes

– Most telescopes are deliberately placed in remote areas– Extensive initiatives in Europe, Japan and Australia to connect;

U.S. is lagging

Some Frustrations

• Telescope connectivity, particularly in U.S. , remains a significant challenge– Westford – 1 Gbps

– GGAO – 1 Gbps

– Arecibo – 155 Gbps

– VLBA – not connected

– GBT – not connected

– CARMA – not connected

– JCMT – not connected

– SMA – not connected

• Much difficulty in securing funding support from NSF Astronomy for e-VLBI– Need to develop convincing science case

Future Directions

• Further EGAE and VSI-E development and deployment• Improved IP protocols for e-VLBI• Optically-switched networks for highly provisioned high-data-rate

pipes• Solving ‘last mile’ problem to U.S. telescopes• Distributed correlation using clusters and/or highly distributed PC’s• Extending to higher bandwidths

– Haystack has Astronomy NSF grant to push for 4Gbps/station– Preparing NSF proposal to extend to 16Gbps/station using new digital-

filter and recording technology

• Continuing to move e-VLBI into routine practice on a global basis

e-VLBI Technical Working Group

• Established at this e-VLBI workshop as group of technical experts, David Lapsley chair• On hold until David Lapsley replacement is on-board

• Hope to re-invigorate at July e-VLBI workshop in Sydney

• Objectives

– Evaluate e-VLBI/VSI-E hardware/software/procedures

– Implement standardized global e-VLBI network performance/monitoring tools

– Provide expert assistance to e-VLBI users

• ~2 members from each major e-VLBI geographical area

Thank you - THE END

Questions?

Antenna/Correlator Connectivity• JIVE Correlator (6 x 1 Gbps)• Haystack (2.5 Gbps)• Kashima, Japan (1 Gbps)• Tsukuba, Japan (1 Gbps)• GGAO, MD (10 Gbps)• Onsala, Sweden (1 Gbps)• Torun, Poland (1 Gbps)• Westerbork, The Netherlands (1 Gbps)• Westford, MA (2 Gbps)• Jodrell Bank (1 Gbps?)• Arecibo, PR (155 Mbps)• Wettzell, Germany (~30 Mbps)• Kokee Park, HA (nominally ~30 Mbps, but problems)• TIGO (~2 Mbps)

In progress:• Australia – plan to connect all major antennas at 10Gbps!• Hobart – agreement reached to install high-speed fiber• NyAlesund – work in progress to provide ~200Mbps link to

NASA/GSFC