Upload
morris-hoover
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
e-VLBI: Connecting the World’s Radio Telescopes
with High-Speed Networks
Alan R. WhitneyMIT Haystack Observatory
Westford, Massachusetts, USA
Traditional VLBIThe Very-Long Baseline Interferometry (VLBI) Technique(with traditional data recording on magnetic tape or disk)
The Global VLBI Array(up to ~20 stations can be used simultaneously)
• Science Missions– Development of the very early universe
– Formation and evolution of galaxies
– Black holes in galactic cores
– Stellar nurseries
– Evolution and formation of elements
VLBI for Astronomy
• Combines data from global telescope array to synthesize a very large, distributed antenna
• Highest resolution technique available to astronomers – tens of microarcseconds (corresponds to resolving dimples on a golf ball at distance 5000 km!)
• Allows detailed studies of the most distant objects in the Universe
• Data is derived from national and international array of large radio telescopes
• Disadvantage: Very large amount of data and huge amount of computing required to make a single image
VLBI for Astronomy
VLBI astronomy example
VLBI astronomy example
Galaxy NGC6251
• Distance: ~500Mlight-yrs
• 1 parsec = 3.262 light-yrs = ~0.1arcseconds
• Allows observations to focus in on energetic core
Chautauqua 2001
Quasars, hotspots, polarization
Resolution: ~10 marcsec
VLBI astronomy example
• Fundamental VLBI measurement is time-of-arrival difference of signals between telescopes in array; typical single-measurement precision is <10 picosec
• Highest precision (few mm) technique available for global tectonic measurements
• Highest precision Earth Orientation and Earth Rotation measurements
• Earth-rotation measurements are important for military/civilian navigation
• Stable reference frame formed by distant quasars• Fundamental calibration for GPS constellation within Celestial
Reference Frame• Highest spatial and time resolution of Earth’s motion in
space for the study of Earth’s interior• International observing program includes ~35 stations
around the world, coordinated by International VLBI Service
• In South American, stations in Concepcion, Chile and Fortaleza, Brazil are regular contributors
VLBI for Geodesy
VLBI astronomy example
Plate Tectonic Motion from VLBI measurements
VLBI astronomy example
Correlation between Earth Rotation and Atmospheric Angular Momemtum
Conclusion: Primary driver of variations in Earth rotation rate is weather!
Disadvantages of Traditional ‘Record & Ship’ VLBI
• Long interval between data collection and results (typically weeks to months)
• Uncertainty of proper equipment operation during experiment
• Expensive media (tape, disks) sometimes lost or damaged in transit
• Limited bandwidth limits sensitivity (few Gbps per station is practical maximum)
A Little History: Mark 4 VLBI data system
16-station VLBI correlator at JIVE in The Netherlands
Enter ‘e-VLBI’ –
Electronic transmission of VLBI data over high-speed global networks!
e-VLBI is not new!
• 1979 – OVRO-Haystack 1 Mb/station; 2400-baud modems (Mark III)• 1980’s – JPL DSN for EOP; 500 kb/s sample rate buffered to disc,
56 kbps transfer over land lines • 1995 – Japanese Keystone project; 256 Mbps over dedicated fiber• 1999 – Europe; 1 Mb/station over Internet using ftp• 2000 - Japan develops 1 Gbps e-VLBI system within Japan over
dedicated links• 1977 – NRAO-Algonquin – 20 Mb/sec real-time satellite link
But the advent in global high-speed networks is completing changing the game!
Scientific Advantages of e-VLBI
• Bandwidth growth potential for higher sensitivity– VLBI sensitivity (SNR) proportional to square root of Bandwidth
resulting in a large increase in number of observable objects(only alternative is bigger antennas – hugely expensive)
– e-VLBI bandwidth potential growth far exceeds recording capability(practical recordable data rate limited to few Gbps)
• Rapid processing turnaround– Astronomy
• Ability to study transient phenomena with feedback to steer observations
– Geodesy• Higher-precision measurements for geophysical investigations
• Better Earth-orientation predictions, particularly UT1, important for military and civilian navigation
Practical Advantages of e-VLBI
• Increased Reliability– remove recording equipment out of field
– remote performance monitor & control capability in near real-time
• Lower Cost– Automated Operation Possible
• eliminates manual handling and shipping of storage media
– Real-time or near-real-time Processing• forestalls growth of storage-capacity requirements with bandwidth growth
– Elimination of recording-media pool (millions of $’s!)
• Avoid unexpected media-shipping interruptions and losses
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system– Mark 5 system developed at MIT Haystack Observatory with support from several
national and international institutions
• Demonstrate e-VLBI in feasibility experiment– ~700 km link between Haystack Observatory and NASA/GSFC
• Develop specialized e-VLBI data-transport formats and protocols– Develop international standard for e-VLBI data format– New IP-based protocol tailored to operate in shared-network ‘background’ to
efficiently using available bandwidth
• Extend e-VLBI to national and global VLBI community
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
• Demonstrate e-VLBI in feasibility experiment
• Develop specialized e-VLBI data-transport formats and protocols
• Extend e-VLBI to national and global VLBI community
Mark 5 VLBI Disk-Based Data System
• 1 Gbps continuous recording/playback to/from set of 8 inexpensive (ATA) disks• Developed at MIT Haystack Observatory with multi-institutional support• Mostly COTS components• Two removable ‘8-pack’ disk modules in single 5U chassis• With currently available 250GB disks – capacity of single ‘8-pack’ 2.0TB; expected to increase to 3.0TB by early 2005 at cost of ~$0.5/GB• GigE connection for real-time and quasi-real-time e-VLBI operations• Inexpensive: <$20K• ~75 Mark 5 systems now installed at stations and correlators
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
• Demonstrate e-VLBI in feasibility experiment
• Develop specialized e-VLBI data-transport formats and protocols
• Extend e-VLBI to national and global VLBI community
Haystack
Westford
NASA/GSFCUSNO
(correlator)
(correlator)
Bossnet 1 Gbps e-VLBI demonstration experiment
(Fall 2002)
• 788 Mbps e-VLBI transfer achieved over shared IP infrastructure, but took much tuning
• Full report at www.haystack.edu/e-vlbi
Initial experiment
Future ~700
km
Westford-GGAO e-VLBI results
• First near-real-time e-VLBI experiment demonstrated on 6 Oct 02– Recorded data at 1152 Mbps on Westford-GGAO baseline
– GGAO disk-to-disk transfer at average 788 Mbps transfer rate
• Direct data transfer experiment demonstrated on 24 Oct 02– Direct transfer of GGAO data to disk at Haystack at 256 Mbps
– Immediate correlation with Westford data
– Nominal fringes observed
• Conclusion– e-VLBI at near Gbps speeds over ordinary shared networks is possible
but still difficult
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
• Demonstrate e-VLBI in feasibility experiment
• Develop specialized e-VLBI data-transport formats and protocols
• Extend e-VLBI to national and global VLBI community
VSI-E
• VSI-E = VLBI Standard Interface for e-VLBI
• Follows on heels of VSI-H and VSI-S specifications over last 3 years
• Goal is to allow compatible interchangeable of data between heterogeneous VLBI data sytems
• VSI-E currently under development by international VSI committee
• RTP protocol has been chosen
• Draft specification currently under discussion.
• Goal: Complete VSI-E specification by mid-2004
• Prototype software will be available June 04
RTP Capabilities• RTP provides an Internet-standard format for:
– Transmission of sampled analog data– Dissemination of session information– Monitoring of network and end system performance (by participants and third parties)– Adaptation to varying network capability / performance– Message Sequencing / reordering– Multi-cast distribution of statistics, control and data
• RTP allows the reuse of many standard monitoring / analysis tools• RTP seen as internet-friendly by the network community:
– attention to efficiency • protocol designed to have minimum overhead for in-band data
– attention to resource constraints • won't use up all your bandwidth with control information
– attention to scaling issues
RTCP Capabilities• Monitors network’s real-time data delivery performance
• Statistics collected from receivers
• Information delivered to–Senders (adapt to prevailing conditions)–Network management (identifies faults, provisioning problems)
• Adaptive, bandwidth-limited design
Possible VSI-E Topologies
New Application-Layer Protocols for e-VLBI
• Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity
• New protocols are being developed which are tailored to e-VLBI characteristics; for example:– Can tolerate some loss of data (perhaps 1% or so) in many cases
– Can tolerate delay in transmission of data in many cases
• ‘Experiment-Guided Adaptive Endpoint’ (EGAE) strategy being developed at Haystack Observatory under 3-year NSF grant:– Will ‘scavenge’ and use ‘secondary’ bandwidth
– ‘Less than best effort’ service will not interfere with high-priority users
– Dr. David Lapsley has joined Haystack staff to lead this effort
Typical bit-rate statistics on Abilene network
Conclusion: Average network usage is only a few % of capacity
Usage >20Mbps less than 1% of the time
100 500 Mbps
0.1
1.0
0.01
0.001
Typical distribution of heavy traffic on Abilene
Conclusion: Heavy usage of network tends to occur in bursts of <2 minutes
secs200 400 1000
1.0
0.9
0.8
0.7
<10% of ‘bulk’ transfers exceed ~100 secs
New Transport Protocols for e-VLBI
• TCP is very inefficient is there are virtually any losses on a network due either to network sharing or due to physical packet losses– e-VLBI is particularly sensitive due to typical long RTT’s
• UDP can used, but is sometimes ‘unfriendly’ to other users
• New transmission protocols are being developed which are much more aggressive, but fair – some examples:– FAST (Caltech)
– Tsunami (Indiana University)
– UDT (UDP-based Data Transfer; Univ. of Illinois)
– High-speed TCP (Sally Floyd)
– XCP (Explicit Control Protocol; Dina Katabi, MIT)
Acceptable Latencies for e-VBLI
• Latency delays must not exceed the ability of the correlator to buffer the data
• Near real-time operation
– At 1 Gbps, disk buffers of large size can be used for; in this case latency is not a real issue.
– At 10 Gbps, disk buffers are not economically feasible and will require real-time correlation
• Real-time operation
– At 1 Gbps, a buffer size of ~0.5GB is currently available from each station;a latency delay of up to ~2 seconds is acceptable
– At 10 Gbps, it is likely that larger buffers will need to be built to accommodate latency delays up to at least 1 second
• Optically switched networks with dedicated wavelengths may eventually put the latency issue to rest as high-performance applications are able to use dedicated-lambda facilities.
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
• Demonstrate e-VLBI in feasibility experiment
• Develop specialized e-VLBI data-transport formats and protocols
• Extend e-VLBI to national and global VLBI community
Westford-to-Kashima e-VLBI experiments
• First Westford/Kashima experiment conducted on 15 Oct 02
– Data recorded on K5 at Kashima and Mark 5 at Westford at 256 Mbps
– Files exchanged over Abilene/GEMnet networks
• Nominal speed expected to be ~20 Mbps, but achieved <2 Mbps for unknown reasons - investigating
– File formats software translated
– Correlation on Mark 4 correlator at Haystack and PC Software correlator at Kashima
– Nominal fringes obtained
• Kashima data is now regularly transmitted to Haystack for processing
Westford to Sweden ‘Real-time’ experiments
• ‘Real-time’ experiments conducted in March/April 2004
– Data transmitted and processed in real-time (i.e. no disk buffering); data transmitted directly from stations to correlator
– First experiment at 32 Mbps due to temporary low-speed connection to Haystack
– Plan to extend these experiments to at least 512 Mbps with GigE links
UT1 ‘Intensive’ e-VLBI• Daily ~1 hour VLBI sessions between
Kokee Park, Hawaii and Wettzell, Germany are used for UT1 measurements
• Data are time sensitive since they are used for predicting UT1
• Currently requires ~4 day turnaround shipping media
• These measurements are an ideal candidate for routine e-VLBI
– Short daily session collect <100 GB of data– Even 100 Mbps will allow transfer in a few
hours
• Work now in progress to make necessary connections
– Network being organized from Kokee Park to USNO;connection speed OC-3
– Data from Mark 5 system in Wettzell will be carried over dedicated fiber at ~30Mbps to Univ. of Regensberg, then to GEANT
First e-VLBI to South America!
• First e-VLBI data transmitted from TIGO in Concepion, Chile to Haystack Observatory in April 2004
• Two scans of ~1.5GB each were transmitted at an average rate of 1.0-1.5 Mbps• Hope to continue this effort and upgrade the link!
A sample of international connections• Possibilities for international connections
– Surfnet – U.S. to Europe at 10 Gbps
– TransPAC/APAN – U.S. to Japan at 2.5 Gbps x2
– GEMnet – U.S. to Japan at 2.5 Gbps (privately operated by NEC)
– AMPATH – connections to telescopes in Chile and Brazil
– AARNET – Recently announced 2x10 Gbps connections to Hawaii and U.S.
– IEEAF –Europe/U.S./Japan link at 10 Gbps
– A sample of others under construction• GLORIAD – connecting China and Russis
• TEIN – Paris to Seoul
• EUMEDCONNECT – Europe to Mediterranean
• NeDAP – Europe to Russia
• ALIS – Europe to Latin and South America
– And many others………..
622 Mbps +10 Gbps
Transoceanic donations to IEEAF (in red)Credit: IEEAF
TransPAC Network(planned upgrade in 2004 to 2xGigE plus OC-48)
Credit: J. Williams, IU
AMPATH: Research and Education Network and International Exchange Point for the Americas
Launched in March 2000 as a project led by Florida International University (FIU), with industry support from Global Crossing (GX), Cisco Systems, Lucent Technologies, Juniper Networks and Terremark Worldwide
Enables wide-bandwidth digital communications between the Abilene network and 10 National Research and Education Networks (NRNs) in South and Central America, the Caribbean and Mexico
Provides connectivity to US research programs in the region
AMPATH is a project of FIU and the National Science Foundation’s Advanced Networking Infrastructure & Research (ANIR) Division
Note: VLBI telescopes currently in Chile and Brazil Credit: Julio Ibarra, FIU
0
UKUKUKUK
FRFRFRFR
CHCHCHCH
SESESESE
ITITITIT RORORORO
HRHRHRHR
EEEEEEEE LVLVLVLV LTLTLTLT
IEIEIEIENLNLNLNL
BEBEBEBE
DEDEDEDE
PLPLPLPL
CZCZCZCZ
HUHUHUHUATATATAT
SISISISI
SKSKSKSK
LULULULU
ESESESES
PTPTPTPT
GRGRGRGR
BGBGBGBG
CYCYCYCY ILILILIL622622 3434
3434
4545
622622
622622
155155155155
155155
155155
3434
3434
155155622622
155155 3434 4545
10 Gbps10 Gbps 2.5 Gbps2.5 Gbps
SE - PoP for Nordunet
GÉANT:The connectivity at 10 Gbps
Aarnet: SXTransport Project in 2004 Connect Major Australian Universities to 10 Gbps Backbone Two 10 Gbps Research Links to the US Aarnet/USLIC Collaboration on Net R&D Starting Now
Credit: George McLaughlin, AARNET
GLORIAD: Global Optical Ring (US-Ru-Cn)“Little Gloriad” (OC3) Launched January 12; to OC192 in 2004
Abilene - Upgrade Completed!
Credit: Internet2
National Light Rail Footprint
15808 Terminal, Regen or OADM siteFiber route
NLRStarting Up NowInitially 4x10 Gb
WavelengthsFuture: to 40x10Gb Waves
Transition beginning now to optical, multi-wavelength R&E networks. Also Note: XWIN (Germany); IEEAF/GEO plan for dark fiber in Europe
PITPIT
PORPOR
FREFRE
RALRAL
WALWAL
NASNASPHOPHO
OLGOLGATLATL
CHICHI
CLECLE
KANKAN
OGDOGD
SACSAC BOSBOSNYCNYC
WDCWDC
STRSTR
DALDAL
DENDEN
LAXLAX
SVLSVL
SEASEA
SDGSDG
JACJAC
What is the future of e-VLBI?
• Global connectivity is increasing at a very rapid rate
• e-VLBI is being aggressively developed in the U.S., Europe and Japan and will likely become standard procedure within the next decade.
• In South America, the ALMA project in Chile will be a large driver for high-bandwidth communications to the rest of the world.
Bandwidth Growth of Int’l Bandwidth Growth of Int’l HENP Networks (US-CERN HENP Networks (US-CERN
Example)Example)
Bandwidth Growth of Int’l Bandwidth Growth of Int’l HENP Networks (US-CERN HENP Networks (US-CERN
Example)Example) Rate of Progress >> Moore’s Law. Rate of Progress >> Moore’s Law. (US-CERN Example)(US-CERN Example)
9.6 kbps Analog9.6 kbps Analog (1985) (1985) 64-256 kbps Digital (1989 - 1994) [X 7 – 27]64-256 kbps Digital (1989 - 1994) [X 7 – 27] 1.5 Mbps Shared (1990-3; IBM) [X 160]1.5 Mbps Shared (1990-3; IBM) [X 160] 2 -4 Mbps2 -4 Mbps (1996-1998) [X 200-400] (1996-1998) [X 200-400] 12-20 Mbps (1999-2000) [X 1.2k-2k]12-20 Mbps (1999-2000) [X 1.2k-2k] 155-310 Mbps155-310 Mbps (2001-2) [X 16k – 32k] (2001-2) [X 16k – 32k] 622 Mbps622 Mbps (2002-3) [X 65k] (2002-3) [X 65k] 2.5 Gbps 2.5 Gbps (2003-4) [X 250k] (2003-4) [X 250k] 10 Gbps 10 Gbps (2005) [X 1M] (2005) [X 1M]
A factor of ~1M over a period of 1985-2005 A factor of ~1M over a period of 1985-2005 (a factor of ~5k during 1995-2005)(a factor of ~5k during 1995-2005) HENP has become a leading applications driver, HENP has become a leading applications driver, and also a co-developer of global networks; and also a co-developer of global networks;
Credit: Harvey Newman, Caltech
What are the problems?
• Biggest problem: ‘Last-mile’ connection of telescopes– Most telescopes are deliberately built in remote locations
– Biggest single obstacle is physical cost of laying fiber
– Cost of terminal equipment is rapidly diminishing, even for 10 Gbps
– A concerted international effort must be made to connect every major telescope in the world to high-speed network
• Some countries require buying services from service providers, which is very expensive; lighting dark fiber is far cheaper, if possible
e-VLBI is one among many applications
• HENP – high-energy physics community is currently heaviest user of international networks, mainly in dissemination of very large data files from major HENP facilities in Europe and U.S.
• Astronomy in general – optical telescopes are now sending high-resolution images in real-time to remote observers around the world;NVO will be among the world’s largest distributed database
• Education – tremendous opportunities here– Remote interactive learning from world experts in all fields
– Demonstration of advanced surgical procedures
– Easy international collaboration for all disciplines
– Master classes in the performing arts, with remote students having access to the world’s great performers and teachers
– Access to global databases and information of all kinds will help level the playing field between rich and poor nations.
Summary of Impact of e-VLBI Program
• Opens new doors for national and international astronomical and geophysical research.
• Represents an excellent match between modern Information Technology and a real science need.
• Motivates the development of a new shared-network protocols that will benefit other similar applications.
• Drives an innovative IT research application and fosters a strong international science collaboration.
The End
Thank you!