Recent results from the Swinburne

supercomputer VLBI software correlator(s)

Steven Tingay, Craig West, Adam Deller, Shinji Horiuchi

Centre for Astrophysics and SupercomputingSwinburne University of Technology

4th eVLBI Workshop: July 12 – 14 2005ATNF Marsfield

http://astronomy.swin.edu.au

Outline of talk• Goals of project.

• Why correlate in software?

• Details of the observing/correlator setup.

• Performance of the software correlator.

• Availability of system to proposers.

• Results for Vela pulsar.

• Results for J0006-0623.

• Results for PKS 0438-436 and PKS 0405-385.

• OH masers

• Wide-field imaging, other pulsars, “Double pulsar” observation on Saturday (ATCA/Parkes)

• Summary.

Goals

• Use supercomputers to demonstrate SKA-related techniques and technologies: process in software the sampled, digitised output of single dish and interferometric radio telescopes (VLBI/eVLBI!!);

• Develop these techniques and technologies and integrate them into existing radio astronomy National Facilities;

• Activities funded by State and Federal bodies and industry partners, in close collaboration with the ATNF, University of Tasmania, the Auckland University of Technology (New Zealand), and the University of Western Australia.

VLBI system

Parkes

RF electronics

~ few GHz

Δν = 16 MHz (56 – 72 MHz)

Sample/Digitise

101001010…

Multibeam correlatorBoard (application-specific device)

ATCA

RF electronics

Δν = 16 MHz (56 – 72 MHz)

Sample/Digitise

101001010…

Dell server-class PC (generic processingdevice)

Apple Xraid disks

Cray XD-1

Beowulf cluster

Why correlate in software?

• Moore’s law over the last ten years has made it possible – hopefully Moore holds on a bit longer;

• Short development time for code – well known problem, well defined algorithms;

• “Embarrassingly parallel” computing problem – suits parallel, multi-processor machines very well – strong cross-fertilisation with HPC and ICT sectors (LOFAR/Blue Gene). Vendor interest in the problem;

• The algorithms are very flexible and it is easier to alter the code to reach new regions of parameter space than alter correlators with fundamental hardware constraints;

• Software correlators suit small to medium sized arrays.

Details of the VLBI setup

– PC-based data recorders (see Chris’s talk)• Dell 1600 SC or similar (now superceded);• Metsahovi Radio Observatory “PC-EVN” VSI boards;• Apple Xraid mass storage (5.5 TB with latest disks)

– Swinburne Beowulf cluster supercomputer• 304 P4 processors;• Gigabit ethernet;• 6 x Apple Xraid units;

– Cray XD-1 chassis• 12 Operton processors;• 6 Xilinx FPGAs (ultimately PGPUs as well)• Co-owned with University of Western Australia and ATNF

Details of the software correlator• Currently runs on Beowulf cluster under MPI;

• Uses Intel Performance Primitive (IPP) libraries for optimised cross-correlation and Fourier transform routines;

• First two versions of code implemented as XF correlators – next generation will be FX (see Adam’s talk next);

• Complex correlation (multithreaded on single processor PCs);

• Post-correlation correction of fractional sample errors on timescales of ~few ms;

• Apply system temperature and gain information on the fly, standard FITS output (AIPS, AIPS++, MIRIAD, DIFMAP etc);

• Arbitrary correlator integration time (down to microseconds) and number of frequency channels (correlated up to 16000 for narrow band maser observations) – applications for pulsars, wide-field imaging, and super-high spectral resolution masers for example;

• Correlator is currently much slower than real-time, depending on the required output (getting much faster – see Adam’s talk);

• Trade off unique capabilities with speed performance!

Speed performance of current XF correlator

* system is i/o bound. Recent

To correlate full Australian LBA: improvement to i/o handling

gives ~x2 speed increase.6 antennas = 15 baselines;8 x 16 MHz (4 x 16 MHz at RCP; 4 x 16 MHz at LCP);128 channels;25 ms integration time;12 hour observation;All polarisation products;300 processor cluster;

~180 hours

# baselines # polarisation

products

Bandwidth (MHz)

# frequency channels

integration time (ms)

observation time (h)

correlation time (h)

# processors

X real-time

1 1 16 128 25 5.5 10 20 1.8

1* 1* 4* 64* 25* 6* 1.5* 20* 0.25*

For next generation FXcorrelator, see Adam’s talk, next. Faster!

Stay tuned for XD-1

Availability of the system

• We can now correlate in software VLBI observations at up to x4 the bandwidth of the S2 tape-based VLBI system (factor of 2 improved sensitivity):

– Aim for 1 Gbps operation (x8 bandwidth) in real time via optical fibre in 2006 (SKA demonstration)

• Data are correlated in software on Swinburne supercomputer – eventually on the end of AARNet;

• New facilities are available as part of the VLBI National Facility from June 2005, supported by the MNRF team at Swinburne, the ATNF, and other VLBI partners (first proposals received):

See http://www.atnf.csiro.au/observers/apply/avail.html

• Attempting first Trans-Tasman VLBI with Auckland University of Technology facilities;

• Fringe-check software (uses software correlator) improves VLBI reliability – see Chris’s talk

Vela pulsar

Average pulse profile. 10 bins overon-pulse region. Bins of ~1 ms in width.Can generate arbitrary number of bins andbin width. Coherent de-dispersion priorto correlation.

2.3 GHz observations, Parkes to Tidbinbilla (test the models and analysis of Gwinn et al. 1997, 2000)

Analyse the amplitude statisticsin each bin and fit scintillation models to estimate size of pulsaremission region. Work in progress.

2.3 GHz (4 x 16 MHz)

ATCA, Parkes, Hobart,Ceduna, Mopra (LBAHDR)

Hartebeestoek (Mark5)

Kashima (K5, 256 MHz - digitally filtered in software to match 16 MHz band – Hiroshi Takeuchi, July 2004, NICT News)

2 milliarcsecond resolution

Science paper in preparation

Results for J0006-0623, PKS 0438-436, and PKS 0405-385

Quasar PKS 0438-436:

2.3 GHz (4 x 16 MHz)

ATCA, Parkes, Hobart,Ceduna, Mopra, Tidbinbilla (LBAHDR)

10 mas resolution

Quasar PKS 0405-385:

Famous intraday variablesource (Kedziora-Chudzer et al. 1997)

2.3 GHz (4 x 16 MHz)

ATCA, Parkes, Hobart, Ceduna,Mopra, Tidbinbilla (LBAHDR)

10 mas resolution

OH masers

• 4 MHz

• 16384 frequency channels = 244 Hz/ch

• 1 second of data

• LL polarisation

• Parkes – ATCA

• 1720 MHz

Widefield imaging/other pulsar experiments

• Observation of NGC4945 in May to show that software correlator can be used for wide-field VLBI (2.3 GHz; 4 x 16 MHz).

• Observation of millisecond pulsar scintillation (GBT, Effelsberg, Arecibo; MarkV – correlated in software at Swinburne (project proposed – pending approval).

• Observation of the “double pulsar” on Saturday (this Saturday!: Parkes/ATCA/Hobart; 1.4 GHz; 8 x 16 MHz – Huygens mode)

– Pulsar “eclipses”;– Parallax/proper motion;– Pulsar nebula.– Phase referenced

Software correlator for realtime fringe checking (see Chris’s talk)

Summary• Swinburne software correlator is operational

and available as part of the Australian VLBI National Facility;– Capable for multiple recording formats (LBAHDR;

Mark V; K5 …….

• The next generation FX version will be much faster than our current XF – real-time Gb?;

• Software correlators are:– Relatively easy to develop – good for niche

users;– Capable and flexible – good for niche science;– Relatively slow – compared to dedicated

hardware;– Great fun!!