Laser Interferometer Gravitational Wave Observatory (LIGO) Data Analysis System

Dr. James Kent Blackburn

LIGO Laboratory California Institute of Technology

Mail Code 18-34 Pasadena, CA 91125

USA

Abstract: - The Laser Interferometer Gravitational Wave Observatory (LIGO) will begin searching the cosmos in 2002 for gravitational waves as predicted by Einstein’s general theory of relativity These signals will characterize ultra relativistic events such as the final moments of inspiral in a tightly bound system of neutron stars and black holes, supernovae, quadrupole moments of spinning neutron stars, signatures from the earliest instance of the Big Bang and other events yet to be imagined. The initial LIGO detector will have a peak strain sensitivity of ~10-21 rms in a detection band of a few hundred Hz centered near 100 Hz. This sensitivity corresponds to measuring a differential displacements of ~ 10-18 meters rms, or 1/1000th the diameter of the nucleus of an atom between LIGO’s 4 kilometer long arms. The typical astrophysical signatures expected will have low signal to noise ratios requiring extensive computational resources to lift the gravitational wave out of the data stream and discern it from instrumental and environmental noise sources. To achieve this detection, LIGO is developing its distributed LIGO Data Analysis System (LDAS) which will interface raw data, database storage, parallel computing, signal processing, and scientific result generation using distributed computing technologies, and a hybridized software module call the LDAS API which combines the interpreted command language Tcl/Tk with the extensible compiled C++ language. Key-Words: - LIGO, gravitational waves, LDAS, data analysis, signal processing, distributed computing.

1 Introduction The general theory of relativity predicts that acceleration of masses generate ripples in the fabric of spacetime that propagate away from their sources at the speed of light in the form of gravitational waves [1,2]. Gravitational waves have yet to be detected directly, though their indirect influence has been observed with great accuracy in the binary pulsar PSR 1913+16 discovered in 1974 by Russell Hulse and Joseph Taylor [3]. Using extremely precise pulsar timing measurements, the decay in the orbital period of the system associated with the dissipation of energy in the form of gravitational waves agrees extraordinarily well with predictions from general relativity. The direct detection of gravitational waves will open a new window on the universe, providing a very different view from that portrayed by electromagnetic radiation. Gravitational waves will tell a story of massive coherent movement of matter interacting witin the highly relativistic and non-linear settings of general relativity’s strong field limit. The information carried to Earth by gravitational waves is virtually unaltered by

interactions with ordinary matter. This is in stark contrast to the description from incoherent interactions of atoms provided through the electromagnetic window on the universe, where photons are scattered, absorbed, and re-emitted along their path to electromagnetic detectors. The opportunity for unanticipated new physics from the direct observation of gravitational radiation is very high. The next generation gravitational radiation detectors based on suspended mass interferometers promise to obtain the needed sensitivity to open this new window on the universe. The Laser Interferometer Gravitational Wave Observatory (LIGO) is a National Science Foundation sponsored project currently under joint development by the California Institute of Technology and the Massachusetts Institute of Technology [4]. The project entails the design and construction of two long baseline LIGO interferometer facilities. The first facility is being built in the state of Washington (see Figure 1), and the second is being built in Livingston Parish, Louisiana (see Figure 2) near Louisiana State University. The two sites are separated by 3030 kilometers assuring a maximum difference in arrival time of 10 milliseconds for gravitational waves seen

in coincidence by the two sites. Each site’s interferometer has evacuated perpendicular arms 4 kilometers long in which resonating laser beams measure the infinitesimal length changes associated with gravitational waves.

Figure 1. Aerial view of LIGO’s Hanford corner station building. The two sites have arms very nearly aligned on the surface of the Earth, allowing for optimal signal strength at both observatories in the event of a signal. To increase statistical confidence in the detection of gravitational waves, the Hanford Observatory will include two interferometer, one of length 4 kilometers, and the other of length 2 kilometers allowing for and additional detection constraint of a 2:1 signal strength ratio between the two spatially overlapping Hanford interferometers. The two LIGO Observatories will be synchronized in time using GPS (Global Positioning System). Each observatory will also include a Physical & Environmental Monitoring (PEM) system to provide hundreds of additional channels of information, which can be used in veto of spurious terrestrial signals. This configuration will guarantee LIGO high detection confidence during a gravitational wave observation.

Figure 2. Aerial view of LIGO’s Livingston corner station building.

The LIGO Data Acquisition System will be responsible for collecting all data, both of scientific and engineering importance, at the observatories. The number of unique channels, which are roughly 500 per interferometer, and the ADC sample rates, which will be as high as 16384 samples per second,

will determine LIGO’s data acquisition rates. Taken to together, it is expected that each interferometer will generate between 3 and 5 megabytes of data per second. Because LIGO is an omni-directional detector and cannot be pointed at sources on the sky, it will collect from the full 4π steradians at once. The benefit of this being that for the most likely detection of rare astrophysical gravitational wave events, LIGO plans to run continuously 24 hours a day, 7 days a week. This integrates to a tremendous volume of data over the course of a year from LIGO’s three interferometers, estimated to be 300 to 500 terabytes. Of this data, LIGO will archive all scientifically useful data. Current plans are to reduce this data by a tenth of the full data rate using compression and discarding of redundant or unrelated data sets. To accommodate this large a volume of continuously collected data and to be able to search for gravitational wave signals as the data is collected from the interferometers, LIGO is developing a data analysis and data archival pipeline known as the LIGO Data Analysis System (LDAS).

2 Gravitational Waves Gravitational waves cause a time dependent strain on spacetime as they pass by. The pattern of this strain is restricted to the plane perpendicular to the direction of propagation. It forces compression along one direction and causes an expansion along the perpendicular direction within the plane. Like the electromagnetic wave, the gravitational wave can be represented by two orthogonal polarizations, h+ and hx, which are illustrated in Figure 3. The amplitude ho of a gravitational wave as approximated by the quadrupole moment is given in equation 1,

(1) nske

oo E

rc

G

dt

Qd

rc

Gh

≅

≅ 42

2

4

where the energy term E is the kinetic energy resulting from non-spherical internal motions of the source. Considers a typical gravitational wave source located in the Virgo cluster, having a distribution of mass on the order of one solar mass moving at a few tenths of the speed of light. This source would at most produce a strain amplitude on the order of 10-20 and most likely would be several orders of magnitude smaller. Ground based interferometers like LIGO will be most sensitive to gravitational waves in a frequency band ranging from 10 Hz to 1kHz. Astrophysics tells us that there are a number of candidate sources for which LIGO will be in a position to detect signature waveform over the background noise. Expectations

for detection are highest in four general categories of signals for which the most well developed physical models exist.

λGW

Figure 3. The left shows the distortion on a body that results from the passage of a gravitational wave. The hx polarization is shown on the right and h+ is shown in the middle. First among these are the chirp signals. These are signals produced by the inspiral of compact binary systems of black holes and or neutron stars. The signal increases in frequency and amplitude as it passes through the detectors bandwidth. The second source is the burst signals. These signals are very short in duration, typically associated with astrophysical events such as type II supernovae where the core deviates sufficiently from axial symmetry to produce a time dependent quadrupole moment. These candidate sources will require coincidence measurements using more than one interferometer. The third category of sources is the periodic sources. These signals have slowly evolving frequencies in the local rest frame of the source over time scales on order of weeks or months. However, because of the rotational acceleration experienced by detectors on the Earth, the signals undergo a complex transformation which Doppler shifts the signal as a function time and position on the sky. Astrophysics tells us that rotating neutron stars with small irregularities on their surfaces within our galaxy are potential sources when signals are integrated for sufficiently long time intervals. The remaining candidate source of gravitational waves is the stochastic background signal. These are signals that were emitted during the earliest instants of universe, known as the Plank epoch, prior to 10-43 seconds. Detection of these signals requires correlation of the background gravitational signals from two or more interferometers. Finally, the most interesting category of sources, which produce gravitational waves are the ones science has yet to imagine. These are the signatures, which lead to new insight into the nature of the universe. If history were to repeat itself, then LIGO’s opening of a new gravitation radiation

window on the universe will surely produce new physics and understanding.

3 Principle of Detection A Michelson interferometer is chosen as the bases for the optical topology. Suspended mirrors act as inertial masses. Fabry-Perot cavities are added to each arm of the interferometer to increase the storage time of the light in each arm and thereby enhance the effect of a gravitational wave induced strain on light travel time [5]. In addition, a recycling mirror is placed between the laser and the Michelson beam splitter in such a way as to constructively return light back into the interferometer (see Figure 4). The exact position and orientation of the mirrors in the interferometer are controlled using servo circuitry which feed back on the mirrors by means of driver coils which actuate small magnets mounted on each. The interferometer is held in a state of “lock” meaning that the light returning from each arm exactly cancel in phase at the output port where the photo-detector is located.

InputMirr or

InputMirr or

EndMirr or

EndMirr or

l 2

l 1

L2

L1

Laser

Photodetector

RecyclingMirr or

Fabry-Perot Arm Cavities (4km)Modest Input P ower (6W)Initial Laser (Nd:Y AG)W avelength (1.06µm)Power Recyc ling (30x)Modest Cavity Finesse (100)

Figure 4. LIGO detector configuration uses suspended mirrors act as test masses to probe the distortions of spacetime produced by gravitational waves.

When a gravitational wave of sufficient amplitude and polarization relative to the orientation of the interferometer arms pass by, it will cause compression in one arm of the interferometer while extending the other arm. One half a gravitational wave cycle later the opposite occurs. The compression in one arm and extension along the other causes the light returning to the beam splitter to no longer be 180o out of phase and light spills out of the interferometer onto the photo-detector. The servos will respond to this light leakage and drive the mirrors towards the null signal point. Tracking the response of the servo circuitry allows for reconstruction of the gravitational wave signal.

4 Data Collection and Archival The LIGO interferometers will continuously collect data for a period of 2 years in the LIGO I Science Run scheduled to begin mid year 2002. Of primary concern to the success of LIGO is having the ability to collect all of the scientifically meaningful data and to archive the data for future analysis and reference. The LIGO interferometers are expected to generate more than 300 terabytes of data each year. Out of this full data set, it is expected that through careful understanding of the LIGO’s multi-channel data and through data compression techniques this volume can be reduced to roughly 30 to 50 terabytes per year.

CACR: CIT, CA

LIGO Data Arch iveH P S S

LIGO: CIT, CA

"Mar t ian Network"

Data Ingest ion

CACR LAN (ATM)

LDAS LAN (ATM)

Genera l Comput ing LAN

meta/Data ServerGateway

ATM Swi tch

ATM Switch

Workstat ions PCs ATM-Ethernet Swi tch

Router

Laptop computer

Control & Monitor 200GB Cache

Tape

Tape Robot

ATM-Fast EthernetSwitch

DataConditioning

Tape

Reduced DataTape Robot

DatabaseServer (Sun)

200GBDatabase

LDASWorkstations

ATM-Fast EthernetSwitch

LDASFrame Server1TB Cache

ATM Swi tch

Robot ic Tape StorageIBM SP2

Gateway

CACR LAN (HIPPI)

Linux Cluster

HP Exemplar Cray SGI Or ig in 2000

Gateway

CACR LAN (Ethernet)

BeowulfCACR

Workstations

LIGO : WAN

(ESnet/vBNS)

Hanford LivingstonMIT LSC

LDAS Binaries, Libraries

& Shared Objects(Exported )

ManagerAPIFrameAPI

Meta(Event)DataAPILightWeightAPI

DataConditionAPI

Control/MonitorAPIEventManagerAPI

DataIngestionAPIDiskCacheAPI

RelationalDatabase

Server

MPI_APIWrapperAPI

GuiAPICommandAPI

WebAPI

Figure 5. LDAS LIGO “off-site” hardware and network components. Balloons identify the LDAS software, which will be running on each hardware element.

LIGO data is acquired across all channels into an international gravitational community format standard known as frames. Each frame has finite extent, typically one second, though longer intervals will also be used. Each frame is written to a file that has a name that uniquely identifies the time and location where the frame’s data was collected. These frames are stored on spinning disk media for roughly a day. From there the frame files are shipped by tape to the LIGO Data Archive located at Caltech (see Figure 5). At Caltech, LIGO will ingest the frame files using IBM’s High Performance Storage System (HPSS) capable of holding 200 terabytes of LIGO I Science Run data. Access to the data will be provided to all LIGO Scientific Collaboration (LSC) members participating in the initial LIGO Science Run. In addition, LIGO will provide computational resources to the LSC for carrying out demanding analyses used to extend and refine the interpretation of gravitational wave signals once they have been identified.

5 Data Analysis Understanding the detector and detection of ravitational waves will be central to LIGO operations during this science run. While new and unexpected discoveries will undoubtedly be a part of LIGO’s legacy, the search for gravitational waves of a known signal shape as predicted by general relativity and astrophysics will be the most computationally demanding mode of data analysis in the initial phases of operations [6]. LIGO is developing the LIGO Data Analysis System (LDAS) to carry on the computational analysis needed to detect gravitational wave signals within the interferometer data streams. Each of the LIGO Observatories will be equipped with an LDAS system (see Figure 6), as will the LIGO Data Analysis center located at Caltech (see Figure 5). In addition, the LDAS system is being developed with sufficient flexibility to allow installation of the software on a variety of hardware platforms supporting the Unix operating system. LDAS is being developed based on an open architecture design that utilizes components from data sharing client-server systems, distributed computation systems, parallel computation systems and relational database information mining systems. The system supports both a pipeline analysis designed to carry out on-site searches for gravitational waves in the data as it is being collected from the interferometers, as well as a user driven interactive analysis designed to carry out novel analysis that may provide greater insight into the unexpected physical behavior of the gravitational universe using the data stored in the off-site LIGO Data Archive.

LIGO: Hanford, WA

"Mar t ian Network"

Data Acquis i t ion

CDS LAN

LDAS LAN (ATM)

Genera l Comput ing LAN

meta/ data ServerGateway

GC ATM Swi tch

ATM Switch

Workstat ions PCs ATM-Ethernet Swi tch

Router

Laptop computer

Control & Monitor 100GB Cache

Tape

Tape RobotLinux Cluster

ATM-Fast EthernetSwitch

DataConditioning

Tape

Reduced DataTape Robot

DatabaseServer (Sun)

50GBDatabase

LIGO: Livingston, LA

"Mar t ian Network"

Data Acquis i t ion

CDS LAN

LDAS LAN (ATM)

Genera l Comput ing LAN

meta/data ServerGateway

GC ATM Swi tch

ATM Switch

Workstat ionsPCsATM-Ethernet Swi tch

Router

Laptop computer

Control & Monitor100GB Cache

Tape

Tape Robot Linux Cluster

ATM-Fast EthernetSwitch

DataConditioning

Tape

Reduced DataTape Robot

DatabaseServer (Sun)

50GBDatabase

LDASWorkstations

LIGO WAN(ESnet / vBNS)

LDASWorkstations

ATM-Fast EthernetSwitch

ATM-Fast EthernetSwitch

CDSFrame Server400GB Cache

CDSFrame Server 400GB Cache

LDAS Binaries, Libraries

& Shared Objects(Exported )

ManagerAPIFrameAPI

Meta(Event)DataAPILightWeightAPI

DataConditionAPI

Control/MonitorAPIEventManagerAPI

DataIngestionAPIDiskeCacheAPI

RelationalDatabase

Server

MPI_APIWrapperAPI

GuiAPICommandAPI

WebAPI

Figure 6. LDAS Hanford & Livingston “on-site” hardware and network components. Balloons identify the LDAS software, which will be running on each hardware element. The basic flow of data for both the on-site and off-site versions of LDAS goes like this:

a) Data enters into the system in the form of Frames. b) A subset of data is then selected out of the frames and concatenated into useful time intervals. Any statistics associated with the frame data are ingested into the LIGO database. c) The data is next preconditioned, preparing it for extensive parallel analysis. Here instrumental effects are eliminated Again any statistics associated with the preconditioning are stored in the LIGO database. d) The data is now sent off to a parallel computing environment where data is analyzed repeatedly by approximately 104 unique filters used in the search for gravitational waves. e) Any filter events that suggest a candidate gravitational wave exists in the data are then collected and systematically compared with instrumental characterization data stored in the database to identify possible terrestrial coincidences, which may mimic a gravitational wave. f) All gravitational wave events that are still strong candidates are entered into the database where they can be compared with gravitational wave events found in other detectors. This cycle continues on each new useful segment of data collected from the interferometers.

TCL/TK Layer: interpretor

Custom API TCL/TK code

Generic TCL/TK code:y communicat ionsy loggingy help

C/C++ Layer: shared object

Custom API C/C++ code

Generic C/C++ code:y ILWD data objectsy distr ibuted objectsy object store/restore

Tcl/Tk Resource File:y evaluated tcl / tky conf igurat iony envi ronment

Figure 7. Block diagram illustrating the common functionality integrated into each LDAS API through the combining of the interpreted Tcl/Tk command language with the compiled C++ class libraries.

The actual details of this pipeline are more complex, requiring communications between separate processes running on separate computers. In order to communicate between unique distributed compute components and still be able to efficiently analyze data, the LDAS software design incorporates a modular component called the LDAS API. It is a stand-alone process, which is built out of the interpreted Tcl/Tk command language, but extended with LIGO data analysis functionality using C++ class libraries that can be dynamically loaded into the Tcl/Tk command language (see Figure 7) and called upon as needed. Each LDAS API has common functionality to communication through Unix sockets any of its commands, messages, alarms, and data. The LDAS APIs also have common logging and state registration functionality. Apart from this, each

LDAS API functions uniquely as one component of the total LDAS distributed software system (see Figure 8).

Figure 8. LDAS software block diagram illustrating all the components, including LDAS APIs, algorithm & I/O libraries, database, parallel programs and users.

The communications of interferometer data is handled in the C++ layer. All interferometer data is represented in each LDAS API as a C++ object. Sending interferometer data out a socket actually results in the object being remotely instantiated through the socket. Using this method of communicating date by communicating objects is remarkably efficient over fast Ethernet. Tests show that for optimal sized objects, transmission rates can reach the nominal performance of the fast Ethernet used and still perform computationally demanding class constructor tasks such as byte swapping efficiently. The parallel algorithms used to search for gravitational waves in the interferometer data will run on a cluster of PC, each using the Linux flavor of the Unix operating system. This hardware technology is viable for LIGO data analysis because of the highly parallel nature of the chirp signal searches which tend to dominate the computational cost for LIGO when excluding the technology limited all sky search for periodic sources [7]. The approach used in detecting chirps is to divide up the parameter space of the waveform into a grid of closely spaced points, fine enough to miss only 1 in 10 signals and calculate all waveforms within some domain of the parameter space. Then using Wiener optimal filtering, each waveform is compared to interferometer’s data stream with the formula in equation 2.

(2) dfefS

ftfot fti

n

⋅⋅⋅= ∫∞

∞−

πρ 2*

)(

)()()(

Here o(f) is the Fourier transform of the interferometer data, t*(f) is the complex conjugate of

the Fourier transform of the calculated wave form, and Sn(f) is the noise spectral density of the interferometer.

Figure 9. The computation power needed to carry out a search for the chirp signal associated with inspiral of a compact binary system. Each of the four curves represents a different frequency bandwidth to use in the search algorithm. The computational requirements for a chirp search of the inspiral of a binary system of neutron stars and or black holes has been calculated for LIGO’s initial sensitivity using a loss of signal detection rate of ten percent [8]. The computational needs couple as a high power law to the mass of the smaller companion within the binary system (see Figure 10). Observation and astrophysics have identified 1.4 solar masses as a lower limit in the parameter space that must be searched out. LDAS intends to look slightly below this limit to account for uncertainties in the neutron star equation of state and the low counting statistics of observations.

The LDAS system will appear to users as a single server to which client software communicates. Users will issue requests to LDAS to carry out various types of analysis ranging from data mining through the database to starting computationally demanding tasks on the cluster of PCs using Message Passing Interface (MPI) parallel algorithms. To support the dynamic needs of users and to manage the LDAS resources in this client-server model. LDAS will use a managerAPI software module to parse user requests, schedule tasks and sequence though the steps needed to complete the request. Upon completion of a request, LDAS will produce data products using the eXtensible Markup Language (XML) to format documents, which LIGO has christened as the LIGO Light-Weight (LIGO_LW) format. Users will be notified of the availability of their data product documents through email along with pointers to the URL or FTP site where the document resides.

Figure 10. Users communicate with LDAS as clients. The managerAPI processes the user request using the underlying LDAS APIs. Once completed, data products from the tasks are delivered to a publicly accessible area for user software to ingest.

6 Acknowledgements The work summarized herein represents the collective efforts of the LIGO Laboratory. Their work in building LIGO is paving the way towards making direct detection of gravitational waves a reality. The National Science Foundation supports LIGO under cooperative agreement PHY-9210038. References: [1] A. Einstein, Reports of the Physical-

Mathematical Sessions of the Royal Prussian Academy of Sciences, 1916, p. 688.

[2] A. Einstein, Reports of the Physical-Mathematical Sessions of the Royal Prussian Academy of Sciences, 1918, p. 154.

[3] R. A. Hulse, J. H. Taylor, Astrophys. J., 195, 1975, pp. L51-L53.

[4] A. Abramovici, et. al., Science, 1992, 256, 1992, p. 325.

[5] R. Drever, Fabry-Perot cavity gravity-wave detectors, The Detection of Gravitational Waves, Cambridge University Press, 1991, pp. 306-317.

[6] K. Blackburn, The Laser Interferometer Gravitational Wave Observatory Project, LIGO, Mathematics of Gravitation Banach Center Publications, 1997, pp. 95-135.

[7] P. Brady, et. al., Phys. Rev. D, 57, 1998, p. 2101.

[8] B. Owen, Phys. Rev. D, 53 1996, p. 6749.