February 13th, 2004HEASARC Users Group
Nick White
HEASARC Status and Plans
Nick White
February 13th, 2004HEASARC Users Group
Nick White
The HEASARC CharterThe HEASARC CharterEstablished December 1990
• Maintain and disseminate data from previous and concurrent high-energy astrophysics missions
• Provide software and data analysis support for these data sets• Maintain and provide the necessary scientific and technical expertise
for the processing and interpretation of the data holding• Develop and maintain multi-mission analysis and support tools• Provide catalogs of observations and ancillary information for the
data holdings• Coordinate data, software and media standards with other
astrophysics sites• Support education and outreach activities in high energy
astrophysics (not part of original charter)
February 13th, 2004HEASARC Users Group
Nick White
The HEASARC and High Energy Astronomy Missions
HEASARC provides the multi-mission infrastructure that is used by parallel GOFs and science support centers in the LHEA and elsewhere
Swift
Chandra data@CXC
HETE2
2GLAST
Integral
February 13th, 2004HEASARC Users Group
Nick White
HEASARC Science Personnel• Nicholas White: HEASARC Head• Steven Murray: HEASARC Deputy Head (at CXC)• Lorella Angelini: Data Restoration, Image and timing
analysis, BeppoSAX, Swift (inc. AKBAR), Astro-E2• Keith Arnaud: Data selection software, spectral analysis• Michael Corcoran: Calibration, GLAST, HETE-2, ROSAT• Stephen Drake: Catalogs, website & archive quality
assurance, EUVE, RXTE, CGRO• James Lochner: Education and public outreach• Thomas McGlynn: Archives, Virtual Observatory,
SkyView• Bill Pence: FITS, Software, HEASOFT, Hera
February 13th, 2004HEASARC Users Group
Nick White
Recent HEASARC Highlights• Hardware:
– Acquisition of 8.4 TB (12 TB raw) TB network appliance (March 2003)– Upgrade to Gigabit network (Oct 2003)
• Science Archive:– XMM downloads now 2nd in volume to RXTE– HETE-2 archive reaches 150 GB and 51 bursts– Specialized Swift Browse interface
• Software:– HEAsoft 5.3 release (November 2003): Improved fv integration with DS9– Browse 7.0 release (January 2004): new upload & download features– XSPEC 12 testing by Integral SPI team (ongoing)
• Catalogs:– GRBCAT master catalog of gamma-ray bursts (Feb 2004)
• Education and Outreach– SkyView SciAm Top 50 award (June 2003: 2nd year in a row)– Teacher’s Hera beta implementation (July 2003)
• Services:– First VO Tool: Data Inventory Service (July 2003)
February 13th, 2004HEASARC Users Group
Nick White
The Physical ArchiveThe Physical Archive
Active Missions
RXTE (1995- )Chandra (1999- ) [data at CXC]HETE-2 (2000- ) Integral (2001-)XMM-Newton (1999- )
Past Missions
Ariel 5 EXOSATASCA GingaBBXRT BeppoSAX CGRO HEAO 1Copernicus HEAO 3COS B OSO 8DXS ROSAT Einstein SAS 2EUVE SAS 3 Vela 5B • Data from 24
missions currently in the archive
• 355 astronomical catalogs & mission tables
• The archive volume was3250 Gigabytes as of the end of 2003
Upcoming Missions
Swift (2004 Launch)Astro-E2 (2005 Launch)GLAST (2007 Launch)
February 13th, 2004HEASARC Users Group
Nick White
Usage & Data StatisticsUsage & Data Statistics
Gigabytes transferred per year1998 1999 2000 2001 2002 2003
Datasets byftp & http 844 1406 1880 2270 2917 3943
Science WebPages & Images 198 234 371 399 580 872
Main E&PO WebPages & Images 203 358 678 1010 3108 7581
February 13th, 2004HEASARC Users Group
Nick White
Data Transfers by Mission over Time
February 13th, 2004HEASARC Users Group
Nick White
Current Web Service Statistics• Archive volume: 3.3 TB (mostly compressed)
– Current ingest rate: 400 GB/yr– End of 2004 anticipated rate: 1.4 TB/year, with increased or new flows from
Swift, Integral, and XMM
• Missions in archive:Ariel-5, ASCA, BBXRT, BeppoSAX, Chandra, Compton, Copernicus, COS-B, DXS, Einstein, EUVE, EXOSAT,
Ginga, HEAO-1, HEAO-3, HETE-2, OSO-8, ROSAT, RXTE, SAS2, SAS3, Vela 5B, XMM-Newton
• Downloads (annual): 3.9 TB of data (by FTP and HTTP), 0.9 TB of science web pages (by HTTP) and 7.6 TB of E&PO web pages
• Browse: 355 local tables (+ ~3000 VizieR); 2,150,000 queries• SkyView: >60 surveys; >1,000,000 images generated
• 26,000 bibliography entries linking ADS to HEASARC datasets (ROSAT, ASCA, XMM-Newton, and some RXTE datasets)
February 13th, 2004HEASARC Users Group
Nick White
HEASARC Hardware Configuration
HEASARC(load balancer)
HEASARC12 GHz Dual Processor
HEASARC22 GHz Dual Processor
More slots available
12 TBNetwork Appliance
Archive
750 GBNetApp
Static web site and shared disks
Web servers
DLT Jukebox
Backup archive on 150 GB DLT tapes (copy to NSSDC)
Safehouse6 TB
AssortedStorage
Online archive backup
InternalMachines
Hera (x5)1 GHz
SkyView (x2)1 GHz
150 GBSurveys (x2)
240 GBExternal
Survey Cache
400 GBUser
Cache
DBMS1 on line
GSFC Portal
Linux, Solaris, Mac, Windows, …
Gig
ab
it
100 Mb
DBMS2 backup
Database (Sybase)2 GHZ dual Processors
February 13th, 2004HEASARC Users Group
Nick White
Web site maintenance• Main HEASARC web site has 15,000 static pages and
hundreds of dynamic pages (excluding the FTP area and data archive) comprising 7 GB in total
• Ensuring valid and up-to-date links pages requires constant attention– In the past, invalid links left unattended grew to ~500.– Concentrated effort has now eliminated all but a few broken or
out of date links (Newly broken links are fixed weekly)
• HEASARC is now 99% Section 508 compliant– Assuring appropriate access to HEASARC Web pages involved
changes to most Web pages– Compliance changes were often beneficial for all users.– Maintenance cost of 508 compliance is <=0.05 FTE effort
February 13th, 2004HEASARC Users Group
Nick White
Browse Catalog and Archive Interface
• Capable of querying all local and VizieR catalogs• Local and remote archives (e.g., CXC) • HTML, text, VOTable, FITS and Excel ouputs• New capabilities:
– Expressions allowed in search criteria ( fluxX/fluxY > 2)– Better integration with SkyView– Upload files of targets (in Browse 7.0)– Download queries for later re-use (7.0)– Dataset ID info added when available (7.0)– Links between tables
February 13th, 2004HEASARC Users Group
Nick White
Current Active Missions• Chandra
– Browse service fully linked to CXC archive• RXTE
– Dominates current archive downloads in terms of GB– Simple XTEIndex table developed and deployed (more simplification needs to be
done)• HETE2
– Some HEASARC-written FITS products for GRBs– Difficult to use most data delivered to archive
• Similarities to data recovery efforts of early HEASARC
• XMM-Newton– In 2003, second biggest HEASARC mission in archive volume & downloads– Archive contains both US & non-US Guest Observer data
• Integral– Use modified version of HEASARC archive software at ISDC
• HEASARC Browse can link to ISDC Browse tables– Archive at HEASARC will open in July 2004
February 13th, 2004HEASARC Users Group
Nick White
Preparations for Swift• New HEASOFT package• MOU delineating HEASARC responsibilities• End to end testing including archive ingest. • Customized mission interfaces:
– Individualized interface to Browse• HTML wrappers for standard Browse scripts
– Links between Browse tables– AKBAR: new service to import external information from
observers• HEASARC responsible for some high-level data products
– e.g., overall burst evolution• Mission requires rapid dissemination of data
February 13th, 2004HEASARC Users Group
Nick White
Swift Browse Interface
February 13th, 2004HEASARC Users Group
Nick White
Preparations for GLAST
• PDMP review
• Consultations on archive and catalog design
• Review proposed FITS formats
February 13th, 2004HEASARC Users Group
Nick White
Standards• FITS
– Subsumed GSFC FITS Office Web site– Develop and coordinate domain conventions for high energy data
(OFWG))• Organize periodic HEADCC meetings
– FITS Libraries Support• CFITSIO is essentially reference implementation of FITS• HEASARC also supports widely used FITS libraries in Java and IDL.• New C++ object implementation based on CFITSIO
• Standard Dataset IDs• VO Standards
– Browse support for VO Cone search– SkyView support for VO Simple Image Access Protocol– VOTable (and FITS) outputs available through Browse– VO Service Metadata descriptions for HEASARC resources
Publication of metadata using Open Archives Initiative (OAI) protocol
February 13th, 2004HEASARC Users Group
Nick White
Software Tools and Services• HEASARC software is frequently downloaded
– fv (FITS browser and editor): 500 downloads/month– CFITSIO library: 500/month– HEASOFT: 200-300/month
• HEASOFT– New packages for Swift and Astro E2 missions under
active development• Use HEADAS in order to provide a simple platform-
independent procedure for new software builds and updates
– XSPEC New major release (12.0) in alpha test• Complete re-write of Xspec internals in object oriented
fashion• Can support coded aperture mask data sets
February 13th, 2004HEASARC Users Group
Nick White
Hera• Provides a full, on-line analysis environment to users.
– Directly connected to archive, no downloads needed– Up to date versions of HEASOFT tools automatically available– Easily upload user data and download results.
• Uploaded data screened to ensure service is not used inappropriately.– Uses standard HEASARC FV FITS browser tool as user interface.
• Why?– Large volume of data (e.g., Integral, RXTE) downloads and/or software
installation may be infeasible for some users – With Hera, users can start using HEAsoft and other software without the
need for local installation• Status
– Initial system operational and being used actively.– Proposal for continued funding has been submitted– Both teacher and student interfaces in beta testing
February 13th, 2004HEASARC Users Group
Nick White
New and Updated Catalogs• ~24 new catalogs or major catalog updates in
last year.– Major correlative catalogs: VERONCAT, GB6, etc– High energy source catalogs: XMMSSC,
BMWHRICAT, INTREFCAT– Master catalogs updated regularly– GRBCAT: Compilation of all gamma-ray bursts
• VizieR connection for transparent access to massive catalogs (2MASS, USNO B, GSC 2)
February 13th, 2004HEASARC Users Group
Nick White
Education and Outreach• Education Web Sites:
– Imagine the Universe (Grades 6-12): 500K hits/month.
– StarChild (Grades 1-8) [in collaboration with L. Whitlock]: 1.25 M hits/month.
• New poster on origin of the elements• X-ray analysis school
– Highly successful and oversubscribed
• Host for APOD Web site• StarChild/Imagine/APOD CDs
February 13th, 2004HEASARC Users Group
Nick White
Other Projects• SkyView
– Java redesign– New surveys
(DSS2, Halpha, SFD) & SDSS soon.
– Request for continued funding submitted
• ClassX– Automated
classification of entire RASS and WGACAT samples of X-ray sources using supervised learning algorithm
Density of source requests in SkyView
February 13th, 2004HEASARC Users Group
Nick White
Virtual Observatory Involvement• Data Inventory Service• HEASARC metadata inventory
– Standardized metadata describing all HEASARC tables,data file types, surveys and services.
– Registered with VO to enable effective discovery of HEASARC resources.
– Table metadata
• HEASARC support for VO standards
February 13th, 2004HEASARC Users Group
Nick White
Data Inventory Service• Access to distributed services that use VO
protocols.– Many sites contributing data
• Dynamic service lookup using VO registries
• Catalog, observations, Images
http://heasarc.gsfc.nasa.gov/vo/
February 13th, 2004HEASARC Users Group
Nick White
Near Term Missions and Deadlines
• Launches– Swift: Fall 2004– Astro-E2: Early 2005– GLAST: Early 2007
• HEASARC Senior Review Proposal Due: March 17, 2004
February 13th, 2004HEASARC Users Group
Nick White
Current and Upcoming IssuesScience
• May need greater emphasis on supporting gamma-ray astronomers as Integral and GLAST Archives come online and become an increasing fraction of the HEASARC’s archive
• Must deal with ever increasing complexity of data and calibration– Coded aperture masks (Swift, HETE2, Integral)– Complex source dependent response (“ + GLAST)– Multi-satellite observations (Con-X, LISA)
February 13th, 2004HEASARC Users Group
Nick White
Current and Upcoming IssuesHardware
• CPU and disk space needs achievable, but bandwidth is a potential limitation– Datasets may be growing faster than bandwidth– Demand for data will spike after GRB detections– Higher speed linkage to Internet possible but costly
• … and doesn’t help users who have slower connections
– Move towards more user processing at HEASARC• Hera, grid computing
• On-line backup for archive is presently in same building as primary HEASARC archive.– Planned move of backup archive to another building when space
and network is ready (Summer 2004)
February 13th, 2004HEASARC Users Group
Nick White
Current and Upcoming Issues
Local Software and Services• Integration of tools and services
– Strengthen SkyView/Browse, Browse/NVODIS, Hera/Browse, … cross-links.
– Enable coupling archive/web services and HEASOFT tools.
• Systematic attention to internal documentation of HEASARC systems