Image SubtractionImage Subtraction

Peter Nugent(LBNL/UCB)Peter Nugent(LBNL/UCB)

or....or....

If I Could Redo If I Could Redo Everything Again for Everything Again for PTF, This Is What I PTF, This Is What I

Would Do...Would Do...

Peter Nugent(LBNL/UCB)Peter Nugent(LBNL/UCB)

Things to KnowThings to Know Understand the instrument and changes to it - de-trending is

key to getting off to a good start: talk to the instrument scientists!

NEVER be happy with what you have: Speed/turn-around Types of db queries References Catalogs (stars, galaxies, etc.)

Know what science the collaboration would like to achieve: Try to accommodate everything from start Be flexible enough to adapt mid-way Always look for new scientific opportunities Learn their science

Do not mix image subtraction with other parts of pipelineiPTF Summer School

You do not need to visit the observatory!

I have processed ~1PB of data (20M ccd chips) between Palomar-QUEST and PTF. I did not have to go to the mountain, the mountain came to me...

PTF PipelinePTF Pipeline

50-100 GBs/night

iPTF Summer School

Image SubtractionImage Subtraction

iPTF Summer School

There are two types of image subtraction and they should not be confused – ever:

Real-Time Goal is to identify transients Photometry should be good, but does not have

to be perfect – in principle it can not be

Final Photometry Good enough to write a paper on cosmology Strives for perfection Major advantage: You know where the object

is... zoom in, pick your calibration stars, make perfect references, etc.

What is out thereWhat is out there

iPTF Summer School

hotpants – by Andy BeckerHigh Order Transform of PSF ANd Template Subtraction

http://www.astro.washington.edu/users/becker/v2.0/hotpants.html

There are a few variants (and you will hear more about one tomorrow) but they all have the same form:

Make a reference image Align and convolve with a new image Perform a subtraction Identify the candidates

hotpantshotpants

iPTF Summer School

hotpants -inim ${new} –hki -n i -c t -tmplim ${refremap} -outim ${sub} -tu ${template_saturation} -iu ${new_sturation} -tl ${template_lower} -il ${input_lower} -r ${2.5*seeing} -rss ${6.0*seeing} -tni ${refremapnoise} -ini ${newnoise} -imi ${submask} -nsx ${nsx} -nsy ${nsy}

hki : verbose output-c t : convolve to template-n i : normalize imagensx & nsy : size of regions within image (128X128 pixels ~ 2.5’)submasks: are key to getting things right (bad pixels kill)

I used the standard 3 gaussian & 6 degree polynomial for the kernel. No need to do more or less.

ReferenceReference

iPTF Summer School

Ideally the reference comes from one image, contributes no noise in the subtraction, and is of comparable seeing.

Nothing is ideal:

PTF had a dead chip. Pointing was atrocious, became ~1’ after

improvements Took ~3 months to obtain images from each field

that could make up a good reference Photometric calibration was USNO B1 catalog! Constantly made an effort to make better

reference images during the survey

Settled on ~7 images, best seeing (but not undersampled) to make reference on a PTF field/chip basis: depth, area & bad pix.

NewNew

iPTF Summer School

Don’t settle for having the survey forced down your throat, complain when things are going wrong!

Demand that fits header keywords are right, say for example the FILTER: this separates you from them

Know what the pointing/survey strategy is ahead of time (hitting M31 30 times in one night causes problems if you are not prepared for it)

Don’t bother with subtractions when they are not needed (|galactic latitude| < 10)

Everything is relative, treat the references as gold for photometric and astrometric calibration. Work out differences with the universe later (HST guide stars, absolute photometric calibration, etc.)

New- Ref = SubNew- Ref = Sub

iPTF Summer School

moon

This will always be a needle in a haystack problem.

New Image Reference Image Subtraction

New- Ref = SubNew- Ref = Sub

iPTF Summer School

Per image we would have ~250 5-σ detections. We would require 2

independent detections.

Use Machine Learning to get rid of the crap... Do not attempt to make the perfect subtraction!

Up to 300 images taken per night ~

1000 sq. deg.

PTF Sky CoveragePTF Sky CoverageReferences were made for ~20000 sq.deg. in R-band (minimum 7 minutes w/ seeing < 3.0” and limiting magnitude > 19.9).

iPTF Summer School

iPTF Summer School

• Hopper (N6): Cray XE6 Opteron w/ 153,216 cores

• Edison (N7): Cray XC30 Intel Ivy Bridge w/ 133,824 cores

• Cori (N8) will be one of the first large Intel KNL systems and will have unique data capabilities. 9,300 single-socket nodes with 60 cores per node and burst buffer (NVRAM) for the entire memory footprint.

• NERSC has a Global Filesystem which is viewable from all compute systems (40GB/s). Very high-speed local scratch space on each of the big-irons (168 GB/s)

• 240 PB tape archive

• Data Transfer nodes using ESnet

• Science Gateway and Database nodes for access outside NERSC

Access though general DOE-HEP call for compute time at NERSC.

3B cpu hrs / year

NERSCNERSC

iPTF Summer School

Why NERSCWhy NERSC• Why buy the cow, when you get the milk for free?

• You always want ~10X the compute you need to run a single night on hand at any time to catch up (network, shutdowns, new refs, etc.)

• The subtractions are the source of all complaints, whether they are justified or not.

– Where are my fields from last night?

– How come it is taking so long to see the subs?

– What is my SN/CV/GRB doing now?

Thus you don’t want computing to be one of them. NERSC operates 24/7 with staff on-call for issues that come up round the clock. As PTF was special, 100 khrs/yr but real-time, we were granted special privileges. Special queues, db’s, global disk space, etc. On average there are 3-4 shutdowns per year: all moved to full moon since 2009.

PipelinePipeline

NERSC GLOBAL FILESYSTEM250TB (170TB used)

DataTransferNodes

ScienceGatewayNode 2

ScienceGatewayNode 1

Observatory PTF Collaboration

via Web

Processing/db

Carver

Subtractions

iPTF Summer School

iPTF Summer School

• Chose a Postgres db with q3c for spatial queries• Based on studies comparing Oracle, mysql and

postgres• Runs at NERSC on their scidb nodes: 32-core nodes

on a ZFS filesystem• This currently houses the iPTF database which has

over ~3M images and ~1.5B detections which are queried in real-time 24/7.

ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include protection against data corruption, support for high storage capacities, efficient data compression, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs.

PTF dbPTF db

iPTF Summer School

q3cq3cQ3C is the plugin for PostgreSQL database, designed for working with large astronomical catalogs or any catalogs of objects on the sphere. Q3C allows you to perform fast circular, elliptical or polygonal searches on the sphere as well as perform fast positional cross-matches and nearest neighbor queries. Similar to htm (Hierarchical Triangular Mesh).

The ideas behind Q3C are described in Koposov et al. (2006)

PTF DatabasePTF Database

iPTF Summer School

PTF DatabasePTF Database

All in 851 nights. An image is an individual chip (~0.7 sq. deg.)The database reached 1 TB.

R-band g-band

images 1.82M 305k

subtractions

1.52M 146k

references 29.2k 6.3k

Candidates

890M 197M

Transients 42945 3120

Turn-aroundTurn-around

What does “real-time” subtractions really mean?

In the last 2 years of PTF, for 95% of the nights all images are processed, subtractions are run, candidates are put into the database and the local universe script is run in < 1hr after observation.

Median turn-around is 30m.

2012-07-06

iPTF Summer School

iPTF Summer School

Palomar 48”

Telescope

Palomar 48”

Telescope

SDSC to ESNETSDSC to ESNET

Astrometric SolutionAstrometric Solution

Reference Image

Creation

Reference Image

Creation

Image Processing

/ Detrending

Image Processing

/ Detrending

Star/Asteroid Rejection

Star/Asteroid Rejection

Image Subtractio

n

Image Subtractio

n

Nightly Image

Stacking

Nightly Image

Stacking

Transient CandidateTransient Candidate

Real-Bogus ML

Screening

Real-Bogus ML

Screening

HPWREN Microwave

Relay

HPWREN Microwave

Relay

NERSC Data Transfer

Node

NERSC Data Transfer

Node

Scanning Page

Scanning Page

Wake Me Up – Real Time

Trigger

Wake Me Up – Real Time

Trigger

Web UI MarshalWeb UI Marshal

Outside Telescope Follow-up

Outside Telescope Follow-up

Outside Database for Triggers

Outside Database for Triggers

40 Minutes

40 Minutes

Computing – I/O

Computing – I/O

Heavy DB

Access

Heavy DB

Access

Networking Data

Transfer

Networking Data

Transfer

500 GB/night

100 TBs of Reference Imaging

1.5B objects in DB

Real-TimeTrigger

Publish to Web

iPTF Summer School

Future SurveysFuture Surveys

ZTF (46 deg.2) iPTF (7.2deg.2)

Telescope AΩ

iPTF/PTF 8.7

DES 11.7

ZTF 42.6

LSST 82.2

ZTF image processing will be more challenging as the goal will be to do everything even faster and it is 12

times more data.

Parallel Parallel Processing/SubtractionsProcessing/Subtractions

All computers will have many cores, and the same All computers will have many cores, and the same amount of memory, 2+ years from now (10-100).amount of memory, 2+ years from now (10-100).

Current pipelines work at the level of one ccd chip Current pipelines work at the level of one ccd chip per core – this will fail in the future.per core – this will fail in the future.

Need to parallelize all aspects of the pipeline Need to parallelize all aspects of the pipeline where possible. Threading is easy for most of this, where possible. Threading is easy for most of this, keeping things in memory where possible is ideal:keeping things in memory where possible is ideal: Astrometric catalogs matchingAstrometric catalogs matching Bad pixel masks, CR’sBad pixel masks, CR’s Flats, biases, masks, etc.Flats, biases, masks, etc. Asteroid rejection (verification)Asteroid rejection (verification) Comparison with historical transientsComparison with historical transients

iPTF Summer School

iPTF Summer School

Bottlenecks…crude Bottlenecks…crude vsvs. . realreal

time

bri

gh

tness

5- data in db

iPTF Summer School

Conclusions - FutureConclusions - Future

LSST - 15TB data/nightOnly one 30-m telescope