Astro-Wise workshop Nov 2005 Astro-Wise workshop Lorentz center 14-18 Nov 2005 Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden, Groningen Lorentz

Astro-Wise workshop Nov 2005

Astro-Wise workshopAstro-Wise workshop

Lorentz center 14-18 Nov 2005Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden,

Groningen

Lorentz center 14-18 Nov 2005Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden,

Groningen

Edwin A. ValentijnEdwin A. Valentijn


GoalsGoals

1. Tutorial to the system– New people

2. Prepare for projects and surveys– What to prepare?– Friday

3. Meet – collaboration- we share1. Code,2. Calibrations3. Hardware 4. projects

1. Tutorial to the system– New people

2. Prepare for projects and surveys– What to prepare?– Friday

3. Meet – collaboration- we share1. Code,2. Calibrations3. Hardware 4. projects


ScopeScope

• AstroWise provides infrastructure• Course nov 2002• AstroWise dec 2006• It is far, but not finished• QC and photometry development

will go on for many years

• AstroWise provides infrastructure• Course nov 2002• AstroWise dec 2006• It is far, but not finished• QC and photometry development

will go on for many years


Basic objectivesWide Field imaging EU

Basic objectivesWide Field imaging EU

Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results

100’s Tbyte of image data and 10’s Tbyte of catalogue dataWith production spread over EU

What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)

Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results

100’s Tbyte of image data and 10’s Tbyte of catalogue dataWith production spread over EU

What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)

(O)MegaCAM(O)MegaCAM


statusstatus

• 2002: VO conference, course - design• Build information system - working• Implemented qualified [email protected], WFC@INT, MDM,

OmegaCAM@ILT• >>Qualify with OmegaCAM@VST-2006• >>tune to run Public Surveys• >>quality control• >>Optimize federation/replication

• 2002: VO conference, course - design• Build information system - working• Implemented qualified [email protected], WFC@INT, MDM,

OmegaCAM@ILT• >>Qualify with OmegaCAM@VST-2006• >>tune to run Public Surveys• >>quality control• >>Optimize federation/replication



OmegaCAMOmegaCAM

• All service observing

• We calibrate the instrument• (not specific observtions)

• OmegaCAM Consortium and AstroWise hosted projects share the calibrations, and the work

• First ILT version is available

• All service observing

• We calibrate the instrument• (not specific observtions)

• OmegaCAM Consortium and AstroWise hosted projects share the calibrations, and the work

• First ILT version is available


AstroWise paradigmAstroWise paradigm

“Classical” paradigm

“hunting” AstroWise Target processing

waterfall model TIER architecture

User hunts upstream

driven by input raw data Driven by query of user

Process in pipelineProcess in bits and pieceson the fly

Operators push data User pulls data

Results in releases Provide information system

Static archives - publishDynamic archives –publish on internet

Raw data - obsolete Raw data is sacred


ParadigmParadigm

raw pixel data pipelines/cal files catalogues

all integrated in one information system• Collected in database – Oracle 10g

– Relational (table oriented db) but used for ++ code

• distributed services Virtual Survey Telescope» processing GRID » Storage GRID » Methods/services GRID

raw pixel data pipelines/cal files catalogues

all integrated in one information system• Collected in database – Oracle 10g

– Relational (table oriented db) but used for ++ code

• distributed services Virtual Survey Telescope» processing GRID » Storage GRID » Methods/services GRID


The avalancheintegrated dynamic db

The avalancheintegrated dynamic db

• on-the fly re-processing for everything • 5LS: 5 Lines Script Awe> prompt• Trend analysis Awe > prompt• All dependent bits are traced “tell_me_everything_tool• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO

• on-the fly re-processing for everything • 5LS: 5 Lines Script Awe> prompt• Trend analysis Awe > prompt• All dependent bits are traced “tell_me_everything_tool• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO


ComponentsComponents


componentscomponents

• Procedures + Cal plan at telescope• Data model -> object model ++ ->++db• Central db ; server/clients

– All I/O except images– Meta data– Source lists = catalogues + associate lists– Links = references = joins

• Fileserver – distributed- via db• Python clients• CVS distributed code base - opipe

• Procedures + Cal plan at telescope• Data model -> object model ++ ->++db• Central db ; server/clients

– All I/O except images– Meta data– Source lists = catalogues + associate lists– Links = references = joins

• Fileserver – distributed- via db• Python clients• CVS distributed code base - opipe


1- Components - procedurescalibration plan integrated

1- Components - procedurescalibration plan integrated




Monitoring Photometric Calibration

Monitoring Photometric Calibration


2 ComponentsData Model

2 ComponentsData Model

Sanity checksSanity checks

Quality controlQuality controlCalibration proceduresCalibration procedures

Image pipelineImage pipeline

Source pipelineSource pipeline


Astro-Wise PipelinesAstro-Wise Pipelines

Photometric pipelinePhotometric pipeline

Bias pipeline

Flatfield pipeline

Image pipeline

Source pipeline


Object modelObject model


Components 3:Database persistency

Components 3:Database persistency

• Native talks PL-SQL• Talks Python• All data items = persistent Classes• Nearly all I/O• Fileserver

• Native talks PL-SQL• Talks Python• All data items = persistent Classes• Nearly all I/O• Fileserver

SourceList

RA0, Dec0

t0

V filterSource

ListV filter

t2

RA0?, Dec0?

SourceList

V filterRA0?, Dec0?

t1

SS

C

I

TES

A

A

O

B filterRA0, Dec0

SourceList

SourceList

V filterRA0, Dec0

SS

C

I

TES

A

A

OS

S

C

I

TES

A

A

O

SourceList

B filterRA0, Dec0

SourceList

B filterRA1, Dec1

SourceList

V filterRA0, Dec0

SourceList

B filterRA1, Dec1

SourceList

B filterRA0, Dec0

SourceListB - VRA0, Dec0

SourceList

V filterRA0, Dec0

t0

SourceList

V filterRA0?, Dec0?

t1

SourceList

V filter

t2

RA0?, Dec0?

SourceList

V filterRA0?, Dec0?

tn

Proper motion

Red quasars

Supernovae

Tbyte source lists brains make the associations

Tbyte source lists brains make the associations

Link -lists

as fast as possible


VST - Virtual Survey TelescopeVST - Virtual Survey Telescope


Intra-operability peer to peerIntra-operability peer to peer

• code base + docs : CVS

• Db: “Advanced Replication” evolving to streaming

• code base + docs : CVS

• Db: “Advanced Replication” evolving to streaming

WRITE

–READ-ONLY

–READ-ONLY

–RE

AD

-ON

LY

–RE

AD

-ON

LY

WRITE

–REPLICATIO

N

–REPLICATION

–RE

PLIC

AT

ION

–RE

PLI

CA

TIO

N


Contents of federationContents of federation

• Raw data– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Co added images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects

• Raw data– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Co added images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects


Do It

Do It


Target processing:++ the make metaphor

Target processing:++ the make metaphor

awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382')

The processing chain is

ReadNoise <-- Bias <-- HotPixels

> class HotPixelMap(ProcessTarget): > > def self.make()

> class ProcessTarget(): > > def get(date, chip) # if not exist/up-to-date then make() > > def exist() # does the target exist? > > def uptodate() # is each dependency up to date?

Fully recursive

awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382')

The processing chain is

ReadNoise <-- Bias <-- HotPixels

> class HotPixelMap(ProcessTarget): > > def self.make()

> class ProcessTarget(): > > def get(date, chip) # if not exist/up-to-date then make() > > def exist() # does the target exist? > > def uptodate() # is each dependency up to date?

Fully recursive


Example 5LSExample 5LS

#Find ScienceFrames for a ccd named ccd53 and filter

Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’)

# From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q]

# get the rms of the used MasterflatAwe> y = [k.flat.imstat.stdev for k in q]

# Make a plot Awe> pylab.scatter(x,y)

#Find ScienceFrames for a ccd named ccd53 and filter

Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’)

# From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q]

# get the rms of the used MasterflatAwe> y = [k.flat.imstat.stdev for k in q]

# Make a plot Awe> pylab.scatter(x,y)


Astro-Wise PORTALAstro-Wise PORTAL


Web services- object viewerWeb services- object viewer


ESO-LVESO-LV


QC - calibration scientist monitoring






Web services- object makerWeb services- object maker


ProgrammeProgramme

• Afternoon Gijs Verdoes Kleijn– Demos– Exercises

• Afternoon Gijs Verdoes Kleijn– Demos– Exercises

Documents

Astro-Wise workshop Nov 2005 Astro-Wise workshop Lorentz center 14-18 Nov 2005 Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden, Groningen Lorentz