View
215
Download
0
Category
Preview:
Citation preview
Virtual Laboratories
VGL and FriendsR.Fraser, T.Rankine, J.Vote, R.Woodcock
AuScope Grid Roadshow 2014
CSIRO | MINERAL RESOURCES FLAGSHIP
Messy desk
2 |
– where’s your analysis?
Let’s map it out…Hardcopy of data
SSH Client
Maths PackageProprietary Package
Transform to a regular grid
Crop data to area of interest
Reformat data for processing
Upload data to HPC resource
Configure job and start
processing
Download results
Get field data Visualise data
3 |
Components of a Virtual Laboratory
Data Services
Toolboxes Compute Services
Enablers Virtual Laboratory
4 |
…to this
5 |
Virtual Geophysics Laboratory
Build “science” from existing
libraries
Run jobCollect and
publish results
Discover (raw) data
Select spatial bounds
Virtual Laboratories – Scientific Workflows
• Automated - Simplify access and integration• Reproducible process• Flexible – reusable, support differing licensing
• Allow scientists to focus on the science and not the tech• Exploiting the Cloud and HPC• Operational or education tool
6 |
Components of a Virtual Laboratory
ToolboxesData Services
Compute Services
Enablers Virtual Laboratory
7 |
Data discovery
8 |
Data discovery
9 |
Some data services support subsetting
Some data services support reformattinge.g. CSV, NetCDF, GeoTIFF
Components of a Virtual Laboratory
Data Services
Toolboxes Compute Services
Enablers Virtual Laboratory
10 |
Toolboxes & Compute Services
11 |
A variety of different Toolboxes (scientific codes) are available
Flexibility in what compute services to utilise
Toolboxes
12 |
The steps so far have been building an environment to run a processing script
...or build from existing templates
Either write your own...
Components of a Virtual Laboratory
Data Services
Toolboxes Compute Services (OGC, Provenance)
Virtual Laboratory
Enablers
13 |
Virtual Geophysics LaboratoryBuild “science” from existing
libraries
Run job
Collect and publish results
Discover (raw) data
Select spatial bounds
Published Record
Enablers/Provenance
14 |
• Standards are used to enable….”enough said”• Provenance – information that describes creation of something
• Process record• Audit trail• Publish the actual method/process used to create the results!
What have we gained?
15 |
Provenance“....documenting
their procedures .......in
VGL it is all recorded for us.”
Data Access - “Geophysicists no
longer have to spend days
searching for data from multiple
sources, downloading........ reformatting....”
Reproducibility “......collaboration and sharing now possible”
Barrier to Entry“......making geophysical
processing and modelling tools available to the
broader community at a far lower cost. It
lowers the skills barrier to entry”
Components of the Virtual Geophysics Laboratory
Data Services
Toolboxes Compute Services
Dynamic Virtual Geophysics
Laboratories
Magnetics
Gravity
DEM
eScript
Underworld
PetascaleCompute
Org.Cloud
NeCTAR Cloud
AmazonCloud
Desktop
Service Orchestration
VGLPortal
ProvenanceMetadata
ScriptingTool
eScript
Mag. Grav.Org.Cloud
VGLPortal
VGLPortal
DEM
Mag. Grav.
PetascaleCompute
Org.Cloud
Underworld
Enablers(OGC, Provenance)
16 |
Repurposing to a Virtual Hazards Laboratory (VHIRL)
Dynamic Virtual Hazards
Laboratories
Magnetics
Gravity
DEM
ANUGA
eScript
Petascale
Org.Cloud
NeCTAR Cloud
AmazonCloud
Desktop
Service Orchestration
VHIRLPortal
ProvenanceMetadata
ScriptingTool
ANUGA
Mag. Grav.Petascale
VHIRLPortal
VHIRLPortal
DEM
Bathy DEM
AmazonCloud
Org.Cloud
Landsat
Bathymetry
eScript
Unchanged
Data Services
Toolboxes Compute Services
Enablers(OGC, Provenance)
17 |
VL: Engagement
• My job (Provenance)• Reproducible process
• My toolbox (Market Place)• Analogous to cloud application store• Automation
• My Field(VL core)• Flexibility
18 |
Recommended