View
224
Download
0
Category
Tags:
Preview:
Citation preview
Chad BerkleyNCEAS
National Center for Ecological Analysis and Synthesis (NCEAS),University of California Santa Barbara
Long Term Ecological Research Network Office, University of New MexicoUniversity of Kansas
San Diego Supercomputer Center
Kepler: A Workflow Tool for Heterogeneous Ecological Data
Analysis
http://seek.ecoinformatics.org December 4, 2003Edinburgh, Scotland
Outline
Quick history SEEK overview Ecological Metadata Language Using workflows in Ecology Workflow editing with Kepler Future visions
History
Late 1990s – patterns noticed in the problems surrounding data synthesis at NCEAS
1999 - Michener et al paper on ecological metadata
2000 – Knowledge Network for Biocomplexity Morpho, Metacat, Ecological Metadata Language Some footholds into workflow creation and execution
2003 – Scientific Environment for Ecological Knowledge (SEEK) Grant Continues the work done on the KNB grant Emphasis on using metadata for advanced data
processing
SEEK approach
General approach to specific ecological problems
Data described with adequate metadata in a grid accessible repository
Reasoning engine (ontology based) to locate and extract data and processes
Modeling system to put it all together and control execution flow
SEEK Components
Ecogrid Analysis Library Metadata and data repository
Semantic Mediation System Controlled semantic vocabulary Ontological discovery system
Analysis and Modeling System (Kepler) Workflow control system Utilizes resources from other components
SEEK Architecture
Ecological Metadata Language
Common language for archiving and transport of datasets
XML based Designed for/by the ecological
community Describes physical and logical
structure of data Also includes project, literature and
software information SEEK will add semantic information
Workflows in SEEK
In the SEEK model, data ingestion/cleaning is metadata driven (specifically with EML)
Output generation includes creating appropriate metadata
The analysis pipeline itself becomes metadata
Metadata driven data ingestion
Key information needed to read and machine process a data file is in the metadata File descriptors (CSV, Excel, RDBMS, etc.) Entity (table) and Attribute (column)
descriptions Name Type (integer, float, string, etc.) Codes (missing values, nulls, etc.) In the future, this will include semantic typing
Metadata revision
Metadata is revised following any transformation
Versioning of metadata and data is very important
This process results in a lineage of the data file as it has been transformed
Typical ecological workflow example
Workflows can automate the integration process if data is described with adequate structured metadata
Homogeneous data integration
Integration of homogeneous or mostly homogeneous data via EML metadata is relatively straightforward
Heterogeneous Data integration
Integration of heterogeneous data requires much more advanced metadata and processing
Attributes must be semantically typed Collection protocols must be known Units and measurement scale must be known Measurement mechanics must be known (i.e. that
Density=Count/Area)
Label data with semantic types Label inputs and outputs of analytical components with semantic types
Use Semantic Mediation System (SMS) to generate transformation steps Beware analytical constraints
Use SMS to discover relevant components Ontology – specification of a conceptualization (a knowledge map)
Semantic typing and ontologies
Data Ontology Workflow Components
Measurement Ontology
Density is part of a larger measurement ontology SEEK’s intent is to create one or more community created
ecological ontologies Creates a controlled vocabulary for ecological metadata More about this in Bertram’s talk
About Kepler
Kepler is the name of the SEEK/SDM additions to the Ptolemy modeling system
Ptolemy was designed by the UC Berkeley EECS department
Primary use is modeling EE circuits Free, opensource, pure Java Flexible design GUI for building
workflows
Kepler
A Kepler model consists of linked “actors” (which correspond to workflow steps)
Timing is controlled by a “director” All actors are written in Java but can
call other applications (such as SAS and MATLAB or native language code via JNI)
Actors can call arbitrary Web (or Grid) Services
Ptolemy already has a very large inventory of actors
Easy to use, drag ‘n drop interface
SEEK Contributions to Kepler (so far)
EML data ingestion actor
Actor design tool
EML data ingestion actor
Ingests any data format described by EML metadata
Converts raw data to Kepler format Data can then be operated on with other
actors Produces one output port for each attribute
in the dataset Individual attributes can then be mapped to
other actors
Ptolemy model with EML ingestion actor
SEEK Contributions to Kepler (so far)
EML data ingestion actor
Actor design tool
Actor design tool
Allows “place-holder” actors to be defined on the fly by non-programmers during workflow creation
Domain scientists can thereby create workflows without programming knowledge
Workflows created with these actors can be executed once their functionality is implemented by a programmer
Allows quick prototyping of workflows by domain scientists
“Place-holder” actors can still be linked to other working actors
Ptolemy and dynamically created actor
How domain scientists will benefit
More fully automated integration systems
A library of pre-defined analytical processes which can be executed on heterogeneous data
Semantic data discovery and processing
Automated unit and measurement scale conversions
A fuller understanding of cross site research implications
Acknowledgements
This material is based upon work supported by:
The National Science Foundation under Grant Numbers 9980154, 9904777, and 0225676 to NCEAS and its collaborators.
The National Center for Ecological Analysis and Synthesis, a Center funded by NSF (Grant Number 0072909), the University of California, and the UC Santa Barbara campus.
Primary Collaborators: University of New Mexico (Long Term Ecological Research Network Office), San Diego Supercomputer Center, University of Kansas (Center for Biodiversity Research)
More info: http://seek.ecoinformatics.org
Questions? IRC: irc.ecoinformatics.org #seek
Recommended