Data Grid Research Group Dept. of Computer Science and Engineering The Ohio State University...

Data Grid Research GroupDept. of Computer Science and EngineeringThe Ohio State UniversityColumbus, Ohio 43210, USA

David Chiu & Gagan Agrawal

Enabling Ad Hoc Queries over Low-Level Scientific

Data Sets

2D. Chiu & G. Agrawal. Enabling Ad Hoc Queries over Low-Level Scientific Data Sets

SSDBM ’09

Presentation Outline

• Motivation‣ Current Trends in Scientific Data Management‣ Problem Discussion

• Data Registration Indexing‣ Metadata Extraction‣ Transformation

• Service Composition

• Conclusion

SSDBM ’09

Increased tremendously over the years

Scientific Data Sets

• The collection of scientific data has increased over the years with new instruments, simulations, etc.

• Data sets are stored in repositories around the globe

• Just within U.S. entities in the geospatial domain‣ NOAA: oceanic, climate, water

quality, ...‣ NASA: ozone, air quality, tropical, ...‣ NRCS: land quality, watershed, ...

SSDBM ’09

Data Repositories

Web or Data Grid InfrastructureMass StorageSystems (MSS)

SSDBM ’09

• Data sets are typically low level, i.e., ‣ Unstructured or semi-structured0101071895 0.34 -2.45 0.50 -0.65 -0.62 -0.71 0.00 -0.96 0101071896 -1.71 0.49 0.27 -0.79 -1.53 0.60 0.09 -2.210101071897 -0.53 0.14 4.32 1.95 -1.55 -1.68 -1.32 -0.690101071898 1.90 -2.64 -1.70 1.11 -2.18 -1.08 -0.53 -0.250101071899 0.44 0.97 1.65 -0.71 -2.02 -2.10 -0.50 -2.030101071900 -1.65 1.19 -1.34 0.57 -1.37 7.00 -0.48 -1.77 . . .

• However, data is well-documented‣ Accompanying XML-based metadata describing data sets is

typically required in today’s repositories

SSDBM ’09

Data Repositories

Mass StorageSystems (MSS)

Grid/Web Services & portals

Web or Data Grid Infrastructure

SSDBM ’09

Data Repositories in the Global Scale

AU ...

D. Chiu & G. Agrawal. Enabling Ad Hoc Queries over Low-Level Scientific Data Sets

SSDBM ’09

What Do the Users Want?

I don’t care where data is located.

I also want to share my own data with others!

Don’t just give me the data, but...

- Transform it - Manipulate it - Compose it with other processes and data sets

And do this with the least amount of work required from me!

SSDBM ’09

System Goals

• To enable queries over low level data sets, which involves:‣ identification of relevant data sets‣ automatic planning for the composition of dependent

services (processes) for derivation

• ... while being non-intrusive to existing schemes, i.e.,‣ avoids a standardized format for storing data sets‣ accommodates heterogeneous metadata‣ this system should - fit - into existing MSS and scientific

computing infrastructures (Data Grid & the Web)

SSDBM ’09

That’s good and all, but...

Challenges

• Not without challenges...‣ dealing with metadata from multiple entities‣ efficiently identifying relevant data sets‣ planning and executing accurate service compositions on

the spot

SSDBM ’09

That’s good and all, but...

Challenges

• Not without challenges...‣ dealing with metadata from multiple entities‣ efficiently identifying relevant data sets‣ planning and executing accurate service compositions on

the spot

DOMAIN KNOWLEDGE & SEMANTICS

• And without question, the need for

SSDBM ’09

The AUSPICE System

AUSPICE: Automatic Service Planning and Execution in Cloud/Grid Environments

SSDBM ’09

The Semantics Layer

A Need for Domain Level Knowledge

• Assume the following service retrieves a satellite image pertaining to (x,y) with resolution respective to r

• Questions to ask the system:‣ How to deduce that this service can be used?‣ How to determine what information is needed for input?‣ Did the user provide enough information to invoke this service?

get_sat_image(double x, double y, double r)

inputsTo inputsToinputsTo

longitude latitude grid_size

outputsTo

satellite image

SSDBM ’09

In the Semantics Layer

Applying Domain Information

Domain concepts can be derivedfrom executing a service

Domain concepts can also be derived from retrieving an

existing data setService parameters representdifferent domain concepts

SSDBM ’09

Data Registration Service

Indexing Data Sets

• Handling heterogeneous metadata

• For instance, just within the geospatial domain,

Country Metadata Standards

US CSDGM

AU, NZ ANZLIC

EU ???

CDN ???

... ...

SSDBM ’09

Indexing Data Sets

• Handling heterogeneous metadata

SSDBM ’09

Indexing Data Sets

• Metadata Transformation

(transform to spatial index)

SSDBM ’09

Indexing Data Sets

• Metadata to DB transformations

insert

SSDBM ’09

Indexing Data Sets

SSDBM ’09

Indexing Data Sets

SSDBM ’09

Indexing Data Sets

SSDBM ’09

In the Semantics Layer

Applying Domain Information

Data registration simplifies identification process within

SSDBM ’09

Indexing Services

• Services (inputs, outputs) are also registered in much the same way

SSDBM ’09

The Planning Layer

Service Composition: An Example

A subset of the ontology (unrolled)

SSDBM ’09

The Planning Layer

Service Composition

begin compSrvc(concept, Q[...])W := ()

//perform DFS starting from conceptlet v := concept be the currently visited node

if v is a data type then W := (W, index.getData(v, Q))

else //v is a servicelet (p1,..,pn) be v’s params

//recursive call on each piW := (W, (v, compSrvc(p1, Q), ... , compSrvc(pn, Q)))

end if

return Wend

SSDBM ’09

The Planning Layer

Service Composition: An Example

Ontology (unrolled)

A Derived Execution Plan This is what data registration provides

SSDBM ’09

Planning Times

SSDBM ’09

Conclusion

• The AUSPICE System...‣ unifies heterogeneous metadata‣ extracts certain metadata attributes and indexes low level

data sets and services for fast access from distributed repositories

‣ automatically composes these services and data sets to answer user queries

• Questions - Comments?‣ David Chiu chiud@cse.ohio-state.edu‣ Gagan Agrawal agrawal@cse.ohio-state.edu

Data Grid Research Group Dept. of Computer Science and Engineering The Ohio State University...

Documents

Date: October 5, 2018 Location Senate Chamber – Ohio UnionDate: October 5, 2018 Location Senate Chamber – Ohio Union Ohio Union Rm 2088A | 1739 N. High Street | Columbus, OH 43210

CURRICULUM VITAE GUIDECURRICULUM VITAE May 2020 Natasha K. Bowen, MSW, Ph.D. Professor, College of Social Work Ohio State University 1947 College Rd. Columbus, Ohio 43210 Office phone:

REPRODUCTION AND HORMONES John D. Harder Department of Evolution, Ecology, and Organismal Biology The Ohio State University Columbus, Ohio 43210 USA

A Tool for Supporting Integration Across Multiple Flat-File Datasets Xuan Zhang, Gagan Agrawal Ohio State University

OHIO STATE ATHLETICS COMMUNICATIONS€¦ · OHIO STATE ATHLETICS COMMUNICATIONS Fawcett Center, 6th Floor | 2400 Olentangy River Rd. | Columbus, Ohio 43210 JANUARY 5 at Georgia 4

Physicscms/dcc/stx.pdf174 WEST 18TH AVE COLUMBUS OHIO 43210 THE OHIO STATE UNIVERSITY ELECTRONICS LAB PHYSICS DEPARTMENT mojo FIFO_R_ENB FIFO_R_CLK On FIFO FIFO_R_CLK FIFO_R_ENB …

Is the West Antarctic Ice Sheet Disintegrating?heimbach/papers_glaciology/jgr_hughes_1973_… · T. HUGHES Institute o] Polar Studies, The Ohio State University, Columbus, Ohio 43210

Demographics Report 2015-2016 - Ohio State University2088 Ohio Union 1739 North High Street Columbus, OH 43210 Phone: 614-292-2101 E-Mail: usg@osu.edu Web: usg.osu.edu Demographics

registrar.osu.edu Kay Lenz, Toledo Mark F. Massmann, ... Upper Sandusky Elnora Marie Upshaw, ... Columbus, Ohio 43210

MS, PhD, CChem, FRSC - Laub BioChem · ms, phd, cchem, frsc formerly department of chemistry the ohio state university columbus, ohio 43210 and department of chemistry the molecular

Light-Weight Data Management Solutions for Scientific Datasets Gagan Agrawal, Yu Su Ohio State Jonathan Woodring, LANL

Osvaldo H. Campanella Department of Food Science and … · 2019. 8. 16. · Osvaldo H. Campanella Department of Food Science and Technology The Ohio State University, Ohio 43210

Nuclear Physics at Ohio Statentg/NucPhys_updated2011_full.pdfNuclear Physics at Ohio State Department of Physics The Ohio State University 191 West Woodruﬀ Avenue Columbus, OH 43210

Department of Physics, The Ohio State University, Columbus ...Department of Physics, The Ohio State University, Columbus, OH 43210, USA Jun Jiangx School of Physics, Shandong University,

The Ohio State University Veterinary Medical Center … Ohio State University Veterinary Medical Center Hospital for Companion Animals 601 Vernon L. Tharp St. Columbus, OH 43210-1089

The Ohio State University School of Health & Rehabilitation Sciences 453 West 10th Avenue Columbus, Ohio 43210 Appointment Line: 614-292-1706

2012users.math.uoc.gr/~frantzikinakis/Ergodic2017/... · Combinatorial and Diophantine Applications of Ergodic Theory VITALY BERGELSON The Ohio State University Columbus, OH 43210,

Efficient Evaluation of XQuery over Streaming Data Xiaogang Li Gagan Agrawal The Ohio State University

43210 1983. - Ohio State University

Supporting High- Performance Data Processing on Flat-Files Xuan Zhang Gagan Agrawal Ohio State University