Embed Size (px)
Citation preview
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
1
12:48 AM 12:48 AM
Dynamic Integrations of Crop Data and Corresponding Meteorological Data
based on A Standardized Data Exchange Framework
Seishi Ninomiya, Atsushi Yamakawa, Xinwen Yu
National Agricultural Research Center, National Agriculture and Food Research Organization, Ja
pan
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
2
12:48 AM 12:48 AM
What is Grid?
• Concept and technology to share, integrate and coordinate distributed computer resources– Software and Hardware– Keeping autonomy of distributed resources– Keeping heterogeneity of distributed resources
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
3
12:48 AM 12:48 AM
Users need to obtain one by one, knowing how to access each
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
4
12:48 AM 12:48 AM
e.g. Data Grid provides you
A virtually integrated huge database
We do not need to know where they are, how to use,…
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
5
12:48 AM 12:48 AM
Concept of Grid System
Case Base
Weather Data 2
Farm Management
Meta Database
The Internet
Agterm Dictionary
User who needs DecisionField Data Monitoring
．Growth Model2
Data BrokerWeather Data 1
．．Growth Model1
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
6
12:48 AM 12:48 AM
Dead storage data Issue
• A lot of digital data sets are continuously produced in agricultural experimental stations – Using ordinal software such as spread sheet applications– But they are likely to be kept in local stations and scientist
level• The data sets are isolated and hardly integrated among
different locations– How to ease data publication for merging and sharing for end
users
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
7
12:48 AM 12:48 AM
Different resource integration issue
• Separated crop data are hard to be integrated with different resources, e.g. weather data
• Heterogeneity– e.g. Models constructed using local data are only
applicable locally
• How to integrate crop data with weather data upon user’s request
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
8
12:48 AM 12:48 AM
If merging and sharing are possible
• End users can freely combine separated data sets from different locations and perform analysis on them
• Datamining over the huge amount of data sets becomes real and we can possibly find out unknown facts
• Integration with completely different resources e.g. weather data becomes also possible
• Constructing new model becomes quite easy
• Integrated data help model test and verification.
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
9
12:48 AM 12:48 AM
Objective 1:Crop database • Spread-sheet based heterogeneous data can be easily
integrated into a database to be shared• No skill is required from use sides and just a part of everyday
data management• Uniformity of tables are not required
Internet
DBMS
Application Server
Servlet Container
EJB Container
Application
MetBroker
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
10
12:48 AM 12:48 AM
Crop DB
Data Extraction
Objective 2: Integration of crop data with corresponding meteorological data
MetBroker
Meteorological DB
IntegrationService
Location & Date
Crop Data
Corresponding weather data
XML/Crop data &weather data
Models/Analysis
SOAP/XML
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
11
12:48 AM 12:48 AM
Basic structure of application
Google Map
Client
Bro
wser
Controller
Crop DBCrop Data Service
Model
ServicesWeb application
MetBroker
AMeDASAMeDAS
AMeDAS
View
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
12
12:48 AM 12:48 AM
MetBroker• MetBroker provide consistent access to heterogeneous
weather DBs• Just to develop a wrapper to link to a new DB
Heterogeneous and Autonomous DBs
Meta Data
Rice Growth Prediction
Farm Management
MetBroker
Pesticide Prediction
Heterogeneity is absorbed by brokers (middleware)
B-DB
C-DB
D-DB
A-DB
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
13
12:48 AM 12:48 AM
e.g. Crop model clients of MetBroker
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
14
12:48 AM 12:48 AM
Over 22,000 stations of 25 databases
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
15
12:48 AM 12:48 AM
Coverage of MetBroker
Country Weather Database＃Stations
From
Frq. ID/Passwd
Japan AMeDAS/MAFFIN 1479 1976 Hourly Free
Japan National Meteorological Observatory 150 1989 Hourly Free
Japan Wakayama Prefec. Rainfall DB 137 2001 Hourly Free
Japan Kanagawa Prefec. Agriculture & Forestry Met. DB 14 1998 Hourly Free
Japan Hokkaido Memoro/MAMEDAS 8 2000 Hourly Free
Japan Chiba Prefec. Agric. Exp. Station 3 1986 Hourly Free
Japan Tottori Prefec. Hort. Exp. Station 3 1986 Hourly Free
Japan National Hokkaido Agriculture Research Center 3 1986 Hourly Free
Japan FieldServer Project1 3 2002 10 min Needed
Japan FieldServer Project2 20< 2002 10 min Needed
US/WMO NOAA/WMO Archive 12000< 1994 Daily Free
USA Oregon Integrated Plant Protection Center (NorthWest) 152 1996 Daily Free
USA Long Term Ecological Research Network (ClimDB) 60 1964 Daily Free
USA Georgia Automated Environmental Monitoring Network 46 1997 Daily Free
USA Florida Automated Weather Network 18 1996 15 min Free
USAWashington State University Public Agricultural Weather
System 60 1987 15 min Needed
Norwayu Planteforsk Crop Research Institute 33 1987 Hourly Free
UK Horticulture Research International 2 1919 Daily Needed
NZ HortPlus Ltd 39 1996 Hourly Free
NZ National Climate Database 6547 1853 Hourly Needed
South Africa South African Sugar Association network 13 1997 Daily Free
Korea Seoul National University Plant Disease and Epidemiology Lab 11 1993 Hourly Free
Taiwan Taiwan Ecological Research Network 7 1995 Daily Needed
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
16
12:48 AM 12:48 AM
Temperature, humidity, solar radiation, soil moisture, leaf wetness, CO2,…
High accuracy Camera WIFI based Web server High extensibility Low cost
Field ServerSensor network node
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
17
12:48 AM 12:48 AM
1
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
18
12:48 AM 12:48 AM
Database Broker Service
Data Brokage DB A
Database Driver
DB B
DB C
DB D
Meta DatabaseWhere, How to use
Data contents
Data Request
Search
Standardized Data
Data SummarizationEx) Daily mean from
hourly data
Data acquisition
Data request translated to DB C
Data Standardization
Data Secondary ProcessingClient
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
19
12:48 AM 12:48 AM
Meta Database based with Web ontology
Intelligent BrokerIntelligent Broker
•Decision Support Services•Simulation Models
Inference Engine
Dynamic DB Dynamic DB WrapperWrapper
Item Definition OWLStation metadata
RDF
Metadata database
Meteorological databases
DBDB
DBDB
2. Request3. Request metadata
4. Request data
1. Register
DBDB
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
20
12:48 AM 12:48 AM
Roles Of the RDF/OWL files for weather databases
Name File type Content
Standard vocabulary
OWL •All standard weather items
•Vocabulary to describe weather stations
Item definition OWL Local vocabulary that is used in each database; correspondence to standard vocabulary
Station metadata RDF Description about all the weather stations included in a particular database
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
21
12:48 AM 12:48 AM
A part of Standard Vocabulary OWL
<owl:Class rdf:ID="DailyMaxAirTemperature"> <rdfs:subClassOf rdf:resource="#MaxAirTemperature"/> <rdfs:subClassOf> <owl:Restriction> <owl:allValuesFrom> <owl:Class rdf:about="#DailyMaximum"/> </owl:allValuesFrom> <owl:onProperty> <owl:ObjectProperty rdf:about="#summaryKind"/> </owl:onProperty> </owl:Restriction> </rdfs:subClassOf></owl:Class>
<owl:Class rdf:about="#DailyMaximum"> <rdfs:subClassOf rdf:resource="#Maximum"/> <rdfs:subClassOf> <owl:Restriction> <owl:allValuesFrom rdf:resource="#Daily"/> <owl:onProperty> <owl:ObjectProperty rdf:about="#duration"/> </owl:onProperty> </owl:Restriction> </rdfs:subClassOf></owl:Class>
Sample file:http://www.agmodel.org/MetBroker.owl
“”DailyMaxAirTemperature” is a subclass of “MaxAirTemperature”
“”DailyMaxAirTemperature” is translated as daily maximum data
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
22
12:48 AM 12:48 AM
Sample of Item Definition OWL of a DB
<met:DailyMaxAirTemperature rdf:ID="ame_day.temp_max"><met:summaryKind rdf:resource="http://www.agmodel.org/MetBroker.owl#DailyMaximumOfSampleEvery10Minutes"/></met:DailyMaxAirTemperature>
<met:HourlySampleAirTemperature rdf:ID="ame_time.temperature"><met:summaryKind rdf:resource="http://www.agmodel.org/MetBroker.owl#SampleOnTheHour"/></met:HourlySampleAirTemperature>
A sample file is available on http://www.agmodel.org/Aclima.owl
Local item name
“ame_day.temp_max” is translated as daily maximum data based on ev
ery 10 minute data
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
23
12:48 AM 12:48 AM
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
24
12:48 AM 12:48 AM
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
25
12:48 AM 12:48 AM
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
26
12:48 AM 12:48 AM
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
27
12:48 AM 12:48 AM
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
28
12:48 AM 12:48 AM
System components
Java Runtime Environment 1.5.04
PostgreSQL7.4
JRE supported OS
JBoss-4.0.3
EJB3.0(DBMS
abstraction)
Struts1.2(Web
Interface)
・ IE, Firefox, etc.・Excel2002, newer
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
29
12:48 AM 12:48 AM
Data standard and transformation
• Data standard was defined using XML schema based on 3 commonly used data formats
• Data in Excel format can be transformed to data objects using XSLT and data schema
• Data objects can be verified based on data schema
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
30
12:48 AM 12:48 AM
Crop data upload and integration
Crop Data ServiceEJB3
Source XML
Crop Data
HistoryData transforming
Data validating
XSLT style sheet
Data Schema
a
b
c
Crop Database
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
31
12:48 AM 12:48 AM
Main menu
• Very simple menu structure• Simple and clean user interface
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
32
12:48 AM 12:48 AM
Data upload
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
33
12:48 AM 12:48 AM
Upload history
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
34
12:48 AM 12:48 AM
Web ApplicationCrop Data
Service EJB3
Crop db
Specifying query conditions then executing data query.
Browsing and/or download queried crop data
Data Query
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
35
12:48 AM 12:48 AM
The mechanism of data integration
Location Table
Longitude, latitude
CropDataService
Data query
Location Time duration
Retrieved crop data
MetBroker
Weather Items
Data query Weather data
Data Integration
Other properties …
Weather stations
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
36
12:48 AM 12:48 AM
AMeDAS
Web Application
Crop Data
Location Table
Crop Data Service EJB3
MetBroker
AMeDASAMeDAS
Integrating crop data and weather data
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
37
12:48 AM 12:48 AM
Weather station selection• Possible candidates of the weather stations
are shown based on location and period of data collection
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
38
12:48 AM 12:48 AM
Location registration
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
39
12:48 AM 12:48 AM
Conclusions
• Crop database– 37 spreadsheet files uploaded and merged– 151072 records– 6486 varieties and lines– 184 properties– 256 different experiment locations– 1980-2004
• Crop data were successfully integrated with corresponding weather data
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
40
12:48 AM 12:48 AM
Conclusions
• Highly efficient utilization of resources• No duplication of databases and programs• Easy updates of databases and programs• Reduction of development and maintenance
cost• More powerful DSS by dynamic and flexible
integration of distributed resources
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
41
12:48 AM 12:48 AM
Thank you very much
http://www.agmodel.org/
http://www.agmodel.org/vocabulary/200602/MetBroker.owl
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
42
12:48 AM 12:48 AM
• Thank you for your attention
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
43
12:48 AM 12:48 AM
Seamless Integration of Field Server with Legacy Databases through MetBroker
気象 DB
気象 DB
気象 DB
気象 DB
FieldServerDB
アプリケーション
アプリケーション
アプリケーションMetBroker
気象 DB
気象 DB
気象 DB
気象 DB
FieldServerDB
アプリケーション
アプリケーション
アプリケーションMetBroker
気象 DB
気象 DB
気象 DB
気象 DB
FieldServerDB
アプリケーション
アプリケーション
アプリケーションMetBroker
W DB
気象 DB
気象 DB
気象 DB
アプリケーション
アプリケーション
アプリケーションMetBrokerMetBroker
Weather DB
Weather DB
FS Weather DB
Client APP
Client APP
Client APP
Weather DB
Station Conf. XML
Weather Data XML
FS Data Archive
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
44
12:48 AM 12:48 AM
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
45
12:48 AM 12:48 AM
Brokers Provided as Web Services
ChizuBrokerMetBroker
DEMBroker
WebService-SOAP/XML
Client ClientClient
WebService-SOAP/XML
WebService-SOAP/XML
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
46
12:48 AM 12:48 AM
New Web Service by Combining Existing Services
WeatherDB
WeatherDB
WeatherDB
Client Program
Client Program
Client Program
DEM DB
DEM DB
DEM
Broker
InterpolationSOAP
Internet
Internet
Internet
Internet
Met-Broker
Interpolation Algorithms
SOAPOver HTTP
• Weather Data Interpolation Service by MetBroker & DEMBroker
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
47
12:48 AM 12:48 AM
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
48
12:48 AM 12:48 AM
Potential for Data Sharing Between DSS
Weather
Data
Crop details
Soils Topography
Variety selection O O O
To dam? O O O
Land use O O O O
Spray for disease O O O O
Irrigation or not O O O
Data NeededDecisions(Clients)
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
49
12:48 AM 12:48 AM
Concept of Agri-Grid System
Case Base
Weather Data 2
Farm Management
Meta Database
The Internet
Agterm Dictionary
User who needs DecisionField Data Monitoring
．Growth Model2
Data BrokerWeather Data 1
．．Growth Model1
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
50
12:48 AM 12:48 AM
Standardized Interface to Link Databases and Models
• XML based data exchange • All resources available as Web services• Easy client development• Cost reduction• Easy integrations of services to create
new services
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
51
12:48 AM 12:48 AM
Enquiry
Answer
言語と文字
Info-Broker
Multi-Lingual- Ontology- Translation system- Dictionary/Thesaurus
User
Anywhere
Thank you for your attention
http://www.agmodel.net/
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
52
12:48 AM 12:48 AM
With Web-based data sharing & integration
• Once experimental data in spreadsheet file (e.g. MS Excel) uploaded, these data will be automatically merged into database, and become sharable over the Internet among different locations
• Crop data can be integrated with weather data– No skill is required– Just a part of everyday data management– Uniformity of tables is not required– Low cost
AFI
TA
200
6/B
an
galo
reA
FITA
20
06
/Bang
alo
re
53
12:48 AM 12:48 AM
• Data integration is realized in our brokerage based Data Grid– MetBroker is the core of the Data Grid
• The application demonstrates the power of Data Grid• Currently only open to authorized users• An open source project will be started based on this system.• This approach can be used to all other crop data sharing and
integration• Crop modelers are using this application to develop robust crop
models
