25
LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Embed Size (px)

Citation preview

Page 1: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA-DIS Working Group Report

LBA Science Steering Committee Meeting

-- LBA-DIS Report --

Ubatuba, SP

May 13-15, 2004

Page 2: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA DIS WG Topics

• Data Registration / Archive Status

• Recommendations from 14th SSC Meeting, Fortaleza

• Interactions with other Projects

• Long-term Archive-Related Activities

• System Update

• Software Development Plan

• LBA DIS Institutional Support

• LBA DIS Composition

Page 3: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA Data Registration Status

Page 4: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA Overall Status:Metadata Registered in Beija-flor

260

40

0

62

8

118

63

133

83

152

80

231

92

325

100

379

207

413

211

418

211

469

215

0

50

100

150

200

250

300

350

400

450

500

06/99 01/00 06/00 10/00 02/01 05/01 11/01 05/02 11/02 05/03 11/03 05/04

# of data sets registered

# of posters registered

N=684

Increased 8.9%

Page 5: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA Overall Status:Data Volume Archived at LBA-DIS / CPTEC

4.00.50.6

14.88.7

1.1

69.0

8.91.2

79.6

9.21.3

104.8

9.31.3

0

20

40

60

80

100

120

Gig

ab

yte

s

11/01 11/02 05/03 11/03 05/04

Unrestricted DataRestricted DataPosters

28.1% increase includes:

CLAIRE DataEUSTACH Flux DataORNL Regional Data

Hyperion DataNepstad Risque Data

SPOT NDVI DataMelack Wetlands Data

Page 6: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA Component Status

122

45

23

50

10

3

0

0

341

22

12

0 100 200 300 400 500 600

US-BR

EU-BR

BR

Data Sets with data available 341 22 12

Data Sets w/ broken links ? 0 0

Data Sets w/o data 50 10 3

Posters 122 45 23

US-BR EU-BR BR

xx

Con

N=38 (9 of 17 Phase 1 teams contributing)

N=77 (13 of 16 Phase 1 teams contributing)

N=513 (58 of 60 Phase 1 teams contributing)

*

* Increase of 12 data sets (LBA Ra-CCI)** Decrease of 10 broken links (Data now at CPTEC)

**

Page 7: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

SSC Recommendationsfrom 14th SSC Meeting, Fortaleza

• Improve data registration/archive performance record for European and Brazilian science teams

• Report on data archived by LBA DIS but not registered on Beija-Flor

• On-line tutorial for Beija-Flor / Metadata Editor

• Each LBA team should name a primary and secondary data contact

• Make LBA tower flux data available to Flux networks: AmeriFlux & FluxNet (12th SSC)

Page 8: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Data Registration/Archive Performance of EU and BR teams

• Broken links on all of the LBA data on Alterra Web Site (~10 data sets)

• LBA tower flux data – (FluxNet & Ameriflux)

• Plan for next 6 months:

• Contact each of the non-contributing Brazilian and European/BR teams to acquire data

• Complete analysis of data in archive but not registered -- theory is that much of the “missing” EU/BR & BR data is archived but not registered

Page 9: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Data Archived but Not Registered

Huge task!!

1. Develop a method to cross-reference [hundreds of] data directories in archive with entries in Beija-flor

2. Record these findings in a “data tracking system”

• Link the B-f registered data sets with corresponding archive directories

3. Identify the archive directories that aren’t represented in Beija-flor – who? what? when? where? …

4. Register the “found” data

Page 10: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

On-line Tutorial and Help Files • LBA Metadata Editor (LME):

• FAQs updated

• Glossary updated

• Tutorial updated and improved

• Beija-flor

• Tutorial updated

• Beija-flor User’s Guide

• Contains all of the above [updated] information

• Available for download as .pdf (in Portuguese & English) on Beija-flor site

Page 11: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Data Contact for each Science Team

• Data Contact for Brazilian-led teams will be PI

• LBA-ECO requested in Annual Reports for Phase 2 teams that Data Contact be designated

• 25 teams complied; PI will be default Data Contact if not specified

• LBA-ECO Phase 1 Teams: PI is the default Data Contact

Page 12: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Interactions with Other Projects

• CEOP (<= LBA) Flux, Surface met, Soil data

• HIBAM (=> LBA) Hydrology data

• FluxNet / Ameriflux networks (<=LBA)

• CODATA Workshop, Campinas (March ’04)

• SIVAM < = > LBA, Manaus, AM (April ’04)

• TEAM Workshop, Belém-PA (March ’04)

• PPBio Infrastructure Plan, Belém-PA (April ’04)

• GEOMA Seminar, Ilhabela, SP (May ’04)

Page 13: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

PantanalReserva Jaru

RondoniaFazenda NS Aparecida

Manaus-K34 Flona-Santarem

Brasilia-Cerrado

Fig. 1. LBA-CEOP Reference sites

Caxiuana

LBA is a CSE (Continental Scale Experiment) for GEWEXCEOP (Coordinated Enhanced Observation Period) Phase 3 in progressOther CSEs: BALTEX, GAPP, etc.

NOT operational

Data N/A

Data N/A

Page 14: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

CEOP 3 Phase I

Study Area Data Contact

Data Coverage (Oct01,2002 to Mar31,2003)

Registered In Beija-flor?

Tower still operational?

Surface Metereological and Radiation Data Archived

in LBA-DIS

Flux Data Archived in

LBA-DIS

Soil temp and

moisture Data

Archived in LBA-DIS

Meteorological Tower Data Archived in

LBA-DIS Brasilia-Cerrado Heloisa S. Miranda n n n n n

Caxiuana Julia Cohen y y y y y

Flona-SantaremHumberto Rocha / Scott Miller y y y y y

Manaus-K34 Hermes Braga Xavier y y n n yPantanal Carlos Cesar de Oliveira n y n n yRondonia Fazenda NS Aparecida Fernando Luiz Cardoso y y y y y

CEOP 3 Phase II

Study Area Data Contact

Data Coverage (Apr01,2003 to Sep30,2003)

Registered In Beija-flor?

Tower still operational?

Surface Metereological and Radiation Data Archived

in LBA-DIS

Flux Data Archived in

LBA-DIS

Soil temp and

moisture Data

Archived in LBA-DIS

Meteorological Tower Data Archived in

LBA-DIS Brasilia-Cerrado n n n n n

Caxiuana y n n n nFlona-Santarem y n n n nManaus-K34 y n n n nPantanal n n n n nRondonia Fazenda NS Aparecida y n n n n

Page 15: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

HIBAM Data Status

• Metadata has been registered in Beija-flor– Data currently accessible from relational database on 3

servers• ANA – Agencia Nacional de Aguas - Brasilia

• IRD – Institute de Recherche por le Development – France

• Golden-Teledetection - France

• Data being copied to LBA DIS / CPTEC for distribution from there as well

• Waiting for authority to distribute from CPTEC

Page 16: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Flux Data StatusLBA Investigator/Site

In

Beija-flor?

Data Available

Data Archived

in LBA-DIS

In AmeriFlux

In

FluxNet

Wofsy/Tapajos Yes Yes Yes Yes --

Goulden/Tapajos Yes Yes Yes Yes --

Fitzjarrald/Santarem Yes Yes, but restricted

Yes No --

Manzi / Rondonia Yes Yes Yes (Agreed)* --

A. Nobre / Manaus Yes Yes Yes -- (Agreed)*

Malhi / Braganca Mangrove characterization but no flux data

Char. data but no flux data

Char. Data only (still no

flux data)

-- Yes

Da Rocha?

Others?

* Requirements TBDColor indicates progress since 11/03

Page 17: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Long-term Archive-related Activities

• LBA-ECO is attempting to link pubs to data sets

• LBA-ECO Data Archive-Related Processes, Roles, & Responsibilities Document

• LBA-ECO Documentation Standards / Doc Tool Status

• Implications for LBA DIS / CPTEC archive

• Documentation standards, completeness?, e.g. fields labeled properly, units of measurement identified?

• Data quality assessment? e.g. calibrations, algorithms used?

• Should we try to differentiate wide range of data/documentation quality within archived data collection? e.g. a rating or “score”?

Page 18: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Submit Data Packages to ORNL for Archive

Work w/ PI to Prepare Data Package

Submit to Archive Panel for Evaluation

Send Data Package “as is” to LBA DIS for archive

ORNL ingests data package into DAAC

NASA ORNL DAAC

Archive

Send Data Package to LBA DIS for archive

Review Metadata

LBA DISArchive

Send data to LBA DIS

Metadata

Register Metadata

Raw Data

Update

Mark as “Ready for Archive”

Preliminary

Data

Final Data

Data SetApproved to be Archived?

NoYes

Send data to LBA DIS

Legend -- RolesPI / Team’s Data ContactData CoordinatorArchive Review PanelLBA DISORNL

LBA-ECO Data Archive Process: Overview

Ancillary FilesData Set

Doc DataMetadata

File

24-February-2004

Page 19: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Legend -- RolesPI / Team’s Data ContactData CoordinatorArchive Review PanelLBA DISORNL

LBA Data Archive Process:Data Archived at LBA DIS / CPTEC

LBA-ECO Data (Preliminary & Final)-- Not Permanently Archived at ORNL

Restricted Data – All Sources

Available to the public

Restricted Access

LBA-ECO Final Documented Data-- Also Permanently Archived at ORNL

European & South American Teams’ Data-- Preliminary… Final Data

Historic Data / Pre-LBA Data

LBA DIS Archive

24-February-2004

Page 20: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

System Updates

• Status of new LBA Home Page

• LBA server almost out of space for new data – Additional disk space requested in 9/03– Purchased and should arrive soon

• New database for Palm users created: contacts, groups, sites

Page 21: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA-Server Hard DrivesWHY are they so expensive?

• Industrial STRENGTH• Faster seek time: 0.55 ms • Faster access time• Faster Transfer Rate : 320 MB/s • Faster Rotational Speed: 10,000 rpm • Faster interface: Ultra320 SCSI • Raid-5 (fault tolerance: ability of a

system to respond gracefully to an unexpected hardware failure)

• About US$ 300 / 36.9 Giga bytes ($550 if bought in Brazil)

LBA-DIS server array of SCSI drives

Page 22: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Perspectives for post-December 2005

• LBA DIS will receive data long after field work is over (as LBA-ECO data is archived at the DAAC)

• LBA-ECO will use Beija-flor / LME after field work is over and data are QA’d & documented, i.e. until data archive is complete

• LBA-ECO computer systems will be migrated to LBA • LBA will plan to adopt freeware platform for data

management in the long-term• Long-term LBA data management issues will be

addressed & post-LBA plans implemented

Page 23: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

Software Development/Migration Plan

• Post-LBA non-proprietary Beija-Flor replacement:LBA DIS will propose a plan to implement an enhanced Beija-Flor system and LME based on freeware platform for long-term data/metadata management after LBA data

• LBA-ECO STI system to be migrated to LBA Central Office (beginning Aug-Sept/2004)

Page 24: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA DIS Institutional Support

• LBA Publication Management – INPA’s Central Library on the Web (Done)• Database configuration/migration at INPA – Client/Server Platforms (Done)• LBA Offices Manaus, Santarém, & CPTEC - Systems and computer infrastructure documentation (Done)• Regional Offices – Systems requirements and plans for computer and network enhancements (Done)• 1st LBA Informatics Week at Santarém – June’04

Page 25: LBA-DIS Working Group Report LBA Science Steering Committee Meeting -- LBA-DIS Report -- Ubatuba, SP May 13-15, 2004

LBA DIS Distribution of Tasks

• LBA Central Office– Responsible for computing infrastructure for DIS– Technical training for staff

• LBA DIS / CPTEC– Data/Metadata archive– Interactions w/ LBA & LBA affiliate science teams re: data– Coordination of updates for Beija-flor & LME

• LBA-ECO Project Office– Coordination with:

• LBA DIS at Central Office & CPTEC • LBA-ECO science team • ORNL DAAC