Data to Support Ocean-Atmosphere Research NCAR Research Data Archive (RDA), Zaihua Ji, NCAR Steven Worley, NCAR Scott Woodruff,

Data to Support Ocean-Atmosphere Data to Support Ocean-Atmosphere ResearchResearch

NCAR Research Data Archive (RDA),NCAR Research Data Archive (RDA),http://www.dss.ucar.eduhttp://www.dss.ucar.edu

Zaihua Ji, NCARZaihua Ji, NCARSteven Worley, NCARSteven Worley, NCAR

Scott Woodruff, NOAA/ESRLScott Woodruff, NOAA/ESRL

Fourth International Ocean-Atmosphere ConferenceFourth International Ocean-Atmosphere ConferenceJuly 4 - 6, 2007, Qingdao, ChinaJuly 4 - 6, 2007, Qingdao, China

2

Presentation OutlinePresentation Outline

• IntroductionIntroduction• OverviewOverview

• Focus, Data Relevant to Ocean-Focus, Data Relevant to Ocean-atmosphere Researchatmosphere Research

• RDA Data Archive and ManagementRDA Data Archive and Management• RDA Data Usage StatisticsRDA Data Usage Statistics• Explore Data Archived in RDAExplore Data Archived in RDA• Future DevelopmentsFuture Developments• SummarySummary

3

IntroductionIntroduction

Definition, Research Data Archive (RDA)Definition, Research Data Archive (RDA)• Dataset collection to support atmospheric and

related geosciences research• Hosted by the Computational and Information

Systems Laboratory (CISL) at National Center For Atmospheric Research (NCAR)

• Built with data curation and stewardship over more than 40 years

• Presently maintained by eight people, educated in meteorology and oceanography

• Basic Scale metrics• + 500 datasets+ 500 datasets• + 800K files+ 800K files• + 120 TB+ 120 TB

4

IntroductionIntroduction

Why RDA Access is Unrestricted Why RDA Access is Unrestricted • Our user community is best served if no restrictions existsOur user community is best served if no restrictions exists• Research needs a very large array of data typesResearch needs a very large array of data types

• Dataset development requires experts - no one can do it allDataset development requires experts - no one can do it all• Many studies require global data - everyone can make a contribution Many studies require global data - everyone can make a contribution

and everyone should benefit.and everyone should benefit.

• Data Management alone has significant associated costs (people, Data Management alone has significant associated costs (people, storage, servers, etc)storage, servers, etc)

• There are near zero funds to purchase dataThere are near zero funds to purchase data

• If service systems are well designed many people can be If service systems are well designed many people can be economically servedeconomically served

• ForemostForemost• We know data sharing works well and we want to be a “best practices” We know data sharing works well and we want to be a “best practices”

example in the international research communityexample in the international research community• We have had successful data collaborations and exchanges with

China• Focused on building and improving ICOADS

5

Overview - Data Relevant to Ocean-Atmosphere Overview - Data Relevant to Ocean-Atmosphere ResearchResearch

Marine observationsMarine observations• International Comprehensive Ocean-Atmosphere Data Set International Comprehensive Ocean-Atmosphere Data Set

(ICOADS)(ICOADS)• Marine surface observations from ships, moored and drifting Marine surface observations from ships, moored and drifting

buoys, and fixed platformsbuoys, and fixed platforms• Very long time series, 1794-2005, over 200 million obs.Very long time series, 1794-2005, over 200 million obs.• Merging and QA/QC of many different datasetsMerging and QA/QC of many different datasets• 20 + year collaborative project between NOAA and NCAR, now 20 + year collaborative project between NOAA and NCAR, now

receiving more international collaborationreceiving more international collaboration• More assistance is welcomedMore assistance is welcomed

• RDA provides two product linesRDA provides two product lines• ObservationsObservations• Monthly summary statistics in 1x1 and 2x2 lat. lon. boxesMonthly summary statistics in 1x1 and 2x2 lat. lon. boxes

• ICOADS is a source for higher level analyzed products, e.g.ICOADS is a source for higher level analyzed products, e.g.• SST analysesSST analyses• Global atmospheric reanalysesGlobal atmospheric reanalyses

6


•Example of ICOADS and productsSST: Dec 1942 ICOADS and two analyses HadISST from UKMO ERSST from NOAA

ICOADS

ERSSTHadISST

7


ICOADS monthly summary statistics

• 10 statistics, e.g. median, mean, nobs…

• 23 observed and derived parameters

• no bias corrections

All source observations are available in a similar way - derive your own statistics

8


WODWOD• Approximately, 1900 and Approximately, 1900 and

forwardforward

• Ocean profile Ocean profile observations from many observations from many international sourcesinternational sources

• Many types (XBT, CTD, Many types (XBT, CTD, ARGO, bottle stations, ARGO, bottle stations, etc)etc)

• Quality checked and on Quality checked and on observed and standard observed and standard depth levelsdepth levels

World Ocean Database (WOD) and World Ocean Atlas (WOA)

Prepared by Ocean Climate Laboratory NODC, Sydney Levitus and collaborators• First edition 1982, now at 2005• Worlds largest collection

Profiling float data added WOD01 WOD05

9


WOAWOA• analyzed data on analyzed data on

standard levels, 1 standard levels, 1 degree resolutiondegree resolution

• climatology, seasonal, climatology, seasonal, monthly products (T, S, monthly products (T, S, O2, nutrients, etc..)O2, nutrients, etc..)

• Some special productsSome special products• annual heat contentannual heat content• mix-layer depthmix-layer depth• high resolution (.25 high resolution (.25

degree) T and S degree) T and S climatologyclimatology

10


Marine Analyses - SST

NOAA Extended Reconstructed SST AnalysisNOAA Extended Reconstructed SST Analysis– 1854 - ongoing, monthly, 2°x 2°, updated monthly – ICOADS and real-time obs.

Hadley Centre HadISST–1850 - ongoing , monthly, 1°x 1°, updated monthly– ICOADS, real-time obs., AVHRR sat. blend for modern period

NOAA Optimum Interpolation Daily SST AnalysisNOAA Optimum Interpolation Daily SST Analysis–1/4 1/4 °x1/4 °, 1985 - ongoing, two products, updated monthly

ICOADS, real-time obs, AVHRR satelliteICOADS, real-time obs, AVHRR satellite ICOADS, real-time obs., AVHRR and AMSR satelliteICOADS, real-time obs., AVHRR and AMSR satellite

NOAA OptimumNOAA Optimum IInterpolation SST Analysisnterpolation SST Analysis– 1981- ongoing , weekly, monthly, 1°x 1°, updated monthly– ICOADS, real-time obs., AVHRR sat. blend

(May ‘07 in background)

11


Marine Analyses - WindMarine Analyses - Wind

Global QSCAT/NCEP Blended Analysis

• Created by Colorado Research Associates

• 0.5° x 0.5 °, 4x daily, 1999 - ongoing

• Vector wind components, and curl

• New versions are prepared when QSCAT algorithms are improved. Currently, a new version is under development.

Many other good sources from Atmospheric Reanalysis datasets

• To be presented later

12


Marine Analyses - WavesMarine Analyses - Waves

Global and North Atlantic Atlas of Ocean Waves

• Created by Sergey Gulev

• 2° x 2 ° and 5° x 5° , monthly, 1958 - 1997

• Significant wave height

• Derived from ICOADS visual wave observations

ERA-40

• 1.5° x 1.5 °, 4x daily, 1957 - 2002

• 2-D wave spectra are available for 12 directions and 25 frequencies

13


Marine Analyses - FluxesMarine Analyses - Fluxes

Global Objectively Analyzed Air-Sea Fluxes (OAFlux)

• Created by Yu and Weller

• 1° x 1°, daily and monthly

• Latent and sensible flux, 1981-2002, net short and long wave radiation 1983-2002

• Derived from in situ observations, satellite data, and NWP and reanalysis output

• Update is planned for 2003 and onwardOcean-atmosphere heat, momentum and freshwater flux

• Created by Josey et al.

• 1° x 1°, climatology

• Derived from in situ observations and adjusted to match ocean heat transport estimates

14


Marine Analyses - Atmospheric ReanalysesMarine Analyses - Atmospheric Reanalyses

Sources

• Many useful surface products at different resolutions. Too many to cover here

• Main dataset title and temporal coverage

• NCEP/NCAR Reanalysis, 1948 - ongoing

• NCEP/DOE Reanalysis, 1979 - ongoing

• ERA-40, 1957-2002

• JRA-25, forthcoming from JMA, 1979 - ongoing

• MERRA, forthcoming from NASA, 1979 - ongoingImportant note about ERA-40

• By agreement RDA can distribute only in U.S., and to a few universities in Canada.

• Copy outside this domain is prohibited

15


Marine Analyses - Ocean ReanalysesMarine Analyses - Ocean Reanalyses

NCEP Global Ocean Data Assimilation System (GODAS)

• 1° x 1°, pentad and monthly, 1980 - ongoing• 1/3 ° resolution ± 10° of equator

• 40 vertical levels, parameters (T, S, ocean current, etc)

GODAS Annual mix layer depth

16


NWP Operational OutputNWP Operational Output

NCEP Global Final Analysis (FNL) • 1° x 1°, 4x daily, 1999 - ongoing • Updated monthly

ECMWF Global Advanced Operational Surface Analysis• 1.125 ° x 1.125°, 4x daily, 1985 - ongoing• Updated bi-annually

TIGGE (THORPEX Interactive Global Grand Ensemble)

• Analyses and forecasts from international NWP centers• Now CMA, ECMWF, NCEP, UKMO and JMA; will add 5

more centers

• native model resolutions (approx. 1°), 1x to 4x daily, all ensemble members

• Parameters; 10m winds, temperatures, surface radiation and fluxes, etc,

• Update, near real time, 2-weeks data online

17

RDA Data Archive and ManagementRDA Data Archive and Management

RDA Data Archive ComponentsRDA Data Archive Components Creating Dataset MetadataCreating Dataset Metadata Populating Datasets with DataPopulating Datasets with Data Updating DatasetsUpdating Datasets Technology of data archivingTechnology of data archiving Accumulation of data archived in RDAAccumulation of data archived in RDA

18

RDA Data Archive ComponentsRDA Data Archive Components

19

Creating Dataset MetadataCreating Dataset Metadata

20

Populating Dataset with DataPopulating Dataset with Data

21

Updating DatasetsUpdating Datasets

22

Archiving Data with State of Art TechnologyArchiving Data with State of Art Technology

Hybrid archiving system with Database Management Hybrid archiving system with Database Management System (DBMS), Database for short, and file System (DBMS), Database for short, and file management systemmanagement system

Comprehensive archiving utility: gathering data, Comprehensive archiving utility: gathering data, archiving to MSS and/or online service, recording file archiving to MSS and/or online service, recording file metadata, and publishing data filelistsmetadata, and publishing data filelists

Automated and complete for periodic data updates: Automated and complete for periodic data updates: downloading, processing/validating, and archiving downloading, processing/validating, and archiving data data

All archiving, web and DBMS utilities are based upon All archiving, web and DBMS utilities are based upon open source softwareopen source software

Both onsite and offsite backup of whole RDABoth onsite and offsite backup of whole RDA

23

Data Archived In RDAData Archived In RDA(MSS: 121.5 TB, Online: 14.9 TB; By 2007-05)(MSS: 121.5 TB, Online: 14.9 TB; By 2007-05)

0

20

40

60

80

100

120

140

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

YEAR

DATA

VO

LU

ME (

TB

)

MSS DataOnline Data

24

RDA Data Usage StatisticsRDA Data Usage Statistics

How data are accessedHow data are accessed What data are accessedWhat data are accessed Who uses the RDA dataWho uses the RDA data

25

How Data Are AccessedHow Data Are Accessed(Downloaded between 2006-06-01 & 2007-05-31)(Downloaded between 2006-06-01 & 2007-05-31)

Download Method

Number of Unique Users

Data Volume Accessed (TB)

MSS 397 54

WEB 4285 31

FTP 772 15

Total 5046 100

26

How Data Are AccessedHow Data Are Accessed(Between 2006-06-01 & 2007-05-31)(Between 2006-06-01 & 2007-05-31)

Data Access Method



MSS 397 54

Direct Online 4655 44

Special Request 219 2

Total 5064 100

27

Top 10 Datasets Accessed by Most Top 10 Datasets Accessed by Most UsersUsers(between 2006-06-01 & 2007-05-31)(between 2006-06-01 & 2007-05-31)

Number Of Unique Users

Dataset Title

1681 NCEP Global Tropospher ic Analyses, 1x1, daily 1999Sep15- present

886 NCEP/NCAR G lobal Reanalysis Products, 1948-continuing 529 NGDC, ETOPO2 Global 2' Elevations

396 NCEP ADP G lobal Surface Observations, Februar y 1975 - continuing

352 Terra inBase, Global 5-Minute Ocean Depth and Land Elevation, from NGDC

302 NCEP Global Tropospher ic Analyses, 2.5x2.5, daily 1997Apr- continuing

296 NCEP ADP G lobal Upper Air Observation Subsets, January 1973 - continuing

236 NCEP North Amer ican Regional Reanalysis (NARR)

235 NCEP Vers ion 2.0 OI G lobal SST and NCDC Extended Reconstructed SS T Analyses

229 Internat ional Comprehensive Ocean-Atmosphere Data Set (ICOADS Re lease 2.3), Global Marine Surface Observations

28

Top 10 Datasets by Download VolumeTop 10 Datasets by Download Volume(between 2006-06-01 & 2007-05-31)(between 2006-06-01 & 2007-05-31)

Data Access Volume (TB)

Dataset Title

25 NCEP Global Tropospher ic Analyses, 1x1, daily 1999Sep15- present

20 NCEP North Amer ican Regional Reanalysis (NARR) 11 NCEP/NCAR G lobal Reanalysis Products, 1948 continuing

6 ERA40 Model Resolution Gridded Upper Air Analysis on Model Levels from ECMWF

4 ERA40 Model Resolution Gridded Upper Air Analysis on Pressure Surfaces from ECMWF

3 Historical Unidata In ternet Data Distribution (IDD) Gridded Model Data, December 2002 - current

3 ERA40 2.5 Degree Latitude-Longitude Gridded Upper Air Analysis on Pressure Surfaces from ECMWF

3 ERA40 Model Resolution Gridded Surface, Vertical Integra ls, and Other Single Level Fields from ECMWF

2 NCEP ADP G lobal Surface Observations, February 1975 - continuing

2 GCIP NCEP Eta model output

29

Who uses RDA DataWho uses RDA Data(Accessed Between 2006-06-01 & 2007-05-31)(Accessed Between 2006-06-01 & 2007-05-31)

Organization Type



NCAR 211 22

University 2999 58

Other 1933 20

Total

5100

100

30

Top 20 Countries/Areas by Most UsersTop 20 Countries/Areas by Most Users(Online data Between 2006-06-01 & 2007-05-31)(Online data Between 2006-06-01 & 2007-05-31)

Country & Area Unique Users Data Volume (GB) UNITED.STATES 1431 11686 P.R.CHINA 655 5846 JAPAN 267 3710 INDIA 187 1657 GERMANY 144 338 CANADA 143 4557 UNITED.KINGDOM 132 213 FRANCE 128 648 ITALY 105 844 TAIWAN 100 1485 SOUTH.KOREA 98 1243 SPAIN 93 932 BRAZIL 78 222 AUSTRALIA 71 70 RUSSIA 52 616 NETHERLANDS 51 88 MEXICO 48 211 THAILAND 39 708 PORTUGAL 35 365 NORWAY 33 35

31

Top 10 Datasets by Most Users FromTop 10 Datasets by Most Users FromP.R.China, Taiwan & Hong KongP.R.China, Taiwan & Hong Kong(Online Data Between 2006-06-01 & 2007-05-31)(Online Data Between 2006-06-01 & 2007-05-31)

Unique Users

Volume (GB)

Dataset Title

484 4157 NCEP Global Tropospheric Analyses, 1x1, daily 1999Sep15- present

149 1131 NCEP/NCAR G lobal Reanalysis Products, 1948 continuing

64 505 QSCAT/NCEP B lended Ocean Winds from Colorado Research Associates (version 4.0)

61 26 NCEP Global Tropospheric Analyses, 2.5x2.5, daily 1997Apr- continuing

53 241 NCEP ADP Global Surface Observations, February 1975 - continuing

52 31 NCEP ADP Global Upper Air Observation Subsets, January 1973 - continuing

33 1 NCEP Version 2.0 OI Global SST and NCDC Extended Reconstructed SST Analyses

32 12 NGDC, ETOPO2 Global 2' Elevations

30 1 Terra inBase, Global 5-Minute Ocean Depth and Land Elevation, from NGDC

24 61 GCIP NCEP Eta model output

32

Monthly Online Users FromMonthly Online Users FromP.R.China, Taiwan & Hong KongP.R.China, Taiwan & Hong Kong(Between 2005-01 & 2007-05)(Between 2005-01 & 2007-05)

0

50

100

150

200

250

Nu

mb

er

Of

Un

iqu

e U

se

rs

20

05

-01

20

05

-02

20

05

-03

20

05

-04

20

05

-05

20

05

-06

20

05

-07

20

05

-08

20

05

-09

20

05

-10

20

05

-11

20

05

-12

20

06

-01

20

06

-02

20

06

-03

20

06

-04

20

06

-05

20

06

-06

20

06

-07

20

06

-08

20

06

-09

20

06

-10

20

06

-11

20

06

-12

20

07

-01

20

07

-02

20

07

-03

20

07

-04

20

07

-05

Year-Month

33

Monthly Online Data by Users From Monthly Online Data by Users From P.R.China, Taiwan & Hong KongP.R.China, Taiwan & Hong Kong(Downloaded Between 2005-01 & 2007-05)(Downloaded Between 2005-01 & 2007-05)

0

200

400

600

800

1000

1200

1400

1600

1800

Da

ta V

olu

me

(G

B)

20

05

-01

20

05

-02

20

05

-03

20

05

-04

20

05

-05

20

05

-06

20

05

-07

20

05

-08

20

05

-09

20

05

-10

20

05

-11

20

05

-12

20

06

-01

20

06

-02

20

06

-03

20

06

-04

20

06

-05

20

06

-06

20

06

-07

20

06

-08

20

06

-09

20

06

-10

20

06

-11

20

06

-12

20

07

-01

20

07

-02

20

07

-03

20

07

-04

20

07

-05

Year-Month

34

Explore Data Archived in RDAExplore Data Archived in RDA

RegistrationRegistration Identify datasetIdentify dataset Get the dataGet the data

– Online dataOnline data– MSS FilelistMSS Filelist– SubsettingSubsetting– Questions and special requestsQuestions and special requests

35

Register Yourself on RDA ServerRegister Yourself on RDA Server

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.









36

Identify Datasets You WantIdentify Datasets You Want






are needed to see this picture.QuickTime™ and a

TIFF (LZW) decompressorare needed to see this picture.







37

How to Access Data For A Specified How to Access Data For A Specified DatasetDataset





38

How to Access Data For A Specified How to Access Data For A Specified DatasetDataset


are needed to see this picture.QuickTime™ and aTIFF (LZW) decompressor





are needed to see this picture.QuickTime™ and aTIFF (LZW) decompressor


39

Access Online DataAccess Online Data



…

…








are needed to see this p icture.



40

MSS File ListMSS File List

…

…













41

Data SubsettingData Subsetting

…

…






are needed to see this picture.QuickTime™ and a

TIFF (LZW) decompressorare needed to see this picture.

42

Future DevelopmentsFuture Developments

New data search tool based on standardized metadata, New data search tool based on standardized metadata, GCMDGCMD

Browse tool to identify the files needed, based on DB Browse tool to identify the files needed, based on DB and file metadata queriesand file metadata queries

Expanded RDA content, e.g. JRA-25, more ocean Expanded RDA content, e.g. JRA-25, more ocean reanalysisreanalysis

User feedback and information sharing at the dataset User feedback and information sharing at the dataset level using web forum technologylevel using web forum technology

More user subsetting and format conversionMore user subsetting and format conversion More open archiving and access service for NSF More open archiving and access service for NSF

projectsprojects– Need specific inclusion and management policiesNeed specific inclusion and management policies

Modernization of web page look and function Modernization of web page look and function

43

SummarySummary

NCAR’s RDA is an open international data resourceNCAR’s RDA is an open international data resource We encourage and facilitate data exchangesWe encourage and facilitate data exchanges We continually build and improved access to the RDAWe continually build and improved access to the RDA Modern DB systems are used for management and Modern DB systems are used for management and

public data servicepublic data service Modern data management methods (DB’s and Modern data management methods (DB’s and

automation) allow:automation) allow:– Systematic growth– Systematic and secure archiving– Accurate usage assessment metrics

We are always open to suggestions for new data We are always open to suggestions for new data resources and service improvements from the usersresources and service improvements from the users

44

Thank YouThank You

Questions?Questions?

Contact:Contact: http://dss.ucar.edu/index.htmlhttp://dss.ucar.edu/index.html Zaihua Ji, [email protected] Ji, [email protected]

Documents

Data to Support Ocean-Atmosphere Research NCAR Research Data Archive (RDA), Zaihua Ji, NCAR Steven Worley, NCAR Scott Woodruff,