RECLAIM and ICOADSRECLAIM and ICOADS

Scott Woodruff1, Philip Brohan2, Eric Freeman3, Elizabeth Kent4, Sandy Lubker1, Clive

Wilkinson5, and Steve Worley6

1) NOAA Earth System Research Laboratory, USA2) Met Office Hadley Centre, UK

3) Sourcecorp and CDMP/NCDC, USA4) National Oceanography Centre, Southampton, UK

5) CDMP/NCDC, USA; & CRU/Univ. of East Anglia, UK6) National Center for Atmospheric Research, USA

1) RECLAIM• RECovery of Logbooks And International

Marine data project• Initiated in 2004, building on EU-funded CLIWOC

(1750-1850) project (ES, NL, UK, and some FR)

•Website: http://icoads.noaa.gov/reclaim/–populated with imaged US and UK publications–detailed UK data inventories, etc.

Recent Accomplishments • Imaging by KNMI in 2006 of C19th Dutch logbooks; planned for

future digitization by CDMP• Jointly funded UK & CDMP effort to image (268K pp) and

digitize (1.5M obs) selected UK Royal Navy (RN) Ship’s Logs for data sparse period (1938-47) ~WW II

Ongoing and Planned UK ProjectsTogether with ACRE, The UK National Archives (TNA), other partners

• English East India Co. Logbooks (>1K; 1790-1834) at British Library; many recorded SLP and AT

• RN Ship’s Logs from ~WW I (1914-23; ~376K days of obs)• H.O. “Remarks Books” (6K, 1800-1909), Met. Registers (1850-),

early balloon data from ships, publications, etc.• Many more logbooks (1669-) untapped at TNA and elsewhere

UK National Archives - Main Series of Logbooks 1669-1899

0

2000

4000

6000

8000

10000

12000

1669-1679 1700-1709 1730-1739 1760-1769 1790-1799 1820-1829 1850-1859 1880-1889

De cade

Number of Logbooks

Ships ' Logs

Maste rs ' Logs

Capta ins ' Logs

2) ICOADS• International Comprehensive Ocean-

Atmosphere Data Set• Original COADS project initiated in 1981

– Joint in US between NOAA (ESRL and NCDC) & NCAR

• Plus extensive international contributions such as:– DWD, JMA, KNMI, UK Met Office and National

Oceanography Center, Southampton

• Website including data and metadata access: http://icoads.noaa.gov/

• Formal links to JCOMM under consideration

Major update in progress:Release 2.5 (~1662-2000)

• 1662-1949 processed (redred) – 1950-69 being checked• 1970-2000 (approx.) by late 2008

GreenGreen spans periods of new sources for R2.5

R2.0

R2.5 (bars): increases in data density: 1800-1969R2.5 (bars): increases in data density: 1800-1969

US MMJ

Japan Kobe

UK RN

UK MDB

R2.0

Evolving national (+ HSST) data mixEvolving national (+ HSST) data mixR1

1985R2.0 2002

R2.5 2008

Thompson et al. 2008, Nature: A large discontinuity in the mid-C20th in observed global-mean surface temp

HSST data (abbrev. fmt.)

R2.5: Decadal changes in spatial density; focus on 1930sR2.5: Decadal changes in spatial density; focus on 1930s(red= + , grey= 0 , blue= –)

HSST deck 156: Previously undetected dups w/ deck 193 (due to reversed sign of gravity correction in HSST) removed – but no fix available for empty box

UK MDB deck 215 (German data): additional previously undetected dups removed, or replacement by better quality data from DWD

Dutch deck 193: empty box (data never received?)

Future Directions:

a) Update Frequency • Aim for monthly extensions

– GTS plus some delayed-mode (DM) data– replace NCEP Real-time data

• Challenges:– NOAA funding pressures – larger role for NCDC

• Historical (CDR) updates not suitable for “operations”

– VOS call sign masking• Stemming from security and commercial concerns• NCEP GTS Dec 2007: all ships masked

– Mandated WMO BUFR transition• could be disruptive to data quality and continuity

DM ship/buoy/oceanographic data through 1997/1997/1996

DM buoy data

GTS

WMO Pub. 47 metadata

Recent data mixture complexity: platform types;Recent data mixture complexity: platform types;plus DM v. GTS dataplus DM v. GTS data

> GTS drifter reports consolidated in DM

Potential for artificial signals from DM vs. GTS data mixture changes

R2.5

Future Directions:

b) Improve Data Quality Control • Existing ICOADS QC

– Out-of-date algorithms and QC limits– Can be insufficiently responsive to genuine

climate signals (“trimming” problem)

• Improvements could be tied into:– Enhanced internationalization (JCOMM):

• E.g., improved VOS data flow: TT-DMVOS

– Proposed “Climate ICOADS” Project• For bias-adjusted/valued-added products

R2.5 ad hoc QC improvement: Trimming limits R2.5 ad hoc QC improvement: Trimming limits (July) for RH: Used 1910-49 for pre-1910 data(July) for RH: Used 1910-49 for pre-1910 data

(End years of the three original trimming data periods (1854-1909, 1910-49, and 1950-79) shown)

QC gaps in other variables e.g. SLP and SST

DM buoy/ODAS data (e.g., Canadian): 1980-date

Chinese/GODAR Ships: 1968-93; 424K

Arctic Drift Stations: 1952-76

US Navy Hourlies: 1952-64; 3M (deck 117)

US Lightship Data: 1891-1986; 430K (~2009)

GMN Obs: 1886-1939

DWD Historical Archive: 1850-1939; 11.2M

Arctic Norwegian Logbook Data: 1867-1912; 125K

German Maury Collection: 1845-67; 544K

English East India Co.: 1790-1834; 285K to <6.8M (~2009)

c) Future Directions:c) Future Directions:Additional Data (Metadata) Rescue & BlendingAdditional Data (Metadata) Rescue & Blending

GreenGreen =digitized

YellowYellow =partially

RedRed =undigitized

But costly format translations w/ careful validation required:

• International Maritime Met. Archive (IMMA) format: has extensibility & flexibility to preserve original data

Conclusions• Much useful RECLAIM & ICOADS work remains, e.g.:

– HSST replacement desirable (as feasible; e.g., Dutch not?)– negotiate exchange of more DWD historical marine archive– improve SST indicator information (e.g., 1969-73 US data)

• IMMA observations: critical foundation for all later work– intensive translation validations costly, but often worthwhile– pre-evaluation of data quality of newly available sources?– possible role for additional international cooperation (e.g., UK)

• CLIMAR-III (6-9 May, Gdynia, Poland)– 3rd JCOMM Workshop on Advances in Marine Climatology– summary being finalized; WMO Bulletin article planned– recommendations relevant to RECLAIM, ICOADS, and ACRE