ICOADS and RECLAIMICOADS and RECLAIM

Scott Woodruff1, Sandy Lubker1,Clive Wilkinson2, Eric Freeman3

1) NOAA Earth System Research Laboratory, USA2) CRU/Univ. of East Anglia, UK; & CDMP/NCDC, USA

3) Sourcecorp and CDMP/NCDC, USA

2nd ACRE Workshop1-3 April 2009, O’Reilly’s Rainforest Retreat, Australia

RECLAIM

(1)• IInternational nternational CComprehensive omprehensive OOcean-cean-AAtmosphere tmosphere DData ata SSetet

• Original COADS project initiated in 1981– Joint in US between NOAA (ESRL and NCDC) & NCAR

• With extensive international contributions such as:– DWD, JMA, KNMI, UK: Met Office and NOC, Southampton

• Website for data and metadata access: http://icoads.noaa.gov/

• Release 2.5 (1662-2007) nearing completion• Formal links to JCOMM still developing

– Focus on improving the flow of delayed-mode ship data

Release 2.5: Status & Historical Inputs• 1662-1979 completed: observations (IMMA format) + products• 1980-2004 being reprocessed and checked• through 2007 by mid-2009 (possibly using simplified procedure)• monthly GTS observational updates soon, connected to R2.5

GreenGreen spans periods of major new ship data sources for R2.5

R2.4R2.5

E.g. R2.5 spatial enhancements

Procedural change for RH to use later climatological QC limits

1985 2002 2009

Data mixture changes: 1878-1950 (for example)Data mixture changes: 1878-1950 (for example)additions/deletions, separately by deckadditions/deletions, separately by deck

US MMJ HSST

DWD

Kobe UK RN

R2.5 Reprocessing Issues• As of 1st ACRE workshop (June 2008):

– 1662-1969 processed (1950-69 preliminary)

• Problems/issues detected; some resolved:– World Ocean Database (WOD05)

• wind speeds (MBT records): zero/missing mixup

• also we improved “SST” derivation from profiles– Choose T value closest to 4M and ≤ 10 m depth– More resultant SST data; help with XBT bias issues

– UK Royal Navy Ship’s Logs 1938-45 (deck 245)

• many in-port observations (valid calm wind speed effect)

– Errors in UK Marine Data Bank (MDB) deck 254 (IMM)• some truncated SST (tenths to whole degrees)

• some doubled wind speeds (kts vs. m/s mixup)

– US Maury early SLP data biased: unresolved• KNMI report (Henrik Wallbrink) in final revision

– Wind speed bias issues, buoy archive problems, etc.

Annual distribution of SLP (% per hPa interval) 1778-1969Annual distribution of SLP (% per hPa interval) 1778-1969Note: anomalous patterns could result purely from geographic effectsNote: anomalous patterns could result purely from geographic effects

nobs

US Maury SLP

Annual distribution of wind speeds (% per m/s) 1750-1969Annual distribution of wind speeds (% per m/s) 1750-1969 Note: anomalous patterns could result purely from geographic effectsNote: anomalous patterns could result purely from geographic effects

WOD05 and UK RN

Recent data mixture complexity: platform types;Recent data mixture complexity: platform types;plus delayed-mode (DM) v. GTS dataplus delayed-mode (DM) v. GTS data

> GTS drifter reports consolidated in DM

For R2.5, large additions of DM ship and buoy data through 2007; possible extension of VOS metadata back to 1966

Goal to Improve Update Frequency: For Contemporary Data

• Soon (in conjunction with R2.5)– Monthly GTS-only updates (NCEP)– Obs only (replace abbreviated “NRT”)

• Future: greater roles NCDC & JCOMM?– NCDC GTS data would partly resolve ship

call sign masking (commercial/security)– More timely integration of DM VOS data– Proposed usage of Lloyd’s commercial

“ship particulars” (ship size/type metadata)

(2)• RECRECovery of overy of LLogbooks ogbooks AAnd nd IInternational nternational MMarine dataarine data

• Initiated in 2004, building on EU-funded CLIWOC (1750-1850) project (ES, NL, UK, and some FR)

Aim: A cooperative international project to image historical ship logbooks and related marine data and metadata, and digitize the meteorological and oceanographic observations for merger into ICOADS and for utilization for climate research

• Website: http://icoads.noaa.gov/reclaim/–populated with imaged US and UK publications–detailed UK data inventories, etc.

RECLAIM

Ongoing RECLAIM Archive Ongoing RECLAIM Archive Research: Clive WilkinsonResearch: Clive Wilkinson

• Linking with ACRE and other UK projects– Imaging and proposed digitization of UK RN World War I

• (1914-23) Ship’s Logs• Potential upcoming high priority for cooperation with CDMP?

– UK Hydrographic Holdings (e.g. Remarks Books)

– UK Met Office Archive (e.g., 4K 19thC Met. Registers)

– Expeditionary, whaling data

• Published data and literature inventory– currently >250 entries– marine and oceanographic interest

• Early platform and instrumental metadata

Example Untapped RN Platform & Instrumental MetadataExample Untapped RN Platform & Instrumental MetadataNote: WMO Pub. 47 (VOS) metadata: only 1955-forwardNote: WMO Pub. 47 (VOS) metadata: only 1955-forward

Early technological issues: digitization of UK, US, and other sources (e.g. using punched cards) often was unable to capture important data and metadata

RECLAIM-related imagingRECLAIM-related imagingOther important published metadataOther important published metadata

• WMO Guide to the Applications of Marine Climatology• Report (1989) on visit to US National Archives• Diaz et al. (1989): Climatic Summaries of OWS• Copies/translations of national historical marine

observing instructions (fr David Parker; C19th-20th)• Later UK coding, decoding, and observing manuals• International Meteorological Tables (1890)• Early editions WMO Manual on Codes (WMO-No. 306)

Note: Some materials held at international partners such as UK NOCS, Met Office, and JMA may be subject to copyright restrictions

Selected Data Rescue & Blending CandidatesSelected Data Rescue & Blending Candidates((digitizeddigitized planned for R2.6, as appropriate) planned for R2.6, as appropriate)

GreenGreen =digitized

YellowYellow =partially

RedRed =undigitizedNote: all require translation (T) to uniform IMMA format

T: 2009

T: 2012?

Avail.?

Future R2.6 for ReanalysesFuture R2.6 for ReanalysesCompletion Target: October 2010Completion Target: October 2010

Two major collections with availability questionsTwo major collections with availability questions

• Greenwich Mean Noon/Simultaneous (~1874-1947)– Several different collections

– Some US WB published• also by: Deutsche Seewarte and the Danske Meteorologiske Institut

– NCDC’s current (partial) availability (T) goal: 2012

• DWD Historical Archive– Early 2008 for R2.5, in response to GCOS WG-SP (CLIVAR,

JCOMM, etc.) endorsements: 1876-1915 data (1.2M)

– Newly digitized data, under DWD’s “HISTOR” project

– 3 July 2008 e-mail to R. Zoellner (member of JCOMM ETMC)• no response yet

– Possible availability of relevant DWD personnel at JCOMM SOT-V (May 2009, Geneva)

Survey of US Logbook CollectionsSurvey of US Logbook Collectionsgreen=blended, yellow=digitized not blended, red=undigitizedgreen=blended, yellow=digitized not blended, red=undigitized

Additional Data Rescue CandidatesAdditional Data Rescue Candidates

• KNMI C19th Logbooks (~170; 9K logbook pages)– imaging (by KNMI) in 2006; on WSSRD

• US Fish Commission and related records• Baltic Sea Marine Surface Observations: 1961-90; 360K• Ukranian Marine Data: 1958-85• Canada/MEDS Daily Seawater: 1914-85• Met. observations w/ US Tide Gauges: ~1854-forward• Finnish (1900-16, 1919-56) & Swedish (~1860-1922) Lightships

– partly digitized by US NODC• Mariner’s Museum (Newport News VA) Private C18th Logbooks

Ongoing questions: How to decide data rescue priorities?Ongoing questions: How to decide data rescue priorities?How to quantify data rescue benefitsHow to quantify data rescue benefits

(e.g., to fluxes, SST, reanalyses)?(e.g., to fluxes, SST, reanalyses)?Limited NOAA resources for historical updatesLimited NOAA resources for historical updates

Conclusions• ICOADS Marine Data Rescue: document in preparation

– catalog important data/metadata findings for future work

• NOAA resources needed for ICOADS historical updates– CDMP imaging/digitization largely funds private contractors

• IMMA observations: critical foundation for all later work– independent translation validations costly, but often worthwhile– pre-evaluation of data quality of newly available sources?

• QC improvements needed– e.g. current QC limits extensively missing for high-latitude data

• Plans for eventual “Climate ICOADS” initiative– bias adjustments -- value-added products