Upload
lillian-baird
View
222
Download
1
Tags:
Embed Size (px)
Citation preview
COL: data acquisition, storage and preservation
Roger [email protected]
www.met.reading.ac.uk/~brugge
Climatological Observers Link
Department of MeteorologyUniversity of Reading
Transmission, presentation and archiving of meteorological data, 5 Oct 2011
Outline
• What is COL?– What the data are used for
• Data gathering– Daily versus monthly data– Old and new methods
• Data quality – station gradings• Data storage
– What is stored?– How is it stored?
• Data preservation– For use by COL– For use by others
What is COL?• June 1950 - An advert in Weather in June 1950 by Tom Suttie
– enquiring about setting up an organisation for the exchange of weather data by amateurs
• 1969 - A chance sighting of this advert by Eric Bonsor– there followed an exchange of letters, and
• January 1970 - first edition of the Directory of Amateur Observers published– Site details, instrumentation and observing routines
• June 1970 - first issue of the Bulletin– 48 stations reporting, describing the weather of May 1970.– Nowadays more than 330 sites each month
• Aim of COL - to publish the monthly bulletin at the earliest opportunity– Achieved for over 41 years now– The COL Bulletin is now a rare source of comprehensive UK monthly
weather summaries.– Membership details added to the Directory of Amateur Observers– Almost 1300 stations are now included, together with photographs of
stations
Data gathering
• Monthly data only– A few exceptions for a daily data table– COL cannot handle the volume of daily data that would
otherwise accrue• Paper
– The original method (15% of returns)• Email
– Also used as a reminding mechanism to most members (15%)• Web (70% of returns)
– Underpinned by a database– PHP5 MyAdmin and SQL >archiving– Provides members with an early snapshot– The way of the future?
Paper form – main data features only shown
Data gathering
• Monthly data only– A few exceptions for a daily data table– The Bulletin
• Paper– The original method (15% of returns)
• Email– Also used as a reminding mechanism to most members (15%)
• Web (70% of returns)– Underpinned by a database– PHP5 MyAdmin and SQL > archiving– Provides members with an early snapshot– The way of the future?
Email reply form – the start of a message for August 2011
Data gathering
• Monthly data only– A few exceptions for a daily data table– The Bulletin
• Paper– The original method (15% of returns)
• Email– Also used as a reminding mechanism to most members (15%)
• Web (70% of returns)– Underpinned by a database– PHP5 MyAdmin and SQL > archiving– Provides members with an early snapshot– The way of the future?
Data quality
• COL welcomes observations from everyone– Multiple stations per town– Wide variety of instrumentation and exposure
– Airports, universities, institutions and the hobbyist
• Instrumentation– Originally based around Stevenson screen, mercury-in-glass
thermometers and Snowdon raingauges– Many good quality sites– Increasing use being made of AWS equipment
• Varying costs, and also performance
• Need a way to distinguish between stations that (even in the same locality) might report widely differing obs– > COL station grading system (Stephen Burt)
Buxton, Norfolk
Stratfield Mortimer
Elderslie
Tivington
Red Wharf Bay, Anglesey
Compton Martin
Data quality
• COL welcomes observations from everyone– Multiple stations per town– Wide variety of instrumentation and exposure
– Airports, universities, institutions and the hobbyist
• Instrumentation– Originally based around Stevenson screen, mercury-in-glass
thermometers and Snowdon raingauges– Increasing use being made of AWS equipment
• Varying costs, and also performance
• Need a way to distinguish between stations that (even in the same locality) might report widely differing obs– > COL station grading system (Stephen Burt)
Station grading system(as used in 2011)
Grading system development over the years has been led by Stephen Burt
COL station grades
A27%
B17%
C233%
U15%
C18%
Incl. SYNOP and CLIMAT sites
• Station grade + photograph + station details = a reasonable idea of data quality
Data storage
• What is stored?• Monthly summaries
– A subset of the data; coverage 1940s onwards• Column-aligned text files• Easily manipulated by Excel and Fortran, etc.• Used to make quick calculations and climatological values
– e.g. recently published 1981-2010 averages
• All monthly data, numbers and text– Database; coverage 2006 onwards
• Generated by SQL/webpage• Little-used
• All back issues of the bulletin available electronically – 40th anniversary CD
Members observations – where do/could they go?
• Lots of high-quality COL observations (especially grade A stations)• Effectively unused or un-archived simply because they are not reporting to
UKMO• Data from these sites are likely to be lost unless special arrangements are
made. Currently paper copies– traditionally,
• sometimes saved in local history libraries• donated to an observer living close by• sent to the local rubbish tip (the majority)
– now: they can be deposited with the Chiltern Observatory Trust• Electronic data
– Monthly – some/all retained by COL– Daily – ‘delete’ key– Either way – file formats soon become outdated
• A proper remote archiving/backup facility is preferable.• Might BADC have a role here?
• Two suggested solutions…
Preserving members observations – Possible solution (1)
• 2011: Many amateur sites are fully/partial AWS • Might consumer-AWS instrument manufacturers agree a
common ‘archive file’ export standard with BADC?– to simplify the archiving process– ensure amateur data is not lost when a disk crashes or the
observer dies.
• If this was a file that could be exported then uploaded to a BADC site, say annually, it would take only minutes.
• A mandatory metadata section should be required.• Might Davis…
– who are the market leaders– who currently provide an export file format for NOAA
• …take a lead on this?
Preserving members observations – Possible solution (2)
• BADC to propose a standard Excel spreadsheet-type template for amateur observers to use.– This would greatly simplify the transmission and storage of
amateur observations to BADC.– A mandatory metadata section should be required, perhaps
including site plan/s and photographs from cardinal points.– Perhaps pilot
• using the existing ‘COL archive’ template materials (hourly, daily, monthly templates have been created)
• and with a voluntary participation of a handful of A-grade COL stations in 2012?
• Issues:– There are as many observer-owned spreadsheet formats as
observers– However, given a lead by BADC, COL members might be
persuaded to modify or convert their formats
More information• This presentation –
– To be placed at www.met.reading.ac.uk/~brugge• Website
– www.met.reading.ac.uk/~brugge/col.html• Weather article
– Brugge, R., 2010. Forty years of the Climatological Observers Link. Weather, 65, 139-143.
• Email the author– [email protected]
• Bulletin – some copies available today• ‘Processed data’: an example
– Burt, S.D. and R. Brugge, 2011. Climatological Averages for 1981-2010 and 2001-2010 for stations appearing in the monthly bulletin of the Climatological Observers Link. ISBN 9780956948502.