Tasks within WP4&

The EDUCE database - A summaryWP1, WP2, WP4 and WP5 tasks

Jussi Kaurola, Niilo Siljamo, Antti ArolaOzone and UV-Radiation Research Group

Finnish meteorological Institute

Contributions: Tim Martin (UG), Brian Gardiner&Peter Kirsch (BAS), Harry Slaper&Peter

den Outer (RIVM), Ola Engelsen (NILU), Martin Stanek (CHMI), Ann Webb&Richard Kift (UMIST),

Massimo Ferraguto (SSF), ...

Tasks within WP4 (deliverables):

4.1. Installation of new hardware, permitting... (FMI)

4.2. Installation of WWW data submission tools (FMI)

4.3. Addition of further Oracle structures for non-spectral and ancillary data (FMI, BAS)

4.4. Provision of web pages to provide search access to the Oracle database and guidance to users (FMI)

4.5. Update and maintain metadata pages for the database, including flag for WMO guidelines (UMIST)

4.6. Testing of and report on all database search and access tools, and publishing of user's guide and manual, with copy made available at web site (UG)

The EDUCE database - A summaryWP1, WP2, WP4 and WP5 tasks:

WP1 & WP2

recent changes in the database

screen shots

submission facilities

retrieval facilities

data integrity

hardware issues

plans for the future

FMI tasklist

summary

FMI task in WP1: "RAF for ozone and albedo"

Antti Arola

"The irradiance ratio of snow-covered to snow-free conditions tells how much snow enhances the irradiance on average.

The average albedo RAF in Sodankylä, calculated from 11 years of data, is about 1.19. In other words, snow enhancement is about 19%"

Report 1.4 (11pages):http://www.muk.uni-hannover.de/~martin/results/deliverables/1.4/wp_1_4.pdf

FMI task in WP2: "2.1An evaluation of the impact of atmospheric

variability on the detection of trends in UV"Antti Arola

The developed methodology offers a way of separating the effect of a single factor, e.g., ozone, cloudiness, aerosols or albedo, on the short- and long-term variability of UV, using measurements of spectral irradiance

Report 2.1 (13 pages):http://www.muk.uni-hannover.de/~martin/results/deliverables/2.1/wp2.1_finalreport.pdf

Arola A., K. Lakkala, A. Bais, J. Kaurola, C. Meleti, and P. Taalas, "Factors affecting short- and long-term changes of spectral UV irradiance at two European stations", JGR, doi:10.1029/2003JD003447, 2003.

Recent changes in the database

quality flags:old data flagged "manually"

incoming data flagged automatically

submission:auxiliary data

retrieval of data:flags included in the retrieval of spectra

broadband data

auxiliary data: daily values (hourly values not yet)

better information of recent submissions, enhanced user options for output (CHMI)

Data submission facilitiesspectral and broadband submission has been in operation since 2001

the submission of auxiliary data was delayed due to problems with (old) software

now the submission works also for aux data

"short and friendly guides" for spectral and auxiliary data

spectral guide should be OK (after several iterations)

auxiliary guide was revised two weeks ago

planned changes:

speeding up of submission process

make it work more as a background process

Data retrieval facilitieschanges made since the last meeting

new oracle table: measurement_type(spectral, broadband, ozone, cloud, dursun, snowcover)

broadband data can be retrieved

daily auxiliary can be retrieved (currently not hourly data)

flags included in headers and can be retrieved separately

some fixes of previously reported problems

planned changesfix problems with hourly auxiliary data

new user requirements for WPI design, give us feedback!

spectral retrieval somewhat slow and error-prone (queue problems). A completely new way to make SQL query and consequent data handling is planned

flagging of one spectrum

149263#056 GREEN GREEN Cloudy 3.639e+02 -1.000e+05 8.380e+01 -1.000e+05 -1.000e+05 GREEN -1.200e-02 14 GREEN 1.700e-02 67 GREEN 1.010e-05 6.600e-06 GREEN 1.850e+00 0 0 0 GREEN CLOUDS 2.860e-01 3.207e-01 SOME_VARIATION_SCAN 9.240e+00 9.766e-04 slf00567 /UVROOT/DB/DBroot/no_ntnu_trh/1997/ol752/11010000.gxa 56

Data integrity: duplicates

what is a duplicate? a measured spectrum which has several copies in

the database several different spectra have the same

measurement time

what is an overlapping spectrum? several different spectra have overlapping

measurement time

both types are included in the data retrievals both types need to be removed

Data integrity: duplicates

duplicates caused by errors in submitted dataautomatic identification: very complex task, not possible with current software

manual work is needed

current status (August run by the BAS tool):belsk bre064 128bru jytot 148855

spurious zeroes

during the flagging spectra with only zero values in irradiance column were recorded

41345 spectra found

only 93 cases with sza < 90 degreesat_boku_sonnblick 2at_boku_vienna 1be_iasb_bru 53fi_fmi_sod 2fr_ustl_vda 5gb_umist_reading 11it_jrc_ispra 27no_nilu_andoya 8

Hardware status

the current hardware is becoming old(purchased in 2000 with $40 000)

disks are almost full, Oracle disk at critical levelrecent submissions have been very large

total disk space 106 GB

Oracle data size 56 GB

Flexstor data size 23 GB

CPU power is adequate, flagging might be lagging a bit in the case of large submissions

disk failure on last Thursday, which was handled very well by the FMI IT department

Planned changes

some hardware upgrading is needed:poor mans solution to disk problem:new disks: 4 x 150 GB (4x600 euro)to current disk box (2 for data, and 2 for mirror)

better solution: new disk array (about 15 keuro)and some new disks. Much more tolerant for errors.

best solution: new server (20 keuro), new disk array (15 keuro) and new disks

a short break in service, a longer break in uploading of new data to Oracle

operating system upgrade in November-December causing 2-3 days break

upgrading of e.g. WPI related software (TOMCAT)

Planned changes...

currently everything is in ozone2.fmi.fi server which is outside FMI firewall ("grey area")

an alternative solution:server with Flexstor and WPI in "grey area"

Oracle inside firewall

a better integration of EUVDB Oracle database to FMI other Oracle databases. Benefits:

security

maintenance

even performance?

FMI task list

follow the link on "General Information Page" updated regularly check the "wish list" if you have ideas for new

improvements

Some remarks:-campaign data

manual submission by FMI retrieval from www pages needs planning and

work

Summary

personal opinion: now the functionalities of the database are rather good

some FMI milestones were delayed considerably the whole project time was more or less

development period, which certainly has caused headache to some users

remaining work: hourly auxliary data planned improvements:

smoother submission of large datasets faster retrieval of spectral data improved WPI design

our interest is to continue with the database maintenance. Funding to be discussed later...