WMO BUFR training course Langen 17-20 April 2007
Slide 1
Slide 1
Ecmwf observations data handling
Milan Dragosavac
WMO BUFR training course Langen 17-20 April 2007
Slide 2
Slide 2
MDB RDB PDBFDB
ODB
IFS
MARS
General applications
Analysis Forecast
DisplayPrintPlot
Data transport
Datatransport
Data acquisition
WMO BUFR training course Langen 17-20 April 2007
Slide 3
Slide 3
Data acquisition
Acquire data
Process data
Archive data ( not MARS)
Outgoing ( send data outside Ecmwf)
WMO BUFR training course Langen 17-20 April 2007
Slide 4
Slide 4
Acquisition process
GTS data
ftp
Other data
WMO BUFR training course Langen 17-20 April 2007
Slide 5
Slide 5
Acquisition outgoing
Qscat satellite data in BUFR form
SSMI brightness temperatures
TOVS 1B calibrated and converted into BUFR format
WMO BUFR training course Langen 17-20 April 2007
Slide 6
Slide 6
MEB feed
Restores any old data
Join segmented bulletins
Write bulletins in MEB
statistics
WMO BUFR training course Langen 17-20 April 2007
Slide 7
Slide 7
Message data base
“STREAM”_YYYYMMDD
AIRC_20070216
AIRS_20070216
AMAP_20070216
GRIB_20070216
75 streams in total
One stream per day
WMO BUFR training course Langen 17-20 April 2007
Slide 8
Slide 8
Message data base attributes
TIME character( 4,1)
OBSTYPE character( 4,1)
TTAAii character( 6,1)
BBB character( 3,1)
CCCC character( 4,1)
_SOURCE character(80,1)
_CHECKSUM longinteger
_TIMESTAMP character(20,1)
WMO BUFR training course Langen 17-20 April 2007
Slide 9
Slide 9
Message data base attributes continued
_data_size longinteger
_bulk_size longinteger
_owner character( 3,1)
_last_access time(0)
_access_number longinteger
_compress character( 1,1)
_data bulk(20,0,512,1)
WMO BUFR training course Langen 17-20 April 2007
Slide 10
Slide 10
Message data base indices
UNIQUE I_timestamp_SYNO_20070215 on (_TIMESTAMP)
UNIQUE I_comp_SYNO_20070215 on (TIME, TTAAii, BBB, _CHECKSUM)
NORMAL (2,62) I_OBSTYPE_SYNO_20070215 on (OBSTYPE)
WMO BUFR training course Langen 17-20 April 2007
Slide 11
Slide 11
Message data base
100 Gbytes allocated + 150 Gbytes for METOP data
Over 22 Gbytes / day
Saved on ECFS according to OBSTYPE
All data can be reprocessed if Data base is lost
HP cluster 4 CPU 64 bit processors
4 operational machines + 1 for development
Raid disks 1.2 Tbytes
WMO BUFR training course Langen 17-20 April 2007
Slide 12
Slide 12
MEB data arrival
Monitoring arrival times of different data types
Analysis cycle
Average number of days
WMO BUFR training course Langen 17-20 April 2007
Slide 13
Slide 13
Message data base data availability
Data availability according to synoptic hour
WMO BUFR training course Langen 17-20 April 2007
Slide 14
Slide 14
MEB x-interface
Allows MEB search
Find corresponding BUFR observation in REB
Expand BUFR and GRIB data
Write data into file
WMO BUFR training course Langen 17-20 April 2007
Slide 15
Slide 15
MEB x-interface
REB BUFR observations corresponding to MEB bulletin
WMO BUFR training course Langen 17-20 April 2007
Slide 16
Slide 16
Preprocessing suite
MEB – message data base feed task
deqc – decoding, quality control and BUFR creation, writing into REB
ex00 – extraction for analysis
bc00 – extraction for boundary condition suite
wmo – ftp WMO station list
WMO BUFR training course Langen 17-20 April 2007
Slide 17
Slide 17
Decoding
Decoding
Quality control
BUFR creation
Global grib fields creation
Writing into REB
WMO BUFR training course Langen 17-20 April 2007
Slide 18
Slide 18
REB extraction for analysis
14 files created
Conventional observations
Satellite data
Sea surface temperatures
Ice analysis
WMO BUFR training course Langen 17-20 April 2007
Slide 19
Slide 19
REB – report data base
One table per observation type per day
AIRC_20070215
ATOV_20070215
B002_20070215
CREX_20070215
TEMP_20070215
SSBT_20070215
QSCA_20070215
statistic
WMO BUFR training course Langen 17-20 April 2007
Slide 20
Slide 20
REB attributes
TIME character(6,1)
IDENT character(9,1)
_BUFR_TYPE integer
_BUFR_SUBTYPE integer
_LATITUDE1 float(6)
_LONGITUDE1 float(6)
_COR1 integer
_QC integer
WMO BUFR training course Langen 17-20 April 2007
Slide 21
Slide 21
REB attributes continued
_TIMESTAMP character(20,1)
NR_SUBSETS integer
_data_size longinteger
_bulk_size longinteger
_owner character(3,1)
_last_access time(0)
_access_number longinteger
_data bulk(20,0,512,1)
WMO BUFR training course Langen 17-20 April 2007
Slide 22
Slide 22
REB indices
UNIQUE I_ts_SYNO_20070215 on (_TIMESTAMP)
UNIQUE I_comp_SYNO_20070215 on (TIME, IDENT, _BUFR_SUBTYPE, _LATITUDE1, _LONGITUDE1
NORMAL (2,15) I_extract_SYNO_20070215 on (TIME, _BUFR_SUBTYPE, IDENT)
WMO BUFR training course Langen 17-20 April 2007
Slide 23
Slide 23
REB reports data base
Just bellow 22 Gbytes / day
100 Gbytes allocated + 150 Gbytes for METOP data
All data can be re-decoded at any time
Data can be accessed through MARS request and Metview
Archived in MARS
WMO BUFR training course Langen 17-20 April 2007
Slide 24
Slide 24
REB x-interface
Search REB
Find original bulletin in MEB
Expand BUFR and GRIB data
Write data to file
Access WMO station list data base
WMO BUFR training course Langen 17-20 April 2007
Slide 25
Slide 25
BUFR x-interface
Expands BUFR message
Any subset can be viewed
Filtering features
WMO BUFR training course Langen 17-20 April 2007
Slide 26
Slide 26
WMO Data Base
WMO_a_master
WMO_a_master_manual
WMO_a_master_tape
WMO BUFR training course Langen 17-20 April 2007
Slide 27
Slide 27
WMO_a_master attributes
Region_country integer not null
Block smallint not null
Station_index integer not null
Station_subindex smallint not null
Principal _station character(1,1)
Station_name text(20,20,30,1)
Latitude real
Longitude real
WMO BUFR training course Langen 17-20 April 2007
Slide 28
Slide 28
WMO_a_master attributes continued
Pressure_elevation integer
Pressure_elevation_flag character(1,1)
Ground_elevation integer
Ground_elevation_flag character(1,1)
Pressure_level smallint
Surf_prog_00 character(2,1)
Surf_prog_03 character(2,1)
Surf_prog_06 character(2,1)
Surf_prog_09 character(2,1)
Surf_prog_12 character(2,1)
WMO BUFR training course Langen 17-20 April 2007
Slide 29
Slide 29
WMO_a_master attributes continued
Surf_prog_15 character(2,1)
Surf_prog_18 character(2,1)
Surf_prog_21 character(2,1)
Ua_type_00 smallint
Ua_type_06 smallint
Ua_type_12 smallint
Ua_type_18 smallint
Modify_flag character(1,1) not null
WMO BUFR training course Langen 17-20 April 2007
Slide 30
Slide 30
WMO_a_master indices
UNIQUE I_wmo_a_master ON (block,station_index, station_subindex)
WMO BUFR training course Langen 17-20 April 2007
Slide 31
Slide 31
WMO station list
WMO BUFR training course Langen 17-20 April 2007
Slide 32
Slide 32
ODB Observational data base
Interface to analysis
Data base contains unpacked bufr data and original bufr data
Loading done in parallel with load balance achieved on supercomputer due to bufr features
Analysis updates ODB
Analysis bufr feedback files created
Archiving of feedback files in MARS
WMO BUFR training course Langen 17-20 April 2007
Slide 33
Slide 33
Loading ODB with observations
Tasks running in parallel
2-16 processors used per task
Load balancing achieved
WMO BUFR training course Langen 17-20 April 2007
Slide 34
Slide 34
ODB to BUFR
odb2bufr family
WMO BUFR training course Langen 17-20 April 2007
Slide 35
Slide 35
Odbviewer
WMO BUFR training course Langen 17-20 April 2007
Slide 36
Slide 36
Odbviewer
WMO BUFR training course Langen 17-20 April 2007
Slide 37
Slide 37
Odbviewer
WMO BUFR training course Langen 17-20 April 2007
Slide 38
Slide 38
Odbviewer
WMO BUFR training course Langen 17-20 April 2007
Slide 39
Slide 39
Odbviewer
WMO BUFR training course Langen 17-20 April 2007
Slide 40
Slide 40
Odbviewer
WMO BUFR training course Langen 17-20 April 2007
Slide 41
Slide 41
Odbviewer
WMO BUFR training course Langen 17-20 April 2007
Slide 42
Slide 42
Future
Sharp increase in volume of observations expected
IASI data with 8641 channels per observation
Data bases have crucial role in observations handling
Binary data representation allows easy handling of observations as bulk objects