Near Real TimeNear Real TimeOcean Ocean

Observations Observations OnlineOnline

the Detailed Escape of SEACOOSthe Detailed Escape of SEACOOS(Southeastern Atlantic Coastal Ocean (Southeastern Atlantic Coastal Ocean

Observing System) Observing System) Data Management and Visualization Data Management and Visualization

SecretsSecrets

Near Real Time Ocean Observations Online : SEACOOS

Software : enginesSoftware : engines

MapserverMapserver Open source from University of MinnesotaOpen source from University of Minnesota USC runs a mixture of versions from 3.6 to USC runs a mixture of versions from 3.6 to

4.24.2 PostgreSQLPostgreSQL

Open source from postgresql.orgOpen source from postgresql.org USC runs version 7.4.1USC runs version 7.4.1

PostGISPostGIS Open source from Refractions.netOpen source from Refractions.net USC runs version 0.8.1USC runs version 0.8.1

Near Real Time Ocean Observations Online : SEACOOS

Software : additionalSoftware : additional

PHPPHP Open source from php.orgOpen source from php.org USC runs version 4.3.2USC runs version 4.3.2

PerlPerl Open source from perl.orgOpen source from perl.org USC runs version 5.8.0USC runs version 5.8.0

MiscellaneousMiscellaneous ANiS and gifsicleANiS and gifsicle ImagemagickImagemagick

Near Real Time Ocean Observations Online : SEACOOS

HardwareHardware application application

serverserver Apache 2.xApache 2.x MapServerMapServer Perl, PHP, misc.Perl, PHP, misc.

database Adatabase A In-situIn-situ RSRS

database Bdatabase B In-situIn-situ model model

outputoutput

data data scoutscout In-situIn-situ RSRS

Near Real Time Ocean Observations Online : SEACOOS

Directory structureDirectory structure

Data that sits on USC filesystemsData that sits on USC filesystems RS imagesRS images cached imagescached images

All files have strict naming All files have strict naming convention that includes timestampconvention that includes timestamp

Near Real Time Ocean Observations Online : SEACOOS

System administrationSystem administration

Databases backed up nightlyDatabases backed up nightly requires some downtime (~ 2 hours)requires some downtime (~ 2 hours)

Databases cleaned up and optimized Databases cleaned up and optimized nightlynightly significant overhead ~ 4 hourssignificant overhead ~ 4 hours

Some backend data massaging Some backend data massaging (mainly for model output (mainly for model output aggregation)aggregation)

Near Real Time Ocean Observations Online : SEACOOS

Database structureDatabase structure One table category per data typeOne table category per data type

e.g. in-situ windse.g. in-situ winds wind_prod contains all wind datawind_prod contains all wind data wind_map contains wind data appropriate for mapswind_map contains wind data appropriate for maps

e.g. QuikSCAT windse.g. QuikSCAT winds more complicated since requires aggregation and more complicated since requires aggregation and

normalizationnormalization e.g. OI SSTe.g. OI SST

table containing pointers to data files on disktable containing pointers to data files on disk

Ancillary tables for specific data typesAncillary tables for specific data types e.g. OI SSTe.g. OI SST

table containing RGB to SST lookup values (for table containing RGB to SST lookup values (for querying purposes)querying purposes)

Near Real Time Ocean Observations Online : SEACOOS

Data processing overviewData processing overview

Data scout (netCDF)Data scout (netCDF) Perl code flattens incoming netCDF into arrays which are Perl code flattens incoming netCDF into arrays which are

turned into SQL INSERT statemtents.turned into SQL INSERT statemtents. Triggers update the new records suitable for Triggers update the new records suitable for

normalization as well as Mapserver display elements.normalization as well as Mapserver display elements. Procedures run to optimize tables for display.Procedures run to optimize tables for display.

RS images (HDF to PNG)RS images (HDF to PNG) begin as HDF but are availed to USC as PNG’sbegin as HDF but are availed to USC as PNG’s Images have standard naming convention and agreed Images have standard naming convention and agreed

upon extents as well as predefined RGB to value, e.g. upon extents as well as predefined RGB to value, e.g. RGB to SST, pairs.RGB to SST, pairs.

Incoming files cause postgreSQL table to be updated with Incoming files cause postgreSQL table to be updated with the new file and timestamp.the new file and timestamp.