This article was downloaded by: [Mr Andrea Bergamasco]On: 29 May 2012, At: 14:21Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Advances in Oceanography andLimnologyPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/taol20

Knowledge discovery in large modeldatasets in the marine environment:the THREDDS Data Server exampleA. Bergamasco a , A. Benetazzo a , S. Carniel a , F.M. Falcieri a , T.Minuzzo a , R.P. Signell b & M. Sclavo aa CNR-ISMAR, Castello 2737/F, 30122 Venice, Italyb U.S. Geological Survey, Woods Hole, MA 02540, USA

Available online: 25 May 2012

To cite this article: A. Bergamasco, A. Benetazzo, S. Carniel, F.M. Falcieri, T. Minuzzo, R.P. Signell& M. Sclavo (2012): Knowledge discovery in large model datasets in the marine environment: theTHREDDS Data Server example, Advances in Oceanography and Limnology, 3:1, 41-50

To link to this article: http://dx.doi.org/10.1080/19475721.2012.669637

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representationthat the contents will be complete or accurate or up to date. The accuracy of anyinstructions, formulae, and drug doses should be independently verified with primarysources. The publisher shall not be liable for any loss, actions, claims, proceedings,demand, or costs or damages whatsoever or howsoever caused arising directly orindirectly in connection with or arising out of the use of this material.

Advances in Oceanography and LimnologyVol. 3, No. 1, June 2012, 41–50

Knowledge discovery in large model datasets in the marineenvironment: the THREDDS Data Server example

A. Bergamascoa, A. Benetazzoa, S. Carniela*, F.M. Falcieria, T. Minuzzoa,R.P. Signellb and M. Sclavoa

aCNR-ISMAR, Castello 2737/F, 30122 Venice, Italy; bU.S. Geological Survey,Woods Hole, MA 02540, USA

(Received 8 December 2011; final version received 15 February 2012)

In order to monitor, describe and understand the marine environment, manyresearch institutions are involved in the acquisition and distribution of oceandata, both from observations and models. Scientists from these institutions arespending too much time looking for, accessing, and reformatting data: they needbetter tools and procedures to make the science they do more efficient. The U.S.Integrated Ocean Observing System (US-IOOS) is working on making largeamounts of distributed data usable in an easy and efficient way. It is essentiallya network of scientists, technicians and technologies designed to acquire, collectand disseminate observational and modelled data resulting from coastal andoceanic marine regions investigations to researchers, stakeholders and policymakers. In order to be successful, this effort requires standard data protocols,web services and standards-based tools. Starting from the US-IOOS approach,which is being adopted throughout much of the oceanographic and meteorolog-ical sectors, we describe here the CNR-ISMAR Venice experience in the directionof setting up a national Italian IOOS framework using the THREDDS(THematic Real-time Environmental Distributed Data Services) Data Server(TDS), a middleware designed to fill the gap between data providers and datausers. The TDS provides services that allow data users to find the data setspertaining to their scientific needs, to access, to visualize and to use them in aneasy way, without downloading files to the local workspace. In order to achievethis, it is necessary that the data providers make their data available in a standardform that the TDS understands, and with sufficient metadata to allow the datato be read and searched in a standard way. The core idea is then to utilize aCommon Data Model (CDM), a unified conceptual model that describes differentdatatypes within each dataset. More specifically, Unidata (www.unidata.ucar.edu) has developed CDM specifications for many of the different kindsof data used by the scientific community, such as grids, profiles, time series, swathdata. These datatypes are aligned the NetCDF Climate and Forecast (CF)Metadata Conventions and with Climate Science Modelling Language (CSML);CF-compliant NetCDF files and GRIB files can be read directly with nomodification, while non compliant files can be modified to meet appropriatemetadata requirements. Once standardized in the CDM, the TDS makes datasetsavailable through a series of web services such as OPeNDAP or Open GeospatialConsortium Web Coverage Service (WCS), allowing the data users to easilyobtain small subsets from large datasets, and to quickly visualize their content byusing tools such as GODIVA2 or Integrated Data Viewer (IDV). In addition,an ISO metadata service is available through the TDS that can be harvested

*Corresponding author. Email: sandro.carniel@cnr.it

ISSN 1947–5721 print/ISSN 1947–573X online

! 2012 Taylor & Francishttp://dx.doi.org/10.1080/19475721.2012.669637http://www.tandfonline.com

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

by catalogue broker services (e.g. GI-cat) to enable distributed search acrossfederated data servers. Example of TDS datasets can be accessed at the CNR-ISMAR Venice site http://tds.ve.ismar.cnr.it:8080/thredds/catalog.html.

Keywords: large model output; NetCDF; CF convention; THREDDS

1. Introduction

The marine environment is characterized by a large number of complex dynamicalprocesses of societal importance, such as sea level rise, coastal flooding, coastal erosion [1],harmful algal blooms and oil spills. In addition, climate change induced effects at globalor regional scales have not been yet completely understood, and as a consequence there is alarge uncertainty on the effects that they may have on coastal zones [2].

Many national and international research bodies and institutions are actively acquiringmarine data both in situ (e.g. current meters, wave riders, tide gauges, etc.) and remote(e.g. from satellite), as well as running complex, integrated numerical models with the aimof monitoring and depicting the status of our seas.

Despite the growing number of datasets produced by observations and modelling,ocean data is still often generated using custom formats, and distributed using a variety ofad hoc methods, making it difficult to efficiently locate and access data from multipleinstitutions. Scientists very often spend considerable efforts in the time consuming activityof localizing and retrieving data; and, even when successful in this, they still have to spenda considerable amount of time to properly organize them before plotting or analyzingthem, because of different formats, conventions, etc.

Luckily, it is possible to use existing tools and techniques to overcome this problem,turning non-standard datasets held at institutions into standard web services in a way thatputs little burden on the data providers [3,4]. These approaches have been applied to theU.S. Integrated Ocean Observing System (US-IOOS, see http://www.ioos.gov) to makecollectively held oceanographic data easy to find and utilize. IOOS is a coordinatednetwork of organizations that work together to acquire, organize and distributeobservational and model data in the coastal ocean, to allow for improved understandingand prediction of the marine environment [5].

CNR-ISMAR Venice is helping to set up a national Italian IOOS framework, with thefocus of making both its data and model results efficiently available to organizations andresearch bodies interested in monitoring and predicting the dynamics of the coastal marineecosystem [6]. The Italian IOOS network is being designed to connect naturally intothe international IOOS framework. Such an infrastructure will help the understandingand forecasting of locally important issues such as the effects of severe meteo storms, theimplications of climate variability effects on global-regional scales, a quantitative riskassessment in coastal areas, etc.

Examples of stake-holders that will immediately benefit from simple and efficientaccess to large ocean dataset and model results are represented by, (a) companies involvedin the managing of marine coastal resources, including fisheries; (b) institutions dealingwith the management emergencies, including search and rescue and civil protectionactivities; (c) marine scientists; (d) ‘‘policymakers’’ at local, regional, national andinternational level; (e) recreational activities.

The IOOS approach is to standardize not on data formats, but on web services, andto approve certain web services for certain types of data. For gridded data, the approvedIOOS services are currently Open Geospatial Consortium (OGC) Web Coverage Service

42 A. Bergamasco et al.

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

(WCS) and the OPeNDAP service, in agreement with the Climate and Forecast (CF)convention [7].

This paper aims at highlighting the efforts that CNR-ISMAR Venice has recentlycarried out on this direction, discussing the basic ideas that have prompted theoceanographic and meteorological communities to the direction of contributing to‘‘knowledge discovery’’ by means of increasing the model data interoperability, as well asdata-model intercomparison and validation.

2. From the Common Data Model to the THREDDS Data Server

To unify the access to scientific data, we start from the basic idea of building a CommonData Model (CDM). Unidata (http://www.unidata.ucar.edu) an US government fundedorganization whose mission is to provide data services, tools and cyberinfrastructure forearth-system, proposed to create a common model for different ‘‘feature types’’ ofcommonly used scientific data (e.g. grids, profiles, time series, etc.). For each feature type,readers can then be constructed to translate data from many different formats on disk intoa common model in memory. An implementation of this conceptual model is given byUnidata CDM written in NetCDF-Java (http://www.unidata.ucar.edu/software/netcdf-java/CDM). The TDS, which utilizes NetCDF-Java, can for example read data intothe ‘‘grid’’ feature type from NetCDF3, NetCDF4, GRIB1, GRIB2, HDF4 and HDF5on disk.

Data already written according the CF convention can be directly read into the CDM,while other data can be adequately modified using the NetCDF Markup Language(NcML) via XML. The CDM additionally provides a standard API that can identify geo-referenced coordinate systems and queries specialized and oriented to commonly used datastructures in the earth science community.

The TDS is a middleware that bridges the gap between data providers and data users bydelivering standardized metadata and data in a variety of standard services. The servicesallow users to access and use data in a common and efficient way, allowing extraction ofjust the data they need, without downloading entire datasets. This is particularlyimportant as datasets from numerical simulations of the atmosphere and ocean aregrowing to hundreds of gigabytes, terabytes or even petabytes.

The adopted interoperability solution for CDM and TDS is shown in Figure 1.The final aim is that of making the use of scientific data simple, allowing a more

efficient exchange of scientific information that is of high interest in several sectors ofimportance. In addition to web services that deliver data, it is necessary to have webservices that allow users to find the data. The TDS ncISO service (ncISO is a package oftools that facilitates the generation of ISO 19115 metadata from NetCDF data sourcesstored in a TDS catalog) can be harvested by catalogue broker services [8] such as GI-CATGeoportal Server (see Figure 1a), and Geonetwork [9]. These catalogue services internallystore the gathered information in a metadata common data model (ISO 19115-2), whichin turn can be accessed using standardized queries for data discovery and access such asOpenSearch or OGC Catalog Services for the Web (CSW). This means the datasets alongwith their data services could easily be picked up and used by geosciences integrationefforts like US-IOOS, INSPIRE or GEOSS.

The brokering approach can further reduce the demand on data providers; by lettingthem provide any of a number of standard services, and independently providing the

Advances in Oceanography and Limnology 43

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

Figure 1. The adopted interoperability solution for CDM and TDS.Panel a: the user issues a query to a catalog broker service (like GI-CAT) that has harvestedmetadata from TDS servers, and stored them in a common data model (CDM) for metadata(in a database). The catalog broker service then returns the metadata for datasets that meet the user’squery constraints.Panel b: the CDM is contained within the TDS, that provides standard data and metadata services.The metadata contains links to the actual standard data services such as OPeNDAP, WCS, etc,so that the application the user can immediately start doing useful things with the results.

44 A. Bergamasco et al.

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

translation into and out of the common data model. While the catalogue services aremetadata brokers, the THREDDS Data Server is a data broker (Figure 1b), allowingmany different formats of files, as well as OPeNDAP service datasets, to be transformedinto a common data model for actual arrays of data.

In addition to allowing non-conforming data to be virtually transformed into acommon data model, the TDS has also another important characteristic that makes thingseasier for data providers and users: aggregation. This means that many individual files ondisk can be virtually joined into a single dataset accessible through the web services. Thusoceanic and atmospheric model output, as well as remote sensing data, which are typicallypresent on a file system as numerous smaller files, can be accessed via a single OPeNDAPor WCS URL. The TDS is simple to install, as it is 100% Java servlet typically deployedon Tomcat. A provider simply verifies that they have Sun Java, downloads and unzipsTomcat, and then deploys the thredds.war file through the Tomcat GUI, a process thattypically takes less than one hour, and sometimes as little as 10 minutes. Configuring theTDS for local datasets, of course, takes longer, but is still straightforward.

The CNR-ISMAR Venice catalogue http://tds.ve.ismar.cnr.it:8080/thredds/catalog.html represents one of the first Italian community examples taking the IOOSapproach. Once a user has selected archive dataset, a description of the dataset (metadata)appears, along with available web services and data viewers. At the moment, the CNR-ISMAR Venice catalogue contains datasets from several different implementations of thecoupled hydrodynamic-wave-sediment model ROMS (www.myroms.org). Datasets areregularly updated, and include cases from different geographic areas (e.g., the Adriaticsea and the Gulf of Lyon). For a thorough description of these test cases, see [1,10].

3. Advantages for the users

Once connected to a dataset on TDS, users can extract just the data they need usinga variety of methods. One popular method is to access data directly from MATLAB,a common tool for scientific analysis and visualization. By using the NCTOOLBOX forMATLAB (http://code.google.com/p/nctoolbox/) the user can extract the data, have themin his MATLAB workspace and easily plot them or perform a more in-depth analysis.Python users can use OPeNDAP access tools contained in packages such as NetCDF4-Python (http://code.google.com/p/netcdf4-python/).

Other users may be more interested in browsing the results than in working directlywith arrays of data. Personnel at the regional agencies of environmental protections(such as ARPA in Italy) may need to check the results of complex and integratednumerical models, for instance forecasting the wave-height or the currents regimes in frontof the Venice littoral zone. Instead of asking a research centre for dedicated productionor posting of these data, the user can connect directly to the TDS and quickly browse thedata using the viewer link for GODIVA2 [11]. GODIVA2 is a OGC Web Map Service(WMS) client that allows users to easily create maps or animations that rely on imagesgenerated TDS WMS service. This combination of quick browse and efficient access notonly saves time for the end user, but allows for more effective use of the data.

Figure 2 presents an example obtained from the above mentioned catalogue underthe ‘‘FIELD_AC test case’’, that stores outputs resulting from an implementation of thehydrodynamic numerical model ROMS coupled to the wave model SWAN in the AdriaticSea. This hindcast was carried out at 500m resolution spanning year 2007, with both

Advances in Oceanography and Limnology 45

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

models forced with high-resolution atmospheric forcing provided by the model COSMO-I7. Further details on the numerical implementations are given in [12].

Using GODIVA2 (which can be directly activated by clicking in the bottom area of themetadata page popping up), we can for instance visualize the field of potential temperatureat the model top level. Moreover, there is also the possibility of using embedded featuresthat takes care of exporting the results on GoogleTM Earth maps, as shown in Figure 3.When it is necessary to combine layers or create more complex visualizations, the user canaccess the data using the Unidata Integrated Data Viewer (IDV), freely available at (http://www.unidata.ucar.edu/software/idv/), as proposed in Figure 4.

Making things easier and more efficient for users can be particularly important duringemergency response situations. After the Fukushima incident, the US Navy rapidly spunup a 1 km NCOM forecast model covering hundreds of km offshore of the Sendai powerplant. The model forecasts were officially made public as NetCDF 3 files, one file foreach forecast time, and packed into 9GB tar.gz files delivered on an FTP site at NCEP.This made the forecast data effectively inaccessible to researchers at sea with limitedbandwidth. To facilitate use, the 9GB files were transferred to a THREDDS Data Server,converted to NetCDF4 (which resulted in a total file size the same as the tar.gz file), and

Figure 2. Using the quick and intuitive GODIVA2 web map service client, users can pick upthe ocean model variables they wish to visualize from a TDS catalog. After arranging it in termsof latitude, longitude, depth and time, GODIVA2 service allows for mapping, drawing sections andproducing time animations. Shown here, is the sea surface temperature from a high-resolution run(500m) of the northern Adriatic sea using ROMS-SWAN model, referring to July 3, 2007.

46 A. Bergamasco et al.

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

Figure 3. The same example shown in Figure 2, exported now to GoogleTM Earth mapping.

Figure 4. Using the open-source software IDV, more complex images can be arranged, such as this3D view of the northern Adriatic topography (in orange) with superimposed the sea surfacetemperature field from a high-resolution run (500m) of the northern Adriatic sea using ROMS-SWAN model, referring to May 28, 2007. The figure also shows averaged (2D) velocity fields (blackarrows, plotted every 10th grid points) and contours of significant wave height (m).

Advances in Oceanography and Limnology 47

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

virtually turned into a single CF Compliant dataset available via the TDS, allowingdistribution through OPeNDAP, WCS, and WMS services. This allowed efficientsub setting and extraction by WHOI researchers at sea, who used the forecast data topredict the movement of radioactive material they identified in surface water samples(see Figure 5). The metadata service also allowed others searching for Fukushima productsto effectively locate this new datasets once it came on line [13].

Figure 5. Using NCTOOLBOX to access data from the TDS via OPeNDAP, displaying surfacecurrent vectors and speed for the Fukushima region. On the bottom is shown the actual scriptin Matlab that acquires and subsamples the data and produces the plot on the top.

48 A. Bergamasco et al.

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

4. Conclusions and recommendations

Following the US IOOS approach, the Italian community is advancing toward a‘‘Euro-Mediterranean IOOS’’. Taking an approach focused on minimizing the burden forboth providers and consumers of data, great strides are being made to overcome theexisting bottleneck in the distribution and use of observational and modelled ocean data.

The TDS represents a free and supported solution that is now allowing data providers(e.g. numerical modellers) to serve the data they produced, with no need of modification,via standard web services. Since the TDS metadata service can be harvested by cataloguebrokering systems, data users can easily access the standardized data by using standardizedqueries for data discovery (e.g. OpenSearch, OGC Catalog Services, etc.). After decidingwhich time/space portion has to be transferred, they can select among a variety of tools,including 3D open-source viewers, to promptly visualize or carefully examine them.

The success of this approach does require some expert knowledge on the part of theperson configuring the TDS. In particular, the standardization and aggregation of existingfiles requires learning the NcML language and understanding the Common Data Modelrequirements. However, this configuration is mostly a one-time effort, so that an expertcan assist in the initial implementation, and then local instances of the TDS can bemaintained by personnel without this detailed knowledge.

To continue to build on this success, we need to address a few challenges. While thisapproach works well for structured grids, standards are just now being introduced forunstructured grids. Standard handling of staggered grid information and velocity vectorsneeds to be improved. And finally, issues related to management and dissemination ofpublic data (i.e. an adequate data policy) can be faced when unlocking large modeloutputs.

The brokering approach to harvest metadata from many different services and to readdata from many different formats into common data models greatly improves modelaccess and interoperability, unlocking information from other fields (e.g., social andeconomic studies). These are very desirable properties in the direction of a ‘‘INSPIREcompliant web service’’ (see also http://inspire.jrc.ec.europa.eu), since they contribute tolower the so-called ‘‘Users and Data Producers entry barriers’’ [8].

Acknowledgements

The authors thank Unidata for the technical support and help. This work was supported by theProject ‘‘MARINA’’, funded by Regione Veneto within the initiatives of the law n. 15/2007. Theactivity was partially supported by Projects PRIN 2008YNPNT9_005 and FIRB ‘‘DECALOGO’’(code #RBFR08D825) and by the Project FIELD_AC, funded by the EC Fp7/2007–2013 undergrant agreement no. 242284.

References

[1] S. Carniel, M. Sclavo, and R. Archetti, Towards validating a last generation, integrated wave-current-sediment numerical model in coastal regions using video measurements, Oceanological andHydrobiol. Studies 40 (2011), pp. 11–20, DOI: 10.2478/s13545-011-0036-1.

[2] D. Bellafiore, E. Bucchignani, S. Gualdi, S. Carniel, V. Djurdjevic, and G. Umgiesser, Assessmentof meteorological climate model inputs for coastal hydrodynamics modeling. Ocean Dyn. 62 (2012),pp. 555–568, DOI: 10.1007/s10236-011-0508-2.

Advances in Oceanography and Limnology 49

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012

[3] R.P. Signell, S. Carniel, J. Chiggiato, I. Janecovic, J. Pullen, and C. Sherwood, Collaborationtools and techniques for large model datasets, J. Marine Sys. 65 (2008), pp. 154–161, DOI:10.1016/j.jmarsys.2007.02.013.

[4] R.P. Signell, Model data interoperability for the United States Integrated Ocean ObservingSystem (IOOS), Proceedings of the 11th International Conference on Estuarine and CoastalModeling, Seattle, WA, USA, 2010. DOI:10.1061/41121(388)14.

[5] S. Rayner, The U.S. Integrated Ocean Observing System in a global context, Marine Tech. Soc. J.44 (2010), pp. 26–31, DOI:10.4031/MTSJ.44.6.1.

[6] A. Bergamasco, S. Carniel, M. Sclavo, and T. Minuzzo, From interoperability to knowledgediscovery using large model datasets in the marine environment: the THREDDS Data Serverexample. Data Flow from Space to Earth 2011 International Conference, Venice, 21–23 March2011. Available at http://www.space.corila.it/Program.htm

[7] J. de La Beaujardiere, C.J. Beegle-Krause, L. Bermudez, S. Hankin, L. Hazard, E. Howlett,S. Le, R. Proctor, R.P. Signell, D. Snowden, and J. Thomas, Ocean and coastal datamanagement, Proc. OceanObs’09: Sustained Ocean Observations and Information for Society(Vol. 2), Venice, Italy, 21–25 September 2009.

[8] S. Nativi, S. Khalsa, B. Domenico, M. Craglia, J. Pearlman, P. Mazzetti, and R. Rew,The Brokering approach for Earth Science Cyberinfrastructure. EarthCube White Paper,Oct 2011. Available at http://earthcube.ning.com/page/whitepapers.

[9] S. Nativi, S. Bigagli, P. Mazzetti, E. Boldrini, and F. Papeschi, GI-cat: a mediation solutionfor building a clearinghouse catalo service. Advanced Geographic Information Systems & WebServices, 2009. GEOWS’09. DOI: 10.1109/GEOWS.2009.34.

[10] A. Boldrin, S. Carniel, M. Giani, M. Marini, F. Bernardi Aubry, A. Campanelli, F. Grilli, andA. Russo, The effect of Bora wind on physical and bio-chemical properties of stratified waters inthe Northern Adriatic, J. Geophys. Res. – Ocean 114 (2009), p. C08S92, DOI: 10.1029/2008JC004837.

[11] J.D. Blower, K. Haines, A. Santokhee, and C.L. Liu, GODIVA2: interactive visualization ofenvironmental data on the Web, Phil. Trans. R. Soc. A 367 (1890) (2009), pp. 1035–1039, DOI:10.1098/rsta.2008.0180.

[12] A. Benetazzo, A. Bergamasco, S. Carniel, A. Russo, and M. Sclavo, CNR-ISMAR contributionin the ‘‘FIELD_AC’’ Project: the Gulf of Venice study case. Data Flow from Space to Earth 2011International Conference, Venice, 21–23 March 2011. Available at http://www.space.corila.it/Program.htm

[13] D. Neufeld and R.P. Signell, Case study Fukushima: open source data discovery in disastermanagement, FOSS4G Conference, Denver, Co, USA, 2011. Available at http://2011.foss4g.org/sessions/case-study-fukushima-open-source-data-discovery-disaster-management

50 A. Bergamasco et al.

Dow

nloa

ded

by [M

r And

rea

Berg

amas

co] a

t 14:

21 2

9 M

ay 2

012