OPeNDAP HyraxBack-End Server (BES)

Customization

ESIP Federation Workshop

15 July 2008Patrick West

James Gallagher

OPeNDAP’s Hyrax Architecture

OLFS

Java Servlet Engine

BES

Unix Daemon

BES Commands

XML- encapsulated object

File system with data files,SQL Database, …

DAP2

THREDDS

HTML

Optional THREDDScatalogs

Module

Module

Module

Module

In what ways can you extend the BES?• Extensions are dynamically loaded from a shared object

module

• Containers and Definitions, and ways of storing them

• Dataset Catalogs

• Interface

• Initialization/Termination

• New commands (like our hello world example)

• New response objects

• New response handlers

• New request handlers (data handlers like hdf4, hdf5, …)

• Aggregation engines

• Methods of returning your data (return as netcdf)

• Reporters

• Exception Handlers

• Debugging

Modules are dynamically loaded into the BES

• Within the BES configuration you specify the modules that you want to load into the BES.

• The module is derived from the C++ abstract class BESAbstractModule.

• The class impelements a C-style function called maker that instantiates the module class

• The class implements an initialization and termination method to set up and clean up the module.

• The module can add named handlers to the different pieces of the BES.

• The module can extend already existing modules, for example extending the DAP2 module.

• The module can modify already existing modules, such as changing the way certain responses are built.

BES Configuration File/usr/local/etc/bes/bes.conf

Running what we have• Let’s see what we have already installed

• besctl start (if it’s not already running)

• bescmdln -h localhost -p 10002

• show help;

• show version;

• show keys;

• set container in catalog values c,data/nc/fnoc1.nc;

• define d as c;

• get dds for d;

• define d as c with c.constraint=“u”;

• get dds for d;

• get ascii for d;

Or use the OLFS

• ${CATALINA_HOME}/bin/startup.sh• Start your browser• Localhost:8080/opendap/• Select data, then nc, then fnoc1.nc• Click the check box next to the variable u• Click the button ‘Get ASCII’

The CSV Data Handler Module

• /root/src/bes-3.6.2/csv-handler• …more ‘data/temperature.csv’

"Station<String>","latitude<Float32>","longitude<Float32>","temperature_K<Float32>","Notes<String>”"CMWM",-34.7,23.7,264.3,"BWWJ",-34.2,21.5,262.1,"Foo”"CWQK",-32.7,22.3,268.4,"CRLM",-33.8,22.1,270.2,"Blah”"FOOB",-32.9,23.4,269.69,"FOOBAR"

Setting up the example

• cd /root/src/bes-3.6.2/csv-handler• autoreconf• ./configure• make install

– This will build and install the module library and the sample data

• make bes-config– This will make the changes to the bes.conf

file

BES Configuration File/usr/local/etc/bes/bes.conf

Restart it all

• ${CATALINA_HOME}/bin/shutdown.sh• besctl stop• besctl start• ${CATALINA_HOME}/bin/startup.sh• Back in the browser

– localhost:8080/opendap/– Select data, then csv, then

temperature.csv– Click the button ‘Get ASCII’

CSVModule class

• Example Module class:– see /root/src/bes-3.6.2/csv-handler/CSVModule.cc

• Implements initialize and terminate methods (and dump)

• Adds: (typical for data handlers)– new request handler– new container– new catalog

Containers• A container represents data

– Typically a data file– Can represent a request to a remote system,

such as WCS

• Has three main parts:1. Symbolic name (‘cool_data’)2. Real name (e.g., a filename)3. Type (nc, ff, csv, ...)

• matches the name of the data/request handler

• Derive from the abstraction BESContainer• The BESFileContainer is built into the BES• You might add ... WCSContainer

Accessing the container

• Implement the access method:

virtual string access() ;

• Returns the data container to be accessed, typically full path to a file.– BESFileContainer returns the stored file name– A WCSContainer might make a WCS request to

the remote server, store the resulting data file, and returns the path to that resulting, cached, data file.

Definitions• Analogous to a view of data in a RDB• Can encompass many containers (available soon)

• Can specify constraints for each container• Can specify data attributes from each container• Specify aggregation engine and command

define d as <c1>[,…,<cn>] [with <c1>.constraint=“…”,<c1>.attributes=“…”, <cn>.constraint=“…”,<cn>.attributes=“…” aggregate using <engine> by <command>];

Dataset Catalogs• OLFS uses BES and THREDDS catalogs for

dataset catalog traversal• Can have multiple catalogs in a single BES

representing any type of catalog (coming soon)

– Filesystem– Database catalog

• Each Catalog is configurable• Catalogs inherit from the abstraction

BESCatalog• Add to the list of catalogs in the module

class:BESCatalogList::TheCatalogList()->add_catalog( catalog ) ;

Typical Catalog Configuration

• Data handlers, e.g. csv-handler, add a BESCatalogDirectory representing a file system.

• In the BES configuration file:– BES.Catalog.catalog.RootDirectory - data root

directory of the catalog named “catalog”– BES.Catalog.catalog.TypeMatch - data types

matched by file extensions– BES.Catalog.catalog.Include - what nodes to

include in the catalog list (usually everything)– BES.Catalog.catalog.Exclude - what nodes to

exclude (e.g. dot files)• Let’s look at /usr/local/etc/bes/bes.conf

Catalogs and Containers

• Can create containers within the context of catalogs using BESContainerStorageCatalog

• Uses BES.Catalog.catalog.TypeMatch to determine data type.

set container in catalog values c,<catalog_file>;

Request Handlers

• Derived from BESRequestHandler• Fills in response objects• Register functions that say “I know how

to fill in some response”

Registering a Request Handler

• Inside the Module class the request handler is registered with the BES

BESRequestHandlerList::TheList()->add_handler( name, handler )

• CSVModule::initialize()

BESRequestHandlerList::TheList()->

add_handler( modname,

new CSVRequestHandler( modname ) ) ;

I know how

• Inside the constructor for the request handler you register functions that know how to fill in responses

add_handler( “name”, function )• CSVRequestHandler::CSVRequestHandler

add_handler( DAS_RESPONSE, CSVRequestHandler::csv_build_das ) ;

add_handler( DDS_RESPONSE, CSVRequestHandler::csv_build_dds ) ;

add_handler( DATA_RESPONSE, CSVRequestHandler::csv_build_data ) ;

add_handler( VERS_RESPONSE, CSVRequestHandler::csv_build_vers ) ;

add_handler( HELP_RESPONSE, CSVRequestHandler::csv_build_help ) ;

How does it work

Definitions def as c1, c2

get das for def

Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ffdef

def

command

Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler

Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func

das

das

csv

csv

das

c1

das

c1

c1

c2

c2c2

das

show help

help

help

help

csv

csv

Putting it together so far

Summary: CSV Data Handler Module

• Look at CSVModule– Adds CSVRequestHandler– Adds BESCatalogDirectory– Adds BESContainerStorageCatalog

• Look at CSVRequestHandler– Adds functions to fill in DAS, DDS, DataDDS,

Version and Help responses

• These functions fill in the respective response objects

25

More information…

• The Following slides provide more information on BES– Interfaces– Callbacks– Request plans– Commands– Response objects– Response handlers

The Interface• There is a single interface into the BES.• Inherits from the abstraction BESInterface

– Installed version uses BESCmdInterface– CEDAR uses BESApacheInterface– Creating BESXMLInterface

• Interface runs a set of steps:– initialize the BES environment– build the execution plan– execute the request plan building the response object– transmit the response object– log the status of the request– report on the request– end the request

BESDataHandlerInterface

• BESDataHandlerInterface structure is carried throughout the execution of the commands– Holds on to the response handler– Stores the list of containers– Holds the action being taken– Holds map of name/value pairs to be used during

execution– Holds an error object if an error/exception occurs

Init/End callbacks

• Can register callbacks during the initialization and end steps

static void BESInterface::add_init_callback( p_bes_init init ) ;

static void BESInterface::add_end_callback( p_bes_end end )

typedef bool (*p_bes_init)( BESDataHandlerInterface &dhi ) ;

typedef void (*p_bes_end)( BESDataHandlerInterface &dhi ) ;

• Examples:– Authentication/authorization

– Initialize database connections

– Clean up database connections and files

Building the request plan

• Derived classes of BESInterface implement:virtual void build_data_request_plan() ;

– BESCmdInterface parses the incoming command string

– BESApacheInterface translates information from apache and parses the incoming command string

– BESXMLInterface parses the XML document

• Creates a response handler to execute the command

Commands• String commands sent from client to server• Built in commands:

– show help; (version, process, status, keys)– set container …– show containers;– delete container <container_name>;– delete containers;– define …– show definitions;– delete definition <def_name>;– delete definitions;– set context <name> to <value>;– show context;

DAP Commands• DAPCommandModule adds:

– get das for <def_name> [return as type];– get dds …– get ddx …– get dods …– show catalog [for “node”]; (info)

• dap-server modules (www,usage,ascii) add:– get info_page …– get ascii …– get html_form …

• Data handlers (nc, ff, csv) don’t add any new commands.

Response Objects

• Derived from BESResponseObject.– BESDASResponseObject– BESInfo

• No methods required to be implemented

• Created by a BESResponseHandler• Filled in by BESResponseHandler or

delegated to BESRequestHandler

Informational Response Objects• Built in informational response objects derived from

BESInfo:1. BESTextInfo

2. BESHTMLInfo

3. BESXMLInfo

4. BESSilentInfo

• Each one formats an informational response according to its type

• Can add new informational response objects.BESInfoList::TheList()->add_info_builder( name, function ) ;

1. Function instantiates the derived BESInfo object.

2. Created in BESResponseHandler instances by calling:BESInfo *info = BESInfoList::TheList()->build_info() ;

1. Set in BES configuration file:

BES.Info.Type=<name> e.g. txt, html, xml

Response Handlers:

• represent a specific response, such as a DAS response, a DDS response, a help response ...

• know how to create the response object (DAS, DDS, BESInfo)

• do not necessarily fill in the response object, but know how the response should be filled in

• BESDASResponseHandler knows to create a DAS object and that it needs to go to each request handler for each container.

• A version response handler knows to create an informational response object and that it needs to go to all registered request handlers.

• A status response handler knows to create a text response object and fill it in with the status of the server.

• know how to transmit the response object

Implementing a Response Handler

• Inherits from the C++ abstract class BESResponseHandler (e.g. BESDASResponseHandler, BESHelpResponseHandler)

• Implement the methods:

virtual void execute( BESDataHandlerInterface &dhi ) ;

– Creates the response object– Does the work to fill in the response object, or delegate to a

request handler to fill in the response object

virtual void transmit( BESTransmitter *transmitter, BESDataHandlerInterface &dhi ) ;

– Transmits the resulting response object using the appropriate method on the transmitter (more later)