Numerical Grid Computations with the OPeNDAP Back End

Server (BES)

• Two part problem; 1. Compute the so called Stokes vector for a given vector parameter

space P using a model F(x,y,λ; P)2. Correct P to improve the fit of the theoretical signal to the observed

signal using non-linear optimization

• The purpose is to determine a vector parameter space for which the model produces the best fit in the Least Square sense.

• The complete problem is solved with a Numerical Library Called MERLIN

• Very computational Intensive, one complete dataset can take hours on a single node.

The Problem

High Performance Solution

• Create multiple versions of the computing node waiting for data to crunch

• Create a central place (controller) to browse spectra- polarimetry data inventories and launch inversions upon user’s request.

• Let the controller decide how to distribute the problem.

• Requires multiple identical processes waiting to be called for duty.

Born to Grid

• Hyrax BES was designed from the beginning with grid services in mind

• Suited for problems that DO NOT REQUIRE coordination of the points after each iteration (see next slide)

• Each point is solved individually regardless of the solution to all other points.

What this problem is not• Do not think of a problem like the Navier-Stokes equation

using a cluster and finite element methods with mesh refinement.

• Think Scatter-Gather or Embarrassing Parallel . The Spectral Inversion Problem does not require boundary information to be exchanged in the solution.

The Dataset: Spectral ImagesSingle element from data cube

Y

X

λ

A slice is nice!

Each node gets a Y time λ slice

Each computational node, that is, each BES gets from the controller an OPeNDAP URL specifying the dataset on which to operate and the specific slice using OPeNDAP constraints.Each computational node returns the solution to the controller as an OPeNDAP data object.Controller collects all solutions and assembles a complete result as a netCDF data file.Time to compute solution scales LINEARLY with number of nodesGrid can be homogeneous or heterogeneous.

We do not call it scatter-gather for nothing

BES Server 1

BES Server 2

.

.

.

BES Server n

GRID

Inversion Engine

Data local or remoteusing OPeNDAP over HTTP

Client

Each server computes the inversion over spectral images (y), all n servers compute the complete set (x,y),

HOW SLICING WORKS IN THE INVERSION ENGINE

APACHE+

MODULE

OUTPUT INVERSION RESULTS

InversionEngine

MERLIN

ANCILLARY DATA:Instrument ProfileDispersion Spectral Bandpass(es)Atomic Data for Spectral Lines

Genetic Algorithm orNeural Network provides

Initial Guess

Data Source

Process

HYRAX BES RUNNING MERLIN AS A MODULE

CALIBRATED INPUT DATA (e.g. ASP, DLSP, HINODE, ...)

Process with Custom Data Filters

to Standard Data Engine Format

Polarization Reference Frame: +Q along Solar E-W

Remote OPeNDAP server

Conclusions• Hyrax BES can dynamically load modules that not only do data serving and manipulation, but server side functionality as well.• The use of multiple servers running Hyrax can easily be implemented to linearly improve the performance. This applies to both server side processing as well as load balancing for serving data in demanding systems like the Earth System Grid.• Hyrax Modules offer a simple way to extend the functionality of your server, as well as seamlessly integrate multiple facets of data serving and processing.

QUESTIONS?