VisPortal Project developer’s experience C.E.Siegerist, J. Shalf, E.W. Bethel NERSC/LBNL...

VisPortal Projectdeveloper’s experience

C.E.Siegerist, J. Shalf, E.W. BethelNERSC/LBNL Visualization Group

T.J. Jankun-Kelley, O. Kreylos, K.L. MaCIPIC/UC Davis

Overview• Grid Introduction • Portal Architecture/Software

components. • Visportal Services/Example Pages• User’s perspective• Conclusions/Future/Caveats• References

The Grid• The Grid is an abstraction that allows

transparent use of distributed computing resources.

• The problem that underlies the Grid concept is coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations.

VisPortal Project• The LBNL/NERSC VisPortal effort explores

ways to deliver Grid-based advanced visualization and data analysis capabilities through a Web Portal interface.

• Using standard Globus Grid middleware and off the shelf web automation, the VisPortal hides the underlying complexity of resource selection and distributed application management.

VisPortal ServicesMotivation: simplify remote visualization, offer point an click solutions for complex distributed applications

• File Management• Vis Applications: Visapult, AMR Renderer,

Web VisSheet.• One button Mpeg• …

Globus• The Globus Project provides software

tools (Globus toolkit) that make it easier to build computational grids and grid-based applications.

Globusrun Example of a file conversion RSL script

% globusrun –r escher.nersc.gov –f convert.rsl -w

& (executable=$GLOBUSRUN_GASS_URL # “/home/cristina/convertscript”)

(enviroment=(GLOBUSRUN_GASS_URL $(GLOBUS_GASS_URL))

#!/sbin/sh GLOBUS_LOCATION=/usr/local/globusexport GLOBUS_LOCATIONPATH=$GLOBUS_LOCATION/bin:$GLOBUS_LOCATION/sbinexport PATHLD_LIBRARY_PATH=$GLOBUS_LOCATION/libexport LD_LIBRARY_PATHIMAGE_MAGICK=/usr/freeware/bin export IMAGE_MAGICKglobus-gass-server -t -u -r -w -c > $mytmpgassfile &sleep 3set remote_gass_server_url=`cat $mytmpgassfile`rm $mytmpgassfile $GLOBUS_LOCATION/bin/globus-url-copy $GLOBUSRUN_GASS_URL/home/cristina/file.jpg file:/tmp/file.jpg$IMAGE_MAGICK/convert /tmp/file.jpg /tmp/file.bmp$GLOBUS_LOCATION/bin/globus-url-copy file:/tmp/file.bmp $GLOBUSRUN_GASS_URL/home/cristina/file.bmpglobus-gass-server-shutdown $remote_gass_server_url rm /tmp/file.bmprm /tmp/file.jpg

GPDK• The Grid Portal Development Kit (GPDK)

is a set of modular, reusable components for accessing Grid services in the form of Java Beans. It derives most of its functionality from the Globus Java Commodity Grid (Java CoG) toolkit.

• Other toolkits: GridPort, GridSphere

Portal ArchitectureHPC

Resources

WebPortal

Parallel VisualAnalysis Server

User client 1

Archival Storage

& Data Cache

User client n

User site(s)HPC Facility

Audio& video

DataGRAMGSIMDS

GRAMGSI

GSIFTP

httpsSOAPGridFTP

CoG

Portal Architecture (thick client)

HPCResources

WebPortal

Parallel VisualAnalysis Server

User client 1

Archival Storage

& Data Cache

User client n

User site(s)HPC Facility

Audio& video

DataGRAMGSIMDS

GRAMGSI

GSIFTP

httpsSOAPGridFTP

CoG

Thick Client

Resources | Users

Internal ArchitectureWeb Server

Tomcat

GPDKPortal Engine

(Servlets)Application Logic Presentation

(JSP)

Grid Service BeansSecurity Job

SubmissionInformation

ServicesData Transfer

User Profiles

Grid Middleware Libraries(e.g. CoG)

globus

Software Components

• Apache 1.3.27• Tomcat 3.3.1• gpdk cvs version• Java CoG v1.0a• mod_jk 1.3

• mod_ssl 2.8.12

• openssl 0.9.6h • globus 2.x• java 1.4.x• MyProxy 1.0• log4j_1.2.7

GPDKDemo Portal

Login Page

• Myproxy: username and password of delegated credential

• Local: load from credential file (grid-proxy-init)

• Update user’sresourcesjob historyemail address

Profile

File Mgmt.

• Foreground transfers

• Background transfers

• Globus File Yanker(Shreyas Cholia)

Launch ApplicationsVisapult

Launch ApplicationsAMR Renderer

Visportal WebSheet

MPEG Generator

• Example of a user driven service: the user connects to our Onyx to run an IDL batch job to produce a movie with data that he has in an IBM SP • The portal provides an interface to select the data file and the mpeg parameters. It transfers the file, launches the IDL job and sends an email to the user with a URL to retrieve the movie.

The portal is running (most of the time)

https://frost.lbl.gov/visportal/servlet/visportal

User’s perspective• Accounts in all the resources.

• Request a user certificate: http://www.doegrids.org/pages/cert-request.html

• Be in the grid-map files of the resources (automated through NIM).

• Create a proxy certificate and login.

Challenges

• Difficult GUI Implementation• Error Recovery• Support• Middleware is a moving target• Resource management

Future• Deploying a subset of services to

NERSC users.• Implementing specific user

workflows. • Upgrade to OGSA/GT3.0• Explore Gridlab/Gridsphere,

portlets.

Soon:visportal.nersc.gov

References• The anatomy of the Grid: Enabling Scalable Virtual Organizations, Intl. J.

Supercomputer Applications, 2001.• The Grid Portal Development Kit, J.Novotny, Cuncurrency: Pract. Exper.

2000; 00:1-7• How the Grid will affect the Architecure of Future Visualization Systems,

J.Shalf and W. Bethel, http://www-vis.lbl.gov/Publications• Building a Portal Using GPDK: A Developers Tutorial, J. Novotny,

http://doesciencegrid.org//public/events/GPDW/slides/gpdk-dev.pdf• Globus, A Metacomputing Infrastructure Toolkit. I. Foster, C.

Kessselman, Internation Journal of Supercomputing applications, 1997.• Deploying Web-based Visual Exploratin Tools on the Grid. T.J.Jankun-

Kelly, O. Kreylos, J. Shalf, K-L. Ma, B. Hammann, K. Joy, E.W. Bethel. IEEE Computer Graphics and Applications, march/april 2003.

• VisPortal, Deploying grid-enabled visualization tools through a web-portal interface. http://www-library.lbl.gov/library/docs/LBNL/529/40/PDF/LBNL-52940.pdf

• Globus: www.globus.org• GPDK: www.doesciencegrid.org